We determine the limits of stability of the homogeneous state of a pattern forming optical system in dependency on the wave number by experimental means. The measurement becomes feasible by adopting a scheme based on a Fourier filtering technique. The system under study is a single-mirror feedback arrangement using sodium vapor as the nonlinear medium. The experiment confirms the existence of multiple instability regions of the homogeneous state expected by theory. The measurements do not agree quantitatively with the marginal stability curve determined by a linear stability analysis of an infinitely extended homogeneous system. We study the system numerically and demonstrate that the results of the simulations for the case of a Gaussian beam can be reproduced by a simple modification of the linear stability analysis which accounts for the finite diameter of the input beam. This explains the wave number dependent systematic deviations between the experiment and the linear stability analysis of the infinitely extended system.