Biomarkers are increasingly used in clinical and epidemiologic studies. Prior to these studies, small pilot studies are often conducted to assess the reproducibility of the biomarker. This article discusses how the results of a pilot study can be used to design subsequent studies when the biomarker is a binary assessment. We consider situations in which the pilot study has two factors (e.g., laboratory and individual) that are either crossed or nested. We discuss how binary random-effects models can be used for estimating the sources of variation and how parameter estimates from these models can be used to appropriately design future studies. We also show that fitting a linear variance components model that ignores the binary nature of the data is a simple alternative method that results in nearly unbiased and moderately efficient estimators of important design parameters. We illustrate the methodology with data from a study assessing the reproducibility of p53 immunohistochemistry in bladder tumors.