Modeling and analysis of MR images of the early developing human brain is a challenge because of the transient nature of different tissue classes during brain growth. To address this issue, a statistical model that can capture the spatial variation of structures over time is needed. Here, we present an approach to building a spatio-temporal model of tissue distribution in the developing brain which can incorporate both developed tissues as well as transient tissue classes such as the germinal matrix by using constrained higher order polynomial models. This spatiotemporal model is created from a set of manual segmentations through groupwise registration and voxelwise non-linear modeling of tissue class membership, that allows us to represent the appearance as well as disappearance of the transient brain structures over time. Applying this model to atlas-based segmentation, we generate age-specific tissue probability maps and use them to initialize an EM segmentation of the fetal brain tissues. The approach is evaluated using clinical MR images of young fetuses with gestational ages ranging from 20.57 to 24.71 weeks. Results indicate improvement in performance of atlas-based EM segmentation provided by higher order temporal models that capture the variation of tissue occurrence over time.