Polarization sensitivity (PS) in vertebrate vision is controversial, perhaps because its underlying mechanism has remained obscure. An issue that might have added to the controversy is that rainbow trout (Oncorhynchus mykiss), which have served as the primary model system for polarization-based orientation, lose their ability to orient relative to celestial polarized-light patterns when parr (fry) transform into migratory smolts (juveniles), which would benefit most from polarization-based orientation. Here we addressed two key questions: (1) what is the mechanism underling PS?, and (2) how can the paradoxical loss of PS in trout smolts be reconciled? We assessed PS from optic nerve recordings in parr and smolts and found that the retinal region with enhanced PS shifted from the ventral retina in parr to the dorsal retina in smolts. This adaptation may allow fish to use the most reliable polarization field encountered at each life stage, the celestial polarization field in the shallow-swimming parr and the depth-insensitive underwater polarization field in the deep-swimming smolts. In addition, we assessed spectral sensitivity across the retina and during ontogeny and fit a cascade retinal model to PS data. We found that differential contribution of two cone detectors with orthogonal PS could drive the variation in PS and that feedback from horizontal cells to cones could explain the differential amplification of PS. This elegant arrangement, in which weak PS of cones is amplified and tuned by retinal networks, allows for PS without interfering with sampling of other visual information and illustrates how sensory systems may simultaneously process disparate aspects of physical environments.