The interplay between magnetism and crystal structures in three CaFe2As2 samples is studied. For the nonmagnetic quenched crystals, different crystalline domains with varying lattice parameters are found, and three phases (orthorhombic, tetragonal, and collapsed tetragonal) coexist between TS=95 K and 45 K. Annealing of the quenched crystals at 350°C leads to a strain relief through a large (~1.3%) expansion of the c-parameter and a small (~0.2%) contraction of the a-parameter, and to local ~0.2 Å displacements at the atomic-level. This annealing procedure results in the most homogeneous crystals for which the antiferromagnetic and orthorhombic phase transitions occur at TN/TS=168(1) K. In the 700°C-annealed crystal, an intermediate strain regime takes place, with tetragonal and orthorhombic structural phases coexisting between 80 to 120 K. The origin of such strong shifts in the transition temperatures are tied to structural parameters. Importantly, with annealing, an increase in the Fe-As length leads to more localized Fe electrons and higher local magnetic moments on Fe ions. Synergistic contribution of other structural parameters, including a decrease in the Fe-Fe distance, and a dramatic increase of the c-parameter, which enhances the Fermi surface nesting in CaFe2As2, are also discussed.