When using atom-centered integration grids, the portion of the grid that belongs to a certain atom also moves when this atom is displaced. In the paper, we investigate the moving-grid effect in the calculation of the harmonic vibrational frequencies when using all-electron full-potential numeric atomic-centered orbitals as the basis set. We find that, unlike the first-order derivative (i.e., forces), the moving-grid effect plays an essential role for the second-order derivatives (i.e., vibrational frequencies). Further analysis reveals that the predominantly diagonal force constant terms are affected, which can be bypassed efficiently by invoking translational symmetry. Our approaches have been demonstrated in both finite (molecules) and extended (periodic) systems.