The magnetoelectric effect in the RX3(BO3)4 system (R = Ho, Eu, Sm, Nd, Gd; X = Fe, Al) varies significantly with the cation R despite very similar structural arrangements. Our structural studies reveal a symmetry reducing tilting of the BO3 planes and of the FeO6 polyhedra in the systems exhibiting low magnetic field induced electric polarization. Neutron scattering measurements reveal a lack of magnetic ordering indicating the primary importance of the atomic structure in the multiferroic behavior of this system.