A solid with larger sound speeds usually exhibits higher lattice thermal conductivity. Here, we report an exception that CuP2 has a quite large mean sound speed of 4155 m s-1, comparable to GaAs, but single crystals show very low lattice thermal conductivity of about 4 W m-1 K-1 at room temperature, one order of magnitude smaller than GaAs. To understand such a puzzling thermal transport behavior, we have thoroughly investigated the atomic structures and lattice dynamics by combining neutron scattering techniques with first-principles simulations. This compound crystallizes in a layered structure where Cu atoms forming dimers are sandwiched in between P atomic networks. In this work, we reveal that Cu atomic dimers vibrate as a rattling mode with frequency around 11 meV, which is manifested to be remarkably anharmonic and strongly scatters acoustic phonons to achieve the low lattice thermal conductivity.