Ultrafast charge carrier dynamics in graphite has been investigated by time-resolved terahertz spectroscopy. Analysis of the transient dielectric function and model calculations show that more than 90% of the initially deposited excitation energy is transferred to a few strongly coupled lattice vibrations within 500 fs. These hot optical phonons also substantially contribute to the striking increase of the Drude relaxation rate observed during the first picosecond after photoexcitation. The subsequent cooling of the hot phonons yields a lifetime estimate of 7 ps for these modes.