Phosphate (Pi) is essential to plant growth and crop yield. However, it remains unknown how Pi homeostasis is maintained during cereal grain filling. Here, we identified a rice grain-filling-controlling PHO1-type Pi transporter, OsPHO1;2, through map-based cloning. Pi efflux activity and its localization to the plasma membrane of seed tissues implicated a specific role for OsPHO1;2 in Pi reallocation during grain filling. Indeed, Pi over-accumulated in developing seeds of the Ospho1;2 mutant, which inhibited the activity of ADP-glucose pyrophosphorylase (AGPase), important for starch synthesis, and the grain-filling defect was alleviated by overexpression of AGPase in Ospho1;2-mutant plants. A conserved function was recognized for the maize transporter ZmPHO1;2. Importantly, ectopic overexpression of OsPHO1;2 enhanced grain yield, especially under low-Pi conditions. Collectively, we discovered a mechanism underlying Pi transport, grain filling and P-use efficiency, providing an efficient strategy for improving grain yield with minimal P-fertilizer input in cereals.