We discuss retrieval of the phase of quantum-mechanical and classical wave fields in the presence of first-order vortices. A practical method of phase retrieval is demonstrated which is robust in the presence of noise. Conditions for the uniqueness of the retrieved phase are discussed and we show that determination of the phase in a given plane requires a series of at least three two-dimensional intensity images at different propagation distances. The method is applicable to a wide range of scenarios such as the imaging of imperfect crystals, quantitative determination of the strength of vortex filaments in high-temperature superconductors, and x-ray and electron holography.