Drying of aqueous polymer solutions is widely used to form polymer films. However, we do not fully understand how concentrated polymer near the drying interfaces decreases the rate of water evaporation. In addition, we do not know how the evaporation kinetics varies when we use different polymers. To understand these topics, we examined drying of polymer solutions with three different polymers, poly(vinyl alcohol) (PVA), poly(vinyl pyrrolidone), and poly(ethylene glycol) in unidirectional drying cells. In each case, the water evaporation rate (J) decreased to 1/10 of the initial rate during evaporation. J-1 was proportional to the amount of polymer transported to the drying interface during the drying time. The slope is defined as resistance factor A and the PVA solutions had larger A values than the other polymers, indicating that J decreased more noticeably in the PVA solutions. The molecular weights of the polymers did not explain A. Concentration profiles of polymers near the drying interfaces were quantified in situ with an optical microscope by exploiting differential interference contrast, from which we estimated the diffusion constants (Ds) of the polymers in the drying solutions. We found that A was negatively correlated with D. This result clearly indicates that the diffusivity and accumulation of dissolved polymer molecules near the drying interface governs the overall drying kinetics of their solutions.