The purpose of the present study was to determine whether the amount and the location of residual tumor are associated with outcome in surgically treated rectal cancer patients who receive preoperative chemoradiation therapy. Forty-three rectal cancer patients who underwent sphincter-saving operations after preoperative chemoradiation therapy were enrolled in the study. The total area of residual tumors was measured using morphometry software, and then the area of the residual tumors located within and beyond the muscular layer was also determined. Associations between clinicopathological features were evaluated. The results showed that the total area of residual tumor and area of residual tumor within the muscular layer were associated with TNM stage, tumor regression, and microscopy features, but not with patient disease-free survival. The area of the residual tumor located beyond the muscular layer was significantly associated with pathological ypT, ypN stage, tumor downstaging, perineural invasion, and the depth of tumor invasion beyond the muscular layer (P < 0.05). Further, large residual tumor area beyond the muscular layer was associated with shorter disease-free survival (P < 0.05). Morphometry of residual tumor area beyond the muscular layer is a new pathological prognostic factor for rectal cancer patients receiving preoperative chemoradiation therapy.