To find useful tools to evaluate the prognosis in colorectal carcinoma (CRC) patients, we investigated the prognostic values of tumor-infiltrating T lymphocyte subsets according to intratumoral subsites as well as clinical or molecular characteristics. Immunohistochemistry for CD8, CD45RO, and FOXP3 was performed, and densities of the T cell subsets in each tissue microarray core (cells/mm2) were measured by image analysis. In the training set (n = 218) of CRC, T cell subset densities in the invasion front were more strongly associated with patient outcome than those in the tumor center. In the validation set (n = 549), T cell subset densities in the invasion front were evaluated. Univariate analysis showed that all three T cell subset densities were significantly associated with longer progression free survival and overall survival time (p < 0.001). In multivariate analysis, a high CD45RO density correlated independently with longer progression free survival (p = 0.011) and overall survival time (p = 0.007) in CRC patients, regardless of tumor location or adjuvant chemotherapy status. Our results showed that CD45RO density in the invasion front was the only independent prognostic factor regarding CRC. However, CD8 and FOXP3 densities were also independent prognostic factors in certain clinical settings. Thus, image analysis of tissue microarray cores in the invasion front of CRC could be used as a valid method for evaluating the prognostic significance of T cell subset densities.