We successfully determined the difference of immune microenvironments between pNENs and pancreatic ductal adenocarcinomas (PDACs), and the histology-dependent variability among pNENs using multispectral fluorescent imaging system. Tumour tissue samples including 52 pNENs and 18 PDACs were investigated. The tumour-infiltrating lymphocytes (TILs), their PD-1 and PD-L1 expression in the pNENs were comprehensively and quantitatively analysed and were subsequently compared with those in PDACs. A principal component analysis revealed that the tissue immune profile is related to tumour histology, with distinct groups being observed for NETs, NECs, and PDACs. While NECs and some PDACs had hot immune microenvironments with abundant TILs, NETs had a cold immune microenvironment with few TILs. Moreover, in NETs, the numbers of intraepithelial PD-1high T cells and PD-L1high Type-II macrophages were elevated according to the grade. Univariate analysis revealed that lymph node metastasis, grade, stage, PD-1high T cells, and PD-L1high Type-II macrophages were predictors for recurrence-free survival (RFS), while grade and PD-1high T cells were prognostic factors for overall survival (OS). We also showed that PD-1high T cells and PD-L1high Type-II macrophages were associated with worse outcome in pNENs. Our results support the WHO 2017 tumour classification criteria, which distinguish between G3 NETs and NECs.