We performed high-resolution photon-energy and polarization-dependent ARPES measurements on ultrathin Bi(111) films [6-180 bilayers (BL), 2.5-70 nm thick] formed on Si(111). In addition to the extensively studied surface states (SSs), the edge of the bulk valence band was clearly measured by using S-polarized light. We found direct evidence that this valence band edge, which forms a hole pocket in the bulk Bi crystal, does not cross the Fermi level for the 180 BL thick film. This is consistent with the predicted semimetal-to-semiconductor transition due to the quantum-size effect [V.B. Sandomirskii, Sov. Phys. JETP 25, 101 (1967)]. However, it became metallic again when the film thickness was decreased (below 30 BL). A plausible explanation for this phenomenon is the modification of the charge neutrality condition due to the size effect of the SSs.