We collected groundwater samples at depths of up to 482 m from three boreholes drilled into sedimentary rock within two formations in Hokkaido, Japan. The prokaryotic community in each subsurface groundwater sample was analysed by microscopic counts and cloning–sequencing the 16S rRNA genes. On total direct counts, there were between 4.61 × 104 and 5.06 × 106 prokaryote cells ml−1 in the samples, which is similar to the numbers observed at the marine subsurface. However, the vertical distribution of the prokaryotes did not show a simple decrease in abundance with increasing depth. A high abundance of cells with significant amounts of RNA was identified in the domain Bacteria using fluorescence in situ hybridization, with a high frequency of dividing cells at the transition zone between the two sedimentary rock formations. Cloning–sequencing analysis showed the predominance of γ-Proteobacteria at this transition zone at 281–312 m. The horizontal heterogeneity of the microbial distribution in the subsurface environment was also demonstrated by a relatively high density of members of the domain Archaea in borehole HDB-4, drilled only 1.5 km northeast of HDB-6 and in the same formation.