We investigated the diversity and distribution of archaeal and bacterial 16S rRNA gene sequences in deep aquifers of mid- to late Miocene hard shale located in the northernmost region of the Japanese archipelago. A major fault in the north-west–south-east (NW–SE) direction runs across the studied area. We collected three groundwater samples from boreholes on the south-west (SW) side of the fault at depths of 296, 374 and 625 m below ground level (m.b.g.l.) and one sample from the north-east (NE) side of the fault at a depth of 458 m.b.g.l. The groundwater samples were observed to be neutral and weakly saline. The total microbial counts after staining with acridine orange were in the order 105−106 cells mL−1 and 103 cells mL−1 in the aquifers to the SW and to the NE of the fault, respectively. A total of 407 archaeal and bacterial 16S rRNA gene sequences (204 and 203 sequences, respectively) were determined for clone libraries constructed from all groundwater samples. Phylogenetic analyses showed that the libraries constructed from the SW aquifers were generally coherent but considerably different from those constructed from the NE aquifer. All of the archaeal clone libraries from the SW aquifers were predominated by a single sequence closely related to the archaeon Methanoculleus chikugoensis, and the corresponding bacterial libraries were mostly predominated by the sequences related to Bacteroidetes, Firmicutes and δ-Proteobacteria. In contrast, the libraries from the NE aquifer were dominated by uncultured environmental archaeal clones with no methanogen sequences and by β-proteobacterial clones with no sequences related to Bacteroidetes and δ-Proteobacteria. Hence, the possible coexistence of methanogens and sulphate reducers in Horonobe deep borehole (HDB) on the SW side is suggested, particularly in HDB-6 (374 m.b.g.l.). Moreover, these organisms might play an important geochemical role in the groundwater obtained from the aquifers.