We investigated 16S rRNA gene diversity at a high sediment depth resolution (every 5 cm, top 30 cm) in an active site of the Kazan mud volcano, East Mediterranean Sea. A total of 242 archaeal and 374 bacterial clones were analysed, which were attributed to 38 and 205 unique phylotypes, respectively (≥98% similarity). Most of the archaeal phylotypes were related to ANME-1, -2 and -3 members originating from habitats where anaerobic oxidation of methane (AOM) occurs, although they occurred in sediment layers with no apparent AOM (below the sulphate depletion depth). Proteobacteria were the most abundant and diverse bacterial group, with the Gammaproteobacteria dominating in most sediment layers and these were related to phylotypes involved in methane cycling. The Deltaproteobacteria included several of the sulphate-reducers related to AOM. The rest of the bacterial phylotypes belonged to 15 known phyla and three unaffiliated groups, with representatives from similar habitats. Diversity index H was in the range 0.56–1.73 and 1.47–3.82 for Archaea and Bacteria, respectively, revealing different depth patterns for the two groups. At 15 and 20 cm below the sea floor, the prokaryotic communities were highly similar, hosting AOM-specific Archaea and Bacteria. Our study revealed different dominant phyla in proximate sediment layers.