Members of the prokaryotic picoplankton are the main drivers of the biogeochemical cycles over large areas of the world's oceans. In order to ascertain changes in picoplankton composition in the euphotic and twilight zones at an ocean basin scale we determined the distribution of 11 marine bacterial and archaeal phyla in three different water layers along a transect across the Atlantic Ocean from South Africa (32.9°S) to the UK (46.4°N) during boreal spring. Depth profiles down to 500 m at 65 stations were analysed by catalysed reporter deposition fluorescence in situ hybridization (CARD-FISH) and automated epifluorescence microscopy. There was no obvious overall difference in microbial community composition between the surface water layer and the deep chlorophyll maximum (DCM) layer. There were, however, significant differences between the two photic water layers and the mesopelagic zone. SAR11 (35 ± 9%) and Prochlorococcus (12 ± 8%) together dominated the surface waters, whereas SAR11 and Crenarchaeota of the marine group I formed equal proportions of the picoplankton community below the DCM (both ∼15%). However, due to their small cell sizes Crenarchaeota contributed distinctly less to total microbial biomass than SAR11 in this mesopelagic water layer. Bacteria from the uncultured Chloroflexi-related clade SAR202 occurred preferentially below the DCM (4–6%). Distinct latitudinal distribution patterns were found both in the photic zone and in the mesopelagic waters: in the photic zone, SAR11 was more abundant in the Northern Atlantic Ocean (up to 45%) than in the Southern Atlantic gyre (∼25%), the biomass of Prochlorococcus peaked in the tropical Atlantic Ocean, and Bacteroidetes and Gammaproteobacteria bloomed in the nutrient-rich northern temperate waters and in the Benguela upwelling. In mesopelagic waters, higher proportions of SAR202 were present in both central gyre regions, whereas Crenarchaeota were clearly more abundant in the upwelling regions and in higher latitudes. Other phylogenetic groups such as the Planctomycetes, marine group II Euryarchaeota and the uncultured clades SAR406, SAR324 and SAR86 rarely exceeded more than 5% of relative abundance.