Magnetic Doppler imaging is currently the most powerful method of interpreting high-resolution spectropolarimetric observations of stars. This technique has provided the very first maps of stellar magnetic field topologies reconstructed from time series of full Stokes vector spectra, revealing the presence of small-scale magnetic fields on the surfaces of Ap stars. These studies were recently criticised by Stift et al., who claimed that magnetic inversions are not robust and are seriously undermined by neglecting a feedback on the Stokes line profiles from the local atmospheric structure in the regions of enhanced metal abundance. We show that Stift et al. misinterpreted published magnetic Doppler imaging results and consistently neglected some of the most fundamental principles behind magnetic mapping. Using state-of-the-art opacity sampling model atmosphere and polarized radiative transfer codes, we demonstrate that the variation of atmospheric structure across the surface of a star with chemical spots affects the local continuum intensity but is negligible for the normalized local Stokes profiles except for the rare situation of a very strong line in an extremely Fe-rich atmosphere. For the disc-integrated spectra of an Ap star with extreme abundance variations, we find that the assumption of a mean model atmosphere leads to moderate errors in Stokes I but is negligible for the circular and linear polarization spectra. Employing a new magnetic inversion code, which incorporates the horizontal variation of atmospheric structure induced by chemical spots, we reconstructed new maps of magnetic field and Fe abundance for the bright Ap star α2 CVn. The resulting distribution of chemical spots changes insignificantly compared to the previous modelling based on a single model atmosphere, while the magnetic field geometry does not change at all. This shows that the assertions by Stift et al. are exaggerated as a consequence of unreasonable assumptions and extrapolations, as well as methodological flaws and inconsistencies of their analysis. Our discussion proves that published magnetic inversions based on a mean stellar atmosphere are highly robust and reliable, and that the presence of small-scale magnetic field structures on the surfaces of Ap stars is indeed real. Incorporating horizontal variations of atmospheric structure in Doppler imaging can marginally improve reconstruction of abundance distributions for stars showing very large iron overabundances. But this costly technique is unnecessary for magnetic mapping with high-resolution polarization spectra.