Based on the Heisenberg model including spin–phonon interaction and using a Green's function technique we have studied the size and transition metal doping effects on the phonon properties of antiferromagnetic NiO nanoparticles. The phonon frequency decreases whereas the phonon damping increases with decreasing of particle size. We have shown that a close correlation exists between microstructural features, the different radii of the dopants, leading to different stress and different spin–phonon interaction, and the phonon properties of the nanoparticles. In dependence of the radii of the doped ions we obtain a decrease or increase of the phonon energy with increasing doping concentration. The phonon damping is always greater compared to the undoped case. Our theory can be applied to all antiferromagnetic nanoparticles.