Silk damping of primordial small-scale perturbations in the photon–baryon fluid due to diffusion of photons inevitably creates spectral distortions in the cosmic microwave background (CMB). With the proposed CMB experiment PIXIE it might become possible to measure these distortions and thereby constrain the primordial power spectrum at comoving wavenumbers 50 ≲ k ≲ 104 Mpc−1. Since primordial fluctuations in the CMB on these scales are completely erased by Silk damping, these distortions may provide the only way to shed light on otherwise unobservable aspects of inflationary physics. A consistent treatment of the primordial dissipation problem requires going to the second order in perturbation theory, while thermalization of these distortions necessitates the consideration of the second order in Compton scattering energy transfer. Here we give a full 2 × 2 treatment for the creation and evolution of spectral distortions due to the acoustic dissipation process, consistently including the effect of polarization and photon mixing in the free-streaming regime. We show that 1/3 of the total energy (9/4 larger than previous estimates) stored in small-scale temperature perturbations imprints observable spectral distortions, while the remaining 2/3 only raises the average CMB temperature, an effect that is unobservable. At high redshift dissipation is mainly mediated through the quadrupole anisotropies, while after recombination peculiar motions are most important. During recombination the damping of the higher multipoles is also significant. We compute the average distortion for several examples using CosmoTherm, analysing their dependence on parameters of the primordial power spectrum. For one of the best-fitting 7-year Wilkinson Microwave Anisotropy Probe cosmologies, with nS = 1.027 and nrun = −0.034, the cooling of baryonic matter practically compensates the heating from acoustic dissipation in the μ-era. We also derive the evolution equations for anisotropic spectral distortions in the first order perturbation theory. We furthermore argue that the first order anisotropies of spectral distortions may dominate over the corresponding second order contributions from recombination if an average fractional distortion ≃10−5 is already present before recombination.