Auditory neurons are often characterized by their spectro-temporal receptive field (STRF), a linear measure that captures overall trends of neural responses to modulations of the spectro-temporal envelopes of sounds. We have previously shown that primary auditory cortex neurons of the awake ferret are better characterized by STRFs followed by a non-trivial non-linearity. This non-linearity is a half-wave rectification followed by a squaring function, indicating that cortical neurons probably encode higher-order statistics of the spectrum of sounds. In this article, we introduce the concept of a contrast receptive field (CRF) and show that neurons in the auditory cortex encode quadratic statistics of the spectro-temporal envelope of sounds, which we call auditory contrast. We reveal phase-dependent contrast tuning in single units. Most units with a reliable STRF also possess a reliable CRF, such that the response to stimulus contrast complements the linear response described by the STRF. The relationship between the STRF and the CRF is analyzed in terms of orthogonality, co-localization in time–frequency, feature orientation selectivity, and output non-linearity interdependence. Our study shows that contrast can be used by auditory cortex neurons to sharpen, in a noise-resistant fashion, their responses to dynamic spectral profiles.