Considerable progress has been made in our understanding of the remarkable fidelity with which the human auditory brainstem represents key acoustic features of the speech signal. The brainstem response to speech can be assessed noninvasively by examining scalp-recorded evoked potentials. Morphologically, two main components of the scalp-recorded brainstem response can be differentiated, a transient onset response and a sustained frequency-following response (FFR). Together, these two components are capable of conveying important segmental and suprasegmental information inherent in the typical speech syllable. Here we examine the putative neural sources of the scalp-recorded brainstem response and review recent evidence that demonstrates that the brainstem response to speech is dynamic in nature and malleable by experience. Finally, we propose a putative mechanism for experience-dependent plasticity at the level of the brainstem.