The study examined the effect of gestational (GA) and postnatal (PNA) age on speech sound perception in infants. Auditory event-related potentials (ERPs) were recorded in response to speech sounds (syllables) in 50 infant NICU patients (born at 24–40 weeks gestation) prior to discharge. Efficiency of speech perception was quantified as absolute difference in mean amplitudes of ERPs in response to vowel (/a/-/u/) and consonant (/b/-/g/, /d/-/g/) contrasts within 150–250, 250–400, 400–700 ms after stimulus onset. Results indicated that both GA and PNA affected speech sound processing. These effects were more pronounced for consonant than vowel contrasts. Increasing PNA was associated with greater sound discrimination in infants born at or after 30 weeks GA, while minimal PNA-related changes were observed for infants with GA less than 30 weeks. Our findings suggest that a certain level of brain maturity at birth is necessary to benefit from postnatal experience in the first 4 months of life, and both gestational and postnatal ages need to be considered when evaluating infant brain responses.