It is well understood that Australian climate is affected by natural climate variability such as El Niño–Southern Oscillation (ENSO), the Indian Ocean Dipole (IOD), and the Southern Annular Mode (SAM), depending on seasons and regions. However, studies on Australian climate extremes associated with natural climate variability remain limited. This study examines the observed impact of natural climate variability on inter-annual changes in seasonal extremes of rainfall and temperature over Australia during 1957–2010. We use non-stationary Generalized Extreme Value (GEV) analysis, where GEV parameters are specified as a linear function of modes of climate variability, and compare results with the case when climate variability is not considered. Results from two station-based observational data sets consistently suggest that extreme responses overall resemble mean responses to climate variability. Anomalously drier and hotter conditions occur over northeastern Australia and the southern coast during El Niño and a positive phase of the IOD in the cold seasons, while wetter and cooler conditions are observed during La Niña and a negative phase of IOD. A positive (negative) phase of the SAM brings wetter and cooler (drier and warmer) conditions over much of the eastern continent in summer. Covariation and relative importance of ENSO and the IOD as well as an inverse relationship between rainfall and daily maximum temperature are also found to hold for extremes. This suggests that teleconnection mechanisms responsible for seasonal mean variations may be at work for inter-annual changes in extremes, providing important implications for climate model evaluations and regional climate change projections.