The direct radiative forcing of the climate system includes effects due to scattering and absorbing aerosols. This study explores how important physical climate characteristics contribute to the magnitudes of the direct radiative forcings (DRF) from anthropogenic sulfate, black carbon, and organic carbon. For this purpose, we employ the GFDL CM2.1 global climate model, which has reasonable aerosol concentrations and reconstruction of twentieth-century climate change. Sulfate and carbonaceous aerosols constitute the most important anthropogenic aerosol perturbations to the climate system and provide striking contrasts between primarily scattering (sulfate and organic carbon) and primarily absorbing (black carbon) species. The quantitative roles of cloud coverage, surface albedo, and relative humidity in governing the sign and magnitude of all-sky top-of-atmosphere (TOA) forcings are examined. Clouds reduce the global mean sulfate TOA DRF by almost 50%, reduce the global mean organic carbon TOA DRF by more than 30%, and increase the global mean black carbon TOA DRF by almost 80%. Sulfate forcing is increased by over 50% as a result of hygroscopic growth, while high-albedo surfaces are found to have only a minor (less than 10%) impact on all global mean forcings. Although the radiative forcing magnitudes are subject to uncertainties in the state of mixing of the aerosol species, it is clear that fundamental physical climate characteristics play a large role in governing aerosol direct radiative forcing magnitudes.