Conservation agriculture can provide a low-cost competitive option to mitigate global warming with reduction or elimination of soil tillage and increase soil organic carbon (SOC). Most studies have evaluated the impact of zero till (ZT) only on surface soil layers (down to 30 cm), and few studies have been performed on the potential for C accumulation in deeper layers (0–100 cm) of tropical and subtropical soils. In order to determine whether the change from conventional tillage (CT) to ZT has induced a net gain in SOC, three long-term experiments (15–26 years) on free-draining Ferralsols in the subtropical region of South Brazil were sampled and the SOC stocks to 30 and 100 cm calculated on an equivalent soil mass basis. In rotations containing intercropped or cover-crop legumes, there were significant accumulations of SOC in ZT soils varying from 5 to 8 Mg ha−1 in comparison with CT management, equivalent to annual soil C accumulation rates of between 0.04 and 0.88 Mg ha−1. However, the potential for soil C accumulation was considerably increased (varying from 0.48 to 1.53 Mg ha−1 yr−1) when considering the soil profile down to 100 cm depth. On average the estimate of soil C accumulation to 100 cm depth was 59% greater than that for soil C accumulated to 30 cm. These findings suggest that increasing sampling depth from 30 cm (as presently recommended by the IPCC) to 100 cm, may increase substantially the estimates of potential CO2 mitigation induced by the change from CT to ZT on the free-draining Ferralsols of the tropics and subtropics. It was evident that that legumes which contributed a net input of biologically fixed N played an important role in promoting soil C accumulation in these soils under ZT, perhaps due to a slow-release of N from decaying surface residues/roots which favored maize root growth.