Soil degradation and subsequent yield decline are the main factors limiting further development of agriculture on the farming–pastoral transition zone of China. A 10-year field experiment was conducted in Inner Mongolia to compare the long-term effects of no-tillage with straw cover (NT), subsoiling with straw cover (ST), rototilling with straw cover (RT) and traditional tillage (TT) using ploughs on soil properties and productivity in a spring wheat–oat cropping system. Long-term conservation tillage increased soil organic matter in the top 20 cm by 21.4%, total N by 31.8% and Olsen’s P by 34.5% in the 0–5 cm layer compared to traditional tillage. Mean percentage of macro-aggregates (>0.25 mm, +20%) and macroporosity (>60 μm, +52.1%) also improved significantly in the 0–30 cm soil layer (P < 0.05). The largest yield improvements coupled with greatest water use efficiency (WUE) were achieved by no-tillage with straw cover. Ten-year mean crop yields increased by 14.0% and WUE improved by 13.5% compared to traditional tillage due to greater soil moisture and improved soil physical and chemical status. These improvements in soil properties and productivity are of considerable importance for the seriously degraded soils in semiarid Inner Mongolia, as well as for food security, sustainable agriculture and carbon storage in the farming–pasture transition regions of China.