Perennial forage legumes, particularly lucerne (Medicago sativa L.), play a significant role in crop/livestock mixed farming systems in the semiarid region of the Loess Plateau of China as stock feed and a source of nitrogen for subsequent crops. However, there is evidence that lucerne reduces soil water deep in the soil profile, thereby reducing subsequent crop productivity. From 2004 to 2010, this study evaluated the forage productivity and water use of two locally adapted perennial legume species, milk vetch (Astragalus adsurgens Pall.) and bush clover (Lespedeza davurica S.), compared with lucerne. The 7-year total and average annual forage yield of milk vetch were 56 and 8 t ha−1 and bush clover was 42 and 6 t ha−1, respectively, significantly lower than lucerne at 91 and 13 t ha−1. However, despite lower water-use efficiencies (16 and 12 kg ha−1 mm−1 for milk vetch and bush clover, respectively, compared to 22 kg ha−1 mm−1 for lucerne), the total 7-year water use in milk vetch and bush clover was 3500 mm and 3490 mm, respectively, which was 135–140 mm less than lucerne. After 7 years, lucerne had extracted water from the upper 5 m soil, whereas bush clover used water mainly from the upper 2 m of the soil profile and milk vetch still had some water available below 3 m. We conclude that while the locally adapted forage legumes were not as productive as lucerne as a source of fodder in mixed cropping/livestock system in this region, they use less water, which may be advantageous in drier regions.