The design and fabrication of synthetic structural components often result in homogeneous materials with uniform microstructures and properties. In contrast, nature has evolved structural composites exhibiting rich heterogeneous architectures and tunable site-specific properties. Creating synthetic systems with the heterogeneous nature of biological materials should enable the fabrication of composites with extended durability under severe mechanical demands or with adequate properties using, for example, a more restricted selection of bioresorbable or environmental-friendly basic building blocks. Here, the heterogeneous structures of the tooth and of the tendon-bone interface are revisited to identify design strategies that have been naturally selected to best respond to the non-uniform stresses distributions typically found in such load-bearing structures. Recent attempts to replicate some of these strategies in man-made materials are also shown to illustrate the variety of unusual properties that can be achieved through the proposed bioinspired approach. Finally, the creation of heterogeneous architectures with local microstructure and properties deliberately tuned to match non-uniform loading conditions is suggested as a new pathway towards the development of “material systems” with unprecedented functionalities and durability in mechanically challenging applications.