Wireless mesh networks provide long-distance wireless network connectivity over heterogeneous devices for greater scalability and availability. However, protecting legitimate long-distance wireless links from wormhole attacks is an important yet challenging security issue in wireless mesh networks. In this paper, we propose a reputation-based cross-layer intrusion detection system to effectively detect various wormhole attacks. The proposed system analyses the behaviours of the routing paths in wireless mesh networks to correctly isolate the malicious wormhole paths from legitimate long-distance wireless links. It uses reputation and cross-layer parameters for comprehensive ability to isolate the wormhole attacks in routing paths. This isolation ensures full utilisation of legitimate long-distance wireless links in wireless mesh networks, which is not possible with the existing wormhole attack detection approaches. Experimental results show that the proposed system increases the detection rate, decreases the false alarm rate, and secures legitimate long-distance wireless links in wireless mesh networks. Copyright © 2014 John Wiley & Sons, Ltd.