Smart metering application has received a lot of attention from the research community lately. Usually, low power and lossy network runs on routing protocol for low power and lossy networks protocol, which constructs a directed acyclic graph structure for its normal operation. In this paper, we devise a wormhole attack scenario in such a network. Furthermore, we propose a Merkle tree-based authentication mechanism, which runs on the notion of constructing a tree of hashed security information. We analyze the approach by formulating the wormhole attack scenario as a graph theoretic problem and show the effectiveness of the proposed Merkle tree-based approach for authenticating communications. Furthermore, we perform simulations in network simulator 2 to observe the network performance by adopting a Merkle tree-based approach to prevent from disrupting the links, and we observe better throughput, reduction in jitter, and end-to-end delay as compared with scenarios with wormhole attacks and no avoidance mechanism. In the end, we take a step toward optimizing the performance of the mechanism by proposing an effective tree traversal algorithm, which works on the notion of electing nodes as a root in a large scale network from where a Merkle tree will be originated. The mechanism with the traversal algorithm assists in managing the authentication in a huge network, which is broken down into many trees avoiding wormhole attacks in the network. Copyright © 2014 John Wiley & Sons, Ltd.