We present a new 3D object tracking algorithm that supports multiple planar and nonplanar objects with real-time processing speed and high accuracy. The main problem of object tracking algorithm is the limitation of the supporting type of target object, slow processing speed, and low tracking accuracy. Our algorithm provides high accuracy and real-time performance while detecting not only planar objects but also nonplanar objects. The real-time performance is accomplished by using Features from Accelerated Segment Test corner detection, region of interest, and parallel processing on a multicore processor. High accuracy is realized by using a scale-invariant feature transform descriptor, random sample consensus, region of interest, and double robust filtering. Copyright © 2013 John Wiley & Sons, Ltd.