The formation of composite materials represents an efficient route to improve the performances of polymers and expand their application scopes. Due to the unique structure and remarkable mechanical, electrical, thermal, optical and catalytic properties, carbon nanotube and graphene have been mostly studied as a second phase to produce high performance polymer composites. Although carbon nanotube and graphene share some advantages in both structure and property, they are also different in many aspects including synthesis of composite material, control in composite structure and interaction with polymer molecule. The resulting composite materials are distinguished in property to meet different applications. This review article mainly describes the preparation, structure, property and application of the two families of composite materials with an emphasis on the difference between them. Some general and effective strategies are summarized for the development of polymer composite materials based on carbon nanotube and graphene.