In this paper, we propose a new security log management scheme for smart surveillance in a multi-camera environment. Basically, our security log consist of descriptions for various behavior properties of moving objects, such as motion type, time, and speed in a merged camera view. To generate such security log, we first analyze the input video frame from each surveillance camera and construct a motion vector of interest points in the frame. By analyzing the motion vector, we recognize moving objects and trace their local behavior in the video. On the basis of this analysis, we can calculate various global behavior features of the objects in the merged camera view, which can be acquired by stitching together the frames from multiple camera inputs. Such global behavior features are captured into security logs, which can be used to smartly carry out various surveillance operations such as retrieving objects whose behavior is similar to a query behavior or whose behavior shows predefined abnormality. Because our scheme treats all the objects in the frame independently, it can handle multiple objects simultaneously. We implemented a prototype system and performed various experiments to demonstrate that our scheme can achieve a reasonable performance. Copyright © 2013 John Wiley & Sons, Ltd.