[Japanese | Thesis | Researches in Minoh Lab | Minoh Lab]
In this paper, we propose a method of selecting frames from a video stream that can be stitched based on optical flow without calculating a homography matrix.
It takes much time to browse a whole video stream. So it is more efficient to browse a panoramic image that is constructed from an image sequence.
``Image mosaicing'' is a technique for constructing a large seamless panoramic image from many source images. It is impossible to stitch all the frames in a video stream taken by a video camera. Existence of a homography matrix between the two images is a theoretical condition of the image mosaicing. The condition restricts either the kinds of camera motion or the situations of a scene. If the camera motion is limited to rotation around the focal point, any images can be stitched. Length of optical flow in this motion does not depend on distance between objects and the image plane. On the contrary, for stitching the images taken under unlimited camera motion, the objects in a scene must be on a plane in 3D space. Since all frames in a video stream do not satisfy the condition, it is necessary to select frames that can be stitched. As a video stream contains many frames, selecting them should be done in a short time.
Most of the conventional methods do not select frames beforehand. These methods determine whether two successive frames can be stitched or not after warping the frames by a homography matrix, by calculating intensity differences in the overlapping region. Therefore these methods cannot determine whether the frames can be stitched or not until the whole procedure is finished. We call the methods ``overlap error accumulation method''.
In this paper, we propose a method of selecting frames that can be stitched by using optical flow. Use of the optical flow does not increase the amount of calculation time in our method, since some of the conventional methods include the optical flow calculation. Our method is faster than the overlap error accumulation method, because it selects frames that can be stitched before warping the frames.
We define five camera motion models. The frames taken under these camera motion models are capable of image mosaicing. The optical flow of each camera motion model has typical features. Our method analyzes the optical flow calculated from two successive frames in the given video stream and determines whether it falls under the five camera motion models or not. The camera motion models are: (1)translation on a plane parallel to the image plane, (2)rotation around the focal point, (3)zoom, (4)translation along the focal axis, and (5)rotation around the focal axis.
We conducted three experiments to discuss three topics: correctness of selecting frames by our method, condition that the calculation time of our method comes shorter than that of the overlap error accumulation method, and the ability of our method against the overlap error accumulation method. In the first experiment, we took scenes following the five camera motion models. 87 % of the frames in the video streams were selected. In the second experiment, we measured average calculation time to select the frames that can be stitched and that of total image mosaicing. We showed that our method can generate panoramic images in shorter calculation time if the video streams contain more than 0.1 % frames that can be stitched. In the third experiment, we applied our method to the frames taken under unlimited camera motions and verified the ability of our method against the overlap error accumulation method. The overlap error accumulation method selected 69 % frames in the video stream . In our method 83 % frames of the 69 % frames were selected. The results show the calculation time of our method is shorter than that of the overlap error accumulation method and verify the validation of our method.