TOP  >  Thesis/Dissertation  >  Virtual Object Manipulation via Real Object of Different Shape

Virtual Object Manipulation via Real Object of Different Shape

The Virtual Studio system enables us to present an audience a synthesized scene composed of a real-life actor, a virtual CG background and virtual CG objects. The actor interacts with the virtual objects within the system by constantly watching a monitor in which the synthesized scene is shown. If the actor could manipulate the virtual objects directly with his hands, a convincing demonstration could be given to the audience, however, manipulation of virtual objects is difficult because the actor in the studio can neither directly see nor touch the virtual objects in the synthesized video. Previous work tackled this problem by using a real object of same shape as the virtual object. In this research, an actor manipulates the real object and the system substitutes the virtual object for the real object in the synthesized video. Using a real object provides the actor with visual information and allows the actor to interact with the object tactually. In this method, each real object is of the same shape as the virtual object it represents. Therefore different real object must be prepared for each virtual object. Furthermore, virtual objects can not transform its shape, although transforming shape of virtual objects is one of the key advantage of the Virtual Studio. In this paper, we aim to manipulate a virtual object represented by a real object with a different shape from the virtual one. By removing the restriction that the real object be of the same shape as its virtual counterpart, the virtual object can transform its shape. However, when the real object is replaced by the virtual object in the synthesized video, the relative positions between the virtual object and the actor's hand are drawn incorrectly. This problem is caused because the relative position between a virtual object and the actor's hand does not correspond to the relative position between the real object and the actor's hand. To overcome this difficulty, our method uses a three dimensional hand model of customizable posture. The position and posture of the actor's hand are estimated from observation of the video stream, and a virtual model of the hand is created with the same position and posture as the actor's hand. Comparing this virtual model hand and the virtual object, the system determines which is in the foreground position to the camera so as to draw them correctly in the synthesized video. Our method is composed of four processes described below. Firstly, the position and posture of the real object is estimated from the observed video and a corresponding virtual object of identical position and posture is created. This estimation can be done by a model-based pose estimation of the object. Secondly, the posture of the actor's hand is estimated. An image of the actor's hand is extracted from the observed video and is compared against a database of hand images in various poses. The image nearest to the actor's hand image is then selected from the database. Thirdly, the position of the actor's hand is estimated. A three dimensional model of the hand is created according to the posture selected in step two. The 3D position of the newly created hand model is adjusted to the observed hand by searching the position which it satisfies the following conditions; the image in which the hand model is projected corresponds to the region the actor's hand is observed in the video the most; the hand model collides with the real object as little as possible. In this way, the position and posture of the real object and the actor's hand are deduced, and then the position and posture of the virtual object and the model hand are decided. Finally, the hand model and the virtual object are compared as to which is in the foreground position to the camera, and then they are drawn correctly in the synthesized video. We made the system according to the proposed method described above. We then created a synthesized video wherein the actor manipulates a virtual object by using a real object which was of a different shape from the virtual object and verified that the virtual object and the actor's hand are correctly drawn in the synthesized video.