[Japanese | Thesis | Researches in Minoh Lab | Minoh Lab]
Computer Graphics (CG) objects are often used for making various video contents. Using CG virtual objects (models of real object or artificial CG objects) and visual effects to them enable to make various video contents. In educational video contents which a lecturer explains to audiences, CG objects help the audiences to understand the explanation.
The virtual studio system enables us to synthesize virtual objects and real ones in the same scene. Current virtual studio system aims that virtual objects appear to only background in the scene. In order to make video contents in which the lecturer has interaction with virtual objects, manipulate these objects freely and explain their in the virtual studio system, my research aims at realizing the direct manipulation of virtual objects in virtual studio.
There are five problems for the virtual object manipulation. Firstly, in the scene created with virtual studio system, the audience can see virtual objects. But the actor in virtual studio is not able to see any virtual objects from his point of view nor to touch his body to virtual ones. In these situations, it is difficult for the actor to act naturally. To overcome this difficulty, it is required to visualize virtual objects and get tactile sensation of virtual objects for the actor.
Next, the system has to detect the interaction between the actor and virtual objects. One kind of interaction is that the actor translates and rotates virtual objects at the hand. Object motion depends on actor's physically motion, in the situation that there are not any objects. So, the system has to detect translational and rotational manipulations of the actor. And I think that an occurrence of CG special effects by actor's pointing to a virtual object is also one of virtual object manipulations. To realize an occurrence of CG special effect, the system has to detect the actor's pointing motion. Translational and rotational manipulations are directly proportional to actor's physical motion, but a manipulation occurring special effects is not proportional and the system detect whether the pointing of actor is ON or OFF. And last problem, the system has to synthesize the actor and virtual objects, in consequence of their each other's occluded part. In the scene that actor manipulate virtual objects at his hand, it is possible that actor's hand occlude the part of virtual object. In this situation, the merging image has to be in coherent.
For handling these problems, it is not desirable that some equipment for sensing actor's interaction are used, because the audience can not come to pay attention in explanation by this device.
To overcome five problems for virtual object manipulation, I propose new method using a real object being same figure to virtual object. When the actor grasps and manipulates the real object in virtual studio, the virtual object substitute for real object in the merged video contents. This method realizes three manipulations. They are translation, rotation and appearing special effects by actor's pointing. About visualization and tactile of virtual object, the actor can visualize and touch the real object. Detection of translation and rotation of virtual object is realized by the model-based pose estimation of object. The estimation method uses markers attached on the real object. Translation and rotation that is the output of pose estimation for real object is applied as the motion of the virtual object. In order to detect pointing motion of actor onto the real object for occurring special effect, we use markers used to the pose estimation. The system detects a marker which is hidden by finger. We assume that the actor in studio exposes the manipulating object to camera (and audience), so the hidden marker is detected from markers that can see from the camera. To merge virtual objects and actor, we use a single color real object. Detecting a region of real object provides the detecting a region where the virtual object is displayed.
We evaluate the effectiveness of the proposed method by the experiment. These manipulations which are translation, rotation and occurring animation are realized. I used a spherical real object and an earth by CG. Using proposed method, we could merge the actor images and virtual objects naturally.