[Japanese | Thesis | Researches in Minoh Lab | Minoh Lab]
On film making, it has become popular to apply visual effects (VFX) using computer graphics. The VFX which emphasize motion states of objects such as a position, velocity and rotation make scenes more powerful. In "Shaolin Soccer", the velocity and rotation of the soccer ball are emphasized through trailing and swirling flame CG. In "THE MATRIX", bullet trajectories are emphasized in the scene of "bullet time" using CG that express how bullets fly through the air.
For adding CG as VFX to real objects in image sequences, it is necessary to obtain their motion states from the image sequences. The purpose of this study is to estimate motion states from real image sequence. It is straightforward to obtain 2-dimensional position and velocity on the image from image sequences. However, 3-dimensional motion states of the object cannot be directly obtained.
For estimating these motion states, knowledge of the target object is used. Several methods use a 3-dimensional shape model. In order to estimate the 3-dimensional position and pose of the object, the shape model is translated and rotated so that feature points which extracted from a frame of the image sequence can match those of the projected image of the shape model. The rotation of the object can be estimated from the difference of the pose of the object between two frames. It is difficult, however, to extract feature points of objects such as a sphere with no texture, so that the matching fails and the motion states cannot be estimated correctly. Even if the matching succeeds, there is still the issue of temporal resolution that the rotation cannot be estimated correctly unless the resolution is high enough to interpolate the rotation.
In this study, we adopt the laws of motion as knowledge of the target object for the motion estimation. Although motion states of the object change at every moment, this change follows the laws of motion. If the motion occurs in known surroundings, the motion is collectively expressed by one set of properties which correspond to the object itself, the surroundings, and the relationship between them and initial states of the motion. We refer to these properties which determine motion states at every time based on the laws of motion as motion properties. We propose a method for the motion estimation using a motion model which has the laws of motion and the motion properties. The motion model is acquired so that the model can simulate an observed motion in the image sequence. This method can estimate motion states independently of the temporal resolution, since the laws of motion express motion continuously.
We verify effectiveness of this method by estimating the rigid motion of a ball through the acquisition of the motion model of the ball. The motion model of the ball includes the gravity, air resistance, lift, and inelastic collision with friction laws of motion. A drag coefficient, coefficient of restitution, coefficient of sliding friction, initial position, initial linear velocity, and initial angular velocity are estimated as motion properties. A trajectory which is the position of the object on the image at each frame of the image sequence along with its captured time is used for estimating motion states. The motion model is modified to be able to reproduce the trajectory in the image sequence by simulating the motion of the object. The motion properties are estimated by minimizing the difference between the input trajectory and the simulated trajectory using Powell's method.
We perform simulation experiments to verify estimated motion states from the trajectory of the motion. Estimated motion states are accurate enough to add CG which emphasize them on the image sequence. The residual between the trajectory extracted from the image sequence and the estimated trajectory when using real image sequence is still very high. The estimated motion properties and motion states have not been weighed up, because their actual values have not been measured. Applying this method for the real image sequence remains to be solved.