[Japanese | Thesis | Researches in Minoh Lab | Minoh Lab]
The use of lecture video archives is becoming more popular at universities. Sets of video cameras to record lectures are often installed on ceilings or walls of the lecture rooms, to prevent the cameras from interfering with lectures. The recorded videos by the cameras around the students are useally presented later to the users just as they were recorded. These videos could not give the users the feeling as if they would attend a real lecture. One of the reasons is that those videos are not recorded from students' viewpoints. Another problem is that the users cannot adjust the directions of the cameras.
Videos that substitutes for a student's view in a lecture room can give the users highly realistic sensation, because the users can virtually look around a lecture room with utilization of such videos. So I consider generation of arbitrary view images of students from multiple shots taken by a set of cameras surrounding the lecture space.
One of the approaches to generate arbitrary view image from multiple shots is to reconstruct model shapes of the objects in a scene from the shots acquired by strongly calibrated cameras and then project the models onto a result image. This approach requires the strong calibration, which determines the projection matrices of each camera. It is difficult to precisely calibrate cameras in large spaces such as lecture rooms. Consequently, it is also difficult to change the camera arrangements or to set up a lecture archiving system in another place.
The other approach requires only weak calibration of cameras, which is acquired more easily than strong calibration. This approach reconstructs shapes in a projective grid space, which is a coordinate system based on the coordinates in two basis images. However, the reconstructed shapes are distorted and cannot be directly associated with the world coordinate system because of the distortion of the projective grid space. A fundamental problem arises in applying such approach to generate arbitrary view images in the lecture rooms.
In this paper, I propose a method for correcting distortion of projective grid spaces using the restrictions based on the arrangement of equipments in the lecture rooms.
Restrictions on locations of top/bottom points of each student are used. It is assumed that all the seats in the lecture room are separated, and only one student can sit on each seat at the same time. All the students are sitting on the center of each seat. With this assumption, each student's location is identified by a seat location. It is also assumed that all the top points of students have the same height. The bottom points are at the same height with the desks, because the lower parts of students than the desks are hidden from the cameras. The equivalent points to top/bottom points are extracted from the projective grid space model as reference points, of which the world coordinates can be estimated based on the restrictions of lecture rooms denoted above. The world coordinates of other points in the projective grid space are estimated by interpolating from neighboring reference points. By associating the projective grid coordinates with the world coordinates, distortion of projective grid space is corrected.
Floors, walls, seats, desks and other objects in the lecture rooms are assumed to be fixed and to have constant shape, and color. So the fixed 3-D shape models of them can be made in advance. Then, the arbitrary view images of the lecture room from student's viewpoint can be generated based on the corrected student's model and the lecture room model.
In order to evaluate the accuracy of the proposed correcting method, I simulated this method with the condition that the shape of students' model is correctly reconstructed and top/bottom points are correctly extracted. The location errors of the points of student models in the world coordinate system were no greater than 2cm with proper basis camera arrangements. I also showed the validity of the proposed method by generating some arbitrary view images with small distortion.