TOP  >  Thesis/Dissertation  >  {sotsurontitle}

Estimating learners' impressions of difficulty level for learning content from their nonverbal behaviors in e-Learning

Recently, ICT (Information Communication Technologies) has widely introduced to educational field with development and spread of computers and networks. One of such examples is e-Learning.

There are two types of e-Learning: One is asynchronous type e-Learning, in which learners individually access to course materials on a web server and learn the materials by themselves on their own convenient time and location. The learning style of asynchronous type e-Learning is a kind of one-way communication with visual information. The other one is synchronous type e-Learning, in which the learners learn some topics or materials with a teacher in real-time on their own convenient location via real-time streamed videos. The teachers and each learner can make two-way communication or dialogues with images and voices between them on one-to-one level in synchronous type e-Learning.

Regarding educations or learning as a kind of communication, one of the most important information for the teachers is the learners' state of mind, which are mainly expressed by their nonverbal behaviors such as facial expressions, eye gaze, posture change of head, and gestures. Especially, the learners' Subjective Impressions of Difficulty level for a Learning content, which we call SIDL for short in this thesis, is much important among the various kinds of state of mind. For instance, the learners' SIDL is used as follows in conventional classroom lectures: During a lecture, the teacher finds the learners whose SIDL is high, or who are feeling confused in the lecture content, and gives them an advice in real-time to relieve their "high SIDL" or confusion. In addition, after the lecture, the teacher improves his/her teaching methods and teaching materials on the part in which many learners felt confused with "high SIDL." However, in asynchronous type e-Learning, the teachers cannot fully use the learners' state of mind including SIDL in this way, because the teachers cannot see the learners' nonverbal behaviors directly in asynchronous type e-Learning. The similar problem can occur in the situation of synchronous type e-Learning depending on the quality (resolution or frame rate) of the real-time videos or the number of the learners who learn same material simultaneously.

In order to cope with the problem, we aim to realize a mechanism for automatically estimating the learners' SIDL from their nonverbal behaviors in this thesis, focusing on the learners in both asynchronous type e-Learning and synchronous type e-Learning. The estimation mechanism is expected to be applied as follows: Analyzing the estimation results for a group of the learners, the teachers can know which parts in the course materials are easily confusing the learners, so that the teachers can improve the confusing parts in the course materials. In addition, giving a support material adaptively to the learners in real-time based on the estimation results, not only the teachers but also e-Learning systems can relieve the learners' confusion automatically, as do the teachers in the classroom lectures or home tutoring.

There is a big problem in estimating the learners' SIDL from their nonverbal behaviors; that is, the correlations between the learners' SIDL and their nonverbal behaviors have been neither trivial nor clarified enough yet. Which kinds of nonverbal behavioral features are effective for estimating SIDL? Do such effective behavioral features depend on individual learner? These questions still have not been answered. Moreover, depending on the two types of e-Learning, which are asynchronous type and synchronous type, the properties of the correlations between the nonverbal behaviors and SIDL would vary. For this reason, in this thesis, we first analyze the properties of the correlations, independently focusing on the asynchronous type e-Learning and synchronous type one. After that, we independently propose a concrete method for estimating SIDL in the case of asynchronous type e-Learning and that of synchronous type e-Learning.

As a part of the above analyses, we first tried to examine whether the human teachers can estimate the learners' SIDL from their nonverbal behaviors or not, showing the learners' videos to the human teachers. In this trial, we aimed to obtain the videos from which the teachers can easily recognize not only the learners' nonverbal behaviors but also the object of the learners' gaze on their monitor, because the learners' eye gaze, which is one of the major nonverbal behaviors, will become more effective by used in conjunction with its target objects. Specifically, we focused on the learners learning in asynchronous type e-Learning, and virtually synthesized their images seen through the monitor with a viewpoint behind the monitor, which we call "Transparent images," by Computer Vision techniques. Then we actually showed the sequences of the Transparent images to several human test-subjects as videos, and instructed the test-subjects to estimate the learners' SIDL. As the result, the test-subjects (or human teachers) could correctly estimate the learners' SIDL from their nonverbal behaviors. In addition, the test-subjects enumerated the following 4 nonverbal behaviors as some of the evidences of their estimation: "leaning head to the side," "gazing at a single object," "moving face," and "mumbling."

Based on the above result, we next analyzed the actual correlations between the nonverbal behaviors and SIDL in order to realize a method for estimating SIDL, focusing on asynchronous type e-Learning. We first defined 6 candidates of the features for the estimation so that each feature directly represents one of the above 4 nonverbal behaviors. Next we examine whether each feature candidate is significantly correlated with the learners' actual SIDL or not. As the result, we confirmed that all of 6 feature candidates are basically correlated with many learners' SIDL in fact. However, we also confirmed that the properties of the correlations vary depending on each individual learner. Therefore, in this thesis, we propose to train a SIDL-estimator independently for each learner by SVM and to estimate each learner's SIDL independently by using each trained estimator. In our experiment, we defined the SIDL-estimator as a two class classifier whose target classes are "high SIDL" and "low SIDL," and achieved the estimation accuracy of 75% on an average by the proposed method. This is almost same as the estimation accuracy of the human teachers, which has been confirmed in the previous research focusing on a conventional classroom lecture.

Finally, we tried to realize a method for estimating the learners' SIDL in synchronous type e-Learning. One of the most characteristic points of the synchronous type e-Learning compared with asynchronous type one is that the two-way dialogues can be held between the teacher and each learner on one-to-one level. Due to the changes of the context of this two-way dialogue and the learners' role (speaker or listener), the correlations between the nonverbal behaviors and SIDL will fluctuate in synchronous type e-Learning. In other words, the nonverbal behavior which has a strong correlation with SIDL in a certain dialogue context would not be necessarily correlated with SIDL in another dialogue context. This means we cannot achieve high estimation accuracy if we use the features directly representing the nonverbal behaviors themselves. In order to overcome the above "fluctuation," we propose to use the occurrence frequencies of the nonverbal behaviors within a certain time span as the features for estimating the learners' SIDL in synchronous type e-Learning. We experimentally analyzed the actual correlations between the occurrence frequencies of the nonverbal behaviors and SIDL. As the result, we confirmed that the occurrence frequencies are basically correlated with many learners' SIDL. However, we also confirmed again that the properties of the correlations vary depending on each individual learner. Based on the result of the analysis, we propose again to train a SIDL-estimator independently for each learner by SVM and to estimate each learner's SIDL independently by using each trained estimator. Note that we define the features as not the nonverbal behaviors themselves but the occurrence frequencies of them in this case. In our experiment, the proposed method could estimate the learners "high SIDL" and "low SIDL" with the accuracy of 72% on an average. This is equal to or more than the humans' estimation accuracy, which has been confirmed in another experiment.

Through the results of the above three experiments, we confirmed the following fact; that is, the learners' SIDL can be estimated from their nonverbal behaviors with the accuracy of around 70% if the correlations between the nonverbal behaviors and SIDL are modeled successfully by machine learning techniques. One of the future works is to try to estimate some other kinds of states of mind. Another future work is to use not only nonverbal behaviors but also some other information such as voice or speech information simultaneously for the similar estimation tasks.