TOP  >  Thesis/Dissertation  >  Analyzing the relationship between learners' comprehension and behavior based on IRT

Analyzing the relationship between learners' comprehension and behavior based on IRT

FD (Faculty Development) for higher education has spread to Japanese universities recently. Many universities evaluate their lectures as part of FD, and lecture evaluation is often regarded as the important part for FD. When evaluating lectures with videos, people generally pay attention only to the way of teaching; however, it is necessary to pay attention more to the learners' response to the instructor's teaching actions in lecture evaluation because a lecture is a kind of communication between an instructor and learners intrinsically. The learners' response to the instructor's teaching actions is observed as their behavior. Assuming that there is a relationship between learners' behavior and their comprehension, we call this relationship ``behavior-comprehension relationship'' in the latter part of this paper. The behavior-comprehension relationship is helpful for lecture evaluation since learners' comprehension can be a criterion for lecture evaluation. In this paper, we try to clarify the behavior-comprehension relationship. To perform this try, we observe the learners' behavior and classify learners by clustering them along to their behavior. We also conduct questionnaire surveys and quizzes in order to measure the learners' comprehension. Finally, we analyze the behavior-comprehension relationship using the clustering result. For the above analysis, we need to measure the objective comprehension of learners; however, the measured comprehension includes a kind of error factors such as individual differences of learners and the differences of quiz difficulty. Therefore, we propose a method for eliminating the error factors with Item Response Theory (IRT). IRT is a pedagogical theory about exam. It fits the relationship between the accuracy rate of an exam and the objective comprehension of applicants to a logistic function. The logistic function enables us to eliminate the differences of quiz difficulty. IRT parameterizes the difficulty of exam. The difficulty parameter decides the form of the logistic function. We also parameterize the individual differences of learners, assuming that the objective comprehension of learners bears a linear relationship to the answer of questionnaire surveys. We estimate both of the individual differences parameters and the difficulty parameters at the same time by fitting the relationship between the accuracy rate of quiz and the answers of questionnaire surveys to the logistic function. The estimated values of these parameters enable us to eliminate the individual and difficulty differences. In the experiments, we carried out the questionnaire surveys and the quizzes in five lectures. In those lectures we carried out 13 pairs of questionnaire survey and quiz to 13 different learners and got 73 data in total. In each pair of the experiments, we used one five-rank questionnaire survey and five true-false quizzes. We classified learners by using k-means method. In that classification, we represented their behavior quantitatively based on frequency of behavior, which was grouped by hand into ``active behavior (Active)'', ``passive behavior (Passive)'', ``loose behavior (Loose)'', ``deviated behavior (Deviated)'' and ``PC inspection and operation (PC)''. We eliminated the individual differences of learners and the differences of quiz difficulty by our proposed method. Our proposed method successfully fitted the relationship between the accuracy rate of quizzes and the answers of questionnaire surveys to the logistic functions. We confirmed the effectiveness of our proposed method by comparing with the result without eliminating the individual and difficulty differences. Next, we analyzed the behavior-comprehension relationship using the clustering result. As the result of this analysis, we discovered the following three types of knowledge: ``Passive, PC and Active have positive correlation with the learners' comprehension,'' ``Loose does not have correlation with learners' comprehension much,'' and ``Deviated has negative correlation with learners' comprehension.'' we compared the differences between the clusters, and discovered the following two types of knowledge: ``The correlation between Deviated and learners' comprehension is not unique,'' and ``PC has negative correlation with learners' comprehension when it appears with Passive and Deviated.'' As a future work, we will analyze the behavior-comprehension relationship in various classes, and examine the generality of the above knowledge.