TOP  >  Thesis/Dissertation  >  Extracting Repetitive Motion on Kitchen Counter with its Location using Load Sensing Table

Extracting Repetitive Motion on Kitchen Counter with its Location using Load Sensing Table

Recently, there has been some research for realizing systems which understand user's activity from observed data and support the users. It is useful for such systems to identify where and when important activities occurred. In case of cooking, motions to process ingredients (processing motions) are important to follow the cooking steps because such motions indicate progress of the cooking steps. It is necessary, at first, to extract the period and the location of each processing motion for the systems to understand what kind of processing motion has been occurred and which ingredient was processed in it.

During the processing motions, such as kneading, cutting, mixing, and smoothing, similar movements are repeatedly observed. We call such processing motions as repetitive motions. The purpose of our method is to extract the period and the location of repetitive motions on kitchen counter.

We observe the repetitive motions by load sensors attached under four corners of a top board. They can observe any physical contacts to the top board during the cooking. Thus it is easy to extract the period of motions touching on the top board by the sensors.

There is a previous method which detects the period and the location of putting/taking motions with load sensors. The period of these motions is detected by thresholding the change of stable load values, and the location of them is estimated as center of gravity of load values. However, in most repetitive motions, nothing is put or taken and thus there is no change of stable load values. Therefore, this method is unsuitable for our goal.

We estimate the location of repetitive motions not from the stable values but from the dynamic values observed during motions. The dynamic value also includes a signal component which does not contribute but disturbs the location estimation. We found that a smoothing filter can cancel this component efficiently. We estimate the locations robustly by this property.

Moreover, a similar pattern of signals is observed repeatedly during a repetitive motion. We extract the repetitive motions based on this property. We investigate three types of features to find such patterns. That is, total weight, center of gravity (COG), and movement of COG. The total weight is the sum of load values observed by the four load sensors. It measures the repetitive pattern of force acting on the top board. The COG is the balance point of the load values and the movement of COG is the difference of COG features observed at two consecutive sampling points. These features measure the repetitive pattern of location.

In the experiment, we evaluated the proposed method on an observation of cooking "boiled dumpling," which requires a variety of repetitive motions: kneading, cutting, mixing filling, and smoothing dough. As a result, we respectively got 88.6%, 87.4%, 83.9% of recall rate, and 87.6%, 76.4%, 80.0% of precision rate for total weight, COG, and movement of COG. The location was estimated with 67.0% of accuracy against 66.4% without the proposed smoothing filter. In addition to that, we also evaluated the motions independently. For kneading and mixing filling, all features work well. For cutting, the results by COG was less accurate than others. In contrast, for smoothing dough, the results by COG was more accurate.

We also evaluated the accuracy of extracting the period and the location of repetitive motions in two short observations of cutting Chinese cabbage and chicken. These motions are expected to be less repetitive than the motions in "boiled dumpling." We got 86.0%, 58.5%, 89.0% of recall rate in cutting Chinese cabbage and 60.0%, 17.3%, 61.0% of recall rate in cutting chicken, for total weight, COG, and movement of COG, respectively.

From the experiments, it turned out that total weight was the best feature for extracting the repetitive motions in entire cooking. It also found that suitable features varied with motions. The motion recognition for the extracted period of repetitive motions is left as a future work.