论文标题
在连续视频中识别美国手语非手册信号语法错误
Recognizing American Sign Language Nonmanual Signal Grammar Errors in Continuous Videos
论文作者
论文摘要
作为开发一种教育工具的一部分，该工具可以通过立即反馈来帮助学生通过独立和互动实践来获得流利的美国手语（ASL），本文介绍了一个近乎实时的系统，以识别连续签名视频中的语法错误，而无需确定整个标志序列。我们的系统自动认识到ASL句子的性能是否包含ASL学生犯的语法错误。我们首先通过3D-RESNET网络识别出独立于多种方式（即手势，面部表情和头部运动）独立于多种方式（即手动手势）和非手动信号的ASL语法元素。然后检查了来自不同方式的语法元素的时间边界，以使用基于滑动窗口的方法来检测ASL语法错误。我们收集了一个连续手语的数据集ASL-HW-RGBD，涵盖了用于培训和测试的ASL语法的不同方面。我们的系统能够从手动手势，面部表情和头部运动中识别ASL-HW-RGBD上的语法元素，并成功地检测出8个ASL语法错误。
As part of the development of an educational tool that can help students achieve fluency in American Sign Language (ASL) through independent and interactive practice with immediate feedback, this paper introduces a near real-time system to recognize grammatical errors in continuous signing videos without necessarily identifying the entire sequence of signs. Our system automatically recognizes if performance of ASL sentences contains grammatical errors made by ASL students. We first recognize the ASL grammatical elements including both manual gestures and nonmanual signals independently from multiple modalities (i.e. hand gestures, facial expressions, and head movements) by 3D-ResNet networks. Then the temporal boundaries of grammatical elements from different modalities are examined to detect ASL grammatical mistakes by using a sliding window-based approach. We have collected a dataset of continuous sign language, ASL-HW-RGBD, covering different aspects of ASL grammars for training and testing. Our system is able to recognize grammatical elements on ASL-HW-RGBD from manual gestures, facial expressions, and head movements and successfully detect 8 ASL grammatical mistakes.
