Linguistically-driven framework for computationally efficient and scalable sign recognition

Files
1069_Paper.pdf(3.34 MB)
Published version
Date
2018
DOI
Authors
Metaxas, Dimitris N.
Dilsizian, Mark
Neidle, Carol
Version
OA Version
Citation
Dimitris N Metaxas, Mark Dilsizian, Carol Neidle. 2018. "Linguistically-driven Framework for Computationally Efficient and Scalable Sign Recognition.." Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)
Abstract
We introduce a new general framework for sign recognition from monocular video using limited quantities of annotated data. The novelty of the hybrid framework we describe here is that we exploit state-of-the art learning methods while also incorporating features based on what we know about the linguistic composition of lexical signs. In particular, we analyze hand shape, orientation, location, and motion trajectories, and then use CRFs to combine this linguistically significant information for purposes of sign recognition. Our robust modeling and recognition of these sub-components of sign production allow an efficient parameterization of the sign recognition problem as compared with purely data-driven methods. This parameterization enables a scalable and extendable time-series learning approach that advances the state of the art in sign recognition, as shown by the results reported here for recognition of isolated, citation-form, lexical signs from American Sign Language (ASL).
Description
License