Publishing house Radiotekhnika

"Publishing house Radiotekhnika":
scientific and technical literature.
Books and journals of publishing houses: IPRZHR, RS-PRESS, SCIENCE-PRESS

Тел.: +7 (495) 625-9241


Real time method of fingerspelling alphabet to text translation


A.A. Volkova – Student, Department «Software of Computer and Information Technologies», Bauman Moscow State Technical University
S.V. Gorin – Associate Professor, Department «Software of Computer and Information Technologies», Bauman Moscow State Technical University

Fingerspelling alphabet to text translation system can be applied to ease communication between healthy people and people with various degrees of hearing impairment. Translation task is reduced to sign extraction and recognition algorithm implementation. Fingerspelling signs can be divided into two groups: dynamic and static. Hand movement halt is a feature defining static sign. Dynamic signs are divided into two types: movement and rotation. Standard and specialized cameras can be applied to solve the task of recognition. Leap Motion camera is used in this work. Characteristic points coordinates of two neighboring frames are being compared to extract a sign in frames sequence. Analyzing characteristic points coordinates we can identify what kind of gesture is shown at the moment, using one of recognition algorithms. Preference has been given to Neural Networks algorithm because of their thoroughness and applicability in solving this class of problems. From Neural Networks algorithms it is more efficient to use Multilayer Perceptron (MLP) as this network is used in classification, has simple realization, high recognition accuracy, doesn’t require special processing of input data received from Leap Motion controller. Therefore, the developed method is a combination of the following methods:
1. source image characteristic points coordinates extraction
2. corresponding sign determination using characteristic points coordinates
3. presence and type of movement determination by characteristic points coordinates changes
4. shown symbol recognition
Two possible camera’s locations have been considered: human to human, camera is below and human to camera. In first case 14 letters haven't been captured against 11 letters in second case. As recognition method is based on characteristic points coordinates and doesn’t depend on hand orientation, hand position has been changed in order to increase the number of captured letters. A sample of 30 people has been collected for MLP training. People with different degrees of fluency in sign language/dactylology have taken part in sampling. The experiments showed insufficient performance of hardware implementation of hand's characteristics extraction, which can create some inconvenience when working with system. Dynamic signs cause greater difficulties.

  1. Gejl'man I.F. Daktitologiya. L.: Leningradskij vosstanovitel'ny'j czentr VOG. 1981. S. 47−76.
  2. Ethnologue – Languages of the World. URL = (data obrashheniya 18.03.2017).
  3. Anderson L., Peterson D. A comparison of some American, British, Australian, and Swedish signs: evidence on historical changes in signs and some family relationships of sign languages. 1979.
  4. Zajczeva G.L. Zhestovaya rech'. Daktilologiya: Uchebnik dlya vuzov. M.: Gumanitar. izdatel'skij czentr Vlados. 2015. S. 24−27.
  5. Leap Motion. URL = (data obrashheniya 27.03.2017).
  6. Volkova A.A. Metod perevoda daktil'ny'x zhestov v tekst v real'nom masshtabe vremeni (diplomnaya rabota). M.: 2017. 41 s.
  7. Berkinblit M.B. Nejronny'e seti. M.: MIROS i VZMSh RAO. 1993.
  8. Makiko F., Yu I., Masami T., Kazuki J. Sign Language Recognition using Leap Motion Controller // Int’l Conf. Par. and Dist. Proc. Tech. and Appl. PDPTA’15.
  9. Kai-Yin F., Nuwan G., Chi-Tsun C., Chi K. A Real-Time ASL Recognition System Using Leap Motion Sensors // 2015 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. 2015. P. 411−414.
  10. Jie H., Wengang Z., Houqiang L., Weiping L. Sign language recognition using real-sense // 2015 IEEE China Summit and International Conference on Signal and Information Processing (ChinaSIP). 2015. P. 166−170.
  11. Jiangqin W., Wen G. The Recognition of Finger-Spelling for Chinese Sign Language // Gesture and Sign Language in Human-Computer Interaction. GW 2001. Lecture Notes in Computer Science. V. 2298.
  12. Deepali N., Milind K. Real time sign language recognition using the leap motion controller // 2016 International Conference on Inventive Computation Technologies (ICICT). 2016.
  13. Mohandes M., Aliyu S., Deriche M. Arabic sign language recognition using the leap motion controller // IEEE 23rd International Symposium on Industrial Electronics (ISIE). 2014. P. 960−965.
  14. Chernen'kij V.M. Algoritmicheskij metod opisaniya diskretny'x proczessov funkczionirovaniya sistem // Informaczionno-izmeritel'ny'e i upravlyayushhie sistemy'. 2016. № 12. S. 11−21.

© Издательство «РАДИОТЕХНИКА», 2004-2017            Тел.: (495) 625-9241                   Designed by [SWAP]Studio