Sequential particle filter with covariance features classified with artificial neural nets for continuous Indian sign language recognition

  • Authors

    • P Praveen Kumar
    • P V.G.D. Prasad Reddy
    • P Srinivasa Rao
    2017-12-21
    https://doi.org/10.14419/ijet.v7i1.1.10163
  • Sequential particle filter tracker, Indian sign language recognition, Covariance features, Artificial neural networks.

  • Machine translation of sign language is a complex and challenging problem in computer vision research. In this work, we propose to handle issues such as hands tracking, feature representation and classification for efficient interpretation of sign language from isolated sign videos. Hands tracking is attempted in a sequential format with one hand after the other by nullifying the effects of head movement using serial particle filter. The estimated hand positions in the video sequence are used to extract the hand portions to create a feature covariance matrix. This matrix is a compact representation of the hand features representing a sign. Adaptability of the feature covariance matrix is explored in developing relationships with new signs without creating a new feature matrix for individual signs. The extracted features are then applied to a neural network classifier which is trained with error backpropagation algorithm. Multiple experiments were conducted on a 181 class signs with 50 sentence formations with 5 different signers. Experimental results show the proposed sequential hand tracking is closer to ground truth. The proposed covariance features resulted in a classification accuracy of 89.34% with the neural network classifier.

  • References

    1. [1] Ong, Sylvie CW, and Surendra Ranganath. "Automatic sign language analysis: A survey and the future beyond lexical meaning." IEEE Transactions on Pattern Analysis & Machine Intelligence 6 (2005): 873-891.

      [2] Liang, Rung-Huei, and Ming Ouhyoung. "A real-time continuous gesture recognition system for sign language." In Automatic Face and Gesture Recognition, 1998. Proceedings. Third IEEE International Conference on, pp. 558-567. IEEE, 1998.

      [3] Mitra, Sushmita, and Tinku Acharya. "Gesture recognition: A survey." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, no. 3 (2007): 311-324.

      [4] Oz, Cemil, and Ming C. Leu. "Recognition of finger spelling of American sign language with artificial neural network using position/orientation sensors and data glove." In International Symposium on Neural Networks, pp. 157-164. Springer, Berlin, Heidelberg, 2005.

      [5] Oz, Cemil, and Ming C. Leu. "American Sign Language word recognition with a sensory glove using artificial neural networks." Engineering Applications of Artificial Intelligence 24, no. 7 (2011): 1204-1213.

      [6] Neelesh SARAWATE, Ming Chan LEU, CemilOZ “A real-time American Sign Language word recognition system based on neural networks and a probabilistic model†in Turkish Journal of Electrical Engineering & Computer Sciences, vol.23, pp 2107 2123, 2015.

      [7] P.V.V.Kishore, S.R.C.Kishore, M.V.D.Prasad “Conglomeration of hand shapes and texture information for recognizing gestures of Indian sign language using feed forward neural networks†International Journal of engineering and Technology, Vol. 5, No. 5, pp.3742-3756, 2013.

      [8] P.V.V.Kishore ,M.V.D.Prasad“Optical Flow Hand Tracking and Active Contour Hand Shape Features for Continuous Sign Language Recognition with Artificial Neural Networks†International Journal of Software Engineering and Its Applications,v. 9, no. 12, pp. 231-250, 2015.

      [9] Tzuu-Hseng S. Li, Min-Chi Kao, and Ping-Huan Kuo “Recognition System for Home-Service-Related Sign Language Using Entropy-Based K-Means Algorithm and ABC-Based HMM†in IEEE transactions on systems, man, and Cybernetics: systems, vol.46, no.1, pp. 150-162.

      [10] Li-Chun Wang, Ru Wang, De-Hui Kong, Bao-Cai Yin, “Similarity Assessment Model for Chinese Sign Language Videos†IEEE Transactions On Multimedia, vol. 16, no. 3, pp. 751-761, 2014.

      [11] P. V. V. Kishore, A. S. C. S. Sastry and A. Kartheek, "Visual-verbal machine interpreter for sign language recognition under versatile video backgrounds," 2014 First International Conference on Networks & Soft Computing (ICNSC2014), Guntur, 2014, pp. 135-140.

      [12] Kishore, P. V. V., M. V. D. Prasad, Ch Raghava Prasad, and R. Rahul. "4-Camera model for sign language recognition using elliptical fourier descriptors and ANN." In Signal Processing And Communication Engineering Systems (SPACES), 2015 International Conference on, pp. 34-38. IEEE, 2015.

      [13] Rao, G. Ananth, and P. V. V. Kishore. "Sign language recognition system simulated for video captured with smart phone front camera." International Journal of Electrical and Computer Engineering 6, no. 5 (2016): 2176.

      [14] P. V. V. Kishore, M. V. D. Prasad, D. A. Kumar and A. S. C. S. Sastry, "Optical Flow Hand Tracking and Active Contour Hand Shape Features for Continuous Sign Language Recognition with Artificial Neural Networks," 2016 IEEE 6th International Conference on Advanced Computing (IACC), Bhimavaram, 2016, pp. 346-351.

      [15] D. A. Kumar, P. V. V. Kishore, A. S. C. S. Sastry and P. R. G. Swamy, "Selfie continuous sign language recognition using neural network," 2016 IEEE Annual India Conference (INDICON), Bangalore, 2016, pp. 1-6.

      [16] Almeida, Sílvia Grasiella Moreira, Frederico Gadelha Guimarães, and Jaime Arturo Ramírez. "Feature extraction in brazilian sign language recognition based on phonological structure and using rgb-d sensors." Expert Systems with Applications 41, no. 16 (2014): 7259-7271.

      [17] Li, Shao-Zi, Bin Yu, Wei Wu, Song-Zhi Su, and Rong-Rong Ji. "Feature learning based on SAE–PCA network for human gesture recognition in RGBD images." Neurocomputing 151 (2015): 565-573.

      [18] Chai, Xiujuan, Guang Li, Xilin Chen, Ming Zhou, Guobin Wu, and Hanjing Li. "Visualcomm: A tool to support communication between deaf and hearing persons with the kinect." In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility, p. 76. ACM, 2013.

      [19] Geng, Lubo, Xin Ma, Haibo Wang, Jason Gu, and Yibin Li. "Chinese sign language recognition with 3D hand motion trajectories and depth images." In Intelligent Control and Automation (WCICA), 2014 11th World Congress on, pp. 1457-1461. IEEE, 2014.

      [20] Nai, Weizhi, Yue Liu, David Rempel, and Yongtian Wang. "Fast hand posture classification using depth features extracted from random line segments." Pattern Recognition 65 (2017): 1-10.

      [21] Zhang, Zhengyou, and Alexey Vladimirovich Kurakin. "Dynamic hand gesture recognition using depth data." U.S. Patent 9,536,135, issued January 3, 2017.

      [22] P. Praveen Kumar, P. V. G. D. Prasad Reddy and P. Srinivasa Rao, “Sign language recognition with multi feature fusion and Adaboost classifierâ€, ARPN Journal of Engineering and Applied Sciences, vol.13, no.4, Feb(2018).

      [23] Shan, Caifeng, Tieniu Tan, and Yucheng Wei. "Real-time hand tracking using a mean shift embedded particle filter." Pattern recognition 40, no. 7 (2007): 1958-1970.

      [24] Mitra, Sushmita, and Tinku Acharya. "Gesture recognition: A survey." IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, no. 3 (2007): 311-324.

      [25] Rautaray, Siddharth S., and Anupam Agrawal. "Vision based hand gesture recognition for human computer interaction: a survey." Artificial Intelligence Review 43, no. 1 (2015): 1-54.

      [26] Lim, Kian Ming, Alan WC Tan, and Shing Chiang Tan. "A feature covariance matrix with serial particle filter for isolated sign language recognition." Expert Systems with Applications 54 (2016): 208-218.

      [27] Viola, Paul, and Michael Jones. "Rapid object detection using a boosted cascade of simple features." In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, vol. 1, pp. I-I. IEEE, 2001.

      [28] Piccardi, Massimo. "Background subtraction techniques: a review." In Systems, man and cybernetics, 2004 IEEE international conference on, vol. 4, pp. 3099-3104. IEEE, 2004.

      [29] Schalkoff, Robert J. Artificial neural networks. Vol. 1. New York: McGraw-Hill, 1997.

      [30] Awad, George, Junwei Han, and Alistair Sutherland. "A unified system for segmentation and tracking of face and hands in sign language recognition." In Pattern Recognition, 2006. ICPR 2006. 18th International Conference on, vol. 1, pp. 239-242. IEEE, 2006.

      [31] Soontranon, N., Supavadee Aramvith, and Thanarat H. Chalidabhongse. "Improved face and hand tracking for sign language recognition." In Information Technology: Coding and Computing, 2005. ITCC 2005. International Conference on, vol. 2, pp. 141-146. IEEE, 2005.

      [32] Shukla, Pushkar, Abhisha Garg, Kshitij Sharma, and Ankush Mittal. "A DTW and Fourier Descriptor based approach for Indian Sign Language recognition." In Image Information Processing (ICIIP), 2015 Third International Conference on, pp. 113-118. IEEE, 2015.

      [33] Kaur, Gurwinder, and Gourav Bathla. "Hand Gesture Recognition based on Invariant Features and Artifical Neural Network." Indian Journal of Science and Technology 9, no. 43 (2016).

      [34] Fu, Xingang, Jiang Lu, Ting Zhang, Chadwell Bonair, and Marvin L. Coats. "Wavelet Enhanced Image Preprocessing and Neural Networks for Hand Gesture Recognition." In Smart City/SocialCom/SustainCom (SmartCity), 2015 IEEE International Conference on, pp. 838-843. IEEE, 2015.

      [35] Kim, Taehwan, Weiran Wang, Hao Tang, and Karen Livescu. "Signer-independent fingerspelling recognition with deep neural network adaptation." In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pp. 6160-6164. IEEE, 2016.

      [36] Starner, Thad, and Alex Pentland. "Real-time american sign language recognition from video using hidden markov models." In Motion-Based Recognition, pp. 227-243. Springer, Dordrecht, 1997.

      [37] Li, Xingyan. "Gesture recognition based on fuzzy C-Means clustering algorithm." Department Of Computer Science The University Of Tennessee Knoxville (2003).

      [38] P. V. V. Kishore, D. A. Kumar, A. S. C. S. Sastry and E. K. Kumar, "Motionlets Matching with Adaptive Kernels for 3D Indian Sign Language Recognition," in IEEE Sensors Journal, vol. PP, no. 99, pp. 1-11. doi: 10.1109/JSEN.2018.2810449.

  • Downloads

  • How to Cite

    Praveen Kumar, P., V.G.D. Prasad Reddy, P., & Srinivasa Rao, P. (2017). Sequential particle filter with covariance features classified with artificial neural nets for continuous Indian sign language recognition. International Journal of Engineering & Technology, 7(1.1), 539-547. https://doi.org/10.14419/ijet.v7i1.1.10163