An Augmented Reality application integrated with hand-tracking to help learn Sign Language
Application Development | Accessibility | Mixed Reality
Tools: Blender, Unity
Device: Magic Leap
Duration: Oct to Dec 2020
Deaf or Hard of Hearing (DHH) individuals choose sign language as their mode of communication, which is different from the spoken language hearing individuals' mode of communication, creating a communication gap between these two communities. As these communities share the same space, like being co-enrolled in academic settings or working as colleagues in the same company, it becomes more than important to reduce the communication gap.
Learning sign language is quite different from learning a spoken language because the spoken language is based on speech-auditory senses, while sign language is based on visual-manual senses. Sign language, along with facial expressions and lip-movements, heavily depend on hand gestures in a 3-dimensional space. Currently available resources like images and videos are restricted to the 2-dimensional domain, making it difficult to master sign language using these resources. Utilizing Mixed Reality can help us break this 2D barrier.
How can we improve the effectiveness of learning sign language using advanced technology?
A viable solution should provide:
An interactive learning method
Remote and constant availablity
Expert supervision not required
Inspired to provide these functionalities, we can look towards using Mixed Reality as a potential solution. The goal of this project is to develop an Augmented Reality (AR) application prototype integrated with Mixed Reality (MR) technologies like hand-tracking.
The application would help the user to visualize hand gestures and movement of a particular sign in AR and would perform hand-tracking to provide real-time feedback on whether the user is reproducing the correct sign or not. To limit the scope for the prototype, this project would focus on helping self-learning British Sign Language (BSL) numbers from 0 to 10.
The application helps users self-learn British Sign Language (BSL) numbers from 0 to 10 by visualizing the signs in AR and the hand-tracking technology provides real-time feedback.
Mixed Reality Device
3D graphics creator
that is 85%
in the next 4 years
According to Global Market Insights, AR could surge up to
people are devoid of auditory senses
Sign Language is the primary mode of communication for individuals from Deaf and Hard-of-Hearing community
A user would wear Magic Leap, a headset-mounted device, and 3D instructional hands would be displayed, in augmented reality, on the screen of the headset. This AR instructional hand would help the user to visualize and guide about a particular sign's hand gesture, and movement in space. The user would now need to reproduce that same gesture, and the headset would perform hand-tracking to recognize the user's hand gesture in real-time to figure out if the user performed the correct gesture or not.
All the signs will be displayed on the home screen. The users would select the sign they want to learn and the sign would open on the next screen.
View Sign Screen
The selected sign gets displayed. Now the user can go to the previous or next sign in the list.
Now the user can view the hand movement of a particular sign by clicking on the “View Animation” button or the user can practice reproducing the sign.
Practice Sign Screen
If the "Practice Sign" button is selected, the system would go to the next screen where the user is asked to reproduce that particular sign. While the user is producing the sign, the system would perform gesture recognition to find if the user has reproduced the correct sign or not.