Around 140,000 people in Germany are dependent on sign language interpreters. But interpreters are not always available. Marius Martin and Michael Darmstadt, graduates of the dual degree program in business informatics at Mainz University of Applied Sciences, wanted to find out whether it was possible to create a sign recognition system suitable for everyday use. The goal was that no extra hardware should be necessary for its use and that the system should allow fluent communication between deaf and hearing people with the help of technical support. Marius and Michael therefore researched a sign recognition system using a simple webcam and neural network. This was a challenge because sign language is complex and, like other languages, differs from country to country and region to region.

Approach

• Use of the Google framework MediaPipe for hand and face recognition
• Creation of training videos for training a neural network (over 3,000 videos from a total of 12 participants). For this purpose, they have created a website that allows even non-professionals to record such videos on the PC.
• Extraction of the position data of the hands and face for each frame of the videos into CSV files.
• Training of a neural network: Using Keras and Weights & Biases + Google Cloud resources, various configurations of neural networks were tested. Approx. 70 days of computing time.
• Building a prototype based on the MediaPipe framework that can be compiled for Linux and later times for mobile devices.
• The prototype is able to recognize 30 different signs of the German sign language. In a controlled environment, they achieved 96.7% accuracy.

Prize of the University Council

The practical orientation and the application of cutting-edge technologies convinced the University Council of the Mainz University of Applied Sciences, which awarded the work a prize. The prototype forms a good basis for further work in the field, which in the future could lead to an interpreter in your pocket that enables easy communication.

From DATAGROUP, Marius received special hardware with which neural networks can be trained, because not every standard computer has the necessary computing power. After graduation, he continues to work at DATAGROUP and is currently employed in the Media division as a consultant and software developer. “The dual study program at DATAGROUP gave me the opportunity to combine theory with practice,” explains Marius. “That’s exactly how we approached our final thesis. It’s nice that I was supported in this at DATAGROUP and still am today when I want to develop further and familiarize myself with new areas.”