Vision
In order to enable a user to learn a subject, one of the best techniques for retaining information with an interface is gamification. It captures the attention of the player and increases his motivation to learn. This is why I wanted to realize a short video game to teach sign language on an interactive mirror.
Project
The sign language video game is an augmented reality game, a visual novel on the augmented mirror. We follow a character during a short adventure in which the user can take choices by making signs in front of the mirror. He can answer to the characters, interact with objects and choose a path.
To make the sprites and animations, I improved my work on the interactive 3D avatar that I developed for the augmented reality communication. I then recorded signs and poses for the characters and tutorials of the game. I implemented the Kalidokit module using mediapipe to make the avatar follow the movements of a user. I also implemented the module on the augmented mirror in an application on its own.
The program allows an easy the creation of the dataset, implementing new models, visualization of training results, as well as a dynamic visualisation of recorded movements. The model is composed of a bidirectional LTSM, a linear, a dropout, a batchnorm1d, a relu, and an other Linear. The AI trained on 16 different signs or 1600 sequences. I reach an accuracy of 87% on the test set and 96% on the training.
I then implemented the IA into the game engine to add control with motion. The data is transmitted from the slr module through gosai’s redis database to the game engine. When the probability of a detected sign exceeds a threshold, the sign is considered as validated by the engine.
I needed a game engine to create a visual novel using P5. I started with the implementation of P5VN in gosai. The engine is basically adaptable for the mirror because it is basically working with p5 and the code was not hard to handle.
I almost completely transformed it to run asynchronously to adapt it to gosai, to allow it to load and play video animations, I adapted it to interact and select menus thanks to sign language and added many other features.
Interest
The interest is in stimulating the motivation of a user to learn sign language, especially while their brain is still young and they have more facility in learning a language. The advantage over some existing games is that the gestures are estimated by computer vision, which makes the application much less restrictive than with equipment like datagloves.
These augmentations on a mirror are good ways to encourage performance, the human ability to link a mental image (a mental sound, a word in your head) to a physical activity (a real sound, a gesture).
These type of applications are motivating, entertaining, and the user practices through meaningful activities.
