Heading
My project was to create a remote visual and audio communication between a person "A" and an other "B" in augmented reality. "B" has a Hololens 2 augmented reality headset and is put in contact with a remote advisor. Using a simple browser, "B" sees a 3D animated avatar of the advisor and can switch to a AR session on his headset. The avatar is supposed to follow certain movements of "A", which are necessary for body expression. The two can chat with each other through a text chat as well as an audio chat.
Related works
More and more communication networks (teams, zooms) are implementing new features in remote conferences. Like the fact that users may not want to share their wallpaper with others, they may not want to share their camera, they can sometimes virtually replace themselves with 3D avatars. For example, the market leaders Teams and Zoom are focusing on this functionality. As of the first half of 2022 for Teams and in the latest available version of Zoom.
Almost every major tech company is rushing to develop a VR or AR headset, believing that what has long been a niche market may be about to grow exponentially. The adoption of Virtual Reality (VR) headsets has accelerated in 2020 and 2021, thanks in part to the strong performance of the Oculus Quest 2.
Realisations:
The project is a remote discussion program. "A" launches a session from a simple browser. "B" connects to the same url address with an augmented reality headset. By pressing "START AR", an augmented reality session opens.
"B" sees an animated avatar that follows "A"'s head, eye and mouth movements. The avatar follows some of "A"'s movements, tilts and rotates his head, shoulders, makes certain facial expressions, follows his mouth movements, the direction of his pupils and his eyelashes. The two users can send text messages to each other and view them, as well as mute each other to speak in audio.
"B" can also view "A"'s avatar and communicate directly on a website without an augmented reality headset, without launching an AR session.
Interest :
Expressing body language and facial expressions without appearing on camera. It also brings a more "fun" aspect to a conversation. The vision of this project is above all to research and innovate in order to rethink our communication and visualisation tools. Some elements could be implemented later, for example a visualisation and manipulation of objects from the hololens. For example, help with the installation and maintenance of an object.
This project has the advantage of being an immersive experience, a way to hide one's appearance, and a new visualisation tool.
The most important part of the project is the management of the avatar from Mediapipe, it is enabled by the Kalidokit solver (https://github.com/yeemachine/kalidokit). It is a JS library allowing the animation of arms, hands, fingers, face, mouth, eyelids, pupil of VRM models (Virtual Reality Model).
Search for a model
The various models are in VRM format. VRM is formulated on the basis of the standard 3D glTF2.0 format for handling humanoid models. It is intended to be particularly expressive. It has a large number of joints, can blink its eyes and animate its mouth. It is very often used in the world of VR games (VR chat for example), or by Vtubers (entertainment broadcasters who use a virtual avatar).
The models used are downloaded from the website "https://hub.vroid.com/en/". They must respect specific conditions of use: use by a third party, downloadable, use as an avatar, commercial use by a company.
