The use of digital technology within homes is increasing, while the latest technological and digital breakthroughs allow significant advancement in this field. This progress creates many opportunities by designing an augmented reality interface for a mirror. We will see how to integrate this interface to an augmented mirror fully. Its use has to be natural and serve as a basis for developing platforms such as sports training, sign language, or medical monitoring of drug intake.
Contribution
The short-term objective was relatively straightforward. First, the mirror has to retrieve the person's position in front of the mirror. It then has to calculate the location of their reflection and display this reflection on the mirror. This way features can eventually be implemented, such as adding a silhouette on the screen.
This kind of coach would be performing a series of exercises that the user would then try to reproduce.
Solution
To retrieve the person's position, they have to be identified on the image: a pose estimation model is used to retrieve the points of interest of the body, the face, and the hands (Google, 2020[9]).
The coordinates of these virtual points had to be converted into real-world coordinates. The Intel D435i stereoscopic camera (Intel RealSense, 2021[10]) was used to know their distance from the screen. The program then calculates the depth of each pixel in the image. Once it obtained the distances, these coordinates were converted to match what the user sees in the reflection.
All these steps are done in a Docker container for compatibility and deployability in case of platform change. It was necessary to send these coordinates to a server outside the container, which retransmits them to a web page on which the results are then displayed.
The module carries out a data augmentation to enrich this dataset and get better results. It applies shift and scale augmentation. Thus the program multiplies the size of our dataset randomly and at will.
At the time of the program's launching, the user sees a short video of a person, which makes him a demonstration of a word in sign language.
He must then repeat this movement by having a return on the device’s camera. He can see in real time the points detected on his body. When he has validated the action, a second tutorial appears about another movement, and he must do it again, and so on.
Minigames to test the pose estimation
"Poke it", "Bounce" and "Danse" are minigames that use the pose estimation feature of the mirror to work. The first requires the user to touch a number of circles with their right hand within a set time. The second displays a bouncing ball that the user can interact with. "Dance" is a module displaying the video of a coach whose movements the user can follow. The video progresses the more he manages to do the right moves.
Dance
Poke It
Bounce
Conclusion
Second Self aims to become a platform on which an application can be easily developed. This mirror opens up a new range of possibilities. It becomes a new media that offers a new way to interact with a user. It represents an excellent base on which we can build new modules related to all types of fields: health, fashion, education, sports, and many others. In short, our mirror is gradually becoming the modular platform of our dreams.
References:
Teeuw, M. (2016). MagicMirror2. Xonay Media. ↩︎
Etienne (2016, January 24). Magic Mirror : Voir l’heure dans un miroir connecté? Cuboak. ↩︎
Bar-Noye, N. (2015, June 16). Innovation israélienne : Miroir gentil miroir, dis-moi qui est la plus belle ? AlianceFR.com. ↩︎
HA, J. (2018, November 15). Coming soon: A revolutionary smart mirror by Panasonic. Medium. ↩︎
Sali, K. (2018, February 12). Le miroir augmenté d’Amazon pour conquérir le marché de la mode. L’echommerces. ↩︎
T. Blum, V. Kleeberger, C. Bichlmeier and N. Navab, "mirracle: An augmented reality magic mirror system for anatomy education," 2012 IEEE Virtual Reality Workshops (VRW), Costa Mesa, CA, USA, 2012, pp. 115-116, doi: 10.1109/VR.2012.6180909. ↩︎
Vera L., Gimeno J., Coma I., Fernández M. (2011) Augmented Mirror: Interactive Augmented Reality System Based on Kinect. In: Campos P., Graham N., Jorge J., Nunes N., Palanque P., Winckler M. (eds) Human-Computer Interaction – INTERACT 2011. INTERACT 2011. Lecture Notes in Computer Science, vol 6949. Springer, Berlin, Heidelberg. ↩︎
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13). Association for Computing Machinery, New York, NY, USA, 311–320. DOI: ↩︎
Google. (2020). Holistic. Mediapipe. ↩︎
