Skip to Main content Skip to Navigation
Journal articles

Anatomical augmented reality with 3D commodity tracking and image-space alignment

Armelle Bauer 1, 2 Debanga Raj Neog 3 Ali-Hamadi Dicko 1, 4 Dinesh K. Pai 3 François Faure 1, 4 Olivier Palombi 1, 5 Jocelyne Troccaz 2
1 IMAGINE - Intuitive Modeling and Animation for Interactive Graphics & Narrative Environments
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann, INPG - Institut National Polytechnique de Grenoble
2 TIMC-GMCAO - Gestes Medico-chirurgicaux Assistés par Ordinateur
TIMC - Techniques de l'Ingénierie Médicale et de la Complexité - Informatique, Mathématiques et Applications, Grenoble - UMR 5525
Abstract : This paper presents a mirror-like augmented reality (AR) system to display the internal anatomy of the current user. Using a single Microsoft V2.0 Kinect (later on referenced as the Kinect), we animate in real-time a user-specific model of internal anatomy according to the user’s motion and we superimpose it onto the user’s color map. Users can visualize their anatomy moving as if they where looking inside their own bodies in real-time. A new calibration procedure to set up and attach a user-specific anatomy to the Kinect body tracking skeleton is introduced. At calibration time, the bone lengths are estimated using a set of poses. By using Kinect data as input, the practical limitation of skin correspondence in prior work is overcome. The generic 3D anatomical model is attached to the internal anatomy registration skeleton, and warped on the depth image using a novel elastic deformer subject to a closest-point registration force and anatomical constraints. The noise in Kinect outputs precludes direct display of realistic human anatomy. Therefore, to enforce anatomical plausibility, a novel filter to reconstruct plausible motions based on fixed bones lengths as well as realistic angular degrees of freedom DOFs) and limits are introduced. Anatomical constraints, applied to the Kinect body tracking skeleton joints, are used to maximize the physical plausibility of the anatomy motion while minimizing the distance to the raw data. At run-time, a simulation loop is used to attract the bones towards the raw data. Skinning shaders efficiently drag the resulting anatomy to the user’s tracked motion. Our user-specific internal anatomy model is validated by comparing the skeleton with segmented MRI images. A user study is established to evaluate the believability of the animated anatomy. As an extension of Bauer et al. (2016), we also propose an image-based algorithm that corrects accumulated inaccuracy of the system steps: motion capture, anatomy transfer, image generation and animation. These inaccuracies show up as occlusion and self-occlusion misalignments of the anatomy regions when superimposed between them and on top of the color map. We also show that the proposed work can efficiently reduce these inaccuracies.
Complete list of metadatas

Cited literature [1 references]  Display  Hide  Download
Contributor : Jocelyne Troccaz <>
Submitted on : Monday, November 13, 2017 - 11:16:45 AM
Last modification on : Tuesday, April 28, 2020 - 3:30:05 PM
Document(s) archivé(s) le : Wednesday, February 14, 2018 - 2:28:10 PM



Armelle Bauer, Debanga Raj Neog, Ali-Hamadi Dicko, Dinesh K. Pai, François Faure, et al.. Anatomical augmented reality with 3D commodity tracking and image-space alignment. Computers and Graphics, Elsevier, 2017, 69, pp.140 - 153. ⟨10.1016/j.cag.2017.10.008⟩. ⟨hal-01633632⟩



Record views


Files downloads