Hand Gesture and Pose Recognition using Deep Learning for Hygiene Monitoring

Autor*in: Armin Lennart Dietz

ISBN: 978-3-95900-381-0

Dissertation, Leibniz Universität Hannover, 2019

Herausgeber*in der Reihe: Eduard Reithmeier

Band-Nr.: IMR 04/2019

Umfang: 152 Seiten, 49 Abbildungen

Schlagworte: Hand hygiene monitoring, disinfection, vision-based system, gesture, action, pose recognition, hand detection, deep learning, Handhygiene Monitoring

Kurzfassung: Hand hygiene is widely considered to be the most effective measure to prevent healthcare associated infections. The World Health Organization recommends a hand rubbing process, consisting of nine hand gestures, to ensure that all areas of the hands are disinfected. Nevertheless, compliance with these recommended instructions for hand hygiene disinfection is low.

In this thesis, an automated optical system to monitor the hand hygiene disinfection process using deep learning is developed. The system consists of  a single depth sensor and three deep neural networks, which are the HandNet, GestureNet and PoseNet. The HandNet is developed to segment individual hand regions from interacting hands and from background scenes. To obtain substantial training data with labeled depth images, a method is presented to automatically assign each image pixel to a class of three hand regions for each hand or to a background class. Image transformations and background swaps further increase the variation of the training data. Based on the segmented image data, the nine gestures of the hand rubbing process and an additional error gesture are classified by the GestureNet. An approach to localize disinfected regions on the hands is proposed. The PoseNet is trained with synthetic depth data of hands and corresponding 3D positions of hand joints. The joints’ positions are then predicted based on depth data and two synthetic 3D hand models are fitted in. By detecting and observing contacted regions of the hand models over time, the assumed local degree of disinfection is visualized.

More than 180 000 images of 39 different subjects are recorded to train and evaluate the neural networks. The results show that fine details of the hand regions are robustly segmented from highly occluded hands and cluttered background scenes. It is also shown, that the data augmentation method is essential for a robust hand detection. The influence of different input modalities to the GestureNet is evaluated quantitatively and qualitatively. The  results show that segmentation masks from the HandNet in addition to the depth data as input to the GestureNet increases the robustness of predictions. For the classification of the ten gestures from a single image, an accuracy of 98% for correct predictions is achieved. It is further shown that the localization of disinfected hand areas is feasible. The presented monitoring system is thus able to inform the user online about the state of disinfection.