Robust Sensor-based Recognition of Human Behavior

Staff - Faculty of Informatics

Date: 6 July 2022 / 16:00 - 18:30


You are cordially invited to attend the PhD Dissertation Defence of Shkurta Gashi on Wednesday 6 July 2022 at 16:00 on Teams.

Personal computing devices – such as smartphones and smartwatches – can continuously and unobtrusively collect sensor data and use it to infer human behavior. Users’ whereabouts, their physical activities, or even their emotions can be inferred from location data, accelerometer readings, or physiological parameters collected using such devices and their built-in sensors. This data-driven recognition of human behavior fuels the development of personal informatics systems, which help people learn about, reflect upon, and possibly change their habits. Such systems include three main logical components. The collection of data from several sensors available in personal devices – which requires the previous identification of measurable expressions of the behavior(s) to be recognized – is related to the sensing component. The modeling component comprises the design of features and behavioral markers that can be computed from the raw sensor data and fed to machine learning algorithms to identify the target behaviors. Finally, the information available about users’ behavior and context is leveraged by the interventions component to provide, e.g., data visualizations, notifications, and recommendations, that have the ultimate goal to help users improve their health, well-being, productivity, and more. Notwithstanding the significant interest in this field from both academic research and industry, the deployment of personal informatics systems in real-life settings is still cumbersome and prone to errors. A first set of challenges stems from the fact that sensor data collected in ambulatory settings through users’ personal devices are often noisy and individual samples or batches of data may be missing due to several technical issues. We refer to challenges posed by noisy data and missing data as sensing challenges. Even when the raw data is free of errors, though, the identification of models to recognize human behavior is hampered by several factors. Sensor data in general – and physiological signals in particular – may vary significantly across subjects and contexts. Further, the practical difficulties related to the collection of ground-truth data about human behavior often leads to a lack of labels to be used for model training and evaluation. We refer to the challenges posed by the individual variability and context-dependency of human behavioral data and to the lack of labeled data as modeling challenges. In this thesis, we address the sensing and modeling challenges described above and provide a set of six novel technical contributions aimed at enabling the robust, sensor-based recognition of human behavior. First, we devise a new methodology to identify and remove noisy data – in particular, artifacts in electrodermal activity data. Second, we investigate the use of imputation strategies to handle missing accelerometer sensor data collected in data traces used for recognizing cooking activities. Third, we investigate the impact of missing data on the recognition accuracy of sleep and wake stages, and rely again on imputation strategies to mitigate the errors caused by incomplete data traces. Fourth, we investigate how population and personalized models influence the performance of sleep detection using sensor data that varies significantly between subjects – in particular electrodermal activity, skin temperature, and accelerometer data. Fifth, we explore the role of social context in human behavior measurement by exploiting the physiological synchrony derived from sensor data to quantify the emotional state during interactions of two persons (e.g., a presenter and an audience member) or of multiple people (e.g., students in a classroom). Sixth, we propose the use of a hierarchical classification approach to discriminate head gestures and facial expressions using accelerometer and gyroscope sensor data collected from ear-mounted devices. Thereby, we explore the impact on the classification performance of a technique called transfer learning, which allows us to cope with the lack of data annotations by using pre-trained machine learning models. To obtain the contributions described above, we collect – and make available to the research community – four data sets. We further ran extensive data analyses on both our own data sets and on others available in the literature. Taken together, the technical contributions and findings described in this thesis can support other researchers and practitioners in the design and implementation of sensor-based systems for human behavior recognition and represent a further step towards making personal informatics systems robust against various sources of errors.

Dissertation Committee

  • Prof. Silvia Santini, Università della Svizzera italiana, Switzerland (Research Advisor)
  • Prof. Fabio Crestani, Università della Svizzera italiana, Switzerland (Internal Member)
  • Prof. Ernst Wit, Università della Svizzera italiana, Switzerland (Internal Member)
  • Prof. Lama Nachman, Intel Labs, USA (External Member)
  • Prof. Kristof Van Laerhoven, University of Siegen, Germany (External Member)