Simulation-Based Testing and Runtime Monitoring for Autonomous Robotic Systems

You are cordially invited to attend the PhD Dissertation Defence of Sajad MazraehKhatiri on Wednesday 10 December 2025 at 15:30 in room D1.14.

Abstract:
The increasing deployment of autonomous robotic systems in critical applications necessitates robust methods for ensuring their safety and reliability. A primary obstacle to this is the “reality gap”, where simulation-based testing, a vital and scalable validation approach, often fails to represent real-world conditions, thus limiting its effectiveness in detecting critical failures before deployment. This dissertation addresses this challenge by proposing and validating a holistic framework for robotic system assurance, composed of two complementary layers: pre-deployment testing and runtime safety monitoring. For pre-deployment assurance, this thesis introduces a novel search-based approach, Surrealist, which leverages operational data to first replicate real-world behaviors in simulation with high fidelity and then generate new challenging and realistic test scenarios that expose safety-critical failures. The effectiveness of this approach is first demonstrated on Unmanned Aerial Vehicles (UAVs), where it automatically discovers critical failure modes. The entire testing workflow is automated and orchestrated by Aerialist, a modular and scalable test bench developed as a core contribution of this dissertation. The generalizability and practical value of the pre-deployment framework are then confirmed through a comprehensive industrial case study at ANYbotics. The framework was successfully adapted from UAVs to the ANYmal quadrupedal robot and integrated into their development workflow. It proved highly effective at uncovering algorithm deficiencies missed by manual methods and provided an objective, repeatable benchmark for comparing software versions, leading to its adoption as an essential pre-release validation gate. This initial success spurred the framework’s further evolution into the company’s MLOps pipeline, enabling rapid, large-scale benchmarking of new ML models by reducing test suite execution times from hours to minutes. To complement pre-deployment testing, this dissertation introduces Superialist, a lightweight, black-box runtime monitor. Its design is grounded in a large-scale empirical study that established a quantifiable, moderate-to-strong correlation between observable “decision uncertainty” and subsequent safety violations, revealing that up to 89% of unsafe states are preceded by anomalous navigation patterns. Superialist uses an autoencoder to detect these anomalous patterns in real-time with up to 96% precision, serving as an effective early warning system that identifies impending safety hazards up to 50 seconds in advance. The contributions of this thesis provide a comprehensive, data-driven methodology to bridge the reality gap, enhancing robotic system safety through rigorous pre-deployment testing and real-time runtime monitoring. This work has also fostered broader community research through the establishment of the international UAV Testing Competition, which is built upon the frameworks developed in this dissertation.

Dissertation Committee:
- Prof. Paolo Tonella, Università della Svizzera italiana, Switzerland (Research Advisor)
- Prof. Sebastiano Panichella, University of Bern, Switzerland (Research co-Advisor)
- Prof. Piotr Krzysztof Didyk, Università della Svizzera italiana, Switzerland (Internal Member)
- Prof. Carlo Alberto Furia, Università della Svizzera italiana, Switzerland (Internal Member)
- Prof. Aitor Arrieta, Mondragon University, Spain (External Member)
- Prof. Sebastian Elbaum, University of Virginia, USA (External Member)