SI Seminar by Nargiz Humbatova

Software Institute

Date: 6 October 2022 / 16:30 - 17:30

USI Campus EST, room D0.03, Sector D

DeepCrime: Mutation Testing of Deep Learning Systems Based on Real Faults

Speaker: Nargiz Humbatova

Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem. Traditional test adequacy criteria are not effective with DL models, whose behaviour is determined by many factors such as training data, model architecture, a set of various hyperparameters, and only marginally by the source code. Hence, researchers have proposed several novel test adequacy criteria tailored for DL systems, including mutation adequacy (mutation testing). However, the newly proposed mutation testing approaches were mainly based on some random changes to the structure or the weights of an already trained DL model. In one of our recent works, we investigated the possibility to simulate the effects of real faults by means of DL-specific mutation operators. We have defined 35 DL mutation operators relying on 3 empirical studies about real faults in DL systems. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults. We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Finally, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool.

Nargiz is a fourth-year PhD student at TAU group, working on the PRECRIME project under the supervision of Professor Paolo Tonella and Assistant Professor Gunel Jahangirova. She obtained her BSc in Mathematics from Moscow State University in 2012. Then she spent almost five years working as a software developer in a bank in Baku, Azerbaijan. Before joining USI, she got an MSc degree in Advanced Computing from the University of Bristol. Currently, her research interests are focused on the topic of Mutation Testing of Deep Learning Systems, automated repair and fault localisation. 

Chair: Aaron Fiechter

ℹ️  The seminar will be in presence for everyone in room D0.03. If you are unable to attend in presence, the SI website contains a link to a Teams video call in the “location" field (i.e., click on “D0.03"). 


In February 2019, the Software Institute started its SI Seminar Series. Every Thursday afternoon, a researcher of the Institute will publicly give a short talk on a software engineering argument of their choice. Examples include, but are not limited to novel interesting papers, seminal papers, personal research overview, discussion of preliminary research ideas, tutorials, and small experiments.

On our YouTube playlist you can watch some of the past seminars. On the SI website you can find more details on the next seminar, the upcoming seminars, and an archive of the past speakers.

Everyone is welcome to attend the seminars organized by the Software Institute.