Testing RAG Systems: Coverage-Guided Adequacy and Metamorphic Oracles
Software Institute
Date: 30 April 2026 / 17:00 - 18:00
USI East Campus, Room D1.13
Speaker: Jinhan Kim, USI
Abstract: Retrieval-Augmented Generation (RAG) systems have become the default way we deploy LLMs over proprietary knowledge, yet the way we test them has not kept pace. Most existing frameworks score answers one query at a time against a fixed snapshot of the corpus, which is useful for benchmarking, but a long way from testing the system as a whole. In this talk I want to take a software-testing view of RAG systems and ask two questions:
First, is our test suite actually exercising the retriever, or are we just hitting the same popular chunks over and over? I’ll introduce Chunk Coverage, an oracle-independent adequacy criterion. Second, what happens when the corpus moves under your feet: updates, stale versions, OCR noise, format drift? I’ll talk about metamorphic testing for RAG.
Biography: Jinhan Kim is a postdoc at TAU lab in Università della Svizzera italiana (USI) led by Prof. Paolo Tonella. He completed his Ph.D. degree from KAIST under the supervision of Prof. Shin Yoo. His research bridges Software Engineering (SE) and Artificial Intelligence (AI), focusing on testing and reliability of AI systems. He develops principled methods to assess and improve the robustness of complex systems, from traditional software to LLMs, aiming to make them reliable and transparent for deployment in safety-critical domains. More information is available at https://jinhan.me/
Chair: Carmen Armenti
*************************
In February 2019, the Software Institute started its SI Seminar Series. Every Thursday afternoon, a researcher of the Institute will publicly give a short talk on a software engineering argument of their choice. Examples include, but are not limited to novel interesting papers, seminal papers, personal research overview, discussion of preliminary research ideas, tutorials, and small experiments.
On our YouTube playlist you can watch some of the past seminars. On the SI website you can find more details on the next seminar, the upcoming seminars, and an archive of the past speakers.