Seminars at the Faculty of Informatics

The Faculty of Informatics is pleased to announce a seminar given by Todd Mytkowicz 


TITLE: Rethinking Performance Evaluation

SPEAKER: Todd Mytkowicz, University of Colorado at Boulder, USA

DATE: Thursday, December 6th, 2007

PLACE: USI Lugano, Room SI - 006, Informatics Building (Via G. Buffi 13)

TIME: 16.00-17.00


Reproducible performance evaluation is an integral part of experimental computer science; if researchers cannot recreate the results from published works they will be unable to compare their methods to the state of the art. In this talk we argue that modern computer systems are so complex that small changes in their initial state can have drastic effects on their runtime performance. This sensitivity means that current methods in experimental performance evaluation do not necessarily provide reproducible results. We propose to increase the reproducibility of performance evaluations by deliberately affecting the initial state of the computer system and looking at average behavior, a technique we call intervention. We employ intervention in the context of evaluating compiler optimizations on the SPEC CINT 2006 benchmark suite. Using this preliminary data, we provide a head to head comparison of intervention and the current state of the art in performance evaluation and show how easy it is for current methods to be led astray.

URL 1: