Perspectives for Large-scale Evaluation Campaigns and the Cross-Language Evaluation Forum

SPEAKER: Nicola Ferro, Assistant Professor, University of Padua, Italy
DATE: Thursday, February 10th, 2011
PLACE: USI Università della Svizzera italiana, room 355, Main building (Via G. Buffi 13)
TIME: 13.30

ABSTRACT:
Large-scale international evaluation campaigns have provided  a  fundamental contribution to the advancement of state-of-the-art techniques and  to the  development of innovative information systems through common evaluation procedures, regular and systematic evaluation cycles, comparison and benchmarking of the adopted approaches and solutions, spreading and exchange of knowledge and know-how. All the major initiatives (TREC, CLEF, NTCIR, FIRE) follow a similar evaluation paradigm and, according to regular  cycles, produce large volumes of valuable scientific data, which derive from their experimental activities. However, these campaigns face many common issues when it comes to the creation, management, analysis, mining, enrichment, archiving, curation, re-use,  and  dissemination of the experimental data, performance measures and figures, and statistical analyses.

The talk will discuss  why   this scientific data should play a central role in the design and planning of an evaluation campaign and how large-scale evaluation campaigns  can adopt interoperable infrastructures to foster the sharing and re-use of such data. In particular, we will present  our experience in designing and developing the DIRECT system, which has been adopted in CLEF since 2005 and represents a first step in this direction. Finally, we will  outline  some  future perspectives for evaluation campaigns related to the activities of the PROMISE network of excellence, which will provide a virtual and open laboratory for conducting participative research and experimentation in which it will be possible to carry out, advance and bring automation into the evaluation and benchmarking of complex multimedia and multilingual information systems, by facilitating management and offering access, curation, preservation, re-use, analysis, visualization, and mining of the collected experimental data.

BIO:
Nicola Ferro is assistant professor in Computer Science at the Department of Information Engineering and at the Faculty of Statistical Sciences of the University of Padua, Italy.  He teaches the courses on Digital Libraries, Information Retrieval, and Databases.  He received a Ph.D. degree in Computer Science from University of Padua in 2005.  He holds a Laurea degree from University of Padua in Telecommunications Engineering. His main research interests are digital libraries and archives, their architectures, interoperability, and evaluation, as well as multilingual information access and its evaluation. He is and has been involved in the overall coordination of the CLEF (Cross Language Evaluation Forum) evaluation campaigns since 2005. He is scientific leader of the DL.org working group on quality in digital libraries. He is programme co-chair of the CLEF 2010 Conference on Multilingual and Multimedia Information Access Evaluation. He is the coordinator of the PROMISE European Network of Excellence on the experimental evaluation of multilingual and multimodal information access systems. He has participated and participates in several national and international projects among which Europeana Connect (multilingual information access services for Europeana and their evaluation), Europeana v 1.0 (multilinguality and annotations in the Europeana Data Model), TrebleCLEF (best practices, collaboration, and evaluation for multilingual information access systems), TELplus (enhanchement of The European Library portal towards Europeana), SAPIR (search in audio visual content using peer-to-peer information retrieval), and DELOS (the European network of excellence on Digital Libraries). He has published more than 70 papers on digital library architectures, interoperability, and services; multilingual information access and its experimental evaluation; the management of the scientific data produced during evaluation campaigns. He is member of ACM and IEEE.

HOST: Prof. Fabio Crestani