Evaluate 2010

Workshop on Experimental Evaluation of Software and Systems in Computer Science

Co-located with SPLASH'10 in Reno/Tahoe, Nevada, October 17-21, 2010.

Share |


Call for Position Statements

We call ourselves 'computer scientists', but are we scientists? If we are scientists, then we must practice the scientific method. This includes a solid experimental evaluation. In our experience, our experimental methodology is ad hoc at best, and nonexistent at worst.

In the last few years, researchers have identified disturbing flaws in the way that experiments are performed in computer science. For example, in the area of performance evaluation of computer systems, our measurements on one system are rarely reproducible on another. As hardware and software grow more complex, this problem just gets worse.

This workshop brings together experts from different areas of computer science to discuss, explore, and attempt to identify the principles of sound experimental evaluation.

The workshop will consist of discussion sessions, which focus on themes such as data collection, data analysis, and reproducibility, with the goal to answer the following questions:

We are asking each participant to submit a short position statement, no more than 1000 words, on the state of experimental evaluation in computer science.


Submissions are closed.

You do not need to write a fully formatted paper. Just submit your short, text-only position statement in the "Abstract" field on the EasyChair submission system: http://www.easychair.org/conferences/?conf=evaluate2010. Do not upload any file; check the "Abstract Only" check box instead. In the "Keywords" section, specify the topics you would be most interested in discussing during the workshop.

The organizing committee will use the submitted position statements to determine who to invite to the workshop. In your position statement, please also answer the following two questions: (1) Why do I want to participate in the workshop? (2) What can I contribute to the workshop?

If you are invited to attend the workshop, your position statement will be released on a password-protected web page that will be accessible to all workshop participants.

Moreover, if you would like to voice your opinion already before this workshop, please leave a comment on our LinkedIn , Facebook , or Xing groups, or contact any of the organizers.

Important Dates

Submission deadline: Monday, August 9, 2010, 23:59:59 (Apia, Samoa Time)
Notification of acceptance: Monday, August 30, 2010
Early registration: mid September, 2010
Workshop: Monday, October 18, 2010


Cliff Click, Chief JVM Architect and Distinguished Engineer at Azul Systems, will give the keynote on experimental evaluation.

Panel Discussion

To learn from the experience and insights on evaluation methodology of our colleagues in different areas of computer science, we will host a virtual panel discussion with renowned experts on experimental evaluation in three distinct areas:

Organizing Committee


Here is a short initial list of prior work related to the Experimental Evaluation of Software and Systems in Computer Science. This list is by no means meant to be exhaustive. One goal of our workshop is to initiate work towards a resource, maybe in the form of a Wiki or an annotated bibliography, that can serve as a reference for researchers who need to experimentally evaluate Software and Systems. If you would like to contribute, post to one of our groups (, , ), contact one of the workshop organizers, or submit a position statement to the workshop.