Cross-Checking Oracles

Cross-Checking Oracles is a technique to generate and execute oracles for testing. The technique exploits the redundancy of the program under test. This redundancy must be expressed in terms of an equivalence between two sequence of method calls. Cross-Checking Oracles works by comparing the results of the executions of two supposedly equivalent sequences.

Evaluation Replication

This section explains how to replicate our experimental evaluation of the Cross-Checking Oracles technique.

Requirements

In order to replicate our experiments, you need a Unix-like operating system (Linux or Mac OS X). You also need:

JDK 1.6
Python 2.7
GNU make

Step by step Experiments Replication

Please note that the replication of the entire evaluation process requires several hours. This step by step tutorial allows you to replicate the evaluation of the Cross-Checking Oracles technique. This tutorial focuses on the evaluation of our technique on Guava's ArrayListMultimap class. Note that each make command can be parallelized using the -j option (see Make documentation).

Download the package: CrossCheckingOraclesExperiments.tar.gz. Extract the archive and go to the proper directory.
```
tar -xzf CrossCheckingOraclesExperiments.tar.gz
cd ICSE-2014-Experiments/experiments/ArrayListMultimap
```
Execute the while set of experiments for the Guava's ArrayListMultimap case study.
```
make all
```

Now you can check the results. Please note that your results could slightly differ from results published in the paper. This is due to the random nature of Randoop (the tool we use to generate the test suites).

Complete Experiment Replication

Download the package: CrossCheckingOraclesExperiments.tar.gz. Extract the package with the command tar -xzf CrossCheckingOraclesExperiments.tar.gz. Under ICSE-2014-Experiments/experiments/ you will find several directories, where each directory corresponds to one case study reported in the paper. Each of this directory contains a makefile (named Makefile) to run our technique on the particular case study. To replicate our experiments on the case studies, follow the steps of the step by step experiment replication.