Weevil: a Distributed System Experiment Automation Tool

Overview

This research project addresses the problem of experimenting with highly distributed systems. We use the term highly distributed system to refer to a system capable of delivering a service to many clients through a large number of distributed access points. A highly distributed system usually consists of a network of components, executing independent and possibly heterogeneous tasks, that collectively realize a coherent service. Examples of such systems are various forms of general-purpose communication systems, application-level overlays, networks of caching servers, peer-to-peer file-sharing systems, distributed databases, replicated file systems, and distributed middleware systems in general.

One difficulty software engineers face later in distributed system development process is how to evaluate and tune the alternative designs and implementations. The only solution is systematic, repeatable experimentation with executable prototypes in realistic execution environments. Thus, many large-scale or emulated networks come into being to provide the testbed for such large-scale experiments. However, the large-scale and complicated nature of large-scale network environments and distributed system usage scenarios makes the control of such experiments difficult.

Our tool, Weevil, is to ease the difficulty in programming distributed experiment control scripts by generating them based on some easy-to-write configuration files. It covers a two-phase experimentation process.

Workload generation: A distributed system workload is a time-ordered list of service calls from clients to system access points. While there are many ways to create such workloads, they are still not flexible enough and unavailable for some scenarios. Our approach to workload generation is to perform an offline simulation of expected client behavior and record the resulting service calls. During an experiment run, these service calls are issued as inputs into the subject system.
Experiment deployment and execution: This activity applies a workload to a system and gathers runtime data. To achieve this, the subject system, the workload, and the framework codes are first deployed across an experimentation testbed and then configured. Next, the system is started and stimulated by the execution of service calls at the times and locations dictated by the workload. When the experiment terminates, output and diagnostic data are collected from across the testbed to facilitate efficient data analysis and reduction.

Weevil consists of a synthetic-workload generator coupled with an environment for managing the deployment and execution of experiments. Weevil is intended to facilitate experimentation activities for distributed systems by providing engineers with a flexible, configurable, automated and, thus, repeatable process for evaluating their software on a networked testbed. Further, Weevil is not targeted at a specific system or application model, but rather it is a generic, programmable tool. This documentation describes a prototype implementation of Weevil.

Licence and Credits

Copyright © 2004-2006 University of Colorado. Weevil is free software. you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

Weevil is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

People

Documents

Weevil User Manual

Workload Generator Reference Manual

SSim Simulation Library:
This library supports message communication between simulated processes. It provides the basic strategy to simulate actor's behavior.
Weevil's Workload-Generation Library:
This library is an extension library of SSim Simulation Library. It provides convenient functions that identify simulated processes with their IDs. It also provides a function to output workload.
Weevil's Workload Generator Conceptual Models:
This reference lists all the elements and its arguments in the workload scenario conceptual model as shown in Figure 2 in the user manual.
Weevil's Workload Generator Declaration Macros:
These macros are used to declare and configure the elements in the conceptual model.
Weevil-Defined Functions:
These functions can be used to ease your work on actor configurations.

Experiment Environment Reference Manual

Weevil's Experiment Environment Conceptual Models:
This reference lists all the elements and its arguments in the experiment conceptual models described in Section 3.3 in the user manual.
Weevil's Experiment Environment Declaration Macros:
These macros are used to declare and configure the elements in the conceptual models.
Some property macros will be defined by these declaration macros. These property macros can be used in the parameters of any declarations.
Some Other Weevil-Defined Property Macros:
These extra property macros are defined based on your experiment configurations besides those defined above in the declaration macro reference. You can use them in your ComponentType and ActorProgram configurations.
Weevil-Defined Functions:
These functions can be used to ease your work on experiment configurations.
Java ActorProgram Library:
You should implement the Java actor program by inplementing the Actor interface in this library.

Publications

Yanyan Wang
Automating Experimentation with Distributed Systems Using Generative Techniques. Ph.D. Thesis, Department of Computer Science, University of Colorado. Aug. 2006.
Yanyan Wang, Matthew J. Rutherford, Antonio Carzaniga, Alexander L. Wolf,
Automating Experimentation on Distributed Testbeds, In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE 2005), Long Beach, California, November 7-11, 2005.
Yanyan Wang, Antonio Carzaniga, Alexander L. Wolf,
Four Enhancements to Automated Distributed System Experimentation Methods. In Proceedings of the Thirtieth International Conference on Software Engineering (ICSE'08). Leipzig, Germany, May 2008.
Yanyan Wang, Matthew J. Rutherford, Antonio Carzaniga, Alexander L. Wolf,
Weevil: a Tool to Automate Experimentation with Distributed Systems. Technical Report CU-CS-980-04. Department of Computer Science, University of Colorado. Oct. 2004.

Acknowledgements

We extended a parallel tool, vxargs, to allow more arguments included in an arbitrary command (instead of only allowing IP address as arguments in the original vxargs). We thank Yun Mao, for providing this useful tool. You can download our extended version of vxargs here.

Feedback

Your feedbacks are very valuable to us. Please don't hesitate to send us email (Yanyan.Wang@colorado.edu) if you have any problems, corrections, or suggestions. Thanks!