Distributed Coordination via ZooKeeper
Staff - Faculty of Informatics
Start date: 11 October 2011
End date: 12 October 2011
The Faculty of Informatics is pleased to announce a seminar given by Flavio Junqueira
DATE: Tuesday, October 11th, 2011
PLACE: USI Università della Svizzera italiana, room A34, Red building (Via G. Buffi 13)
Building distributed systems is hard. They often involve using algorithms and techniques that are intricate, and system developers often do not have the time (or will) to deal with convoluted theory details of distributed algorithms. Process communication, partition of data and computation, request routing, and configuration management are all issues that a developer of large distributed systems has to face. Such issues become even more complex when system and network failures are part of the mix, which are not uncommon in data-center environments. The recent trend of cloud computing brings even more challenges in the form of elasticity, the ability to add and remove servers, and scale.
To mitigate the challenges of developing and running large distributed systems at Yahoo!, we developed a coordination system called ZooKeeper. Since the first time it was deployed in production four years ago, it has grown in popularity both inside and outside of Yahoo!. ZooKeeper is currently an open source project with developers and contributors from multiple companies (http://zookeeper.apache.org/).
The popularity of ZooKeeper stems from its simple, yet powerful API and its excellent performance. The API of ZooKeeper has been inspired by file system APIs, which are often familiar to developers, thus simplifying adoption. There are some key differences, however, that lead to an efficient implementation and enable extensive use of ZooKeeper. One such an important feature is watchers. ZooKeeper foregoes client caches and uses watchers to notify clients of changes to the ZooKeeper state instead. Cache invalidation in large systems can create the convoy effect, which has negative and important performance consequences. ZooKeeper is also a crash-tolerant system and relies upon an efficient broadcast protocol, Zab, to maintain the state of replicas consistent. One key feature of Zab is enabling multiple outstanding client operations.
In this presentation, we review some of the key design choices we have made for ZooKeeper. We discuss a few practical examples to explain where and how it has been used, present some evaluation results, and share some of the experience with conducting a project from research to widespread use.
Flavio Junqueira is a Research Scientist with Yahoo Research and leads the Scalable Computing group in Barcelona, Spain. He holds a PhD degree from University of California San Diego (UCSD) in computer science, a MSc degree in Electrical Engineering from COPPE/UFRJ in Rio de Janeiro, and a BS degree cum laude in Electrical Engineering from UFRJ in Rio de Janeiro. His main research interest is distributed systems and algorithms, and he has focused on topics such as dependability, concurrency, and replication. Additionally, he has worked on projects related to the modeling of failures and vulnerabilities in distributed systems, the design of distributed algorithms, and the design of large-scale distributed systems for Web search. He is the recipient of a number of awards and nominations, such as the CSE Department best PhD dissertation award, a nomination to the ACM PhD Dissertation award, and the best paper awards at ACM CIKM 2009 and USENIX ATC 2010. He is an active contributor to open source projects, such as Hadoop, ZooKeeper, BookKeeper, and S4 hosted by the Apache Software Foundation.
HOST: Prof. Fernando Pedone