Implementation of a transport system with stateful consumers

Abstract

When handling large amounts of data, parallel processing is key. Distributed computation makes Big Data work, but it presents new challenges for scaling and reliability. These challenges are made even more interesting when computations are stateful, and done in near real-time as in stream processing. To make a stateful distributed system scalable and resilient to failures, a framework for message passing and state persistence is needed. As of today no such system is available but the components are all there to build one. This thesis gives an outline of how such a framework could be constructed using Apache ZooKeeper and Apache Kafka, describing algorithms and components which could be used for such a system. Also provided are some experimental performance benchmarks of Kafka running in Amazon's EC2 cloud, to use as a measuring stick for its viability as a transport system for messages.