A GUI for application design and performance reporting of data streaming applications

Abstract

As a result of more connected devices and users and applications moving to the cloud, every year the amount of data needed to be processed by different computer systems increases. The troubles with trying to store all this information, and a growing demand for real-time processing not possible with traditional batch processing, has led to an increase in the popularity of stream processing as a way of handling large amounts of data in a real-time fashion with low latency and high throughput. The number of Stream Processing Engines (SPEs) available for processing this streaming data is increasing and each has its own programming style and conventions, resulting in individually developed SPEs that differ in many aspects. In general, there is a lack of a common set of tools supporting developers to design, visualise and monitor streaming applications. For instance, a streaming application executed by an SPE can be modelled as a Directed Acyclic Graph (DAG), but not all SPEs have a tool to visualise this graph. This thesis concerns the design and implementation of a framework in the form of a Graphical User Interface (GUI) as a first step towards a unified view of streaming applications, aiding developers with visual representations, statistics, and code generation for different SPEs by abstracting away the implementation details. Using Java and JavaFX, a framework is developed and tested with two SPEs (Apache Flink and Liebre) and then evaluated in terms of performance, functionality, and generality. Presented is a functional framework allowing for a user to visualise DAGs of existing streaming applications, viewing live and offline statistics of executions, and the possibility to design a streaming application graphically and generating the corresponding Java code. The implementation is general and can be adapted to work with other popular SPEs. The resulting framework is a first step towards a unified view of streaming applications from different SPEs and adds a tool for developers to use, enabling for increased productivity and better understanding while working with both new and already existing streaming applications.