Benchmarking of Serverless Application Performance across Cloud Providers: An In-depth Understanding of Reasons for Differences

Abstract

Serverless computing has emerged as a new compelling cloud computing model for deploying modern applications, creating demand for benchmarking serverless platforms to help practitioners make a suitable choice. Existing works related to serverless benchmarking primarily focus on microbenchmarking to measure an individual aspect of function performance such as CPU speed and cold start. Some studies propose an application-centric benchmarking framework but lack an in-depth analysis of the application performance difference across cloud providers. Furthermore, none of the related studies provides details about addressing benchmark fairness. In contrast, this thesis presents a methodology to design a serverless application benchmark for a fair comparison between two leading cloud providers: AWS and Azure. The benchmark execution generates detailed traces constituting the end-toend execution duration which enables drill-down analysis on how the application performs differently across cloud platforms. The main finding shows that storage triggering can substantially impact the end-to-end latency, and the performance difference between cloud platforms.