Predictive Resource Tuning of Apache Spark Jobs in Kubernetes - A Machine Learning driven approach, with uncertainty aware predictions

Abstract

Apache Spark is a widely used distributed data processing engine for large-scale analytics and machine learning workloads, typically executed in multi-node cluster environments. Kubernetes is increasingly used to deploy and manage these workloads in container orchestration, resource scheduling, and scalable cluster management. Despite this infrastructure, Spark workloads require users to specify CPU and memory allocations in advance. In practice, these resource requests are often overestimated to prevent job failures under uncertain workload requirements. This leads to inefficient resource utilization in shared clusters, reduced overall throughput, and increased operational cost in multi-tenant environments. This thesis addresses the problem of static and overly conservative resource provisioning for Spark jobs running on Kubernetes.

This thesis proposes a machine learning-based resource tuning system that learns from historical execution behavior to predict more accurate resource allocations for defined workload types. In addition, the system incorporates uncertainty awareness to ensure that resource adjustments are applied more conservatively when prediction confidence is low. The proposed approach aims to reduce resource overprovisioning while maintaining execution stability, thereby improving cluster efficiency in distributed Spark environments.

The results demonstrate that the tuning approach reduced memory from 64 GB to 8 GB and cores from 4 to 1 per executor for a designed test job. The total savings was 616 GB of memory and 30 cores per such job after tuning for 40 iterations. This increased resource utilization, although execution time increased by approximately 55%. Uncertainty-aware Wilson intervals were successfully incorporated into memory tuning but not core tuning due to its continuous cost-based optimization objective.