Automation and Orchestration for Machine Learning Pipelines A study of Machine Learning Scaling: Exploring Micro-service architecture with Kubernetes
| dc.contributor.author | Melberg, Filip, VASILIKI KOSTARA | |
| dc.contributor.department | Chalmers tekniska högskola / Institutionen för fysik | sv |
| dc.contributor.department | Chalmers University of Technology / Department of Physics | en |
| dc.contributor.examiner | Volpe, Giovanni | |
| dc.contributor.supervisor | Volpe, Giovanni | |
| dc.date.accessioned | 2024-06-12T11:44:08Z | |
| dc.date.available | 2024-06-12T11:44:08Z | |
| dc.date.issued | 2024 | |
| dc.date.submitted | ||
| dc.description.abstract | lthough Machine Learning (ML) has been around for many decades, its popu larity has grown tremendously in recent years. Today’s requirements show a great need for the development and management of ML projects beyond algorithms and coding. The aim of this thesis is to investigate how a minimal team of engineers can create and maintain a ML pipeline. To this end, we will explore how a Machine Learning Operations (MLOps) pipeline could be created using containerization and container orchestration of micro-services. After relevant research, the result is a minimal, on-premises Kubernetes cluster set up on physical servers and Virtual Machines (VMs) running the Ubuntu Operating System (OS). The cluster consists of a master and two worker nodes, which are used for two main ML frameworks. Populating the cluster with more nodes is straightforward, which makes scaling a simple task. Additionally, a locally shared folder on the network is mounted in the cluster as an external storage and the cluster is configured to access either a local or a cloud-provided container registry. Once the cluster is set up and run ning, an application is launched to train the YOLOv5 model on a custom dataset. Later, Distributed Data Parallel (DDP) training is performed on the cluster using PyTorch, TorchX, PyTorch Lightning and Volcano. | |
| dc.identifier.coursecode | TIFX05 | |
| dc.identifier.uri | http://hdl.handle.net/20.500.12380/307806 | |
| dc.language.iso | eng | |
| dc.setspec.uppsok | PhysicsChemistryMaths | |
| dc.subject | DevOps, Docker, Kubernetes, Micro-service, ML, MLOps, PyTorch, YOLO | |
| dc.title | Automation and Orchestration for Machine Learning Pipelines A study of Machine Learning Scaling: Exploring Micro-service architecture with Kubernetes | |
| dc.type.degree | Examensarbete för masterexamen | sv |
| dc.type.degree | Master's Thesis | en |
| dc.type.uppsok | H | |
| local.programme | Complex adaptive systems (MPCAS), MSc |