Profiling and Visualizing CPU-Based LLM Inference in llama.cpp

Abstract

Large language models (LLMs) are increasingly deployed on local hardware rather than through cloud services, motivated by concerns around data privacy, network dependence, and operational cost. In many local deployments, the CPU serves as the primary or sole compute resource, since dedicated accelerators such as GPUs are not always available. Running LLMs on CPUs is technically demanding, as in ference performance is shaped by complex interactions between the workload and the underlying hardware. Identifying performance bottlenecks is therefore essential for improving local LLM inference, but is difficult without structured measurement and analysis. This bachelor’s thesis presents a profiling and visualization tool for characterizing LLM inference on CPUs using an inference serving framework called llama.cpp. The tool collects hardware performance counters, which are low-level CPU metrics such as cache accesses and floating-point operations, at four levels of granularity: the entire inference run, the prefill phase in which the user’s input prompt is processed and the decode phase in which the response is generated token by token, individual decoder blocks, and individual tensor operations. Measurements are stored in a structured database and can be explored through a graphical dashboard, allowing users to inspect runtime, memory traffic, cache behavior, and other metrics across different quantization formats and model sizes. By applying roofline analysis, the tool distinguishes between compute-bound and memory-bound behavior and supports the identification of performance bottlenecks during local LLM inference.