Performance Bottleneck Evaluation of llama.cpp on Jetson and H100

Abstract

Efficient large language model inference depends strongly on the interaction between workload shape, numerical precision, serving configuration, and hardware platform. This thesis evaluates the performance bottlenecks of llama.cpp on two contrasting NVIDIA platforms: the datacenter-class H100 GPU and the edge-oriented Jetson AGX Orin. The study uses Llama 3.1 8B models in BF16, Q8_0, and Q4_K_M formats, and separates inference into prefill and decode phases using controlled single-sequence workloads and concurrent serving experiments. The evaluation first establishes baseline performance across balanced, prefill-heavy, and decode-heavy workloads. It then applies targeted profiling with Nsight Systems, server-side timing logs, and power measurements to explain the observed behavior. On H100, the results show a stable phase-dependent precision trade-off: BF16 is most effective for long-prefill workloads because execution is dominated by optimized BF16 GEMM and attention kernels, while Q4_K_M is more favorable for decode-heavy workloads where execution shifts to repeated matrix-vector kernels. Flash Attention further improves long-prefill throughput, but has a smaller effect on decode. On Jetson Orin, the dominant tuning problem is different. Performance and energy efficiency depend strongly on the selected power mode. The results show that 50W provides a strong energy-oriented operating point, while MAX mode gives the highest throughput and lowest latency. Orin also shows power-mode-dependent precision behavior: Q8_0 remains competitive at lower power, while Q4_K_M becomes more favorable for decode at higher power modes. Concurrent serving experiments further reveal a trade-off between throughput, energy per token, and time to first token. Overall, this thesis shows that inference optimization cannot rely on a single global configuration. Instead, effective deployment requires phase-aware, platform-aware, and power-aware tuning. The final guidelines recommend precision, Flash Attention, power mode, and concurrency settings based on the dominant workload and deployment objective.