Applying Conformal Prediction for LLM Multi-Label Text Classification

Abstract

This thesis investigates how conformal prediction can be used to improve the robustness and interpretability of multi-label text classification with large language models (LLMs). Using a dataset of Wikipedia comments annotated for multiple types of toxicity, a binary relevance approach is combined with inductive conformal prediction to produce label-wise prediction sets with formal coverage guarantees. Two data splitting strategies are explored to study the trade-off between model accuracy and calibration quality: one prioritising LLM fine-tuning, the other prioritising calibration set size. Results show that conformal prediction enables meaningful uncertainty quantification, including abstention on ambiguous inputs, while maintaining reliable coverage across a range of significance levels. The analysis also highlights challenges related to rare labels, label imbalance, and the sensitivity of validity guarantees to shifts in annotation quality and dataset distribution over time. Overall, the study supports the practical use of conformal prediction as a safeguard mechanism for LLM-based classifiers, especially in settings where predictive reliability and human oversight are both critical.