Integrating Vision–Language Models with Medical Imaging and Clinical Data for Improved Lymphoma Diagnosis

Abstract

Recent advancements in deep learning have fundamentally transformed the field of medical imaging. Along with the increasing burden of cancer worldwide, the demand for automated diagnostic tools has surged. An example of such process is the Lymphoma Artificial Reader System (LARS), an ensemble model composed of 10 ResNet34-based models trained on over 17,000 [18F] FDG-PET/CT scans. Although LARS has achieved impressive diagnostic accuracy, it relies solely on image features and lacks the capability to understand and utilize patient metadata (such as age, sex, and smoking history) and treatment information. However, these structured clinical records play a crucial role in diagnostic decisions in daily clinical practice. In order to bridge this gap, we propose a new vision–language version of LARS (namely MM-LARS) capable of incorporating patient context via a tabular-totext transformation. By converting structured patient data into natural language prompts, we effectively fuse clinical text with dual-view PET images. Through extensive ablation experiments, we demonstrate that this multimodal approach not only improves overall diagnostic performance over vision-only baselines, but also significantly boosts model specificity. Notably, the inclusion of clinical text successfully corrects false-positive misdiagnoses caused by ambiguous visual features. Our work provides practical insight into the next generation of multimodal diagnostic AI, offering a framework that is potentially generalizable beyond lymphoma.