Adapting Without Forgetting

Abstract

Automated cell segmentation in microscopy images is a critical task in pharmaceutical research and biological discovery. Existing supervised approaches require extensive annotated data and must be retrained for each new cell type or imaging condition. Foundation models offer a promising alternative through their ability to generalise across diverse domains without retraining. However, a systematic comparison of their generalisation across diverse microscopy conditions has been lacking. Moreover, adapting such models to specific datasets through fine-tuning risks catastrophic forgetting, potentially undermining the very generalisation that makes them valuable. This thesis benchmarks two state-of-the-art foundation models for cell segmentation, CellSAM and Cellpose-SAM, across several datasets spanning multiple imaging modalities. Cellpose-SAM consistently outperformed CellSAM, particularly at stricter evaluation thresholds, attributed to its flow-based instance separation, which handles touching and overlapping cells more effectively than CellSAM’s promptbased approach. Cellpose-SAM was subsequently fine-tuned using QLoRA, a parameter-efficient method that adapts only a small fraction of the model’s weights. Fine-tuning on a small curated dataset yielded substantial improvement on challenging microscopy images while retaining near-baseline performance on held-out general data, demonstrating that catastrophic forgetting can be avoided with careful, parameter-efficient finetuning. The entire training process required approximately 15 minutes on a single A100 GPU, illustrating that meaningful adaptation of large vision foundation models is achievable with modest computational resources.