Segmentation in X-ray Fluoroscopy Utilizing Virtual Simulations of Cardiovascular Procedures

Abstract

Abstract This project aims to assess the quality of simulated fluoroscopy images from the company Mentice and their surgical simulator as training data for neural networks. To do this, a simulated dataset was collected by extracting images and corresponding information from the simulation to create labels for different objects in the image automatically. The objects segmented in the images are catheters. The evaluation uses a dataset from diagnostic and Trans-catheter Aortic Valve Implantation (TAVI) in vivo procedures. A model called Neural Neighbor Style Transfer was used to adapt the domain of the images to make them look more like real images to improve the performance. To perform segmentation two models U-Net and YOLOv8 were used. The major finding was that a Dice score of 0.8803 was achieved using pretraining on style-transfer images using YOLOv8. It was also found that by pre-training using simulated images performance was on average increased by 0.3%/0.6% (Dice/IoU) for simulated images and 0.9%/1.2% for style-transfered images. A model trained purely on simulated images could achieve a Dice score of 0.5182. Overall it can be concluded that the simulated images help improve performance and style-transfer also shows promise for improving the metrics.