Hybrid Video and Instance Segmentation for Surgical Scene Understanding in Robotic Mitral Valve Surgery
In the authors' words
Objective: Artificial intelligence-based surgical scene segmentation is fundamental to surgical scene understanding. However, the optimal segmentation strategy for robotic cardiac surgery remains unknown. We developed and evaluated a hybrid segmentation framework combining Mask R- CNN and Segment Anything Model 2 (SAM2) for surgical scene recognition during the entry phase of robotic mitral valve surgery. Methods: Intraoperative videos from ten robotic mitral valve operations performed at two institutions were analyzed. Seven anatomical structures and surgical object classes (ascending aorta, diaphragm, lung, pericardium, right atrium, robotic arms, and sutures) were manually annotated from 2172 image frames. We created three patient-level training, validation, and test splits after ruling out any split where an object class was completely missing from validation or test. Three segmentation approaches were compared: Mask R-CNN, Mask R-CNN-prompted SAM2 for all seven classes, and a hybrid late-fusion model combining SAM2 for anatomical structures and Mask R-CNN for robotic arms and sutures. Model performance was assessed using pixel accuracy, mean intersection over union (IoU), and Dice score. Results: Across the three patient-level splits, the hybrid late-fusion model achieved the highest average overall segmentation performance, with pixel accuracy of 0.79-0.91, mean IoU of 0.64-0.74, and mean Dice of 0.76-0.84. At the object level, late-fusion Dice scores across the three splits were 0.86-0.96 for aorta, 0.38-0.90 for diaphragm, 0.77-0.92 for lung, 0.48-0.77 for pericardium, 0.68- 0.90 for right atrium, 0.95-0.97 for robotic arms, and 0.66-0.76 for sutures. Mask R-CNN- prompted SAM2 outperformed Mask R-CNN for the aorta in all three splits and for the lung, pericardium, diaphragm, and right atrium in two of three splits, whereas Mask R-CNN was more reliable for sutures, including Dice scores of 0.76 vs 0.21 and 0.66 vs 0.53 in Splits 2 and 3. By combining prompted SAM2 for anatomy with Mask R-CNN for robotic arms and sutures, late fusion achieved the highest mean Dice in two of three splits (0.84 and 0.76) and the highest mean Dice averaged across all three splits; all-object SAM2 propagation performed slightly better in the remaining split (0.81 vs 0.80). Conclusions: A hybrid segmentation framework combining SAM2 and Mask R-CNN achieved the highest overall segmentation performance in two of three patient-level splits and performed comparably to the best-performing model in the remaining split. These findings suggest that object-specific hybrid segmentation strategies may improve surgical scene segmentation during robotic mitral valve surgery.
Appeared: Thursday, November 19. medRxiv. Preprint, not yet peer-reviewed.