Browser-Based Video Supports Scalable Computer Vision Analysis for Open Surgical Skill Assessment
In the authors' words
Purpose: Objective assessment of open surgical skills remains limited by expert availability. Machine learning (ML)-based hand keypoint detection offers potential for automated, on-demand assessment, but performance in accessible browser-based recording environments (laptop-webcam video recorded in a web browser) is unknown. We hypothesized that browser-based recordings would achieve acceptable keypoint localization (percentage of correct keypoints [PCK] >70%). Methods: Retrospective technical validation of prospectively collected video from 14 PGY-1 interns at one academic center (July to August 2024). We annotated 203 frames (101 browser-based, 1280x720/854x480; 102 overhead, 3840x2160/1920x1080), yielding 6,232 ground-truth keypoints. An internally developed pipeline and MediaPipe Hands were evaluated. The primary endpoint was PCK normalized to the index-proximal bone; bounding-box-normalized PCK, detection rate, and per-region accuracy were also evaluated. Results: Primary PCK, normalized to a fixed anatomical hand scale, was 85.9% (95% CI, 80.3-91.0) for browser-based recordings and 81.5% (95% CI, 76.7-87.0) for overhead recordings. Secondary bounding-box-normalized PCK showed a similar pattern (91.6% vs 90.6%), and detection rates were high in both environments (94.9% vs 97.7%). In contrast, MediaPipe detection rates were only 5.3% for browser-based recordings and 15.4% for overhead recordings, resulting in substantially lower accuracy. Conclusion: Hand keypoint detection performance was comparable between browser-based and overhead recordings using the internally developed pipeline. These findings suggest that browser-based video acquisition can provide data of sufficient quality for ML-based hand keypoint detection, offering a scalable pathway for surgical skill assessment without specialized recording hardware.
Appeared: Wednesday, November 11. medRxiv. Preprint, not yet peer-reviewed.