Browser Video Holds Up for Surgical Skill Tracking
Open surgical skill assessment has long depended on expert review, a scarce and expensive resource. This preprint tests whether a much simpler input stream — laptop webcam video captured directly in a browser — can support computer vision analysis well enough to measure hand motion with useful precision. The answer, at least in this technical validation, is yes for the internally developed pipeline and no for MediaPipe Hands in this setting.
Investigators evaluated video from 14 PGY-1 interns at a single academic center, using recordings collected prospectively during July and August 2024. They annotated 203 frames in total, split between 101 browser-based frames at 1280×720 or 854×480 resolution and 102 overhead frames at 3840×2160 or 1920×1080 resolution. Those annotations produced 6,232 ground-truth keypoints, giving the model a direct test of whether a lower-friction recording setup could still preserve the visual detail needed for anatomical localization.
The clinical significance sits in accessibility. If a browser can capture video good enough for keypoint localization, surgical assessment may move closer to the bedside, the simulation lab, or even the trainee’s own workstation without requiring specialized camera rigs.
Anatomical Precision Stayed High Without Specialized Hardware
The primary endpoint was percentage of correct keypoints, normalized to the index-proximal bone, a fixed anatomical reference that avoids some of the distortion introduced by variable framing. In browser-based recordings, the internally developed pipeline achieved a PCK of 85.9% with a 95% confidence interval of 80.3% to 91.0%. Overhead recordings came in at 81.5% with a 95% confidence interval of 76.7% to 87.0%.
Those figures matter because they show that the lower-cost environment did not collapse performance. The browser-based setup did not outperform overhead video by a dramatic margin, but it also did not suffer the kind of degradation that would make automated assessment unreliable. In practical terms, hand localization remained in a range that suggests anatomical detail survived the transition from dedicated overhead capture to ordinary browser video.
A second metric told a similar story. When PCK was normalized to bounding box size rather than anatomical scale, browser-based video reached 91.6% and overhead video reached 90.6%. Detection rates were also strong in both settings, at 94.9% and 97.7%, respectively. Together, those numbers suggest that the core pipeline handled the visual environment robustly even when the input came from a more modest recording platform.
MediaPipe Failed Where the Custom Pipeline Succeeded
MediaPipe Hands performed very differently. Detection rates fell to 5.3% in browser-based video and 15.4% in overhead video, leaving accuracy far below the internally developed system.
That gap is the clearest signal in the study. General-purpose computer vision tools can look attractive on paper, but surgical video is unforgiving: hands overlap, instruments obscure fingers, lighting shifts, and the operative field is crowded. In this technical environment, a pipeline trained or tuned for the task outperformed a more generalized hand detector by a wide margin. The result supports a common reality in clinical AI development: model choice matters less than whether the model has been adapted to the specific imaging conditions of the procedure.
The implication is not simply that one algorithm worked better than another. It is that browser-based surgical video may be good enough to support a purpose-built machine learning workflow, provided the detection system is designed for the visual constraints of open surgery rather than consumer hand-tracking tasks.
Why Browser Capture Could Change Skill Assessment Workflow
Open surgical skill evaluation usually depends on expert raters who must watch, score, and compare performance across trainees. That process is labor-intensive and difficult to scale. A browser-based workflow lowers the barrier to recording and can turn almost any standard computer into a data collection point. For a training program, that means motion analysis might be gathered more routinely, with less dependence on custom hardware and less friction around deployment.
The biological and technical logic is straightforward. Surgical skill is expressed in the kinematics of the hands — stability, precision, economy of movement, and the relationship between hand position and operative targets. If video quality is sufficient for keypoint detection, it becomes possible to quantify those movements at scale and potentially link them to education, feedback, and performance benchmarking. The present validation does not prove clinical outcome improvement, but it does establish a feasible input pathway for downstream educational tools.
The study also matters because it tested an ordinary capture environment rather than an idealized laboratory setup. That makes the result more relevant to real-world training programs, where simplicity often determines whether a platform survives beyond the pilot phase.
What the Validation Does and Does Not Prove
This is a retrospective technical validation, not a clinical outcomes study. The cohort was small, limited to 14 interns at one academic center, and the frame annotations covered a discrete sample of video rather than a large continuous archive. The data also came from a narrow training window, so performance in different lighting conditions, among more experienced surgeons, or across a wider range of procedures remains untested.
Data access was restricted under the IRB-approved protocol, and the analysis code and pipeline are available from the corresponding author on reasonable request. Ethical approval came from the Institutional Review Board of Beth Israel Deaconess Medical Center, and the preprint states that relevant consent and reporting requirements were followed.
Still, the technical message is clear: browser-based video acquisition did not prevent accurate hand keypoint detection when paired with an internally developed pipeline. The limitation is not that the method failed at the hardware level. The more pressing question is how widely that performance will generalize once the system is exposed to different operators, procedures, and recording conditions.