Expert Performance on a Hip Wire Navigation Simulator Provides a Basis for Establishing Benchmarks to Define Skill Proficiency

Steven Long; Geb W Thomas; Donald D Anderson; Jessica M Downes; Emily A Wagstrom; Matthew Karam

doi:10.1097/CORR.0000000000003877

Simulator-based training addresses a pressing need to move surgical skill acquisition from the operating room to the laboratory, reducing patient risk associated with traditional trial-and-error training methods. The Core Requisites of Orthopedic Wire Navigation Skills (CROWNS) curriculum employs a hybrid reality simulator to teach surgeons the foundational skill of placing a guidewire in bone via fluoroscopic imaging. Simulator-based training has already been shown to improve subsequent operating room performance, but objective benchmarks to assess skill proficiency are lacking. The objective of this study was to measure and compare expert and novice performance on a simulated hip wire navigation task as a basis to define candidate proficiency training benchmarks. This study was designed as an experimental assessment of a surgical simulation platform to differentiate between skill levels during a proctored examination. In all, 113 surgeons participated, including 68 orthopaedic residents (novices) and 45 experts (28 Orthopaedic Trauma Association fellows and 17 practicing orthopaedic surgeons). To achieve the primary objective of measuring and comparing expert and novice performance, all participants performed a simulated hip wire navigation task replicating guide wire placement for intertrochanteric fracture fixation. Experts completed three simulated patient scenarios: one represented a straightforward case with clear imaging and a typical femoral neck-shaft angle (∼ 135°), another required a steeper guidewire trajectory to match an increased neck-shaft angle, and another involved a slightly malreduced fracture with an imperfect lateral view, requiring greater judgment to determine appropriate wire placement. Novices completed only the straightforward case scenario. Performance metrics included tip-apex distance (TAD), fluoroscopic image count, time, decision errors, the angle of off-target adjustments, and the previously reported Image-based Decision Error Analysis (IDEA) composite score. The IDEA score is normalized, meaning that a participant's score reflects the number of SDs above or below the mean performance across a large resident cohort. Assuming a normal distribution of scores, 95% of participants would be expected to score between -2 and +2. Higher values indicate better performance generally associated with more surgical experience. The score combines three normalized components-TAD, angle error, and decision errors-where lower TAD and smaller or fewer errors yield a more positive score. Statistical analyses compared novice and expert cohorts to help inform proficiency benchmark recommendations. The expert cohort's simulator-based IDEA scores were then compared with prior intraoperative IDEA scores from residents. The expert cohort completed all three simulated patient scenarios with a mean ± SD time of 117 ± 60 seconds, using 17 ± 8 images, and achieving a TAD of 13 ± 5 mm. The expert cohort had 5 ± 4 decision errors, angle errors of 1.4° ± 1.8°, and an IDEA score of 0.65 ± 0.4. The novice cohort completed their single patient scenario with a time of 207 ± 88 seconds, using 22 ± 10 images, and achieving a TAD of 20 ± 7 mm. The novice cohort had 14 ± 7 decision errors, angle errors of 3.2° ± 3.0°, and an IDEA score of -0.21 ± 0.7. When comparing the performance of the expert and novice cohorts in the single shared case scenario, the expert cohort achieved a substantially higher IDEA score (0.62 ± 0.3 and -0.21 ± 0.7, respectively, mean difference 0.83 [95% confidence interval 0.58 to 1.07]; p < 0.001). Based on prior research linking simulator performance to actual surgical experience, the expert IDEA mean score of 0.65 was roughly comparable to a performance expected in novices after their first 11 completed wire navigation surgical cases. An IDEA score of 0.5 is proposed as a balanced proficiency benchmark, achieved by 75% of experts but only 13% of novices. The CROWNS simulator and IDEA composite score reliably distinguished expert from novice performance, underscoring the need for deliberate practice and targeted feedback to close the novice-expert gap. A benchmark IDEA score of 0.5 offers a rigorous yet attainable goal for residents, representing readiness for safe and efficient operating room participation. Objective, simulation-based proficiency standards-such as attaining an IDEA score of 0.5 for hip fracture pinning-can guide competency-based progression, ensuring that residents achieve the technical skill required for safe operating room performance. Future research should evaluate how such benchmarks translate to real-world outcomes and whether they can be optimized to further improve surgical training and patient safety.

Expert Performance on a Hip Wire Navigation Simulator Provides a Basis for Establishing Benchmarks to Define Skill Proficiency

View Online

Abstract

Details

Metrics