LLMs Move from Case Prompts to Complete OSCE Stations
Objective Structured Clinical Examinations sit at the center of competency-based medical education, but building them consumes substantial faculty time. A new feasibility study tested whether a large language model could generate not just a case stem, but a full station package: candidate instructions, standardized patient script, and examiner marking scheme in one structured call. The question is no longer whether generative AI can write educational text. It is whether it can assemble the moving parts of an assessment that medical schools trust to judge competence.
Three board-certified internists reviewed 24 generated cases spanning eight internal medicine diagnoses and three learner levels. They used a 12-item Likert instrument to rate the stations across multiple domains, creating a direct expert check on whether synthetic material could approximate faculty-built OSCE content. Ratings were high across the board, and several domains landed in a range that would be hard to dismiss as speculative or merely experimental. Communication behaviors scored 4.64 out of 5, EPA alignment scored 4.62 out of 5, and overall value scored 4.61 out of 5.
One domain resisted the technology more than the others.
Where the Model Performed Best and Why That Matters
The strongest scores clustered around areas where language structure and role clarity matter most. Communication behaviors are highly scriptable: a standardized patient can express concern, respond to empathy, and reveal history in a way that maps neatly onto prompt-driven generation. EPA alignment also translated well because the station can be built around an observable clinical task rather than a broad essay-style judgment. In practice, that makes sense. Entrustable professional activities are operationalized through concrete behaviors, and language models are naturally suited to assembling scenario-specific prompts, expectations, and scoring anchors from a structured input.
The overall value score of 4.61 suggests faculty reviewers saw utility in the output, not just novelty. For programs that struggle to produce enough stations for repeated cycles, that matters because station design is often the bottleneck, not the delivery platform. A tool that can generate the skeleton of a station may reduce the friction that limits frequent formative assessment, especially when faculty oversight is still applied at the final stage.
The technology was not equally strong in every dimension. Guideline concordance scored lowest at 4.24, and raters showed the greatest disagreement there. In medical education, guideline fidelity is not a cosmetic feature; it determines whether a case reflects accepted practice or quietly drifts into outdated or idiosyncratic teaching. A model can draft a plausible heart failure or chest pain station, yet still miss subtle changes in diagnostic pathway, treatment sequencing, or red-flag logic that experts would catch immediately. The gap reveals a practical boundary: synthetic stations can look clinically polished while still needing human review where clinical doctrine is precise and version-sensitive.
A Single Structured Call Is Not the Same as Autonomous Assessment Design
The study design is important because it tested a complete generation workflow rather than isolated fragments. A single structured call produced full OSCE stations, and those stations were then judged by experienced internists across diagnoses and training levels. That workflow is what makes the project operationally interesting. Educational systems do not need a model that can only brainstorm cases. They need one that can reliably produce coherent, schema-validated material that fits a station template from start to finish.
Still, this was a feasibility study, and the sample is necessarily modest. Twenty-four cases can establish whether the concept works, but they cannot settle how the tool performs across broader specialty domains, different faculty groups, or the full range of diagnostic nuance that shows up in authentic examination development. The sample also does not answer whether students perform differently on AI-generated stations, whether these stations measure the same competencies as faculty-written ones, or whether repeated use introduces unintended patterning in assessment content.
A brief sentence captures the current state well: the stations were acceptable to experts, but acceptability is not equivalence.
Faculty Oversight Remains Essential at the Calibration Layer
The lowest-scoring domain points to the place where human educators still add the most value: calibration. Guideline concordance is not just about factual correctness. It includes deciding what level of detail belongs in a learner-level station, what should be expected of a junior learner versus a more advanced trainee, and how to align scoring with current standards rather than generic clinical intuition. Even when a model generates a coherent scenario, it can miss the subtle editorial judgments that separate a usable teaching case from a defensible assessment instrument.
For medical educators, the implication is straightforward. LLMs may be well positioned to accelerate the first draft of OSCE development, especially for candidate instructions, patient scripts, and marking schemes that follow recognizable patterns. The human role shifts rather than disappears. Faculty still need to verify guideline accuracy, adjust difficulty, and ensure the station measures the intended EPA rather than merely producing a polished script. In that sense, the model behaves less like an assessment author and more like a high-speed drafting assistant that can supply structure before expert refinement begins.
The study also links directly to reproducibility. Because the output comes from a structured call and was assessed using a fixed rating instrument, the workflow is easier to standardize than ad hoc prompt experimentation. That opens the possibility of building a repeatable infrastructure for station generation, which is a more durable contribution than a one-off example. Even so, the evidence stops short of showing that the generated stations are ready to replace conventional development pathways.
What This Means for Competency-Based Training
Competency-based medical education depends on scalable, defensible evaluation. OSCEs help solve that problem, but they are expensive in faculty time and editorial effort. This study suggests that LLMs can meaningfully compress the early drafting phase while preserving a quality level that experts judged acceptable across several domains. The strongest performance in communication and EPA alignment fits the strengths of current language models: pattern recognition, structured text generation, and task framing. The weakest performance in guideline concordance marks the edge of the system’s reliability and the point where educational governance must remain firmly human.
What the data do not show is equally important. There is no demonstration here of learner outcomes, psychometric validity, inter-station reliability, or performance across other specialties. There is also no evidence that a generated station automatically earns the right to enter a high-stakes exam. The study instead establishes a narrower claim: complete OSCE stations can be generated from a single structured prompt in a way that experts consider acceptable, provided that faculty remain in charge of the clinical and curricular fine print.