Automated reminders improved evaluation completion, but feedback still lagged
Emergency medicine clerkships depend on frequent, rapid feedback, yet the administrative machinery behind those evaluations can be punishingly manual. At Thomas Jefferson University, third-year medical students were required to initiate digital evaluation requests after each clinical shift, leaving clerkship staff to chase down incomplete forms week after week. The burden was large enough to justify a redesign of the process itself.
Investigators introduced an automated reminder system built on existing survey tools, Microsoft Excel, and Microsoft Power Automate, then compared performance across two 48-week periods before and after implementation in September 2024. The result was a modest but statistically significant rise in evaluation completion, paired with a slower pace to completion. In practical terms, the system made more evaluators respond, but it did not make them respond sooner.
Completion rose, especially among attending physicians
Across the study window, 5,747 evaluations were sent to attending and resident evaluators. Overall completion increased from 71.4% before automation to 75.1% after implementation, with p < 0.001. The larger gain came from attending physicians: completion climbed from 73.9% to 81.6%, also with p < 0.001.
That pattern fits the operational logic of the intervention. Automated nudges can reduce forgetfulness, create a persistent cue in an overloaded workflow, and lower the friction of an action that competes with clinical care, documentation, and paging. In a setting like emergency medicine, where evaluators move quickly between patients and tasks, even a simple reminder loop may recover evaluations that would otherwise disappear into the inbox.
Resident evaluator completion improved less, and the intervention did not erase the underlying delay in how quickly evaluations were returned. Median time to completion for resident evaluations increased from 2.67 days to 3.98 days. Attending completion time also lengthened, from 2.59 days to 2.74 days, with p = 0.002.
Feedback arrived more often, but not more quickly.
Automation trimmed administrative burden without solving timeliness
The pre-intervention process demanded close monitoring by the clerkship administrator, who identified incomplete evaluations at the end of each week and sent reminder emails manually. Escalation sometimes involved the clerkship director and other senior stakeholders, including program directors and departmental leadership. The institution estimated that the manual process required about 12 hours per month of administrator effort, not counting calls, texts, or videoconferencing for stubbornly incomplete cases.
The automated workflow changed that sequence by linking Qualtrics with Excel and Power Automate. Once a student sent a request, the system logged it automatically, tracked completion in real time, and sent reminder emails every 2 days for incomplete evaluations younger than 28 days. From an administrative standpoint, the design substituted repeat human labor with a rules-based digital process that could run continuously in the background.
A theoretical cost analysis suggested the shift could yield net savings of $2,850 in the first year and $9,560 over 3 years. Those estimates were based on institutional experience, state salary estimates, and projected programming development and maintenance costs. The financial logic is straightforward: clerical time is expensive when repeated every week, and small efficiencies accumulate quickly in a high-volume clerkship.
Faster systems do not automatically create better feedback
The educational consequence is less straightforward. Evaluation completion matters because written feedback helps students recognize gaps, reinforce strengths, and prepare for future rotations and residency applications. But delayed feedback loses some of its formative value. A comment delivered several days later may be less actionable than one returned while the clinical encounter is still fresh in the student’s mind.
This study exposes that tension clearly. Automation improved the probability that evaluators would eventually respond, yet the median delay grew rather than shrank. A reminder system can move the denominator in the right direction without fixing the educational interval that determines whether feedback still feels immediate enough to change behavior. The design may also nudge evaluators to defer the task until a later reminder arrives, especially when the workflow competes with patient care and documentation.
In that sense, the intervention solved a compliance problem more than a pedagogic one. Medical educators seeking timelier feedback may need to combine reminders with other mechanisms, such as clearer expectations at the point of care, tighter turnaround goals, or simpler evaluation formats that reduce the cognitive cost of completion.
What this means for clerkship operations
The study has a specific institutional focus, but the operational lesson extends beyond a single emergency medicine rotation. Clerkships across specialties face the same pattern: evaluation systems depend on busy clinicians, and manual chasing drains staff time while creating uneven completion rates. Automation offers a scalable way to reduce repetitive administrative work, especially when the reminder process is tied to systems already in use.
Still, the evidence here argues against assuming that automation alone improves the educational quality of feedback. The study did not assess cognitive burden or evaluation quality, and those outcomes will matter if schools want to know whether the system changes what students actually receive, not just whether forms are eventually submitted. The gains in completion rate were real, but they were modest, and timeliness moved in the wrong direction.
For clerkship leaders, the most concrete question is whether an automated reminder engine should be viewed as a back-office efficiency tool or as a genuine feedback intervention. This study supports the first claim more strongly than the second. Further work will need to determine whether reducing administrative burden can be paired with faster, more useful feedback at the bedside and in the classroom, rather than merely increasing the number of forms that finally get finished.