TimelyCare Uses AI to Filter Risk Without Handing Away Clinical Judgment
TimelyCare’s experiment with health care AI began with a simple operational problem and a high-stakes clinical setting. On an anonymous peer-support forum used by students across multiple colleges, too many posts were being flagged for review. The system generated noise, not just alerts, and every false positive consumed time from staff who needed to focus on the small subset of messages that might signal genuine distress. AI became attractive not because it could replace human judgment, but because it might reduce the volume enough for clinicians to see the posts that mattered.
The organization’s broader response was shaped by a second problem: how to govern these tools before they were deeply embedded in daily workflow. TimelyCare became one of the first five organizations to earn URAC’s Health Care AI Accreditation, and the first in behavioral health. That distinction is more than a badge. In a field where an overlooked warning can have immediate consequences, the quality system around an algorithm becomes part of patient safety, not a side process.
A Peer Community That Needed Better Signal Detection
The Peer Community feature sits at the center of the most clinically sensitive use case described in the material. Students can post anonymously, and other students can respond. The challenge is obvious: anonymity can encourage honest disclosure, but it also makes it harder to know when a message crosses from ordinary peer conversation into potential crisis. TimelyCare initially built a version that flagged a substantial number of posts, only to discover that false positives created more work than value. Human review was being pulled toward low-yield content, which is exactly the kind of drift that can weaken a safety net.
AI-assisted review was introduced to analyze posts and distinguish those that should be escalated for human attention from those that could remain in the community. Development was iterative, involving prompt writing, testing, adjustment, and retesting. The process was not a single deployment event; it was a sequence of refinements designed to make the model more discriminating while preserving clinical escalation where needed. In behavioral health, a high-recall system that floods clinicians with low-risk material can be nearly as dangerous as a narrow one that misses warning signs. Both failure modes distort the response.
A concerning post needs a trained human at the end of the chain.
Accreditation Standards Shaped the Build, Not Just the Review
Heather Geracie, Vice President of Clinical Excellence, already worked inside a quality assurance structure supported by URAC’s Telehealth Accreditation. The AI effort asked a different set of questions. How should the system be tested? How often should it be retested? What happens when model performance changes over time? Those concerns move beyond ordinary clinical workflow into issues such as bias, drift, documentation, and technical accountability. Geracie’s background as a registered nurse with emergency medicine experience gave her a strong clinical lens, but the accreditation process added structure for questions that sit at the intersection of care delivery and engineering.
The value of the standards was practical. They gave clinical leaders a framework to bring into conversations with product and engineering teams while the tools were still being shaped. Roughly six months of policy and procedure development followed, with collaboration across departments. Product and engineering brought technical expertise. Clinical teams brought experience with evidence-based care, quality review, and escalation. The point was not simply to approve a tool after it existed; it was to define what acceptable performance should look like before the organization relied on it operationally.
Chart Auditing at Scale Depends on Prioritization, Not Automation Alone
TimelyCare also built an AI chart-auditing tool to help a small quality team review care at larger scale. Geracie said the organization expected to exceed 300,000 telehealth visits in a year, a volume that makes manual review of every chart unrealistic. The tool uses the clinical practice guidelines, policies, and scoring methodology already embedded in the organization’s Quality Assurance Program. Completed charts are scored from one to five, with higher scores identifying records that may deserve closer review.
The score is only a starting point. Human reviewers then apply clinical judgment to determine whether the case calls for policy-driven feedback, coaching, or corrective action. This is a familiar quality model in another form: the algorithm does not replace review, but it helps rank where attention should go first. For a small quality team, that can change the difference between sporadic sampling and a more disciplined, risk-based process. In telehealth, where volume can outpace staff capacity, prioritization is not a luxury. It is the mechanism that keeps oversight from collapsing under its own workload.
Response Pathways Matter as Much as Detection Thresholds
An alert is only as useful as the workflow behind it. For the Peer Community, a concerning post must move quickly to a professional who can decide whether a student needs support. For chart review, a flagged record must be examined in a way that feeds back into care improvement. If no one owns the downstream step, the tool becomes a notification engine rather than a safety intervention. TimelyCare’s model tries to avoid that trap by defining who receives each flag, how quickly review should occur, and what actions may follow.
The organization also planned for misses. Periodic auditing of unflagged posts and lower-scoring charts helps expose blind spots and refine detection over time. Ongoing evaluation includes response times, false negatives, and differences between clinician assessment and AI output. Those measures are essential because a model that performs well at launch can weaken as language, user behavior, or clinical patterns shift. In mental health settings, the content itself can evolve quickly, and model drift can quietly erode sensitivity to the kinds of language that matter most.
Documented oversight ties the whole structure together. Testing procedures, system modifications, and follow-up actions all need to be recorded. Without that discipline, it becomes impossible to know whether the tool is improving care or simply changing the appearance of efficiency. Behavioral health triage leaves little room for ambiguity. If AI is going to help sort the stream of messages and charts, clinicians still have to own the response when the system flags risk, and they have to do so with clear accountability for what the model sees, what it misses, and how those decisions are checked over time.