What Should an AI Agent Do When a Patient Says "The Usual"?

A patient calls and says she needs her refill, the usual one, and can she come in around the same time as last month.

A person handling that call does something so quickly it barely registers as thinking. They pull the chart. They see two active medications, one refilled six weeks ago and one three months ago, and infer she means the recent one. They notice the last visit was a Tuesday at 3 p.m. with a specific provider. They check whether that provider still has Tuesday afternoon slots. They notice the medication needs a clinical review this cycle. They handle it.

Almost none of that was in what she said. It came from the record, from context, and from knowing what "the usual" means for this patient in this practice.

This is the actual problem healthcare AI agents have to solve. Recognizing that she wants a refill and an appointment is the easy part, and it has been the easy part for a while. Everything after that recognition, the pulling and checking and inferring and confirming and doing, is where a request either becomes an outcome or becomes a task in somebody's queue.

This article is about that machinery. It is more technical than most pieces on this subject, deliberately, because the questions that separate healthcare AI agents that work in production from ones that demo well are mechanical questions.

Intent Is Not an Instruction

Here is the distinction the whole article rests on.

An intent is what a person wants. An instruction is an unambiguous, executable command against a specific system. The gap between them is filled by data the person did not supply and rules they do not know.

"Book me in with Dr. Rivera next week" is an intent. The instruction it maps to specifies a provider identifier, a visit type, a duration determined by that visit type, a location, a date and time drawn from real availability, and a reason for visit, validated against rules governing what that provider can be booked for on that day.

Every one of those fields has to come from somewhere. Some come from the conversation. Most come from the system of record. A few come from rules that exist only in a scheduling protocol document, or in the head of whoever has worked the front desk longest.

Healthcare AI agents that treat the conversation as the instruction will produce confident, wrong actions. That is the core failure mode, and it does not look like an error in a transcript. The conversation reads perfectly. The appointment is simply in the wrong slot type, which nobody discovers until the patient arrives.

The Seven Steps Between a Request and a Result

Here is the mechanical path. Each step is a place where healthcare AI agents either hold up or quietly fail.

One: Recognizing Intent

Classifying what the person wants from natural speech, including when they want several things at once and when the first thing they say is not the main thing.

This is the step language models do well, and it is where healthcare AI agents are usually judged. It is also, unhelpfully, the only step visible in a demo.

Two: Disambiguation and Gap Filling

Determining what information is missing and getting it, without interrogating the patient. "The usual" needs resolving. A date of birth needs confirming. A plan change needs surfacing before it becomes a problem at the counter.

The design question is what to ask versus what to look up. Every question asked is friction the patient experiences; every fact looked up is friction removed. Strong systems ask as little as possible because they can retrieve most of it.

Three: Grounding Against the Record

The most important step and the one most often skipped. Grounding means the agent's understanding of current state comes from the system of record, not from the conversation or from the model's own reasoning.

What appointments exist. What the coverage is today. What medications are active. What the provider's real availability is right now. Healthcare AI agents that reason about availability without reading it will eventually offer a slot that does not exist, and will do so fluently.

Four: Applying Rules

Deciding what is permissible. Visit type by provider, duration by visit type, authorization requirements by plan and service, escalation triggers by clinical content.

Rules should be deterministic and explicit. This is not a step to delegate to a model's judgment, because the same input must produce the same decision every time, and it must be auditable afterward. Our comparison of generative AI versus traditional automation covers where each approach belongs.

Five: Executing the Write

Actually changing something in a production system: creating the appointment, submitting the authorization, updating coverage.

Two properties matter and both are unglamorous. Idempotency means an operation can be retried safely without creating a second appointment. Retry logic means a portal that times out gets tried again rather than silently abandoning the request. Healthcare AI agents without these produce either duplicates or gaps, and both erode staff trust immediately.

Six: Verifying the Outcome

Reading back to confirm the change actually landed. An API call that returns success is not proof; a record that reflects the change is.

This step is frequently absent, and its absence is why some products report high completion rates that operations teams do not recognize. Attempted and completed are different states, and only verification distinguishes them.

Seven: Closing the Loop

Confirming to the patient in plain language, generating downstream tasks, writing a structured summary to the chart, and logging the action for audit.

Skipping this is what produces the second call, where the patient rings back to check the first call worked.

Why Probabilistic Systems Need Deterministic Guardrails

Now the central engineering argument.

Language models are probabilistic. Given the same input twice they may produce slightly different output, and they are optimized to produce plausible responses rather than to refuse when uncertain. That property is exactly what makes them good at step one and step two, where flexibility is the whole point.

It is exactly wrong for steps four and five. A scheduling rule must produce the same answer every time. An authorization submission must not vary in its content because of sampling. A write to a chart must be correct rather than plausible.

The architecture that works puts a deterministic layer around the probabilistic one, which is how reliable healthcare AI agents are built. The model interprets and gathers. Rule-based logic decides and executes. Voice AI sits at the front of that stack rather than constituting it. Validation sits between them, checking the model's extracted values against the record before anything is written.

This has practical consequences for how confidence should be handled. A well-built agent should have a threshold below which it does not act, and the correct behavior below that threshold is to ask a clarifying question or escalate, not to proceed with its best guess. A system that never escalates is not confident, it is unconstrained.

It also explains a finding that puzzles people. MGMA's May 2026 poll of practice leaders found that where AI had not improved productivity, the explanations offered centered on adoption and integration friction, including inconsistent use and interoperability problems that break workflows. The model was rarely the reported constraint. The scaffolding around it was.

Where Completion Actually Breaks

Six failure modes account for most of what goes wrong in production. Each has a specific engineering answer.

‍Ungrounded assumptions. The agent acts on what the patient said rather than what the record shows. A patient certain their insurance is unchanged is frequently wrong. The answer is reading current state before every decision that depends on it.

‍Stale reads. State is read at the start of a two-minute conversation and acted on at the end, by which time a colleague has booked the slot. The answer is re-validating immediately before the write.

‍Partial completion. Three of five steps run. The appointment exists but intake forms never went out. The answer is transactional thinking: define the unit of work, and treat anything less than all of it as incomplete rather than as success.

‍Silent timeouts. A payer portal stops responding and the request evaporates with no alert. Given that CAQH CORE reports only around 35 percent of medical prior authorizations run fully electronically through the standard transaction, a great deal of payer interaction still happens through exactly the portals most likely to fail this way. The answer is explicit failure states and alerting.

‍Unhandled edge cases. Duplicate patient records, a plan the rules do not cover, a provider on unexpected leave. The answer is a defined default: escalate rather than improvise.

‍Escalation without context. The transfer happens but the receiving staff member gets a notification rather than the conversation, so the patient repeats everything. The answer is treating the payload as part of the escalation rather than as a nicety.

Notice that none of these are model failures. Every one is an execution failure, which is why evaluating healthcare AI agents on conversational quality tells you so little about how they will behave in month six.

What to Ask About Mechanics

Conversations about healthcare AI agents improve considerably when the questions get specific. These are the ones worth asking.

‍Show me a write landing, not a screenshot. Ask to watch a record change in a test environment, then refresh and see it persist.

‍What happens when the write fails? Listen for whether there is a defined failure state, who gets alerted, and how quickly. Vagueness here is the answer.

‍Is the operation idempotent? Ask directly what happens if the same request is submitted twice after a timeout. If the answer is a duplicate appointment, you now know something important.

‍Do you verify after writing? Ask whether success is determined by an API response or by reading the record back.

‍What is the confidence threshold for acting? Ask what happens below it, and what share of interactions fall there.

‍Where do the rules live? Rules held in prompt text behave differently from rules held in deterministic logic. Ask which, and ask how a rule change is deployed and tested.

‍What does the audit trail contain? Every action taken in a clinical or financial system should be reconstructable afterward.

For the integration-side questions that sit underneath all of this, our guide to integrating AI with your EHR covers what to verify before deployment.

Here's How Confido Health Can Help

This article traced the path from a patient's intent to a verified outcome and identified where healthcare AI agents break along it. Confido Health's healthcare AI agents are engineered around that path rather than around the conversation at the front of it.

Here is what Confido Health delivers:

• Grounded execution against your systems, reading current state from your EHR or PMS before acting, so decisions reflect what is true now rather than what was asserted on the call
• Deterministic rule enforcement for scheduling logic, visit types, provider preferences, payer requirements, and escalation triggers, applied consistently and auditably rather than inferred
• Integration-first approach with 40+ EHR and PMS systems including Epic, Athenahealth, and eClinicalWorks, with verified write-back across scheduling, eligibility, prior authorization, referral intake, refills, and payments
• Structured escalation with full context, warm transferring or routing a task carrying the entire conversation, so a person never restarts from a blank screen
• A complete audit trail of transcripts, actions, and write-backs, so every action taken in your systems can be reconstructed
• Empathetic, natural conversations with 97 percent patient satisfaction, in more than 20 languages, answering every call on the first ring
• Proven ROI, with up to 70 percent reduction in staff call burden, 60 percent reduction in cancellations, 80 percent reduction in manual administrative work, 75 percent faster prior authorization processing, and a 15 to 20 percent increase in revenue collections
• Live in under 30 days using expert-approved templates co-built with practicing physicians and operations leaders

Confido Health is more than a tool. It is the execution layer that carries a patient's intent all the way to a verified outcome inside your systems.

Want to watch a write actually land in your environment rather than in a slide? Let's get started today.

Still in research mode? Start with our explainer on what an AI voice agent is, then see the front desk KPIs worth tracking.

Frequently Asked Questions

How do healthcare AI agents complete a task end to end?

Through seven steps: recognizing intent, filling information gaps, grounding against the system of record, applying deterministic rules, executing the write, verifying the change landed, and closing the loop with the patient and downstream tasks. Recognition is the easiest step and the only visible one in demos.

What is grounding in healthcare AI agents?

Grounding means the agent derives its understanding of current state from your EHR or PMS rather than from the conversation or its own reasoning. Without it, an agent can fluently offer an appointment slot that does not exist or act on coverage details that changed last month.

Why do AI agents need deterministic rules if the model is capable?

Language models are probabilistic and optimized to produce plausible output. Scheduling rules and payer requirements must produce identical, auditable results every time. The working architecture uses the model to interpret and gather information, and deterministic logic to decide and execute.

What is idempotency and why does it matter in healthcare?

Idempotency means an operation can be safely repeated without duplicating its effect. It matters because timeouts are common in payer portals and clinical systems. Without it, a retry produces a second appointment or a duplicate authorization, which creates more work than the automation saved.

What is read-after-write verification?

It is reading the record back after an action to confirm the change actually persisted. An API response indicating success is not the same as a record reflecting the change. Verification is what distinguishes an attempted action from a completed one in completion reporting.

Why do AI deployments underperform in healthcare?

Usually because of execution and integration rather than model quality. MGMA polling found integration friction, inconsistent adoption, and interoperability problems breaking workflows among the leading explanations where AI had not improved productivity. Healthcare AI agents fail at the scaffolding around the model far more often than at the model itself.

What should an AI agent do when it is uncertain?

Ask a clarifying question or escalate with full context. A well-designed agent has a confidence threshold below which it does not act. Systems that never escalate are not more capable, they are unconstrained, which is a considerably worse property in a clinical or financial system.

What is partial completion?

Partial completion is when some steps of a workflow run and others do not, such as an appointment created without intake forms sent. It frequently gets reported as success. The fix is defining the unit of work explicitly and treating anything short of all steps as incomplete.

How is completing a task different from understanding a request?

Understanding is a language capability that improves as models improve and generalizes across customers. Completion is an integration and state-management capability rebuilt for each EHR, rule set, and exception path. Two products can sound identical and behave very differently in production.

What technical questions should you ask an AI vendor?

Ask to see a write land and persist, what happens when a write fails, whether operations are idempotent, whether outcomes are verified by reading back, what the confidence threshold is, where rules live, and what the audit trail contains. Failure behavior reveals more than accuracy claims.