From Utterance to Outcome: Pragmatic Competence in Task-Oriented Voice Agents

A thought paper on why voice agents misread what callers mean rather than what they say, and the architecture that closes the gap.

The failure is interpretation, not transcription

A caller says "I only have two pills left." Nothing in that sentence is a request. It is a report about a pill bottle, and a competent pharmacy agent treats it as a request for medication before the supply runs out.

Large language models handle this kind of inference well in isolation. What degrades in production is everything around it: holding the inferred goal across an authentication interruption, distinguishing a goal the agent may assume from a fact it must verify, and knowing when the backend has actually done what the caller asked.

Deployed task-oriented voice agents fail more often at pragmatics than at parsing. The operative failure is not mishearing the words but misidentifying the outcome the speaker wants, losing that outcome across turns, or acting on an inference the speaker never authorized.

The gap sits between three things that are easy to conflate.

An agent can score well on the first and fail the caller on the third. Empathetic phrasing is not evidence of task success, and conversational fluency has made this failure harder to detect, not easier.

The worked domain here is outpatient specialty pharmacy. Callers request refills, chase shipments, report that a label says no refills remain, and mention travel dates that silently change the deadline. The stakes are ordinary but real: a missed interpretation means a patient without medication. Pharmacy is a useful test case because its workflow vocabulary is precise — prescription, fill, refill, renewal, order, shipment, prior authorization — while callers use none of it consistently. The distance between what is said and what must be executed is wide and legible.

What follows concerns the design of agents built on general-purpose models with tool access, not the training of new models. Every mechanism proposed is implementable as prompt structure, tool contract, or evaluation harness. That constraint is deliberate: it is the situation of nearly every team deploying a voice agent today.

What the literature already settled

The problem is old. What is new is that the machinery for solving it now sits inside a general-purpose model, which makes the engineering question one of scaffolding rather than representation.

Grice's account of implicature established that speakers routinely mean more than they say, and that hearers recover the surplus by assuming cooperative intent. Austin and Searle supplied the complementary machinery: utterances are actions, and an indirect speech act performs one act by way of another — "can you check that" is a request wearing a question's clothes. Relevance theory reframed the inference as optimization, and the Rational Speech Acts framework later formalized it as recursive probabilistic reasoning between speaker and listener models, which is the closest formal analogue to what a prompted model does implicitly.

Meaning in dialogue is not transmitted but established. Clark and Brennan's grounding account treats each contribution as requiring evidence of understanding before the pair proceeds, and reference to an object is itself a collaborative achievement refined over turns. This is directly relevant to a phrase like "the other one," which has no referent outside the shared record of what has been discussed.

Early computational work modeled utterances as evidence about the speaker's plan, inferring goals and obstacles rather than classifying sentence types. Bratman's account of intention as commitment that constrains future deliberation underwrites the claim below that a stated goal should persist until achieved or withdrawn, not until the next turn.

Modern task-oriented systems replaced plan inference with slot-filling and dialogue state tracking, standardized by corpora such as MultiWOZ. The Action-Based Conversations Dataset is nearer to the present problem: its agents must satisfy customer requests while adhering to company policy, and a considerable gap to human performance remained.

The spoken-dialogue tradition had already reached the architectural conclusion this piece restates. RavenClaw separated the specification of the task from domain-independent discourse behaviours — grounding, error recovery, topic resumption — so that the latter could be reused across five deployed domains. Its non-understanding studies also established that recovery strategy, not recognition accuracy alone, drives task success in voice.

On clarification, conversational search treats the question as a first-class action: Aliannejadi and colleagues showed that a single well-chosen question can substantially improve retrieval, and MIMICS extended this to large-scale logs. Work on language models addresses the prior question of when to ask — CLAM clarifies selectively, Zhang and Choi estimate the utility of asking from entropy over candidate intents, and CLAMBER finds current models weak at it.

Separating the utterance from the outcome

An agent should decompose each contribution into six distinct things, because each carries a different burden of proof.

The first three are interpretive. The last three are what keep interpretation from becoming improvisation.

The asymmetry principle

The central design rule is that these layers warrant different evidentiary standards: infer the goal generously, and infer facts and authorization conservatively.

Generous goal inference is what makes an agent useful. A caller who says "it says no refills" is not asking for a lexicographic account of the label; the agent should proceed as though the underlying need is more medication, and say so in a way the caller can correct.

Conservative fact and authority inference is what makes the agent safe. Mentioning travel is not a request to ship elsewhere. Saying "I'm out" is not confirmation of which prescription. A goal is a hypothesis the agent can act on provisionally and cheaply revise; a fact or an authorization is a claim that must be established before an irreversible action depends on it.

The justification is a cost argument. Mis-inferring the goal costs a turn: the caller corrects, and the agent adjusts. Mis-inferring a fact or authorization costs an incorrect state change — a duplicate order, a shipment to the wrong address, a prescription request the prescriber never received. One error is conversational, the other transactional. Agents that apply a single confidence threshold to both either interrogate cooperative callers or act on things nobody asked for.

Expressions like "fine, whatever" are the clearest case where the two standards diverge. Read as goal evidence, the phrase is informative: something is unresolved. Read as authorization, it is worthless. An agent should let such an utterance escalate its sense of urgency and never let it satisfy a confirmation requirement.

Competence comes in layers, and they fail differently

Pragmatic competence in a commercial agent is not one capability but three, stacked, and instantiated over live state. Separating them matters because they have different owners, different update cadences, and different transfer properties across clients.

A pharmacy knowledge base can explain what a renewal is. It cannot say how this pharmacy submits one, whether one already exists for this patient, or what the backend confirmed thirty seconds ago. The distinction is between semantic knowledge and operational truth. Conflating them produces the characteristic failure of knowledge-grounded agents: a correct, general, useless answer delivered to someone with a specific open problem.

Each layer has a signature failure, which is why diagnosing production errors requires attributing them to a layer before fixing anything.

• Human communication failure — the agent answers the sentence and misses the request.
• Industry failure — the agent understands the need but picks the wrong workflow, treating a renewal as a refill.
• Company failure — the agent picks the right workflow and executes it against the wrong policy or the wrong tool.
• State failure — everything is correct except that the agent is reasoning about a snapshot that has since changed.

A single transcript labelled "the agent didn't understand" can be any of the four. Teams that do not separate them tend to respond to all of them by adding text to a prompt.

The top layer is the reusable product. The bottom layer is the integration work. The middle is the reason vertical specialization is defensible. Deciding where a given rule belongs is consequential: encode a client-specific policy in the industry layer and it will be silently wrong for the next client in the same vertical. A workable heuristic is that anything with a named owner inside the client organization belongs in the company layer.

Goals have to survive the conversation

The most common observable failure in deployed voice agents is not misinterpretation. It is amnesia. The caller states a goal, a sub-procedure intervenes, and the agent emerges from the sub-procedure asking what the caller was calling about.

Authentication is the canonical case. A caller who opens with "I need my prescription filled" has stated the goal; verifying identity is a prerequisite to serving it, not a new conversation. On completion the agent should resume — "Thanks, now let's get that refill started" — rather than restart.

The remedy is a small structured record, maintained continuously and cheaply. It does not require a reasoning essay per turn or an additional model call per sentence.

Two fields do most of the work. Uncertainty is what a clarification policy consumes. Progress is what prevents the agent from reporting success that the backend never granted.

A goal ends in exactly one of three ways, and an agent that cannot distinguish them will either abandon live goals or pursue dead ones.

An interruption returns to the active goal. Only the caller withdraws a goal, and only a verified backend result resolves one. A blocked goal is handed over with its record intact, which is what makes a transfer feel like continuation rather than a restart.

Callers also raise more than one thing. "While I have you" introduces a second goal without retiring the first, and the common failure is to let the newer goal silently displace the older one. The record should hold a short stack, with an explicit return: resolve the interjected goal, then name the original before continuing. Naming it is not politeness — it is the grounding evidence that lets the caller correct a wrong resumption.

Ask only when the answer changes what you do

Not all ambiguity is worth a question. The operative test is not whether the agent is uncertain about meaning but whether the candidate meanings imply different actions. Ask when two plausible readings would lead to materially different actions. Otherwise resolve from context and proceed, stating the reading so the caller can correct it.

This differs from the formulation in the clarification literature, where the target is usually uncertainty over the user's information need. In a transactional setting the decision is cheaper to make: the action space is small, enumerable, and known in advance.

The policy has to be tuned against three distinct errors, not one. Under-asking means the agent acts on a guess where the readings diverge; this produces wrong actions and is the expensive failure. Over-asking means the agent interrogates a caller who has already been clear; this is what callers describe as the agent "not listening," and it is worse in voice than in text, where a question costs several seconds of a stranger's patience. Mis-asking means the agent asks a question that does not discriminate between the candidate actions — "Can you tell me more?" is the common instance.

A question earns its place only if the answer changes what the agent does next. That criterion is testable directly: for each clarification in a transcript sample, check whether both possible answers lead to the same action.

A clarifying question should also narrow the target while preserving the goal. "Which medication — the metformin or the lisinopril?" keeps the refill alive; "What can I help you with?" discards it. This is the point at which the clarification policy and the goal record meet: clarification reads from Uncertainty and writes to Target, and it should never write to Goal.

Stated as an expected-cost comparison, the policy has explicit inputs. Take the candidate readings and their probabilities, and the action each implies. If every reading implies the same action, act — the readings differ in meaning but not in consequence. Otherwise weigh the cost of asking against the expected cost of acting on the most probable reading, which is the cost of that action under each reading in which it is the wrong one, weighted by that reading's probability. Ask when the expected cost of acting exceeds the cost of the question.

The formulation is not new. Horvitz and Paek proposed mediating clarification in spoken-language systems through explicit models of user goals and the expected utility of asking, within a broader account of mixed-initiative interaction. What the transactional setting changes is that the cost terms are largely known: the cost of a wrong address change or a duplicate order is a matter of record, and the cost of a question in voice is a few seconds of a caller's patience — small, but not zero.

Two consequences follow. The asymmetry principle above falls out of the cost structure: goal errors carry low action cost and are cheap to act on, while fact and authorization errors carry high action cost and warrant a question. And the threshold is tunable per action class rather than globally, which is how it should be configured in the company layer.

A status label is not an observation

Many apparent understanding failures are failures of the information supplied to the agent. A tool that returns nothing but a bare status — on hold — has not described the world. It has handed the model a token and left it to invent the business rule that token implies, and models are obliging enough to do so, fluently and wrongly.

A response should answer five questions the agent would otherwise guess at.

This is the same argument made in practitioner guidance on tool design: error messages and result formats are prompts, and their wording measurably changes agent behaviour.

Under-specified observations fail in both directions, and fixing one direction alone tends to worsen the other. A status label should not become an invented business rule — the agent must not infer from on hold that refills are impossible when the backend would accept one. Equally, the caller's evident desire for a refill should not cause the agent to disregard a genuine restriction. The generous-goal, conservative-fact asymmetry is enforceable only if the facts are actually legible.

The most consequential distinction in that table is the last. Asynchronous pharmacy operations — a renewal request sent to a prescriber, a prior authorization, a shipment booking — complete long after the call ends. An agent that cannot distinguish submitted from accepted from completed will close calls with confident resolutions that the backend never granted. Verified resolution requires the tool to say which of the three occurred, and the agent to say the same to the caller.

Evaluate outcomes, not responses

Pragmatic competence cannot be measured by rating responses. A reply can be fluent, empathetic, on-topic, and still leave the caller without medication. The evaluation must be anchored to outcomes, which is the design principle behind state-comparison benchmarks such as τ-bench — which scores agents by comparing final database state to an annotated goal state, and reports that strong function-calling models complete fewer than half of its customer-service tasks.

The test set should be drawn from de-identified transcript excerpts, not written by the team building the agent. Synthetic cases encode the authors' model of how callers speak, which is precisely the thing under test. Domain staff — pharmacy operations, in this case — label each excerpt on five dimensions: what the caller wanted; which facts in the transcript support that reading; what, if anything, needed clarification; the correct next action; and what would count as verified completion.

The second of those matters more than it appears. Without it, a disagreement between annotators is irresolvable, and the set silently encodes one person's intuitions.

Each underlying need should appear in several forms, because production traffic does. A set that contains only well-formed requests measures a distribution that does not exist on the phone.

Four measures, reported separately. Collapsing them into a single score hides which layer is failing.

Average performance understates the risk in a voice deployment, where the same caller with the same need may be handled differently on successive attempts. τ-bench's pass^k metric — success across repeated trials of the same task — is the right shape for this, and its authors report substantial degradation as k increases. For a pharmacy agent, the relevant question is not whether a refill request usually succeeds. It is whether it succeeds every time for a patient who will run out on Friday.

What this implies for one agent, and one platform

The architecture above was derived from a production deployment — a bilingual pharmacy agent handling inbound patient calls for a specialty pharmacy — and it generalizes unevenly.

For that agent specifically, the proposals reduce to four, in dependency order.

1. Persistent caller goals. A goal stated before authentication survives it. Resumption names the goal rather than asking what the call is about.
2. Explicit status-to-action rules. Every prescription status the backend can return is mapped, by the pharmacy, to what the agent may attempt and what it must say. No status reaches the model without a rule attached.
3. Clearer tool results. Tool responses carry reason, permissibility, blocking condition, available actions, and outcome — with submitted, accepted, and completed distinguished.
4. A transcript-based evaluation set. Labelled by pharmacy operations staff, covering the paraphrase variants above, scored on the four metrics.

The second is the dependency. Without it, the third has nothing to report and the fourth has no ground truth to score against.

For a platform serving multiple verticals, the reusable asset is the top layer plus the scaffolding: the interpretation frame, the goal record, the clarification policy, the tool-response contract, and the evaluation harness. None of these are pharmacy-specific. What does not transfer is the content of the company layer — the status vocabulary, the workflows, the routing rules, the definition of done. The practical implication is that the per-client work should be a structured elicitation of that layer, not a prompt rewrite.

Stated plainly: a shared system for understanding human goals, supplemented by industry workflows and each company's operating rules, is a more durable product than a collection of individually tuned prompts. The asset is the elicitation format and the evaluation harness, because they are what make the tenth deployment cheaper than the first. A prompt that works is an artifact; a method for producing prompts that work is a platform.

What this does not yet establish

The architecture is proposed, not proven. It is derived from production experience with a single agent in a single vertical, and the evaluation protocol above has not yet been run against it. Numbers would change the status of every claim here.

Four limitations deserve naming.

• No comparative evidence. Nothing here establishes that the goal record outperforms a well-written monolithic prompt. That comparison is the obvious first experiment.
• Single domain. Pharmacy has unusually precise workflow vocabulary. The layered architecture may not decompose as cleanly where the industry layer is thin or contested.
• Voice-specific effects are under-treated. Transcription errors, barge-in, and latency interact with clarification policy in ways this piece does not model. A question that is cheap in text is expensive in speech.
• Annotation cost. The evaluation protocol asks operations staff to label transcripts. That is real time from people with other jobs, and the protocol says nothing about how much labelled data suffices.

Three directions follow. First, an ablation over the components — goal record, status-to-action rules, tool-response contract — measured on verified resolution. Second, calibration of the clarification policy against caller-experienced cost, where an unnecessary question and a wrong action are weighted by what each actually costs. Third, an examination of whether the company layer can be elicited semi-automatically from historical transcripts and existing standard operating procedures, which is the difference between a method and a product.

A worked trace

The following exchange shows the goal record evolving turn by turn. It is constructed, not transcribed, and simplified. Names and drugs are placeholders.

Three things in the trace are the point. At turn 4 the agent asks a question that narrows the target without discarding the goal. At turn 7 it acts on the goal but asks about the address, because that is a fact with a state change attached. At turn 10 it reports submitted rather than done, because the tool told it the difference and the caller needs to know it.

References

Pragmatics and dialogue theory

Allen, J. F., & Perrault, C. R. (1980). Analyzing intention in utterances. Artificial Intelligence, 15(3), 143–178.

Austin, J. L. (1962). How to Do Things with Words. Oxford University Press.

Bratman, M. E. (1987). Intention, Plans, and Practical Reason. Harvard University Press.

Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 127–149). American Psychological Association.

Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition, 22(1), 1–39.

Cohen, P. R., & Perrault, C. R. (1979). Elements of a plan-based theory of speech acts. Cognitive Science, 3(3), 177–212.

Frank, M. C., & Goodman, N. D. (2012). Predicting pragmatic reasoning in language games. *Science*, 336(6084), 998.

Grice, H. P. (1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and Semantics 3: Speech Acts (pp. 41–58). Academic Press.

Schegloff, E. A., Jefferson, G., & Sacks, H. (1977). The preference for self-correction in the organization of repair in conversation. Language, 53(2), 361–382.

Searle, J. R. (1975). Indirect speech acts. In P. Cole & J. L. Morgan (Eds.), Syntax and Semantics 3: Speech Acts (pp. 59–82). Academic Press.

Sperber, D., & Wilson, D. (1986). Relevance: Communication and Cognition. Blackwell.

Task-oriented dialogue and clarification

Aliannejadi, M., Zamani, H., Crestani, F., & Croft, W. B. (2019). Asking clarifying questions in open-domain information-seeking conversations. *SIGIR '19*, 475–484.

Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M. (2018). MultiWOZ — a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. *EMNLP 2018*, 5016–5026.

Chen, D., Chen, H., Yang, Y., Lin, A., & Yu, Z. (2021). Action-Based Conversations Dataset: A corpus for building more in-depth task-oriented dialogue systems. *NAACL 2021*, 3002–3017.

Kuhn, L., Gal, Y., & Farquhar, S. (2022). CLAM: Selective clarification for ambiguous questions with generative language models. arXiv:2212.07769.

Zamani, H., Mitra, B., Chen, E., Lueck, G., Diaz, F., Bennett, P. N., Craswell, N., & Dumais, S. (2020). MIMICS: A large-scale data collection for search clarification. arXiv:2006.10174.

Zhang, M. J., & Choi, E. (2023). Clarify when necessary: Resolving ambiguity through interaction with LMs.

Zhang, T., Qin, K., Deng, Y., et al. (2024). CLAMBER: A benchmark of identifying and clarifying ambiguous information needs in large language models. arXiv:2405.12063.

Agent architecture, tools, and evaluation

Anthropic. (2024). Building effective agents.

Anthropic. (2025). Writing effective tools for agents — with agents.

Anthropic. (2025). Effective context engineering for AI agents.

Barres, V., Dong, H., Ray, S., Si, X., & Narasimhan, K. (2025). τ²-Bench: Evaluating conversational agents in a dual-control environment. arXiv:2506.07982.

Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv:2406.12045.

Spoken dialogue and mixed initiative

Bohus, D., & Rudnicky, A. I. (2005). Sorry, I didn't catch that! An investigation of non-understanding errors and recovery strategies. *SIGdial 2005*, 128–143.

Bohus, D., & Rudnicky, A. I. (2009). The RavenClaw dialog management framework: Architecture and systems. *Computer Speech & Language*, 23(3), 332–361.

Horvitz, E. (1999). Principles of mixed-initiative user interfaces. *CHI '99*, 159–166.

Horvitz, E., & Paek, T. (2001). Harnessing models of users' goals to mediate clarification dialog in spoken language systems. *User Modeling 2001*.

Sources accessed 17 September 2026. Pre-1990 references are cited from the standard editions and are not linked.