"What is the top-selling product?"


Ask that of an agent connected to an orders database and you get back a confident SQL query, a chart, and a name. The problem is that the name changes depending on which SQL the model wrote — and every SQL is defensible. Highest revenue is a MacBook Pro at $1.24M. Highest units moved is a USB-C Cable at 84,200. Most distinct orders is a Phone Case at 31,450. Same question, three different products, three different SQLs.


The interesting engineering here is not how to detect that the question is ambiguous. It is what the system does the moment it decides the model is not allowed to guess.

One Question, Three SQLs, Three Different Products

Without the modal loop

The model reads "top-selling," silently picks a metric, and writes SQL against it.

  • the answer is a single number and a single product name
  • the metric that produced it is invisible to the user
  • all three interpretations produce plausible-looking output
  • the user finds out only if they cross-check a chart they weren't expecting

With the modal loop

The model recognises the shape "ranking language without a metric," stops writing SQL, and calls a tool.

  • the chat input disappears and a wizard appears
  • the user picks Revenue, Units, or Orders — or types their own
  • the follow-up SQL embeds the choice as a literal
  • the answer that comes back is anchored to a decision the user made

Below is the same three-way split, live. Switch tabs to see the SQL each interpretation would generate and the row that would come back.


"What is the top-selling product?"

Same question. Three interpretations. Three different answers.

sql
SELECT products.name, SUM(order_items.unit_price * order_items.quantity) AS revenue
FROM order_items
JOIN products ON order_items.product_id = products.id
GROUP BY products.name
ORDER BY revenue DESC
LIMIT 1
ProductRevenue
MacBook Pro$1,240,000

Three plausible SQLs. Three top-selling products. One plain-English question. The fix is not smarter parsing. It is a modal loop that forces the model to hand the decision back.

One Flag Runs the Whole Modal Loop


◆Key Takeaway

Flipping needsApproval to true on the clarification tool activates three behaviours at once: the auto-approval short-circuit is skipped, the pending-tool detector fires and hands the wizard renderer an input-available tool part, and while any pending tool exists the chat composer is hidden and the wizard sticks to the bottom of the conversation. Flipping it back to false reverts them together.


There is a normal fast path for render-only tools. A chart tool, for example, gets its output auto-written to the string "rendered" the moment the model calls it — the frontend paints the chart and the conversation moves on. The clarification tool falls off that fast path because of the flag. The auto-approval guard checks needsApproval !== true and refuses to write an output. The tool stays in the input-available state until a human answers.


One flag, three downstream behaviours, all keyed off the same registry entry. That is the whole gate.


The gate is scoped to a live chat session. The tool is registered against the AI SDK's chat client and lives inside the input-available / output-available state machine. Outside that context — batch runs, dashboard refreshes, scheduled reports — the tool has no meaning, the pending detector has no messages to walk, and the loop cannot fire. Ambiguity in those contexts has to be prevented at a different layer.

From Ambiguous Prompt to Literal-Substituted SQL

The loop from the user's first question to the SQL that runs after they answer is six beats. Each one is a small, boring piece of glue — that is the point.


  1. 1

    The system prompt injects the ambiguity archetypes. A capability named intentClarification contributes a fragment with one blanket guardrail ("do not act on presumed user intentions") and four when / ask / reason entries the model is expected to recognise. This runs on every user-facing chat.

  2. 2

    The model calls render_ask_user_question. Input is validated against a Zod discriminated union of choice and query_choice questions. The registry entry gates the tool with needsApproval: true.

  3. 3

    The pending-tool detector swaps the composer for the wizard. A hook walks the message list in reverse, finds the first input-available tool part whose registry entry needs approval, and hands it to the wizard renderer. The wizard sticks to the bottom of the conversation; the normal composer is hidden.

  4. 4

    The auto-approval short-circuit is skipped. The guard that would write "rendered" for a render* tool checks needsApproval !== true and falls through. The tool stays pending until a human answers.

  5. 5

    The wizard serialises the answer, including any custom text or notes. A helper wraps the answers and — if the user typed a custom answer or added notes — attaches a memoryHint string. The tool output is written back and the AI SDK auto-fires the next turn.

  6. 6

    The follow-up SQL embeds the choice as a literal. A separate guardrail tells the model that when the user picks "United States" for country, it must write WHERE country = 'United States' — not WHERE country = $1 — because the dashboard frontend cannot substitute parameters at render time.


The tool output the AI SDK sees at the end of step 5 looks like this — same shape whether the user picked a predefined option or typed their own. The memoryHint is present only when there is user-authored text:


json
{
  "answers": [
    {
      "type": "query_choice",
      "question": "Which country?",
      "choice": { "label": "United States", "value": "United States" },
      "notes": "For this dashboard always use US only; we exclude Puerto Rico."
    }
  ],
  "memoryHint": "The user provided a custom answer or added notes. This may contain personal preferences, corrections, or context worth saving to memory."
}

Nothing in the six-step path is clever on its own. What makes the loop work is that the flag from section 2 wires them together — the moment needsApproval: true is set, steps 3 and 4 activate, and once the tool call has been emitted no code path advances the turn until a human answers. The model cannot decide to skip ahead mid-loop. What it can do is not emit the call in the first place — that failure mode gets its own section below.


The memoryHint string in the payload above is deterministic, not model-inferred: a helper called hasCustomContent checks for a trimmed, non-empty freeText or notes on any answer, and the hint is either present verbatim or absent. The model does not decide whether it fires.

Two Question Types, Not Four

The schema accepts exactly two shapes: choice and query_choice. A choice question is a static enumeration — timeframes, metrics, boolean toggles. A query_choice question runs a live SELECT to populate its options — customer names, account IDs, product SKUs — so the user picks a real value the model can embed verbatim on the next turn.


Both types share a header, a question string, and an "Other" fallback. They diverge only in where the options come from. That is the reason the schema is a discriminated union rather than a single flexible object: a static timeframe list looks nothing like a live-database lookup, and earlier attempts to mash the two shapes into one produced a model that kept malforming its inputs.A normalizeQuestionType preprocessor runs before Zod validation and absorbs the most common model error: it infers type: "choice" when the input has options and no type, and type: "query_choice" when it has sql and no type. The model can omit the discriminator and still succeed.


A valid choice tool call, on the wire:

json
{
  "questions": [
    {
      "type": "choice",
      "question": "Which timeframe?",
      "header": "Timeframe",
      "multiSelect": false,
      "options": [{ "label": "Last 30 days" }, { "label": "Last quarter" }]
    }
  ]
}

The wizard renders one question per step, so the array is min(1). Validation refuses empty submissions and refuses "Other" checked without any custom text — the model cannot ask a question that the user is then unable to answer meaningfully.

The Guardrails Name Shapes, Not Answers

The prompt-level rules do not try to teach the model what ambiguity is in general. They hand it a short list of shapes it is expected to recognise. There are four:


  • Timeframe. The request references recency or trends without a concrete timeframe.
  • Ranking metric. Ranking language like top / best / active is used without a clear metric.
  • Insufficient info. The user is providing insufficient or ambiguous information about the data.
  • Custom visualization intent. The user requests a custom visualization, dashboard, or interactive display that standard charts cannot represent.

Each entry ships as a when / ask / reason triple, so the model is not just told the shape — it is also given a template question to ask and a justification for asking it. The blanket rule sits above the four: do not act on presumed user intentions, use render_ask_user_question before proceeding.


The literal-substitution rule at the other end of the loop is the same technique in a different place.The literal-substitution rule is prose in the system prompt, not a runtime check. If the model ignores it and writes WHERE country = $1, a raw SQL runner will still execute the query — the dashboard is what breaks. Section 7 pays this off. The prompt tells the model: when the user picks "United States," write it as a string literal, because the frontend has no parameter substitution step and the SQL that renders in the dashboard is the SQL you emit.


Neither guardrail attempts general reasoning. They enumerate the shapes the model must handle correctly and count on the model to match against them.

The Failure Window the System Cannot Close

The model just doesn't call the tool.


This is the largest failure window, and it is pure omission. The user asks a superlative question with no metric, the next turn runs SQL directly, no render_ask_user_question in between. The guardrail is prose in the system prompt; nothing between the model's decision and the SQL runner enforces it at runtime.



What exists instead is a diagnostic that runs offline against saved conversation traces. It grep-triangulates on ambiguous-superlative regexes ("top", "best", "most", "active" without a nearby metric), flags any turn that ran SQL without a preceding clarification call, and reports a pattern code. The diagnostic assumes trace persistence — no saved trace, no detection. A deterministic preprocessor that flagged ambiguous prompts before the model saw them is called out in the diagnostic's own notes as an aspirational next step. It has not been built and no committed design exists for it.


The reader who wants to copy this loop should copy this fact too: the model can and does miss ambiguous prompts, and the runtime does not know.

Three Smaller Failure Windows

The other ways the loop breaks are smaller and each has a partial mitigation.


The tool call arrives malformed. The normalizeQuestionType preprocessor catches the most common model error — a missing type field. Any other schema violation surfaces as an error state inside the wizard body, which shows the raw input as JSON. The tool then sits input-available forever — the chat input stays hidden, and only a page reload or a manual retry unblocks the user. There is no "regenerate," "skip," or "fall back to text input" affordance. This is a real dead-end when it fires.


The user answers "Other" with unmeaningful text. The wizard accepts any non-empty string. The memoryHint fires and the model may write a memory. Weak or noisy memory is possible. A separate knowledge-review capability tries to counter this with a prose rule — "if the content is ambiguous, ask one targeted follow-up instead of saving weak memory" — but enforcement is prose-level; a determined model can still write a bad memory. The system does not detect the noise; the next turn simply carries it.


The follow-up SQL parameterises anyway. The literal-substitution rule is prose. If the model ignores it, a raw SQL runner still executes the query — the dashboard is what breaks, because the frontend has no parameter substitution step. Nothing catches this before render; a broken dashboard render is the detector.


Live-row options fail to load. A query_choice question fetches its options against the connected database. If the SQL errors or returns zero rows, the user only sees the "Other" fallback, and the agent gets whatever they type — which defeats the point of picking a real value. The wizard does not distinguish "no rows because the filter is empty" from "no rows because the SQL was wrong."


What a Standalone Checker Page Would Need

The visualiser above is a hand-authored artifact of one blog post. Three fictional SQLs against a fictional schema, hard-coded to make one specific point. Turning that into a real "checker page" — a natural-language question in, three plausible SQLs out — is a distinct build, and the current repo does not have the pieces yet.


Four pieces would need to land. First, a live database adapter so the SQLs the checker enumerates run against real tables, not a fictional catalogue. Second, more ambiguity archetypes than the current four: the guardrails enumerate timeframe, ranking metric, insufficient info, and custom visualization intent, and do not enumerate entity-name ambiguity ("show me revenue for Acme" — which of three Acmes?), timezone ambiguity, aggregation-level ambiguity ("by month" — calendar or fiscal), or unit ambiguity ("in dollars" — USD, local currency, constant dollars). The archetypes a real checker would surface are broader than the guardrails currently cover.


Third, a repair pass for database-level ambiguity. The current accuracy work explicitly notes that Postgres, MySQL, MariaDB, and SQL Server have no registered recoverers — an ambiguous column "id" error on a JOIN has no automatic qualifier fix. A checker that flags database-side ambiguity would want this pass, and it does not exist.


Fourth, a recovery UI for the wizard's malformed-input dead-end. The current behaviour — chat stuck, page reload required — is acceptable for a rare model failure inside a conversational flow. It is not acceptable for a checker page whose whole purpose is inspecting the model's tool calls.


None of these are hard, and each is a small enough chunk to ship independently. The point of writing the loop this way — one flag, six steps, four archetypes — was to make each piece replaceable without touching the others.