A simple trick for your RAG system can save a lot of latency.

The benchmark result that made this worth shipping was simple: P90 dropped from about 8.0 seconds to 4.9 seconds.

The change was not a better embedding model, a smaller prompt, or a new vector database.

A user sends one message. Before the assistant can safely answer, the system has to run a policy layer: meaningless-message detection, escalation, topic checks, dissatisfaction checks, routing, and procedure matching.

Only after that does RAG retrieve documents and call the answer model.

Drawn the obvious way, the request path is linear:

That shape is easy to trust. The system knows which branch owns the turn before an answer exists.

It is also slow. The user waits for policy time plus RAG time.

The obvious improvement is to parallelize the policy work first.

In an ecommerce assistant, that means checking safety, classifying intent, choosing between product help, order status, returns, refunds, and support-ticket paths, and matching any curated procedure at the same time. Once that work says "this is a product-help question," RAG can retrieve documents and write the answer.

That is better than the linear path. The policy layer no longer pays every classifier one after another.

But the RAG work still starts after the agent decision. If RAG takes five seconds, those five seconds still sit after the policy fanout.

The next move is to start a possible RAG answer before that decision is finished.

Now the common product-help path waits for the slower branch, not the sum of both branches.

But this creates the real problem: RAG may finish before the system knows whether RAG should answer.

The gate decides whether the prepared RAG answer becomes the response for this turn, or whether it gets discarded because another branch won.

Here, "flush" means opening the gate and releasing the prepared RAG answer to the response path.

Until that happens, the answer is just work the system prepared early. Once the gate flushes it, the answer becomes part of the user-visible turn.

So the system starts the slow answer early, then waits to release it until the orchestrator has enough information to commit it.

The actual fanout

The production graph is not "guardrails, then RAG." It is a bundle of concurrent work around one customer turn.

Some work asks, "What kind of message is this?" Some work asks, "Which agent owns this?" Some work asks, "Is there a curated procedure for this exact case?" RAG asks, "If I had to answer, what would I say?"

The important part is that only the orchestrator may turn those answers into a customer-visible action.

The gate buffers events

The gate is a small state machine.

While it is buffering, RAG can write. The customer sees nothing.

The buffer is a holding area for events that may never become part of the conversation.

When the orchestrator decides RAG is the right answer, it opens the gate.

flushed = await gated_writer.open_and_flush()

That call sends every buffered event to the real writer in order. Then it changes the state to open.

When the orchestrator decides RAG should lose, it drops the gate.

await gated_writer.drop()

The buffered events vanish. The state becomes dropped.

If the speculative task tries to write one more token while cancellation lands, the gate swallows it.

The orchestrator gatekeeps the flush

The gate must not open because RAG finished.

RAG finishing only proves that the model produced something. It does not prove this turn belongs to the product-help path. It does not prove a curated procedure failed to match.

It also does not prove the speculative prompt had all the final instructions. A support-handoff offer, for example, may be known only after sentiment and policy finish.

So the flush waits for the orchestrator verdict.

The order is the design. The customer sees text only after the decision says RAG won.

Side effects wait too.

The prepared product-help answer can do more than produce text. It can close a trace, write cache data, update metrics, increment counters, or record session state.

Those acts are output. They just do not appear in the chat window.

If the buffered RAG answer is dropped, its pending side effects should be dropped too.

Otherwise the system can commit trace data, metrics, counters, or session state for an answer the user never saw.

So the order is: flush the answer first, then commit the side effects for that flushed answer.

A plausible answer can still be the wrong producer

The curated procedure case is the one that makes the gate feel less like a latency helper and more like a control boundary.

Imagine the user asks about a return window. Plain RAG starts producing a general answer from knowledge-base passages.

At the same time, the procedure matcher finds a curated operational case: ask for the order state, use a specific macro, include the returns portal link.

The speculative answer may be plausible. It is still the wrong producer.

Latency wins count only when they preserve the owner of the reply.

The gate has to count what leaked

There is one ugly edge after the gate opens.

Suppose the orchestrator approves RAG and flushes buffered tokens. The customer has now seen the first part of the answer.

Then the speculative RAG task fails before returning its final response object.

The system might be tempted to fall back to the old sequential path.

That is safe only if nothing has reached the client.

If some tokens already flushed, a fallback would duplicate or contradict the partial answer already on the wire.

So the gate tracks what leaked.

With total_written = 0, fallback is cheap.

With total_written > 0, the client has history. The system cannot replay a second answer into the same stream.

The gate code is small

The gate only needs three behaviors.

def __call__(self, event):
    if state == "open":
        underlying(event)
        total_written += 1
        return

    if state == "dropped":
        return

    buffer.append(event)

async def open_and_flush(self):
    for event in buffer:
        underlying(event)
        total_written += 1
    buffer.clear()
    state = "open"

async def drop(self):
    buffer.clear()
    state = "dropped"

At runtime, each event follows the gate state.

No speculative event reaches the user until the orchestrator authorizes it. No discarded branch can keep writing after it has lost.

The buffer also needs a cap.

If guardrails stall and the LLM keeps streaming, the API process should not store an infinite answer. A bounded buffer turns that failure into a degraded stream rather than a memory problem.

What changed in my mental model

I used to think of streaming as a transport detail: a nicer way to send the answer as it arrives.

In a speculative pipeline, streaming is part of the decision system.

The rules become practical:

• A speculative producer may prepare output. It may not write to the real response writer or commit side effects.
• The orchestrator owns the flush.
• Every losing branch calls drop.
• Side effects wait behind the same decision as tokens.
• The system records whether anything reached the client before fallback.

These rules sound fussy until you read one failed transcript.

The customer sees half an answer, then a handoff. The model starts a confident product-help response, then the system realizes this was a support-ticket case.

Or a guardrail rejects a turn after the first sentence already crossed the wire.

The useful latency trick

Parallel guardrails with speculative RAG has a clean performance shape.

The latency improvement comes from waiting for max(policy branch, RAG branch) instead of policy branch + RAG branch.

The tradeoff is model cost.

Speculative RAG sometimes spends retrieval and generation tokens on an answer that will be dropped. If routing chooses returns, if a curated procedure wins, or if a support handoff needs a different response, that speculative answer was useful only as a latency bet.

For this workflow, I think that tradeoff is worth it. The user feels the common product-help path immediately, and the gate keeps the extra work from becoming a wrong answer.

For rejected or rerendered paths, the gate has to drop the buffered answer before another producer runs.

The gate keeps the routing and policy outcome separate from model speed.

RAG can be fast and still lose. A procedure can arrive late and still win. A guardrail can reject after the first token exists in memory, and the user still never sees it.

If you build this

The invariant is the thing to keep in your head: nothing from speculative RAG should survive unless that RAG answer was actually served.

That includes tokens, trace updates, retrieval-cache writes, counters, metrics, and session flags. If the prepared answer is dropped, its pending side effects should be dropped with it.

In code review, I would look for three paths:

• RAG wins: open_and_flush(), await the final response, then commit deferred side effects.
• RAG loses: drop(), cancel the speculative task, and never commit deferred side effects.
• RAG partly flushed and then fails: do not replay a second answer into the same response.

The important line is open_and_flush(). After that runs, the buffered answer has reached the response path, so every fallback decision has to account for what the user may already have seen.