TestForge Blog
← All Posts
#ai #agent #streaming #sse #websocket

AI Agent Streaming Design — Should You Use SSE or WebSocket?

In AI Agent services, user trust depends not only on the final answer but on how progress is shown during execution. This post compares SSE and WebSocket for token streaming, step status, tool execution events, and intermediate results, with practical guidance for real product teams.

TestForge Team ·

Why streaming matters

In AI Agent products, users care less about raw latency than whether the system feels alive while it works.

A typical Agent flow includes:

  • question interpretation
  • retrieval
  • tool execution
  • intermediate aggregation
  • final answer generation

If you wait until all of that is complete, the service feels slow and opaque.

Separate the event types first

A practical streaming design usually needs more than plain text tokens.

1. token stream

  • partial text generated by the model

2. status events

  • retrieving
  • running_tool
  • summarizing
  • completed

3. structured intermediate results

  • retrieved documents
  • computed values
  • chart data
  • warnings

Streaming in an AI Agent is really an event-model design problem.

When SSE is a good fit

SSE is a server-to-client streaming model over HTTP.

Advantages:

  • simple to implement
  • works well with standard web infrastructure
  • ideal for token streaming
  • easy browser integration

Good fits for SSE:

  • answer token streaming
  • progress updates
  • retrieved citations
  • one request, one response flow

Example:

event: status
data: {"step":"retrieving"}

event: citation
data: {"title":"10-Q Filing","source":"sec"}

event: token
data: {"text":"NVIDIA "}

event: token
data: {"text":"shows rising concentration risk "}

event: done
data: {"latency_ms":8420}

When WebSocket becomes necessary

WebSocket supports bidirectional messaging, so it fits more interactive systems.

Advantages:

  • server and client can keep exchanging events
  • useful for long-lived sessions
  • better for collaborative or highly interactive workspaces

Good fits for WebSocket:

  • user-controlled cancellation or resume
  • tool execution that asks for more input mid-run
  • collaborative workspaces
  • heavy two-way synchronization

For many Q&A-style Agents, though, WebSocket is often more than you need.

The practical decision rule

Choose based on complexity versus value.

Use SSE first when:

  • one user request maps to one streamed answer
  • only server-to-client events are needed
  • you want a fast MVP
  • you want simpler infrastructure behavior

Use WebSocket when:

  • bidirectional interaction is essential
  • users need to control an in-flight run
  • sessions are long-lived and heavily stateful
  • your product behaves more like a real-time workspace than a chat interface

In many real teams, the best path is start with SSE, move to WebSocket only if the interaction model truly demands it.

Design the backend event model clearly

The execution engine may produce many internal events, but the UI should only receive a clean subset.

Recommended event types:

  • status
  • token
  • artifact
  • citation
  • warning
  • error
  • done

That separation makes the frontend much easier to build.

For example:

  • status drives the progress indicator
  • token updates the answer body
  • artifact updates a side panel
  • citation builds a source list
  • warning renders trust signals

Tool-using Agents need better streaming UX

RAG and tool-using Agents often spend time across multiple steps:

  • retrieval for 2 seconds
  • API calls for 3 seconds
  • final generation for 5 seconds

If total response time is 10 seconds, the UI should show:

  • searching
  • analyzing
  • generating

rather than a blank wait.

Cancellation and timeout should be part of the design

Streaming without cancellation is incomplete.

You usually want:

  • user-driven cancel support
  • best-effort backend cancellation
  • timeout handling
  • partial-result fallback when possible

Even with SSE, disconnect handling and server-side cancellation matter.

Common implementation mistakes

  • mixing tokens and structured events into one state bucket
  • relying on connection close instead of an explicit done event
  • duplicating output after reconnect
  • unstable auto-scroll in long answers
  • showing citations out of sync with the answer

It is usually better to separate:

  • stream state
  • message state
  • artifact state

Early MVP

  • frontend: Next.js
  • backend: FastAPI
  • transport: SSE
  • events: status, token, citation, done

Growth phase

  • add tool execution logs
  • support cancellation
  • persist sessions
  • improve retry behavior

Advanced phase

  • background job queues
  • WebSocket control channels
  • collaborative real-time sessions

Closing thoughts

Streaming in AI Agents is not a cosmetic effect. It is part of trust, perceived speed, and product clarity.

In practice:

  • SSE is enough for many Agent products
  • WebSocket is best when true bidirectional interaction is required
  • the stream should include status, evidence, and tool output, not just tokens

A strong Agent experience is not only about a good final answer. It is also about making the path to that answer understandable while it is happening.