advanced 9 min read

System Design: AI Chat & LLM Token Streaming UI

System design architecture for modern AI interfaces (ChatGPT, Claude, Cursor) handling Server-Sent Events (SSE) token streaming, markdown parsing without layout jumps, optimistic message dispatch, cancel generation (AbortController), and tool call invocation UI.

Why it matters in interviews

AI-enabled web experiences are the #1 trending hiring requirement across ConveGenius, OpenAI, Microsoft, and Google. Interviewers assess your ability to stream chunks over HTTP, handle partial JSON tool executions, and maintain smooth auto-scrolling.

Visual & Interactive Explanation

AI Token Streaming & Tool Execution Lifecycle

Sequence & Data Exchange Flow

React Chat UI

Next.js Route

LLM API

Tool Sandbox

React Chat UINext.js Route

request

1. POST /api/chat (Prompt + History + AbortSignal)

Next.js RouteLLM API

request

2. Proxies prompt with streaming=true

LLM APINext.js Route

response

3. Streams tokens: 'The', ' solution', ' is...'

Next.js RouteReact Chat UI

response

4. SSE chunks -> Decoded in ReadableStream -> Append to buffer

React Chat UITool Sandbox

render

5. LLM emits tool_call: 'render_chart' -> Live Artifact renders!

Code Examples & Implementation

Streaming Fetch Reader with AbortController

import { useState, useRef } from 'react';

export function ChatStreamer() {
  const [messages, setMessages] = useState<{ role: 'user' | 'assistant'; content: string }[]>([]);
  const [isStreaming, setIsStreaming] = useState(false);
  const abortCtrlRef = useRef<AbortController | null>(null);

  const sendMessage = async (userPrompt: string) => {
    // 1. Optimistic user message
    setMessages(prev => [...prev, { role: 'user', content: userPrompt }, { role: 'assistant', content: '' }]);
    setIsStreaming(true);

    abortCtrlRef.current = new AbortController();

    try {
      const response = await fetch('/api/ai/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ prompt: userPrompt }),
        signal: abortCtrlRef.current.signal,
      });

      if (!response.body) throw new Error('No stream response body');

      const reader = response.body.getReader();
      const decoder = new TextDecoder('utf-8');
      let assistantText = '';

      while (true) {
        const { value, done } = await reader.read();
        if (done) break;

        const chunk = decoder.decode(value, { stream: true });
        assistantText += chunk;

        // Update assistant message streaming content
        setMessages(prev => {
          const updated = [...prev];
          updated[updated.length - 1] = { role: 'assistant', content: assistantText };
          return updated;
        });
      }
    } catch (err: any) {
      if (err.name !== 'AbortError') {
        console.error('Streaming failed', err);
      }
    } finally {
      setIsStreaming(false);
      abortCtrlRef.current = null;
    }
  };

  const stopGenerating = () => {
    abortCtrlRef.current?.abort();
  };

  return (
    <div>
      {/* Messages list with incremental markdown */}
      <button onClick={() => sendMessage('Explain React Fiber')}>Send</button>
      {isStreaming && <button onClick={stopGenerating}>Stop Generating</button>}
    </div>
  );
}

Interview-Ready Answers

An AI streaming UI uses `fetch()` with `ReadableStream` or SSE. As chunks arrive, a text buffer appends tokens, triggering throttled React state updates. Markdown renders incrementally, and an auto-scroll anchor sticks to bottom only if the user hasn't manually scrolled up.

Official Documentation & Specifications

Next.jsVercel: AI SDK Streaming and Tool Calling Architecture

MDNMDN: Streams API and ReadableStream

Follow-up Questions & Deep Dives

Common Mistakes & Anti-Patterns

Re-parsing the entire markdown document from character 0 on every single character token instead of chunk throttling
Not providing an AbortController 'Stop' button (wasting expensive LLM API credits when user wants to cancel)
Forcing aggressive auto-scroll that forcibly yanks the user down when they are trying to read earlier messages

Real-World Architectural Scenario

You are building an AI IDE chat assistant that outputs both explanation text and live code artifact widgets. How do you structure the streaming protocol?

Rate Your Readiness

Rate how comfortably you can explain this in an interview to update your global readiness gauge:

Rate your confidence: