Streaming LLM responses is the one change that makes an AI feature feel fast, even when the model behind it isn't. Instead of making users stare at a spinner for several seconds while the full answer is generated, you push tokens to the screen as they arrive. In this guide I'll walk through how I build streaming LLM responses in Next.js, both from a route handler and from a separate Node.js backend, plus the production problems that tutorials usually skip.

I've written a lot on this site about on-device AI and LLM integration in Android apps. This post is the web side of the same problem. In my freelance work on Upwork, a large share of clients are US startups, and when they ask for an AI feature, the brief almost always includes some version of "make it feel like ChatGPT." What they mean, most of the time, is streaming.

Why Streaming LLM Responses Matter in AI App Development

An LLM generates text token by token. A non-streaming request waits for the very last token before returning anything. For a long answer, that wait can easily stretch into many seconds. Streaming doesn't make the total generation faster, but it changes the number users actually perceive: time to first token.

From a product point of view, streaming gives you three things:

  • Perceived speed. Users start reading almost immediately, so the wait feels shorter.
  • Early cancellation. If the answer is heading in the wrong direction, users can hit stop. If you wire it correctly, you also stop paying for tokens nobody will read.
  • Trust. A visible, progressing answer feels alive. A frozen spinner feels broken, and users refresh, which often means a duplicate request and duplicate cost.

The trade-off is complexity. You now deal with partial data, connections that drop midway, proxies that buffer your output, and the question of when to save the final message. None of it is hard, but all of it needs deliberate handling.

Route Handler or Separate Node.js Backend?

There are two setups I use, depending on the project.

Option 1: Next.js route handler

If the app is already a Next.js app and the AI feature is mostly "send messages, get an answer," a route handler in the App Router is the simplest path. Route handlers return standard Response objects, and a Response can wrap a ReadableStream. No extra server, no CORS, and auth lives in the same place as the rest of the app.

Option 2: Separate Node.js (Express) backend

If there's already a Node.js API, or the AI logic involves tool calls, database writes, queues or a mobile client that also needs the stream, I put it in the backend and use Server-Sent Events (SSE). SSE is plain HTTP, works with any client that can read a response body, and is much simpler to operate than WebSockets for one-way server-to-client text.

My rule of thumb

One web client and simple prompts: route handler. Multiple clients (web plus Android), long-running jobs or an existing API: dedicated Node.js backend with SSE. Pick the one that matches where the rest of the business logic already lives.

Streaming LLM Responses From the Server

Here's a Next.js route handler using the official OpenAI Node SDK. The same pattern works with other providers whose SDKs expose an async iterable of chunks.

// app/api/chat/route.ts
import OpenAI from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export const runtime = "nodejs";
export const dynamic = "force-dynamic";

const SYSTEM_PROMPT = "You are a concise assistant for our product.";

export async function POST(req: Request) {
  const { history } = await req.json();

  const completion = await openai.chat.completions.create(
    {
      model: "gpt-4o-mini",
      messages: [{ role: "system", content: SYSTEM_PROMPT }, ...history],
      stream: true,
      stream_options: { include_usage: true },
    },
    { signal: req.signal } // abort upstream if the client disconnects
  );

  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of completion) {
          const text = chunk.choices[0]?.delta?.content;
          if (text) controller.enqueue(encoder.encode(text));
          if (chunk.usage) {
            console.log("total tokens:", chunk.usage.total_tokens);
          }
        }
        controller.close();
      } catch (err) {
        controller.error(err);
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
    },
  });
}

A few details worth calling out:

  • The system prompt is built on the server. Never accept a system prompt from the browser. Anyone can open dev tools and rewrite it.
  • req.signal is passed to the SDK. When the user closes the tab or hits stop, the upstream request is aborted too, instead of quietly generating tokens you'll still be billed for.
  • include_usage sends token counts in the final chunk, which is where I log usage per user.

For an Express backend, I switch to SSE so each event carries structured JSON. That makes it easy to add done and error events later without inventing a custom protocol.

// server/routes/chat.js
app.post("/api/chat", requireAuth, async (req, res) => {
  const abort = new AbortController();
  res.on("close", () => abort.abort());

  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.setHeader("X-Accel-Buffering", "no");
  res.flushHeaders();

  const send = (event, data) =>
    res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`);

  let fullText = "";

  try {
    const completion = await openai.chat.completions.create(
      { model: "gpt-4o-mini", messages: buildMessages(req), stream: true },
      { signal: abort.signal }
    );

    for await (const chunk of completion) {
      const text = chunk.choices[0]?.delta?.content;
      if (text) {
        fullText += text;
        send("token", { text });
      }
    }

    await saveMessage(req.user.id, fullText); // persist once, at the end
    send("done", {});
  } catch (err) {
    if (!abort.signal.aborted) {
      send("error", { message: "Generation failed. Please try again." });
    }
  } finally {
    res.end();
  }
});

Notice that I accumulate fullText and save it once the stream finishes. Writing to the database on every token is a common mistake. It hammers the database and leaves half-written messages behind when a connection drops.

Consuming the Stream on the Client

You don't need a library to read a stream in the browser. fetch gives you res.body, which is a ReadableStream. Here's a small React hook for the route handler version above, including a stop button via AbortController.

"use client";
import { useRef, useState } from "react";

type Msg = { role: "user" | "assistant"; content: string };

export function useChatStream() {
  const [output, setOutput] = useState("");
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);
  const abortRef = useRef<AbortController | null>(null);

  async function send(history: Msg[]) {
    abortRef.current = new AbortController();
    setOutput("");
    setError(null);
    setLoading(true);

    try {
      const res = await fetch("/api/chat", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ history }),
        signal: abortRef.current.signal,
      });
      if (!res.ok || !res.body) throw new Error("Request failed");

      const reader = res.body.getReader();
      const decoder = new TextDecoder();

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        const text = decoder.decode(value, { stream: true });
        setOutput((prev) => prev + text);
      }
    } catch (err) {
      if ((err as Error).name !== "AbortError") {
        setError("Something went wrong. Please try again.");
      }
    } finally {
      setLoading(false);
    }
  }

  const stop = () => abortRef.current?.abort();

  return { output, loading, error, send, stop };
}

Two small things make a big difference here. First, decode(value, { stream: true }) handles multi-byte characters that get split across chunks. Without it, emoji and non-English text occasionally render as garbage. Second, an aborted request is not an error from the user's perspective, so I don't show an error message for it.

If you're consuming the SSE version, you can parse event: and data: lines yourself from the same reader loop. I avoid the browser's built-in EventSource for chat because it only supports GET requests and can't send a JSON body or custom auth headers.

Rendering tip

If you render Markdown, re-parsing the whole message on every token can get janky with long answers. Throttle UI updates slightly, or render plain text while streaming and switch to full Markdown once the stream completes.

Production Gotchas: Buffering, Cancellation and Cost

Streaming that works perfectly on localhost often arrives in one big lump once deployed. Here's my checklist when that happens.

1. Something is buffering your response

Reverse proxies like Nginx buffer responses by default, and compression middleware can hold bytes until it has enough to compress. The X-Accel-Buffering: no header handles Nginx. For Express, make sure the compression middleware skips your streaming routes, or flush explicitly. Some CDNs and hosting layers also need streaming enabled in their config.

2. Serverless timeouts

Serverless functions have maximum execution times. Long generations, especially with tool calls or retrieval steps in front of them, can hit that limit and cut the stream mid-sentence. Check your platform's limits before you promise a client long-form outputs. If the work is genuinely long-running, move it to a backend process or a background job and stream progress from there.

3. Cancellation must reach the provider

Stopping the stream in the UI is easy. Stopping the upstream LLM call is what saves money. Always pass an abort signal to the SDK and tie it to the client connection, as in both server examples above.

4. Rate limits and abuse

A public streaming endpoint is an open tap on your API bill. Require auth, rate-limit per user, cap max_tokens, and limit how much history the client can send. I trim conversation history on the server rather than trusting whatever array the browser posts.

Don't skip this

An unauthenticated streaming endpoint with no rate limiting is one of the fastest ways to turn an AI demo into an unexpected invoice. Add auth and limits before the feature ships, not after.

5. Handle mid-stream failures honestly

Once you've sent a 200 status and started streaming, you can't change the status code. Errors have to travel inside the stream, which is why the SSE version sends an explicit error event. In the UI, keep whatever partial text arrived, show a clear message, and offer a retry.

A note for freelancers

If you're freelancing in AI app development, this is the kind of detail that separates a demo from a deliverable. When I scope LLM integration work for clients, I list streaming, cancellation, usage logging and rate limiting as explicit line items. Clients rarely ask for them by name, but they notice immediately when they're missing. Explaining these trade-offs up front also helps set realistic expectations about hosting limits and API costs before anyone is surprised by them.

Key Takeaways

  • Streaming LLM responses improves time to first token and perceived speed. It doesn't make total generation faster, but users experience it as faster.
  • Use a Next.js route handler for simple, web-only features. Use a Node.js backend with SSE when multiple clients or heavier business logic are involved.
  • Always propagate cancellation to the LLM provider with an abort signal, and save the final message once rather than per token.
  • Most deployment bugs are buffering issues: check proxies, compression middleware and serverless timeouts.
  • Protect streaming endpoints with auth, rate limits and token caps before shipping, especially for client projects.