Almost every AI feature I've built for a client has worked great in the demo and then raised a quiet, uncomfortable question a few weeks after launch: why is the API bill this high? LLM cost optimization is the part of AI app development that nobody puts in the pitch deck, but it's often what decides whether an AI feature survives its first quarter. In this post I'll walk through how I approach LLM cost optimization in Node.js backends: measuring token usage, trimming prompts, routing requests to cheaper models, caching, and enforcing per-user budgets, with real code you can adapt.

I've spent 8+ years building Android, React, Next.js and Node.js products, and most of my freelance work on Upwork has been for US clients who care a lot about unit economics. The patterns below come from that work. None of them are exotic. They're just the things that are easy to skip when you're racing to ship.

Why LLM Bills Surprise Clients After Launch

Traditional backend costs scale with infrastructure: servers, databases, bandwidth. They grow slowly and predictably. LLM costs scale with tokens, and tokens scale with user behavior, prompt design and a dozen small engineering decisions that nobody reviews.

Here's what I typically see when I audit an AI feature that's costing more than expected:

  • Bloated system prompts. A prompt that started at a few lines grows into pages of instructions, examples and edge cases, and every single request pays for all of it.
  • Full chat history on every turn. The entire conversation gets resent each time, so long sessions get progressively more expensive.
  • One model for everything. The most capable (and most expensive) model handles tasks a small model could do perfectly well, like classification or extracting a date.
  • No output limits. The model is free to write essays when the UI only shows two sentences.
  • No per-user guardrails. A handful of power users, or one script hammering an endpoint, can dominate the bill.

None of these are bugs in the traditional sense. Tests pass, the feature works, users are happy. That's exactly why they slip through.

Measure Token Usage Before You Optimize Anything

The first rule I follow: don't guess where the money goes. Every LLM call should be logged with enough context to answer three questions: which feature made the call, which user triggered it, and how many input and output tokens it used.

Most provider APIs return a usage object with token counts on every response. Capture it. I store a row per call in a simple llm_usage table with columns for user ID, feature name, model, input tokens, output tokens, whether it was a cache hit, and a timestamp. Once that data exists, a single grouped query tells you which features are expensive and whether input or output tokens dominate.

Tip

Tag every call with a feature string like "summarize-note" or "chat-reply". Without it, your usage data tells you that you're spending money, but not where to cut.

In my experience, the results are usually lopsided. One or two features account for most of the spend, and the fix for each is different. A summarization feature might be input-heavy (huge documents going in), while a chat assistant might be output-heavy (verbose replies). You can't pick the right technique until you know which problem you have.

LLM Cost Optimization Techniques That Actually Work

Once you know where tokens go, these are the levers I reach for, roughly in order of effort versus payoff.

1. Trim the system prompt

Go through the system prompt line by line and ask whether each instruction actually changes the output. Duplicate rules, long few-shot examples that could be one short example, and instructions for edge cases that never happen all cost money on every request. Rewriting a prompt to be shorter and clearer often improves quality too.

2. Cap output tokens

Set an explicit max output token limit per feature, and tell the model the expected length in the prompt. If the UI shows a short reply, ask for a short reply. Output tokens are typically priced higher than input tokens, so this one matters more than people expect.

3. Route by task, not by habit

Not every request needs your biggest model. Classification, tagging, extraction, short rewrites and intent detection usually work fine on a smaller, cheaper model. Reserve the large model for genuinely hard reasoning or long-form generation. A simple tier system in your wrapper makes this a one-line change per feature.

4. Summarize or window the conversation history

For chat features, don't resend every message forever. Keep the last few turns verbatim and replace older turns with a running summary. That keeps the context useful while stopping the cost of each turn from climbing indefinitely.

5. Cache what repeats

If the same input produces an acceptable answer every time (product descriptions, FAQ answers, document summaries, translations of static content), cache it. Exact-match caching with a hash of the model, prompt and input is simple and safe. Semantic caching is more powerful but needs careful thresholds so you don't serve a wrong answer to a slightly different question.

6. Move non-urgent work to batches

Some providers offer discounted batch processing for jobs that don't need an instant response, like nightly tagging or bulk summarization. If a user isn't waiting on the result, it probably doesn't need to run in real time.

Watch out

Every optimization can degrade quality. Before switching a feature to a smaller model or a shorter prompt, run a handful of real inputs through both versions and compare the outputs side by side. Cheap and wrong is worse than expensive and right.

Caching, Routing and Per-User Budgets in Node.js

Here's a trimmed version of the wrapper I tend to put in front of every LLM call in a Node.js backend. It handles model tiers, exact-match caching in Redis, a daily per-user token budget, output caps and usage logging. Feature code never calls the provider SDK directly; it calls ask().

// llm.ts
import OpenAI from "openai";
import crypto from "node:crypto";
import { redis } from "./redis";
import { db } from "./db";

const client = new OpenAI();

// Keep model names in env so you can swap them without a deploy
const MODELS = {
  small: process.env.LLM_SMALL_MODEL!,
  large: process.env.LLM_LARGE_MODEL!,
};

type Tier = keyof typeof MODELS;

interface AskOptions {
  userId: string;
  feature: string;
  system: string;
  input: string;
  tier?: Tier;
  maxOutputTokens?: number;
  cacheTtlSeconds?: number;
}

const DAILY_TOKEN_BUDGET = Number(process.env.DAILY_TOKEN_BUDGET ?? 50000);

export class BudgetExceededError extends Error {
  constructor(public userId: string) {
    super(`Daily AI budget reached for user ${userId}`);
  }
}

export async function ask(opts: AskOptions): Promise<string> {
  const {
    userId,
    feature,
    system,
    input,
    tier = "small",
    maxOutputTokens = 400,
    cacheTtlSeconds = 0,
  } = opts;
  const model = MODELS[tier];

  // 1. Exact-match cache
  const cacheKey =
    "llm:" +
    crypto.createHash("sha256").update(`${model}|${system}|${input}`).digest("hex");

  if (cacheTtlSeconds > 0) {
    const cached = await redis.get(cacheKey);
    if (cached) {
      await logUsage({ userId, feature, model, inputTokens: 0, outputTokens: 0, cached: true });
      return cached;
    }
  }

  // 2. Per-user daily budget
  const day = new Date().toISOString().slice(0, 10);
  const budgetKey = `budget:${userId}:${day}`;
  const used = Number(await redis.get(budgetKey)) || 0;
  if (used >= DAILY_TOKEN_BUDGET) {
    throw new BudgetExceededError(userId);
  }

  // 3. The actual call, with an output cap
  const res = await client.chat.completions.create({
    model,
    max_tokens: maxOutputTokens,
    messages: [
      { role: "system", content: system },
      { role: "user", content: input },
    ],
  });

  const text = res.choices[0]?.message?.content ?? "";
  const inputTokens = res.usage?.prompt_tokens ?? 0;
  const outputTokens = res.usage?.completion_tokens ?? 0;

  // 4. Update budget and cache
  await redis
    .multi()
    .incrby(budgetKey, inputTokens + outputTokens)
    .expire(budgetKey, 60 * 60 * 26)
    .exec();

  if (cacheTtlSeconds > 0 && text) {
    await redis.set(cacheKey, text, "EX", cacheTtlSeconds);
  }

  // 5. Log for cost analysis
  await logUsage({ userId, feature, model, inputTokens, outputTokens, cached: false });
  return text;
}

async function logUsage(row: {
  userId: string;
  feature: string;
  model: string;
  inputTokens: number;
  outputTokens: number;
  cached: boolean;
}) {
  await db.llmUsage.create({ data: { ...row, createdAt: new Date() } });
}

Using it from a feature looks like this. A tagging endpoint runs on the small model with a tight output cap and a long cache, while a detailed report uses the large model:

// Cheap, repeatable task: small model, short output, cached for a day
const tags = await ask({
  userId,
  feature: "tag-note",
  system: "Return 3-5 comma-separated topic tags. No other text.",
  input: note.body,
  tier: "small",
  maxOutputTokens: 30,
  cacheTtlSeconds: 60 * 60 * 24,
});

// Hard reasoning task: large model, larger output, no cache
const report = await ask({
  userId,
  feature: "weekly-report",
  system: reportPrompt,
  input: JSON.stringify(weekData),
  tier: "large",
  maxOutputTokens: 800,
});

A few notes from using this pattern in real projects:

  • The budget check is approximate. Concurrent requests can slightly overshoot the limit, since the check and the increment aren't atomic with the API call. For most products that's fine; the goal is to stop runaway usage, not to bill to the exact token.
  • Handle BudgetExceededError gracefully. Show the user a friendly message or degrade to a non-AI fallback. A raw 500 error makes the guardrail feel like a bug.
  • Include the model in the cache key. When you swap models, old cached answers from the previous model shouldn't leak through unnoticed.
  • Don't cache personalized or sensitive outputs unless the key includes the user ID and you're comfortable storing that content.

LLM Cost Optimization as a Freelance Selling Point

This is where the technical work turns into a business advantage. Plenty of developers can call an LLM API now. Far fewer can explain to a founder what that feature will cost per user per month and how they'll keep it under control. In my experience, clients notice the difference immediately.

Here's how I bring LLM cost optimization into client conversations:

  1. Raise it in the first call. When a client wants an AI feature, I ask how they plan to monetize it and whether usage limits fit the product. That question alone signals that I'm thinking about their business, not just the code.
  2. Estimate before building. Using a rough average of input and output tokens per request and expected usage, I give a ballpark range for monthly API cost. I'm clear it's an estimate, not a promise, because real user behavior always differs from assumptions.
  3. Ship the usage dashboard with the feature. A simple admin view of spend by feature and top users turns a scary invoice into something the client can reason about.
  4. Offer optimization as follow-up work. After launch, real usage data reveals what to trim. That's a natural, genuinely useful reason to keep working together.
Clients don't just want an AI feature. They want an AI feature they can afford to keep running.

In the Build With Aamir community, I often tell developers that the fastest way to stand out to US clients is to think one level above the ticket. For AI work, cost is that level. It won't guarantee you win a contract, but it puts you in a different conversation than developers who only talk about which model is newest.

Key Takeaways

  • Measure first: log tokens per call with user and feature tags so you know exactly which parts of your app drive the bill.
  • Pull the cheap levers early: trim system prompts, cap output tokens and route simple tasks to smaller models before reaching for complex solutions.
  • Centralize LLM calls behind one Node.js wrapper that handles caching, budgets and logging, so every feature gets guardrails by default.
  • Protect quality: compare outputs on real inputs before and after every optimization, because cheap and wrong is not a win.
  • Use cost awareness as a differentiator with clients by estimating spend, shipping a usage dashboard and offering post-launch optimization.