LLM API Cost Optimization: How to Cut Your AI Bill Without Hurting Quality
Short answer: The levers that actually move an LLM bill, roughly in order of effort-to-payoff, are prompt caching (up to a 90-97.5% discount on repeated prefix tokens, depending on the model), batch processing (a flat 50% discount on anything that doesn't need an instant answer), output control (output tokens cost about 5x input tokens on Claude models, so verbose responses are an expensive default), and model routing (sending simple requests to a cheaper model instead of your default one). Semantic caching can add more on top, but its savings are workload-dependent and worth measuring before you build it.
What this means for you
- Turn on prompt caching for your system prompt and any repeated context first — it is the lowest-risk change with the fastest payoff.
- Move anything that does not need a same-second answer to a batch API before paying full price for it.
- Do not route requests to a cheaper model without an evaluation set confirming quality holds — cheap-but-wrong costs more than expensive-and-right.
- Instrument cost per operation before optimizing anything. You cannot cut what you have not measured.
Nobody plans for their AI feature to cost four times what it did at launch, but it happens constantly, usually not because usage grew four times but because nobody turned on the cost controls providers already offer for free or at a flat discount. That is a different question from the one in our AI integration cost guide, which covers what it costs to build a feature. This is about what it costs to run it, every month, for as long as it keeps getting traffic.
Why the bill gets away from you
A few patterns show up repeatedly in production LLM spend, and none of them involve anything exotic.
Output tokens are the expensive ones
On Anthropic's own pricing page, Claude Opus 5.5, Sonnet 5.5, and Haiku 5.5 all price output tokens at roughly 5x their input token price. A verbose response habit, or a system prompt that encourages long explanations by default, taxes every single request at that multiplier, whether the user wanted the extra text or not.
Nobody tracks cost per operation
Most teams watch total monthly API spend, not cost broken down by feature or request type. That tells you the bill went up, not which of your twelve LLM-powered features caused it, so the fix usually starts with better instrumentation, not a smarter model.
Agentic loops multiply quietly
A multi-step agent that retries, re-reads context, or calls a tool and feeds the result back into the model can burn through tokens in ways a single request never would. Each step looks cheap in isolation; the loop is not.
Prompt caching: turn this on first
If your requests repeat a system prompt, a long document, or RAG context across calls, prompt caching is usually the single highest-leverage change available, and it changes nothing about how your application behaves. Per Anthropic's documentation, a cache read costs 10% of the base input price on most Claude models, 5% on Claude Opus 5.5 and Sonnet 5.5, and 2.5% on Claude Fable 5.1 and Mythos 5.1. Writing to the cache costs more upfront: 1.25x the base price for a 5-minute cache, 2x for a 1-hour cache. Because a cache read is so much cheaper than a fresh request, one write followed by one read already costs less than two uncached requests, and every additional read after that is close to free by comparison. The catch is a minimum prompt length to qualify at all, ranging from 512 to 4,096 tokens depending on the model, so caching a short system prompt alone may not clear the bar; pairing it with the long context or document you are already sending usually does.
Batch processing: 50% off anything that can wait
Not every LLM call needs to come back in under a second. Nightly summarization, bulk classification, embedding generation, and large evaluation runs can all tolerate a delay. Anthropic's Message Batches API charges a flat 50% less than the standard synchronous price for exactly this kind of work, with most batches finishing in under an hour. If a meaningful share of your LLM traffic is already asynchronous in practice, just not routed through a batch endpoint, that is a pure cost reduction with no behavior change for the user, who was never waiting on it anyway.
Output control: the token type you are overpaying for
Since output tokens run roughly 5x the price of input tokens on Claude models, the fastest lever most teams skip is simply asking for less output. Set an explicit, deliberate token limit instead of relying on the default. Where a user-facing explanation is not actually required, request structured output (a defined JSON schema) instead of free-flowing prose; our guide on structured output vs. free text covers how to decide when a schema is the right call. A model that would otherwise pad its answer with caveats and restatements, multiplied across every request, is a cost you are choosing by default rather than on purpose.
Model routing: not every request needs your best model
Classification, extraction, short FAQ answers, and other narrow, well-defined tasks often do not need your most capable (and most expensive) model. Routing those requests to a smaller, cheaper model while reserving the frontier model for genuinely hard requests is a legitimate lever, but it is a quality decision wearing a cost-saving label. Ship it only behind an evaluation set that scores the cheaper model against your own real traffic, not a vendor's benchmark; our LLM evaluation guide covers building that evaluation set. A cheaper model that gets 8% more requests wrong costs more in support tickets, rework, and lost trust than the tokens it saved.
Semantic caching: promising, unproven on your traffic
Semantic caching returns a stored answer when a new request means roughly the same thing as one you have already answered, typically measured by embedding similarity against a threshold, rather than requiring an exact text match. Vendor blog posts cite wildly different savings figures for this (anywhere from 30% to 70%, by different sources, on different workloads), which is a sign to treat those specific numbers skeptically. The more rigorous treatment is a 2026 academic paper, Continuous Semantic Caching for Low-Cost LLM Serving, which formalizes the idea over a continuous query space with theoretical guarantees rather than a fixed list of known questions. Before building this yourself, check whether your actual traffic has a meaningful rate of near-duplicate requests; a support chatbot answering the same handful of questions repeatedly is a good candidate, a highly personalized recommendation engine usually is not. If you do build it, tag every cache entry with tenant, user, or date metadata, because the failure mode here is not just wasted savings, it is serving one customer's cached answer inside another customer's different context.
What to avoid
- Optimizing before instrumenting. If you cannot see cost broken down by feature or request type, you are guessing at which lever to pull first.
- Routing to a cheaper model without an eval set. The savings are real, but so is the quality risk, and only one of those shows up on the invoice.
- Caching across tenants or user contexts. This is a correctness bug wearing a cost optimization, and it is far more expensive to clean up than the caching ever saved.
A practical order to do this in
- Instrument cost per operation so you know which feature or request type is actually driving spend.
- Turn on provider-native prompt caching for your system prompt, long context, and RAG documents.
- Set explicit output limits and move to structured output wherever free-form prose is not required.
- Move async-tolerant work to a batch API for the flat discount.
- Add model routing, but only behind an evaluation harness scored on your own traffic.
- Consider semantic caching only after confirming your traffic has a real rate of near-duplicate queries worth deduplicating.
None of this requires switching providers or rearchitecting the product. If you want a second opinion on where your own AI spend is actually going and which of these levers applies to your stack, that is part of what we scope during AI integration work, timeline and approach depend on your current architecture and traffic, and get set during discovery rather than guessed up front.
Frequently asked questions
What is the single most effective way to reduce LLM API costs?
Prompt caching, for most production apps. If your requests repeat a system prompt, long context, or RAG documents across calls, caching lets you pay the full input price once and roughly 10% of it (90% off) on every repeat read for most models, some models discount cache reads by 95-97.5%. It is also the lowest-risk change: you are not swapping models or changing behavior, just skipping redundant processing of tokens you already sent.
How much can prompt caching actually save?
On Anthropic's own pricing, a cache read costs 10% of the base input price on most Claude models, 5% on Claude Opus 5.5 and Sonnet 5.5, and 2.5% on Claude Fable 5.1 and Mythos 5.1. Writing to the cache costs more upfront, 1.25x base price for a 5-minute cache, 2x for a 1-hour cache, so the math only pays off once you reuse it: one reused 5-minute cache already costs less than two uncached requests. Prompts need to clear a minimum length (512 to 4,096 tokens depending on the model) to be eligible at all.
Is it safe to route requests to a cheaper model to save money?
Only if you measure it. Routing simple, well-defined tasks (classification, extraction, short FAQ answers) to a smaller model while reserving your frontier model for genuinely hard requests is a real cost lever, but it is a quality decision disguised as a cost one. Ship it behind an evaluation set that scores both models on your actual traffic first, see our guide on LLM evaluation. A cheaper model that gives wrong answers costs more in support tickets and rework than the tokens it saved.
What is semantic caching, and is it worth implementing?
Semantic caching returns a stored answer when a new request means roughly the same thing as a past one, not just when the text matches exactly, typically by comparing embedding similarity against a threshold. A 2026 academic paper formalizes this over a continuous query space with performance guarantees, which is a more rigorous treatment than the vendor blog posts that cite conflicting savings ranges for it. It is worth building only after you have confirmed your traffic actually has a meaningful rate of near-duplicate queries, and only if you tag cache entries by tenant, user, or date so you never serve one customer's cached answer to another's different context.
How long does it take to implement LLM cost optimization for an existing product?
It depends on how your app is architected today, how much request volume and instrumentation already exists, and which levers apply. Turning on provider-native prompt caching and tightening output limits is often a small, fast change; adding model routing behind a proper evaluation harness is a larger project. Scope and timeline get set during discovery against your actual codebase and traffic, not estimated in the abstract.
Sources
- Anthropic — Prompt caching documentation — Cache read discounts (90% on most models, 95% on Opus 5.5/Sonnet 5.5, 97.5% on Fable 5.1/Mythos 5.1), cache write multipliers (1.25x for 5-minute, 2x for 1-hour caches), and minimum cacheable prompt lengths (512-4,096 tokens depending on model).
- Anthropic — Batch processing documentation — The Message Batches API's flat 50% cost reduction for asynchronous requests, with most batches completing in under an hour.
- Anthropic — Claude pricing — Base input and output token prices confirming output tokens cost roughly 5x input tokens across Claude Opus 5.5, Sonnet 5.5, and Haiku 5.5.
- Continuous Semantic Caching for Low-Cost LLM Serving (arXiv, 2026) — Academic treatment of semantic caching over a continuous query space with theoretical performance guarantees, used here as a more rigorous counterpoint to conflicting vendor-reported savings percentages.
Need help putting this into practice?
Tech Programmer builds and ships this work for startups and enterprises. Tell us what you are trying to do and we will tell you what it takes.
Related reading
- Vector Database vs. pgvector: Do You Actually Need a Dedicated One?Most teams adding semantic search reach for a dedicated vector database before checking whether the Postgres instance they already run can do the job. pgvector is a Postgres extension, not a new service: it adds a vector column type and an HNSW index to a database you're already operating. It stays comfortable for most teams under roughly 10 million vectors. A dedicated store like Qdrant, Weaviate, or Pinecone earns its place once your corpus genuinely outgrows that, hybrid search or multi-tenant isolation is core to the product rather than a nice-to-have, or you want a vendor to own the ops instead of you.
- Structured Output vs Free Text: When Should an LLM Return JSON?Free text is the right model output when a person reads the answer. A JSON schema is the right output when your code must parse fields and pass them to a UI, a database, or another API. JSON mode only promises that the text parses as JSON. Structured outputs go further and constrain the model to a schema. Neither one makes the values true. Check refusals and truncated responses, then test the fields the way you test any other AI feature.
- MCP vs Custom API Integration: When Should an AI Agent Use Each?The Model Context Protocol (MCP) is an open standard so AI applications can discover and call tools, data, and prompts through a shared server. A custom API or in-process tool call is code your product owns: smaller tool lists, per-user auth, and tighter control. Prefer custom tools when one product agent is the only caller. Prefer MCP when several clients — Claude, ChatGPT, an IDE, an internal copilot — must reuse the same capabilities. A practical path is to ship the domain API first, then add a thin MCP server once a second real client appears.