All insights
Token spend behaves like infrastructure, not like a SaaS seat. Caching, routing and the three decisions that move the number.
Most teams budget an AI feature the way they budget a SaaS tool: a monthly number, roughly flat. Inference does not behave like that. It scales with usage, with prompt length, with how chatty the model is and with how many retries the agent made at 2am. It behaves like infrastructure.
The teams that get caught out are the ones who forecast it as a seat. Three decisions move the number more than any other.
Cache what does not change
System prompts, retrieved documents and tool schemas are resent on every call. Prompt caching turns that repeated prefix into a fraction of the price, and structuring prompts so the stable part comes first is a one-day change with a double-digit percentage effect.
Route by difficulty, not by default
Most requests do not need the most capable model. Classification, extraction and short rewrites run well on small models at a tenth of the cost. A router that sends the easy majority to a small model and escalates the rest is the single largest lever we have, and it usually improves latency as a side effect.
Put a meter where the spend is
Cost is invisible until it is attributed. Tag every call with the feature, the customer and the run, and put the result on the same dashboard as the product metrics. Once a product manager can see that one feature costs four times the others, the conversation about whether it earns that changes on its own.
Forecast like capacity, not like a subscription
Model the bill as tokens per active user per day, multiplied out, with a p95 and a ceiling. Review it monthly the way you would review cloud spend. Treated that way, inference is a manageable line item. Treated as a seat, it is a surprise.
