Reducing AI application costs without losing sight of quality
A practical guide to model routing, prompt caching and evaluation, prompted by October 2026 API updates.
Start with the complete cost
AI application cost is more than a model's price per token. Include retrieval, external tools, repeated requests, retries, infrastructure and the time people spend correcting poor output. Compare costs per successfully completed task, alongside the quality threshold that defines success.
In October 2026, OpenAI added a 12-hour prompt-caching option to its API. Anthropic also announced pricing changes alongside Claude Haiku 5.5. These options may help certain workloads, but savings depend on prompt reuse, pricing terms and request patterns.
Test practical changes one at a time
Measure a baseline on a representative evaluation set before changing anything. Try shorter context, fewer retrieved passages, prompt caching, a smaller model for simpler work or batch processing when immediate responses are unnecessary.
A cache only helps when requests reuse the relevant input and the extra write cost is justified. Check provider documentation for retention, pricing and data requirements before enabling it.
- Track quality, latency and cost for the same task set.
- Check whether fewer retrieved passages omit required evidence.
- Escalate uncertain or high-impact work to a person or stronger model.
- Version prompts, model choices and evaluation results.
Make a decision you can reverse
Roll changes out gradually with monitoring and an easy rollback. Re-evaluate when source data, model versions or user behaviour changes. Avoid promising a fixed percentage reduction without measuring the actual workload.
Have a real workflow in mind?
Explore an AI discovery engagement, a focused RAG pilot or fractional engineering support with BuildrLab.