The $28,000 Wake-Up Call
Month one of our AI tutoring feature in production: 340,000 API calls. Invoice: $28,000. The client’s annual AI budget was $36,000. We had a serious problem to solve quickly.
Twelve months later, the same feature handles 1 million daily requests at $0.00019 per query — a 93% cost reduction while serving 3× more users. This is everything we changed.
Layer 1: Semantic Caching (73% Cache Hit Rate)
We embed every incoming question using text-embedding-3-small and query Pinecone for semantically similar previous questions. Any match above 0.92 cosine similarity returns a cached answer instantly — no LLM call, no latency, no cost. Students in an edtech context ask remarkably similar questions. Within six weeks, semantic caching was handling 73% of all queries.
Layer 2: Tiered Model Routing
A fine-tuned DistilBERT classifier assigns every question a complexity score from 1 to 5, running on CPU in under 20ms. Scores 1–2 route to GPT-3.5-turbo. Scores 4–5 route to GPT-4o. The remaining 27% of uncached queries cost 60% less in model fees alone due to this routing.
Layer 3: Prompt Compression
40% of our average prompt was boilerplate context repeated verbatim on every call. LLMLingua compressed the context window from 1,800 to 1,080 tokens on average without measurable quality loss on our evaluation benchmark.
The Numbers
Month one: $28,000 for 340K calls. Month twelve: $5,800 for 30M calls. P99 latency: 380ms. Zero user-facing incidents through two 8× traffic spikes.