LLM Cost Optimisation: How We Reduced GPT-4 Costs by 65%

When Genstori came to us, they were burning $41,800 per month on GPT-4 API calls. Their AI content platform was processing roughly 50,000 requests daily — and the costs were growing faster than their revenue.

Six weeks later, that bill was $14,620. Same platform. Same request volume. Same output quality. 65% lower cost.

Here's exactly what we did — every strategy, every decision, and the code that made it happen.

The Problem: Why LLM Costs Spiral

Before diving into solutions, it's worth understanding why LLM costs get out of control so quickly. Most teams build their first AI feature the obvious way: user sends query → call GPT-4 API → return response. It works. It's fast to build. And it's expensive.

The three biggest cost drivers we see in production AI systems:

• Repeated queries — The same (or near-identical) questions asked thousands of times per day, each generating a fresh, expensive API call
• Wrong model for the task — Using GPT-4o (expensive) for tasks that GPT-4o Mini (10× cheaper) handles equally well
• Bloated prompts — System prompts and context windows that are 3× longer than they need to be

Strategy 1: Redis Semantic Caching

This was our biggest single win — a 28% cost reduction on its own. The insight: a large percentage of LLM queries in any production system are either identical or semantically equivalent. "What's the refund policy?" and "How do I get a refund?" should return the same answer.

We implemented two layers of caching:

Layer 1 — Exact Match Cache (Redis)

For identical queries, we cache the response in Redis with a TTL based on content freshness requirements. Exact matches return instantly at zero LLM cost.

import redis
import hashlib
import json

class LLMCache:
    def __init__(self):
        self.redis = redis.Redis(host='localhost', port=6379)
        self.ttl = 3600  # 1 hour default

    def get_cache_key(self, prompt: str, model: str) -> str:
        content = f"{model}:{prompt}"
        return hashlib.sha256(content.encode()).hexdigest()

    def get(self, prompt: str, model: str):
        key = self.get_cache_key(prompt, model)
        cached = self.redis.get(key)
        if cached:
            return json.loads(cached)
        return None

    def set(self, prompt: str, model: str, response: dict):
        key = self.get_cache_key(prompt, model)
        self.redis.setex(key, self.ttl, json.dumps(response))

Layer 2 — Semantic Cache (Embeddings)

For similar-but-not-identical queries, we use embedding similarity. If a new query is >92% similar to a cached query, we return the cached response. This alone eliminated 40% of our redundant API calls.

Strategy 2: Intelligent Model Routing

Not every query needs GPT-4o. Most don't. The problem is that teams default to the most powerful model because it's the safest choice — not the most cost-effective one.

We built a lightweight classification layer that routes queries to the right model:

Result: 80% of queries now run on GPT-4o Mini (17× cheaper than GPT-4o). The classifier itself costs almost nothing — it's a tiny fine-tuned model that runs locally.

def route_to_model(query: str, context_length: int) -> str:
    # Complex tasks → GPT-4o
    complex_indicators = [
        "analyze", "compare", "reasoning",
        "multi-step", "code", "explain why"
    ]

    # Long context → GPT-4o
    if context_length > 8000:
        return "gpt-4o"

    # Complex reasoning → GPT-4o
    if any(ind in query.lower() for ind in complex_indicators):
        return "gpt-4o"

    # Default: GPT-4o Mini (17x cheaper)
    return "gpt-4o-mini"

Strategy 3: Prompt Compression

This one surprised us. The system prompt for Genstori's AI was 2,847 tokens. After compression — removing redundancy, tightening instructions, removing examples that weren't needed — it was 890 tokens. 69% smaller.

Since you pay per token in both input AND output, every token in your system prompt is paid for on every single API call. At 50,000 daily requests, a 1,957-token reduction saved:

• 1,957 tokens × 50,000 requests × 30 days = 2.9 billion tokens/month
• At GPT-4o pricing: 2.9B tokens × $2.50/1M = $7,250/month saved

Strategy 4: Response Streaming + Early Termination

For use cases where users don't need full responses, streaming with early termination reduces output tokens — and you only pay for tokens generated, not the full potential response.

For Genstori's content summarisation feature, users typically read the first 200 words before deciding to expand. We implemented streaming and let users stop generation early — reducing average output tokens by 35% on that feature alone.

Strategy 5: Batch Processing for Async Tasks

OpenAI's Batch API offers 50% cost reduction for non-real-time tasks. Any request that doesn't need an immediate response — background analysis, scheduled reports, bulk content generation — should use the Batch API.

For Genstori, 30% of their API calls were background tasks (content indexing, metadata generation, categorisation). Moving these to the Batch API halved their cost on those requests — without any user-facing impact.

from openai import OpenAI
import json

client = OpenAI()

# Create batch file
batch_requests = [
    {
        "custom_id": f"task-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [{
                "role": "user",
                "content": task["prompt"]
            }]
        }
    }
    for i, task in enumerate(background_tasks)
]

# Submit batch — 50% cheaper, 24hr turnaround
batch = client.batches.create(
    input_file_id=upload_batch_file(batch_requests),
    endpoint="/v1/chat/completions",
    completion_window="24h"
)

Full Results — Before vs After

Frequently Asked Questions

What to Do Next

If you're running an AI product in production, start with this priority order:

1. Audit your prompts — count your system prompt tokens. If it's over 1,000 tokens, it probably has room to compress.
2. Implement exact-match caching — Redis, one afternoon of work, immediate cost reduction.
3. Set up model routing — classify queries by complexity, route simple ones to Mini.
4. Move async tasks to Batch API — instant 50% reduction on non-real-time calls.
5. Add semantic caching — the most impactful but also most complex step. Worth it at scale.

Do them in order. Each one pays for the next. After all five, you'll be looking at a very different API bill.