How to Reduce LLM Token Costs — 7 Practical Strategies

1. Convert structured data from JSON to TOON

If your prompt includes structured data — user records, API responses, product catalogs, log entries — converting it to TOON before sending is the fastest way to cut tokens. Uniform arrays of objects typically save 40–60%. The conversion is instant and lossless.

Impact: 30–60% reduction on data-heavy prompts.
Effort: Low — paste JSON, copy TOON, done.
Tool: JSONtoTOON converter (free).

2. Use prompt caching for repeated context

Both Anthropic (Claude) and OpenAI (GPT-4) offer prompt caching. If your system prompt or context is the same across many calls, the provider caches the token computation and charges you a fraction of the input price on subsequent calls.

• Anthropic's prompt caching charges ~10% of input tokens on cache hits
• OpenAI charges 50% of input price on cached prompt prefixes

Structure your prompt so stable content (instructions, schema definitions, background data) comes first, and variable content (the user query) comes last. This maximises the cached prefix length.

Impact: 50–90% reduction on the cached portion of your prompt.
Effort: Medium — requires prompt restructuring.

3. Trim verbose system prompts

System prompts often accumulate instructions, caveats, and examples over time. Audit yours and remove:

• Redundant instructions (telling the model the same thing twice)
• Excessive whitespace and blank lines
• Verbose prose where a terse instruction works equally well
• Examples that are similar to each other (keep one representative, remove duplicates)

Impact: 10–30% depending on prompt quality.
Effort: Low — manual review takes 30 minutes.

4. Batch small requests

Multiple independent questions can often be combined into one prompt. Instead of ten separate calls asking "classify this text as positive/negative", send all ten texts in a single call and ask the model to classify each one.

Token overhead (model loading, formatting) is paid once per call, not per item. Batching 10 items into 1 call often cuts total tokens by 20–30% and API call count by 90%.

Impact: 20–30% on token count, 80–90% on per-call overhead.
Effort: Medium — requires prompt redesign.

5. Choose the right model for each task

Not every task requires GPT-4o or Claude 3.5 Sonnet. Classification, extraction, summarisation of short texts, and formatting tasks can often be handled by smaller models at 5–20× lower cost per token.

• Use a small model (GPT-4o-mini, Claude 3.5 Haiku, Llama 3.2) for routing, classification, and simple extraction
• Reserve large models for complex reasoning, code generation, and nuanced writing

Impact: 5–20× cost reduction on eligible tasks.
Effort: Low — swap model ID and test quality.

6. Trim context to what is actually needed

Conversation-based apps often send the entire conversation history with every turn. Consider:

• Summarising earlier turns once they exceed a threshold
• Dropping turns that are no longer relevant to the current query
• Using retrieval (RAG) to pull only the most relevant documents rather than always including everything

Impact: 20–50% on long-running conversations.
Effort: High — requires summarisation pipeline.

7. Set appropriate max_tokens and stop sequences

Output tokens cost the same or more than input tokens. If your task only needs a short answer, set max_tokens to a reasonable limit and add a stop sequence. This prevents runaway generation on edge cases.

For structured output tasks (JSON extraction, classification), a smallmax_tokens value (e.g., 100–300) combined with a JSON stop sequence significantly reduces average output token count.

Impact: Variable — highest for tasks with variable-length output.
Effort: Very low.

How can I reduce LLM API token costs?

The highest-impact single change for data-heavy prompts is converting structured data from JSON to TOON format, which reduces token count by 30–60% on arrays. Combined with prompt caching and model selection, most applications can achieve 50–80% cost reduction without degrading output quality.

Does TOON format work with all LLMs?

Yes. TOON is plain text. Any LLM — GPT-4o, Claude 3.5, Llama 3, Gemini — can read TOON data. Add a brief format description to your system prompt and the model reads it correctly.