RAG Optimization and Performance

Tune a RAG system for quality, speed and cost: chunk, embedding, retrieval, reranking and context optimization, plus caching, latency, throughput and cost control.

20 minAdvancedCode Examples
The Core Idea: A RAG system that is accurate but takes ten seconds, or that costs more than the problem is worth, will not survive in production. Optimization is the work of improving quality, latency and cost without sacrificing one for another by accident. The key discipline is simple: measure first, change one thing at a time, and check quality again after every speed or cost improvement.

1. Intermediate: Where Time and Money Go

Before optimizing, find out where the budget is actually spent. In a typical query the large costs are the language model call and, if used, the reranker. Embedding the query and searching the index are usually small.
StageLatencyCost driver
Query embeddingSmallTokens in the query
Vector searchSmall, grows with index sizeCompute and memory for the index
RerankingGrows with the number of candidatesModel calls per candidate
GenerationUsually the largest sharePrompt tokens plus answer tokens
Rule: instrument every stage, measure on realistic traffic, and optimize the stage that dominates. Guessing leads to speeding up what was already fast.

2. Intermediate: Optimizing for Quality

Each component of the pipeline has its own knobs. Change one at a time and measure on the golden dataset.
ComponentWhat to tuneHow you know it helped
ChunkingSize, overlap, structure rulesContext recall rises
EmbeddingsModel choice, dimensions, query and document settingsHit rate at k rises
RetrievalTop-k, hybrid weights, filters, thresholdsRecall rises without more noise
RerankingNumber of candidates, reranker modelMRR and precision at the prompt size rise
ContextNumber and order of chunks, compressionFaithfulness and correctness rise, tokens fall
One change at a time: if you change chunking and the reranker together and quality moves, you will not know which change caused it.

3. Intermediate: Caching

The fastest request is the one you never repeat. Caches store results that are expensive to compute and likely to be needed again.
Embedding cache
Store the vector for each normalized query, so repeated questions skip the embedding call.
Retrieval cache
Store the retrieved chunk ids for a query, valid until the index changes.
Semantic cache
Reuse a stored answer when a new question is very similar to an earlier one. Large savings, but a wrong match returns a wrong answer.
Cache safely: invalidate cached answers when their source documents change, and include user permissions in the cache key, otherwise one user may receive an answer built from documents another user is allowed to see. Normalizing queries first, such as lowercasing and removing punctuation, raises the hit rate.

4. Intermediate: Latency and Throughput

Latency is how long one request takes. Throughput is how many requests the system completes per second. They are related by a simple rule of queues: the number of requests in progress at once equals the arrival rate times the time each takes.
concurrent requests=requests per second×seconds per request\text{concurrent requests} = \text{requests per second} \times \text{seconds per request}
At 20 requests per second and 1.5 seconds per request, about 30 requests are in flight at any moment, so slower requests need more capacity. Track latency by percentile, such as p50 and p95, not by average. A p95 of 4 seconds means one user in twenty waits that long, even if the average looks fine.
Parallelize independent work, for example dense and sparse search at the same time.
Reduce candidates sent to the reranker, since its time grows with their number.
Stream the answer as it is generated so the user sees the first words early.
Use smaller models for routing and grading steps.
Batch embedding work during ingestion.

5. Advanced: Optimizing Cost Without Losing Quality

Prompt tokens usually dominate the cost of a query. The most effective savings come from sending less text that matters more.
LeverEffectRisk
Fewer, better chunks (reranking)Fewer prompt tokensMay cut a needed chunk
Contextual compressionShorter chunksMay drop a qualifier
Route easy questions to a cheaper modelLarge savings on common questionsA wrong route lowers quality
CachingSkips whole callsStale or mismatched answers
Smaller or quantized embeddingsLess storage and faster searchSmall recall loss
Track a pair of numbers: cost per query and quality on the golden set, together. A cost cut that quietly lowers correctness is a hidden bill that users pay.

6. Visual Explanation: Spending Less Without Getting Worse

Panel A shows the three goals and their main levers. Panel B uses the numbers from the code below to compare latency and cost before and after tuning. Panel C is a lever explorer.
Panel A · Three GoalsOptimize them together
Quality
Better answers
Chunks, embeddings, reranking, context
Speed
Lower latency
Caching, parallelism, fewer candidates, streaming
Cost
Fewer tokens
Compression, routing, caching, fewer chunks
Panel B · Before and AfterIllustrative numbers from the code in section 7
Latency before: 50 candidates, no cache
1050 ms
Latency after: 20 candidates, embedding cached
780 ms
Cost plan A: 8 chunks of 300 tokens
1.3 units per query
Cost plan B: 4 chunks of 150 tokens
0.4 units per query
Panel C · Lever ExplorerClick a lever to open it
Click to see why averages are not enough for latency▼
Users remember the slow requests. Percentiles such as p95 show how the slowest few percent behave, which the average hides completely.
How to read it: Panel A names the goals, Panel B shows real before and after numbers, and Panel C lists what each lever saves and what to watch.

7. Python Implementation: Cache Hit Rate, a Latency Model and a Cost Model

The program builds three small models. First, an embedding cache keyed by a normalized query shows how spelling and punctuation differences are absorbed. Second, a latency model adds the time of each stage, with reranking proportional to the candidate count, and compares two configurations. Third, a cost model converts chunk count and chunk size into prompt tokens and into a monthly total. The timings and prices are illustrative, not measurements of any real system.
rag_optimization.pyPython 3.10+ · Standard library only
def normalize(query):    return " ".join("".join(ch for ch in query.lower() if ch.isalnum() or ch == " ").split()) cache = dict()stats = [0, 0] def embed_cached(query):    key = normalize(query)    if key in cache:        stats[0] += 1        return "hit"    cache[key] = True    stats[1] += 1    return "miss" queries = ["How do refunds work?", "how do refunds work", "What is the shipping time?", "How do REFUNDS work?", "what is the shipping time"]for q in queries:    embed_cached(q)print("Embedding cache: hits", stats[0], "| misses", stats[1], "| hit rate", round(stats[0] / len(queries), 2)) # A simple latency model, in millisecondsdef latency(candidates, embed_cached_flag):    embed = 0 if embed_cached_flag else 30    search = 20    rerank = 8 * candidates    generate = 600    return embed + search + rerank + generate before = latency(50, False)after = latency(20, True)print("Latency before:", before, "ms | after:", after, "ms | saved:", before - after, "ms") # A simple cost model with illustrative prices (units per 1000 tokens)def cost_per_query(chunks, tokens_per_chunk, price_per_1k=0.5):    prompt_tokens = 200 + chunks * tokens_per_chunk    return prompt_tokens, prompt_tokens / 1000 * price_per_1k tokens_a, cost_a = cost_per_query(8, 300)tokens_b, cost_b = cost_per_query(4, 150)print("Plan A: 8 chunks x 300 tokens =", tokens_a, "tokens, cost", round(cost_a, 2), "units")print("Plan B: 4 chunks x 150 tokens =", tokens_b, "tokens, cost", round(cost_b, 2), "units")monthly = 100000print("Per month at", monthly, "queries:", round(cost_a * monthly), "vs", round(cost_b * monthly), "units")
Expected output
Embedding cache: hits 3 | misses 2 | hit rate 0.6Latency before: 1050 ms | after: 780 ms | saved: 270 msPlan A: 8 chunks x 300 tokens = 2600 tokens, cost 1.3 unitsPlan B: 4 chunks x 150 tokens = 800 tokens, cost 0.4 unitsPer month at 100000 queries: 130000 vs 40000 units
Five queries contained only two distinct questions, so normalization turned three of them into cache hits. Cutting rerank candidates from 50 to 20 and skipping the cached embedding saves 270 milliseconds in the model. Halving the number of chunks and the length of each, for example with reranking and compression, cuts the prompt from 2,600 to 800 tokens, so the monthly bill falls from 130,000 to 40,000 units in this example. In a real project you would verify on the golden set that quality did not drop.

Knowledge Check

Q1. Why track p95 latency instead of only the average?
Q2. What must a cache key include when documents have permissions?
Q3. A change reduces cost by 70 percent. What should you check next?

Key Points

✓
Measure each stage first; generation and reranking usually dominate time and cost.
✓
Tune chunking, embeddings, retrieval, reranking and context one change at a time on the golden dataset.
✓
Embedding, retrieval and semantic caches save work; invalidate them when documents change and include permissions in the key.
✓
Latency is time per request, throughput is requests per second, and concurrency is their product.
✓
Report latency by percentile, not by average.
✓
Prompt tokens dominate cost: use reranking, compression, routing and caching to send less and better text.
✓
Track cost and quality together so savings never hide a loss in correctness.

The Big Picture

Optimization is a loop of measuring, changing one thing and measuring again. With an evaluation set to protect quality and a profile to show where time and money go, you can make a RAG system faster and cheaper with confidence.
Profile → Pick the biggest stage → Change one thing → Re-measure quality and cost → Keep or revert
Up Next: Production RAG Architecture. You will design the backend: APIs, scaling the vector database, batch and real-time ingestion, updates and deletions, freshness, versioned knowledge bases and fallback strategies.