What we learnt adding a semantic cache to a RAG service

We wanted our retrieval-augmented generation service to feel faster and cost less to run, but without letting cached answers become stale, overconfident, or just subtly wrong.

That is the awkward part of caching in a RAG system. Every user question can trigger retrieval, ranking, prompt construction, and a call to a generative model. That is exactly what you want when the question is new, nuanced, or dependent on freshly changed content. It is less satisfying when the service has already answered the same question, or a close rephrasing of it, many times before.

That was the motivation behind adding a semantic cache to the RAG service.

The idea sounds simple: if two questions mean the same thing, reuse the previous answer instead of running the full RAG pipeline again. In practice, the most interesting part of the work was not storing and retrieving vectors. It was deciding when reuse is safe, and when an answer is no longer safe to reuse.

Why semantic caching?

Traditional caches usually work with exact keys: same request in, same response out. Natural language does not behave like that.

Users might ask:

"How do I reconcile a bank account?"
"Can you explain bank reconciliation?"
"What are the steps for doing a bank rec?"

Those requests are different strings, but they may have the same intent. A semantic cache stores an embedding of the question, then uses vector similarity to find previously answered questions with a similar meaning.

For a RAG service, this can help with three things:

Latency: a cache hit avoids the slower retrieval and generation path.
Cost: fewer generation calls means lower model usage.
User experience: repeated questions feel much more responsive.

In an offline simulation over historical query patterns, we saw enough repeated intent to justify the work. The exact hit rate depends heavily on threshold, traffic mix and cache warm-up, so we treated the numbers as design input rather than a production promise.

This pattern also appears in AWS's reference architecture for Amazon OpenSearch Serverless and Amazon Bedrock: Build a read-through semantic cache with Amazon OpenSearch Serverless and Amazon Bedrock. The broad shape was familiar, but the service-specific details were where most of the engineering judgement lived.

The architecture at a glance

At a high level, the cache sits in front of the expensive part of the RAG workflow.

%%{init: {"themeVariables": {"fontSize": "18px"}}}%%
flowchart TD
    user["User query"] --> api["RAG API"]
    api --> cache["Semantic cache lookup"]
    cache --> embed["Embedding model"]
    embed --> vector["Vector search index"]
    vector --> decision{"Safe cache hit?"}
    decision -- yes --> cached["Return cached answer"]
    decision -- no --> rag["Run retrieval + generation"]
    rag --> write["Write answer to cache"]
    write --> response["Return answer"]

The main pieces are:

FastAPI service for the RAG API and workflow orchestration.
Local embedding model for query embeddings, loaded with sentence-transformers.
Amazon OpenSearch Serverless as the vector search store for cache entries.
AWS-hosted RAG path using retrieval infrastructure and foundation model calls.
Background writes so a successful RAG response can be stored without delaying the user response.
Management and invalidation APIs so cached answers can be listed and removed when content changes.
Observability for hit rate, miss reasons, latency and cache errors.

The cache is "read-through" from the user's point of view. The request checks the cache first. If there is a safe hit, the service returns the cached answer. If not, it runs the normal RAG path and schedules the cache write afterwards.

The word "safe" matters. A lookup can miss even when the vector search finds something similar. Common reasons include a low similarity score, a narrow gap between the best two matches, an incompatible request scope, an invalidated entry, or a cache version mismatch.

The first big decision: precision over hit rate

Semantic caching has an awkward failure mode. A missed cache hit is usually acceptable: the service does more work than necessary, but still answers through the normal RAG path. A bad cache hit is different. It can return an answer for the wrong intent.

That shaped the design. The cache is opportunistic, not authoritative. We preferred false negatives over false positives.

The AWS blog describes this as the balance between cache hits and cache collisions. That framing became central to the implementation. A similarity threshold alone was not enough; we also needed extra checks that reflected the domain and the way users ask questions.

Our hit decision combined several signals:

A similarity threshold, so weak matches miss.
A gap check, so the top result needs to be clearly better than the next candidate.
Scope filters, so answers are not reused across incompatible contexts.
Cache versioning, so embeddings from different model or service versions are not compared.
Lexical guardrails for known domain conflicts, so close accounting terms with different meanings do not collide.
Optional validation for borderline hits, where the extra latency is worth the confidence.

One useful mental model was to look for questions that sound close but should not share an answer. Accounts payable and accounts receivable, or apply and unapply, can sit close together in embedding space while pointing to different product behaviour. In those cases, the right cache behaviour is to miss.

That made the cache more conservative than the simplest possible implementation. It also made it safer. The trade-off was deliberate: a missed cache hit costs time; a wrong cache hit costs trust.

Why OpenSearch Serverless?

We considered Redis-style vector caching, but Amazon OpenSearch Serverless was a better fit for this service.

The main reasons were pragmatic:

We needed vector similarity search plus metadata filtering.
We already had operational familiarity with OpenSearch in the platform.
OpenSearch gave us useful query capabilities for listing, filtering and invalidating entries.
A managed serverless store avoided introducing another self-managed vector component.

There were trade-offs. In our implementation, OpenSearch Serverless behaved differently from self-managed OpenSearch in enough places that some operations needed to be modelled around eventual visibility and API-level constraints. Those constraints affected the design of invalidation, duplicate handling and background writes.

The lesson here was simple: a managed service can remove a lot of undifferentiated work, but it does not remove the need to design around its edges.

Local embeddings were worth it

One decision that surprised us was choosing a local embedding model rather than calling a managed embedding API for every lookup.

For a semantic cache, embedding latency is on the critical path. If generating the embedding takes too long, the cache loses a lot of its value. Local embeddings gave us lower and more predictable latency, avoided per-request embedding API cost, and removed a network dependency from the hot path.

That did introduce its own costs:

The model has to be downloaded, versioned and packaged.
Container images are larger.
Memory use and startup behaviour need attention.
Model changes need to be treated as cache compatibility events.

We handled that by downloading the model during the Docker build, loading it at service startup, and including the embedding model identifier in the cache version. When the model changes, old entries are automatically ignored because they belong to a different embedding space.

This was a good example of an engineering trade-off that only made sense in context. We started with the instinct that a managed API would be simpler, then changed our view once the embedding call became part of the critical path. For a low-throughput prototype, a managed embedding API might still be simpler. For a latency-sensitive cache lookup, local inference was a strong fit.

Invalidation mattered more than expected

Caching generated answers is only useful while those answers are still valid. This became more important than we first expected.

The service already had content update flows, so the cache needed to respect them. We added invalidation paths for content changes and management APIs for explicit cache removal. Cache writes happen asynchronously after the response, so we also had to think about races: what happens if a response is scheduled to be cached, then the underlying content is invalidated before the write runs?

In the normal path, the answer was to record invalidation timestamps and check them before executing delayed writes. If a write was scheduled before a relevant invalidation, it is skipped.

That detail is easy to miss in a diagram, but it matters. Without it, a background task could repopulate the cache with an answer from just before the content changed.

Observability is part of the feature

With a semantic cache, "hit" and "miss" are not enough.

We needed to know why a decision happened:

Was the score below threshold?
Was the top result too close to the next result?
Did a domain guardrail reject the hit?
Was an entry ignored because it had been invalidated?
Was optional validation attempted?
Did the cache fail open and continue through the RAG path?

Those reasons became metrics and structured logs. That made it possible to monitor the cache as a product behaviour rather than just an infrastructure component.

It also made tuning safer. If we change a threshold, we can watch the miss reasons and hit rate rather than guessing from anecdotal examples. The observability work changed the conversation from "is the cache working?" to "why did the cache choose not to answer?"

What I would tell someone building this again

The core pattern is straightforward. The hard parts are in the boundaries.

Start with the failure mode. A wrong cached answer is usually worse than a slow correct answer. Design the hit decision around that.
Do not rely on similarity alone. Thresholds are useful, but metadata scope, versioning, gap checks and domain-specific guardrails make the cache safer.
Treat invalidation as a first-class workflow. Generated answers are only as fresh as the content and prompt context behind them.
Make miss reasons observable. You cannot tune a semantic cache well if all misses look the same.
Benchmark with real usage patterns where possible. Synthetic examples are useful for proving the mechanism, but repeated user intent is what makes the cache valuable.
Be honest about managed-service constraints. Serverless vector search is powerful, but service-specific behaviour around visibility, deletion and pagination can still shape the design.
Version the embedding space. If the embedding model changes, the old vectors are not comparable. Make that impossible to forget.
Treat background writes and invalidation as one workflow. If they are designed separately, stale answers can reappear through timing gaps.

Final thoughts

Semantic caching is one of those ideas that looks like an optimisation but quickly becomes a correctness problem.

The architecture is useful because it can reduce latency and cost, but the design only works if the cache knows when not to answer. In our case, the most valuable engineering work was not the vector lookup itself. It was the surrounding discipline: conservative hit decisions, scoped cache keys, safe invalidation, background-write handling and observability.

The result is a cache that behaves more like an assistant to the RAG pipeline than a replacement for it. When it is confident, it helps. When it is not, it gets out of the way.

References

AWS Machine Learning Blog: Build a read-through semantic cache with Amazon OpenSearch Serverless and Amazon Bedrock
AWS documentation: Amazon OpenSearch Serverless
OpenSearch documentation: k-NN search
FastAPI documentation: Background Tasks
Sentence Transformers documentation: Sentence Transformers