Caching
Caching
Repeated prompts, answered in milliseconds
AI-FW's opt-in completion cache re-serves exact and near-duplicate prompts without another upstream call, cutting latency and token spend. Strict isolation means cached answers never cross models, endpoints, or tenants.
Highlights
- Exact cache hit, first request: 200, 0.64 s; identical second request: 200, 0.0075 s (about 80x faster), byte-identical bodies, log confirms Completion cache exact hit
- Semantic tier re-serves near-duplicate prompts with local embeddings
- Cache scopes per agent, group, model, or global, first match wins
- Streams and flagged content are never cached; cached bodies still pass response inspection
0.64 s
First request (uncached)
0.0075 s
Identical second request (exact cache hit)
~80x
Faster response on cache hit
byte-identical
Bodies, per the transaction log
See caching in your environment
The AI-FW gateway deploys into your own environment in minutes. Talk to us for a guided demo or jump straight into the docs.