Self-hosted AI gateway: the buyer's checklist
Most AI gateways can route a request and report on it. Fewer can prove who sent it, keep the content inside your network, and refuse to forward traffic when inspection is unhealthy. Use this checklist to tell the difference before you commit.
Where does inspection actually run?
This is the first question, because it decides your data residency answer. Some products route your prompt to a hosted classification service and return a verdict. That means a third party sees the content, and inspection stops working when their service does.
- Ask: does any part of your prompt leave my environment for inspection?
- Ask: what happens to policy enforcement when that service is unavailable?
- Prefer: inspection that runs in the same process as your traffic, with no external dependency
Can it prove who called?
A gateway sees an API key. A governance layer sees an identity. If an auditor asks which agent made a call, under whose authority, an anonymous key is not an answer.
- API keys with issue, label, revoke and hash-only storage
- Certificate-based authentication (mutual TLS) and self-enrolment for agents
- Group and tag scoping so policy applies to a workload, not a person's memory
What is recorded, and what is retained?
Audit value comes from decisions, not from copies of your prompts. A gateway that stores full content has created a new data set to protect, and a new breach surface.
- Metadata-only transaction records: identity, model, decision, rule, latency, outcome
- Truncated previews rather than full bodies
- Export to your own SIEM (syslog over TLS or a pull API) so records land where you already monitor
What happens when something fails?
The default on failure is the most important design decision in the product, and it is usually buried in the documentation.
- Fail-closed: no traffic while inspection is unhealthy, rather than forwarding it unscanned
- Per-model resilience (retries, failover, distribution) treated as availability, not as a way to bypass policy
- Explicit, documented error behaviour instead of silent pass-through
Can it govern the surfaces beyond chat?
Coding agents, tool calls and productivity copilots are AI traffic. If governance only covers the chat endpoint, the gap is where your most privileged callers live.
- Developer tooling: coding agents and assistant hooks governed by the same policy
- Tool calls: which tools an agent may invoke, with what arguments, under whose identity
- Productivity suites: catalogue, agent registry and usage from one admin surface
How do you operate it?
Governance you cannot run at 3am is a liability. Favour a small operational surface with a deployment story your team already knows.
- Runs as a container in your own environment, in front of your existing identity stack
- Configuration through an admin interface, applied immediately and surviving restarts
- A published, machine-readable licence so limits are visible rather than negotiated
Common questions
The gateway, its policy engine and its data store run inside your environment, in front of your existing network and identity stack. Requests leave your network only to the model providers you configured.
Only to reach the model backends you choose. Inspection, policy evaluation and logging run locally, so policy enforcement does not depend on a third-party service being reachable.
It can be deployed with no outbound access beyond your own model endpoints. Guardrail evaluation is local, which is what makes that possible.
Validate it yourself with our Technical Plan
A step-by-step guide to running your own proof of concept and a full head-to-head comparison, in your own environment, so you can decide on evidence rather than on a feature table.