# AI-FW - full documentation
AI-FW is an AI firewall and governance gateway, self-hosted in your environment, that identifies, authorizes, inspects, governs, and audits all AI traffic. This file contains the full text of the knowledge base published at https://aifw.io/docs.
Source: https://aifw.io · Docs source: https://github.com/securetron-gh/aifw
---
# Settings reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/settings
Every tab of the Settings page - General, Access, Identity, Model, Routing, Guardrails & Inspection, Performance, Audit, Export, Integrations, and Licensing.
The Settings page organizes every runtime configuration option into tabs. All
settings are database-backed, apply instantly, and survive restarts.
## General
Gateway-wide basics, including the product version shown in the UI.
## Access
- **Authentication mode**: `required` (no token returns 401), `optional`
(anonymous allowed, tokens validated), or `passthrough` (any non-empty key
accepted and forwarded verbatim).
- **Key mode**: `model` (registered model keys win upstream) or `client`
(the client's key is forwarded).
- **Streaming**: `buffered` (full response scanned before delivery) or
`stream` (live tokens, mid-stream blocks terminate the stream).
- **Manage API keys**: issue, generate, label, and revoke admin-issued keys.
## Identity
OIDC SSO provider configuration (authority, client ID, secret, scopes, sign-in
mode, role map), SCIM provisioning, and forwarded-headers trust settings.
## Model
Default model, default backend, and the **Reject Unregistered Models** strict
toggle.
## Routing
Conditional routing rules, model groups, agent groups, and agent tags used by
the routing engine.
## Guardrails & Inspection
Prompt and response scanning, the semantic analysis tier (backend, endpoint,
model, timeouts, fail-closed toggle), prompt compression, fail-open options,
and Claude inference hooks.
## Performance
Reliability defaults, retry behavior, and per-model resilience defaults.
## Audit
Logging policy (per-direction content storage, retention), audit events, and
the Audit Logs page size.
## Export
Log sources and the syslog collector (transport, host, port, facility,
APP-NAME).
## Integrations
M365 Copilot admin bridge (app-only credentials and test connection) and the
endpoint gateway (images, audio, files, fine-tuning, batches).
## Licensing
The edition of this gateway and its token allowance.
- **Edition badge**: shows `Community` or `Licensed`. The edition is set when
the gateway is deployed and is read-only here.
- **Usage vs allowance**: current token usage compared with the published
monthly allowance for the Community Edition (see the
[Community Edition license](/docs/community-edition/license)).
- **Quota toggle**: enable or disable cap enforcement for this gateway.
- **Upgrade**: a link to [contact sales](https://aifw.io/contact) for the
licensed edition.
The Dashboard also shows a **Community Quota** stat card with usage against
the allowance when the edition is Community.
- [Community Edition license](/docs/community-edition/license)
- [Identity & access](/docs/guides/identity-access)
- [Audit logs & export](/docs/guides/audit-logs-export)
- [Model routing](/docs/guides/model-routing)
---
# Model Inventory reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/model-inventory
The Model Inventory page and every option: filters, enable/disable, add/edit/delete, protocols, resilience, compression, advanced parameters, groups, and routing rules.
The Model Inventory is the single place where models are managed.
## List and filters
The inventory table lists every registered, disabled, and discovered model.
Server-side filters narrow the list by **search** (model id, provider, or
backend URL) and by **status** (registered, disabled, discovered), composing
with pagination.
## Model actions
- **Enable / Disable** - flips a model between `registered` and `disabled`.
Requests for a disabled model are rejected with `400 model_disabled`.
- **Add custom model** - model ID, provider, backend URL, optional API key, and
protocol.
- **Edit** - update provider, backend URL, API key, protocol, and per-model
settings.
- **Register** - promote a discovered (shadow) model to registered.
- **Delete** - remove a custom model.
## Per-model settings (Edit modal)
| Setting | What it controls |
|---|---|
| Protocol | `openai` or `anthropic` (facades translate between them) |
| Key mode | Override the upstream key behavior for this model |
| Streaming mode | Override buffered vs live streaming for this model |
| Resilience | Retry limit, retry statuses, reliability mode (off/failover/distribute), primary share, alternate endpoints |
| Compression | Per-model override (inherit / on / off) for prompt compression |
| Advanced parameters | Inject-when-absent or force overrides for request parameters |
## Model groups and routing rules
The page also hosts the **Model Groups** card (create, edit, delete, member
checkboxes) and the **Routing Rules** card (ordered list with enable/disable,
reorder, and edit/delete). See the [model routing guide](/docs/guides/model-routing).
## Shadow AI
Unknown models seen in traffic appear under **Shadow AI (unregistered)** and on
the dashboard, so you can review and register them.
- [Configure models & keys](/docs/how-to/configure-models-keys)
- [Model routing & registry](/docs/guides/model-routing)
- [Reliability & caching](/docs/guides/reliability-caching)
---
# Rules Manager reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/rules-manager
Built-in rules, custom rules, rule types and actions, per-rule options, priority, and AI-assisted rule creation.
The Rules Manager is where guardrail rules are created and governed.
## Built-in rules
The built-in "System" rules cover the common vectors: jailbreak detection, PII
masking, toxicity filtering, and data-exfiltration blocking. Each can be
enabled or disabled from the Rules Manager.
## Custom rules
Create, edit, enable/disable, and delete custom rules with:
| Option | What it controls |
|---|---|
| **Type** | Regex / keyword, or semantic (natural-language intent) |
| **Action** | `block`, `mask`, `log_only` (accept, no content persisted), or `accept` (accept and log) |
| **Severity** | Your own triage label |
| **Pattern / intent** | The regex, keyword, or intent description |
| **Skip semantic check** | Opt a matching request out of the semantic tier |
| **Source IP / subnet** | Apply the rule only inside a CIDR |
| **Destination IP / subnet** | Apply the rule only when the backend resolves inside a CIDR |
| **Key mode / streaming** | Pin upstream key behavior and streaming mode for matching requests |
| **Rule scoping** | Apply to agents (ids, globs, tags, groups), exclude agents, exclude source IPs |
## Evaluation
Rules are evaluated in a fixed order: built-ins, then block rules, mask rules,
accept/log rules, then the semantic tier, then the implied default action.
See the [guardrails guide](/docs/guides/prompt-response-guardrails) for the
full order. Deny always wins over accept.
## Create Rule with AI
The **Create Rule with AI** card drafts a rule from a plain-language objective
(keyword or semantic), with an editable preview before the rule is committed.
See [Create rules with AI](/docs/how-to/create-rules-with-ai).
## Prompt Analyzer
The **Prompt Analyzer** card tests any prompt against the enabled inbound rule
base without sending it upstream. It is safe under any license or quota state:
no model call is made and no log rows are written.
- **Matched rules**: each text, regex, or keyword rule that fires, with its
action and severity.
- **Semantic scores**: per-rule cosine scores with block and flag banding,
plus the overall verdict (block, flagged, or allowed).
- **Masked preview**: the redacted prompt when mask rules apply.
Paste a prompt and run the analysis to see exactly which rules would fire and
why, before you commit the rules to production.
- [Prompt & response guardrails](/docs/guides/prompt-response-guardrails)
- [Semantic intent analysis](/docs/guides/semantic-intent-analysis)
---
# Risk Profiles reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/risk-profiles
User, agent, and IP risk leaderboards, category risk cards, risk auto-block, and reset actions.
The Risk Profiles page surfaces risk per identity and per category.
## Leaderboards
| Surface | Shows |
|---|---|
| **Users** | Risk score, semantic flags and blocks, violation count, first/last seen, identity badge (verified vs header), per-user reset, drill-down into recent transactions |
| **Agents** | Risk scores (block violations only), requests, violations, band, reset |
| **IP addresses** | Per-source-IP risk from blocks and semantic flags, with drill-down into the users and agents behind each IP |
Summary cards: Total Users, New AI Users, High-Risk Users, and Semantic Flags.
## Category risk cards
All-time and 7-day detections per category with top entities and recent
transactions:
- **Semantic Risk Analysis** - blocks vs flags per semantic rule
- **PII Risk** - SSNs, emails, API keys in traffic
- **PCI (Card) Risk** - payment card numbers
- **Health Data Risk** - medical terms and identifiers (heuristic, not an
attestation)
## Risk auto-block
Configure the count-based guard: violations within a rolling window trigger a
cooldown block, with the switch, count, window, and duration configurable.
See [Risk profiles & auto-block](/docs/guides/risk-profiles).
- [Risk profiles & auto-block](/docs/guides/risk-profiles)
- [Audit logs & export](/docs/guides/audit-logs-export)
---
# Audit Logs reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/audit-logs
The transaction log, filters, the Events tab, and page-size settings.
The Audit Logs page shows the metadata-only transaction trail.
## Transactions
Every request is logged with model, identity, source IP, latency, decision, and
truncated previews per the logging policy. Server-side filters narrow the list
by user, agent, source IP, model, block code, and free-text search, with
server-side pagination (default 50 rows per page, Prev/Next controls).
## Events
The **Events** tab records security and application-change events: logins,
logouts, failed credential attempts, configuration changes (secrets masked),
rule mutations, user and key lifecycle events, and M365 bridge mutations.
## Export
From the same page, exports feed the pull API and syslog collector. See
[Audit logs & export](/docs/guides/audit-logs-export) for the full details.
- [Audit logs & export](/docs/guides/audit-logs-export)
- [Observability & dashboard](/docs/guides/observability-dashboard)
---
# Dashboard reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/dashboard
Every card and control on the AI-FW Dashboard - traffic stats, latency, guardrail snapshot, 24-hour metrics, and charts.
The AI-FW Dashboard is the operational view of gateway traffic. Every metric is
built from metadata and truncated previews; raw content is never stored.
- **Traffic stats**: allowed, blocked, block rate, P95 latency, registered
models, tracked agents, tracked users.
- **Traffic-mix donut** and **latency sparkline**.
- **Guardrail snapshot**: a stacked bar of block types.
- **24-hour metrics row**: semantic flags and blocks, text-rule blocks and
deny-by-default, unique callers, token volume, and tokens saved by
compression.
- **Charts**: prompt-lifecycle Sankey, 7-day by 24-hour heatmap, requests and
tokens by hour, top models and agents by requests, top models and IPs by
tokens.
- **Live Activity view**: transactions as they happen, including real streamed
content previews.
- **Community Quota card**: when the edition is Community, token usage against
the published monthly allowance with the current license status.
See [Observability & dashboard](/docs/guides/observability-dashboard) for the
full walkthrough.
- [Observability & dashboard](/docs/guides/observability-dashboard)
- [Audit Logs reference](/docs/admin/audit-logs)
---
# M365 Copilot admin reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/m365-copilot
The M365 Copilot page - agent catalog, agent registry, usage analytics, and interaction export with app-only authentication.
The M365 Copilot page manages the Copilot admin surfaces through Microsoft
Graph with app-only (client-credentials) authentication.
- **Connection**: enable the bridge and enter Tenant ID / Client ID / Client
Secret; **Test connection** acquires an app-only token and lists the agent
catalog to prove the application permissions work.
- **Agent catalog (packages)**: live inventory of agents in the organization
with per-agent details (name, type, platform, hosts, publisher, version,
blocked state, deployment).
- **Agent registry**: create, view, update, and delete agent registrations,
including syncing agents registered on AI-FW with M365 (Beta endpoint).
- **Usage analytics**: per-user activity, user-count summaries, and daily
user-count trends for D7 / D30 / D90 / D180.
- **Interaction export**: export a user's Copilot prompt and response history.
Every mutation and report load is audited, fail-closed on Graph errors, and the
client secret is masked. Mock Graph is honored only in non-production
environments.
See [M365 Copilot admin & agent sync](/docs/tutorials/m365-copilot-bridge).
- [M365 Copilot admin & agent sync](/docs/tutorials/m365-copilot-bridge)
- [Identity & access](/docs/guides/identity-access)
---
# Users reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/users
Local users, roles, passwords, SSO, SCIM provisioning, and user groups.
The User Management page controls who can use the admin surface.
## Local users
- **Add / update** - create users, assign roles, set passwords, and mark
accounts active or inactive.
- **Remove** - delete users, with self-delete and last-admin protection.
- Passwords are stored as salted hashes (PBKDF2).
## Roles
| Role | Access |
|---|---|
| **Admin** | Full access to everything |
| **Read-Only** | View the entire UI, all mutations blocked |
| **AI-FW Admin** | Manage the AI firewall surface only |
| **Agent-Trust Admin** | Manage the agent inventory and trust only |
## SSO & provisioning
- **OIDC SSO** - provider configuration (authority, client ID, secret, scopes,
callback), sign-in mode (`local`, `oidc`, or `both`), and a runtime role map.
Unknown or absent claims map to Read-Only.
- **SCIM 2.0** - enable with a bearer secret; users and groups provision from
your identity provider, with roles assigned from matching groups.
## User groups
Named groups of users (local usernames and/or OIDC email or UPN) that share
one or more roles. Members inherit the group's roles at sign-in; the effective
role is the user's own role plus their groups' roles. Group changes apply on
the next sign-in and all mutations are audited.
See [Identity & access](/docs/guides/identity-access).
- [Identity & access](/docs/guides/identity-access)
- [Admin-issued API keys](/docs/guides/agent-api-keys)
---
# Agent API keys reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/agent-api-keys
The Agent API Keys page - add, generate, label, and revoke keys with hashed storage and per-key identity.
The Agent API Keys page manages admin-issued keys that agents and tools send as
`Authorization: Bearer ` or `x-api-key`.
- **Add or Generate** - paste an existing key or generate a new one, with an
optional label.
- **Revoke** - immediate, no waiting.
- Keys are stored as **SHA-256 hashes only**, never plaintext.
- Behavior per authentication mode: enforced in `required`, validated in
`optional`, not checked in `passthrough`.
- Keys authenticate the caller and are **not** forwarded upstream (use key mode
`model` for registered-model keys).
- Audit and risk identity is the key's unique registry id, displayed as
`label (id)` on the Audit Logs page.
- Each key gets its own completion-cache namespace.
See [Admin-issued API keys](/docs/guides/agent-api-keys).
- [Admin-issued API keys](/docs/guides/agent-api-keys)
- [Identity & access](/docs/guides/identity-access)
---
# Agents reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/agents
The agent registry admin page - onboarding modes, identity proof, approval, groups and tags, webhooks, and deboarding.
The Agents page manages the agent registry and trust.
## Registry policies
- **Onboarding mode** - open, approval-required, or admin-only registration.
- **Identity proof level** - whether agents must present a public key or
certificate thumbprint.
- **Registration enforcement** - whether unregistered agents are rejected.
- **Allowed authentication methods** - which of API keys, JWT, mTLS, or
Kerberos agents may use.
## Agent records
Each agent record shows its identity, owner and sponsor, tags, usage and data
classification, status, and webhook. Admins can edit most fields; the identity
proof is set by the client at registration and cannot be edited.
## Groups and tags
- **Agent groups** - named collections of agents (explicit ids and/or tags),
optionally owned by a user or user-group.
- **Tags** - free-form labels used by routing rules, rule scoping, and groups.
## Webhooks and deboarding
Agents can register an outbound webhook for notifications. Deboarding removes
an agent from the registry and suspends its identity, without requiring key
rotation for other agents.
See [Agent self-enrollment](/docs/tutorials/agent-self-enrollment) and the
[A2A protocol reference](/docs/api-reference/a2a-agent-protocol).
- [Agent self-enrollment](/docs/tutorials/agent-self-enrollment)
- [Model routing & registry](/docs/guides/model-routing)
---
# CA integrations reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/ca-integrations
SCEP and ACME enrollment, the central CA trust store, and server-certificate enrollment.
The CA Integrations page manages the certificate infrastructure the platform
relies on.
## CA trust store
A central store of CA certificates used across the platform:
- CA chains fetched from **SCEP** (`GetCACert`) or **ACME** (finalize chain).
- Uploaded **root and intermediate PEMs** for trust anchors.
- The same store supplies **agent-mTLS trust anchors**, so runtime validation
of agent certificates happens against one unified source.
## Server-certificate enrollment
The gateway's server authentication (TLS) certificate is enrolled through a CA
integration (SCEP or ACME, server-side key path) and stored as a PFX. The mTLS
listener binds it with hot rotation and file or self-signed fallbacks.
## Integrations
Add, edit, and remove CA integration profiles (name, URL, credentials), and
manage which integrations the gateway uses for enrollment.
- [Identity & access](/docs/guides/identity-access)
- [Agent self-enrollment](/docs/tutorials/agent-self-enrollment)
---
# Agent Trust reference
Section: Admin Reference · URL: https://aifw.io/docs/admin/agent-trust
The Agent Trust dashboard - agent inventory, approvals, tasks, and trust settings.
The Agent Trust surface manages agents and the A2A protocol operations.
## Agent Trust dashboard
An overview of registered agents, their status, recent tasks, and trust
signals, including reputation and risk per agent.
## Admin approvals
Pending agent registrations that require approval appear here when the
registry is in approval mode, with accept or reject actions.
## Tasks
A2A tasks (certificate requests and other operations) are visible with their
state, result artifacts, and failure reasons. See the
[A2A protocol reference](/docs/api-reference/a2a-agent-protocol).
## Trust settings
Registry policy (onboarding mode, identity proof level, enforcement, allowed
authentication methods) is configured from the Agent Trust settings.
- [Agents reference](/docs/admin/agents)
- [Agent self-enrollment](/docs/tutorials/agent-self-enrollment)
- [A2A agent protocol](/docs/api-reference/a2a-agent-protocol)
---
# OpenAI-compatible API
Section: API Reference · URL: https://aifw.io/docs/api-reference/openai-compatible-api
The /v1 endpoints, chat completions, request headers, error codes, streaming, and cache headers.
The gateway exposes OpenAI-compatible endpoints at its `/v1` base URL. Any client
that speaks the OpenAI API can point at the gateway instead of the provider.
## Endpoints
| Method & path | Purpose |
|---|---|
| `POST /v1/chat/completions` | OpenAI-style chat completions (native, and via facade to Anthropic backends) |
| `POST /v1/responses` | Responses API relay for newer SDKs; adapted to chat, run through the same pipeline |
| `POST /v1/embeddings` | OpenAI-style embeddings (routed to a configured embeddings provider) |
| `GET /v1/models` | List registered models |
Common OpenAI-compatible route aliases are accepted alongside the canonical
paths, so existing SDKs and tooling connect without changes.
## Endpoint Gateway (images, audio, files, fine-tuning, batches)
Beyond chat, messages, and embeddings, the gateway can expose the remaining
OpenAI-compatible surfaces. These are **opt-in and disabled by default**;
disabled endpoints return `404` with code `endpoint_disabled`. Enable each kind
in **Settings -> Endpoint Gateway**:
| Surface | Gate | Behavior |
|---|---|---|
| `/v1/images/generations`, `edits`, `variations` | images | JSON prompt scanned, masked, and compressed like a chat prompt; response metadata scanned; inspection-gated cache for generations |
| `/v1/audio/speech` | audio | Input scanned like a chat prompt, then relayed |
| `/v1/audio/transcriptions`, `translations` | passthrough | Multipart governed pass-through with key injection and audit |
| `/v1/files` | files | Text files decoded and scanned; violations block the upload, binary files pass through |
| `/v1/fine_tuning/jobs`, `/v1/batches` (+ cancel) | passthrough | Governed pass-through with key injection and audit |
The media content itself (pixels, audio waveforms) is never analyzed, only the
surrounding text. Every request records the endpoint kind in the transaction
log.
## Request headers
| Header | Purpose |
|---|---|
| `Authorization: Bearer ` | Client authentication and/or upstream key per the configured auth and key modes |
| `X-Agent-ID` | Agent identity, tracking, routing rules, risk, cache scoping |
| `X-User-Id` | User identity for tracking (authenticated token `sub` claims are preferred) |
| `x-aifw-cache-refresh` | Bypass the cache lookup for this request (the write is still allowed) |
## Standard request
```json
POST /v1/chat/completions
{
"model": "gpt-4o",
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Hello!" }
],
"stream": false
}
```
| Field | Notes |
|---|---|
| `model` | Optional, omitted values are filled from the configured default model |
| `messages` | Standard chat messages; `tool_calls`/`tool` sequences pass through byte-identical |
| `stream` | `true` applies the effective streaming mode (buffered or live) |
## Error codes
Blocked or rejected requests return `HTTP 400` with a structured error:
```json
{
"error": {
"message": "The transaction was intercepted and blocked by corporate AI security policy.",
"type": "guardrail_violation",
"code": "prompt_injection_detected",
"details": { "policy_id": "BUILTIN-JAILBREAK", "timestamp": "2025-01-15T10:00:00Z" }
}
}
```
Common codes:
| Code | Meaning |
|---|---|
| `guardrail_violation` | Blocked by a rule (built-in or custom); details carry the policy ID |
| `prompt_injection_detected` | Jailbreak/prompt-injection rule matched |
| `semantic_rule_violation:` | Semantic tier scored above the block threshold |
| `model_disabled` | The model is registered but not enabled |
| `model_not_registered` | Strict mode is on and the model isn't in the registry |
| `backend_not_configured` | No backend URL anywhere for the resolved model |
| `default_policy_block` | Deny-by-default fired (nothing matched, default action is block) |
Blocked responses also carry the `X-Gateway-Security: Violated` header. Auth
failures return `401`; unknown routes `404`.
## Streaming
With `stream: true`, responses use server-sent events exactly like the OpenAI
format, ending with `data: [DONE]`. The gateway applies the effective streaming
mode (see [identity & access](/docs/guides/identity-access#streaming-modes)):
- **buffered** (default): the full response is scanned before any token is
delivered; a violation returns `400` before delivery.
- **stream**: tokens flow live; a mid-stream violation terminates the SSE stream
with a `guardrail_violation` error event. Tokens already delivered cannot be
retracted.
## Caching
When a cache scope is configured (see the
[reliability guide](/docs/guides/reliability-caching)), responses may be served
from the completion cache. Cache hits are recorded in the transaction log, and
cached bodies still pass through response inspection on replay.
- [Connect an OpenAI SDK](/docs/tutorials/connect-openai-sdk), client examples
- [Anthropic Messages API](/docs/api-reference/anthropic-messages-api), the other native protocol
---
# Anthropic Messages API
Section: API Reference · URL: https://aifw.io/docs/api-reference/anthropic-messages-api
The /v1/messages endpoint, Anthropic-native chat, x-api-key auth, facades, and streaming.
The gateway speaks the **Anthropic Messages protocol** natively, so Claude-native
clients (Claude Code, Claude CoWork, Claude Desktop) can route through the same
inspection pipeline as OpenAI-compatible clients.
## Endpoint
| Method & path | Purpose |
|---|---|
| `POST /v1/messages` | Anthropic-style messages (native, and via facade to OpenAI-style backends) |
## Authentication
Claude clients authenticate with either:
- `x-api-key: ` (console keys), or
- `Authorization: Bearer ` (OAuth-style tokens).
The gateway accepts both, applies the configured auth mode, and relays to the
registered Claude model's backend with `anthropic-version: 2023-06-01` and the
resolved key (rule → model → global precedence).
## Standard request
```json
POST /v1/messages
{
"model": "claude-sonnet-4-20250514",
"max_tokens": 4096,
"system": "You are a helpful assistant.",
"messages": [
{ "role": "user", "content": "Hello!" }
],
"stream": false
}
```
The response uses the standard Anthropic schema (`content` blocks, `stop_reason`,
usage metadata).
## Model protocol metadata
Each registered model carries an `apiStyle`, `openai` (default) or `anthropic`:
- Model IDs starting with `claude-` are registered as `anthropic` automatically.
- The protocol column in the Model Inventory can change it per model.
## Facades
The gateway translates between the two protocols transparently:
- **Anthropic client → OpenAI-style model** (`/v1/messages` to a `gpt-*`
backend): the Messages request is translated to `chat/completions` and the
response/SSE back to Messages events (`content_block_delta`).
- **OpenAI client → Claude backend** (`/v1/chat/completions` to a `claude-*`
backend): the request is translated to the Messages schema, `system` and
`developer` messages hoisted into the initial system message, `max_tokens`
defaulting to 4096, `temperature` clamped to 0–1, and the response translated
back to the OpenAI shape (`tool_use` → `tool_calls`).
Both directions run through the same inspection pipeline; rule enforcement and
the semantic tier are protocol-agnostic.
## Streaming
`stream: true` works with Anthropic `content_block_delta` frames, scanned
token-by-token. A mid-stream violation terminates the SSE stream with a
`guardrail_violation` error event (in the buffered mode, the full response is
scanned before delivery and a violation returns `400`).
## Error shape
Blocked requests return `HTTP 400` with an error translated to the Anthropic error
shape (and OpenAI errors → Anthropic shape on the reverse facade), so
Anthropic-SDK clients can parse failures the same way they parse provider
failures.
- [Claude Code, Cursor & MCP tools](/docs/tutorials/claude-code-cursor-mcp), client setup
- [OpenAI-compatible API](/docs/api-reference/openai-compatible-api), the other native protocol
---
# A2A agent protocol
Section: API Reference · URL: https://aifw.io/docs/api-reference/a2a-agent-protocol
The Agent2Agent (A2A) surface, the agent card, registration, task-based certificate operations, MCP, and webhooks.
Agent Trust implements the **Agent2Agent (A2A) v1.0.0** protocol, letting AI
agents onboard with an identity and perform PKI operations, certificate issuance,
renewal, and revocation, through a task-based interaction model.
## Agent card
Every deployment publishes a machine-readable agent card:
```
GET /.well-known/agent-card.json
```
It advertises the A2A endpoints, the authentication schemes accepted (`bearerAuth`
and/or `mtlsAuth`), and the skills an agent can use. Clients should read the card
at startup, it is the single source of truth for the deployment.
## Authentication
| Scheme | How |
|---|---|
| **Bearer / JWT** | `Authorization: Bearer ` against your configured signing secret; the `sub` claim becomes the agent identity |
| **mTLS** | Client certificate on the mTLS listener; certificate identity (SPIFFE URI → subject key identifier → common name) maps to the agent identity |
| **Kerberos** | Domain-joined service accounts via `Authorization: Negotiate` |
Auth mode is configurable: `off` (anonymous, IP-based identity, backward
compatible), `jwt`, or `oidc` (browser session or mTLS cert).
## Register an agent
```
POST /a2a/v1/register
```
| Field | Notes |
|---|---|
| `agentId` | Unique agent identity; bound to the caller on first registration |
| `displayName` / `agentCardJson` | Human-readable identity and capabilities |
| `publicKeyPem` / `certThumbprint` | Identity proof, required when the registry policy requires it; set by the client, not editable by admins |
| `tags[]` | Free-form labels used by routing rules and agent groups |
| `webhookUrl` | Optional HTTPS outbound webhook for notifications (requires an authenticated caller) |
The registry binds `agentId` to the registering caller's identity; a different
caller cannot take it over. Register every agent with a unique `agentId`, agents
sharing an IP without one collapse into a single identity.
## Tasks
Certificate operations are **tasks**, asynchronous, pollable units of work:
| Method & path | Purpose |
|---|---|
| `POST /a2a/v1/tasks/send` | Create a task with a user message (e.g. a CSR file part) |
| `GET /a2a/v1/tasks/{taskId}` | Poll task state and result artifacts |
| `POST /a2a/v1/tasks/{taskId}/cancel` | Cancel a running task |
| `POST /a2a/v1/tasks/{taskId}/subscribe` | SSE stream of task updates |
Send a CSR as a file part, poll until `completed`, then retrieve the issued
certificate chain artifact. See the
[self-enrollment tutorial](/docs/tutorials/agent-self-enrollment) for the full
round-trip with real request/response shapes.
## Streams & idempotency
- SSE streaming for `message:stream` and `subscribe` endpoints.
- **Idempotency** via an `X-Idempotency-Key` header, retries with the same key
don't create duplicate tasks.
## Webhooks
Agents can register an outbound **webhook** (HTTPS, public host, authenticated
caller) to receive notifications instead of polling. Webhook deliveries are
verifiable and failures are surfaced in the agent's record and the audit log.
## MCP compliance tools
MCP-enabled agents can also call the gateway's compliance tools directly over
MCP, including checking a prompt *before* sending it to a model and acting on the
verdict (`allow` / `deny` / `flag` / `mask`). See the
[Claude Code, Cursor & MCP tutorial](/docs/tutorials/claude-code-cursor-mcp).
## Task states
| State | Meaning |
|---|---|
| `submitted` | Task created, waiting |
| `working` | Task being processed |
| `input-required` | Task needs more input from the agent |
| `completed` | Done, result artifacts available |
| `canceled` | Canceled by the agent or admin |
| `failed` / `rejected` | Failed, or rejected (e.g. while the identity is blocked) |
| `auth-required` | The agent must authenticate before the task proceeds |
- [Agent self-enrollment (CSR + mTLS)](/docs/tutorials/agent-self-enrollment), end-to-end walkthrough
- [Identity & access](/docs/guides/identity-access), auth modes and roles
---
# OpenAI-compatible models & endpoints
Section: API Reference · URL: https://aifw.io/docs/api-reference/openai-compatible-endpoints
200+ OpenAI-compatible models and their vendor endpoints - Claude, OpenAI, Azure, Grok, Meta Llama, DeepSeek, AWS Bedrock, Qwen, Z.ai, and more.
The AI-FW Model Inventory accepts any OpenAI-compatible backend URL. This
reference lists the current model IDs and endpoints for the major vendors,
taken from each vendor's public documentation. **Model IDs change frequently, so
always verify against the vendor's live model list before configuring a
model.**
Auth is `Authorization: Bearer ` unless noted. Claude models are reachable
from OpenAI-compatible clients through the AI-FW Anthropic facade.
## OpenAI
Base URL: `https://api.openai.com/v1`
- Flagship: `gpt-5.6-sol` (alias `gpt-5.6`), `gpt-5.6-terra`, `gpt-5.6-luna`
- Specialized: `gpt-5.6-cyber`, `gpt-daybreak-red-latest`, `gpt-daybreak-blue-latest`, `gpt-image-2`, `gpt-realtime-2.1`, `gpt-realtime-2.1-mini`, `gpt-realtime-2`, `gpt-realtime-translate`, `gpt-realtime-1.5`, `gpt-4o-mini-tts`, `gpt-transcribe`, `gpt-live-transcribe`, `gpt-realtime-whisper`, `gpt-4o-transcribe`, `gpt-4o-mini-transcribe`
- Stable families (verify): `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `o3`, `o4-mini`, `text-embedding-3-small`, `text-embedding-3-large`
```bash
curl https://api.openai.com/v1/chat/completions -H "Authorization: Bearer $OPENAI_KEY" \
-d '{"model":"gpt-5.6-luna","messages":[{"role":"user","content":"Hi"}]}'
```
## Azure OpenAI / Microsoft Foundry
Base URL: `https://.openai.azure.com/openai/v1` (deployment-based, `api-version` required). Auth: `api-key` header.
- `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-chat-latest`, `gpt-5.5`, `gpt-5.4`, `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-5.4-pro`, `gpt-5.3-chat`, `gpt-5.3-codex`, `gpt-5.2`, `gpt-5.2-codex`, `gpt-5.1`, `gpt-5.1-chat`, `gpt-5.1-codex`, `gpt-5.1-codex-mini`, `gpt-5`, `gpt-5-mini`, `gpt-5-nano`, `sora-2`, `gpt-4.1`, `gpt-4.1-mini`, `gpt-4.1-nano`, `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `codex-mini`, `computer-use-preview`
- Partner models sold by Azure: Grok (`grok-4.6`, `grok-4.3`, `grok-4-20-reasoning`, `grok-4-20-non-reasoning`, `grok-4.1-fast-reasoning`, `grok-4.1-fast-non-reasoning`, `grok-4`, `grok-code-fast-1`), Mistral (`Mistral-Large-3`, `mistral-medium-3-5`, `mistral-document-ai-2512`, `mistral-ocr-4-0`), Moonshot (`Kimi-K2.7-Code`, `Kimi-K2.6`, `Kimi-K2.5`), `MAI-Image-2.5`
## Claude (Anthropic)
Native base URL: `https://api.anthropic.com` (Anthropic `/v1/messages` format; auth `x-api-key`). Reachable from OpenAI-compatible clients via the AI-FW facade.
- Current: `claude-fable-5-1`, `claude-opus-5`, `claude-sonnet-5`, `claude-haiku-4-5` (dated `claude-haiku-4-5-20251001`)
- Legacy (still available): `claude-opus-4-8`, `claude-opus-4-7`, `claude-opus-4-6`, `claude-opus-4-5`, `claude-sonnet-4-6`, `claude-sonnet-4-5`, `claude-fable-5`
- Bedrock IDs: `anthropic.claude-opus-5`, `anthropic.claude-sonnet-5`, `anthropic.claude-haiku-4-5`
## X (Grok)
Base URL: `https://api.x.ai/v1`
- `grok-4.6`, `grok-4.5`, `grok-4.3`, `grok-4.20-0309-reasoning`, `grok-4.20-0309-non-reasoning`, `grok-4.20-multi-agent-0309`, `grok-build-0.1`
- Media: `grok-imagine-image-2.0`, `grok-imagine-image`, `grok-imagine-image-quality`, `grok-imagine-video-1.5`, `grok-imagine-video`, `grok-voice-think-fast-2.0`
- Aliases: `` and `-latest` track the newest release.
## Meta (Llama)
Meta has no first-party OpenAI-compatible API; Llama models are served by hosting partners:
- **Groq**: `https://api.groq.com/openai/v1` with `llama-3.1-8b-instant`, `llama-3.3-70b-versatile`
- **AWS Bedrock**: `meta.llama3-*`, `meta.llama3.1-*`, `meta.llama3.2-*`, `meta.llama3.3-*`, `meta.llama4-*`
- **Together / Fireworks / DeepInfra / SambaNova**: `meta-llama/llama-3.3-70b-instruct` and other Llama 3.x/4 families (verify IDs)
## AWS Bedrock
OpenAI-compatible via the Bedrock Runtime (`InvokeModel`/`Converse`) with `accept: application/json`; model IDs use provider prefixes.
- Amazon: `amazon.nova-*` (Nova 2 Lite, Nova 2 Sonic, Nova Pro, Nova Premier, Nova Lite, Nova Micro, Nova Canvas, Nova Reel, Nova Sonic), `amazon.titan-*` (text embeddings, multimodal embeddings, image)
- Anthropic: `anthropic.claude-*` (5.x, 4.x, 3.x as listed above)
- Meta: `meta.llama*`; Mistral: `mistral.*`; Cohere: `cohere.command-*`, `cohere.embed-*`, `cohere.rerank-*`; AI21: `ai21.jamba-*`; DeepSeek: `deepseek.*`; xAI: `grok-4.6`, `grok-4.3`; Z.AI: `zai.glm-*`; OpenAI: `openai.gpt-*`; Qwen: `qwen.*`; NVIDIA, MiniMax, Moonshot, Stability, Writer, TwelveLabs also available.
## DeepSeek
Base URL (OpenAI format): `https://api.deepseek.com` (also `/anthropic` for Anthropic format)
- `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`
- On Bedrock: `deepseek-v3-2`, `deepseek-v3-1`, `deepseek-r1`
## Mistral
Base URL: `https://api.mistral.ai/v1`
- `mistral-medium-3-5-26-04`, `mistral-small-4-0-26-03`, `mistral-large-3-25-12`, `ministral-3-14b-25-12`, `ministral-3-8b-25-12`, `ministral-3-3b-25-12`, `codestral-25-08`, `mistral-embed-23-12`, `codestral-embed-25-05`, `voxtral-mini-transcribe-26-02`, `voxtral-small-25-07`, `ocr-4-1`, `ocr-4-0`, `mistral-moderation-26-03`, `shieldstral-1-0`, `zai-glm-5-2` (Z.ai GLM hosted by Mistral)
## Google Gemini
OpenAI-compatible base URL: `https://generativelanguage.googleapis.com/v1beta/openai/`
- `gemini-3.7-flash`, `gemini-3.6-flash`, `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-flash-lite`, `gemini-3.1-pro-preview`, `gemini-3-flash-preview`, `gemini-3.5-transcribe`, `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-2.5-flash-lite`
- Media: `gemini-3.1-flash-image` (Nano Banana 2), `gemini-2.5-flash-image` (Nano Banana), `veo-3.1-generate-preview`, `lyria-3-pro-preview`
- Embeddings: `gemini-embedding-2-preview`, `gemini-embedding-001`
## Groq
Base URL: `https://api.groq.com/openai/v1`
- Production: `llama-3.1-8b-instant`, `llama-3.3-70b-versatile`, `openai/gpt-oss-120b`, `openai/gpt-oss-20b`, `whisper-large-v3`, `whisper-large-v3-turbo`; systems: `groq/compound`, `groq/compound-mini`
- Preview: `qwen/qwen3.6-27b`, `qwen/qwen3.8-27b`, `openai/gpt-oss-safeguard-20b`, `minimaxai/minimax-m2.7`, `canopylabs/orpheus-v1-english`, `meta-llama/llama-prompt-guard-2-*`
- Live list: `GET https://api.groq.com/openai/v1/models`
## Qwen (Alibaba Cloud Model Studio / DashScope)
OpenAI-compatible base URL: `https://dashscope.aliyuncs.com/compatible-mode/v1`
- Text: `qwen3.8-max`, `qwen3.7-plus`, `qwen3.8-flash`
- Omni/vision: `qwen3.5-omni-plus`, `qwen3.5-omni-plus-realtime`
- Image/video: `qwen-image-3.0-pro`, `wan2.7-image-pro`, `wan3.0-video`, `happyhorse-1.1-t2v`, `happyhorse-1.1-i2v`, `happyhorse-1.1-r2v`
- Audio: `qwen-audio-3.0-tts-plus`, `qwen-audio-3.0-asr-flash-streaming`, `qwen-audio-3.0-asr-flash-filetrans`, `qwen-audio-3.0-realtime-plus`
- Embeddings: `text-embedding-v4`, `qwen3.7-text-embedding`, `tongyi-embedding-vision-plus`, `qwen3-rerank`
- Third-party on Model Studio: `deepseek-v4-pro`, `deepseek-v4-flash`, `kimi-k2.7-code`, `kimi-k3`, `glm-5.2`, `ZHIPU/GLM-5.3`, `MiniMax-M2.5`
- Also hosted on Groq (`qwen/qwen3.6-27b`, `qwen/qwen3.8-27b`) and Bedrock (`qwen.*`)
## Z.ai (GLM)
OpenAI-compatible base URL: `https://open.bigmodel.cn/api/paas/v4` (Z.ai API)
- Text: `glm-5.3`, `glm-5.3-flash`, `glm-5.2`, `glm-5.1`, `glm-5`, `glm-4.7`, `glm-4.7-flash`, `glm-4.7-flashx`, `glm-4.6`, `glm-4.5`, `glm-4.5-x`, `glm-4.5-air`, `glm-4.5-airx`, `glm-4.5-flash`, `glm-4-32b-0414-128k`
- Vision: `glm-4.6v`, `glm-4.6v-flash`, `glm-4.6v-flashx`, `glm-4.5v`, `autoglm-phone-multilingual`
- OCR/audio: `glm-ocr`, `glm-asr-2512`; media: `glm-image`, `cogview-4`, `cogvideox-3`
- Also on Bedrock (`zai.glm-*`) and Mistral (`zai-glm-5-2`)
## Other OpenAI-compatible hosts (verify IDs)
| Vendor | Base URL | Representative models |
|---|---|---|
| OpenRouter | `https://openrouter.ai/api/v1` | 300+ models: `openai/gpt-5.6-luna`, `anthropic/claude-sonnet-5`, `google/gemini-3.7-flash`, `meta-llama/llama-3.3-70b-instruct`, `x-ai/grok-4.6`, `deepseek/deepseek-chat`, `qwen/qwen3-*` |
| Together AI | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo`, `Qwen/Qwen2.5-72B-Instruct`, `deepseek-ai/DeepSeek-V3` |
| Fireworks AI | `https://api.fireworks.ai/inference/v1` | `accounts/fireworks/models/llama-v3p3-70b-instruct`, `qwen2p5-72b-instruct`, `deepseek-v3` |
| DeepInfra | `https://api.deepinfra.com/v1/openai` | `meta-llama/Llama-3.3-70B-Instruct`, `Qwen/Qwen2.5-72B-Instruct`, `deepseek-ai/DeepSeek-V3` |
| Cerebras | `https://api.cerebras.ai/v1` | `llama-3.3-70b`, `llama3.1-8b`, `gpt-oss-120b` |
| NVIDIA NIM | `https://integrate.api.nvidia.com/v1` | `meta/llama-3.3-70b-instruct`, `nvidia/llama-3.1-nemotron-*`, `deepseek-ai/deepseek-r1` |
| SambaNova | `https://api.sambanova.ai/v1` | `Meta-Llama-3.3-70B-Instruct`, `Qwen/Qwen3-235B-A22B` |
| Novita AI | `https://api.novita.ai/v3/openai` | `meta-llama/llama-3.3-70b-instruct`, `qwen/qwen-2.5-72b-instruct` |
| Perplexity | `https://api.perplexity.ai` | `sonar-pro`, `sonar` |
| Ollama (local) | `http://localhost:11434/v1` | any pulled tag, e.g. `llama3.3`, `qwen3` |
| LM Studio (local) | `http://localhost:1234/v1` | any loaded model |
| vLLM (self-host) | `http://:8000/v1` | any served model |
Register any of these backend URLs in the Model Inventory with the matching
model ID and provider key, then route traffic through the gateway. See
[Configure models & keys](/docs/how-to/configure-models-keys).
---
# Community Edition license
Section: Community Edition · URL: https://aifw.io/docs/community-edition/license
The AI-FW Community Edition is free with a published monthly allowance of 100,000,000 tokens, enforced by the gateway against a machine-readable license document.
# Community Edition license
The AI-FW Community Edition is free to use. This page is the full terms
reference; the gateway enforces the allowance it reads from the published
license documents, with this page as the authoritative source.
## Monthly token allowance
The Community Edition may process **100,000,000 tokens per month**, measured
over a rolling 30-day window. The count is the total of prompt and completion
tokens for every allowed transaction the gateway processes while running the
Community Edition.
The allowance is published so it can be updated without a gateway release. The
gateway re-reads the license documents hourly and applies the current value. If
the published sources are unreachable, the gateway keeps the last known-good
cap, and falls back to a conservative default until a fetch succeeds.
## What counts against the allowance
- Prompt tokens for every allowed request, before or after any gateway
processing.
- Completion tokens for every allowed response, streamed or not.
- Tokens consumed by tool calls and their results within a request.
Requests that fail authentication or inspection are not processed, so they do
not consume tokens. Tokens are not refunded when a response is interrupted.
## Enforcement
Enforcement is gateway-side and requires no license key, account, or phone
home beyond reading the public license documents.
- Under the allowance, requests flow normally.
- Above the allowance, a 7-day grace period still passes requests while the
gateway flags the overage on the Dashboard and in response headers.
- Past the grace period, the gateway returns `429 license_quota_exceeded`
until usage falls back inside the rolling 30-day window.
The gateway never routes traffic around itself to bypass the allowance. If you
see license-quota errors in normal use, check the token meter on the Dashboard
to confirm where usage stands within the window.
## Updates to the allowance
The license documents carry a version number. When AI-FW changes the
allowance or the terms, the version increments and the gateway logs the
change. The current version is always visible at the license URLs below.
- License document (machine-readable):
https://aifw.io/community-edition/license.json
- License overview (human-readable):
https://aifw.io/community-edition/license
## Scaling beyond the Community Edition
The Community Edition is free. For higher limits, guaranteed throughput, or
enterprise features, [contact sales](https://aifw.io/contact). The full terms
are published by AI-FW, a product of
[Securetron.net](https://securetron.net).
---
# What is AI-FW?
Section: Getting Started · URL: https://aifw.io/docs/getting-started/what-is-aifw
An overview of the AI-FW gateway, what it does, how it fits into your stack, and why it is fail-closed by design.
AI-FW is an **AI firewall and governance gateway** that sits between your AI
agents and the large-language-model backends they call. Every prompt and response
flows through the gateway, where it is inspected, scored against policy, routed to
the right model, and audited, all in real time, with no agent-side code changes.
## What it does
| Capability | What it means for you |
|---|---|
| **Prompt inspection** | Jailbreak and prompt-injection attempts are blocked, PII is masked, and your own rules are enforced before anything reaches the model. |
| **Response inspection** | Toxicity, leaked credentials, and data exfiltration (system prompts, internal IPs, private keys) are caught on the way back. |
| **Model routing** | Each request is routed to the right model and backend, with per-model API keys held server-side. |
| **Semantic intent analysis** | Traffic is scored against natural-language policies, not just matched against keywords. |
| **Identity & access** | Agents and users authenticate with API keys, JWT, mutual TLS, or Kerberos; RBAC scopes the admin surface. |
| **Risk & audit** | Rolling risk scores per user, agent, and IP; automatic blocking; a metadata-only audit log. |
| **Reliability & caching** | Per-model retries, failover, and load distribution, plus an opt-in completion cache. |
## Two surfaces
AI-FW brings two integrated surfaces together:
**AI-FW, the AI firewall & governance gateway.** The traffic layer described above:
an OpenAI-compatible and Anthropic-compatible endpoint that agents point at instead
of the model provider directly.
**Agent Trust, the agent registry and A2A protocol.** A registry where AI agents
onboard with an identity, discover services, and request PKI operations,
certificate issuance, renewal, and revocation, over the Agent2Agent (A2A)
task-based protocol. Agents authenticate with certificates (mTLS), and the same
governance, risk, and audit machinery applies to them.
## How it fits into your stack
```
Your agents & tools (OpenAI SDKs, Claude Code, Cursor, MCP, custom agents)
│
▼
┌───────────────────────┐
│ AI-FW gateway │ inspect → govern → route → audit
└───────────────────────┘
│
▼
LLM backends (OpenAI, DeepSeek, Anthropic, Mistral, Groq,
OpenRouter, Ollama, vLLM, LM Studio, …)
```
Clients keep using their normal SDKs, the only change is the base URL. The gateway
speaks the **OpenAI chat-completions protocol** and the **Anthropic Messages
protocol**, and translates between them, so Claude-native tools can use OpenAI-style
models and vice-versa.
## The operating philosophy
AI-FW is **fail-closed by design**:
- A scanner error or missing configuration **blocks** the request, it never
forwards traffic unfiltered.
- The semantic tier blocks on any scoring error or timeout rather than passing a
request through unjudged.
- New models start **disabled** until an administrator explicitly enables them.
- Raw prompt and response content is **never persisted**, the audit log keeps
metadata and truncated previews only.
That means an outage or misconfiguration can never silently turn your AI pipeline
into an ungoverned one.
Ready to try it? Follow the [quick start](/docs/getting-started/quick-start) to get
a gateway running in minutes.
---
# Quick start
Section: Getting Started · URL: https://aifw.io/docs/getting-started/quick-start
Get an AI-FW gateway running in minutes, start the service, configure a model, and inspect your first prompt.
This guide gets an AI-FW gateway running locally and routing real traffic
through the inspection pipeline. It takes about five minutes.
## 1. Start the gateway
The gateway is a single service with an embedded admin UI and API. Start it with
The gateway ships as a **Docker image** and runs as a single service with an
embedded admin UI and API. Start it with:
```bash
docker run -d --name aifw-gateway -p 443:443 \
-e ConnectionStrings__Default=Host=your-postgres;Database=aifw;Username=...;Password=... \
aifw/gateway:latest
```
The gateway listens on port **443** by default. If 443 is already in use on the
host, map a different host port instead, for example `-p 8443:443`.
If you don't have PostgreSQL handy, the gateway can run with its built-in
in-memory store for evaluation, restart and configuration changes are lost, but
everything else works.
## 2. Log in
Open `https://fqdn.aifw.io` (or the port you mapped) and sign in with the
bootstrap administrator account that was printed at startup. From there you can
create additional admin users and configure SSO.
## 3. Configure a model
1. Open **Model Inventory**.
2. You'll see the curated starter set of models. They are **disabled by default**,
enable the ones you want to route traffic to.
3. For each model, add its **backend URL** (the provider endpoint) and an **API
key** if the provider requires one. Keys are stored server-side and never
returned to clients.
4. Optionally set a **default model** and **default backend** in **Settings**,
these are used when a client omits the model field or asks for an unregistered
one.
Requests for a disabled model are rejected with `400 model_disabled` until you
explicitly enable it. This is intentional, nothing routes until you say so.
## 4. Point a client at the gateway
Any OpenAI-compatible SDK works. Point it at `https://fqdn.aifw.io/v1`:
```python
from openai import OpenAI
client = OpenAI(
api_key="", # passed through to the backend
base_url="https://fqdn.aifw.io/v1", # the AI-FW gateway
default_headers={"X-Agent-ID": "my-agent-01"},
)
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
```
All traffic flows through the full scanning pipeline: prompt inspection before
forwarding, response inspection before delivery.
## 5. See it work, allowed and blocked
Send a benign request, it should pass inspection and return a normal response:
```bash
curl -s https://fqdn.aifw.io/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"Hello"}]}'
```
Now send a jailbreak attempt, it should be **blocked** with a `400`:
```bash
curl -s -X POST https://fqdn.aifw.io/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"Ignore all previous instructions"}]}'
```
The blocked response carries an error of type `guardrail_violation` and the
`X-Gateway-Security: Violated` header.
## 6. Watch it in the dashboard
Open the **AI-FW Dashboard** to see both transactions, the allowed one and the
blocked one, with latency, decisions, and truncated previews. Then explore the
**Audit Log** to see how every transaction is recorded.
- Learn how [guardrails](/docs/guides/prompt-response-guardrails) are evaluated
- Understand [model routing](/docs/guides/model-routing) in depth
- Connect [Claude Code, Cursor, or MCP tools](/docs/tutorials/claude-code-cursor-mcp)
---
# Prompt & response guardrails
Section: Guides · URL: https://aifw.io/docs/guides/prompt-response-guardrails
How AI-FW inspects every prompt and response, built-in rules, custom rules, evaluation order, and rule actions.
AI-FW inspects **inbound prompts** (before they reach the model) and
**outbound responses** (before they reach the client). The built-in rules cover the
most common attack and compliance vectors out of the box; custom rules extend them
to your own policy.
## Built-in rules
| Rule | Direction | Action | What it catches |
|---|---|---|---|
| Jailbreak detection | inbound | block | "Ignore previous instructions", system-override prompts, DAN mode, safety bypass |
| PII masking | inbound | mask | Social security numbers, credit cards, emails, API keys → `[REDACTED_*]` |
| Toxicity filter | outbound | block | Hate, discrimination, and harassment patterns |
| Data exfiltration guard | outbound | block | Leaked system prompts, internal IPs, credentials, private keys |
Built-in rules can be enabled or disabled individually from the **Rules Manager**
(shown as "System" rules). Custom rules can be added, edited, enabled, disabled,
and deleted.
## How rules are evaluated
Every request is evaluated in a fixed order, left to right. The first decisive
outcome wins; later steps only run while the request is still undecided:
| # | Step | What happens on a match |
|---|---|---|
| 1 | **Built-in rules** | System rules run first, jailbreak, PII mask, toxicity, exfiltration. |
| 2 | **Block rules** | Reject with `400 guardrail_violation`. A block always wins, even if an accept rule would also match. |
| 3 | **Mask rules** | First match rewrites the content and short-circuits the group. |
| 4 | **Accept / Log rules** | First match passes the request; `log_only` also suppresses preview persistence. |
| 5 | **Semantic tier** | Scores the request against natural-language policies, see the [semantic analysis guide](/docs/guides/semantic-intent-analysis). |
| 6 | **Implied rule** | Terminal step: apply the configured default action, allow, or block (deny-by-default). |
Deny-overrides are absolute: if a block rule matches, nothing else runs, even an
accept rule that would also match. Only text-type rules (regex / keyword / token
limits / model / user) can match in the block group.
## Rule actions
Every custom rule has one of four actions:
| Action | Effect |
|---|---|
| `block` | Reject with `400 guardrail_violation` (always wins over other actions) |
| `mask` | Rewrite matching text with a replacement value before forwarding |
| `log_only` | Accept & no log, pass through; the match is recorded but no content is persisted |
| `accept` | Accept & log, pass through; prompt and response previews are stored in the audit trail |
## Per-rule options
- **Skip semantic check**, when enabled and the rule matches, the request bypasses
the semantic intent tier.
- **Source IP / subnet**, the rule applies only when the client's IP is inside the
CIDR (e.g. `10.0.0.0/8`, `192.168.1.5`).
- **Destination IP / subnet**, the rule applies only when the resolved backend IP
of the model is inside the CIDR.
- **Key mode & streaming override**, a rule can pin the upstream key behavior and
streaming mode for requests it matches (see the [identity guide](/docs/guides/identity-access)).
## What a blocked request looks like
Blocked transactions return `HTTP 400` with a structured error:
```json
HTTP 400
{
"error": {
"message": "The transaction was intercepted and blocked by corporate AI security policy.",
"type": "guardrail_violation",
"code": "prompt_injection_detected",
"details": {
"policy_id": "BUILTIN-JAILBREAK",
"timestamp": "2025-01-15T10:00:00Z"
}
}
}
```
plus the `X-Gateway-Security: Violated` response header.
## Structure-preserving rewrites
When PII masking or the default-model substitution rewrites a request, the rewrite
is **structure-preserving**: replacements are applied to JSON string values and the
`model` field only. Tool calls, function calls, streaming options, and every other
field pass through byte-identical, so `assistant(tool_calls)` →
`tool(tool_call_id)` sequences survive untouched to the backend.
## Fail-open options (opt-in escape hatches)
The gateway is **fail-closed by default**: a scanner exception or missing
configuration blocks the request rather than forwarding it unguarded. For teams
that must prioritize availability over safety in specific situations, Settings
-> **Fail-open Options** exposes three admin toggles, **all default OFF**:
| Toggle | When ON |
|---|---|
| Scanner exceptions | A prompt-scanner exception allows the request with an audit entry instead of blocking it |
| Response-scan exceptions | A response-scanner exception forwards the response unscanned (never cached) |
| Compression gate | If the semantic gate is unavailable, compression is applied ungated instead of keeping the original (a below-threshold drift is still rejected) |
Every fail-open event is logged with a warning and audited, so you can always
see when the safety net was lifted.
- [Semantic intent analysis](/docs/guides/semantic-intent-analysis), scoring traffic by meaning
- [Create rules with AI](/docs/how-to/create-rules-with-ai), draft rules from an objective
---
# Model routing & registry
Section: Guides · URL: https://aifw.io/docs/guides/model-routing
How AI-FW decides which model handles a request, the model inventory, default model and backend, strict mode, and conditional routing rules.
The gateway makes two independent decisions for every request: **which model** to
use, and **where** to send it. Both are governed by the model registry, a central
inventory of every model your organization is allowed to call.
## The model registry
Models are not read from config files. They live in the **Model Inventory**:
- A curated starter set is seeded on first run, each with its provider and backend
URL. They ship **disabled**, enable the ones you want to route traffic to.
- **Add custom model**, model ID, provider, backend URL, optional API key.
- **Edit**, update provider, backend URL, or API key of any registered model.
- **Register**, promote a discovered (shadow) model to registered.
- **Delete**, remove a custom model.
Each model also carries a **protocol** (`openai` or `anthropic`). Model IDs that
start with `claude-` default to the Anthropic protocol automatically; you can
change it per model.
## Default model & default backend
| Setting | Question it answers | Where |
|---|---|---|
| **Default model** | Client omitted the `model` field, which model do we use? | Settings → Default Model |
| **Default backend** | The resolved model has no backend URL of its own, where do we send it? | Settings → Default Backend |
The Settings page groups everything into tabs: General, Access, Identity,
Model, Routing, Guardrails & Inspection, Performance, Audit, Export, and
Integrations. The guides reference them by name, for example
**Settings -> Default Model** or **Settings -> Access Control**.
The full decision tree:
```
1. Client sends model "X"?
no → insert the DEFAULT MODEL
2. Is model X registered with its own backend URL?
yes → route to model X's backend, inject model X's key
no → route to the DEFAULT BACKEND (inject default key if set)
3. No backend anywhere?
→ 400 backend_not_configured
```
## Strict mode
The **"Reject Unregistered Models"** toggle in Settings → Default Model controls
what happens when a client asks for a model that isn't in the registry:
- **OFF (default):** unregistered models fall back to the default backend.
- **ON:** requests for unregistered models are rejected with `400
model_not_registered`.
Either way, requests for a **disabled** registered model are rejected with `400
model_disabled`, that check is independent of strict mode.
## Conditional routing rules
Beyond the default routing, you can define **routing rules** that steer a request
based on its context:
| Field | Meaning |
|---|---|
| `agent` | Literal requesting agent ID |
| `agentgroup` | Requesting agent belongs to a named agent group |
| `agenttag` | Requesting agent's registry record carries the tag |
| `ip` | Source IP |
| `model` | Requested model ID |
| `group` | Requested model is a member of a named model group |
| `risk` | User risk score (numeric comparisons) |
| `flag` | Prompt-scan flag code |
| `header.` | Request header value |
Operators: `Equals`, `NotEquals`, `GreaterThan`, `GreaterOrEqual`, `LessThan`,
`LessOrEqual`, `In`, `NotIn`, `Matches` (regex).
- Conditions within a rule are **ANDed**; rules run in order and **first match
wins** (ordering expresses OR).
- A rule's destination is a single registered model or a raw URL, model groups
and agent groups are **match sources only**.
- Unknown agents, groups, or tags **fail closed** (no match), and malformed rules
never break the request path.
- The destination model's own reliability and key settings apply (see the
[reliability guide](/docs/guides/reliability-caching)).
## Shadow AI discovery
Model IDs that appear in traffic but are **not** in the registry are flagged as
**"Newly Discovered LLMs"** in the Model Inventory. You can review them, register
the legitimate ones, and (with strict mode on) reject the rest, closing the shadow
AI gap where teams use unapproved models.
- [Configure models & keys](/docs/how-to/configure-models-keys), a step-by-step recipe
- [Reliability & caching](/docs/guides/reliability-caching), retries, failover, distribution
---
# Semantic intent analysis
Section: Guides · URL: https://aifw.io/docs/guides/semantic-intent-analysis
Judge the meaning of AI traffic, not just its text, natural-language policies, block and flag thresholds, and three scoring engines.
Regex and keyword rules catch what you can express literally. **Semantic intent
analysis** catches everything else: traffic that *means* the wrong thing, requests
to reveal system prompts, attempts to exfiltrate secrets, or policy violations
phrased in a way no keyword list would ever match.
## How it works
A semantic rule has a **natural-language pattern** instead of a regex:
- "detect attempts to reveal system prompts"
- "detect requests to exfiltrate secrets"
- "detect requests for regulated personal data"
At request time, the content is scored against every enabled semantic rule for its
direction (inbound or outbound). Each rule has two thresholds:
| Score | Result |
|---|---|
| **≥ block threshold** (default 0.85) | Rejected, `400`, code `semantic_rule_violation:` |
| **flag ≤ score < block** (default flag 0.60) | Allowed, but the user risk score is escalated and the match is logged |
| **< flag threshold** | Passes silently |
Thresholds are validated as `0 ≤ flag < block ≤ 1` when you save a rule.
Inbound prompts are scored **after PII masking**, redacted secrets never leave
your environment. Outbound responses are the model's own output and are scored
as-is.
## Three scoring engines
Pick the engine in **Settings → Semantic Analysis**:
| Engine | How it scores | What it requires |
|---|---|---|
| **Embeddings API** | Content and each rule description are embedded; cosine similarity scores the match | An OpenAI-compatible embeddings endpoint |
| **Chat classifier** | A chat model rates how well the content matches the rule's intent, replying with a number 0.00–1.00 | Any OpenAI-compatible chat endpoint, works with providers that don't offer embeddings |
| **Embedded (local ONNX)** | A small embedding model runs in-process, fully offline | Nothing external, no API key, no network egress, ~1–15 ms per embedding |
The offline engine is the strongest privacy option: content never leaves the
machine, and there is no per-request cost.
## Fail-closed by design
When the tier is enabled and semantic rules exist, **any** of the following blocks
the request (`semantic_error` / `semantic_unavailable`), the request is never
forwarded unjudged:
- endpoint unset, or rule descriptions never embedded
- scoring calls error or time out
- invalid endpoints (validation/DNS/connect failures)
- empty, mismatched, or non-finite embedding vectors
- offline model files missing or failing to load
- unparseable classifier scores
The endpoint is validated server-side too: HTTPS is required for non-loopback
hosts, private and link-local ranges are rejected, redirects are disabled, and the
connection is pinned to the validated IP at connect time.
## Performance
The tier is engineered to stay out of the way:
- Chat-classifier calls run with **bounded concurrency** instead of serially
(concurrency default 3).
- A **bounded content-embedding cache** skips the network embed for repeated
prompts.
- Long content (>4K characters) is scored as **head + tail segments** (max cosine)
so intent cannot be smuggled past a truncation point.
## Enablement
The tier is **off by default**. Turn it on in **Settings → Semantic Analysis**:
choose the scoring backend, the endpoint and model (or none for the offline
engine), per-request timeouts, and the fail-closed toggle (default on). The API key
is read from server configuration, it is never stored in the database or returned
to clients.
- [Prompt & response guardrails](/docs/guides/prompt-response-guardrails), where the semantic tier fits in the evaluation order
- [Risk profiles & auto-block](/docs/guides/risk-profiles), how flags escalate risk scores
---
# Identity & access
Section: Guides · URL: https://aifw.io/docs/guides/identity-access
Authentication modes, API keys, key precedence, streaming modes, mTLS, Kerberos, OIDC SSO, SCIM, and admin roles.
AI-FW authenticates two kinds of callers: **human users** (admins, via the
web UI) and **machine agents** (via the gateway API). This guide covers both, plus
the role model that scopes the admin surface.
## Gateway authentication modes
**Settings → Access Control → Authentication Mode** controls what the gateway
requires from API callers:
| Mode | Behavior |
|---|---|
| `required` | No Bearer token → `401`. Every caller must authenticate. |
| `optional` | Anonymous callers allowed, but any token sent must be valid. |
| `passthrough` | Any non-empty Bearer key is accepted on presence alone and forwarded verbatim upstream. |
The gateway accepts `Authorization: Bearer` tokens and, for Anthropic-style
clients, the `x-api-key` header.
## Upstream key precedence
Which API key reaches the model backend is a separate decision, resolved as
**rule → model → global** (most specific wins):
1. **Rule-level**, a key mode on a routing/guardrail rule applies when that rule
matches.
2. **Model-level**, the key mode set per model in the Model Inventory.
3. **Global**, Settings → Access Control → Key Mode, default `model`.
| Effective mode | Upstream `Authorization` header |
|---|---|
| `model` | The model's registered key wins; the client's token is used only when the model has no key |
| `client` | The client's Bearer key is forwarded verbatim; no client key → no auth header |
With key mode `client`, the gateway no longer guarantees that only approved keys
reach the backend, callers supply their own. Enable deliberately, e.g. together
with `passthrough` auth for bring-your-own-key scenarios.
## Streaming modes
Requests with `stream: true` are handled per the effective streaming mode
(**rule → model → global**, same chain as key mode):
| Mode | Behavior |
|---|---|
| `buffered` (default) | The full response is buffered and scanned before **any** token is delivered, strictest. A violation returns `HTTP 400` before delivery. |
| `stream` | Tokens are relayed live through a sliding-window scanner; a mid-stream violation aborts the connection and terminates the SSE stream with a `guardrail_violation` event. |
Requests without `stream: true` are always buffered regardless of mode.
## Agent authentication
Agents can prove identity in four ways:
- **API keys**, admin-issued keys created in the Agent API Keys page. Each key
carries a unique key ID and is used for identity, cache isolation, and risk.
- **JWT**, Bearer JWTs validated against your configured signing secret; the
`sub` claim becomes the agent identity.
- **mTLS (RFC 8705)**, client certificates on a separate TLS listener; the
certificate identity (SPIFFE URI → subject key identifier → common name) maps to
the agent identity.
- **Kerberos**, domain-joined Windows service-account agents authenticate with
`Authorization: Negotiate` and can enroll for an mTLS client certificate. UPN and
source-subnet allowlists are configurable.
## Admin roles & the web UI
The admin surface is protected by role-based access control with four roles:
| Role | Access |
|---|---|
| **Admin** | Full access to everything |
| **Read-Only** | View the entire UI; all mutations blocked, controls greyed out |
| **AI-FW Admin** | Manage the AI firewall surface only, rules, models, settings, audit logs |
| **Agent-Trust Admin** | Manage the agent inventory and trust, no access to the AI firewall section |
Local users are managed on the User Management page, with per-user roles and
password resets.
## SSO & provisioning
**OIDC SSO**, configure a generic OpenID Connect provider (authority, client ID,
secret, scopes). Sign-in mode can be `local`, `oidc`, or `both`. A runtime role map
translates identity-provider claims into AI-FW roles; unknown or absent claims
map to **Read-Only** by default.
**SCIM 2.0 provisioning**, expose user and group provisioning to your identity
provider (Entra ID, Okta, …) with a bearer secret. `userName` maps to the local
username, `active` to the enabled state, and a group whose display name matches a
role assigns that role to its members. SCIM-provisioned users are passwordless,
they authenticate via SSO.
- [Configure models & keys](/docs/how-to/configure-models-keys), key handling per model
- [Risk profiles & auto-block](/docs/guides/risk-profiles), risk scoring of identities
---
# Risk profiles & auto-block
Section: Guides · URL: https://aifw.io/docs/guides/risk-profiles
Rolling risk scores for users, agents, and IPs, with automatic blocking, category risk cards, and a metadata-only audit trail.
AI-FW tracks **risk** per identity, users, agents, and source IPs, so risky
behavior escalates automatically and can block traffic before it causes damage.
## How risk scores work
- Risk increases with **violations** (blocked requests, denied authorizations,
failed tasks, CA errors) and with **semantic flags**.
- Scores **decay over time** (48-hour exponential decay, capped at 100) so a past
mistake doesn't permanently label an identity.
- **Only verified identities drive escalation**: a token claim (`sub`) escalates;
client-supplied headers are never trusted for risk. The UI shows an **Identity**
badge, Verified (token claim) vs Header (client-supplied).
## Three risk surfaces
| Surface | What it shows |
|---|---|
| **User risk leaderboard** | Risk score, semantic flags + blocks, violation count, first/last seen, per-user reset, and a drill-down into recent transactions. |
| **Risky agents** | Agent risk scores (block violations only, agent IDs are client-controlled, so semantic flags stay excluded), requests, violations, band, reset. |
| **Risky IP addresses** | Per-source-IP risk, created automatically on **blocks and semantic flags** (the least spoofable signal), with drill-down showing which users/agents came from that IP. |
Summary cards show Total Users, New AI Users, High-Risk Users, and Semantic Flags.
## Category risk cards
Risk is also broken down by category, each with all-time and 7-day detections,
last-detected time, top entities, and the most recent transactions:
- **Semantic Risk Analysis**, blocks vs flags per semantic rule
- **PII Risk**, SSNs, emails, API keys in traffic
- **PCI (Card) Risk**, payment card numbers
- **Health Data Risk**, medical terms and identifiers
Health-data detection is a keyword/identifier heuristic for visibility, not a
compliance attestation. Treat it as a signal, not a certification.
## Risk auto-block
The **Risk Auto-Block** card adds a count-based guard on top of score-based
blocking: an identity that accumulates **N violations within a rolling window of T
seconds** is automatically blocked for a cooldown period.
| Setting | Default |
|---|---|
| Master switch | on |
| Violations (N) | 5 |
| Window (T, seconds) | 300 |
| Block duration (seconds) | 3600 |
- When the block fires, an **alert is written to the audit log** with the
identity, violation count, and window.
- While blocked, the orchestrator rejects the identity's tasks.
- Violations older than the window drop out of the count; the switch fully
disables the guard.
- Score-based thresholds are unchanged: requires-auth at 10, block at 50, with a
1-hour cooldown.
## Audit trail
Every transaction lands in the **audit log** with metadata only: model, identity,
source IP, latency, decision, and truncated previews (raw content is never
persisted). The Activity log previews real streamed content as it flows, and the
**Events** tab records configuration and security changes,
including `risk_auto_blocked` alerts, so you always know who changed what, and
when.
- [Semantic intent analysis](/docs/guides/semantic-intent-analysis), flags that feed risk scores
- [Identity & access](/docs/guides/identity-access), verified vs header identities
---
# Reliability & caching
Section: Guides · URL: https://aifw.io/docs/guides/reliability-caching
Per-model resilience (retries, failover, distribution) and the opt-in completion cache, exact and semantic, with strict tenant isolation.
The gateway is the single point of failure for every AI call you make, so
reliability and latency matter. AI-FW gives each model its own resilience
policy, and an opt-in completion cache keeps repeated prompts fast and cheap.
## Per-model resilience
Configured in **Model Inventory → Edit → Resilience** (all off by default):
| Setting | Meaning |
|---|---|
| `RetryLimit` | Max retries against the *same* backend on retryable statuses (0 = no retries) |
| `RetryStatuses` | CSV of retryable HTTP codes; empty = 429 + all 5xx |
| `ReliabilityMode` | `off` (primary only), `failover` (ordered alternates), `distribute` (weighted random across the pool) |
| `PrimaryShare` | Relative weight of the primary backend in `distribute` |
| `AlternateEndpoints` | Raw URLs or registered model IDs (their own backend + key are used), each with a share |
Semantics:
- **Retries** also cover network failures and upstream timeouts; caller
cancellation is propagated, never retried.
- **Failover** tries alternates in order; the first acceptable response wins; if
everything fails, the last upstream error is passed through.
- **Distribute** picks one endpoint per request with probability share ÷ total.
- Off mode + retry limit 0 = exactly one upstream call.
- Streams are never cached mid-flight, retry/failover applies only up to the
first response byte.
- Every attempt is recorded in the transaction log (`RelayAttempts` /
`RelayFailover`), so you can see failover in action.
## The completion cache
An opt-in response cache for chat completions and messages traffic, in two tiers:
| Tier | What it does |
|---|---|
| **Exact** | Re-serves byte-identical prompts (default) |
| **Semantic** | Uses local embeddings to re-serve near-duplicate prompts (per-scope threshold, default 0.98) |
**Cache policy scopes**, caching is off until at least one scope exists. Each
scope is `global`, `model`, `modelgroup`, `agent`, or `agentgroup`, with an
enabled flag and an optional semantic threshold. Precedence for a request:
**agent > agentgroup > model > modelgroup > global** (first match wins).
**The cache key** is a hash of the endpoint path + the forwarded (post-mask) body,
plus the resolved destination backend when the toggle is on (default, so
different routed destinations never share cached completions). Auth headers are
never part of the key.
## Safety rules (both tiers)
- Never caches streams, flagged or PII-masked prompts, blocked/violating
responses, or non-JSON/non-2xx upstreams.
- The cache write is deferred to the response middleware, a guardrail-violating
body is never cached.
- Cached bodies flow through response inspection on replay, cached content never
bypasses scanning.
- `x-aifw-cache-refresh` bypasses the lookup but still allows the write.
## Isolation & privacy
- The cache identity comes **only from the authenticated principal**, never from
client-declared headers, a spoofed header cannot select another tenant's
policy or cached completions.
- **Semantic matches are strictly scoped**: an entry only matches a request with
the same endpoint, resolved model, destination backend, and authenticated
caller identity, near-duplicate prompts never cross models, endpoints, or
tenants.
- API-key principals get their own isolated cache namespace (keyed by the unique
key ID).
- **Anonymous caching is off by default.** When enabled, anonymous callers are
partitioned by source IP (best-effort) and never get the semantic tier.
- Total byte budget and entry cap are bounded so worst-case RAM stays controlled.
## Known limitations
- Caching applies to the native OpenAI chat path and the native Anthropic Messages
path; the two **facade** paths (OpenAI client → Claude backend, and Anthropic
client → OpenAI backend) bypass the cache.
- A 2xx response whose body is not JSON (or is empty) is never cached.
- [Model routing & registry](/docs/guides/model-routing), routing rules that interact with resilience
- [API reference](/docs/api-reference/openai-compatible-api), cache-related headers
---
# Prompt compression
Section: Guides · URL: https://aifw.io/docs/guides/prompt-compression
Cut upstream token spend with semantic-gated prompt compression, rule, aggressive, and LLM tiers that never change the meaning.
**Prompt compression** reduces upstream token spend by removing filler and
politeness from prompts before they are forwarded to the model. Every rewrite is
guarded by a **semantic safety gate**, so the meaning never changes, if the gate
cannot confirm the rewrite, the original prompt is sent unchanged.
## How it works
Compression runs on the request body before forwarding, per message (chat
completions and `/v1/messages`). Three tiers are available:
| Tier | What it does | Default |
|---|---|---|
| **Rule tier** | Strips a curated filler list (politeness, hedges, and filler like "please", "kindly", "i was wondering if", "basically") at word boundaries, collapses whitespace, cleans punctuation, normalizes the first letter. Content words are never touched. | On |
| **Aggressive tier** | Adds articles (a/an/the, never before quantity words like "a few"), `and` to comma (never between numbers, so "one hundred and fifty" is preserved), and semicolon/colon to comma between words (times like "10:30" and URL ports are untouched). | Opt-in |
| **LLM tier** | When a compressor model is configured, the prompt is rewritten by that model with a strict 16K-character cap and temperature 0. Any failure, timeout, or gate rejection degrades to the gated rule result, never a worse prompt. | Optional |
## The semantic safety gate
Compression is only applied when it is safe to do so:
- The compressed text is embedded and compared against the original with cosine
similarity; the rewrite must meet the configured threshold (**default 0.92**).
- If the score falls below the threshold, or the gate is unavailable, the
**original is kept** (fail-closed).
- Savings must also exceed the configured minimum (**default 5%**) or the rewrite
is skipped entirely.
- The aggressive tier is gated with its own slightly lower threshold (default
0.85), because removing articles lowers embedding similarity; if rejected, it
degrades to the filler-only result.
## Observability
Every applied compression is recorded in the transaction log with the mode used
(rule or LLM) and the estimated tokens saved. The dashboard shows a **Tokens
saved (24h)** total, so you can see exactly what the feature is returning.
## Configuration
- **Settings -> Prompt Compression**: global toggle (off by default), similarity
threshold, minimum savings, the aggressive-rules toggle, and an optional
compressor model.
- **Model Inventory**: per-model override (inherit / on / off), so compression can
be enabled or disabled for individual models.
- **Scope**: chat and `/v1/messages` only. Embeddings input is never compressed -
the vectors would change.
- [Reliability & caching](/docs/guides/reliability-caching), the other cost lever: cache hits skip upstream calls entirely
- [Semantic intent analysis](/docs/guides/semantic-intent-analysis), the same embedding machinery that powers the safety gate
---
# Claude inference hooks
Section: Guides · URL: https://aifw.io/docs/guides/claude-inference-hooks
Act as the AI security server for Anthropic's Inference Hooks - verified, scanned, and machine-actionable verdicts before inference proceeds.
AI-FW can act as the **AI security server** for Anthropic's Inference Hooks
(Claude Enterprise). Anthropic POSTs each governed prompt to the hook endpoint
and waits for an allow or deny verdict before inference proceeds, so Claude
Code and CoWork agents self-police against the same policy as everything else.
## How it works
- **Ingestion** - Anthropic POSTs each prompt frame to `POST /hooks/claude` as
a Standard Webhook.
- **Verification** - every request is signed (HMAC-SHA256 over
`{id}.{timestamp}.{body}` with the configured secret), with a 300-second
timestamp tolerance and constant-time comparison. Unsigned, bad, or replayed
requests are rejected. An optional previous secret is accepted during
rotation.
- **Parsing** - the transcript is extracted from text, tool-use, tool-result,
and thinking blocks. Unknown blocks and fields are skipped, and unknown event
types return `allow` for forward compatibility.
- **Decision** - the transcript is scanned with the same inbound rules as the
gateway:
| Outcome | Verdict |
|---|---|
| Pass | `allow` |
| Block | `deny` with the admin-configured deny reason |
| Flag | `allow`, logged |
| Mask rule | `allow` by default (hooks cannot rewrite); can be set to deny via Settings |
## Configuration
In **Settings -> Claude Inference Hooks**:
- **Enable toggle** for the hook endpoint
- **Signing secret** (plus an optional previous secret for rotation)
- **Mask action** - allow or deny for prompt frames that would be masked
- **Source-IP allowlist** (Anthropic egress ranges)
- **Deny message** shown to the agent when a prompt is denied
- **Endpoint URL**
The endpoint must be **HTTPS on port 443** on a publicly routable host (no
tunnels) and must respond within the **5-second verdict budget**; verdicts must
return HTTP 200.
## Audit
- Enabling, disabling, and secret changes are recorded as configuration-change
events in the Audit Events tab.
- Every hook event writes a transaction-log row keyed by request ID (deduplicated
on the webhook ID) and joinable to Anthropic's Activity Feed denial records via
the reference ID.
- Prompt content follows the same logging policy as gateway traffic.
- [Claude Code, Cursor & MCP tools](/docs/tutorials/claude-code-cursor-mcp)
- [Prompt & response guardrails](/docs/guides/prompt-response-guardrails)
---
# Observability & dashboard
Section: Guides · URL: https://aifw.io/docs/guides/observability-dashboard
The AI-FW dashboard - traffic stats, latency, guardrail snapshot, 24-hour metrics, and the live Activity view.
The AI-FW Dashboard gives operators a live picture of everything flowing through
the gateway. Raw content is never stored, so every metric here is built from
metadata and truncated previews.
## Traffic stats
The top stat cards show:
- **Allowed** and **blocked** request counts, plus the **block rate**
- **P95 latency** across recent traffic
- **Registered models**, **tracked agents**, and **tracked users**
## Traffic mix and latency
- A **traffic-mix donut** breaks traffic down by category.
- A **latency sparkline** shows response times over time.
- The **Guardrail Snapshot** is a stacked bar of block types, so you can see at a
glance what is being rejected and why.
## 24-hour metrics row
The 24h metrics row tracks the signals that matter day to day:
- **Semantic flags** and semantic blocks
- **Text-rule blocks** plus deny-by-default rejections
- **Unique callers** and **token volume**
- **Tokens saved (24h)** from prompt compression
## Charts
Below the stat cards, inline charts add depth:
- A **prompt-lifecycle Sankey** - requests flowing to allowed or blocked, then to
clean, flagged, text-rule, semantic, or default-policy outcomes
- A **7-day by 24-hour traffic heatmap**
- **Requests by hour** (stacked area) and **token usage by hour** (stacked bars)
- **Top models** and **top agents** by requests
- **Top models** and **top IPs** by tokens
## Live Activity view
The Activity log shows transactions as they happen, including **real streamed
content previews** for streaming requests. Every row carries the decision,
latency, model, identity, and truncated previews per the logging policy.
## Reports
Usage and traffic reports can be generated from the dashboard for review or
export, complementing the raw audit and access log APIs.
- [Audit logs & export](/docs/guides/audit-logs-export)
- [Risk profiles & auto-block](/docs/guides/risk-profiles)
---
# Audit logs & export
Section: Guides · URL: https://aifw.io/docs/guides/audit-logs-export
The metadata-only audit trail - logging policy, retention, audit events, and exporting to your SIEM via pull API or syslog.
Every transaction that flows through the gateway lands in a **metadata-only
audit trail**: model, identity, source IP, latency, decision, and truncated
previews. Raw content is never persisted unless your logging policy says
otherwise, and the trail can be exported to your SIEM for review.
## Logging policy
Settings -> Logging controls what the transaction log stores per direction,
independently for prompts and responses:
| Mode | What is stored |
|---|---|
| **None** | Metadata only, no content |
| **Partial** (default) | Content truncated to 2000 characters |
| **Full** | Raw content up to the column cap (a deliberate privacy tradeoff) |
Rule-level "Accept & No Log" always overrides, so sensitive traffic can opt out
of content storage entirely.
- **Retention** - transaction-log rows older than the configured window are
purged (default 30 days, `0` keeps everything). The purge runs at startup and
daily.
- **Audit Logs page size** - server-side pagination on the Audit Logs page
(default 50 rows per page).
## Audit events
The **Events** tab records security and application-change events with actor,
action, target, outcome, and detail:
- **login / logout / auth_failed** - web-UI logins, logouts, and failed
credential attempts (with source IP)
- **config_changed** - every settings change, with values **masked for
secrets** (API keys, tokens, secrets, passwords)
- **rule_created / rule_updated / rule_deleted** - Rules Manager mutations
- **user / role / key lifecycle** events and M365 bridge mutations
## Log sources
Four identifiable log types can be enabled or disabled independently, both for
the pull API and for syslog forwarding (Settings -> Log Sources & Export):
| Type | What it carries |
|---|---|
| App logs | Application log lines (category and task) |
| Security events | Audit events (actor, action, target, outcome, detail) |
| Access logs | Per-request outcome, mode, code, user, agent, path, IP, latency |
| Transactions | Model, allowed/blocked, flag and block codes, tokens, IP |
A disabled type returns `404` from the API and is never forwarded.
## Pull API
Admin-authenticated endpoints expose each log type (`/admin/audit`,
`/admin/security-events`, `/admin/access-logs`, `/admin/transactions`) with:
- a `count` limit (1-1000)
- per-type filters (actor, action, outcome, user, agent, source IP, free-text
search, model, block code)
Payloads are metadata only; transaction previews follow the logging policy.
## Syslog forwarding
Settings -> Syslog Collector pushes selected types to your SIEM:
- **Transport**: UDP (RFC 5424 datagram), TCP (plain, octet-counted), or TLS
(RFC 5425 with optional client certificate and CA pinning)
- **Host and port** (default 514, 6514 for TLS)
- Configurable **facility** (default `local0`) and **APP-NAME**
Forwarding is **non-blocking**: stores enqueue into a bounded channel and never
wait on the collector; overflow is dropped and counted. The background
forwarder retries with backoff and applies configuration changes live.
**Never forwarded**: secrets (audit details are masked at the source) and raw
prompt or completion content, only the metadata the logging policy permits.
- [Observability & dashboard](/docs/guides/observability-dashboard)
- [Risk profiles & auto-block](/docs/guides/risk-profiles)
---
# Admin-issued API keys
Section: Guides · URL: https://aifw.io/docs/guides/agent-api-keys
Issue API keys for agent and tool access - hashed storage, immediate revocation, and per-key identity in the audit trail.
Admins can issue **API keys** that clients send as `Authorization: Bearer `
or `x-api-key` to authenticate to the gateway endpoints. Keys work in addition
to JWT and identity-provider mechanisms.
## Manage keys
Open **Settings -> Access Control -> Manage API Keys** (available in every
authentication mode):
- **Add** - paste an existing key, or **Generate** a new one, with an optional
label for your own reference.
- **Revoke** - revocation is immediate.
Keys are stored as **SHA-256 hashes only** and are never stored in plaintext.
## Behavior per authentication mode
| Mode | How keys are treated |
|---|---|
| **Required** | Enforced - a missing or invalid token returns `401` |
| **Optional** | Validated - a key is checked when present |
| **Passthrough** | Not checked - presence only, and the client's token is forwarded upstream verbatim |
## Upstream behavior
API keys **authenticate the caller**; they are not forwarded upstream. With key
mode `client`, the client's bearer (the API key) would be forwarded upstream,
so use key mode `model` for registered-model keys instead.
## Identity in the audit trail
The audit and risk identity for an API-key client is the key's **unique
registry id**, not its label. The label is carried as a name for new traffic
(Risk Profiles shows it), and the Audit Logs page resolves the id to
`label (id)` at display time, so a truncated id renders as something like
`VSCode-01 (f54f7921...)`.
API keys also isolate the completion cache per key (each key gets its own cache
namespace), so cached answers never leak across keys.
- [Identity & access](/docs/guides/identity-access)
- [Audit logs & export](/docs/guides/audit-logs-export)
---
# Configure models & API keys
Section: How-To · URL: https://aifw.io/docs/how-to/configure-models-keys
A recipe for the Model Inventory, add models, set per-model keys, enable them, and understand key precedence.
This recipe walks through configuring the model registry so traffic routes where
you want it, with the right keys.
## 1. Open the Model Inventory
The **Model Inventory** page is the single place where models are managed. On
first run, a curated starter set is seeded (GPT family, o-series, DeepSeek, and
friends), each with its provider and backend URL.
Starter models ship **disabled**. Requests for a disabled model are rejected with
`400 model_disabled` until you explicitly enable it. This fail-closed default means
nothing routes until you say so.
## 2. Enable the models you use
Use the Enable/Disable control (or the status API) to flip each model to
`registered`. Only then will requests route to it.
## 3. Add a custom model
If your model isn't in the starter set, a private fine-tune, a self-hosted vLLM
or Ollama endpoint, use **Add Custom Model**:
| Field | Purpose |
|---|---|
| Model ID | The identifier clients send in the `model` field |
| Provider | A label for your own reference |
| Backend URL | Where the model actually lives |
| API key | Optional, the provider key, stored server-side |
| Protocol | `openai` or `anthropic` (`claude-*` IDs default to Anthropic) |
## 4. Set the default model & backend
In **Settings**:
- **Default model**, filled in when a client omits the `model` field.
- **Default backend**, the fallback destination for models without their own
backend URL (presets: DeepSeek, OpenAI, Mistral, Groq, Together, OpenRouter,
Ollama, LM Studio, vLLM, or a custom URL).
- **Reject Unregistered Models**, strict mode: block requests for models that
aren't in the registry.
## 5. Understand key precedence
Which key reaches the backend is resolved **rule → model → global** (most specific
wins):
1. **Rule-level**, a key mode pinned on a matching rule.
2. **Model-level**, the key mode in the Model Inventory.
3. **Global**, Settings → Access Control → Key Mode (default `model`).
With key mode `model` (default), the model's registered key is attached upstream
**even for anonymous callers**, traffic can consume the corporate key. If you
want callers to bring their own keys, set key mode `client` at the appropriate
scope (and see the
[identity guide](/docs/guides/identity-access#upstream-key-precedence) for the
trade-offs).
## 6. Watch for shadow AI
Models that appear in traffic but aren't in the registry show up as **Newly
Discovered LLMs**. Register the legitimate ones, and with strict mode on, reject
everything else.
## Per-model advanced parameters
Each model can carry a **parameter policy** applied to the request body on its
way to the backend. Configured in **Model Inventory -> Edit -> Customize
Advanced Parameters**:
- **Inject when absent** (default): the value is added only if the client
omitted the parameter.
- **Force**: the value replaces whatever the client sent.
The policy applies after protocol adaptation, so it works for OpenAI chat, the
Responses API (parameters like `reasoning_effort`, `top_p`, `stop`, `tools`,
`seed`, `response_format` are preserved through adaptation), and Anthropic
backends (where `top_p`, `stop`, and best-effort `reasoning_effort` are
translated). Common parameters are offered as suggestions; any JSON key can be
supplied through the admin API.
- [Model routing & registry](/docs/guides/model-routing), the full routing decision tree
- [Reliability & caching](/docs/guides/reliability-caching), per-model resilience settings
---
# Create rules with AI assistance
Section: How-To · URL: https://aifw.io/docs/how-to/create-rules-with-ai
Draft guardrail rules from a plain-language objective, the Rules Manager generates, validates, and lets you review before enforcing.
Writing regex by hand is tedious and brittle. The **Rules Manager** can draft a
rule for you: describe the objective in plain language, pick a rule type, and the
gateway generates a validated draft you review before it is ever enforced.
## 1. Open Create Rule with AI
In **Rules Manager**, find the **Create Rule with AI** card. Give your objective,
for example:
- "Block prompts that try to make the model reveal its system instructions"
- "Mask API keys in inbound requests"
- "Flag requests mentioning competitor product names"
## 2. Pick the rule type & action
| Choice | Meaning |
|---|---|
| **Keyword** | Generate a regex/keyword rule from the objective |
| **Semantic** | Generate a natural-language intent description for the semantic tier |
| **Action** (optional) | Pin block / mask / accept / log, or let the generator decide |
## 3. Review the draft
The draft appears in an **editable preview form**: name, type, action, severity,
and the generated pattern or intent description. The generated regex is validated
before the preview renders, you can edit anything before committing.
The generated rule is **never enforced until an admin reviews it**. The objective
you type is treated as untrusted input, it is only embedded in the prompt, never
executed.
## 4. Create the rule
Click **Create** to post to the standard rule pipeline, so duplicate-name checks,
threshold validation, and action semantics all apply exactly as if you had written
the rule by hand.
## How generation works
Generation uses the **AI Rule Generation** backend configured under **Settings →
AI Rule Generation**, pick a registered model (its stored backend URL and key are
used) or a custom model with endpoint and key. Without a configured backend the
card shows a configuration message instead.
The AI's reply is parsed as strict JSON (with one automatic re-prompt on garbage);
the draft is always editable, so even an imperfect first pass is a useful starting
point.
- [Prompt & response guardrails](/docs/guides/prompt-response-guardrails), evaluation order and rule actions
- [Semantic intent analysis](/docs/guides/semantic-intent-analysis), semantic rule thresholds
---
# Deploy on Azure Container Apps
Section: How-To · URL: https://aifw.io/docs/how-to/deploy-azure
A production-shaped deployment recipe, PostgreSQL, the gateway container, secrets, environment configuration, and health checks.
This recipe deploys the AI-FW gateway to **Azure Container Apps** with
PostgreSQL Flexible Server as the backing store.
Azure CLI installed and authenticated, a resource group, and a container
registry (ACR) holding your gateway image.
## 1. Provision PostgreSQL
Create a PostgreSQL Flexible Server with TLS required:
| Setting | Recommended |
|---|---|
| Tier | Burstable (e.g. `Standard_B1ms`) for evaluation; scale up for production |
| Storage | 32 GiB minimum |
| Backup retention | 7 days; geo-backup and high availability per your DR policy |
| Network | Private network for production; allow Azure services for simplest connectivity |
| Firewall | Only the services that must connect |
Store the connection string, including the password, as a container-app secret,
never in the image, pipeline output, or repository.
## 2. Push the gateway image
Build the gateway container and push it to your registry with a unique tag (for
example, your CI build ID). The container listens on the configured HTTP port
(`443` in the example below).
## 3. Create the Container App
Create the app with **external HTTPS ingress** to the gateway's HTTP port:
```bash
az containerapp create \
--resource-group \
--environment \
--name AI-FW-gateway \
--image /: \
--ingress external \
--target-port 443 \
--min-replicas 1 --max-replicas 1 \
--secrets postgres-connection= jwt-secret= \
--env-vars \
ASPNETCORE_ENVIRONMENT=Production \
ASPNETCORE_HTTP_PORTS=443 \
ConnectionStrings__PostgreSQL=secretref:postgres-connection \
AiFw__JwtSecret=secretref:jwt-secret
```
Notes:
- **Secrets** are referenced with `secretref:` and are never visible in app
configuration dumps.
- The **JWT secret** signs tokens for API-key and JWT authentication, generate a
strong random value and rotate it deliberately.
- Set **fail-closed** configuration (`AiFw__FailClosed=true`) to keep the
gateway's deny-by-default posture in production.
## 4. Configure the admin account
Right after first deployment, sign in and **change the bootstrap administrator
password** (User Management), then create your own admin users. If you use SSO,
configure OIDC now and switch the sign-in mode.
For production, restrict ingress with an IP allow-list or a private network,
exposing the admin UI to the public internet is only acceptable for short-lived
evaluations.
## 5. Verify health
1. Wait for the newest revision to become healthy.
2. Read the generated FQDN.
3. Call `https:///api/health`, expect `200`.
4. Call `https:///ready`, expect `200` (this is the readiness probe used by
the platform).
5. Confirm the running image matches the tag you pushed.
## 6. Optional: enable agent mTLS
Application-level **mTLS** (client certificates for agents) is configured through
the gateway's TLS settings. In a Container Apps deployment, enabling raw TCP
ingress and custom hostnames gives you full control over the certificate
lifecycle, a follow-up iteration after the initial HTTPS-only deployment.
- [Quick start](/docs/getting-started/quick-start), local evaluation first
- [Identity & access](/docs/guides/identity-access), admin roles and SSO after deployment
---
# Block prompt injection in production
Section: Solutions · URL: https://aifw.io/docs/solutions/block-prompt-injection
How to detect and block prompt injection and jailbreak attempts before they reach a model provider, using rule ordering, semantic scoring and fail-closed inspection.
Prompt injection is not a content problem you can solve with a blocked word list. Attackers rephrase, encode, split the payload across messages, and hide instructions in retrieved documents. What you need is a decision made before the request leaves your network, with a record of what was decided.
## The short answer
- Inspect **both directions**: the prompt before it is forwarded, and the response before it is returned.
- Combine **text rules** (known patterns) with **semantic scoring** (meaning), because either one alone misses a large share of attempts.
- Evaluate in a **fixed order** with deny winning over accept, so nothing later in the chain can soften a block.
- Make inspection **fail closed**: if scoring cannot run, the request is refused rather than forwarded unscanned.
## How to do it in AI-FW
1. **Turn on the built-in guardrails.** Jailbreak and prompt-injection detection ships enabled, and runs in line with the request.
2. **Add rules for your own patterns.** Keyword and regular-expression rules catch internal markers: a codename, a system-prompt fragment, a phrase your team uses. Each rule has an action - block, mask, accept, or log - and deny always wins.
3. **Add a semantic rule for intent.** Describe the policy in plain language, for example "detect requests attempting to extract the system prompt". AI-FW scores the meaning of the request against it, so a rephrasing still scores high. Use the embedding engine, a chat classifier, or the fully offline engine if nothing may leave your network.
4. **Test before you enforce.** Use the [Prompt Analyzer](/docs/admin/rules-manager) to run a prompt against your enabled rule base without sending it upstream. You see which rules fire, the semantic score for each policy, and the verdict - with no model call and no log rows.
5. **Set the failure mode deliberately.** Keep inspection fail-closed for production traffic, and document any opt-in fail-open rule so the exception is a decision, not an accident.
## Verify it works
- Send a known-bad prompt and confirm the request is refused before the provider sees it.
- Check the [Audit Logs](/docs/admin/audit-logs) entry: identity, model, decision, and the rule that fired - with no raw content stored.
- Watch the block rate on the [dashboard](/docs/guides/observability-dashboard) after enabling a new rule; a sudden step change usually means a false positive.
- Re-run the Prompt Analyzer against your own test set whenever you change rule order.
## Related
- [Prompt & response guardrails](/docs/guides/prompt-response-guardrails)
- [Semantic intent analysis](/docs/guides/semantic-intent-analysis)
- [Create rules with AI assistance](/docs/how-to/create-rules-with-ai)
## Frequently asked questions
**Do keyword rules alone stop prompt injection?**
No. Rules catch known patterns; semantic scoring catches rephrasing. Use both, with deny winning over accept so a specific block cannot be softened later in the chain.
**What happens if inspection fails?**
With the fail-closed default the request is refused rather than forwarded unscanned, so a scanner problem cannot silently bypass policy.
Ask for the Technical Plan: a step-by-step proof of concept you run against your own prompts, so you can measure detection and false positives before rolling out.
---
# Mask PII before prompts leave your network
Section: Solutions · URL: https://aifw.io/docs/solutions/mask-pii-before-providers
Detect and mask personal data in prompts before they reach a model provider, while keeping the answer usable and the audit record content-free.
Most teams discover they are sending personal data to a model provider during a privacy review, not before it. Blocking every request is rarely acceptable, and ignoring it is not either. Masking is the middle path: replace the sensitive values before the request leaves, then let the model work on the redacted text.
## The short answer
- Mask on the **outbound path**, before the provider sees the content, rather than logging afterwards.
- Decide per category: names, email addresses, phone numbers, account numbers, health data, card data.
- Keep the **structure** of the request intact so tool calls and JSON bodies still parse.
- Record the **decision** (masked, which categories) without storing the original content.
## How to do it in AI-FW
1. **Enable masking rules** for the categories you care about. Built-in detection covers the common cases; custom rules handle your own identifiers such as employee numbers or internal case references.
2. **Choose the action per rule.** `mask` replaces the value; `block` refuses the request; `log` records the event without changing the prompt. Deny wins over accept, so a broad allow cannot override a specific mask.
3. **Scope by identity where it matters.** A finance workload and a documentation-writing assistant can have different policies for the same category.
4. **Check structure preservation.** Masked content is rewritten in place, so structured payloads and tool calls survive the round trip.
5. **Keep the audit trail content-free.** The transaction records that masking happened and which categories applied. Raw prompts are not persisted.
## Verify it works
- Send a prompt containing a test email address and confirm the provider receives the redacted form.
- Confirm the response is still usable: the model should answer normally with placeholders in place.
- Inspect the [Audit Logs](/docs/admin/audit-logs) entry for the masking decision, and confirm no original value appears.
- Review [Risk Profiles](/docs/admin/risk-profiles) for the PII category to see how often masking fires across your teams.
## Related
- [Prompt & response guardrails](/docs/guides/prompt-response-guardrails)
- [Audit logs & export](/docs/guides/audit-logs-export)
- [Identity & access](/docs/guides/identity-access)
## Frequently asked questions
**Does masking break the answer?**
No. Values are replaced in place and the request structure is preserved, so the model answers normally with placeholders where the sensitive values were.
**Is the original value stored for audit?**
No. The transaction records that masking applied and which categories fired. Raw prompt content is not persisted.
Inspection and masking run locally, so no third-party service sees your prompts. See [Data residency for AI traffic](/comparison/ai-data-residency) for the questions to put in an RFP.
---
# Govern MCP tool calls by identity
Section: Solutions · URL: https://aifw.io/docs/solutions/govern-mcp-tool-calls
Decide which MCP tools an agent may call, with what arguments, and under whose identity, and keep an auditable record of every tool invocation.
A tool call is not a message. It changes state: it files a ticket, edits a record, sends a message, runs a query. Once agents can call tools, the question stops being "what did the model say" and becomes "what was permitted to happen, on whose authority".
## The short answer
- Treat tool calls as **governed actions** with an identity, not as part of the conversation.
- Decide **per identity** which tools are allowed, and which arguments are acceptable.
- Keep one policy surface instead of per-client allowlists that drift.
- Record tool name, caller, decision and outcome - without storing the payload.
## How to do it in AI-FW
1. **Register the client.** MCP tools reach AI-FW through the same authenticated path as any other caller, so the agent has an identity (API key, JWT, or certificate) rather than a shared token.
2. **Scope tools to groups.** Put agents with similar jobs in a group and attach tool policy to the group. A support agent and a finance agent should not inherit the same tool set by accident.
3. **Decide by tool and argument.** Allow a search tool for everyone, restrict a write tool to a specific group, and mask or block sensitive argument patterns. Policy is data, not code, so it changes without a deployment.
4. **Keep inspection in the path.** Prompts and responses are still scanned, so a tool call cannot become a way to smuggle content past the guardrails.
5. **Use the same audit trail.** Every invocation lands in the transaction record with the identity and the decision, so a single query answers who called which tool.
## Verify it works
- Call a blocked tool and confirm the refusal comes back as a policy decision, not a client-side error.
- Check [Audit Logs](/docs/admin/audit-logs) for the tool name, the caller and the outcome.
- Review which identities actually invoke write tools; unused privileges are the easiest thing to remove.
## Related
- [MCP tools](/product/mcp-tools)
- [Agents reference](/docs/admin/agents)
- [A2A agent protocol](/docs/api-reference/a2a-agent-protocol)
## Frequently asked questions
**Can we allow a tool for one agent group only?**
Yes. Tool policy attaches to identities and groups, so a support agent and a finance agent can have different tool sets without a per-client allowlist.
**Are tool arguments inspected?**
Yes. A tool call passes through the same inspection as any other request, so content cannot be smuggled past the guardrails through a tool invocation.
Coding agents, MCP tools and Microsoft 365 Copilot can all sit behind one policy engine. See [Governing the AI surfaces beyond chat](/comparison/governing-agent-surfaces).
---
# Find and stop shadow AI usage
Section: Solutions · URL: https://aifw.io/docs/solutions/stop-shadow-ai
Discover unregistered models and ungoverned clients, then close the gap with a strict model registry, per-identity policy and a record of every attempt.
Shadow AI rarely looks like rebellion. It looks like a team that needed an answer, had a company card, and signed up for a model API. The traffic exists; the governance does not. The first step is making it visible, and the second is deciding what to do about it.
## The short answer
- You cannot govern what you cannot see: **log every model request** through one gateway.
- A **strict registry** turns "unknown model" from a silent success into a refusal you can act on.
- Attribute usage to an **identity**, so an unregistered client has a name attached.
- Expect the first week to be noisy: most shadow usage is benign and simply needs an approved path.
## How to do it in AI-FW
1. **Put the gateway in the path.** Point clients at one base URL. OpenAI-compatible and Anthropic-compatible clients work with a base-URL change, which is usually a one-line edit per team.
2. **Turn on strict model registration.** With `Reject unregistered models` enabled, a request for a model nobody approved is refused rather than forwarded. That converts invisible usage into an explicit, attributable event.
3. **Watch the flags, not just the blocks.** Shadow AI appears in the [dashboard](/docs/guides/observability-dashboard) as unregistered-model attempts, and in [Audit Logs](/docs/admin/audit-logs) with the caller identity.
4. **Give the team a legitimate path.** Register the model that was being used unofficially, attach a per-model key so credentials stay server-side, and route it under policy. The goal is adoption, not a permanent block.
5. **Add risk scoring.** Repeated unregistered attempts raise the risk score for that identity, which can trigger an automatic cooldown while you resolve it.
## Verify it works
- Confirm unregistered-model attempts appear in the transaction log with an identity attached.
- Check that registered models still serve normally: a strict registry should not break approved traffic.
- Review [Model Inventory](/docs/admin/model-inventory) monthly for models that are registered but unused, and keys that should be rotated.
## Related
- [Shadow AI](/product/shadow-ai)
- [Model routing & registry](/docs/guides/model-routing)
- [Risk profiles & auto-block](/docs/guides/risk-profiles)
## Frequently asked questions
**Will strict model registration break approved traffic?**
No. Registered models continue to serve normally. Only requests for unregistered models are refused, and each one is attributed to an identity so you can act on it.
**How do we onboard a model a team is already using?**
Register it with its provider and key, then let it route under policy. The goal is adoption through an approved path rather than a permanent block.
The [self-hosted buyer's checklist](/comparison/self-hosted-ai-gateway) covers what to look for in the discovery controls, and [Configure models & keys](/docs/how-to/configure-models-keys) walks through registering them.
---
# Give agents real identities with mTLS
Section: Solutions · URL: https://aifw.io/docs/solutions/agent-identities-mtls
Issue certificates to agents through a CSR enrollment flow, then authorize every model call against that identity instead of a shared API key.
An API key proves that someone holds a string. It does not say which agent is calling, whether that agent is still supposed to exist, or how to revoke it without rotating a credential every other workload shares. Agents that act autonomously need the same trust model you already use for servers.
## The short answer
- Give each agent its **own identity**: a key pair and a certificate, not a copied token.
- Let agents **enroll themselves** with a certificate signing request, approved by policy.
- Authorize on the **certificate identity**, so revocation does not require rotating shared secrets.
- Keep keys, certificates and policy in one place so suspension is immediate.
## How to do it in AI-FW
1. **Choose the authentication mode per agent.** API key, JWT, mutual TLS, or Kerberos for domain-joined machines. Different agent populations can use different modes.
2. **Run self-enrollment.** The agent generates a key pair, presents a certificate signing request to AI-FW, and receives a certificate. The private key never leaves the agent.
3. **Attach policy to the identity.** Roles, model access, and tool permissions are granted to the agent or its group, not to every holder of a shared key.
4. **Suspend by identity.** If an agent misbehaves, revoke its certificate. Every other workload keeps working - no organisation-wide rotation.
5. **Bring your own CA if you prefer.** AI-FW integrates with certificate authority workflows, so issued certificates can follow the process your PKI team already runs.
## Verify it works
- Enroll a test agent and confirm it can call a model, then authorize a second request with a certificate that is not registered: it should be refused.
- Revoke the test agent's certificate and confirm the next call fails while other agents continue normally.
- Check the [Agents reference](/docs/admin/agents) and [Agent Trust](/docs/admin/agent-trust) for inventory, approvals and lifecycle state.
- Confirm the audit record ties each call to the certificate identity.
## Related
- [Agent self-enrollment tutorial](/docs/tutorials/agent-self-enrollment)
- [Identity & access](/docs/guides/identity-access)
- [CA integrations](/docs/admin/ca-integrations)
## Frequently asked questions
**Does an agent's private key leave the agent?**
No. The agent generates the key pair and presents only a certificate signing request, so the private key never leaves the agent.
**How do we revoke a single agent?**
Revoke its certificate. Other agents keep working, with no shared secret that has to be rotated across the fleet.
Treating agents as identities rather than clients is what makes agent-to-agent work provable. See [AI gateway vs AI firewall](/comparison/ai-gateway-vs-ai-firewall).
---
# Cut LLM token costs without a safety hole
Section: Solutions · URL: https://aifw.io/docs/solutions/cut-token-costs-safely
Reduce token spend with semantic-gated prompt compression and inspection-gated caching, without letting cost controls bypass guardrails.
Token spend grows quietly: more teams, longer prompts, retries, and the same question asked five times because nobody cached the answer. The easy savings are the dangerous ones - caching a response that should have been blocked, or compressing a prompt until its meaning changes.
## The short answer
- Compress **only when a semantic gate confirms** the meaning survived; otherwise send the original.
- Cache **only content that passed inspection**, with keys scoped by identity so answers cannot leak between callers.
- Never cache streams or flagged content.
- Measure savings per model, not as one blended number, so you can see which change is actually paying.
## How to do it in AI-FW
1. **Turn on the completion cache.** Exact matches are served without an upstream call; near-duplicates are matched semantically. Cache keys are identity-salted and scoped per agent, group, model or globally, first match wins.
2. **Keep inspection in front of the cache.** Blocked or policy-violating content is never stored, and cached answers are re-checked against the identity that is asking, so a hit cannot cross a boundary.
3. **Enable prompt compression.** Long prompts are compressed under a gate: the compressed version is scored against the original, and it is used only when meaning is preserved. If the gate cannot run, the original is kept - fail closed, as usual.
4. **Watch the gate budget.** On very large prompts the gate runs within a per-request budget. When it is exhausted, remaining segments are left untouched, which keeps latency and embedding cost bounded.
5. **Route by cost as well as capability.** Send routine workloads to a cheaper model, keep regulated data on an approved provider, and express both as routing rules rather than tribal knowledge.
## Verify it works
- Compare tokens billed across a week with cache and compression off, then on, for the same workload.
- Confirm no blocked response ever appears as a cache hit.
- Spot-check compressed prompts for a few workloads: the answers should be equivalent, and any rejected gate should leave the original text in place.
- Review the cost model in [Total cost of ownership](/comparison/ai-gateway-tco) for the four lines that decide payback.
## Related
- [Reliability & caching](/docs/guides/reliability-caching)
- [Prompt compression](/docs/guides/prompt-compression)
- [Completion cache](/product/caching)
## Frequently asked questions
**Can a cached answer bypass inspection?**
No. Blocked or flagged content never enters the cache, and cached answers are evaluated against the identity making the request, so a hit cannot cross a boundary.
**What happens when compression cannot be verified?**
The original prompt is sent. Compression applies only when the semantic gate confirms the meaning survived.
The Community Edition includes caching, compression and the dashboard, with a published token allowance - enough to measure real savings before a commercial agreement.
---
# Connect an OpenAI SDK
Section: Tutorials · URL: https://aifw.io/docs/tutorials/connect-openai-sdk
Point an OpenAI-compatible client at the gateway, endpoint mode for full inspection, proxy mode for convenience.
There are two ways for OpenAI-compatible clients to reach the gateway:
1. **Endpoint mode** (recommended), point the client at the gateway's `/v1`
base URL. Full content inspection on every request.
2. **Proxy mode**, configure the gateway as an HTTP forward proxy.
## Endpoint mode
Any OpenAI-compatible SDK works. The only change is the base URL.
### Python
```python
from openai import OpenAI
client = OpenAI(
api_key="", # passed through to the backend
base_url="http://:443/v1", # the AI-FW gateway
default_headers={"X-Agent-ID": "my-agent-01"},
)
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
```
### Node.js
```js
import OpenAI from "openai";
const client = new OpenAI({
apiKey: "",
baseURL: "http://:443/v1",
defaultHeaders: { "X-Agent-ID": "my-agent-01" },
});
const resp = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: "Hello!" }],
});
console.log(resp.choices[0].message.content);
```
### Plain curl
```bash
curl http://:443/v1/chat/completions \
-H "Content-Type: application/json" \
-H "X-Agent-ID: my-agent-01" \
-d '{
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello!"}]
}'
```
## Request headers
| Header | Purpose |
|---|---|
| `Authorization: Bearer ` | Client authentication and/or upstream key (see auth modes) |
| `X-Agent-ID` | Agent identity for tracking, routing rules, and risk |
| `X-User-Id` | User identity for tracking (token `sub` claims are preferred) |
| `x-aifw-cache-refresh` | Bypass the cache lookup for this request (still allows the write) |
## Proxy mode
Configure the gateway as an HTTP forward proxy for clients that support one:
```bash
curl --proxy http://:443 https://api.openai.com/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'
```
| Proxy mode | Behavior | Inspection |
|---|---|---|
| Absolute-form HTTP | Normalized to `/v1/*`, full pipeline runs | ✅ Full |
| CONNECT tunnel (HTTPS) | Raw TCP tunnel to an allow-listed backend | ⚠️ Metadata only, TLS is end-to-end encrypted |
**CONNECT allow-list**, tunnels only open to the default backend host, registered
model backends, and extra hosts you add in **Settings → Forward Proxy → Allowed
Tunnel Hosts**. Anything else gets `403`. The whole forward proxy can be disabled.
Content inspection requires seeing the traffic. For HTTPS, use endpoint mode, the
CONNECT tunnel is end-to-end encrypted and only metadata is logged.
## Streaming
`stream: true` works exactly as with any OpenAI backend. The gateway's streaming
mode (buffered or live) is applied per request, see the
[identity & access guide](/docs/guides/identity-access#streaming-modes) for the
trade-off between strictness and latency.
- [API reference, OpenAI-compatible endpoints](/docs/api-reference/openai-compatible-api)
- [Quick start](/docs/getting-started/quick-start)
---
# Claude Code, Cursor & MCP tools
Section: Tutorials · URL: https://aifw.io/docs/tutorials/claude-code-cursor-mcp
Route Claude-native clients, Cursor, and MCP tools through the gateway, Anthropic protocol support, facades, and the MCP compliance interface.
The gateway speaks **both** the OpenAI chat-completions protocol and the
**Anthropic Messages protocol**, so Claude-native clients get the same rules,
semantic tier, key precedence, streaming, and audit logging as OpenAI-compatible
clients.
## Endpoints at a glance
| Client protocol | Gateway endpoint | Examples |
|---|---|---|
| OpenAI `chat/completions` | `POST /v1/chat/completions` | OpenAI SDKs, Cline, VS Code, Copilot |
| Anthropic `messages` | `POST /v1/messages` | Claude Code, Claude CoWork, Claude Desktop |
Each registered model carries a protocol (`openai` or `anthropic`); `claude-*`
model IDs default to Anthropic automatically.
## Claude Code / CoWork
Point the client at the gateway with its base URL, no other client change:
```bash
export ANTHROPIC_BASE_URL=http://:443
export ANTHROPIC_AUTH_TOKEN= # optional; see auth modes
```
Claude sends `POST /v1/messages` with `x-api-key` (console key) or
`Authorization: Bearer` (OAuth). The gateway accepts both, relays to the registered
Claude model's backend, and applies the resolved key and rules.
## Cursor / Cline / VS Code
Set the OpenAI base URL to `http://:443/v1` and pick a registered
model. Because the gateway can route `claude-*` models behind an OpenAI-style
client (and vice-versa), you can use Claude models from OpenAI-style editors and
OpenAI models from Anthropic-style clients.
## Protocol facades
The gateway translates between the two protocols:
- **OpenAI client → Claude backend**: the request is translated to the Anthropic
Messages schema (system, messages, tools → `input_schema`) and the response/SSE
translated back to the OpenAI shape. `temperature` is clamped to 0–1, and
`system`/`developer` messages are hoisted into the initial system message.
- **Anthropic client → OpenAI backend**: the Messages request is translated to
`chat/completions` and the response/SSE back to Messages events.
Both directions run through the same inspection pipeline, rule enforcement and
the semantic tier are protocol-agnostic. Streaming works on both endpoints, with
`content_block_delta` frames scanned token-by-token; a mid-stream violation
terminates the SSE stream with a `guardrail_violation` event.
## MCP tools
The gateway exposes its **compliance tooling over MCP** so MCP-enabled agents can
call it directly:
- `initialize`, protocol handshake
- `list tools`, discover available compliance tools
- `check prompt`, send a prompt for inspection *before* it goes to a model, and
act on the verdict
Verdicts are machine-actionable: `allow`, `deny`, `flag`, or `mask`, so an agent
can self-police before a single token is spent. Each verdict includes the matched
policy and (for denies) the reason.
## Verify it works
- `POST /v1/messages` with a `claude-*` model → the gateway relays to the
Claude backend.
- Ask for a model with the wrong protocol → the facade translates transparently.
- Send a jailbreak attempt through either protocol → `400 guardrail_violation`.
- [API reference, Anthropic Messages](/docs/api-reference/anthropic-messages-api)
- [Quick start](/docs/getting-started/quick-start)
---
# M365 Copilot admin & agent sync
Section: Tutorials · URL: https://aifw.io/docs/tutorials/m365-copilot-bridge
Manage Microsoft 365 Copilot admin surfaces from AI-FW with app-only auth - agent catalog, agent registry, usage reports, and interaction export.
The **M365 Copilot admin** bridge connects AI-FW to Microsoft 365 Copilot
admin surfaces through Microsoft Graph with **application permissions**
(app-only, client-credentials) from your Entra app registration. No user
sign-in, no delegated consent, no redirect URI.
What it manages:
- **Copilot agent catalog (packages)** - live inventory of agents in the
organization, with per-agent details (name, type, platform, hosts, publisher,
version, blocked state, deployment).
- **Agent registry** - create, view, update, and delete agent registrations.
This is how you **sync agents registered on AI-FW with M365**.
- **Usage analytics** - per-user activity, user-count summaries, and daily
user-count trends (D7 / D30 / D90 / D180).
- **Interaction export** - export a user's Copilot prompt and response history
for review.
The Agent Registry endpoint is `/beta` and in preview: Microsoft states that
beta APIs are subject to change and not supported for production use. The
catalog, usage reports, and interaction export are `/v1.0` (GA).
## What changed
Earlier versions of the bridge used a delegated authorization-code flow and
managed the Copilot **policy-settings** API (allow web search, chat pinning,
image generation) with drift detection and push/pull. That flow was removed:
Microsoft does not support Application permissions for the policy-settings API,
so those chat-level guardrails can never be managed app-only and are out of
scope. Deployments on the old flow must re-provision the app registration with
the application permissions below.
## 1. One-time setup in Microsoft Entra
1. Register an app in Microsoft Entra (or use an existing Enterprise App) and
grant the **application** permissions:
| Permission | Surface |
|---|---|
| `AgentRegistration.ReadWrite.All` | Agent registry |
| `CopilotPackages.ReadWrite.All` | Agent catalog |
| `Reports.Read.All` | Usage reports |
| `AiEnterpriseInteraction.Read.All` | Interaction export |
Complete **admin consent** in the Entra portal. No redirect URI is needed,
app-only auth never redirects a browser.
2. In AI-FW: **Settings → M365 Copilot Admin** → enable the bridge and enter the
**Tenant ID / Client ID / Client Secret** from the app registration.
3. Click **Test connection**. AI-FW acquires an app-only token and lists the
Copilot agent catalog to prove the application permissions work.
## 2. Work with the M365 Copilot page
Once connected, the **M365 Copilot** page shows:
- the **agent catalog** with per-agent details,
- the **agent registry** for creating and managing agent registrations,
including syncing agents registered on AI-FW with M365,
- **usage analytics** for the selected period,
- **interaction export** for a user's Copilot history.
## Behavior and safety
- **App-only everywhere** - every Graph call carries a client-credentials token,
cached until expiry. There is no user, refresh token, or redirect URI.
- **Fail-closed** - Graph errors abort the action and surface in the UI; the
client secret is masked and never returned to the browser.
- Every registry mutation and report load is written to the audit log.
- **Mock Graph** routes all Graph calls to a built-in in-memory mock for demos
and automated tests. It is only honored in non-production environments, so
the bridge can never be silently skipped in production.
## Scope note
App-only is limited to the Copilot admin APIs Microsoft supports with
application permissions: agent catalog, agent registry, usage reports, and
interaction export. The **policy-settings** API (`copilotPolicySetting`) is
Application = Not supported per Microsoft docs, so chat-level guardrails (allow
web search, chat pinning, image generation) cannot be managed app-only and are
out of scope for this bridge.
- [Identity & access](/docs/guides/identity-access) - the roles that can manage the bridge
- [Risk profiles & auto-block](/docs/guides/risk-profiles) - auditing the bridge's changes
---
# Agent self-enrollment (CSR + mTLS)
Section: Tutorials · URL: https://aifw.io/docs/tutorials/agent-self-enrollment
Walk an AI agent through the full enrollment round-trip, discover, register, request a certificate with a CSR, and authenticate with mTLS.
Agent Trust lets AI agents onboard themselves and obtain **PKI certificates** over
the Agent2Agent (A2A) task-based protocol. This tutorial walks through the full
round-trip: discovery → registration → certificate request (CSR) → certificate
extraction → mTLS authentication.
## 1. Discover, learn the API
Every AI-FW deployment publishes an **agent card** describing its
capabilities and authentication schemes:
```bash
curl http:///.well-known/agent-card.json
```
The card advertises the A2A endpoints, the authentication schemes it accepts
(bearer and/or mTLS), and the skills the agent can use. Read it before building
your client, it is the single source of truth for the deployment.
## 2. Register, introduce yourself
Register with a unique `agentId` and the identity proof (your public key):
```json
POST /a2a/v1/register
{
"agentId": "my-agent@corp.example",
"displayName": "My Agent",
"publicKeyPem": "-----BEGIN PUBLIC KEY----- …",
"usage": "Document Q&A assistant for legal",
"dataClassification": "Internal",
"tags": ["legal", "qa"]
}
```
Key facts:
- **Your identity is bound to the caller**, the first registrant's authenticated
identity (or source IP) owns that `agentId`. Another caller cannot take it over.
- Give every agent a **unique `agentId`**. Agents sharing an IP with no `agentId`
collapse into a single identity; the same `agentId` from two hosts is rejected.
- Identity proof (public key or certificate thumbprint) is set by the client at
registration and **cannot be edited by admins**, rotate it by re-registering
from the same sender.
## 3. Request the certificate, send your CSR
Generate a key pair and a CSR locally, **the private key never leaves the agent**:
```bash
openssl req -new -newkey rsa:2048 -nodes \
-keyout agent.key -out agent.csr \
-subj "/CN=my-agent@corp.example"
```
Then send the CSR to the certificate service as an A2A task:
```json
POST /a2a/v1/tasks/send
{
"contextId": "enrollment-001",
"message": {
"role": "user",
"parts": [{
"kind": "file",
"mimeType": "application/x-pem-file",
"name": "agent.csr",
"bytes": ""
}]
}
}
```
## 4. Poll the task
The certificate service processes the request asynchronously. Poll the task state
until it completes:
```json
GET /a2a/v1/tasks/{taskId}
```
Expect `completed` with the issued certificate chain as the result artifact.
## 5. Extract the certificate
Retrieve the artifact and write the chain to disk:
```bash
# certificate chain: leaf + intermediate certs (no root, correct)
# the leaf's SANs must cover the gateway host you'll connect to
curl -o cert.pem http:///a2a/v1/tasks/{taskId}/artifacts/{artifactId}
```
`agent.key` was generated in step 3 and stayed on the agent; `agent.csr` was the
request input. The issued `cert.pem` is the only new piece.
## 6. Authenticate with mTLS
Use the certificate for mutual TLS against the gateway:
```bash
curl --cert cert.pem --key agent.key \
https:///a2a/v1/agent/card \
--cacert ca-bundle.pem
```
The gateway validates your certificate against its **CA trust store** (the same
store used for server certificates and other agents), maps the certificate
identity to your agent record, and authenticates you. Because runtime validation
is against the trust store, certificate rotation keeps working without a registry
change.
## The round-trip at a glance
```
discover (.well-known/agent-card.json)
→ register (POST /a2a/v1/register, bind identity)
→ send CSR (POST /a2a/v1/tasks/send)
→ poll task (GET /a2a/v1/tasks/{taskId})
→ extract chain (cert artifact)
→ use it (mTLS for every call)
```
- [API reference, A2A agent protocol](/docs/api-reference/a2a-agent-protocol)
- [Identity & access](/docs/guides/identity-access), mTLS, Kerberos, and API keys
---