Mitrity LLM Gateway

The Mitrity LLM Gateway checks the requests your agents send to LLM providers. You run it in your own cluster, and you point each agent's provider base_url at the gateway instead of at the provider.

For each request, the gateway:

1. accepts the agent's TLS connection, using its own certificate;
2. runs the MITRITY decision engine on the parsed prompt;
3. redacts sensitive data in the request body, in place, if a DLP rule matches;
4. opens a new TLS connection to the real provider and forwards the request;
5. scans the model's response, including streamed responses, before the agent sees it.

This is the fourth deployment surface, next to Mitrity Gateway, Mitrity MCP Sidecar and the Mitrity Mesh Authorizer. It is the same decision engine at a different decision point. The Mesh Authorizer cannot rewrite a request body on the wire, but the Mitrity LLM Gateway owns the body. Matched secrets and PII are scrubbed before the prompt ever leaves your cluster, so redaction happens on the wire, in both directions.

Use it when your agents call Anthropic, OpenAI, Azure OpenAI, Vertex AI, Amazon Bedrock or an OpenAI-compatible host directly (not through an MCP tool), and you want mission-scope, threat-intel and DLP governance on that traffic. You also get usage and cost visibility per agent and per model.

What it governs

Every request to a configured provider path runs the same decision engine that powers the other MITRITY surfaces:

A request path with no configured provider returns 404. It is never forwarded anywhere.

Prompt injection on prompts

The injection scan runs on every prompt. The policy's llm_injection_mode decides what a hit does:

• enforce blocks the model call;
• alert, the default, lets the call proceed and records the event;
• off skips the scan for prompts.

Tool calls are governed by the policy's injection_mode, and this setting does not affect them.

The default is alert for two reasons. An injection in a prompt does no harm by itself: the harm is the tool call it induces, and that stays enforced. And agents that read logs, web pages or tickets legitimately see injection-shaped text.

Setting injection_mode to off turns the scan off everywhere, prompts included.

Guardrails: model access, cost & rate

Beyond inspecting each prompt, the gateway enforces per-policy guardrails on LLM traffic. You configure all of them in the dashboard, under Policies → (a policy) → LLM Guardrails. They apply to every agent bound to that policy, and the gateway enforces them before the call reaches the provider:

• Model allowlists / denylists: restrict which providers and models an agent may call. For example, allow only anthropic:claude-*, deny azure-openai:gpt-4o, or allow vertex:gemini-2.5-*. You author them as llm:send:<provider>:<model> allow/deny rules. The policy's default action decides between "allow all except denied" and "deny all except allowed." An off-allowlist call is denied with a provider-shaped 403.

• Per-call token caps: cap the output tokens a single call may request (max_output_tokens). A request over the cap is denied before it reaches the provider.

• Cost budgets (token / USD): a per-policy budget over a rolling day / week / month window, measured from actual usage. Set a token budget (exact), a USD budget (best-effort, priced from the model catalog), or both. Add a warn threshold (default 80%) and an exceeded action:

  • block: over-budget calls are denied (403);
  • hold: over-budget calls are held for approval (409), and you approve them to keep spending;
  • alert-only: calls proceed, and you are notified.

  On the policy, the dashboard shows a live burn-down (used vs limit, % consumed, current window). Spend alerts fire to Slack/email once at the warn threshold, and once when the budget is exceeded.

• Rate limits: a per-policy cap on LLM calls per minute per agent. Requests over the limit get a 429.

Budgets and rate limits fail open: an agent with no budget or limit configured is never blocked, because cost control is opt-in. Identity and DLP, by contrast, fail closed.

These fine-grained controls are specific to the LLM Gateway, which parses the request and response body. The Mesh Authorizer can enforce a coarse egress allowlist to provider hosts (for example, block api.openai.com entirely), but it cannot see the model, the token counts or the spend. For model / token / budget / rate governance, route LLM traffic through the gateway.

Supported providers

Anthropic and OpenAI are configured by default. Remove one from the configuration to 404 its path.

Azure OpenAI, Vertex AI and OpenAI-compatible hosts have no default upstream: you name your own resource, regional endpoint or host. Bedrock's regional endpoint follows from the region you configure.

Each upstream URL must be https://, because the gateway opens a fresh TLS connection toward the provider. You can point an upstream at a regional or proxy endpoint, if your provider contract requires one.

Two providers that could answer the same path are refused when the gateway starts, rather than guessed. That happens with:

• OpenAI and a compatible host, both on /v1;
• Azure's v1 surface and a compatible host on /openai/v1;
• Azure's v1 surface and Bedrock.

Give the compatible host a mount of its own, and keep Azure on its deployment surface next to Bedrock.

A policy names a model by its provider key: anthropic, openai, azure-openai, vertex, aws-bedrock or openai-compatible. The same key appears on every usage record, in the LLM BOM and in the price catalog, so vertex:gemini-2.5-pro means the same thing wherever you see it.

Agents keep their own provider API key. The gateway forwards it verbatim to the provider, and only changes where the traffic flows. To point an agent at the gateway, you change its base URL and make it trust the gateway's CA:

# Anthropic SDKs:
export ANTHROPIC_BASE_URL=https://<gateway-host>:8443
# OpenAI SDKs (note the /v1 -- OpenAI clients append /chat/completions):
export OPENAI_BASE_URL=https://<gateway-host>:8443/v1
# An OpenAI-compatible host: the gateway plus the mount you configured for it
export OPENAI_BASE_URL=https://<gateway-host>:8443/mistral/v1
# Amazon Bedrock through an OpenAI SDK: the gateway plus /openai/v1, with your
# Bedrock API key as the bearer token
export OPENAI_BASE_URL=https://<gateway-host>:8443/openai/v1
export OPENAI_API_KEY=$AWS_BEARER_TOKEN_BEDROCK
# Azure OpenAI SDKs: pass the gateway as the endpoint
#   AzureOpenAI(azure_endpoint="https://<gateway-host>:8443", ...)
# Trust the gateway CA (or use your SDK's CA option):
export SSL_CERT_FILE=$PWD/mitrity-llm-gateway-ca.crt

# Vertex AI (google-genai): keep vertexai=True with your project and location;
# the SDK builds the full /v1/projects/.../models/{model}:generateContent path
# and sends ?alt=sse on streams, so only the base URL changes.
from google import genai
from google.genai import types

client = genai.Client(
    vertexai=True, project="my-gcp-project", location="europe-north1",
    http_options=types.HttpOptions(base_url="https://<gateway-host>:8443"),
)

Azure OpenAI and AI Foundry

Azure has no default upstream, because every subscription has its own resource. So the upstream URL is required. It must be an *.openai.azure.com or *.services.ai.azure.com host, unless you explicitly allow a private endpoint. Both surfaces are served by default; enable only the one you use if you would rather the other return 404. The api-version query parameter rides through untouched, as does every other query parameter.

Map each deployment to the model it serves. An Azure deployment name is a label you chose, and the model behind it appears nowhere in the request. Without a mapping, the gateway cannot name the model. A policy rule that names models, such as a model allowlist or a price-catalog entry, then has nothing to match. So the gateway resolves the deployment to a model before it judges the call:

providers:
  azure-openai:
    upstream_url: "https://your-resource.openai.azure.com"
    deployments:
      gpt-4o-prod: "gpt-4o"
      gpt-4o-mini-batch: "gpt-4o-mini"

A call to a deployment that is not in the map is denied (403). The refusal is recorded on the audit trail with the deployment name, so you can see exactly what to add. on_unknown_deployment: allow forwards those calls instead. Use it only while you finish the map, because a call whose model the gateway cannot name is a call your model allowlists do not restrict.

The dashboard, the usage records, the LLM BOM and every policy rule see the resolved model: an Azure call through gpt-4o-prod is governed and priced as azure-openai:gpt-4o. The request body that reaches Azure is the body your agent wrote, minus anything DLP redacted. The gateway never adds a field that your SDK did not send.

Copilot Studio and Microsoft 365 Copilot are not proxied here. Their models run inside Microsoft, so there is no customer-controlled hop for the gateway to sit on. Azure OpenAI / AI Foundry traffic is protected inline at the gateway. Copilot Studio and Microsoft 365 Copilot agents are discovered, inventoried and posture-assessed through the Microsoft 365 connector instead.

Vertex AI

Vertex AI is governed on the Gemini generateContent and streamGenerateContent methods. The model is read from the request path (.../models/gemini-2.5-pro:generateContent). So a policy rule such as llm:send:vertex:gemini-2.5-* applies before the call is forwarded, and the usage record names vertex:gemini-2.5-pro. Your agents keep their own Google credentials, and the bearer token is forwarded as-is.

There is no default upstream: set the regional endpoint that you call today. A shared gateway can be pinned to one project and region. A request path that names any other project or region is answered 404 and is never forwarded.

providers:
  vertex:
    upstream_url: "https://europe-north1-aiplatform.googleapis.com"
    project: "my-gcp-project"      # optional pin
    location: "europe-north1"      # optional pin

Two things to know about Vertex traffic:

• Streaming works best with alt=sse. Every Google SDK sends it, and the gateway then applies the streaming holdback scan frame by frame. Without alt=sse, Vertex returns one JSON array instead of an event stream. The gateway buffers that body and scans it whole before delivering it. The response is still governed and still redacted, but without the time-to-first-token benefit.
• Thinking tokens count as output. Gemini bills the tokens a model spends reasoning as output tokens, and so does the gateway. output_tokens on a Vertex usage record is the sum of the candidate and thinking token counts that Vertex reports, and your token budgets and the estimated cost are measured against it.

rawPredict (partner models) and Vertex's own OpenAI-compatible endpoint are not served and return 404.

Amazon Bedrock

Bedrock is governed on its OpenAI-compatible surface: /openai/v1/chat/completions, authenticated with a Bedrock API key as the bearer token (AWS_BEARER_TOKEN_BEDROCK). Traffic on that surface gets the same governance as OpenAI's own. The prompt is decomposed, scanned and redacted in place, the response is scanned, and the call is metered. Your agents keep their own Bedrock API key, and the gateway forwards it as-is.

providers:
  aws-bedrock:
    region: "us-west-2"          # the regional endpoint is derived from it
    auth: "passthrough"

Models are governed by their base id. Bedrock model ids carry a version suffix with a colon (anthropic.claude-sonnet-4-5-20260514-v1:0), and a cross-region inference profile adds a region prefix (us., eu., apac., us-gov.). The gateway strips the prefix for the verdict, the usage record and the price catalog. A call to us.anthropic.claude-sonnet-4-5-20260514-v1:0 is governed as aws-bedrock:anthropic.claude-sonnet-4-5-20260514-v1:0, so one rule covers every regional profile of a model. The gateway still forwards the body with the id your agent wrote. Every usage record from Bedrock also carries how the gateway authenticated (passthrough).

What this release does not cover. Bedrock's native Converse / ConverseStream and InvokeModel surfaces sign the request body with SigV4. These are the surfaces that boto3's converse() and LangChain's ChatBedrock call. A gateway that redacts such a body must re-sign it with a credential of its own, and that adapter has not shipped. So those surfaces and the resign mode are recognized in configuration, and refused at startup with a message that says so. An agent that calls Bedrock through an OpenAI SDK against the compatible endpoint is governed today. An agent on boto3 or ChatBedrock is not.

Bedrock and Azure's v1 surface publish the same path. With both configured, the gateway refuses to start, so keep Azure on its deployment surface.

OpenAI-compatible providers

Most model hosts outside the big four publish the OpenAI Chat Completions wire format under their own origin: Mistral, Groq, Together, OpenRouter and every self-hosted vLLM. The gateway governs them all as one provider, openai-compatible. They get the same prompt decomposition, DLP redaction, streaming scan and usage metering as OpenAI, pointed at the host you name.

You choose three things:

• the upstream origin;
• the path prefix that the host serves its API under (/v1 for most; the gateway forwards to it as-is);
• the mount: the path this gateway serves the host on, which defaults to the prefix.

A call to <mount>/chat/completions reaches the host at <path_prefix>/chat/completions. Your agent's OpenAI SDK uses the gateway plus the mount as its base URL.

providers:
  openai-compatible:
    upstream_url: "https://api.mistral.ai"
    path_prefix: "/v1"                    # where the host serves its API (default)
    mount: "/mistral/v1"                  # where this gateway serves it (default: path_prefix)
    models:                               # optional
      - "mistral-large-latest"
      - "codestral-latest"

Only one provider can own a path. OpenAI is configured on /v1 by default, and Azure's v1 surface answers on /openai/v1 when enabled. So a compatible host that serves either layout needs a mount of its own. On a collision, the gateway refuses to start rather than guess which upstream a prompt belongs to. Leave mount unset to serve the host on its own prefix, when nothing else owns it.

One gateway serves one compatible host; a second host is a second gateway deployment. Agents keep their own key for the host. The gateway forwards the Authorization header as-is (or none, for a host that needs none), and only changes where the traffic flows.

Models are governed by the name the request uses. A call is authored as llm:send:openai-compatible:<model>, for example llm:send:openai-compatible:mistral-large-latest. The usage record, the LLM BOM and the dashboard carry openai-compatible:<model> exactly as the request spelled it.

A body that names no model is refused (403), and the refusal is recorded, so you can see which SDK call to fix. With a models list, a call that names a model outside it is refused the same way, and the model is recorded so that the list can grow.

Model ids that contain a / (meta-llama/Llama-3.3-70B-Instruct-Turbo, anthropic/claude-3.5-sonnet) need a rule that crosses it. A glob * does not, so write llm:send:openai-compatible:*/* or a regex: rule.

Cost is counted, not estimated. The price catalog carries no entries for these hosts, because list prices differ per host and per plan. So their calls are metered (tokens, budgets and rate limits all apply) and counted in uncosted_calls on the usage summary, rather than priced. Token budgets are exact; a USD budget does not see these calls.

A self-hosted vLLM must be reachable over TLS. Terminate TLS in front of vLLM (an ingress, a mesh sidecar) rather than pointing the upstream at plaintext: the gateway refuses an http:// upstream that is not a loopback test stub.

This is not interception: the agent connects to the gateway by name and verifies the gateway's certificate against the gateway's CA, like any other HTTPS endpoint.

Agents hold no provider keys

By default, an agent authenticates to its provider with its own API key, and the gateway forwards that key as-is (auth: passthrough). A key that an agent holds is a key that can leak, through a prompt, a tool result, a log line or a compromised container.

Brokered keys remove the key from the agent altogether. Under auth: brokered, on each call, the gateway:

• resolves the provider key from the MITRITY credential broker, as a credential definition granted to the agent's mission, exactly as the MCP gateway resolves ${credential:<id>} in tool arguments;
• injects the key into the provider's credential header;
• strips every credential header that the agent presented.

The agent configures its SDK with any placeholder key, or none. The provider only ever sees the brokered one.

providers:
  openai:
    auth: brokered
    brokered_key:
      value: "Bearer ${credential:openai-prod}"   # header defaults to Authorization
  anthropic:
    auth: brokered
    brokered_key:
      value: "${credential:anthropic-prod}"       # header defaults to x-api-key
credentials:
  injection_enabled: true

openai-prod and anthropic-prod are credential definitions in the dashboard, granted to the mission that the calling agent runs under. Rotate the value there, and the gateway switches to the new one at its next heartbeat, within 30 seconds. Remove the grant, and the agent's calls are denied within the same 30 seconds. The agent never had the key, so there is nothing to chase.

The header defaults to what each provider reads (x-api-key for Anthropic, api-key for Azure OpenAI, Authorization elsewhere), and you can set it explicitly. The value template may carry several placeholders. A literal key in the template is refused, because a key in configuration is what brokering exists to avoid.

What the gateway guarantees under brokered mode:

• The key is resolved only for an allowed call. Resolution happens after the policy verdict and after DLP redaction, so a denied call never touches the broker.
• Resolution failure denies. A credential that the broker does not resolve for the calling agent (not granted, revoked, broker unreachable) is a 403 in the provider's error envelope. Nothing is forwarded, neither without a credential nor with the raw placeholder. The refusal is recorded on the audit trail with the credential id (credential_unresolved), never a value.
• The value goes to the provider and nowhere else. It appears in no log line, no audit event and no response.
• Every usage record says which mode the call ran under. auth_mode is passthrough or brokered on every LLM usage record, so you can tell which agents still hold their own keys.

Brokering is per provider. Leave a provider on passthrough, and its agents keep forwarding their own keys, next to a brokered provider on the same gateway. Brokered mode needs credentials.injection_enabled: true on the gateway, which wires the broker. With a brokered provider and no broker, the gateway refuses to start.

Model aliases and failover

A model alias is a name your agents send as their model, such as fast or support-default, that the gateway serves from an ordered list of upstreams. It lets you move agents between models, regions or providers without changing them, and keeps them answering when one provider is down or rate-limiting.

Model aliases are part of the Enterprise plan. On another plan, adding or changing an alias is refused with 402 and the code plan_upgrade_required. Aliases you already have are kept, and you can still list and delete them, but your gateways don't receive them: every call goes to the provider it addresses, with the model it names. An agent that sends an alias of its own, such as fast, then gets that provider's error for an unknown model. When your organization moves to Enterprise, your gateways receive its aliases on their next heartbeat, within about 30 seconds.

You define aliases with the /llm-routes API, for your whole organization or for one policy. A policy's alias overrides the organization's alias of the same name for that policy's agents. An alias can also use the name of a real model, such as gpt-4o: agents that already ask for that model then get failover with no change on their side. Each alias lists up to eight members. A member is a target and a model:

{
  "alias": "fast",
  "strategy": "weighted",
  "members": [
    {"target": "openai",   "model": "gpt-4o-mini", "priority": 1, "weight": 100},
    {"target": "azure-eu", "model": "gpt-4o-mini", "priority": 2}
  ],
  "failover": {"max_attempts": 2, "retry_on": ["server_error"]}
}

Targets are the upstreams the gateway itself declares, so an alias can choose among them but never add one:

• Every entry under providers is a target named by its provider key (openai, anthropic, ...).
• targets declares more: another region or account, or an OpenAI-compatible host. A target takes its provider's own settings plus provider and optional labels. It has no request path of its own: agents reach it only through an alias.

providers:
  openai: {}
targets:
  azure-eu:
    provider: azure-openai
    upstream_url: "https://contoso-eu.openai.azure.com"
    auth: brokered
    brokered_key:
      value: "${credential:azure-eu}"
    labels:
      region: eu
routing:
  enabled: true        # off by default: every call is then served by the provider it addresses
  max_attempts: 3      # 1-5; an alias may ask for fewer

How a call with an alias is served:

• Members are tried by priority, lowest first. Within one priority, weighted spreads calls by weight (300 against 100 is three to one), and latency tries the fastest upstream first.
• A request is only ever sent within its own API: an OpenAI Chat Completions call goes to OpenAI, Azure OpenAI (the v1 surface), Amazon Bedrock's OpenAI-compatible surface or an OpenAI-compatible host, and an Anthropic Messages call to Anthropic. Only the model field of the body changes. Vertex AI and Azure's deployment URLs name the model in the path, so they are not used for aliases.
• Every member a call is sent to is governed as that call: the model allowlist, the destination allowlist, DLP, budgets and holds all apply to it. A member your policy refuses is skipped, and the call goes to the next one. When the policy refuses every member, the call is denied exactly as it would be without the alias.
• Each member gets its own credential. Under auth: brokered, every target uses its own brokered key. A target under auth: passthrough receives the agent's own key, so it is used only when it is the provider the agent addressed, or when both carry the same credential_scope, your statement that they accept the same key.

When the next member is tried:

• Always on a connection failure, a 429, or a 502, 503 or 529. The provider never processed the request, so trying elsewhere costs nothing.
• On a 500 or 504, a timeout (failover.attempt_timeout_ms) or a 401/403, only when the alias lists the class in failover.retry_on (server_error, timeout, auth_error). These can come after the provider processed the prompt, so trying again elsewhere may be billed twice.
• Never on another 4xx, such as a content-filter refusal. The same request fails the same way everywhere, and retrying a refusal on a less strict provider would get around your controls.
• Never after a streamed response started. A stream that answered 2xx belongs to the agent. A failed attempt is never shown to the agent: when every member fails, the agent gets the last provider's error.

The gateway caps attempts at the smaller of the alias's and its own max_attempts, and in any 10 seconds spends no more retries than 20% of its calls plus 30, so a failing provider cannot multiply your traffic.

Every usage record names the target that served the call (llm_target). For a call made with an alias it also names the alias (llm_route), the number of attempts and the path (llm_failover_path, for example openai=unavailable,azure-eu=ok). The model on the record is the model that actually served the call, so budgets and cost estimates count what you paid for.

Provider health

The gateway measures every upstream from the calls it sends, with or without aliases:

• failures by kind, over the last 5 minutes;
• latency, the time until the provider answers, which for a call that is not streamed includes the whole generation;
• the models a provider is rate-limiting.

Each target is reported as healthy, degraded (at least 20% of recent calls failed, or latency is over health.latency_degraded_ms), auth_error (the last call was refused 401 or 403), unreachable (the last probe failed) or ejected.

A target is ejected after 5 failures in a row, or when at least half of 20 or more calls in 30 seconds failed. An ejected target goes to the back of the line for 30 seconds, then gets one trial call. The wait doubles each time it fails again, up to 5 minutes. A 429 moves just that model of that target to the back until the provider's Retry-After passes. Health only changes the order of an alias's members; it never refuses a call, so a call whose every member is ejected still tries them. Health is measured and reported on every plan, but without model aliases, ejection, throttling and probes don't change where a call goes.

Optional probes (health.probes.enabled) send each target one GET to its origin every health.probes.interval (30 seconds by default), with no credential, no body and no query, to spot an idle upstream that cannot be reached. A probe never ejects a target or brings it back; only real calls do.

Each gateway reports this with every heartbeat. The /llm-gateway/targets API returns every gateway's targets with their state, counts, latency and last probe, merged across replicas, plus the alias members a gateway could not use and why (for example a target it does not declare).

Identity: how the gateway knows who is calling

There are two identity paths, and the gateway fails closed when neither yields an identity:

1. SPIFFE via mTLS (primary). When the agents run in an Istio mesh, the gateway requires and verifies a client certificate on the TLS handshake. The verified certificate's spiffe:// identity (namespace + ServiceAccount) resolves the calling agent, against the same control-plane binding that the Mesh Authorizer uses.
2. MITRITY API key (fallback, for non-mesh agents). The agent presents its MITRITY-issued key in a configured header, and the value resolves against the same control-plane index. The header is never forwarded to the provider and never logged. Agents on this path cannot present a client certificate, so client-certificate verification must be relaxed for them at install time.

A valid identity that maps to no governed agent is denied by default. An allow mode exists strictly as a migration escape hatch. An absent identity is always denied, regardless of configuration.

Mesh mode: one hop governs the whole fleet

The gateway runs in one of two identity modes, set by llmGateway.identityMode:

• mesh (default): the centralized, N-to-1 egress hop. A single gateway Service governs every agent in the environment. It resolves each caller per request (SPIFFE mTLS, or the API-key fallback) against a (namespace, ServiceAccount) → agent index. Then it enforces that agent's policy: its model allowlist, token caps, budgets, rate limits and DLP. This is the mode for the shared in-cluster LLM-egress hop. It is how mesh customers get the body-level LLM governance that the Mesh Authorizer cannot provide, because the Mesh Authorizer is L3/L4 and never sees the prompt.
• colocated: the 1-to-1 sidecar. The gateway serves a single configured agent, and trusts a same-pod loopback call as that agent. It is used when a gateway is deployed next to exactly one workload.

Mesh mode needs a mesh-edge credential. To govern the fleet, the gateway must hold every served agent's policy, and it gets those policies from its own heartbeat. So in mesh mode, llmGateway.agentId must be your environment's mesh-edge credential: an agent that you create with is_mesh_edge: true.

A mesh-edge agent's heartbeat returns the mission profiles of every mesh-bound agent in its environment (fleet provisioning). That is what populates the resolution index. If you point the gateway at a plain agent, it receives only that one profile, so every other caller resolves to nothing and is denied.

Delivery is scoped to the mesh edge's own environment, for least privilege. The flag can be set only through the agent API, never by an agent that registers itself. This is the same fleet-provisioning mechanism that the Mesh Authorizer uses to serve many agents from one deployment.

Once the gateway is installed, confirm that the fleet is flowing: the gateway logs heartbeat synced mesh fleet with the mesh-bound profile count. A mesh gateway that never logs it is almost always pointed at a non-mesh-edge credential.

Response DLP and streaming

Response-phase DLP scans the text that the model generates, before it reaches the agent. It complements the request-phase prompt scan. It is on by default.

• Non-streaming responses are buffered whole and scanned, then blocked, redacted or delivered. This is fail-closed: a model output that the gateway cannot fully read, parse or (after a redact match) rewrite is never delivered.

• Streaming (SSE) responses use a holdback-window scan. The gateway withholds the last few kilobytes of generated text from the agent, so that a secret straddling chunk boundaries can still be matched, redacted or blocked before delivery. Stream framing (roles, tool calls, usage frames, pings) passes through untouched.

• Thinking is inspected, never rewritten. A model's reasoning is signed by the provider, or replayed by the agent on the next turn as if it were. Examples are Anthropic thinking blocks, Vertex thought parts and the reasoning_content that an OpenAI-compatible host returns. So the gateway cannot scrub reasoning the way it scrubs text. It scans it instead, on every leg:

  • in a prompt that replays a thinking block;
  • in a buffered response;
  • in a stream, where thinking frames are held back whole under the same window, and released byte for byte once the thinking behind them has been scanned.

  A DLP pattern that would redact thinking denies the call or terminates the stream instead, and nothing partial is delivered. A block pattern blocks and a flag pattern records, as usual. Anthropic's redacted_thinking blocks are ciphertext and pass through uninspected.

Streaming holdback: the bounded-window caveat

The holdback window is bounded (default 2 KiB, configurable from 256 bytes to 1 MiB), so that the gateway never buffers an entire stream. As a consequence, a single DLP match longer than the window that straddles the window edge can escape the streaming scan. Size the window above your longest expected pattern match; a typical credential or PII token is far under 2 KiB.

Non-streaming responses are unaffected, because they are always scanned whole. Raising the window trades a little time-to-first-token latency for a wider straddle guarantee.

Privacy guarantee

Prompts and model responses never leave your cluster, except to the LLM provider that you explicitly configured as the upstream. The gateway evaluates everything locally, with the same in-process decision engine as the other MITRITY binaries. Only two things reach the MITRITY control plane:

• verdict metadata: decision, policy and pattern IDs, provider, model, redaction counts;
• token counts.

Each gateway also reports the health of its upstreams: per-target state, counts and latency. Prompt and response content is never included in events, logs, health reports or the dashboard.

The agent's provider API key is forwarded only to the provider, or, under brokered mode, replaced by a key that the agent never held. The MITRITY identity header (API-key fallback) is never forwarded upstream and never logged.

Fail-closed behavior

The gateway is inline on the agent-to-provider path, and it fails closed:

• A request with no resolvable identity, or an identity bound to no governed agent, is denied.
• A mission-scope, DLP-block or threat-intel verdict on the prompt denies the call. The error is provider-shaped, in that provider's own JSON envelope, so agent SDKs surface it cleanly.
• A policy verdict that requires human approval is denied at the wire, because a synchronous proxy has no pending state. Approve the request in the dashboard approval queue, and the agent retries.
• Gateway down = governed traffic down. Agents whose base_url points at an unavailable gateway cannot reach the provider at all. That is the fail-closed contract. Run at least two replicas (the default), and keep them close to the agents.

LLM usage, cost & BOM

Each governed call emits a usage record, keyed to the calling agent. The record holds the provider, the model, the input/output/total token counts, the decision (allowed, blocked, or redacted) and the redaction count. These records power the LLM Gateway page, under Monitoring in the dashboard sidebar:

• Per-call usage records: every LLM Gateway-governed call, filterable by agent, provider, model, decision and time range. A blocked call is recorded with zero tokens, because it was never forwarded.
• Token spend trends and estimated cost: usage aggregated by model, agent or day. estimated_cost_usd is computed from a static price catalog keyed by (provider, model) that ships with platform releases. Prices are estimates, never billing data. A group's estimate is null when no call in it matched a catalog entry, and uncosted_calls counts the calls excluded for that reason.
• LLM Bill of Materials (BOM): a per-agent model inventory, with one entry per distinct (agent, provider, model) ever observed through the gateway. Each entry has first-seen/last-seen timestamps and lifetime call and token totals. The BOM answers "which agents talk to which models", the same way the AI Bill of Materials answers it for tools.

The same usage numbers drive the per-policy cost budgets, the burn-down panel and the spend alerts in Guardrails. What the gateway measures is also what it enforces.

To check that those guardrails hold under pressure, and not only in configuration, point a Red Teaming campaign at the route. It sends adversarial attempts through this gateway, on a budget that you set. It records the attempts that the guardrails let through as findings, each with a remediation.

No prompt or response content is ever included. Usage records are metadata only, consistent with the privacy guarantee above. The same data is available programmatically through the /llm-usage API endpoints.

Tracing with OpenTelemetry

The gateway can trace every call it governs and send the spans to your own OpenTelemetry collector over OTLP/HTTP. Tracing is off until you turn it on:

observability:
  tracing:
    enabled: true
    endpoint: "https://otel-collector.observability:4318"   # spans go to <endpoint>/v1/traces
    sample_ratio: 0.1          # share of calls traced when the agent sent no sampled traceparent
    propagate_upstream: true   # send the provider the gateway's traceparent
    headers_file: ""           # optional collector auth, one key=value per line, under /run/secrets or /etc/mitrity

An http:// endpoint also needs insecure: true. The standard variables OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME and OTEL_TRACES_SAMPLER_ARG override the file.

Each call is one trace:

• llm_gateway.request: the whole call, as the agent saw it;
• llm_gateway.admission: identity, limits, budget and the policy decision;
• llm_gateway.redaction: the DLP rewrite of the prompt, when it changed something;
• llm_gateway.upstream: one span per provider attempt, with the target, the attempt number and how it ended;
• llm_gateway.stream or llm_gateway.response: reading and delivering the answer.

Spans carry provider and model names, token counts, status codes, the decision and a decision code, the agent's id and, for an alias, the alias and its target. Spans never carry prompt or response text, header values, query strings, request paths or credentials. Exporting never slows or fails a call: when the collector is down or slow, spans are dropped and counted.

The gateway joins the agent's trace when the request carries a valid W3C traceparent, and passes its own on to the provider. With propagate_upstream: false the provider receives neither traceparent nor tracestate. Usage records carry the call's trace_id, so a record leads to its trace. A caller that sends its own traceparent chooses that id, so treat it as a lookup aid, not as evidence.

Deployment

The Mitrity LLM Gateway ships as a Helm chart, published as a cosign-signed OCI artifact with every MITRITY edge release:

helm install mitrity-llm-gateway \
  oci://ghcr.io/mitrity-io/charts/mitrity-llm-gateway --version <X.Y.Z> \
  --namespace mitrity-system --create-namespace \
  --set llmGateway.apiKey=$MITRITY_API_KEY

Key facts:

• Available from MITRITY edge release v0.13.0. The chart version tracks the release tag (vX.Y.Z → chart version X.Y.Z).
• Off by default. The gateway capability is flag-gated in the edge binary, and installing the chart is the opt-in.
• Mesh mode wants a mesh-edge credential. In the default identityMode: mesh, set llmGateway.agentId to an agent created with is_mesh_edge: true, so that its heartbeat provisions the fleet (see Mesh mode above). In mesh mode, startup fails fast if agentId is empty.
• cert-manager provisions the gateway's own server certificate on the default TLS path; you can bring your own TLS Secret instead. Every hostname that agents will use in base_url must be on the certificate.
• Istio is optional. The gateway does not require a mesh. When agents run in an Istio mesh, issue the gateway's certificate from the mesh trust domain, so that in-mesh agents present SPIFFE client certificates automatically. Agents need no changes beyond the base URL.
• Secure by default. The gateway refuses to open a plaintext listener on a non-loopback address. An egress gate that sees prompt bodies never runs unencrypted on the network.

The chart README packaged with the release covers the full install, provider configuration, certificate setup, a troubleshooting table and signature/SBOM verification.

Related Documentation

• Integration Modes: Mitrity Gateway and Mitrity MCP Sidecar
• Mesh Enforcement (Istio): the in-mesh governance surface and its capability table
• Per-Agent Identity in the Mesh: the namespace + ServiceAccount binding that the SPIFFE identity path uses
• API Overview: the /llm-usage endpoints for usage records, summaries and the BOM, /llm-routes for model aliases and /llm-gateway/targets for upstream health
• Red Teaming: governed adversarial campaigns that test these guardrails on a live route
• Microsoft 365 Copilot and Copilot Studio: the connector that discovers, inventories and posture-assesses the Microsoft agents that never leave Microsoft