Docs / Gateway・Sidecar・メッシュ / Host Observer

Host Observer

The coverage page lists the execution paths that do not cross a MITRITY decision point: an agent that shells out from its own code, a framework tool that is not hooked, a script that keeps running after the tool call returned. None of those produce a decision, so none of them appear in the audit log on their own.

mitrity-host-observer covers those paths on Linux hosts. It watches every process the kernel executes, decides — deterministically, from process ancestry — whether that execution passed through a MITRITY binary, and reports the ones that did not as findings. The rule it is built on:

An execution is governed if and only if its process ancestry passes through a MITRITY entrance. Ungoverned execution is a finding, not a heuristic.

It adds no new decision point and makes no decisions of its own. It detects, after the fact, and never prevents: it loads no eBPF program, holds no capability and blocks nothing. Every sentence in the dashboard about it says "detected", never "prevented".

What it sees

The observer consumes Cilium Tetragon's event stream on the host — over Tetragon's gRPC socket, or from its JSON export file where the socket is not mounted. Three event types matter:

Event	What the observer takes from it
`process_exec`	The execution: pid, in-container pid, Tetragon's process identity, the binary, the arguments, `cwd`, uid, container and pod identity, and the ancestor chain. Everything else is built on this.
`process_exit`	Marks the ancestry-cache entry exited. Nothing is shipped for it.
`process_kprobe`	The file and network sensors, when the accompanying `TracingPolicy` is installed: writes, reads, renames, unlinks and `chmod` on the sensitive path set, and TCP connects.

Tetragon records the fork-time parent, so a process that calls setsid, double-forks and reparents to pid 1 keeps its lineage on every later event. That property is what the whole attribution model rests on: daemonizing does not launder an execution.

One observer runs per host — a DaemonSet pod per node, a systemd unit per VM, a compose service in the demo. It sees every process on the host: every agent's, and everything that is not an agent.

How attribution works

Each edge (mitrity-gateway, mitrity-mcp-sidecar) reports its own process identity on its heartbeat: the host's boot id, its container id, its own pid, the pid of the agent process it serves, the pids of the upstream servers it spawned, the hash of its binary, and the workspace roots it is enforcing. The observer receives those identities on its own heartbeat as the roster of its environment, and classifies every execution against it:

Classification	Meaning
`governed`	An ancestor of the process is the agent's gateway or sidecar (matched by attested pid, or by name confirmed against the roster), an attested upstream, the process is itself a corroborated MITRITY entrance — or the control plane joined it to an admission decision (below).
`ungoverned`	The process is inside an agent's process tree — under the attested agent pid or in its container — and no entrance is in its ancestry. This is the finding.
`unattributed`	Outside every known agent tree, or the chain could not be completed. Recorded when a runtime rule matches, and always when a MITRITY entrance the roster could not name or confirm is in its chain (`mitrity_ancestor`) — a look-alike entrance must not hide the tree below it; otherwise dropped on the host. An incomplete chain is never reported as `ungoverned` — the unseen link might have been the entrance.

Two details make the classification reliable rather than merely plausible:

A name is not proof. An agent can name any binary mitrity-gateway, or exec a copy of the real one. An entrance seen by name is held against the roster entry for its container: a binary hash the observer can read and that differs from the attested one refutes it, and the attested gateway pid decides — an entrance that does not match the pid its roster entry attests is unconfirmed, unattributed, never governed (a copy of the genuine binary hashes the same), and the observer asks the control plane for a fresh roster so a real restart is confirmed within seconds. Only when nothing can hold the name to account — no attested pid, or a runtime where pids cannot be compared — is the name believed. A non-entrance MITRITY binary (mitrity-hook, mitrity-llm-gateway, mitrity-extauthz, the observer itself) never attributes on its name.
The admission join. A hooked Bash on Claude Code is decided by the admission API and then executed by the framework itself, not by the gateway — by ancestry alone every admitted command would read ungoverned. So the control plane joins an ungoverned execution to an allowed admission decision for the same agent when the digest of an inline shell body in the tree equals the decision's digest and the decision is within five minutes before the execution. Each decision is consumed by exactly one execution: a command re-run after a single admission is ungoverned, which is the correct classification. With builtin_exec_routing: governed_shell, the command runs under the gateway and is governed by ancestry — no join needed.

Framework helpers get an exemption at the rule, not at the classification: rg, git and mitrity-hook spawned directly by a Claude Code or Claude Agent SDK agent process (the framework runs the admission hook from the agent process for every tool call, so by ancestry that exec is ungoverned — and it is the one MITRITY binary that is legitimately there), with no execution feature in the command and every path inside the workspace, are stored ungoverned with helper: true, excluded from the ungoverned_execution finding and from the 24-hour count, and still visible in the event stream. Every other rule still applies to them.

What leaves the host

The control plane learns the shape of what ran, never your data. Before an event is queued, in this order:

Secret-flag masking. The value of any --password=, --token=, --api-key=, --secret=, --credential=, --auth= style flag (and the argument after the bare form) becomes [REDACTED:flag]; Authorization: and Bearer values lose the credential.
DLP. Every argument passes the edge's built-in DLP pattern set plus your tenant's own regex patterns, delivered to the observer on its heartbeat. A match becomes [REDACTED:<category>].
Canonicalization. The redacted argv is reserialized with the same shell grammar the command-analysis pipeline uses, producing the canonical command, its digest, the base commands, the execution features, the resolved paths and the inline-body digests the admission join compares.

What is on the wire per event: the canonical command, argv0 and base, the resolved paths, the features, the classification and its basis, an ancestry summary (basenames and roles, nearest first), container and pod identity, uid, cwd, the binary path and timestamps. An event over the 16 KiB per-event cap sheds detail in a fixed order rather than disappearing, and every cut is flagged (paths_truncated, detail_shed for script paths, bases and features, ancestry_truncated, canonical_truncated) so a cut is never read as an absence — a workload can choose when shedding happens by padding its arguments. The argv itself is treated the same way: one cut by Tetragon or by the observer's caps is flagged argv_truncated, and one the observer could not decode unambiguously (an argument carrying a quote) is flagged argv_ambiguous — its canonical command, digest, paths and features then describe one possible split, so the control plane joins no admission on it, never treats it as a framework helper, and records a command-pattern match on it at low confidence. Either way the event is sent unsampled, never dropped.

Never on the wire: environment variables (the observer never even requests the field), file contents, tool output, the raw arguments string, script bodies, capability or credential sets, Kubernetes labels and annotations. The observer's own logs never carry argv, canonical text, paths or its API key.

Governed executions are sampled (one in twenty per agent by default, plus the first of every entrance session and every rule match), each carrying the count it stands for, so the totals stay exact while a busy build does not flood the pipeline. Ungoverned executions are always sent. Everything the observer could not send — a full queue, a per-agent rate cap — is counted and reported on the next batch, and the Runtime page shows the drop count per host: a gap is always visible, never silently rendered as coverage.

Rules and findings

The control plane evaluates eight built-in rules on every stored ungoverned and unattributed event. Governed events are exempt — their governance happened at the entrance. Every rule records and notifies; none blocks.

Rule	Severity	Matches when
`ungoverned_execution`	high	Always, for a non-helper `ungoverned` exec. The classification is the finding.
`destructive_outside_workspace`	high	`rm` with a recursive flag or `--no-preserve-root`, `find … -delete` or `shred`, with a target outside every workspace root the edge enforces. No roots ⇒ every path qualifies; `/`, `~`, `$HOME` and a root-level glob always qualify.
`remote_fetch_to_shell`	critical	`curl` or `wget` feeding a shell — a pipe, a process substitution `<(curl …)`, or a command substitution as the inline body.
`setuid_bit`	high	`chmod` or `install -m` setting `u+s`, `g+s` or the `4000`/`2000` bits; or the `chmod` sensor seeing `S_ISUID`/`S_ISGID`.
`autorun_write`	high	A write, rename, unlink or `chmod` on an autorun location — cron tables, systemd units, shell rc files, `.git/hooks`, `package.json` lifecycle scripts, `.claude/settings*.json`, `.mcp.json` — by a file sensor, or by `crontab`, `systemctl enable`, `tee`, `cp`, `mv`, `install` or a redirect.
`non_allowlisted_connect`	medium	A TCP connect to a public address not covered by the agent's DLP destination allowlist. Only evaluated when the allowlist is non-empty and wildcard-free; a wildcard entry makes the rule undecidable and the summary says so.
`credential_file_read`	high	A read of `/etc/shadow`, `/etc/sudoers`, `~/.ssh/id_`, `~/.aws/credentials`, `~/.config/gcloud/*`, `~/.kube/config`, `~/.docker/config.json`, `~/.netrc`, `~/.git-credentials`, `~/.npmrc`, `~/.pypirc` or any `.env` file — by the read sensor, or by a command whose paths include one.
`command_pattern_match`	the indicator's	An enabled `command_pattern` threat indicator visible to your tenant matches the canonical command — the same matcher the edge uses, with your suppressions and severity overrides applied.

Two host-level checks are findings without a rule: observer_offline (medium) when an observer that reported within the last 24 hours has been silent for 10 minutes — an attacker with root can stop the observer and Tetragon, so unexplained silence is treated as an incident — and sensor_unavailable (medium) when a sensor the installed TracingPolicy declares fails to load for two consecutive heartbeats. Rules that depend on a missing sensor are not evaluated on that host, and the Runtime page names them per host rather than rendering them as clean.

A finding has a lifecycle: one open finding per dedup key (the agent or host, plus the path, destination or indicator the rule names). A recurrence increments its occurrence count, moves its last-seen time and refreshes its examples; a finding resolved by an owner or manager — or automatically, for the two host-level checks — that recurs opens a new finding, so a repeat offender's history is a list, not one mutable row. Each carries a fixed remediation.

Three notification kinds ride on the notification engine — routed per tenant under Settings → Notifications (Slack, email) and muted per user from the bell: runtime.ungoverned_execution (once per agent per 15 minutes, with up to five example commands), runtime.rule_match (once per agent or host and rule per hour; critical for critical and high rules) and runtime.observer_offline (once per host per episode). Each links to the finding.

The Runtime page and the coverage badge

Runtime in the dashboard has three tabs:

Overview — every host that reported in the last 24 hours: whether its observer is present (a heartbeat within 5 minutes), its status, Tetragon and observer versions, the sensors and their state, which rules are not being evaluated there and why, the agents rostered on it, and the drop count. Below it, every agent with a runtime context: governed, ungoverned, helper and admitted execution counts for the window, the last event, and open findings by severity.
Events — the stream, newest first, with the classification and its basis, the canonical command, path or destination, and the rules that matched. Helper executions are hidden by default. Expand a row for the process, the ancestry and the exec detail.
Findings — open by default, highest severity first, with the message, the remediation and the code-specific detail. Owners and managers resolve a finding with a note.

On the agent page, the Execution coverage card's observed axis gains the observer row: present or silent, the last event, and the number of ungoverned executions in the last 24 hours — and the Runtime findings card lists the agent's open findings. The observer feeds two coverage findings: observer_ungoverned_execution (high — the badge reads degraded while the 24-hour window holds an ungoverned exec) and observer_offline (medium — partial while the host's observer is silent). Presence on its own never improves a badge: the observer corroborates the attested axis, it does not replace it, because an observer sees process trees, not intent.

Installing it

Prerequisites

Linux with a BTF-enabled kernel — 4.19 or newer; CONFIG_SECURITY_PATH=y for the file sensors (true of every mainstream distribution kernel).
Tetragon 1.3 or newer, installed by you, with ancestors enabled so every event carries the full chain, and process namespaces enabled so an edge that cannot read its own container id — every container on cgroup v2 Kubernetes — is joined by its mount namespace: helm install tetragon cilium/tetragon -n kube-system --set tetragon.processAncestors.enabled=base,kprobe --set tetragon.enableProcessNs=true. Tetragon 1.1 and 1.2 work in a degraded cache mode where chains that predate the observer's start are incomplete and read unattributed; the host shows as degraded. Without --enable-process-ns, such an edge's tree reads unattributed and the observer warns once.
An observer credential: an agent whose is_runtime_observer is true, created through the tenant API (POST /agents with "is_runtime_observer": true; the flag is not on the agent form) — never through edge self-registration. One per environment serves any number of hosts; it needs no policy, no mission scope and no mesh identity. Its key authenticates the observer, and it decides the roster: the agents in its environment are what the observer can attribute. An agent from another environment on the same host is unattributed.

Cloud Run and other serverless runtimes have no kernel access, so the observer does not apply there; the coverage page states that the observer axis is absent for such agents rather than implying otherwise.

Kubernetes (Helm)

The chart runs a DaemonSet on every Linux node, mounts Tetragon's socket directory read-only from the host, and installs the TracingPolicy for the kprobe sensors when tracingPolicies.enabled is true. No hostPID, no privileged, no capabilities, a read-only root filesystem.

helm install mitrity-host-observer oci://ghcr.io/mitrity-io/charts/mitrity-host-observer \
  --namespace mitrity-system --create-namespace \
  --set observer.agentId=<observer-agent-id> \
  --set observer.existingSecret=mitrity-observer-key   # a Secret with key `api-key`

Value	Default	Meaning
`observer.agentId`	— (required)	The observer credential's agent id.
`observer.apiKey` / `observer.existingSecret`	—	The key inline (the chart creates a Secret) or an existing Secret with key `api-key`.
`observer.controlPlaneUrl`	`https://api.mitrity.com`
`observer.heartbeatInterval`	`30s`
`observer.governedSampleEvery`	`20`	Governed sampling rate; `1` sends every governed exec.
`tetragon.source`	`grpc`	`grpc` or `export_file`.
`tetragon.hostSocketDir`	`/var/run/tetragon`	Mounted read-only at the same path.
`tracingPolicies.enabled`	`true`	Install the `TracingPolicy` for the file and connect sensors (needs Tetragon's CRDs).
`networkPolicy.enabled`	`false`	Egress to the control plane and DNS only.
`securityContext.runAsUser`	`0`	Tetragon's socket is root-owned; the observer runs as uid 0 with every capability dropped. To run non-root, point Tetragon at a group-readable socket directory and set this.

The image is ghcr.io/mitrity-io/mitrity-host-observer:vX.Y.Z (distroless, cosign-signed); the raw binary is published as an OCI artifact at ghcr.io/mitrity-io/binaries/mitrity-host-observer:vX.Y.Z for oras pull.

Docker Compose

The demo bundle runs Tetragon privileged on the host pid and cgroup namespaces, and the observer unprivileged next to it, sharing the socket volume:

services:
  tetragon:
    image: quay.io/cilium/tetragon:v1.7.0
    privileged: true
    pid: host
    cgroup: host
    volumes:
      - /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf:ro
      - tetragon-run:/var/run/tetragon
      - ./tetragon/tetragon.conf.d:/etc/tetragon/tetragon.conf.d:ro   # enable-ancestors: base,kprobe · enable-process-ns: true
      - ./tetragon/tetragon.tp.d:/etc/tetragon/tetragon.tp.d:ro       # the observer's TracingPolicy
  mitrity-host-observer:
    image: ghcr.io/mitrity-io/mitrity-host-observer:latest
    depends_on: [tetragon]
    environment:
      MITRITY_OBSERVER_AGENT_ID: ${OBSERVER_AGENT_ID}
      MITRITY_OBSERVER_API_KEY: ${OBSERVER_API_KEY}
      MITRITY_OBSERVER_CONTROL_PLANE_URL: https://api.mitrity.com
      MITRITY_OBSERVER_HOST_ID: demo-host
    volumes:
      - tetragon-run:/var/run/tetragon:ro
    cap_drop: [ALL]
    read_only: true
    tmpfs: [/tmp, /run/mitrity]                 # /run/mitrity holds the health socket
volumes:
  tetragon-run: {}

Your agent and gateway containers need no change: the edge derives its container id from its own cgroup, and the observer sees the same id from Tetragon.

systemd (VMs)

Install Tetragon from its release tarball with ancestors enabled and the observer's TracingPolicy in /etc/tetragon/tetragon.tp.d/, put the binary at /usr/local/bin/mitrity-host-observer, the configuration at /etc/mitrity/host-observer.yaml and the key in /etc/mitrity/host-observer.env (mode 0600):

[Unit]
Description=MITRITY host observer
After=network-online.target tetragon.service
Wants=network-online.target
Requires=tetragon.service

[Service]
Type=simple
ExecStart=/usr/local/bin/mitrity-host-observer --config /etc/mitrity/host-observer.yaml
EnvironmentFile=-/etc/mitrity/host-observer.env
Restart=always
RestartSec=5
User=root
CapabilityBoundingSet=
AmbientCapabilities=
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=yes
RuntimeDirectory=mitrity
RuntimeDirectoryMode=0700
ProtectKernelTunables=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
ReadOnlyPaths=/var/run/tetragon /var/log/tetragon

[Install]
WantedBy=multi-user.target

ProtectHome=read-only is required: on a VM the observer verifies entrance executables under home directories. RuntimeDirectory=mitrity creates /run/mitrity for the health socket, which ProtectSystem=strict would otherwise keep read-only.

Configuration essentials

agent_id: "<observer-agent-id>"
api_key: "${MITRITY_OBSERVER_API_KEY}"      # from the file or the environment — never on the command line
control_plane:
  base_url: "https://api.mitrity.com"
observer:
  heartbeat_interval: "30s"
  tetragon:
    source: "grpc"                            # grpc | export_file
    address: "unix:///var/run/tetragon/tetragon.sock"
  sampling:
    governed_every: 20
  unattributed: "rule_matches"                # rule_matches (sent on a rule match, and always with mitrity_ancestor) | drop (never sent)
  health:
    listen_addr: "unix:/run/mitrity/host-observer.sock"   # /healthz /readyz /metrics — a 0600 socket; loopback TCP only when set explicitly

Every key has a default and a bound; a value outside its bounds is a start-up error naming the key, not a clamp. The environment prefix is MITRITY_OBSERVER_. The health listener is a Unix socket by default (unix:/run/mitrity/host-observer.sock, mode 0600, in a directory the observer creates when it is missing — a systemd RuntimeDirectory, a container tmpfs); loopback TCP (127.0.0.1:9477, what the Helm chart pins, private to the pod's network namespace) only when set explicitly, and a relative socket path or any other address refuses to start. control_plane.base_url must be https://; allow_insecure: true permits plaintext only to a private host (a cluster service, an RFC 1918 address) for a stub control plane, never a public one. The observer also refuses a Tetragon address that is not a Unix socket or loopback: Tetragon's gRPC API is unauthenticated, and the observer will not read it across a network.

What it does not do

Stated plainly, because a control that quietly misses something is worse than one that names the gap:

It does not prevent anything. No kill-on-match, no synchronous kernel verdict, no Sigkill/Override action. Detection latency is the batch flush (5 seconds) plus ingest; the promise is "flagged within 30 seconds".
Linux only. macOS and Windows hosts, and serverless runtimes without kernel access — Cloud Run, Lambda — are not observed. Sandboxed container runtimes (gVisor, Kata) run their own kernel and are not observed either.
No DNS or UDP egress, no raw sockets, no body inspection. The connect sensor sees TCP connects; the rule evaluates destinations against your allowlist and nothing more.
Pipe topology between sibling processes is not reconstructed. A Python agent that opens curl and sh as two separate pipes shows as two executions; the remote_fetch_to_shell shape is only visible when a shell expresses it.
Ancestry is only as complete as Tetragon's cache. A chain broken before Tetragon started reads unattributed, never ungoverned.
One observer credential serves one environment. A host shared by agents of several environments needs one observer per environment.
The join to admissions is by digest and time window, not by a capability the framework carries. A framework that rewrites the command text before executing it reads ungoverned until it is routed through the governed shell.

Troubleshooting

What you see	What it means	What to do
Host offline, `observer_offline` finding	No heartbeat for 10 minutes from an observer that was reporting.	Check the DaemonSet pod or the unit, and Tetragon. Root can stop both — treat unexplained silence as an incident. The finding auto-resolves on the next heartbeat.
Host degraded, ancestors disabled	Tetragon runs without `--enable-ancestors`; the observer is in cache mode.	Enable `processAncestors` on Tetragon (1.3+). Pre-existing process trees read `unattributed` until then.
A sensor is unavailable, `sensor_unavailable` finding	The `TracingPolicy` is installed but that hook failed to load on this kernel.	Reinstall the policy; check the Tetragon version and the kernel's `CONFIG_SECURITY_PATH`. Rules that need the sensor are listed as not evaluated for the host.
A sensor is not installed	No `TracingPolicy` declares it. Expected on hosts without the kprobe policy; no finding.	Set `tracingPolicies.enabled=true` (Helm) or drop the policy into `tetragon.tp.d/` if you want the file and connect rules.
Every execution on a host is `unattributed`	The roster is empty: no agent on the host has heartbeated a runtime context on this boot, or the observer credential is in a different environment than the agents. Or the edge runs in a cgroup-namespaced container (cgroup v2 Kubernetes) and Tetragon runs without `--enable-process-ns`, so the mount namespace the edge attested cannot be joined.	Check that the edge is heartbeating, that the observer agent's environment matches the agents', and that Tetragon has `--enable-process-ns`.
Governed counts look low	Governed executions are sampled; the count is the sum of the samples' weights and is exact.	Nothing — or set `observer.governedSampleEvery=1` to send every one.
`dropped` is non-zero for a host	The queue or the per-agent rate cap evicted events; they were counted, never silently lost.	Raise `batch.queue_max_events` or the per-agent cap, or accept it: the drops are reported so coverage is never overstated.

Governance Coverage — what MITRITY governs and what it does not; the execution-coverage badge the observer feeds
Governed Shell — the execution path on which routed commands are governed by ancestry, no join needed
Writing Policies — the command grammar the observer reuses, workspace_roots, autorun_write_action and the destination allowlist the rules evaluate against
Deployment Guide — installing the edge binaries whose heartbeats carry the runtime context the observer attributes against