PRIVATE AI
OPERATIONS LAB

An implemented engineering lab

Let AI investigate.
Keep recovery
with the operator.

A distributed engineering team needs help diagnosing its private Git service. The assistant needs useful evidence. It does not need administrative authority.

Follow the real incident Inspect the design

3fixed, read-only MCP tools

1Linux VPS · $12/month compute

0administrative tools in MCP

Humanseparate SSH authorization

01 / System design

Useful evidence across
deliberate trust boundaries.

The private service, the diagnostic interface and the recovery path have different jobs. This public page serves only reviewed static files.

Public visitors reach only a static site. Inside the private lab, OpenCode reaches MCP over Tailscale Serve on HTTPS port 443. MCP reads a sanitized snapshot and fixed health endpoints. A separate operator publisher reads Docker state. A human operator uses separately authorized Tailscale SSH to restore Gitea.
One host, separate containers and paths. Network admission, tool permissions and operator authority are independent controls.
A

Private connectivity

Tailscale Serve terminates tailnet-only HTTPS. Explicit grants admit the diagnostic identity to TCP 443. An excluded node is denied even while the allowed path is healthy. No Funnel is used.

B

Bounded diagnostics

The Python MCP server exposes three tools with no arguments. Fixed health endpoints and a read-only snapshot replace arbitrary shell commands, paths and network targets.

C

Separate recovery

A human identity has a separate Tailscale SSH rule in check mode for the existing root account. The AI node has no SSH grant. Root recovery retains administrative authority.

02 / Actual failure, actual diagnosis

Gitea went down.
The evidence stayed available.

OpenCode with an OpenAI model used genuine MCP responses from an isolated diagnostic client. The operator controlled the fault and recovery independently, with a recovery guard active.

  1. Establish the baseline

    The AI called the three diagnostic tools. Gitea and proxy health were both HTTP 200; state and log snapshots were fresh.

    AI / read

  2. Introduce a real outage

    The separate operator stopped the Gitea container. Caddy and MCP remained running. A real proxy request returned HTTP 502.

    Operator / stop

  3. Correlate independent observations

    The AI observed Gitea exited, proxy healthy, 502 / 200 health and sanitized upstream-error events. It identified the outage without claiming to know why Gitea stopped.

    AI / diagnose

  4. Recover through human SSH

    The authorized operator restored Gitea through Tailscale SSH. No MCP tool initiated recovery. The independent timeout guard did not need to fire.

    Operator / restore

  5. Verify the recovery

    The AI called MCP again: 200 / 200, running state and fresh snapshots. Separate anonymous application-API checks confirmed the same test repository ID and README blob.

    AI / verify

“The new evidence indicates a Gitea outage while the proxy itself remains healthy.”

Actual OpenCode model response during the incident · 22 Oct 2026

Nine actual MCP calls across baseline, outage and recovery. Repository persistence was checked by the operator, not by the three diagnostic tools. This proves continuity of the compared fixture, not integrity of every stored record.

03 / Authorization under test

An allowed path.
A genuinely denied identity.

These results came from distinct tailnet nodes and roles, with healthy allowed controls. A timeout by itself would not establish an authorization denial.

Recorded M4 access checks · not a live probe
Client / pathExpectedObserved
Diagnostic identity → private MCP / HTTPSAllowPASS Discovery and all three tools worked
Diagnostic identity → server SSHDenyPASS Connection denied by network policy
Excluded node → private MCP / HTTPSDenyPASS Denied alongside healthy allowed controls
Human operator → root / Tailscale SSHAllow with check rulePASS Actual recovery completed
Human operator → unapproved SSH userDenyPASS Explicit SSH policy rejection
Public network → private service portsDenyPASS M4 public probes blocked; no cloud inbound rules
MCP caller → shell / restart / file toolsUnavailablePASS Unknown-tool responses; extra arguments rejected

The public-port row describes M4 before public-site deployment. M5 adds only a separate static HTTPS listener; it requires fresh public/private regression checks before acceptance.

Grants admit a connection. They do not assign MCP tool roles.

Every admitted MCP caller gets the same three read-only diagnostics. Human SSH authorization is a separate policy. The OpenCode agent also denies shell, file access, delegation and other tools; those settings alone are not operating-system isolation.

04 / Implementation & evidence

Small surface.
Explicit limits.

Real service behavior is more useful than a scripted answer. Narrow interfaces make the investigation inspectable and the limitations visible.

service_health

Live health

Two fixed HTTP endpoints. No redirects or caller-supplied targets. App health is distinguished from proxy health.

recent_logs

Selected events

Up to ten sanitized log events. No raw messages, headers, tokens or repository contents are returned. Events remain untrusted data.

operational_state

Selected state

Container state, health and restart counts from a timestamped snapshot. Missing, malformed or stale data produces an explicit safe error.

Acquisition limits and authority ownership
  • Health requests: two-second timeout and 4 KiB per endpoint; four-second whole-tool deadline.
  • Snapshot: fixed regular file, 32 KiB maximum; stale after 20 seconds. The publisher normally polls every two seconds.
  • Logs: fixed Gitea/Caddy services, 120-second window, at most 40 acquired lines per service; structured allowlist projection before publication.
  • Structured tool output: 4 KiB maximum; MCP wire response: 12 KiB maximum. Tool input accepts no additional properties.
  • The operator-side publisher has Docker authority. MCP cannot start it or request refreshes and mounts only the snapshot directory read-only. MCP has no Docker socket, credentials or Gitea storage mounts.
Design tradeoffs and limits of the evidence
  • Containers and distinct tailnet nodes constrain the exercised paths. They do not isolate the lab from its trusted Docker administrator or remove shared-kernel risks.
  • The OpenAI provider receives prompts and sanitized diagnostic responses. This is not local model inference or a claim of provider-side data isolation.
  • SQLite and persistent volumes fit a small, single-host demonstration. Restart persistence is not a backup or disaster-recovery strategy.
  • Logs, service content and tool responses are untrusted. An exercised injection fixture was rejected; that does not prove general prompt-injection immunity.
  • Point-in-time health and one verified repository fixture do not prove sustained availability, complete data integrity or production readiness.
  • No autonomous remediation, arbitrary shell tools, tenant platform or custom model. Codex is the development agent; OpenCode is the demonstration client.
Reproducibility and recorded versions

Gitea 28.2.0 rootless, Caddy 2.11.7, Python 3.12.15, official MCP SDK 2.3.0, OpenCode 1.19.3 and Tailscale 1.104.2. MCP SDK client negotiated protocol 2026-07-28; the OpenCode connection's negotiated protocol was not separately recorded. Container images and Python dependencies are pinned.

Evidence is a reviewed, de-identified projection of the accepted M4 checkpoint. It includes actual timestamps and outcomes, rather than private node addresses, authentication artifacts or raw service logs.

Read the compact evidence record

The customer outcome

Faster investigation.
A clear handoff to the person authorized to act.

The assistant explains what it observed, where the evidence stops, and when a human operator is needed.