Problem
On-call responders must connect alerts, deployments, metrics, logs, runbooks, and permissions under time pressure. Letting probabilistic output directly mutate infrastructure creates a second incident risk.
Contribution
- Separated AI proposal from execution authority with an in-process capability created only by a deterministic gate.
- Designed at-least-once processing around Redis suppression, PostgreSQL uniqueness, RabbitMQ manual acknowledgements, and idempotent strategies.
- Built a production-shaped digital twin and a public, dry-run operator experience.
Case note 01
The boundary
The model gets typed, bounded, read-only tools. It can classify evidence and propose a response, but it never receives a mutating tool. Every proposed action re-enters a deterministic guardrail gate.
Case note 02
Correct under retries
The system treats network and process failure as normal. Fast duplicate suppression lives in Redis; durable correctness lives in PostgreSQL; the queue acknowledges only after durable processing.
- A database uniqueness constraint prevents duplicate incidents.
- An action claim commits before an external side effect.
- The execution strategy is idempotent across the crash uncertainty window.
Case note 03
Honest limitation
The public environment is a deterministic digital twin and stays in dry-run mode. It proves the decision path without granting anonymous visitors authority over real infrastructure.