Runbooks

Incident runbook: Aqueduct CCIP delivery failure

Aqueduct is the cross-chain collateral bridge (Arbitrum Sepolia ⇄

incident-aqueduct-ccip.md

Incident runbook: Archive weekly backtest cron failed

The archive service runs the weekly Atrium-vs-baseline backtest +

incident-archive.md

Incident runbook: Chaos Mode misfire

Chaos Mode is the judge-facing demo lever that injects a controlled

incident-chaos.md

Incident runbook: Codex API down

Codex is the x402-payable read API surface . When

incident-codex.md

Incident, keeper failure

Trigger: vigil.keeper.miss.count > 0, or fewer than 2 of 3 keepers reporting healthy.

incident-keeper.md

Incident runbook: Lantern proof-of-reserves stalled

Lantern publishes hourly Merkle attestations of the Coffer vault

incident-lantern.md

Incident runbook: Notifier service silent

The notifier fans out alerts from Scribe to Telegram / Discord / email /

incident-notifier.md

Incident, oracle drift / staleness

Trigger: plinth.oracle.disagreement.count > 5/hour, or any OracleStaleError on Plinth.

incident-oracle.md

Incident Response Procedure

Phase 12, general incident response for Atrium production.

incident-response.md

Incident runbook: Scribe (subgraph) indexing lag

Scribe is the Atrium subgraph deployed to Graph Studio. Every UI

incident-scribe.md

Caddy on Droplet

Install and configure Caddy as reverse proxy on the Atrium daemons droplet.

caddy-on-droplet.md

Codex Deploy Runbook

The CODEXPAYTOADDRESS is the on-chain address that receives x402 USDC payments for Codex API access.

codex-deploy.md

Deploy runbook

Pre-flight:

deploy.md

Runbook, deployer EOA key rotation

Rotates the deployer EOA that holds admin privileges on all Atrium contracts.

deployer-key-rotation.md

DNS Provisioning

Records to configure for useatrium.me via your registrar .

dns-provisioning.md

DigitalOcean Droplet Setup

Bring-up procedure for the Atrium off-chain services droplet .

do-droplet-setup.md

Launch smoke test

Manual pre-launch checklist. Run through each item on the day of launch.

launch-smoke-test.md

Soft-Launch Runbook

Phase 12, go-live procedure for Atrium beta.

soft-launch.md

Vercel Deployment Protection Migration

Toggle production from auth-walled to public when ready for testnet launch.

vercel-deployment-protection.md

Vercel Environment Variable Scoping

Prevent sensitive secrets from leaking into Vercel Preview deployments. Preview

vercel-env-scoping.md

Runbook, key rotation

Applies to: Lantern signing key, Codex backend HMAC key, Keeper bot keys.

key-rotation.md

On-call rotation

Three founders rotate weekly. Pager goes through Discord @oncall

on-call-rotation.md

PGP Key Generation

Generate the security@useatrium.me PGP key for vulnerability disclosure encryption.

pgp-key-generation.md

Discord Alerting Topology

Phase 12, consolidated alerting routes to #ops-alerts.

discord-alerts.md

Discord Server Setup

Human-action runbook for creating the Atrium Discord server.

discord-server-setup.md

Honeybadger Setup

Phase 12 observability, heartbeat monitoring for all Atrium daemons.

honeybadger-setup.md

New Relic Setup

Phase 12 observability, APM + RUM for all Atrium services.

new-relic-setup.md

Sentry Integrations

Phase 12, Discord + Slack webhook setup for Sentry critical error alerts.

sentry-integrations.md

Status Page Setup

Upptime-based status page at status.useatrium.me.

status-page-setup.md

BrowserStack Setup for Atrium E2E

Atrium uses BrowserStack Automate with Playwright's CDP connector for cross-browser E2E testing. This supplements the local Chromium + mobile-safari projects.

browserstack-setup.md

Email routing setup

Status: not yet configured

email-routing.md

Loadtest EOA Runbook

The loadtest service uses a dedicated testnet EOA that is separate from the deployer. This limits blast radius: if the key leaks, only loadtest funds are compromised.

loadtest-eoa.md

Runbook, GitHub Student Pack tool activation

Priority-ordered checklist for activating Student Pack tools.

student-pack-setup.md

Vigil-Keeper Testnet Stake

Procedure for staking the keeper EOA on Arbitrum Sepolia so the vigil-keeper service can execute liquidations.

vigil-keeper-testnet-stake.md