Incident response

This page describes how we detect, communicate, and recover from incidents that affect customer-facing service. Linked from the Security & privacy chapter and from the public status page.

What counts as an incident

We use three levels and respond differently to each:

Routine maintenance windows are tracked separately as severity = maintenance on the status page and are announced 72 h ahead when planned.

The 1-2-4 playbook

For confirmed critical or major incidents we follow a fixed cadence so customers know when to expect updates:

1 hour — customer notification

A first notification goes out by email to tenant owners affected within 1 hour of confirmation. Tone is short and concrete:

• What is broken (function, not internal cause)
• Who is affected (all customers / specific tenants / specific feature)
• What customers should not try in the meantime
• A link to status.dppagent.com for ongoing updates

Magic words to avoid in the first notification: root cause, solved, no impact. We do not yet know.

2 hours — initial scope statement

Within 2 hours we post or send a scope statement that answers:

• What data was involved (e.g. resolver responses only, no write path)
• Which tenants are affected (all, region X, list of N)
• What timeline the issue started (UTC timestamp)
• Working hypothesis of the cause (always labelled as preliminary)

This goes into the corresponding status_incidents row's detail field and is visible on the public status page.

4 hours — public post

Within 4 hours of confirmation, the incident is publicly visible on status.dppagent.com regardless of whether it has been resolved. We update the post on the same page until the incident is resolved.

When the incident closes we set resolved_at on the row and post a closing comment in the detail field describing what was done. A post-mortem follows for any critical incident or any major incident lasting more than 2 hours.

Detection

Detection sources, in rough order of how often they catch things:

1. Public status probes (every 60 s) — /api/cron/status-sample writes per-service uptime rows; if a probe goes down for 2 consecutive minutes a watcher fires.
2. Vercel deployment + function alerts — error-rate thresholds on the resolver, AI Suggest, and integration cron.
3. Customer reports — via support email or in-app. A high absolute volume in a short window triggers triage.
4. Internal staff usage — during business hours we notice broken flows ourselves.

A single source can confirm a minor; we require two independent signals (or a single critical-class signal like resolver 5xx) before opening a major or critical row.

Roles during an incident

For each incident we name:

• Incident commander — coordinates response, makes the call-vs-pager decisions, writes the customer notification.
• Technical lead — investigates the cause, drives the fix, reports findings up.
• Communications — drafts and sends customer email, updates the public status page.

For solo-incident hours the commander wears all three hats. The roles are about discipline, not headcount.

Post-mortem

Every critical incident gets a written post-mortem within 7 days. Major incidents lasting >2 hours also do. The post-mortem covers:

• Timeline (UTC, minute precision, including detection lag)
• Cause (immediate + contributing factors)
• What the user experienced
• What we changed in code, infrastructure, or process to prevent recurrence
• Open follow-ups (linked to the tracking system)

We share post-mortems with affected customers on request, under NDA where relevant.

Tabletop exercises

We run unannounced internal tabletops on a quarterly target cadence to keep the playbook from rotting. The scenarios cover at least one each per year of:

• Resolver outage during peak hours
• Database unreachable / failover
• Suspected unauthorised access to customer data
• A single tenant's integration credential leak

Tabletop write-ups are internal but referenced anonymously in the quarterly security update.

Where customers see what

• status.dppagent.com — current health, active incidents, uptime history (24h / 7d / 30d).
• Email to tenant owners — the 1h and 2h communications above, plus the resolution notice.
• Customer security review pack — historical incident summary, available on request to enterprise prospects.

The internal tracking system is not customer-facing; what is customer-facing is the status page and the email cadence.