# Incident Response Runbook · Wave 8 · C6 · 2026-07-13

## Purpose

A structured procedure for handling security / availability /
integrity incidents on KOKON. Every incident, no matter how small,
is recorded on the tamper-evident governance ledger so a regulator
can walk the timeline from first alert to resolution.

## Severity ladder

| Sev  | Meaning                                                         | Response SLA | Public disclosure?     |
| ---- | --------------------------------------------------------------- | ------------ | ---------------------- |
| S1   | Confirmed data compromise, ledger tampering, or prod outage     | 15 min       | Regulator + subjects   |
| S2   | Suspected compromise; degraded prod (fail-closed working)       | 1 hour       | Regulator (48h window) |
| S3   | Anomaly, missed SLA, or drill finding                           | 24 hours     | Internal only          |

## Decision tree

```
                    Alert / detection signal
                          │
                          ▼
        ┌─────────────────┴─────────────────┐
        │      Does chain still verify?      │
        └─────────────────┬─────────────────┘
                yes │        │ no
                    ▼        ▼
       ┌────────────┐  ┌────────────────────┐
       │  Is prod   │  │  S1 · GOV LEDGER   │
       │ reachable? │  │  TAMPERING         │
       └────┬───────┘  │  Freeze writes,    │
      yes  │  no       │  activate call     │
           ▼   ▼       │  tree, capture     │
   ┌──────┐ ┌─────┐    │  evidence bundle   │
   │ Are  │ │ S1 · │    └────────────────────┘
   │ SLO's│ │ PROD │
   │ green│ │ DOWN │
   └──┬───┘ └─────┘
      │
   ┌──┴──┐
   │yes  │ no → S2 (degraded)
   │     │
   │     ▼
   ▼    (investigate SLO breach source)
   S3
```

## Immediate actions per severity

### S1 · Confirmed compromise / outage

1. **Freeze**: `sudo systemctl stop kokon-control-company.service`
   if writes must be blocked. If reads-only mode acceptable, skip.
2. **Capture**: `curl -o /var/incident/evidence-$(date -u +%FT%TZ).json
   http://127.0.0.1:18091/api/control/audit/evidence-bundle?days=90`
3. **Backup**: `sudo /home/arin/kokon-control-company/scripts/backup-full-prod.sh`
   → snapshot of current state before any remediation.
4. **Notify**: incident channel (Slack / PagerDuty). Announce to the
   call tree below within 15 minutes.
5. **Record**: POST to `/api/control/audit/gov-ledger` with
   `kind=INCIDENT_S1`, `outcome=FAIL`, `classification=RESTRICTED_INTERNAL`,
   payload = incident description.

### S2 · Suspected / degraded

1. **Capture** evidence bundle (as above).
2. **Investigate** via `/observability/traces`, `/slo/status`,
   `/arin-beacon`, `/liveness`.
3. **Record**: `kind=INCIDENT_S2` on the gov ledger.
4. **Notify** ops channel; no external disclosure yet.

### S3 · Anomaly / SLA miss / drill finding

1. **Record**: `kind=INCIDENT_S3` on the gov ledger.
2. **Ticket**: internal follow-up; investigation on a business-day
   timeline.

## Call tree

| Role                | Primary                          | Backup                | Contact channel |
| ------------------- | -------------------------------- | --------------------- | --------------- |
| Incident commander  | Ops lead                         | Product lead          | PagerDuty       |
| Security lead       | Security engineer                | Backend architect     | Signal          |
| Legal / compliance  | Data protection officer          | External counsel      | Phone           |
| Regulator liaison   | Compliance officer               | CEO                   | Encrypted email |
| Data subject notice | Compliance officer               | Legal counsel         | Encrypted email |

_Fill in real names + numbers before the runbook is signed off._

## Communication templates

### Internal-first alert (Slack / PagerDuty)

```
[S{1|2|3}] KOKON incident detected at {UTC}
Signal: {gov chain / liveness / SLO / drill}
Impact: {reads / writes / brain / auditor}
Owner: {incident commander}
Evidence: {URL to bundle}
Bridge: {conf link}
```

### Regulator notification (S1/S2 only)

```
Dear {regulator},

At {UTC timestamp} we detected {S1/S2 event} affecting KOKON
Control Company services. Preliminary impact assessment:
- Duration:  {ongoing / from X to Y}
- Users affected: {count / classes}
- Data at rest:  {intact / possibly-touched / compromised}
- Chain state:   {CLEAN / BROKEN at seq N}

Evidence bundle attached, signed with our ed25519 public key you
previously pinned. We will follow up within 24/48 hours with a
completed incident report.

- {Regulator liaison name}
```

### Data subject notification (S1 confirmed data compromise)

```
Subject: Important notice about your KOKON data

At {UTC} we detected a security incident that may have exposed
information you provided to us. Affected data classes:
{PII / TRANSACTION / BEHAVIOURAL / …}

We have:
- Frozen the affected systems
- Restored from clean backup dated {UTC}
- Reported to {regulator} at {UTC}

Your rights:
- Request an export of your data:  /ui/privacy → Export
- Request full erasure:            /ui/privacy → Erase
- Contact our DPO: {email}

We deeply regret this occurred.
```

## Forensic capture procedure

The evidence bundle is the load-bearing artefact. It contains:

1. Chain verify result at capture time (`chain_verify`)
2. Every gov ledger row in the last 90 days
3. Build SHA + build time
4. ed25519 signature over the digest (regulator can verify offline)

Additional forensic artefacts to grab AFTER the bundle:

- Full-DB backup: `scripts/backup-full-prod.sh`
- Systemd journal since incident time:
  `sudo journalctl -u kokon-control-company.service --since '{UTC}' > incident.log`
- Nginx access log tail:
  `sudo tail -n 5000 /var/log/nginx/access.log > incident-nginx.log`
- OTel span dump: `curl http://127.0.0.1:18091/api/control/observability/traces?limit=4096
  > incident-traces.json`

All artefacts land in `/var/incident/{incident-id}/` — chmod 400,
owned root. Regulator delivery via encrypted email or SFTP.

## Post-incident review (all severities)

Within 7 days of resolution:

1. Root cause: what broke, why, and why we didn't catch it earlier.
2. Impact: users affected, data touched, downtime, SLA misses.
3. Timeline: alert → detection → containment → remediation.
4. Improvements: what changes to make so it doesn't happen again.
5. Update this runbook if the incident revealed a gap.
6. Post `docs/runbook/logs/{date}-incident-{id}.md` with the above.

The review itself is recorded on-chain via
`kind=INCIDENT_REVIEW`, `outcome=PASS`, referencing the original
incident's gov ledger seq.

## Practice drills

The runbook is only credible if it's practiced. Wave 1 · F6 chaos
drill exercises the mechanical parts (kill socat / service / nginx);
the human parts must be practiced separately:

- Quarterly tabletop drill: simulate an S1, follow the tree for real,
  time each stage.
- Annual live drill (out of hours): pull the ARIN gateway and see
  how long the fail-closed path takes to surface.
- Every drill logged as `kind=IR_TABLETOP_DRILL` / `IR_LIVE_DRILL`
  on the gov ledger.

## Related documents

- `docs/CONTROLS.md` — controls matrix mapping incidents to
  controls that should have prevented / caught them.
- `docs/PRIVACY-NOTICE.md` — the public statement to which data
  subjects are pointed.
- `docs/DPIA-TEMPLATE.md` — DPIA process invoked when a new data
  flow is introduced or an incident reveals a gap.
- `docs/runbook/secret-rotation.md` — Class A rotation (e.g. if a
  token was leaked during the incident).
- `docs/runbook/logs/2026-07-12-chaos-drill.md` — reference for
  recovery-time budget under mechanical failure.