# Backup & disaster-recovery runbook

**Targets.** Recovery point objective (RPO): 24 hours (nightly backup; run `backup:run` before risky work such as
year-end close or an update). Recovery time objective (RTO): 4 hours to a working system on a new server.
Tighten RPO for the hosted platform with MariaDB binary logs or a managed database's point-in-time recovery.

## What a backup contains

`storage/app/backups/backup-YYYYMMDD-HHMMSS.ecb` (one file):
- `db.sql`: consistent `mysqldump --single-transaction` including the **integrity triggers** (posted journal
  lines immutable, audit log append-only) and routines;
- `files/…`: uploaded files under `storage/app/private` and `storage/app/public`;
- `manifest.json`: SHA-256 of every part and row counts of key tables (tenants, users, members, journals,
  journal lines, giving entries, audit logs).

The archive is encrypted with AES-256-GCM in 1 MB chunks under `BACKUP_KEY`. Each chunk's number and a last-chunk
flag are authenticated, so tampering, reordering or truncation is detected and the file will not decrypt.

## The two keys: keep them OFF the server

| Key | Needed for | Where it lives |
|---|---|---|
| `BACKUP_KEY` | opening any backup | `.env` + church password manager + sealed copy with a trustee |
| `APP_KEY` | reading encrypted columns (welfare notes, gateway/WhatsApp secrets, 2FA) and sessions | same |

A backup without `BACKUP_KEY` is useless. A restored database without the original `APP_KEY` loses the encrypted
fields (users can reset 2FA; gateway keys can be re-entered; **welfare case notes cannot be recovered**).

## Schedule & retention

- 02:00 daily `backup:run`; Sunday 03:00 `backup:verify` (decrypt + check every checksum, live DB untouched).
- Local retention: 14 daily + 8 weekly + 12 monthly (`config/backup.php`).
- Off-site copy (3-2-1 rule: 3 copies, 2 media, 1 off-site): `BACKUP_OFFSITE_DISK` (S3-compatible bucket such
  as Backblaze B2, Wasabi or Cloudflare R2; install `league/flysystem-aws-s3-v3` and add the disk in
  `config/filesystems.php`) and/or `BACKUP_COPY_TO` (synced folder, external drive). Use bucket versioning or
  object lock so ransomware on the server cannot delete old backups.
- The platform console shows the last result; investigate any red banner the same day.

## Commands

```bash
php artisan backup:run                        # make one now
php artisan backup:list                       # what is held locally
php artisan backup:verify [file]              # prove a backup opens and is complete
php artisan backup:restore <file> --force     # type RESTORE; maintenance mode; safety backup first
```

`backup:restore` takes the app down, makes a safety backup of the current state, loads the dump, copies the
files back, checks the restored row counts against the manifest, then brings the app up.

## Scenario playbooks

**A. Someone deleted or changed data by mistake.** Ledger entries are never deleted (reversals only), so first
look in the Audit trail. If a restore is still needed, restore the latest good backup **into a scratch database**
(see drill below), extract what is needed, and re-enter it. Do not roll back the live books for one record.

**B. Server lost, disk failed or ransomware.**
1. Provision a new server (docs/DEPLOYMENT.md), same app version as the backup (manifest shows the last migration).
2. Put the original `APP_KEY` and `BACKUP_KEY` in `.env`; `php artisan config:cache`.
3. Copy the newest off-site `.ecb` into `storage/app/backups/`; `php artisan backup:verify <file>`.
4. Create the empty database, then `php artisan backup:restore <file> --force`.
5. `php artisan migrate --force` (no-op if versions match), `php artisan queue:restart`.
6. Re-point DNS; re-check the payment and WhatsApp webhook URLs; send a test message.
7. Tell church administrators the recovery point (backup time); anything after it is re-entered from source documents.

**C. Database corrupt but server fine.** `backup:verify` the newest backup, then `backup:restore` it.

**D. A backup key is exposed.** Generate a new key (`backup:key`), take a fresh backup, keep the old key in the
password manager only until every backup made with it has aged out of retention, then destroy it.

## Quarterly restore drill (record the result in the board pack)

```bash
php artisan backup:verify
# Restore into a throwaway database, never the live one:
mysql -e "CREATE DATABASE church_restore_drill"
DB_DATABASE=church_restore_drill php artisan config:clear   # or use a copy of the app pointed at it
php artisan backup:restore <file> --force
# Check: sign in, open the trial balance as at the backup date, compare with the last printed board pack.
mysql -e "DROP DATABASE church_restore_drill"
php artisan config:cache
```
Last drill on this machine (13 Oct 2026): backup decrypted, restored into a scratch database, row counts matched,
the immutability trigger was active after restore, scratch database dropped.