Backup, disaster recovery and continuity for self-hosted CMDash

Who this is for. Anyone running CMDash under a self-hosted license: on a Docker host (self-host.md) or in your own AWS account (self-host-aws.md). It sets out the backup and recovery policy that applies to your installation, and the commands that carry it out.

CMDash sets one disaster recovery and business continuity policy for its hosted service and for every licensed installation. On the hosted service CMDash carries it out. On a self-hosted installation you do, with the tools that ship with CMDash. This page is that policy, written for the operator of a self-hosted installation.

The policy

A backup that has never been restored is a belief, not a backup. The drill is part of the policy, not an extra.

What to back up

Docker host

The commands below run in deploy/onprem/. The operator runbook that ships with CMDash (docs/runbooks/onprem-install.md, sections 5, 6 and 9) has every option.

Set up the scheduled backup

1. Create the encryption key on a different machine and keep it there. With age: age-keygen -o backup-identity.txt, then a MAC key: openssl rand -hex 32 > backup-mac.key.

2. Put the public half on the host: set CMDASH_BACKUP_AGE_RECIPIENT (or CMDASH_BACKUP_PASSPHRASE_FILE) in cmdash.env. Without one, the timer fails every run by design and writes nothing.

3. Install the timer:

   bashCopy

   cd deploy/onprem/systemd
   # Edit WorkingDirectory= and User= in cmdash-backup.service to match your install first.
   sudo cp cmdash-backup.service cmdash-backup.timer /etc/systemd/system/
   sudo systemctl daemon-reload
   sudo systemctl enable --now cmdash-backup.timer
   systemctl list-timers cmdash-backup.timer     # confirm it is armed
   

4. Copy each new archive and its .sha256 and .mac files from backups/ to off-host storage on the same daily cadence. Keep the MAC key away from that storage.

A backup missed while the host was off runs at the next boot (Persistent=true). A failed run fails the unit; check it with systemctl status cmdash-backup and alert on it with your usual monitoring.

Restore

cd deploy/onprem
./restore.sh backups/cmdash-onprem-backup-<ts>.tar.gz.age --age-identity backup-identity.txt --mac-key-file backup-mac.key
# or, passphrase mode:
./restore.sh backups/cmdash-onprem-backup-<ts>.tar.gz.gpg --passphrase-file /root/.cmdash-backup-pass

Restore onto a stack that install.sh has already set up, running the same release the backup came from (or a newer one). restore.sh checks the MAC and decrypts before it changes anything, so a wrong key or a tampered archive stops early. It is destructive: it replaces the database and the object store, asks you to type RESTORE, re-applies the schema and row-level security policies, and health-checks the api at the end. It writes the backed-up cmdash.env and license to ./restored-secrets/ for you to review; it never overwrites your live configuration.

To rebuild a lost host: install CMDash on the new host with install.sh, put back cmdash.env and the license from your secrets copy, then run restore.sh with the newest off-host archive.

Upgrades and rollback

• One-click update (Settings, Updates). The updater rolls the application one service at a time behind a health check and rolls back to your current version automatically if the new one fails it. It only offers releases that need no database migration.
• Manual upgrade. Back up, set IMAGE_TAG to the new release in cmdash.env, and re-run install.sh. Apply any SQL file the release notes name with ./install.sh --pre-push-sql <file>.
• Manual rollback. Set IMAGE_TAG back to the previous release and re-run install.sh. If the release changed the database, restore the backup you took before the upgrade.

Your own AWS account

The CloudFormation stack runs the daily backup to its backups bucket and, on the production and GovCloud templates, keeps RDS automated backups for 7 days and a final snapshot when the stack is deleted. The restore steps, for both the eval and the RDS shapes, are in self-host-aws.md under "Backups", and upgrades and their rollback under "Upgrades". What remains yours:

• Copy the eval template's app-secrets.env somewhere safe.
• Decide whether you need copies in a second region or account, and set them up (S3 replication, RDS snapshot copy).
• Run the restore drill.

The restore drill (once per quarter)

1. On a scratch host or stack running the same release, start from a fresh install.
2. Fetch the newest off-host archive and the decryption key from where each is kept. If one person cannot get both quickly, fix that first: that is the finding.
3. Restore it, note the start and end times, and confirm: the health check passes, you can sign in, an integration that stores a credential still connects, and the restored cmdash.env carries the CREDENTIAL_ENCRYPTION_KEY you expect.
4. Run the restore again with a wrong key and confirm it refuses before changing anything.
5. Record the date, the archive, who ran it and the measured time. That time is your tested RTO; compare it with your target.
6. Delete the scratch install and restored-secrets/.

CMDash's Continuity & recovery section can hold the record: a recovery register entry for CMDash with your declared RTO and RPO, and each drill's result as the tested value.

Who does what

Related

• self-host.md: installing on a Docker host.
• self-host-aws.md: installing in your own AWS account, its backups and upgrades.
• deploy/onprem/README.md: the installer's reference, including backup.sh and restore.sh.
• docs/runbooks/onprem-install.md: the operator runbook (backup and restore, upgrade, scheduled backups).