Guide

Backup and restore

Your workspace's database can be copied to private backup storage on a schedule. This guide covers turning backups on, checking the copies, restoring one, and what stops backups.

4 min read

Turn backups onCopy link to this section

Backups run once your administrator connects private backup storage in the hosting settings and restarts the workspace. Until then nothing leaves the workspace, and the status page says that data lives only as long as the running workspace.

Your administrator should also set a permanent encryption key. Credentials stored in the workspace are encrypted with it, and each snapshot is sealed with AES-256-GCM under a key derived from it before it is uploaded, so backup storage holds ciphertext only. Keep a copy of the key somewhere else: a sealed snapshot, and the credentials in any restored database, cannot be read without the same key. Without it the workspace makes a key for one run only, and snapshots are uploaded unsealed; the status page and /ops say so.

How often snapshots are takenCopy link to this section

Every 10 minutes, when the database has changed since the last snapshot. One more is taken a minute after start-up and one as the workspace shuts down. Which snapshots are kept follows a retention rule, by default 24h,7d,4w: the newest one in each of the last 24 hours, 7 days and 4 weeks that hold one. The newest snapshot, the one this database was restored from and the newest "before restore" one are always kept. A change that is only uptime beats is uploaded at most once every 6 hours.

Each snapshot is named by the time it was taken, in UTC. At start-up the newest snapshot is restored into the empty database automatically, after waiting up to 30 seconds for the instance being replaced to upload its final one.

A large workspace is stored as a set of numbered parts sealed with a checksum, and the set is listed, copied, checked and restored as one snapshot; earlier snapshots restore just as they did. The backup log shows the size stored beside the size of the database dump, and says so well ahead of time when backups are approaching a limit.

Check the snapshotsCopy link to this section

The Backup section of the status page says whether backups are on, which snapshot this boot restored from, how many snapshots this boot took, when the last one was taken and its last error. Signed in, the Backups card on the operations page (/ops) lists every snapshot in backup storage with its size. Back up now takes one at once, and Download database saves a copy of the live database to your computer.

Restore a snapshot from /opsCopy link to this section

On the Backups card choose Restore… next to a snapshot and type restore to confirm. The whole workspace database is replaced by that snapshot, and the workspace is unavailable for about a minute.

  1. The snapshot is decrypted and loaded into a scratch database first. If that fails, for example because it was sealed with another key or is damaged, nothing is changed.
  2. The current state is kept twice: a local copy and a "before restore" snapshot in backup storage, so you can restore it back.
  3. The workspace and scheduled routines are paused, the database is replaced and brought up to date, and both start again.
  4. The sign-in settings stay as they are now: an older snapshot does not bring back a changed password, a revoked session or a spent recovery code.

The weekly test restoreCopy link to this section

Once every 7 days the newest snapshot is loaded into a scratch database, every table's row count is compared with the live database, and the scratch database is dropped. The live database is only read. The check runs hourly and starts a test when the last one is a week old; Run a test restore on /ops runs one now. The result is row counts match, restored; some tables changed since the snapshot (rows written since the snapshot), or failed with its error, and the status page shows the last one.

What pauses backupsCopy link to this section

WhenWhat happens
The restore at start-up failedAutomatic backups stop, so a snapshot of the empty database never replaces the good one. The status page says "Paused: the restore failed, so the good snapshot is being kept". Fix the cause and restart. Back up now on /ops overrides the pause on purpose and resumes backups, and it uploads the database as it is now.
No permanent encryption key is setBackups are not paused, but they are uploaded unsealed until the key is set, and the stored credentials in them cannot be read after the next restart.
A restore is runningOnly the restore's own "before restore" snapshot is taken until it finishes.
A backup is already running, or the database is not up yetThe timer skips that turn; Back up now says so.
Backup storage is not connectedBackups are off, and the status page says data lives only as long as the running workspace.

The weekly test restore waits too while the API is down, a restore is running or backups are paused. The messages these states show are listed in Troubleshooting.

Last updated