Guide

Encryption keys

What the workspace's encryption key and its backup key protect, why the encryption key has to stay the same from one start to the next, and how to change it without losing anything.

3 min read

Two keys from one secretCopy link to this section

  • The encryption key is a secret that whoever runs the workspace sets in its hosting settings. Stored credentials (vault secrets and model provider keys), identity provider settings, saved logins and webhook secrets are encrypted with it before they are written, and so is the two-factor seed of the operations page. The sessions of the workspace's sign-in page are signed with a key derived from it.
  • The backup key is not set separately: it is derived from the encryption key. Every database snapshot is sealed with it (AES-256-GCM) before it leaves the workspace, so backup storage only ever holds ciphertext.

Neither key is shown in the workspace or on the operations page, and secret values are replaced in the logs.

Why the encryption key must persistCopy link to this section

When no permanent key is set, the workspace makes one for each start. That keeps it running, but with three costs:

  • Credentials saved during that run cannot be read after the next restart, because the key that sealed them is gone.
  • Snapshots are uploaded unsealed, because a snapshot sealed with a key that disappears could never be restored.
  • Two-factor sign-in for the operations page cannot be turned on, because its seed would be lost at the next restart.

The operations page (/ops) then warns that stored credentials will be lost on the next restart. Keep a copy of the key somewhere other than the workspace, such as your password manager: a restored database's credentials, and a sealed snapshot, open only with the key they were sealed with.

Make a keyCopy link to this section

Any 32 random bytes written in base64 will do. On most computers this prints one:

openssl rand -base64 32

Set it as the workspace's encryption key in the hosting settings and restart. From then on snapshots are sealed and the status page lists them as encrypted.

Change the keyCopy link to this section

  1. Make a new key as above, and keep the current one to hand.
  2. In the hosting settings, set the new key as the encryption key and the current one as the previous encryption key, then restart.
  3. Reading keeps working at once: whatever the new key cannot open is tried with the previous one. While it starts, the workspace re-encrypts every stored secret the previous key opens under the new key, in batches.
  4. Follow it on Admin › System status, under encryption key rotation: how many values were re-encrypted, were already on the new key, failed, and remain. A value that neither key opens is counted as failed and left exactly as it was; check that the previous key is the one those values were stored with.
  5. API keys created before the change keep working while the previous key is set. Reissue them, and once the page says nothing remains on the previous key, remove it and restart.

Two things to plan for:

  • Changing the key signs everyone out of the workspace sign-in page, because its sessions are signed with a key derived from it.
  • Snapshots sealed under the old key still restore while it is set as the previous key. Keep a copy of the old key for as long as you keep snapshots taken before the change.

If the key is lostCopy link to this section

Nothing can recover a lost key. The running workspace keeps everything that is not sealed with it, but every stored credential has to be entered again, and snapshots sealed under the lost key can no longer be restored. Set a new key, restart, enter the credentials again, and take a fresh snapshot with Back up now on /ops. Backups themselves are covered in Backup and restore.

Last updated