config-public/docs/secrets.md

Secrets

Three separate things, handled three different ways. The split matters: the first is needed to reach a host, the second is needed by the software on it, and the third is a mistake.

1. The deploy SSH key

Ansible knows exactly one thing about credentials: SSH_KEY_PATH, a path to a 0600 private key. inventory/group_vars/all.yml feeds it to ansible_ssh_private_key_file and nothing else in the repository touches key material.

Everything about where the key comes from lives in scripts/load-ssh-key.sh, which prints a path and never prints the key:

export SSH_KEY_PATH="$(scripts/load-ssh-key.sh)"         # env:   already on disk
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh ci)"      # ci:    from a CI secret
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh vault)"   # vault: from Vault / OpenBao

Add a backend there rather than teaching a playbook about secrets. That seam is the reason the same playbooks run unchanged from a laptop, from GitHub Actions, from GitLab CI and from Gitea Actions.

Whichever backend you use, the matching public key must be in the deploy user's ~/.ssh/authorized_keys on every host, or the key is useless.

Storing it in CI

Store the key base64-encoded, as a single line:

base64 -w0 < ~/.ssh/deploy_key            # Linux
base64 < ~/.ssh/deploy_key | tr -d '\n'   # macOS

# Windows PowerShell
[Convert]::ToBase64String([IO.File]::ReadAllBytes("$HOME\.ssh\deploy_key"))

On Windows use exactly that. certutil -encode wraps its output in a header and line breaks, and Get-Content re-encodes the bytes — both produce something that looks like base64 and decodes to an unusable key. Do not re-wrap the output either; it is one long line by design.

Then set it as SSH_PRIVATE_KEY_B64:

Platform	Where	Notes
GitHub	Settings → Environments → production → Secrets	Scoped to the environment, so only jobs naming it can read it
GitLab	Settings → CI/CD → Variables	Mark Masked and Protected
Gitea	Settings → Actions → Secrets	No environments and no masking controls: any run in the repository can read it. See ci-gitea.md

Why base64 rather than the raw key? GitLab can only mask a value that is a single line with no whitespace, and an OpenSSH private key is neither. GitHub redacts secrets from logs regardless, but a single-line value survives being passed between steps without surprises. One encoding satisfies both.

A raw SSH_PRIVATE_KEY still works — as a GitLab File-type variable, as a GitHub multi-line secret, or as a path on disk. The loader accepts all three. It just cannot be masked on GitLab, so the key can appear in a job log if something echoes it.

Storing it in a vault

vault reads a HashiCorp Vault or OpenBao KV v2 secret. The expected layout, overridable with VAULT_KV_MOUNT / VAULT_SECRET_PATH / VAULT_SECRET_FIELD (or the BAO_* spellings):

kv/infra/ansible/deploy-key      field: private_key

A read-only policy is enough:

path "kv/data/infra/ansible/deploy-key" {
  capabilities = ["read"]
}

Keep the vault off the deploy path. If the vault runs on a host this repository deploys to, and it seals on reboot, then a deploy that fetches its key from the vault cannot run — including the deploy that would fix that host. Host it elsewhere, or auto-unseal it. A CI secret is the dependable route.

That is also why the deploy key does not live in the stacks' OpenBao (§2), even when one runs on the host. You need SSH to reach the host the vault runs on, so a key fetched from that vault could never get you there in the first place. Two stores, deliberately: CI holds the key that reaches the host, the vault holds what the stacks need once you are there.

2. Per-stack secrets

Stack secrets never live in this repository. There are two ways to get them onto a host, and both end in the same place: a .env next to the stack's compose file, which Compose reads and the sync never touches (stack_exclude).

Either way, compose files read each value with Compose's required-variable syntax:

    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?missing from .env}

That is worth copying. Without the :?, a missing variable expands to the empty string and Postgres starts with no password at all — a failure that looks like success until someone finds it. With it, Compose refuses to start and names the variable. Each stack also ships a .env.example listing the names with no values, as documentation.

From OpenBao, with stack_secrets (recommended)

A playbook names what it needs, and the compose_stack role renders it:

    stack_secrets:
      SECRET_KEY:
        path: stacks/webapp
        field: secret_key

The role reads kv/stacks/webapp from http://127.0.0.1:8200 on the target host, at the end of the sync, and writes {{ stack_dest }}/.env at 0600. Nothing is placed by hand, and nothing — plaintext or encrypted — is committed. webapp.yml and webapp-staging.yml are the worked examples, reading from the openbao stack on the same host.

A sealed vault, an unset token, a 403 or 404 from the vault, a field that is not there, and a value that cannot be safely written to a .env all fail before the stack starts, each naming what to fix. No task prints a value: the reads are no_log, and the asserts that explain a failure see only names, paths and status codes.

Why the read happens on the host

Because the vault is published to 127.0.0.1:8200 and nowhere else. Ansible tasks already run on the host, so the store never needs a public hostname, never needs a certificate, and does not care whether the deploy came from a laptop or a CI runner. ansible.cfg sets pipelining = True, so module arguments are piped to the remote interpreter over stdin and the token is never written to a file on the host. That covers the token specifically — it does not make every transfer file-free: ansible.builtin.copy still lands its content as a 0600 temp file under ~/.ansible/tmp on both ends, removed when the run ends, regardless of pipelining.

The consequence is one vault per host that reads from it, or a vault_addr the host can reach. In this repository only app runs one; metrics, which also runs on edge, uses a hand-placed .env instead (below).

The token

VAULT_TOKEN is read from the control node's environment: your shell locally, and a CI secret in a runner. It carries a stacks-read policy granting read on kv/data/stacks/* and nothing else. It is not the root token.

The name is not configurable. compose_stack reads it with lookup('env', 'VAULT_TOKEN'), and the deploy workflows pass it through as an environment variable rather than interpolating it into a run: body — an expression is substituted before the shell sees it, so a value containing shell metacharacters would be executed rather than assigned.

Where to put it, and how long it lasts, are steps 5 and 6 of Bootstrapping the vault below.

Unsealing

OpenBao has no self-contained auto-unseal; it needs an external KMS or a second vault. The example stack is unsealed by hand:

ssh [email protected]
docker exec -it openbao bao operator unseal   # three times, three different keys

(docker exec inherits the container's environment, including the BAO_ADDR set in docker-compose.yml — so none of the commands on this page need their own -e BAO_ADDR=.... Without it the bao CLI defaults to https://127.0.0.1:8200 and every one of these fails with a TLS handshake error, since the listener has TLS disabled.)

After a reboot the vault is sealed and any deploy that reads a secret fails, naming the vault and pointing here. That is the intended behaviour: a sealed vault is a loud, safe failure.

Adding a secret

Putting the root token and the secret value on the command line, as -e BAO_TOKEN=<root> and <field>=<value>, lands both in the host's process table and in the operator's shell history. Log in interactively instead, and pipe the value in on stdin so it never appears as an argument:

ssh [email protected]
docker exec -it openbao bao login   # prompts for the root token

read -rs SECRET
printf '%s' "$SECRET" | docker exec -i openbao bao kv put kv/stacks/<stack> <field>=-
unset SECRET

read -rs does not echo and does not reach shell history, and printf '%s' adds no trailing newline. That last part matters. <field>=- stores stdin byte for byte, including the newline you press Enter for, and the role refuses to render a value containing a newline or a single quote — a .env line it cannot quote safely is one Compose would misread. The deploy then fails on Verify no secret value contains a newline or a single quote, which is the guard working, but it is a round trip you can skip.

Generating a value has the same trap, because every generator ends its output with a newline:

openssl rand -hex 32 | tr -d '\n' | docker exec -i openbao bao kv put kv/stacks/<stack> <field>=-

kv put replaces every field at that path. To add a second field beside an existing one, use bao kv patch instead, or the first is lost.

Then add the entry to the playbook's stack_secrets and deploy. Rotating is the same kv put (or kv patch) followed by a redeploy.

Removing an entry from stack_secrets stops the render on the next deploy; it does not delete the .env already on the host, which keeps serving its stale value until something else removes it.

A changed openbao.hcl needs a manual restart

Restarting OpenBao seals it. openbao.yml therefore reports that its payload changed rather than restarting the container, because an automatic restart in the middle of site.yml would seal the vault and fail every stack imported after it. Restart and unseal deliberately.

Bootstrapping the vault

Hosting the vault on the host it serves is circular, so the order is fixed and the middle of it is manual. This is the whole sequence, once per vault.

1. Prepare the host and deploy the stack.

cd ansible
ansible-playbook playbooks/bootstrap.yml -u <admin-user> -k -K -e deploy_sudo=true
ansible-playbook playbooks/openbao.yml

deploy_sudo is not optional here. openbao.yml hands data/ to uid 100, and a stack_dirs entry with an owner is the only thing the role escalates for. Without it the deploy stops at Create the extra directories the stack needs with Missing sudo password. The stack declares no stack_secrets of its own — it is the root of trust and cannot depend on itself — so it comes up with nothing in place. See bootstrap.md.

2. Initialise, once, ever.

ssh [email protected]
docker exec -it openbao bao operator init

Five unseal keys and one root token are printed once. They are not written to the host and cannot be recovered. Put all six in your password manager before doing anything else: three of the five unseal the vault, and losing three makes the storage unreadable permanently.

3. Unseal.

docker exec -it openbao bao operator unseal   # three times, three different keys
docker exec openbao bao status                # expect: Sealed  false

4. Create the mount and the read-only policy.

Log in interactively rather than passing -e BAO_TOKEN=<root>, for the reason under Adding a secret above — an argument lands in the host's process table.

docker exec -it openbao bao login   # prompts for the root token

docker exec openbao bao secrets enable -path=kv kv-v2

printf 'path "kv/data/stacks/*" {\n  capabilities = ["read"]\n}\n' \
  | docker exec -i openbao bao policy write stacks-read -

5. Mint the deploy token and store it.

docker exec openbao bao token create \
  -policy=stacks-read -period=768h -display-name=ansible

This token, never the root one, is VAULT_TOKEN. Locally it is an export:

export VAULT_TOKEN=<the stacks-read token>

In CI it is a secret of the same name:

Platform	Where	Notes
GitHub	Settings → Environments → production → Secrets	Scoped to the environment, so only jobs naming it can read it
GitLab	Settings → CI/CD → Variables	Mark Masked and Protected
Gitea	Settings → Actions → Secrets	No environments and no masking controls: any run in the repository can read it — which is precisely why the token is scoped. See ci-gitea.md

Unlike the SSH key in §1, this one needs no base64: a vault token is already a single line with no whitespace, which is the only thing GitLab's masking requires.

6. Diarise the expiry.

-period=768h makes the token renewable indefinitely; it does not make it permanent. It lapses 32 days after it was last renewed, and the first sign is a deploy failing on the vault read after a quiet month.

Re-minting is the reliable fix — repeat step 5 and replace the secret. To renew instead, log in as the token rather than passing it as an argument, where the host's process table would keep it:

docker exec -it openbao bao login   # paste the stacks-read token
docker exec openbao bao token renew # renews the token you logged in with
docker exec openbao sh -c 'rm -f ~/.bao-token'

A longer -period at mint time is the other lever, traded against how long a leaked token stays useful.

7. Prove the token cannot write.

docker exec -e BAO_TOKEN=<the new token> openbao bao kv put kv/stacks/canary value=nope

This must fail with a 403. If it succeeds the policy is wrong and anything holding the deploy token can write to the store. Nothing else surfaces a policy typo, so do not skip it.

8. Load the secrets and converge.

Once per secret, per Adding a secret above — mind the trailing newline. For the examples that is postgres_password and secret_key under kv/stacks/webapp, and the same two under kv/stacks/webapp-staging:

openssl rand -hex 32 | tr -d '\n' \
  | docker exec -i openbao bao kv put kv/stacks/webapp postgres_password=-
openssl rand -hex 32 | tr -d '\n' \
  | docker exec -i openbao bao kv patch kv/stacks/webapp secret_key=-

Then converge with ansible-playbook playbooks/site.yml from ansible/.

When you are finished, bao login has left the root token in the container at ~/.bao-token; remove it with docker exec openbao sh -c 'rm -f ~/.bao-token'.

Steps 2–7 are manual and stay manual. That is the accepted cost of not having a second machine to hold the keys.

By hand, on the host

A stack with no stack_secrets never talks to a vault, and reads a .env someone placed on the host. Editing one in this repository has no effect, and a host's .env is never overwritten by a deploy. To change one, edit it on the host and redeploy.

metrics works this way, because it runs on edge too and edge has no vault. Its playbook asserts the file exists before doing anything, so the error arrives before the payload is synced rather than halfway through starting the stack:

  pre_tasks:
    - name: Look for the metrics .env on the host
      ansible.builtin.stat:
        path: "{{ stack_root }}/metrics/.env"
      register: metrics_env
      changed_when: false

    - name: Verify the metrics .env is present on the host
      ansible.builtin.assert:
        that:
          - metrics_env.stat.exists
        fail_msg: >-
          {{ stack_root }}/metrics/.env is missing on {{ inventory_hostname }}.
        quiet: true

Do this for any hand-placed stack whose compose file has no safe defaults.

3. Secrets committed to the repository

There is no third category. This section exists because it is the failure mode this layout is designed to prevent, and it is worth being blunt about why.

A credential written directly into a compose file:

• cannot be excluded from the sync. stack_exclude keeps .env off the wire; it can do nothing about a value inside docker-compose.yml.
• is in every clone, forever. Git history is the point of git. Deleting the file in a later commit changes nothing about the credential.
• cannot be rotated without a deploy, so it usually is not rotated at all.
• escapes its blast radius. A personal access token carries its creator's access to everything they can reach, not just the project it was added to. A cloud application secret outlives the VM it was used on.

If you find one, the fix is to rotate the credential, not to delete the file. Removing it stops new clones carrying it and changes nothing else. Then move it to a .env on the host and read it with ${VAR:?}.

The guard

The lint workflow — .github/workflows/lint.yml, .gitea/workflows/lint.yml and the lint job in .gitlab-ci.yml — runs gitleaks on every push, over the working tree and the full history: a secret that was committed and then removed is still leaked, and the scan should say so. .gitleaks.toml allowlists only *.env.example files and this document.

On GitHub it runs the upstream Docker image rather than the marketplace action on purpose: the action requires a licence key for organisation-owned repositories and silently does nothing without one, which is the worst possible behaviour for a secret scanner. The Gitea port runs the pinned release binary instead, because a job there is itself a container with no Docker socket.

Do not add an allowlist entry to make a finding go away. Rotate the credential.