config-public/docs/secrets.md

Secrets

Three separate things, handled three different ways. The split matters: the first is needed to reach a host, the second is needed by the software on it, and the third is a mistake.

1. The deploy SSH key

Ansible knows exactly one thing about credentials: SSH_KEY_PATH, a path to a 0600 private key. inventory/group_vars/all.yml feeds it to ansible_ssh_private_key_file and nothing else in the repository touches key material.

Everything about where the key comes from lives in scripts/load-ssh-key.sh, which prints a path and never prints the key:

export SSH_KEY_PATH="$(scripts/load-ssh-key.sh)"         # env:   already on disk
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh ci)"      # ci:    from a CI secret
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh vault)"   # vault: from Vault / OpenBao

Add a backend there rather than teaching a playbook about secrets. That seam is the reason the same playbooks run unchanged from a laptop, from GitHub Actions, from GitLab CI and from Gitea Actions.

Whichever backend you use, the matching public key must be in the deploy user's ~/.ssh/authorized_keys on every host, or the key is useless.

Storing it in CI

Store the key base64-encoded, as a single line:

base64 -w0 < ~/.ssh/deploy_key            # Linux
base64 < ~/.ssh/deploy_key | tr -d '\n'   # macOS

# Windows PowerShell
[Convert]::ToBase64String([IO.File]::ReadAllBytes("$HOME\.ssh\deploy_key"))

On Windows use exactly that. certutil -encode wraps its output in a header and line breaks, and Get-Content re-encodes the bytes — both produce something that looks like base64 and decodes to an unusable key. Do not re-wrap the output either; it is one long line by design.

Then set it as SSH_PRIVATE_KEY_B64:

Platform	Where	Notes
GitHub	Settings → Environments → production → Secrets	Scoped to the environment, so only jobs naming it can read it
GitLab	Settings → CI/CD → Variables	Mark Masked and Protected
Gitea	Settings → Actions → Secrets	No environments and no masking controls: any run in the repository can read it. See ci-gitea.md

Why base64 rather than the raw key? GitLab can only mask a value that is a single line with no whitespace, and an OpenSSH private key is neither. GitHub redacts secrets from logs regardless, but a single-line value survives being passed between steps without surprises. One encoding satisfies both.

A raw SSH_PRIVATE_KEY still works — as a GitLab File-type variable, as a GitHub multi-line secret, or as a path on disk. The loader accepts all three. It just cannot be masked on GitLab, so the key can appear in a job log if something echoes it.

Storing it in a vault

vault reads a HashiCorp Vault or OpenBao KV v2 secret. The expected layout, overridable with VAULT_KV_MOUNT / VAULT_SECRET_PATH / VAULT_SECRET_FIELD (or the BAO_* spellings):

kv/infra/ansible/deploy-key      field: private_key

A read-only policy is enough:

path "kv/data/infra/ansible/deploy-key" {
  capabilities = ["read"]
}

Keep the vault off the deploy path. If the vault runs on a host this repository deploys to, and it seals on reboot, then a deploy that fetches its key from the vault cannot run — including the deploy that would fix that host. Host it elsewhere, or auto-unseal it. A CI secret is the dependable route.

2. Per-stack .env files

Stack secrets live on the hosts, not here.

The role excludes .env from every sync (stack_exclude), so editing one in this repository has no effect, and a host's .env is never overwritten by a deploy. To change one, edit it on the host and redeploy.

Each example stack that needs credentials ships a .env.example listing the variable names with no values, and reads them in its compose file with Compose's required-variable syntax:

    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set it in .env on the host}

That is worth copying. Without the :?, a missing variable expands to the empty string and Postgres starts with no password at all — a failure that looks like success until someone finds it. With it, Compose refuses to start and names the variable.

webapp.yml goes one step further and asserts the file exists on the host before doing anything, so the error arrives before the payload is synced rather than halfway through starting the stack:

  pre_tasks:
    - name: Look for the webapp .env on the host
      ansible.builtin.stat:
        path: "{{ webapp_dir }}/.env"
      register: webapp_env
      changed_when: false

    - name: Verify the webapp .env is present on the host
      ansible.builtin.assert:
        that:
          - webapp_env.stat.exists
        fail_msg: >-
          {{ webapp_dir }}/.env is missing on {{ inventory_hostname }}.
        quiet: true

Do this for any stack whose compose file has no safe defaults.

3. Secrets committed to the repository

There is no third category. This section exists because it is the failure mode this layout is designed to prevent, and it is worth being blunt about why.

A credential written directly into a compose file:

• cannot be excluded from the sync. stack_exclude keeps .env off the wire; it can do nothing about a value inside docker-compose.yml.
• is in every clone, forever. Git history is the point of git. Deleting the file in a later commit changes nothing about the credential.
• cannot be rotated without a deploy, so it usually is not rotated at all.
• escapes its blast radius. A personal access token carries its creator's access to everything they can reach, not just the project it was added to. A cloud application secret outlives the VM it was used on.

If you find one, the fix is to rotate the credential, not to delete the file. Removing it stops new clones carrying it and changes nothing else. Then move it to a .env on the host and read it with ${VAR:?}.

The guard

The lint workflow — .github/workflows/lint.yml, .gitea/workflows/lint.yml and the lint job in .gitlab-ci.yml — runs gitleaks on every push, over the working tree and the full history: a secret that was committed and then removed is still leaked, and the scan should say so. .gitleaks.toml allowlists only *.env.example files and this document.

On GitHub it runs the upstream Docker image rather than the marketplace action on purpose: the action requires a licence key for organisation-owned repositories and silently does nothing without one, which is the worst possible behaviour for a secret scanner. The Gitea port runs the pinned release binary instead, because a job there is itself a container with no Docker socket.

Do not add an allowlist entry to make a finding go away. Rotate the credential.