config-public/docs/secrets.md

# Secrets

Three separate things, handled three different ways. The split matters: the first
is needed to *reach* a host, the second is needed by the software *on* it, and
the third is a mistake.

## 1. The deploy SSH key

Ansible knows exactly one thing about credentials: **`SSH_KEY_PATH`**, a path to
a `0600` private key. `inventory/group_vars/all.yml` feeds it to
`ansible_ssh_private_key_file` and nothing else in the repository touches key
material.

Everything about *where the key comes from* lives in `scripts/load-ssh-key.sh`,
which prints a path and never prints the key:

```bash
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh)"         # env:   already on disk
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh ci)"      # ci:    from a CI secret
export SSH_KEY_PATH="$(scripts/load-ssh-key.sh vault)"   # vault: from Vault / OpenBao
```

Add a backend there rather than teaching a playbook about secrets. That seam is
the reason the same playbooks run unchanged from a laptop, from GitHub Actions,
from GitLab CI and from Gitea Actions.

Whichever backend you use, the matching **public** key must be in the deploy
user's `~/.ssh/authorized_keys` on every host, or the key is useless.

### Storing it in CI

Store the key **base64-encoded**, as a single line:

```bash
base64 -w0 < ~/.ssh/deploy_key            # Linux
base64 < ~/.ssh/deploy_key | tr -d '\n'   # macOS
```

```powershell
# Windows PowerShell
[Convert]::ToBase64String([IO.File]::ReadAllBytes("$HOME\.ssh\deploy_key"))
```

On Windows use exactly that. `certutil -encode` wraps its output in a header and
line breaks, and `Get-Content` re-encodes the bytes — both produce something that
looks like base64 and decodes to an unusable key. Do not re-wrap the output
either; it is one long line by design.

Then set it as `SSH_PRIVATE_KEY_B64`:

| Platform | Where | Notes |
| --- | --- | --- |
| GitHub | Settings → Environments → `production` → Secrets | Scoped to the environment, so only jobs naming it can read it |
| GitLab | Settings → CI/CD → Variables | Mark **Masked** and **Protected** |
| Gitea | Settings → Actions → Secrets | No environments and no masking controls: any run in the repository can read it. See [ci-gitea.md](ci-gitea.md) |

Why base64 rather than the raw key? GitLab can only *mask* a value that is a
single line with no whitespace, and an OpenSSH private key is neither. GitHub
redacts secrets from logs regardless, but a single-line value survives being
passed between steps without surprises. One encoding satisfies both.

A raw `SSH_PRIVATE_KEY` still works — as a GitLab File-type variable, as a
GitHub multi-line secret, or as a path on disk. The loader accepts all three. It
just cannot be masked on GitLab, so the key can appear in a job log if something
echoes it.

### Storing it in a vault

`vault` reads a HashiCorp Vault or OpenBao KV v2 secret. The expected layout,
overridable with `VAULT_KV_MOUNT` / `VAULT_SECRET_PATH` / `VAULT_SECRET_FIELD`
(or the `BAO_*` spellings):

```
kv/infra/ansible/deploy-key      field: private_key
```

A read-only policy is enough:

```hcl
path "kv/data/infra/ansible/deploy-key" {
  capabilities = ["read"]
}
```

**Keep the vault off the deploy path.** If the vault runs on a host this
repository deploys to, and it seals on reboot, then a deploy that fetches its key
from the vault cannot run — including the deploy that would fix that host. Host
it elsewhere, or auto-unseal it. A CI secret is the dependable route.

That is also why **the deploy key does not live in the stacks' OpenBao** (§2),
even when one runs on the host. You need SSH to reach the host the vault runs
on, so a key fetched from that vault could never get you there in the first
place. Two stores, deliberately: CI holds the key that reaches the host, the
vault holds what the stacks need once you are there.

## 2. Per-stack secrets

Stack secrets never live in this repository. There are two ways to get them onto
a host, and both end in the same place: a `.env` next to the stack's compose
file, which Compose reads and the sync never touches (`stack_exclude`).

Either way, compose files read each value with Compose's required-variable
syntax:

```yaml
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?missing from .env}
```

That is worth copying. Without the `:?`, a missing variable expands to the empty
string and Postgres starts with **no password at all** — a failure that looks
like success until someone finds it. With it, Compose refuses to start and names
the variable. Each stack also ships a `.env.example` listing the names with no
values, as documentation.

### From OpenBao, with `stack_secrets` (recommended)

A playbook names what it needs, and the `compose_stack` role renders it:

```yaml
    stack_secrets:
      SECRET_KEY:
        path: stacks/webapp
        field: secret_key
```

The role reads `kv/stacks/webapp` from `http://127.0.0.1:8200` **on the target
host**, at the end of the sync, and writes `{{ stack_dest }}/.env` at `0600`.
Nothing is placed by hand, and nothing — plaintext or encrypted — is committed.
`webapp.yml` and `webapp-staging.yml` are the worked examples, reading from the
`openbao` stack on the same host.

A sealed vault, an unset token, a 403 or 404 from the vault, a field that is
not there, and a value that cannot be safely written to a `.env` all fail before
the stack starts, each naming what to fix. No task prints a value: the reads are
`no_log`, and the asserts that explain a failure see only names, paths and
status codes.

#### Why the read happens on the host

Because the vault is published to `127.0.0.1:8200` and nowhere else. Ansible
tasks already run on the host, so the store never needs a public hostname, never
needs a certificate, and does not care whether the deploy came from a laptop or
a CI runner. `ansible.cfg` sets `pipelining = True`, so module arguments are
piped to the remote interpreter over stdin and the token is never written to a
file on the host. That covers the token specifically — it does not make every
transfer file-free: `ansible.builtin.copy` still lands its content as a `0600`
temp file under `~/.ansible/tmp` on both ends, removed when the run ends,
regardless of pipelining.

The consequence is **one vault per host that reads from it**, or a `vault_addr`
the host can reach. In this repository only `app` runs one; `metrics`, which
also runs on `edge`, uses a hand-placed `.env` instead (below).

#### The token

`VAULT_TOKEN` is read from the control node's environment: your shell locally,
and a CI secret in a runner. It carries a `stacks-read` policy granting `read`
on `kv/data/stacks/*` and nothing else. It is **not** the root token.

The name is not configurable. `compose_stack` reads it with
`lookup('env', 'VAULT_TOKEN')`, and the deploy workflows pass it through as an
environment variable rather than interpolating it into a `run:` body — an
expression is substituted before the shell sees it, so a value containing shell
metacharacters would be executed rather than assigned.

Where to put it, and how long it lasts, are steps 5 and 6 of *Bootstrapping the
vault* below.

#### Unsealing

OpenBao has no self-contained auto-unseal; it needs an external KMS or a second
vault. The example stack is unsealed by hand:

```bash
ssh [email protected]
docker exec -it openbao bao operator unseal   # three times, three different keys
```

(`docker exec` inherits the container's environment, including the `BAO_ADDR`
set in `docker-compose.yml` — so none of the commands on this page need their
own `-e BAO_ADDR=...`. Without it the `bao` CLI defaults to
`https://127.0.0.1:8200` and every one of these fails with a TLS handshake
error, since the listener has TLS disabled.)

After a reboot the vault is sealed and any deploy that reads a secret fails,
naming the vault and pointing here. That is the intended behaviour: a sealed
vault is a loud, safe failure.

#### Adding a secret

Putting the root token and the secret value on the command line, as
`-e BAO_TOKEN=<root>` and `<field>=<value>`, lands both in the host's process
table and in the operator's shell history. Log in interactively instead, and
pipe the value in on stdin so it never appears as an argument:

```bash
ssh [email protected]
docker exec -it openbao bao login   # prompts for the root token

read -rs SECRET
printf '%s' "$SECRET" | docker exec -i openbao bao kv put kv/stacks/<stack> <field>=-
unset SECRET
```

`read -rs` does not echo and does not reach shell history, and `printf '%s'`
adds no trailing newline. **That last part matters.** `<field>=-` stores stdin
byte for byte, including the newline you press Enter for, and the role refuses
to render a value containing a newline or a single quote — a `.env` line it
cannot quote safely is one Compose would misread. The deploy then fails on
*Verify no secret value contains a newline or a single quote*, which is the
guard working, but it is a round trip you can skip.

Generating a value has the same trap, because every generator ends its output
with a newline:

```bash
openssl rand -hex 32 | tr -d '\n' | docker exec -i openbao bao kv put kv/stacks/<stack> <field>=-
```

`kv put` replaces every field at that path. To add a second field beside an
existing one, use `bao kv patch` instead, or the first is lost.

Then add the entry to the playbook's `stack_secrets` and deploy. Rotating is the
same `kv put` (or `kv patch`) followed by a redeploy.

Removing an entry from `stack_secrets` stops the render on the next deploy; it
does not delete the `.env` already on the host, which keeps serving its stale
value until something else removes it.

#### A changed openbao.hcl needs a manual restart

Restarting OpenBao seals it. `openbao.yml` therefore reports that its payload
changed rather than restarting the container, because an automatic restart in
the middle of `site.yml` would seal the vault and fail every stack imported
after it. Restart and unseal deliberately.

#### Bootstrapping the vault

Hosting the vault on the host it serves is circular, so the order is fixed and
the middle of it is manual. This is the whole sequence, once per vault.

**1. Prepare the host and deploy the stack.**

```bash
cd ansible
ansible-playbook playbooks/bootstrap.yml -u <admin-user> -k -K -e deploy_sudo=true
ansible-playbook playbooks/openbao.yml
```

`deploy_sudo` is not optional here. `openbao.yml` hands `data/` to uid 100, and a
`stack_dirs` entry with an `owner` is the only thing the role escalates for.
Without it the deploy stops at *Create the extra directories the stack needs*
with `Missing sudo password`. The stack declares no `stack_secrets` of its own —
it is the root of trust and cannot depend on itself — so it comes up with
nothing in place. See [bootstrap.md](bootstrap.md).

**2. Initialise, once, ever.**

```bash
ssh [email protected]
docker exec -it openbao bao operator init
```

Five unseal keys and one root token are printed **once**. They are not written
to the host and cannot be recovered. Put all six in your password manager before
doing anything else: three of the five unseal the vault, and losing three makes
the storage unreadable permanently.

**3. Unseal.**

```bash
docker exec -it openbao bao operator unseal   # three times, three different keys
docker exec openbao bao status                # expect: Sealed  false
```

**4. Create the mount and the read-only policy.**

Log in interactively rather than passing `-e BAO_TOKEN=<root>`, for the reason
under *Adding a secret* above — an argument lands in the host's process table.

```bash
docker exec -it openbao bao login   # prompts for the root token

docker exec openbao bao secrets enable -path=kv kv-v2

printf 'path "kv/data/stacks/*" {\n  capabilities = ["read"]\n}\n' \
  | docker exec -i openbao bao policy write stacks-read -
```

**5. Mint the deploy token and store it.**

```bash
docker exec openbao bao token create \
  -policy=stacks-read -period=768h -display-name=ansible
```

This token, never the root one, is `VAULT_TOKEN`. Locally it is an export:

```bash
export VAULT_TOKEN=<the stacks-read token>
```

In CI it is a secret of the same name:

| Platform | Where | Notes |
| --- | --- | --- |
| GitHub | Settings → Environments → `production` → Secrets | Scoped to the environment, so only jobs naming it can read it |
| GitLab | Settings → CI/CD → Variables | Mark **Masked** and **Protected** |
| Gitea | Settings → Actions → Secrets | No environments and no masking controls: any run in the repository can read it — which is precisely why the token is scoped. See [ci-gitea.md](ci-gitea.md) |

Unlike the SSH key in §1, this one needs no base64: a vault token is already a
single line with no whitespace, which is the only thing GitLab's masking
requires.

**6. Diarise the expiry.**

`-period=768h` makes the token renewable indefinitely; it does not make it
permanent. It lapses 32 days after it was last renewed, and the first sign is a
deploy failing on the vault read after a quiet month.

Re-minting is the reliable fix — repeat step 5 and replace the secret. To renew
instead, log in as the token rather than passing it as an argument, where the
host's process table would keep it:

```bash
docker exec -it openbao bao login   # paste the stacks-read token
docker exec openbao bao token renew # renews the token you logged in with
docker exec openbao sh -c 'rm -f ~/.bao-token'
```

A longer `-period` at mint time is the other lever, traded against how long a
leaked token stays useful.

**7. Prove the token cannot write.**

```bash
docker exec -e BAO_TOKEN=<the new token> openbao bao kv put kv/stacks/canary value=nope
```

This must fail with a 403. If it succeeds the policy is wrong and anything
holding the deploy token can write to the store. Nothing else surfaces a policy
typo, so do not skip it.

**8. Load the secrets and converge.**

Once per secret, per *Adding a secret* above — mind the trailing newline. For
the examples that is `postgres_password` and `secret_key` under
`kv/stacks/webapp`, and the same two under `kv/stacks/webapp-staging`:

```bash
openssl rand -hex 32 | tr -d '\n' \
  | docker exec -i openbao bao kv put kv/stacks/webapp postgres_password=-
openssl rand -hex 32 | tr -d '\n' \
  | docker exec -i openbao bao kv patch kv/stacks/webapp secret_key=-
```

Then converge with `ansible-playbook playbooks/site.yml` from `ansible/`.

When you are finished, `bao login` has left the root token in the container at
`~/.bao-token`; remove it with `docker exec openbao sh -c 'rm -f ~/.bao-token'`.

Steps 2–7 are manual and stay manual. That is the accepted cost of not having a
second machine to hold the keys.

### By hand, on the host

A stack with no `stack_secrets` never talks to a vault, and reads a `.env`
someone placed on the host. Editing one in this repository has no effect, and a
host's `.env` is never overwritten by a deploy. To change one, edit it on the
host and redeploy.

`metrics` works this way, because it runs on `edge` too and `edge` has no
vault. Its playbook asserts the file exists before doing anything, so the error
arrives before the payload is synced rather than halfway through starting the
stack:

```yaml
  pre_tasks:
    - name: Look for the metrics .env on the host
      ansible.builtin.stat:
        path: "{{ stack_root }}/metrics/.env"
      register: metrics_env
      changed_when: false

    - name: Verify the metrics .env is present on the host
      ansible.builtin.assert:
        that:
          - metrics_env.stat.exists
        fail_msg: >-
          {{ stack_root }}/metrics/.env is missing on {{ inventory_hostname }}.
        quiet: true
```

Do this for any hand-placed stack whose compose file has no safe defaults.

## 3. Secrets committed to the repository

There is no third category. This section exists because it is the failure mode
this layout is designed to prevent, and it is worth being blunt about why.

A credential written directly into a compose file:

- **cannot be excluded from the sync.** `stack_exclude` keeps `.env` off the
  wire; it can do nothing about a value inside `docker-compose.yml`.
- **is in every clone, forever.** Git history is the point of git. Deleting the
  file in a later commit changes nothing about the credential.
- **cannot be rotated without a deploy**, so it usually is not rotated at all.
- **escapes its blast radius.** A personal access token carries its creator's
  access to everything they can reach, not just the project it was added to. A
  cloud application secret outlives the VM it was used on.

If you find one, the fix is to **rotate the credential**, not to delete the file.
Removing it stops new clones carrying it and changes nothing else. Then move it
to a `.env` on the host and read it with `${VAR:?}`.

### The guard

The `lint` workflow — [`.github/workflows/lint.yml`](../.github/workflows/lint.yml),
[`.gitea/workflows/lint.yml`](../.gitea/workflows/lint.yml) and the `lint` job in
`.gitlab-ci.yml` — runs [gitleaks](https://github.com/gitleaks/gitleaks) on every
push, over the working tree **and** the full history: a secret that was committed
and then removed is still leaked, and the scan should say so. `.gitleaks.toml`
allowlists only `*.env.example` files and this document.

On GitHub it runs the upstream Docker image rather than the marketplace action on
purpose: the action requires a licence key for organisation-owned repositories
and silently does nothing without one, which is the worst possible behaviour for
a secret scanner. The Gitea port runs the pinned release binary instead, because
a job there is itself a container with no Docker socket.

Do not add an allowlist entry to make a finding go away. Rotate the credential.