lucasw89/config-public
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add the host bootstrap: deploy user, key, docker access, stack root

Browse Source 
bootstrap.yml prepares a host as an admin via -u; new-deploy-key.sh makes the
key pair and refuses to write inside the repo. CI deploys refuse the bootstrap
playbook, since the dispatch API ignores the dropdown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed 2026-09-30 15:07:31 +02:00
1 parent 78b227f32d
commit 792f7cde48
10 files changed
+443 -9
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitea/workflows/deploy.yml
+4
View File
@@ -113,6 +113,10 @@ jobs:
set -eu
case "$STACK" in
''|*[!a-z0-9_-]*) echo "refusing STACK='$STACK'"; exit 1 ;;
# Has a playbook, so the test -f below would pass it, but it must
# run as an administrator — see docs/bootstrap.md. The dropdown
# leaves it out; the API does not.
bootstrap) echo "refusing STACK='$STACK': run it by hand"; exit 1 ;;
esac
case "$MODE" in
check|sync|deploy) ;;
.github/workflows/deploy.yml
+4
View File
@@ -97,6 +97,10 @@ jobs:
set -eu
case "$STACK" in
''|*[!a-z0-9_-]*) echo "refusing STACK='$STACK'"; exit 1 ;;
# Has a playbook, so the test -f below would pass it, but it must
# run as an administrator — see docs/bootstrap.md. The dropdown
# leaves it out; the API does not.
bootstrap) echo "refusing STACK='$STACK': run it by hand"; exit 1 ;;
esac
case "$MODE" in
check|sync|deploy) ;;
.gitlab-ci.yml
+3
View File
@@ -100,6 +100,9 @@ ansible:ping:
# run webapp's backup and migration against the wrong stack.
- STACK="${STACK_FIXED:-${STACK:-}}"
- case "$STACK" in ''|*[!a-z0-9_-]*) echo "refusing STACK='$STACK'"; exit 1;; esac
# bootstrap has a playbook but must run as an administrator, by hand -- see
# docs/bootstrap.md.
- case "$STACK" in bootstrap) echo "refusing STACK='$STACK': run it by hand"; exit 1;; esac
# PULL reaches the same command line, unquoted, from the same untrusted place
# as STACK, so it gets the same treatment: an unchecked value can smuggle in
# extra `-e` overrides (stack_dest, ansible_host) and redirect a deploy.
AGENTS.md
+2 -1
View File
@@ -102,7 +102,8 @@ stack needs something new; do not work around it with loose tasks.
Worked examples for each: `static-site.yml` (the minimum), `reverse-proxy.yml`
(`stack_dirs`, reload), `metrics.yml` (`stack_src`, per-host maps, ownership),
`webapp.yml` (two phases, `stack_files`, `stack_build`), `webapp-staging.yml`
(`stack_src` + `stack_dest` + `stack_env`), `banner.yml` (not a stack at all).
(`stack_src` + `stack_dest` + `stack_env`), `banner.yml` (not a stack at all),
`bootstrap.yml` (preparing a host; see `docs/bootstrap.md`).
#### `stack_dirs`
README.md
+46 -2
View File
@@ -64,13 +64,15 @@ ansible/
├── roles/compose_stack/ # the one shared "sync payload, bring stack up" role
└── playbooks/
├── site.yml # every stack, in dependency order
├── bootstrap.yml # prepares a host; run once, by an admin
└── <stack>.yml # one per stack, runnable on its own
server/<host>/<stack>/ # the payload: compose files and config, copied verbatim
server/shared/ # payloads used by more than one host
scripts/load-ssh-key.sh # produces the deploy key path (file / CI / vault)
scripts/new-deploy-key.sh # generates the deploy key pair
scripts/check.sh # the full static gate — what CI runs
skills/adding-a-stack/ # step-by-step guide to adding a service
docs/ # secrets, CI setup, connectivity, triggering deploys
docs/ # bootstrap, secrets, CI setup, connectivity, deploys
AGENTS.md # the conventions and the role contract, in full
```
@@ -86,10 +88,15 @@ AGENTS.md # the conventions and the role contract, in full
> lives in a world-writable directory, and a runner's build directory is exactly
> that.
**A host that has never been deployed to needs preparing first** — a `deploy`
user, its key, the `docker` group, rsync, and a writable `/srv/stacks`. That is
`scripts/new-deploy-key.sh` and `playbooks/bootstrap.yml`, once per host; see
[docs/bootstrap.md](docs/bootstrap.md).
```bash
cd ansible
ansible-galaxy install -r requirements.yml
export SSH_KEY_PATH=~/.ssh/your_deploy_key # see Secrets, below
export SSH_KEY_PATH=~/.ssh/deploy_key # see Secrets, below
```
Then check what a deploy *would* do before doing it:
@@ -98,6 +105,40 @@ Then check what a deploy *would* do before doing it:
ansible-playbook --check --diff playbooks/metrics.yml
```
### Passwordless sudo
Almost everything a deploy does runs as `deploy`, unescalated, and that is
deliberate: a bind mount created by root is one the container cannot write.
The exception is a `stack_dirs` entry that sets `owner` or `group`. Handing a
directory to another uid needs root, so the role escalates for exactly those
entries and nothing else. `metrics.yml` does it for Grafana (uid 472), because
an image that does not run as root cannot repair its own data directory from
the inside.
Without it, those deploys stop here:
```
TASK [compose_stack : Create the extra directories the stack needs]
[ERROR]: Task failed: Missing sudo password
```
There is no password to supply — `deploy` has none. Grant `NOPASSWD` once, as
an administrator, either over SSH or on the host itself:
```bash
cd ansible
ansible-playbook playbooks/bootstrap.yml -u <admin-user> -k -K -e deploy_sudo=true
ansible-playbook playbooks/bootstrap.yml -c local -K -e deploy_sudo=true
```
Leaving `-e deploy_sudo=true` off is the easy mistake: the run still ends green
and the only sign is `skipped=1` in the recap. Confirm it took with
`ssh deploy@<host> sudo -n true && echo ok`.
This cannot come from CI, and that is the point: the runner holds only the
unprivileged deploy key. Why `-u` works, what `NOPASSWD:ALL` costs and how to
narrow it are in [docs/bootstrap.md](docs/bootstrap.md).
### On Windows
Ansible has no native Windows control node — `ansible-core` needs a POSIX system,
@@ -262,6 +303,9 @@ is in [`skills/adding-a-stack/SKILL.md`](skills/adding-a-stack/SKILL.md).
| A stack refuses to start, naming a variable | Its `.env` is missing on the host — that is the design working |
| `Permission denied (publickey)` | The matching public key is not in the deploy user's `authorized_keys` |
| `Connection timed out` | A network path problem, not a key problem — see [`docs/connectivity.md`](docs/connectivity.md) |
| `Failed to find required executable "rsync"` | `synchronize` needs rsync at **both** ends; read the PATH in the error to tell the runner from the host |
| `Missing sudo password` on *Create the extra directories* | The host was bootstrapped without `-e deploy_sudo=true` — see [Passwordless sudo](#passwordless-sudo) |
| `"/srv/stacks/<stack>" is not a directory` | A file is in the way; the role says which. A first `--check` skips Compose instead — see [`docs/bootstrap.md`](docs/bootstrap.md) |
## License
ansible/playbooks/bootstrap.yml
+124
View File
@@ -0,0 +1,124 @@
---
# Prepare a host for everything else in this repository: the deploy user, its
# key, its access to Docker, and the directory the stacks live in.
#
# The one playbook that cannot connect as the deploy user, because it is what
# creates it. Run it as an administrator who already has access:
#
# ansible-playbook playbooks/bootstrap.yml -u <admin-user> -k -K
#
# -u is what makes this work, and it works only because the deploy account is
# `remote_user` in ansible.cfg rather than ansible_user in the inventory. An
# inventory variable outranks -u; a config default does not. From the host
# itself, `-c local -K` skips SSH and the question with it.
#
# -k asks for the SSH password and -K for the sudo password; drop either if that
# account already uses a key or has passwordless sudo. SSH_KEY_PATH is not
# needed — an empty ansible_ssh_private_key_file is ignored by the ssh plugin,
# so group_vars/all.yml does not get in the way here either.
#
# Deliberately not imported by site.yml and not offered in the CI deploy
# dropdown: a converge run and the runner both authenticate as the deploy user,
# which is exactly what does not exist yet.
- name: Bootstrap a host for deployment
hosts: all
gather_facts: false
become: true
tags: [bootstrap]
vars:
deploy_user: deploy
deploy_pubkey_file: ~/.ssh/deploy_key.pub
# The compose_stack role escalates only when a stack_dirs entry sets owner
# or group, which is how a bind mount gets the uid its container runs as.
# metrics and openbao do that, so a host running either wants
# -e deploy_sudo=true. Still off by default: this grants NOPASSWD:ALL, and a
# host with no such stack has no reason to carry it.
deploy_sudo: false
tasks:
- name: Look for the public key on the control node
ansible.builtin.stat:
path: "{{ deploy_pubkey_file | expanduser }}"
register: bootstrap_pubkey
delegate_to: localhost
become: false
changed_when: false
# Before the user is half-created, rather than after.
- name: Verify the public key exists
ansible.builtin.assert:
that:
- bootstrap_pubkey.stat.exists | default(false)
fail_msg: >-
no public key at {{ deploy_pubkey_file }} —
run scripts/new-deploy-key.sh first
quiet: true
- name: Look for the docker group on the host
ansible.builtin.getent:
database: group
key: docker
fail_key: false
# Without this the user task fails with "Group docker does not exist", which
# is true and says nothing about the actual problem.
- name: Verify Docker is installed
ansible.builtin.assert:
that:
- ansible_facts.getent_group.docker is defined
fail_msg: >-
no docker group on {{ inventory_hostname }} — install Docker first
quiet: true
# The payload sync is ansible.posix.synchronize, which needs rsync at both
# ends. Installed rather than asserted, unlike Docker above: rsync is one
# package from the base repositories, where installing Docker means choosing
# repositories and versions and is a decision this playbook should not make
# on your behalf.
- name: Install rsync
ansible.builtin.package:
name: rsync
state: present
# In the docker group because the role runs compose unescalated: that is what
# keeps the bind mounts from being created root-owned.
- name: Create the deploy user
ansible.builtin.user:
name: "{{ deploy_user }}"
groups: docker
append: true
shell: /bin/bash
create_home: true
- name: Authorise the deploy key
ansible.posix.authorized_key:
user: "{{ deploy_user }}"
key: "{{ lookup('ansible.builtin.file', deploy_pubkey_file | expanduser) }}"
# Never exclusive. Rewriting the whole file would drop any other key that
# is currently somebody's only way onto this host.
exclusive: false
# The role creates {{ stack_root }}/<stack> unescalated, and /srv is
# root-owned, so without this the first deploy fails on a permission error
# that looks nothing like its cause.
- name: Create the stack root
ansible.builtin.file:
path: "{{ stack_root }}"
state: directory
owner: "{{ deploy_user }}"
group: "{{ deploy_user }}"
mode: "0755"
- name: Allow the deploy user to escalate
ansible.builtin.copy:
dest: "/etc/sudoers.d/{{ deploy_user }}"
content: "{{ deploy_user }} ALL=(ALL) NOPASSWD:ALL\n"
owner: root
group: root
mode: "0440"
# A malformed sudoers file can lock everyone out of a host permanently.
# validate runs visudo against the candidate and refuses to install it
# if it does not parse.
validate: visudo -cf %s
when: deploy_sudo | bool
docs/architecture.md
+7 -5
View File
@@ -45,11 +45,13 @@ the same logic, extend the contract, do not add a second role.** When a stack
needs something the contract cannot express, that is information — say so, rather
than working around it with loose tasks.
`banner.yml` is the deliberate exception. It is not a Compose stack at all: one
shared file dropped into another stack's directory. Forcing it through a role
built around "sync a directory, then `docker compose up`" would mean weakening
the role for one caller. Plain tasks are the right answer, and the fact that
there is exactly one such playbook is the signal that the contract is holding.
`banner.yml` and `bootstrap.yml` are the deliberate exceptions. Neither is a
Compose stack at all: one drops a single shared file into another stack's
directory, the other prepares a host — a user, a key, a directory, a sudoers
file. Forcing either through a role built around "sync a directory, then
`docker compose up`" would mean weakening the role for one caller. Plain tasks
are the right answer, and the fact that there are so few such playbooks is the
signal that the contract is holding.
## Two entry points, not two playbooks
docs/bootstrap.md
+155
View File
@@ -0,0 +1,155 @@
# Preparing a host
Everything else in this repository assumes a host that already has a `deploy`
user, a key it trusts, access to Docker, and a writable stack root. This is how
that gets made. It runs once per host.
There is a chicken-and-egg problem in the middle of it: every other playbook
connects **as** `deploy`, so the playbook that creates `deploy` cannot. It
connects as an administrator you already have access with, and it is the only
thing in this repository that does.
## 1. Generate the deploy key
```bash
scripts/new-deploy-key.sh # ~/.ssh/deploy_key
scripts/new-deploy-key.sh ~/.ssh/other # somewhere else
```
`ed25519`, and no passphrase — nothing in CI or in a playbook run can type one,
and an encrypted key fails much later with an authentication error that says
nothing about a passphrase.
The script prints the paths and the public key. It does not print the private
key unless you pass `--print-secret`, for the same reason
[`load-ssh-key.sh`](../scripts/load-ssh-key.sh) does not: key material should
stay out of terminals, logs and the process list.
It refuses two things. It will not overwrite an existing key without `--force`,
because replacing a live deploy key locks you out of every host that still
trusts the old one. And it will not write anywhere inside this repository:
gitleaks scans the working tree **and** the full history, so a key committed
once stays leaked after you delete the file.
One key serves every host and every stack. Rotating it is therefore an
all-hosts operation — replace the line in each `authorized_keys`, then update
the one CI secret.
## 2. Prepare the host
```bash
cd ansible
ansible-playbook playbooks/bootstrap.yml -u <admin-user> -k -K -e deploy_sudo=true
```
`deploy_sudo=true` is what grants the deploy user passwordless sudo, and it is
off by default. Leave it off only on a host where no stack sets `owner` or
`group` in `stack_dirs` — those entries are the only thing the role escalates
for, and `metrics.yml` and `openbao.yml` both have them. Omit it and the run
still ends green, with `skipped=1` in the recap as the only sign; the
consequence turns up much later, in a different playbook, as
`Missing sudo password`.
`-u` is honoured because the deploy account is `remote_user` in `ansible.cfg`
and not `ansible_user` in the inventory. That distinction exists for this
playbook: an inventory variable outranks a command-line `-u`, so naming the
account there would make the flag a silent no-op and send the run in as the
user that does not exist yet — or, on a host already bootstrapped, as the one
that cannot escalate.
Running it on the host itself avoids SSH altogether:
```bash
cd ansible
ansible-playbook playbooks/bootstrap.yml -c local -K -e deploy_sudo=true
```
`-c local` executes as whoever invoked it and ignores `ansible_user` altogether.
`-k` asks for that account's SSH password, `-K` for its sudo password; drop
either if it already uses a key or has passwordless sudo. `SSH_KEY_PATH` is not
needed here. The public key is read from whichever machine runs Ansible, so an
on-host run needs `deploy_pubkey_file` to point at a copy that is there.
| Variable | Default | |
| --- | --- | --- |
| `deploy_user` | `deploy` | Must match `remote_user` in `ansible/ansible.cfg` |
| `deploy_pubkey_file` | `~/.ssh/deploy_key.pub` | Read from the control node |
| `deploy_sudo` | `false` | Passwordless sudo for the deploy user |
What it does, and why each part matters:
- **Creates `deploy` in the `docker` group.** The `compose_stack` role runs
Compose unescalated, which is what stops bind mounts being created
root-owned. Without group membership every Docker call fails on the socket.
- **Installs rsync.** The payload sync is `ansible.posix.synchronize`, which
shells out to rsync **at both ends**. Missing it on the target fails the first
deploy; missing it on the machine running Ansible fails the same task with the
same message, so read the PATH in the error to tell the two apart.
- **Installs the public key**, never exclusively — rewriting the whole
`authorized_keys` would drop a key that is currently somebody's only way in.
- **Creates `{{ stack_root }}` owned by `deploy`.** This is the one that bites
silently: the role creates `/srv/stacks/<stack>` unescalated and `/srv` is
root-owned, so skipping it produces a permission error on first deploy that
looks nothing like its cause.
- **Grants passwordless sudo only when you ask.** The role escalates when a
`stack_dirs` entry sets `owner` or `group`, which is how a bind mount gets the
uid its container runs as. `metrics.yml` and `openbao.yml` both need it, so
any host running either wants `-e deploy_sudo=true`. The sudoers file is
written through `visudo -cf`, because a malformed one can lock everyone out of
a host permanently. `NOPASSWD:ALL` is broad — narrowing it to the commands the
`file` module actually invokes is a reasonable hardening once a host settles.
It is not imported by `site.yml` and not offered in the CI deploy dropdown.
Both authenticate as `deploy` — which on a fresh host does not exist yet, and on
a bootstrapped one is the unprivileged account this playbook exists to grant
privilege to. Either way it cannot bootstrap itself. Making CI able to would
mean storing an administrator's credential where the runner can reach it, a
larger and permanent grant in place of one interactive command.
**`--check` does not work on a host that has never been bootstrapped.** It stops
at *Authorise the deploy key* with `Either user must exist or you must provide
full path to key file in check mode`: check mode did not really create the user,
so the module has no home directory to inspect. That is Ansible's limitation,
not a fault in the run. On an already-bootstrapped host check mode is clean.
### rsync on the machine running Ansible
The half this playbook cannot do for you. `synchronize` runs rsync locally and
pushes, so whatever drives the deploy needs the binary as well:
- **A CI runner.** GitHub's hosted runners ship it. The GitLab jobs install it
in the `.ansible` `before_script`. The Gitea deploy installs it through the
toolchain action, because the act_runner images do not carry it. A runner
without it fails with `Failed to find required executable "rsync"` and a PATH
full of `/opt/hostedtoolcache/...`, which is the giveaway that the message is
about the runner and not about your host.
- **Your workstation**, if you deploy from it: `apt install rsync`, or the
equivalent.
### The first deploy of a stack cannot be a check run
`--check` creates nothing, so on a host that has never run a given stack there
is no project directory for Compose to inspect. The role notices and skips the
compose step with a note rather than failing, and the payload diff above it is
still reported — but the stack itself is only really created by a real deploy.
## 3. Hand the key to CI
```bash
base64 -w0 < ~/.ssh/deploy_key
```
Store it as `SSH_PRIVATE_KEY_B64`. Where that lives differs per platform —
[secrets.md](secrets.md) has the table, the other backends, and what not to use
on Windows.
## 4. Confirm it before deploying anything
Run the **connectivity** workflow first (`ansible:ping` on GitLab).
`SUCCESS => {"ping": "pong"}` proves
the runner decoded the key, reached port 22, authenticated, and ran Python on
the far side — everything a deploy needs except the playbook.
If it fails, the message tells you which of three different problems you have;
[connectivity.md](connectivity.md) has the table.
scripts/new-deploy-key.sh
Executable
+97
View File
@@ -0,0 +1,97 @@
#!/usr/bin/env bash
#
# Generate the Ansible deploy key pair.
#
# scripts/new-deploy-key.sh # ~/.ssh/deploy_key
# scripts/new-deploy-key.sh ~/.ssh/other # somewhere else
# scripts/new-deploy-key.sh --force # replace an existing key
# scripts/new-deploy-key.sh --print-secret # also print the value for CI
#
# Prints the paths and the public key. It does not print the private key unless
# you ask for it: load-ssh-key.sh keeps key material off stdout and out of logs
# and the process list, and this script holds the same line.
#
# The public key belongs in the deploy user's authorized_keys, which
# ansible/playbooks/bootstrap.yml does for you. The private key belongs in the
# CI secret SSH_PRIVATE_KEY_B64. See docs/bootstrap.md and docs/secrets.md.
set -euo pipefail
umask 077
log() { printf 'new-deploy-key: %s\n' "$*" >&2; }
die() { log "$*"; exit 1; }
force=0
print_secret=0
key_path=""
while [ $# -gt 0 ]; do
case "$1" in
--force) force=1 ;;
--print-secret) print_secret=1 ;;
-h|--help) sed -n '3,9p' "$0" | sed 's/^#[[:space:]]\{0,1\}//'; exit 0 ;;
-*) die "unknown option: $1" ;;
*)
[ -z "$key_path" ] || die "only one key path may be given"
key_path="$1"
;;
esac
shift
done
key_path="${key_path:-$HOME/.ssh/deploy_key}"
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
key_dir="$(dirname "$key_path")"
[ -d "$key_dir" ] || die "no such directory: $key_dir"
key_dir="$(cd "$key_dir" && pwd)"
# gitleaks scans the working tree *and* the full history, and a key committed
# once stays leaked after the file is deleted. Refuse before that can happen.
case "$key_dir/" in
"$repo_root"/*) die "refusing to write a private key inside the repository: $repo_root" ;;
esac
if [ -e "$key_path" ] && [ "$force" -ne 1 ]; then
die "$key_path already exists. Replacing it locks out every host that still
trusts the old key, so pass --force if that is what you want."
fi
# An `if`, not `[ ... ] && rm ...`: an AND-list whose test fails returns
# non-zero, which under `set -e` is a trap this repository has been bitten by
# before -- see the same note in .gitea/workflows/deploy.yml.
if [ "$force" -eq 1 ]; then
rm -f "$key_path" "$key_path.pub"
fi
# ed25519, and no passphrase: nothing in CI or in a playbook run can type one,
# and an encrypted key fails later with an authentication error that says
# nothing about the passphrase.
ssh-keygen -t ed25519 -f "$key_path" -N "" -C "ansible-deploy@$(basename "$repo_root")" >/dev/null
printf 'private key %s\n' "$key_path"
printf 'public key %s\n\n' "$key_path.pub"
cat "$key_path.pub"
cat <<INSTRUCTIONS
Next:
1. Put it on the host, and create everything else it needs:
cd ansible
ansible-playbook playbooks/bootstrap.yml -u <admin-user> -k -K \\
-e deploy_pubkey_file=$key_path.pub
2. Store the private key as the CI secret SSH_PRIVATE_KEY_B64 (where, per
platform, is in docs/secrets.md):
base64 -w0 < $key_path
3. Confirm it works before any deploy: the connectivity workflow on GitHub
or Gitea, the ansible:ping job on GitLab.
INSTRUCTIONS
if [ "$print_secret" -eq 1 ]; then
printf '\nSSH_PRIVATE_KEY_B64:\n'
base64 -w0 < "$key_path"
printf '\n'
fi
skills/adding-a-stack/SKILL.md
+1 -1
View File
@@ -45,7 +45,7 @@ files and one button. `AGENTS.md` holds the conventions and the full
| If | Do | Example |
| --- | --- | --- |
| It is not a Compose stack — one file into another stack's directory, or a system service | Write plain tasks. Do not force the role | `banner.yml` |
| It is not a Compose stack — one file into another stack's directory, host preparation, or a system service | Write plain tasks. Do not force the role | `banner.yml`, `bootstrap.yml` |
| Something must happen between the payload landing and the stack starting (backup, migration) | Use the two entry points, `tasks_from: sync` then `tasks_from: up`, and gate the middle on `pull \| default('policy') == 'always'` | `webapp.yml` |
| The payload is bind-mounted config, which Compose never recreates a container for | Add a `post_tasks` reload keyed on `compose_stack_synced` | `reverse-proxy.yml`, `metrics.yml` |
| The compose file is not named `docker-compose.yml`, or there are several | `stack_files` | `webapp.yml` |