diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..3a29d90 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,43 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "permissions": { + "allow": [ + "Bash(make check:*)", + "Bash(scripts/check.sh:*)", + "Bash(./scripts/check.sh:*)", + "Bash(uvx ansible-lint:*)", + "Bash(uvx yamllint:*)", + "Bash(ansible-lint:*)", + "Bash(yamllint:*)", + "Bash(shellcheck:*)", + "Bash(ansible-doc:*)", + "Bash(ansible-config:*)", + "Bash(ansible-inventory:*)", + "Bash(ansible-galaxy install:*)", + "Bash(uvx --from ansible-core ansible-playbook --syntax-check:*)", + "Bash(ansible-playbook --syntax-check:*)", + "Bash(git status:*)", + "Bash(git diff:*)", + "Bash(git log:*)", + "Bash(git branch:*)" + ], + "ask": [ + "Bash(ansible-playbook:*)", + "Bash(ansible:*)", + "Bash(ssh:*)", + "Bash(scp:*)", + "Bash(rsync:*)" + ], + "deny": [ + "Read(./**/.env)", + "Read(./**/*.env)", + "Read(./**/vault.yml)", + "Read(./**/*vault_pass*)", + "Read(./**/id_rsa*)", + "Read(./**/id_ed25519*)", + "Read(./**/*.pem)", + "Bash(ansible-vault view:*)", + "Bash(ansible-vault decrypt:*)" + ] + } +} diff --git a/.claude/skills/adding-a-stack/SKILL.md b/.claude/skills/adding-a-stack/SKILL.md new file mode 100644 index 0000000..703b01d --- /dev/null +++ b/.claude/skills/adding-a-stack/SKILL.md @@ -0,0 +1,10 @@ +--- +name: adding-a-stack +description: Use when adding a new service to this repository, moving a service to a different host, or bringing a hand-deployed service under Ansible and CI. +--- + +# Adding a stack + +The instructions live in [`skills/adding-a-stack/SKILL.md`](../../../skills/adding-a-stack/SKILL.md), +outside `.claude/` so that every agent reads the same file. Read it and follow +it. Do not duplicate guidance here. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7417221 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,231 @@ +# AGENTS.md + +Instructions for LLMs and AI agents working in this repository. Humans should +read `README.md` first; this file assumes you already know what the repo is for. + +This is the one instruction file. `CLAUDE.md` and `GEMINI.md` are pointers to it, +and `.claude/skills/` points at `skills/`. Put content here or in `skills/`, +never in a pointer — the whole arrangement exists so that every agent, and every +human, reads the same text. + +## What this repository is + +A pattern for deploying Docker Compose stacks to a small number of Linux hosts +with Ansible, published as a working example. Every service on every host is a +Compose stack living in a directory under `/srv/stacks/` on the target. This repo +holds the files that get copied there, and the playbooks that copy them. + +| Host | Address | Runs | +| ------ | ------------------ | ----------------------------------- | +| `edge` | `edge.example.com` | The reverse proxy, metrics | +| `app` | `app.example.com` | The applications behind it, metrics | + +Both use SSH user `deploy` on port 22. + +Everything under `server/` and every host name is an **example**. `example.com` +is reserved by RFC 2606 and can never resolve to a real service. Replace them +with yours; the framework around them is the part meant to be reused. + +## Vocabulary + +- **host** — one machine (`edge`, `app`). The top-level directories under + `server/` and the inventory names must always agree, because the role resolves + a payload as `server/{{ inventory_hostname }}/{{ stack_name }}`. +- **stack** — one Docker Compose deployment on one host (`metrics`, `webapp`). +- **payload** — the files under `server///`, copied to the host + verbatim. Payloads are *data*. Do not template them, do not restructure them, + and do not lint them as Ansible content. + +## Skills + +Step-by-step guides for recurring jobs live in `skills//SKILL.md`, outside +any vendor directory. + +| Skill | Use when | +| --- | --- | +| `adding-a-stack` | Adding a new service, moving one to a different host, or bringing a hand-deployed service under Ansible and CI | + +## Playbook rules + +These are the conventions. Follow them; do not invent per-stack variations. + +1. **One playbook per stack**, at `ansible/playbooks/.yml`. If the same + stack runs on several hosts it is still one playbook with a host group, not + one per host. `metrics.yml` is the example. +2. **Self-contained.** Every playbook must run standalone as + `ansible-playbook playbooks/.yml` with no required extra vars — `pull` + is the one optional flag. It declares its own `hosts`, its own vars, and + everything the stack needs (directories, networks, compose files). Never rely + on another playbook having run first. Where that means two playbooks create + the same directory, that is correct and the repetition is deliberate. +3. **Tag every play** with the stack name, so `site.yml --tags metrics` works. +4. **Reuse goes in `roles/compose_stack`, not in copy-pasted tasks.** If two + playbooks need the same logic, extend the role's variable contract rather than + adding a second role. Keep the role count at one unless there is a genuinely + different shape of work. +5. **Modules, not shell.** `community.docker.docker_compose_v2` rather than + `docker compose up`; `ansible.builtin.file` rather than `mkdir -m 777`. Reach + for `command`/`shell` only when no module exists, and then set `changed_when` + explicitly. +6. **Fully-qualified collection names** everywhere (`ansible.builtin.copy`, not + `copy`). +7. **Name every task**, sentence case, imperative: `Sync the metrics payload`, + not `sync files`. +8. **Idempotent.** A second run must report zero changes, and `--check --diff` + must not error. +9. **No secrets in the repository.** `.env` files stay on the hosts. Never commit + a key, a password, or a token — including inside a compose file, where no + exclude can help you. See `docs/secrets.md`. + +### The `compose_stack` role contract + +Playbooks configure the role through these variables. Extend this list when a +stack needs something new; do not work around it with loose tasks. + +| Variable | Default | Purpose | +| ---------------- | ----------------------------------------------------- | ------------------------------------------- | +| `stack_name` | *required* | Stack identifier, used for logs and tags | +| `stack_src` | `server/{{ inventory_hostname }}/{{ stack_name }}` | Payload directory in this repository | +| `stack_dest` | `{{ stack_root }}/{{ stack_name }}` | Destination directory on the host | +| `stack_dirs` | `[]` | Directories to pre-create (see below) | +| `stack_networks` | `[]` | External Docker networks to ensure exist | +| `stack_files` | `[docker-compose.yml]` | Compose files passed to the module | +| `stack_exclude` | `[.env]` | Paths never synced to the host | +| `stack_prune` | `false` | Delete host files absent from the payload | +| `stack_env` | `{}` | Extra env for compose; `PWD` is always set | +| `stack_pull` | `{{ pull \| default('policy') }}` | Per-invocation override of the pull flag | +| `stack_build` | `policy` | `always` for stacks built from a Dockerfile | +| `stack_state` | `present` | `present`, `absent`, or `restarted` | + +Worked examples for each: `static-site.yml` (the minimum), `reverse-proxy.yml` +(`stack_dirs`, reload), `metrics.yml` (`stack_src`, per-host maps, ownership), +`webapp.yml` (two phases, `stack_files`, `stack_build`), `webapp-staging.yml` +(`stack_src` + `stack_dest` + `stack_env`), `banner.yml` (not a stack at all). + +#### `stack_dirs` + +Entries take a `path` plus optional `mode`, `owner`, `group` and `recurse`. All +four are applied **only when given**, so a directory that already exists on the +host is never re-chmodded — several bind mounts are created root-owned by Docker +on first start, and Postgres refuses to start if its data directory is not +`0700`/`0750`. + +The role escalates **only when an entry sets `owner` or `group`.** Everything +else is created as the deploy user, which is the whole point of pre-creating a +bind mount: a root-owned directory is exactly what you are trying to avoid. A +stack that genuinely needs another uid says so with `owner`/`group` — see +`metrics.yml`, which hands `grafana-data` to `472` with `recurse: true` and no +`mode`. + +Put that in `stack_dirs` rather than a `pre_tasks` chown. `pre_tasks` run before +the role creates the stack directory, and `ansible.builtin.file` stamps the +attributes it was given onto every parent directory it creates on the way down — +so an escalated chown of a child leaves the stack directory itself owned by that +uid, and the payload sync, which runs as the deploy user, cannot write into it. + +#### `compose_stack_synced` + +After the sync the role sets this fact. Use it to fire a reload for payloads that +are bind-mounted configuration, which Compose never recreates a container for: + +```yaml + post_tasks: + - name: Reload nginx so changed vhost and conf files take effect + ansible.builtin.command: + cmd: docker exec reverse-proxy nginx -s reload + when: compose_stack_synced | default(false) + changed_when: true +``` + +Without the `when`, every run reports a change and the stack is never idempotent. + +#### Doing maintenance between sync and up + +`roles: - role: compose_stack` syncs and then starts, which is what almost every +stack wants. When something has to happen in between — a backup, a schema +migration — use the two entry points rather than inventing a second mechanism: + +```yaml + tasks: + - name: Sync the payload + ansible.builtin.include_role: + name: compose_stack + tasks_from: sync + + - name: Migrate the database + # ... gated on an update run + - name: Start the stack + ansible.builtin.include_role: + name: compose_stack + tasks_from: up +``` + +Declare the `stack_*` vars at **play** level so both phases see them; inside +`roles:` they are scoped to that invocation and the second include will not see +them. `webapp.yml` is the worked example. + +### Deploy mode versus update mode + +Every playbook serves two purposes and must support both without being edited: + +```bash +ansible-playbook playbooks/webapp.yml # bring the stack up +ansible-playbook playbooks/webapp.yml -e pull=always # pull newer images +``` + +`pull` is a single repo-wide extra-var passed straight through to +`community.docker.docker_compose_v2`. Valid values are `policy` (the default — +Compose decides, so nothing is fetched for images already present), `always`, +`missing` and `never`. Read it as `pull | default('policy')` at every use site so +the playbook still runs when the var is undefined. + +This is also the hook for the heavier work a stack needs only when its images +actually move. Gate those tasks on the same flag rather than creating a separate +`*-update.yml` playbook: + +```yaml +- name: Back up the database before migrating + ansible.builtin.command: + cmd: docker compose -f docker-compose.yml -f docker-compose.migrate.yml run --rm backup + chdir: /srv/stacks/webapp + when: pull | default('policy') == 'always' + changed_when: true +``` + +Two consequences worth stating plainly: **a default run must never destroy or +migrate anything**, and **an update run must be safe to repeat.** + +## Safety + +- **Never run a playbook against a real host unless the user asks in that turn.** + Run `--check --diff` first and say what it would change. +- **Never read or print the contents of `.env` files, vault files, or SSH keys.** + If a task genuinely needs one, ask. +- Do not `git push`, open pull requests, or commit unless asked. + +## Verifying your work + +No Ansible tooling is assumed to be installed in a fresh checkout. One command +runs everything CI runs: + +```bash +make check # or: scripts/check.sh +``` + +It runs `yamllint` from the repository root, installs the Galaxy collections, +runs `ansible-lint` at the `production` profile, and syntax-checks every +playbook. Locally it reaches for `uvx` so the repo needs no virtualenv of its +own; set `RUNNER=installed` if the tools are already on `PATH`. + +Two things worth knowing if you run the commands by hand instead: + +- **Run every ansible command from the `ansible/` directory.** `ansible.cfg` is + only discovered in the current directory, and it is what puts `roles/` on the + roles path. From the repository root you get + `The role 'compose_stack' was not found`. +- **Install the collections before linting**, or `community.docker` and + `ansible.posix` resolve to nothing and the fully-qualified-name checks pass + vacuously. + +Lint and syntax-check are the definition of "done" for a playbook. They are +static and safe; run them yourself rather than asking the user to. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..dc970e2 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,6 @@ +# CLAUDE.md + +The instructions for this repository live in `AGENTS.md`, so that every agent and +editor reads the same file. Do not duplicate guidance here — edit `AGENTS.md`. + +@AGENTS.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..4062f35 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,62 @@ +# Contributing + +## Running the checks + +One command, and it is the same one CI runs: + +```bash +make check # or: scripts/check.sh +``` + +It runs `yamllint` over the repository, installs the Galaxy collections, runs +`ansible-lint` at the `production` profile, and syntax-checks every playbook. + +Locally it reaches for [`uv`](https://docs.astral.sh/uv/) so this repository +needs no virtualenv of its own. If you already have the tools on `PATH`, use +`RUNNER=installed scripts/check.sh` to skip that. + +The version pins live in `scripts/check.sh` and nowhere else; both CI +configurations ask the script for them, so a workflow cannot drift from what you +ran locally. + +**Lint and syntax-check are the definition of done for a playbook.** They are +static and safe. + +## What the checks will not catch + +Nothing here talks to a host, so the gate cannot tell you that a deploy works. +Before proposing a change to a playbook, run it against your own host with +`--check --diff` and say in the pull request what it reported. If you cannot, +say that too — it is useful information, and better than an implied claim. + +## Conventions + +They are in [`AGENTS.md`](AGENTS.md), which is written for AI agents and is +equally the contributor guide. The short version: + +- One playbook per stack, self-contained, tagged with the stack name. +- Reuse goes in the `compose_stack` role's variable contract, not in copy-pasted + tasks or a second role. +- Modules, not `shell`. Fully-qualified collection names. Every task named. +- A second run must report zero changes. +- No secrets in the repository — not in a `.env`, and especially not inline in a + compose file. See [`docs/secrets.md`](docs/secrets.md). + +[`skills/adding-a-stack/SKILL.md`](skills/adding-a-stack/SKILL.md) is the +step-by-step version for the most common change. + +## Changing the examples + +The six example stacks exist to demonstrate specific features of the role +contract — the table in [`server/README.md`](server/README.md) says which. If you +change one, check that whatever it was demonstrating is still demonstrated +somewhere, or the documentation that points at it stops being true. + +New examples are welcome if they show something the current six do not. An +example that shows the same thing a seventh way is a cost, not a contribution. + +## Reporting a security issue + +Do not open a public issue for a vulnerability. Note that the example +credentials, hostnames and addresses in this repository are all fictional; +`example.com` is reserved by RFC 2606 and cannot resolve. diff --git a/GEMINI.md b/GEMINI.md new file mode 100644 index 0000000..547b370 --- /dev/null +++ b/GEMINI.md @@ -0,0 +1,6 @@ +# GEMINI.md + +The instructions for this repository live in `AGENTS.md`, so that every agent and +editor reads the same file. Do not duplicate guidance here — edit `AGENTS.md`. + +@AGENTS.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..4ccdc3a --- /dev/null +++ b/README.md @@ -0,0 +1,256 @@ +# compose-stack-deploy + +A small, complete pattern for deploying Docker Compose stacks to a handful of +Linux hosts with Ansible — and for letting CI do it safely, one stack at a time, +on purpose rather than on every push. + +It is deliberately not a framework. There is **one** Ansible role, about 90 lines +of tasks, and a convention about where files live. Everything else in this +repository is documentation and six worked examples. + +``` +server/app/webapp/ → rsync → /srv/stacks/webapp/ on app +ansible/playbooks/webapp.yml docker compose up -d +``` + +## Is this for you? + +It fits if you have somewhere between two and twenty services on a few hosts, +each one a `docker-compose.yml`, and you are currently deploying them by hand or +with a shell script you no longer trust. + +It is the wrong tool if you want orchestration, scheduling, or rolling updates. +That is Kubernetes or Nomad, and you should use one of those. + +## The idea, in three parts + +**Payloads are data.** A stack's compose file and its configuration live under +`server///` and are copied to the host *verbatim*. Nothing is +templated. You can read the file in the repo and know exactly what is on the +host, and you can `docker compose up` it by hand in an emergency. + +**One role, one contract.** Everything a stack can need — networks to create, +directories to pre-create, which compose files, what to exclude from the sync, +whether to pull — is a variable on a single role. A new service is usually four +lines: + +```yaml +- name: Deploy the static site + hosts: app + gather_facts: false + tags: [static-site] + + roles: + - role: compose_stack + vars: + stack_name: static-site + stack_networks: [proxy-net] +``` + +**One key, one seam.** Ansible knows exactly one thing about credentials: +`SSH_KEY_PATH` points at a `0600` private key. A single script decides where that +key comes from — a file, a CI secret, a vault — so adding a backend never touches +a playbook. + +## Layout + +``` +ansible/ +├── ansible.cfg # read only from this directory — see below +├── requirements.yml # Galaxy collections +├── inventory/ +│ ├── hosts.yml +│ └── group_vars/all.yml # connection settings, stack_root, the pull default +├── roles/compose_stack/ # the one shared "sync payload, bring stack up" role +└── playbooks/ + ├── site.yml # every stack, in dependency order + └── .yml # one per stack, runnable on its own +server/// # the payload: compose files and config, copied verbatim +server/shared/ # payloads used by more than one host +scripts/load-ssh-key.sh # produces the deploy key path (file / CI / vault) +scripts/check.sh # the full static gate — what CI runs +skills/adding-a-stack/ # step-by-step guide to adding a service +docs/ # secrets, CI setup, connectivity, triggering deploys +AGENTS.md # the conventions and the role contract, in full +``` + +## Getting started + +> **Run every ansible command from the `ansible/` directory.** `ansible.cfg` is +> only discovered in the current working directory, and it is what points at the +> inventory and puts `roles/` on the roles path. From the repository root you +> will get `The role 'compose_stack' was not found`. +> +> Alternatively export `ANSIBLE_CONFIG=/path/to/repo/ansible/ansible.cfg`, which +> works from anywhere. CI has to do this: Ansible ignores an `ansible.cfg` that +> lives in a world-writable directory, and a runner's build directory is exactly +> that. + +```bash +cd ansible +ansible-galaxy install -r requirements.yml +export SSH_KEY_PATH=~/.ssh/your_deploy_key # see Secrets, below +``` + +Then check what a deploy *would* do before doing it: + +```bash +ansible-playbook --check --diff playbooks/metrics.yml +``` + +### On Windows + +Ansible has no native Windows control node — `ansible-core` needs a POSIX system, +and the scripts here are shell scripts. Use **WSL2** and run everything inside +it; the commands above then work unchanged. + +Two things that will cost you an afternoon if you skip them: + +- **Clone into the WSL filesystem** (`~/config`), not `/mnt/c/...`. On the + Windows drive `chmod 600` does not stick, and OpenSSH refuses a private key + whose permissions are too open. It is also markedly slower. +- **Never open a private key or a shell script in Notepad.** It rewrites the file + with CRLF line endings, which makes SSH reject the key and makes deployed + scripts fail on the host with `bad interpreter: /bin/bash^M`. The + `.gitattributes` here keeps LF endings on checkout, but it cannot protect a + file you edit outside git. + +## Deploying + +```bash +# One stack +ansible-playbook playbooks/metrics.yml + +# Everything, in dependency order +ansible-playbook playbooks/site.yml + +# One stack out of site.yml +ansible-playbook playbooks/site.yml --tags metrics +``` + +### Deploy versus update + +This distinction is the one piece of the design worth internalising. + +A plain run brings the stack up with the images already on the host. It never +pulls, migrates, backs up, or destroys anything. Run it as often as you like. + +```bash +ansible-playbook playbooks/webapp.yml -e pull=always +``` + +`pull=always` fetches newer images and recreates changed containers. It is *also* +the switch for the heavier per-stack work: `webapp` only runs its backup and +database migration on an update run. + +If your images are tagged `latest`, a plain deploy will **not** pick up a new +build — Compose is content with the image it already has. Use `pull=always` when +you mean to update. + +Verify with `--check --diff` first. These are production servers. + +## The examples + +Six stacks, each chosen to demonstrate one thing. Read them in this order. + +| Stack | Hosts | What it shows | +| --- | --- | --- | +| [`static-site`](ansible/playbooks/static-site.yml) | app | The minimum: a name and a network. **Copy this one.** | +| [`reverse-proxy`](ansible/playbooks/reverse-proxy.yml) | edge | Pre-creating bind mounts; reloading after a config-only change | +| [`metrics`](ansible/playbooks/metrics.yml) | edge, app | One payload shared by two hosts; per-host variable maps; directories owned by a container's uid | +| [`webapp`](ansible/playbooks/webapp.yml) | app | Backup and migrate *between* sync and up, gated on an update run; a second compose file; building from a Dockerfile; requiring a host-side `.env` | +| [`webapp-staging`](ansible/playbooks/webapp-staging.yml) | app | A second instance of the same payload, elsewhere, with different settings | +| [`banner`](ansible/playbooks/banner.yml) | edge, app | The escape hatch: something that is not a Compose stack at all | + +`server/` is data — see [`server/README.md`](server/README.md). + +## Secrets + +Three separate things, handled three different ways. Full detail in +[`docs/secrets.md`](docs/secrets.md). + +**1. The deploy SSH key.** Ansible only ever reads `SSH_KEY_PATH`. +`scripts/load-ssh-key.sh` is the only thing that knows where the key comes from; +it prints a path and never prints key material. + +```bash +export SSH_KEY_PATH="$(scripts/load-ssh-key.sh)" # key already on disk +export SSH_KEY_PATH="$(scripts/load-ssh-key.sh ci)" # from a CI secret +export SSH_KEY_PATH="$(scripts/load-ssh-key.sh vault)" # from Vault / OpenBao +``` + +**2. Per-stack `.env` files.** Stack secrets live on the hosts, not here. The +role excludes `.env` from every sync, so editing one in the repo has no effect +and a host's `.env` is never overwritten. The example stacks use Compose's +`${VAR:?message}` syntax, so a stack refuses to start with a missing credential +rather than booting with an empty password. + +**3. Nothing else.** There is no third category, and that is the point. A secret +written directly into a compose file cannot be excluded from the sync, cannot be +rotated without a commit, and is in every clone of the repository forever. CI +runs [gitleaks](https://github.com/gitleaks/gitleaks) over the working tree *and* +the history on every push to keep it that way. + +## CI/CD + +All three platforms are maintained, and all three do the same two things. + +| | GitHub Actions | GitLab CI | Gitea Actions | +| --- | --- | --- | --- | +| Static gate | [`lint.yml`](.github/workflows/lint.yml) | `lint` job | [`lint.yml`](.gitea/workflows/lint.yml) | +| Deploy | [`deploy.yml`](.github/workflows/deploy.yml) | per-stack buttons | [`deploy.yml`](.gitea/workflows/deploy.yml) | +| Diagnostics | [`connectivity.yml`](.github/workflows/connectivity.yml) | `ansible:ping` | [`connectivity.yml`](.gitea/workflows/connectivity.yml) | +| Setup guide | [`ci-github-actions.md`](docs/ci-github-actions.md) | [`ci-gitlab.md`](docs/ci-gitlab.md) | [`ci-gitea.md`](docs/ci-gitea.md) | + +Gitea reads `.gitea/workflows/` *instead of* `.github/workflows/` — it stops at +the first workflow directory that exists — so the two never both run. Delete the +platforms you are not using rather than letting them rot. + +The rules that matter, on all three: + +- **The lint gate runs on every push** and must pass. It needs no access to any + host. It is `scripts/check.sh` — the same command you run locally — so it + cannot drift from what you tested. +- **Deploys are never automatic.** Merging changes what *would* be deployed; a + person still decides when. Every deploy path is manual or externally triggered. +- **Check before deploy.** All three expose a check mode that runs + `--check --diff` and touches nothing. +- **To update rather than deploy**, set the pull policy to `always`. +- **Untrusted input is validated.** The stack name and pull policy reach a shell + command and, for externally triggered deploys, come from another repository. + Both are matched against a fixed pattern before anything runs. + +Another repository can ask this one to redeploy a stack after building a new +image — see [`docs/triggering-deploys.md`](docs/triggering-deploys.md). + +## Adding a stack + +1. Put the compose file and its configuration in `server///`. +2. Add `ansible/playbooks/.yml`. Copy `static-site.yml`. +3. Add it to `site.yml`, in dependency order. +4. Add it to CI: the `stack` dropdown in `.github/workflows/deploy.yml` and in + `.gitea/workflows/deploy.yml`, and a `check:`/`deploy:` pair in + `.gitlab-ci.yml`. Skip this and the stack works from a workstation but is + invisible in CI, which is how services end up hand-deployed. +5. `make check`, then `--check --diff` against the host. + +The long version, with a table of which contract variable solves which problem, +is in [`skills/adding-a-stack/SKILL.md`](skills/adding-a-stack/SKILL.md). +[`AGENTS.md`](AGENTS.md) has the conventions and the full role contract. + +## Troubleshooting + +| Symptom | Cause | +| --- | --- | +| `The role 'compose_stack' was not found` | Not running from the `ansible/` directory | +| `SSH_KEY_PATH does not point at a readable file` | Key not loaded, or a CI job that cannot see the secret | +| A deploy succeeds but the service is unchanged | Image tag is `latest` and you did not pass `-e pull=always` | +| Config file copied but nothing happens | Bind-mounted config; the stack needs a reload, which the playbook does only when the sync changed something | +| `no payload directory at ...` | `stack_src` does not match where the files actually are | +| A stack refuses to start, naming a variable | Its `.env` is missing on the host — that is the design working | +| `Permission denied (publickey)` | The matching public key is not in the deploy user's `authorized_keys` | +| `Connection timed out` | A network path problem, not a key problem — see [`docs/connectivity.md`](docs/connectivity.md) | + +## License + +MIT. See [LICENSE](LICENSE). diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..3a3e7c9 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,138 @@ +# Why it is shaped this way + +Short notes on the decisions that are not obvious from the code, and what each +one is buying. Read `AGENTS.md` for the rules; this is the reasoning behind them. + +## Payloads are data, not templates + +A stack's compose file is copied to the host **verbatim**. Nothing is templated, +nothing is generated. + +This is the decision most likely to look wrong at first, because templating is +what configuration management is *for*. The argument against it here: + +- **You can read the repository and know what is on the host.** With templating + you know what the host will get after rendering, which is not the same thing + at three in the morning. +- **You can `docker compose up` it by hand.** When the deploy path is broken — + and the deploy path is what breaks — the payload is still a working compose + project you can run from an SSH session. +- **Upstream projects ship compose files.** Nextcloud, Gitea, Discourse and + most self-hosted software give you one. Keeping it verbatim means you can diff + against theirs when you upgrade. Templating it means you re-derive your changes + every time. +- **Per-host differences belong in the playbook.** Which networks exist, which + directories to create, which compose files to use: these are deployment + concerns and the role contract expresses all of them. See `metrics.yml`, which + covers two hosts from one payload without templating anything. + +The cost is real: genuinely dynamic values have to come from the environment +rather than from Ansible. In practice that means a `.env` on the host, which is +where the secrets had to live anyway. + +## One role, and a contract + +There is one role. Every stack-specific need is a variable on it, and the list is +documented in `AGENTS.md` as a contract. + +The alternative — a role per stack, or a role per *kind* of stack — is how these +repositories usually grow, and the failure mode is consistent: twelve roles that +are each 90% the same, diverging slowly, so that a fix to the sync logic has to +be applied twelve times and is applied to nine. + +The rule that keeps it honest is in `AGENTS.md` rule 4: **if two playbooks need +the same logic, extend the contract, do not add a second role.** When a stack +needs something the contract cannot express, that is information — say so, rather +than working around it with loose tasks. + +`banner.yml` is the deliberate exception. It is not a Compose stack at all: one +shared file dropped into another stack's directory. Forcing it through a role +built around "sync a directory, then `docker compose up`" would mean weakening +the role for one caller. Plain tasks are the right answer, and the fact that +there is exactly one such playbook is the signal that the contract is holding. + +## Two entry points, not two playbooks + +`compose_stack` exposes `sync` and `up` separately, so a playbook that must do +work in between — a backup, a schema migration — can interleave its own tasks. + +The alternative is a second playbook, `webapp-update.yml`, and the reason to +avoid it is that the two immediately drift. The update playbook gains a network +the deploy playbook does not have; someone fixes a directory in one and not the +other. One playbook that behaves differently under a flag cannot drift from +itself. + +## Deploy and update are one flag + +`pull` is a single repository-wide extra-var. `policy` is the default and means +"Compose decides", so nothing is fetched for an image already present. `always` +fetches newer images **and** gates the heavier per-stack work. + +Two properties follow, and both are load-bearing: + +- **A default run must never destroy or migrate anything.** You should be able to + run `site.yml` at any time, against everything, without thinking about it. That + is what makes it useful for convergence after a manual change. +- **An update run must be safe to repeat.** It is not transactional. Something + will fail halfway through, and the fix is to run it again. + +## The `SSH_KEY_PATH` seam + +Ansible knows one thing about credentials: a path to a private key. One script +decides where that key comes from. + +The payoff is that the same playbooks run unchanged from a laptop, from GitHub +Actions, from GitLab CI and from Gitea Actions, and that adding a secret backend +is a change to one +file that no playbook imports. The script prints a path and never prints key +material, so it is safe to call in a CI log. + +The constraint that makes it work is worth stating: **nothing else in the +repository touches key material.** The moment a playbook learns how to read a +vault, the seam is gone. + +## Nothing deploys on push + +Every deploy path is manual or externally triggered. Merging changes what *would* +be deployed; a person still decides when. + +This is a judgement call, not a universal truth. It is right for a handful of +long-lived stateful services where a bad deploy means restoring a database, and +where the person merging is often not the person who should be watching the +deploy. It is wrong for a fleet of stateless services with good rollback, where +continuous deployment is the whole point. + +If you adopt this repository and your situation is the second one, the change is +small — add a `push` trigger to the deploy workflow — but make it deliberately. + +## Bind mounts and the directories nobody creates + +A surprising amount of the role contract exists because of one Docker behaviour: +**a bind mount whose host path does not exist is created by Docker, owned by +root.** The container then cannot write to it, and the failure surfaces as an +application error rather than a permissions one. + +`stack_dirs` pre-creates them as the deploy user. Three details are the result of +getting this wrong: + +- **Attributes apply only when set.** A directory that already exists on the host + keeps its permissions. Re-chmodding a live bind mount is how you break a + Postgres data directory, which refuses to start unless it is `0700` or `0750`. +- **Escalate only for `owner`/`group`.** Creating these as root defeats the + purpose. The role escalates only when an entry actually asks for another uid. +- **It belongs in `stack_dirs`, not `pre_tasks`.** `pre_tasks` run before the role + creates the stack directory, and `ansible.builtin.file` stamps its attributes + onto every parent it creates on the way down. An escalated chown of a child + leaves the *stack* directory owned by that uid, and the payload sync — running + as the deploy user — then cannot write into it. + +## Bind-mounted config needs a reload + +Compose recreates a container when the compose file changes. It does not know or +care that a file bind-mounted *into* the container changed, so a new nginx vhost +or Prometheus scrape config lands on the host and has no effect. + +The role sets `compose_stack_synced` after the sync, and playbooks key a reload +off it. The `when:` is what keeps the playbook idempotent — without it every run +reports a change forever, and "changed=0 means nothing happened" stops being +true, which is the only cheap signal these deploys have. diff --git a/docs/connectivity.md b/docs/connectivity.md new file mode 100644 index 0000000..cc5e6ba --- /dev/null +++ b/docs/connectivity.md @@ -0,0 +1,105 @@ +# When CI cannot reach a host + +A deploy job that **hangs and then times out** is a different problem from one +that fails, and the distinction tells you where to look. + +| What you see | What it means | Where the problem is | +| --- | --- | --- | +| `Permission denied (publickey)` | You reached sshd and it rejected the key | The key, or `authorized_keys` | +| `Connection refused` | You reached the host; nothing is listening on that port | sshd is down, or on another port | +| `Connection timed out` | The packets vanished in transit | The network path | + +Only the third is a network problem. No amount of re-pasting the deploy key will +fix it, and it is worth being sure which one you have before spending an +afternoon on the wrong thing. + +## The probe + +`connectivity.yml` has two jobs. + +**`ping`** uses the real deploy key and asks Ansible to talk to every host in the +inventory. Success looks like `SUCCESS => {"ping": "pong"}` for each one, which +proves the runner decoded the key, reached port 22, authenticated, and ran Python +on the far side. That is everything a deploy needs except the playbook. Run this +first, always. + +**`probe`** uses no credentials at all and answers a narrower question: *where do +the packets stop?* Run it when `ping` times out. It reports: + +| Section | Question | +| --- | --- | +| 0 | Does the runner have working DNS and outbound HTTPS at all? And what is its egress address? | +| 1 | Does each target host answer with an SSH banner? | +| 2 | Does SSH to a known-good third party work — is outbound 22 blocked wholesale? | +| 3 | For each host and port: open, refused, or filtered? | + +## Reading the result + +| What you see | What it means | +| --- | --- | +| Section 0 has no DNS or no external HTTPS | The runner has no usable egress at all. Talk to whoever runs it. | +| Section 2 gets `SSH-2.0-...` but section 1 does not | Outbound 22 works generally; your hosts are dropping this runner specifically. Take the egress address to your firewall or cloud security group. | +| Section 2 also gets no reply | Outbound 22 is blocked wholesale on the runner's network. | +| Every port but 443 `filtered` in section 3 | An egress allowlist. No alternative SSH port will help — see below. | + +Two details in the probe are deliberate and worth keeping if you rewrite it. + +**It asks for the SSH banner, not just a TCP connect.** A bare connect can be +answered by a transparent proxy that then says nothing, which looks like success +and is not. A real sshd greets first, so `SSH-2.0-...` is the only honest proof +you reached the actual server. + +**It distinguishes `refused` from `filtered`.** "Refused" means you reached the +host and nothing was listening, so that port is *allowed out*. "Filtered" means +the packets vanished. That difference is what tells you whether you are looking +at a service problem or a network policy, and it is the one thing a plain +`timeout` in a deploy log never tells you. + +## Fixing it + +### Move the runner, not the port + +If you find an egress allowlist, resist the urge to move sshd to a different +port. An allowlist permits a *set* of destination ports; it is not a block on 22, +so there is nowhere to move to. Test this rather than assuming it — the probe's +section 3 will tell you in a minute. + +The reliable fix is a runner that sits somewhere with a working path to the +hosts. A CI runner only makes **outbound HTTPS** connections to its coordinator, +which is why this works where an inbound exception would need a policy change and +someone else's timeline. + +```bash +# GitHub: Settings > Actions > Runners > New self-hosted runner +# GitLab: +sudo gitlab-runner register --url https://gitlab.example.com --token +# Gitea: +act_runner register --instance https://gitea.example.com --token \ + --labels deploy-runner:docker://docker.gitea.com/runner-images:ubuntu-latest +``` + +Then point the workflows at it — `DEPLOY_RUNNER` on GitHub, the `tags:` block in +`.gitlab-ci.yml` on GitLab, and the `runs-on:` label in `.gitea/workflows/` on +Gitea, which has to be edited in the file because Gitea takes no expression +there. All three are described in the CI setup docs. + +Two prerequisites, easy to forget: the host firewall or cloud security group must +allow SSH **from the runner's address**, and the deploy key must be authorised +for the deploy user on each host. + +### Or ask for an exception + +Ask whoever runs the network to allow the runner's egress address outbound to +your hosts on port 22. Less work on your side, but it depends on someone else's +policy and timeline, and it has to be renewed whenever the runner moves. + +### In the meantime + +Deploy from a workstation that can reach the hosts. The playbooks are identical +either way — that is the point of the `SSH_KEY_PATH` seam. + +## A note on ICMP + +Many cloud providers drop ICMP by default, so a failed `ping` to a host proves +nothing on its own. The probe does not test it for that reason. Judge +reachability by the SSH banner and the port states. diff --git a/docs/secrets.md b/docs/secrets.md new file mode 100644 index 0000000..09af76d --- /dev/null +++ b/docs/secrets.md @@ -0,0 +1,170 @@ +# Secrets + +Three separate things, handled three different ways. The split matters: the first +is needed to *reach* a host, the second is needed by the software *on* it, and +the third is a mistake. + +## 1. The deploy SSH key + +Ansible knows exactly one thing about credentials: **`SSH_KEY_PATH`**, a path to +a `0600` private key. `inventory/group_vars/all.yml` feeds it to +`ansible_ssh_private_key_file` and nothing else in the repository touches key +material. + +Everything about *where the key comes from* lives in `scripts/load-ssh-key.sh`, +which prints a path and never prints the key: + +```bash +export SSH_KEY_PATH="$(scripts/load-ssh-key.sh)" # env: already on disk +export SSH_KEY_PATH="$(scripts/load-ssh-key.sh ci)" # ci: from a CI secret +export SSH_KEY_PATH="$(scripts/load-ssh-key.sh vault)" # vault: from Vault / OpenBao +``` + +Add a backend there rather than teaching a playbook about secrets. That seam is +the reason the same playbooks run unchanged from a laptop, from GitHub Actions, +from GitLab CI and from Gitea Actions. + +Whichever backend you use, the matching **public** key must be in the deploy +user's `~/.ssh/authorized_keys` on every host, or the key is useless. + +### Storing it in CI + +Store the key **base64-encoded**, as a single line: + +```bash +base64 -w0 < ~/.ssh/deploy_key # Linux +base64 < ~/.ssh/deploy_key | tr -d '\n' # macOS +``` + +```powershell +# Windows PowerShell +[Convert]::ToBase64String([IO.File]::ReadAllBytes("$HOME\.ssh\deploy_key")) +``` + +On Windows use exactly that. `certutil -encode` wraps its output in a header and +line breaks, and `Get-Content` re-encodes the bytes — both produce something that +looks like base64 and decodes to an unusable key. Do not re-wrap the output +either; it is one long line by design. + +Then set it as `SSH_PRIVATE_KEY_B64`: + +| Platform | Where | Notes | +| --- | --- | --- | +| GitHub | Settings → Environments → `production` → Secrets | Scoped to the environment, so only jobs naming it can read it | +| GitLab | Settings → CI/CD → Variables | Mark **Masked** and **Protected** | +| Gitea | Settings → Actions → Secrets | No environments and no masking controls: any run in the repository can read it. See [ci-gitea.md](ci-gitea.md) | + +Why base64 rather than the raw key? GitLab can only *mask* a value that is a +single line with no whitespace, and an OpenSSH private key is neither. GitHub +redacts secrets from logs regardless, but a single-line value survives being +passed between steps without surprises. One encoding satisfies both. + +A raw `SSH_PRIVATE_KEY` still works — as a GitLab File-type variable, as a +GitHub multi-line secret, or as a path on disk. The loader accepts all three. It +just cannot be masked on GitLab, so the key can appear in a job log if something +echoes it. + +### Storing it in a vault + +`vault` reads a HashiCorp Vault or OpenBao KV v2 secret. The expected layout, +overridable with `VAULT_KV_MOUNT` / `VAULT_SECRET_PATH` / `VAULT_SECRET_FIELD` +(or the `BAO_*` spellings): + +``` +kv/infra/ansible/deploy-key field: private_key +``` + +A read-only policy is enough: + +```hcl +path "kv/data/infra/ansible/deploy-key" { + capabilities = ["read"] +} +``` + +**Keep the vault off the deploy path.** If the vault runs on a host this +repository deploys to, and it seals on reboot, then a deploy that fetches its key +from the vault cannot run — including the deploy that would fix that host. Host +it elsewhere, or auto-unseal it. A CI secret is the dependable route. + +## 2. Per-stack `.env` files + +Stack secrets live on the hosts, not here. + +The role excludes `.env` from every sync (`stack_exclude`), so editing one in +this repository has no effect, and a host's `.env` is never overwritten by a +deploy. To change one, edit it on the host and redeploy. + +Each example stack that needs credentials ships a `.env.example` listing the +variable names with no values, and reads them in its compose file with Compose's +required-variable syntax: + +```yaml + environment: + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set it in .env on the host} +``` + +That is worth copying. Without the `:?`, a missing variable expands to the empty +string and Postgres starts with **no password at all** — a failure that looks +like success until someone finds it. With it, Compose refuses to start and names +the variable. + +`webapp.yml` goes one step further and asserts the file exists on the host before +doing anything, so the error arrives before the payload is synced rather than +halfway through starting the stack: + +```yaml + pre_tasks: + - name: Look for the webapp .env on the host + ansible.builtin.stat: + path: "{{ webapp_dir }}/.env" + register: webapp_env + changed_when: false + + - name: Verify the webapp .env is present on the host + ansible.builtin.assert: + that: + - webapp_env.stat.exists + fail_msg: >- + {{ webapp_dir }}/.env is missing on {{ inventory_hostname }}. + quiet: true +``` + +Do this for any stack whose compose file has no safe defaults. + +## 3. Secrets committed to the repository + +There is no third category. This section exists because it is the failure mode +this layout is designed to prevent, and it is worth being blunt about why. + +A credential written directly into a compose file: + +- **cannot be excluded from the sync.** `stack_exclude` keeps `.env` off the + wire; it can do nothing about a value inside `docker-compose.yml`. +- **is in every clone, forever.** Git history is the point of git. Deleting the + file in a later commit changes nothing about the credential. +- **cannot be rotated without a deploy**, so it usually is not rotated at all. +- **escapes its blast radius.** A personal access token carries its creator's + access to everything they can reach, not just the project it was added to. A + cloud application secret outlives the VM it was used on. + +If you find one, the fix is to **rotate the credential**, not to delete the file. +Removing it stops new clones carrying it and changes nothing else. Then move it +to a `.env` on the host and read it with `${VAR:?}`. + +### The guard + +The `lint` workflow — [`.github/workflows/lint.yml`](../.github/workflows/lint.yml), +[`.gitea/workflows/lint.yml`](../.gitea/workflows/lint.yml) and the `lint` job in +`.gitlab-ci.yml` — runs [gitleaks](https://github.com/gitleaks/gitleaks) on every +push, over the working tree **and** the full history: a secret that was committed +and then removed is still leaked, and the scan should say so. `.gitleaks.toml` +allowlists only `*.env.example` files and this document. + +On GitHub it runs the upstream Docker image rather than the marketplace action on +purpose: the action requires a licence key for organisation-owned repositories +and silently does nothing without one, which is the worst possible behaviour for +a secret scanner. The Gitea port runs the pinned release binary instead, because +a job there is itself a container with no Docker socket. + +Do not add an allowlist entry to make a finding go away. Rotate the credential. diff --git a/docs/triggering-deploys.md b/docs/triggering-deploys.md new file mode 100644 index 0000000..8ff6b2a --- /dev/null +++ b/docs/triggering-deploys.md @@ -0,0 +1,195 @@ +# Triggering a deploy from another repository + +The case: an application repository builds a new image, pushes it, and then asks +*this* repository to redeploy the stack that runs it. + +``` +application repo config repo +────────────────────── ────────────────────────── +build → push :latest ─── trigger ───► deploy + stack=webapp └─ ansible-playbook + pull=always playbooks/webapp.yml + -e pull=always +``` + +`pull=always` is not optional here. The image tag has not changed — only its +contents have — and Compose is perfectly happy with the image it already has. A +trigger that forgets it produces a deploy that does nothing and reports success. + +## GitHub Actions + +`deploy.yml` accepts a `repository_dispatch` event of type `deploy`. + +### Caller + +```yaml + - name: Redeploy the stack + env: + GH_TOKEN: ${{ secrets.CONFIG_REPO_TOKEN }} + run: | + gh api repos/OWNER/config/dispatches \ + --field event_type=deploy \ + --field 'client_payload[stack]=webapp' \ + --field 'client_payload[pull]=always' +``` + +`CONFIG_REPO_TOKEN` is a fine-grained personal access token, or a GitHub App +installation token, with **Contents: read and write** on the config repository. +The default `GITHUB_TOKEN` cannot dispatch to another repository. + +### What happens downstream + +`deploy.yml` validates the payload before doing anything: + +1. rejects `stack` unless it matches `[a-z0-9_-]+`, and `pull` unless it is one + of `policy`, `always`, `missing`, `never`, +2. fails if `ansible/playbooks/.yml` does not exist, +3. materialises the SSH key with `scripts/load-ssh-key.sh`, +4. runs the playbook. + +The values arrive from another repository and end up on a command line, so they +are treated as hostile. See "Why untrusted input is validated" in +[ci-github-actions.md](ci-github-actions.md). + +### The caller cannot wait + +A `repository_dispatch` returns as soon as GitHub accepts the event. The response +contains no run id, so the caller cannot poll for a result either. If the build +pipeline must go red when the deploy fails, you have two options: + +- **Put the deploy in the same repository** as the build and use `workflow_call`, + which does propagate failure. +- **Poll.** After dispatching, list workflow runs for `deploy.yml` created after + your dispatch timestamp and wait on the newest. This is fiddly and racy; only + do it if you genuinely need the coupling. + +Most of the time you do not. A deploy that fails is visible in the config +repository's Actions tab and can notify on failure there. + +## GitLab CI + +`deploy:triggered` exists only in pipelines that were triggered from outside, so +it can never fire on a push here. + +### Multi-project trigger (preferred) + +In the upstream project: + +```yaml +deploy-config: + stage: deploy + trigger: + project: your-group/config + branch: main # must be a protected branch — see below + strategy: depend # upstream waits and mirrors the result + variables: + STACK: webapp + PULL: always +``` + +`strategy: depend` is what makes the build pipeline report the *deployment's* +result. Drop it and the trigger job goes green the moment the request is +accepted, whether or not the deploy worked. + +This authenticates with `CI_JOB_TOKEN`, so **the config project must allowlist +the upstream project** under Settings → CI/CD → Job token permissions. Recent +GitLab versions deny cross-project job-token access by default, and the failure +is a **404** rather than a permissions error — which is the most common cause of +"the trigger does nothing". + +### Trigger token + +For callers that are not GitLab CI — a webhook, a cron box, a person with +`curl`. Create the token under Settings → CI/CD → Pipeline trigger tokens. + +```bash +curl --request POST \ + --form token="$DEPLOY_TRIGGER_TOKEN" \ + --form ref=main \ + --form "variables[STACK]=webapp" \ + --form "variables[PULL]=always" \ + "https://gitlab.example.com/api/v4/projects/your-group%2Fconfig/trigger/pipeline" +``` + +The project path must be URL-encoded (`%2F` for the slash), or use the numeric +project ID. The response is the created pipeline as JSON; `.web_url` takes you +straight to it. + +### One-time setup + +1. **Allowlist the caller** under Job token permissions, or the trigger 404s. +2. **Protect the target branch**, and mark the deploy variables Protected. + Protected variables only exist on protected refs; trigger a feature branch and + the key comes back empty. +3. **Give the upstream project Developer access**, or the job token is refused + regardless of the allowlist. + +## Gitea Actions + +Gitea has no `repository_dispatch`. The way in is the **workflow dispatch API**, +which fills in the same form a person would use under Actions → deploy → Run +workflow. It needs Gitea 1.24 or newer. + +### Caller + +```bash +curl -X POST \ + -H "Authorization: token $CONFIG_REPO_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "ref": "main", + "inputs": {"stack": "webapp", "mode": "deploy", "pull": "always"} + }' \ + https://gitea.example.com/api/v1/repos/OWNER/config/actions/workflows/deploy.yml/dispatches +``` + +Three things are easy to get wrong here: + +- **`ref` is required**, and it is the ref the playbooks are read from as well as + the one the run is attributed to. Point it at your default branch. +- **`mode` falls back to `check`.** Omit it and the trigger runs + `--check --diff`, reports what it would have done, and goes green having + deployed nothing. This is the Gitea equivalent of forgetting `pull=always`, and + it fails just as quietly. (The workflow applies that fallback itself: an + input's declared `default:` belongs to the dispatch form, not to the API.) +- **The job's own `GITEA_TOKEN` cannot do this.** It is scoped to the repository + the job runs in. `CONFIG_REPO_TOKEN` is an access token belonging to a user + with write access to the config repository — Settings → Applications → Access + tokens, with the `write:repository` scope — stored as a secret in the calling + repository. + +The workflow name in the path is the file name, `deploy.yml`. + +### The caller can wait, if it wants to + +Add `?return_run_details=true` and the response carries the run's id and URL, so +a caller can poll `/api/v1/repos/OWNER/config/actions/runs/` until it +finishes and fail its own pipeline on the result. That is more than GitHub's +`repository_dispatch` offers, and more than most callers need: a deploy that +fails is visible in the config repository's Actions tab and can notify from +there. + +### What happens downstream + +Exactly what happens on GitHub — the same validation is in +`.gitea/workflows/deploy.yml`, and it matters more here. A `type: choice` input +constrains the *form* and not the API, so a dispatch call may put any string in +`stack`. + +## Before you enable this + +**A trigger is a deploy.** Anyone holding the token, and any repository on the +allowlist, can deploy any stack in this repository. The validation guard stops +them running *arbitrary commands*; it does not stop them deploying a stack they +had no business touching. Scope the allowlist deliberately, and rotate the token +when someone leaves. + +**Confirm the playbook exists first.** The downstream job fails the pipeline by +design when it does not, which is correct but makes for a confusing first +experience if you enable the trigger before converting the stack. + +## Checking it worked + +The Ansible recap at the end of the job log is the real answer. `changed=0` means +the images had not moved — which, for a trigger that fired after a successful +build, almost always means `pull=always` was not passed. diff --git a/skills/adding-a-stack/SKILL.md b/skills/adding-a-stack/SKILL.md new file mode 100644 index 0000000..9adb585 --- /dev/null +++ b/skills/adding-a-stack/SKILL.md @@ -0,0 +1,85 @@ +--- +name: adding-a-stack +description: Use when adding a new service to this repository, moving a service to a different host, or bringing a hand-deployed service under Ansible and CI. +--- + +# Adding a stack + +A **stack** is one Docker Compose deployment on one host. Adding one touches four +files and one button. `AGENTS.md` holds the conventions and the full +`compose_stack` role contract — read it first; this file is the order of work. + +## Steps + +1. **Payload.** Put the compose file and its configuration in + `server///`. Payloads are copied verbatim, so do not template or + restructure them. Never commit a `.env`, a key, or a secret written inline in + a compose file — read `docs/secrets.md` before you decide something is fine to + commit. +2. **Playbook.** Create `ansible/playbooks/.yml`. Copy `static-site.yml`; + most stacks are a name and a network. Tag the play with the stack name. +3. **Register it.** Add an `import_playbook` entry to `ansible/playbooks/site.yml` + in dependency order — the reverse proxy owns `proxy-net` and the vhost + configuration, so it stays first. +4. **Give it a button.** All three CI configurations need the stack by name, and + none of them can generate the list: + - `.github/workflows/deploy.yml` — add it to the `stack` input's `options:`. + - `.gitea/workflows/deploy.yml` — the same `options:` list again. Gitea reads + `.gitea/workflows/` *instead of* `.github/workflows/`, so the two files are + never both in play and never both wrong at once — which is exactly why one + of them gets forgotten. + - `.gitlab-ci.yml` — add a `check:` and `deploy:` pair extending + `.check` and `.deploy`; copy the pair above them, three lines each. Pin the + stack with **`STACK_FIXED`**, never `STACK` — a pipeline variable outranks a + job's YAML `variables:`, so a button named with `STACK` is hijacked by any + pipeline triggered with a different one. + + Delete whichever platforms you are not using rather than letting them rot. +5. **Verify statically.** `make check` from the repository root. It must pass; + this is the definition of done for the playbook. +6. **Deploy.** Run the check mode, read the diff, then deploy. Never run a + playbook against a host unless the person you are working with asked for it in + that turn. + +## Does it fit the shared role? + +| If | Do | Example | +| --- | --- | --- | +| It is not a Compose stack — one file into another stack's directory, or a system service | Write plain tasks. Do not force the role | `banner.yml` | +| Something must happen between the payload landing and the stack starting (backup, migration) | Use the two entry points, `tasks_from: sync` then `tasks_from: up`, and gate the middle on `pull \| default('policy') == 'always'` | `webapp.yml` | +| The payload is bind-mounted config, which Compose never recreates a container for | Add a `post_tasks` reload keyed on `compose_stack_synced` | `reverse-proxy.yml`, `metrics.yml` | +| The compose file is not named `docker-compose.yml`, or there are several | `stack_files` | `webapp.yml` | +| The same payload serves several hosts, or several instances | `stack_src`, and `stack_dest` for the second instance | `metrics.yml`, `webapp-staging.yml` | +| The stack needs a `.env` that is deliberately never shipped | `stat` + `assert` that it exists on the host first | `webapp.yml` | +| It needs an external Docker network | `stack_networks` | any of them | +| A bind mount would otherwise be created root-owned by Docker | `stack_dirs`; omit `mode` so an existing directory is left alone | `reverse-proxy.yml` | +| A bind mount must be owned by a uid from inside the container | `stack_dirs` with `owner`/`group` and `recurse`, and no `mode` | `metrics.yml` | +| The image is built from a Dockerfile in the payload | `stack_build`, set to `always` only on an update run | `webapp.yml` | +| Hosts differ in some small way | A dict keyed by `inventory_hostname`, looked up in the role vars | `metrics.yml` | + +Extend the role's variable contract rather than adding a second role or loose +tasks. If a stack needs something the contract cannot express, say so instead of +working around it — that is information about the contract. + +## Common mistakes + +- **Skipping step 3 or 4.** The stack then works from a workstation and is + invisible in CI, which is exactly how services end up hand-deployed and + undocumented. This is the most common failure and the least visible. +- **Pre-creating a Postgres data directory** in `stack_dirs`. Postgres refuses to + start unless it is `0700` or `0750`, so leave it to the image. +- **Setting `mode` on a bind mount that already has data.** It rewrites the + permissions of everything the container has written there. Set `owner`/`group` + with `recurse` instead, and no `mode`. +- **A `pre_tasks` chown instead of `stack_dirs`.** `pre_tasks` run before the + stack directory exists, and the escalated `file` task stamps its owner onto + every parent it creates — leaving the payload sync unable to write into the + stack's own directory. +- **Expecting `.env` to reach the host.** It is excluded from every sync by + design. Secrets live on the host. +- **A reload task with no `when: compose_stack_synced`.** Every run then reports + a change, the playbook is never idempotent, and `changed=0` stops meaning + anything. +- **Putting the `stack_*` vars inside `roles:` for a two-phase playbook.** They + are scoped to that invocation and the second `include_role` will not see them. + Declare them at play level.