From 35b1378080d4d1bb2a3386724836ec5b589cb14c Mon Sep 17 00:00:00 2001 From: Lucas Winther Date: Sat, 3 Oct 2026 17:49:12 +0200 Subject: [PATCH] Add stackctl for running stacks by hand on a host One shared script installed into /srv/stacks on every host, with per-host groups in server//stack-groups. Tested in check.sh. Also two new common mistakes in the adding-a-stack skill. Co-Authored-By: Claude Opus 5.5 --- .gitea/workflows/deploy.yml | 1 + .github/workflows/deploy.yml | 1 + .gitlab-ci.yml | 10 ++ AGENTS.md | 3 +- README.md | 29 +++++ ansible/playbooks/site.yml | 2 + ansible/playbooks/stackctl.yml | 35 ++++++ scripts/check.sh | 3 + scripts/test-stackctl.sh | 115 ++++++++++++++++++ server/README.md | 5 +- server/app/stack-groups | 12 ++ server/edge/stack-groups | 18 +++ server/shared/components/stackctl | 194 ++++++++++++++++++++++++++++++ skills/adding-a-stack/SKILL.md | 18 ++- 14 files changed, 442 insertions(+), 4 deletions(-) create mode 100644 ansible/playbooks/stackctl.yml create mode 100755 scripts/test-stackctl.sh create mode 100644 server/app/stack-groups create mode 100644 server/edge/stack-groups create mode 100755 server/shared/components/stackctl diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml index 404fea6..a0e10b2 100644 --- a/.gitea/workflows/deploy.yml +++ b/.gitea/workflows/deploy.yml @@ -33,6 +33,7 @@ on: - site - reverse-proxy - banner + - stackctl - metrics - openbao - static-site diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index fab1007..52b16d0 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -25,6 +25,7 @@ on: - site - reverse-proxy - banner + - stackctl - metrics - openbao - static-site diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index b22fb5a..6f2e0ad 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -209,6 +209,16 @@ deploy:banner: variables: STACK_FIXED: banner +check:stackctl: + extends: .check + variables: + STACK_FIXED: stackctl + +deploy:stackctl: + extends: .deploy + variables: + STACK_FIXED: stackctl + check:metrics: extends: .check variables: diff --git a/AGENTS.md b/AGENTS.md index 3b4c75e..057bf79 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -110,7 +110,8 @@ Worked examples for each: `static-site.yml` (the minimum), `reverse-proxy.yml` `openbao.yml` (owner-only escalation; a notice rather than a restart on `compose_stack_synced`), `webapp-staging.yml` (`stack_src` + `stack_dest` + `stack_env`), `banner.yml` (not a stack at all), -`bootstrap.yml` (preparing a host; see `docs/bootstrap.md`). +`stackctl.yml` (not a stack either: one shared file and one per-host file into +`stack_root`), `bootstrap.yml` (preparing a host; see `docs/bootstrap.md`). #### `stack_dirs` diff --git a/README.md b/README.md index 736cb5f..46e7e2c 100644 --- a/README.md +++ b/README.md @@ -65,9 +65,12 @@ ansible/ └── playbooks/ ├── site.yml # every stack, in dependency order ├── bootstrap.yml # prepares a host; run once, by an admin + ├── stackctl.yml # installs stackctl and each host's stack-groups └── .yml # one per stack, runnable on its own server/// # the payload: compose files and config, copied verbatim server/shared/ # payloads used by more than one host +server//stack-groups # that host's groups for stackctl +scripts/test-stackctl.sh # tests for stackctl, run by check.sh scripts/load-ssh-key.sh # produces the deploy key path (file / CI / vault) scripts/new-deploy-key.sh # generates the deploy key pair scripts/check.sh # the full static gate — what CI runs @@ -220,6 +223,32 @@ Seven stacks, each chosen to demonstrate one thing. Read them in this order. `server/` is data — see [`server/README.md`](server/README.md). +## On the host: stackctl + +`/srv/stacks/stackctl` brings stacks up and down by hand, by name or by group, +without visiting each directory — for maintenance on a host, not for deploys, +which still go through Ansible. + +```bash +/srv/stacks/stackctl up apps # a group from stack-groups +/srv/stacks/stackctl down monitoring webapp # groups and stacks mix +/srv/stacks/stackctl -y restart all # -y skips the openbao question +/srv/stacks/stackctl ps # running/total containers per stack +/srv/stacks/stackctl logs webapp # follows one stack +/srv/stacks/stackctl list # the groups, and every stack found +``` + +Each stack is plain `docker compose` in its own directory, so it uses the same +`.env` and project name a deploy does. A stack whose playbook passes other +compose files through `stack_files`, or other settings through `stack_env`, is +not started identically; the script's header says exactly what carries over. + +Groups are per host, in `server//stack-groups`, along with the start order +(`down` reverses it) and the stacks that need confirming before `down` or +`restart`. The script is one file for every host, in +`server/shared/components/stackctl`. `playbooks/stackctl.yml`, which `site.yml` +runs, installs both; edit them here, not on the host. + ## Secrets Three separate things, handled three different ways. Full detail in diff --git a/ansible/playbooks/site.yml b/ansible/playbooks/site.yml index 0dbcd35..83b8ae4 100644 --- a/ansible/playbooks/site.yml +++ b/ansible/playbooks/site.yml @@ -15,6 +15,8 @@ ansible.builtin.import_playbook: reverse-proxy.yml - name: Default vhost banner ansible.builtin.import_playbook: banner.yml +- name: Stackctl + ansible.builtin.import_playbook: stackctl.yml - name: Metrics ansible.builtin.import_playbook: metrics.yml - name: OpenBao diff --git a/ansible/playbooks/stackctl.yml b/ansible/playbooks/stackctl.yml new file mode 100644 index 0000000..5ba54e0 --- /dev/null +++ b/ansible/playbooks/stackctl.yml @@ -0,0 +1,35 @@ +--- +# stackctl, and each host's groups for it, into the stack root. Not a Compose +# stack: two files and no `docker compose up`, so plain tasks rather than the +# role, as with banner.yml. +# +# The script is one file shared by every host, under server/shared/components/. +# The groups are per host, at server//stack-groups, because each host runs +# different stacks; a host without one still gets the script, and "all" and +# stack names work there without any groups. See "On the host: stackctl" in the +# README. +- name: Install stackctl + hosts: + - edge + - app + gather_facts: false + tags: [stackctl] + + vars: + stackctl_src: "{{ repo_root }}/server/shared/components/stackctl" + stackctl_groups_src: "{{ repo_root }}/server/{{ inventory_hostname }}/stack-groups" + + tasks: + - name: Install the stackctl script + ansible.builtin.copy: + src: "{{ stackctl_src }}" + dest: "{{ stack_root }}/stackctl" + mode: "0755" + + # `is file` is evaluated on the control node, where the repository is. + - name: Install this host's stack groups + ansible.builtin.copy: + src: "{{ stackctl_groups_src }}" + dest: "{{ stack_root }}/stack-groups" + mode: "0644" + when: stackctl_groups_src is file diff --git a/scripts/check.sh b/scripts/check.sh index 8ab8063..fa5bc71 100755 --- a/scripts/check.sh +++ b/scripts/check.sh @@ -58,6 +58,9 @@ step() { printf '\n== %s ==\n' "$*"; } # From the repository root, where .yamllint and its ignore list live. ansible-lint # runs yamllint too, but only over files it recognises as Ansible content; this # covers the workflows, the CI configuration and everything else. +step "stackctl tests" +scripts/test-stackctl.sh + step "yamllint" "${yamllint_cmd[@]}" . diff --git a/scripts/test-stackctl.sh b/scripts/test-stackctl.sh new file mode 100755 index 0000000..6d93b7e --- /dev/null +++ b/scripts/test-stackctl.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# +# Tests for server/shared/components/stackctl, against a throwaway stack root and a stub +# `docker` that records what it was asked to do instead of doing it. +# +# scripts/test-stackctl.sh +# +# Run by scripts/check.sh. Needs nothing but bash. +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +work="$(mktemp -d)" +trap 'rm -rf "$work"' EXIT + +root="$work/stacks" +bin="$work/bin" +log="$work/calls" +mkdir -p "$root" "$bin" +cp "$repo_root/server/shared/components/stackctl" "$root/stackctl" + +for s in reverse-proxy openbao webapp grafana prometheus static-site wiki; do + mkdir -p "$root/$s" + touch "$root/$s/docker-compose.yml" +done +mkdir -p "$root/not-a-stack" + +cat > "$root/stack-groups" <<'EOF' +# a comment +order: reverse-proxy openbao +confirm: openbao + +system: openbao webapp +monitoring: grafana prometheus +monitoring: ghost +sites: static-site +EOF + +# Records " " and fails for any stack named in $FAIL. +cat > "$bin/docker" <<'EOF' +#!/usr/bin/env bash +stack="$(basename "$PWD")" +echo "$stack $*" >> "$STACKCTL_TEST_LOG" +case " ${FAIL:-} " in *" $stack "*) exit 1 ;; esac +if [ "$2" = ps ]; then printf 'running\nexited\n'; fi +EOF +chmod +x "$bin/docker" + +export PATH="$bin:$PATH" STACKCTL_TEST_LOG="$log" + +failures=0 +run() { : > "$log"; set +e; out="$("$root/stackctl" "$@" 2>&1 // a stack that runs on exactly one host server/shared// a payload used by more than one host or instance -server/shared/components/ single files that belong to another stack's directory +server/shared/components/ single files that belong to another stack's directory, + or to the stack root itself +server//stack-groups that host's stackctl groups; a file, not a stack ``` The directory names under `server/` must match the inventory host names, because @@ -32,6 +34,7 @@ is the reusable part. | `app/openbao/` | `openbao.yml` | The vault, on `127.0.0.1:8200` only. `config/openbao.hcl` is bind-mounted; a change needs a manual restart and unseal. | | `shared/metrics/` | `metrics.yml` | Prometheus and Grafana, identical on both hosts, so it lives here rather than being duplicated per host. Its `.env` is placed on each host by hand. | | `shared/components/banner.html` | `banner.yml` | A single file that belongs to the reverse proxy's directory on every host. | +| `shared/components/stackctl`, `/stack-groups` | `stackctl.yml` | The by-hand stack runner, installed into `/srv/stacks` on every host, with each host's groups beside it. Not stacks. | ## Conventions worth copying diff --git a/server/app/stack-groups b/server/app/stack-groups new file mode 100644 index 0000000..6b016e3 --- /dev/null +++ b/server/app/stack-groups @@ -0,0 +1,12 @@ +# Groups for stackctl on app. Format and reserved names: see the same file in +# server/edge/. +# +# OpenBao starts first because the stacks that use stack_secrets read it during a +# deploy, and it needs confirming before down or restart: restarting it seals it. + +order: openbao +confirm: openbao + +system: openbao +apps: static-site webapp webapp-staging +monitoring: metrics diff --git a/server/edge/stack-groups b/server/edge/stack-groups new file mode 100644 index 0000000..00cc683 --- /dev/null +++ b/server/edge/stack-groups @@ -0,0 +1,18 @@ +# Groups for stackctl on edge, installed next to it in /srv/stacks. One group +# per line: +# +# name: stack stack ... +# +# A name on several lines collects all of them. A stack may be in any number of +# groups, or none; "all" always means every stack with a compose file. Each host +# has its own file, because each host has its own stacks. +# +# Two names are reserved: +# order: started first, in this order; everything else follows +# alphabetically, and down runs the whole list in reverse. +# confirm: down and restart ask before touching these, unless given -y. + +order: reverse-proxy + +proxy: reverse-proxy +monitoring: metrics diff --git a/server/shared/components/stackctl b/server/shared/components/stackctl new file mode 100755 index 0000000..8c70205 --- /dev/null +++ b/server/shared/components/stackctl @@ -0,0 +1,194 @@ +#!/usr/bin/env bash +# +# Start, stop and inspect the stacks under this directory by hand, one at a time +# or by group. Installed to /srv/stacks on every host by +# ansible/playbooks/stackctl.yml; each host's groups live next to it in +# stack-groups. +# +# stackctl up apps # every stack in the apps group +# stackctl -y down all # everything, without asking about openbao +# stackctl pull static-site && stackctl up static-site +# stackctl ps # running containers per stack +# stackctl logs webapp +# +# Each stack is plain `docker compose` in its own directory: the default compose +# file, the .env beside it, the directory name as the project name, and PWD set +# to the directory. That is what an Ansible deploy does for any stack that +# leaves stack_files and stack_env at their defaults, or sets only +# COMPOSE_PROJECT_NAME to the directory name. A stack that relies on anything +# else from those two is not started the same way here. +# +# Ansible remains how stacks are deployed; this is for working on the host. +set -euo pipefail + +root="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)" +groups_file="$root/stack-groups" + +usage() { + cat >&2 < [target...] + +commands: + up docker compose up -d + down docker compose down + restart docker compose restart + pull docker compose pull + ps running containers per stack (default target: all) + logs follow the logs of exactly one stack + list the groups and every stack found + +target: a stack, a group from $groups_file, or "all". +-y skips the confirmation before down or restart of a stack under "confirm:". +EOF + exit 2 +} + +die() { echo "stackctl: $*" >&2; exit "${2:-2}"; } +warn() { echo "stackctl: $*" >&2; } + +yes=false +while getopts ':yh' opt; do + case "$opt" in + y) yes=true ;; + *) usage ;; + esac +done +shift $((OPTIND - 1)) +[ $# -ge 1 ] || usage +command="$1" +shift + +case "$command" in + up | down | restart | pull | logs) ;; + ps | list) [ $# -ge 1 ] || set -- all ;; + *) warn "unknown command: $command"; usage ;; +esac +[ $# -ge 1 ] || die "$command needs a target: a stack, a group, or all" + +# Every directory holding a compose file is a stack. +stacks=() +for dir in "$root"/*/; do + [ -f "$dir/docker-compose.yml" ] && stacks+=("$(basename "$dir")") +done + +is_stack() { + local s + for s in "${stacks[@]}"; do [ "$s" = "$1" ] && return 0; done + return 1 +} + +# stack-groups: "name: stack stack ...", one group per line. A name given on +# several lines collects all of them. "order" and "confirm" are reserved. +declare -A group +order=() +confirm=() +if [ -f "$groups_file" ]; then + while IFS= read -r line || [ -n "$line" ]; do + line="${line%%#*}" + [[ "$line" == *:* ]] || continue + name="$(echo "${line%%:*}" | tr -d '[:space:]')" + read -ra members <<<"${line#*:}" + case "$name" in + order) order+=("${members[@]}") ;; + confirm) confirm+=("${members[@]}") ;; + *) group[$name]="${group[$name]:-} ${members[*]}" ;; + esac + done <"$groups_file" +fi + +# Resolve targets to stacks, without duplicates. +declare -A chosen +for target in "$@"; do + case "$target" in + all) + for s in "${stacks[@]}"; do chosen[$s]=1; done + ;; + order | confirm) + die "\"$target\" is reserved in stack-groups, not a target" + ;; + *) + if [ -n "${group[$target]+x}" ]; then + for s in ${group[$target]}; do + if is_stack "$s"; then + chosen[$s]=1 + else + warn "group $target names $s, which has no compose file here; skipping" + fi + done + elif is_stack "$target"; then + chosen[$target]=1 + else + die "no stack or group called $target (stackctl list shows both)" + fi + ;; + esac +done + +# Stacks under "order:" first, in that order, then the rest alphabetically. +rank() { + local i + for i in "${!order[@]}"; do + [ "${order[$i]}" = "$1" ] && { printf '%04d' "$i"; return; } + done + printf '9999' +} +mapfile -t selected < <( + for s in "${!chosen[@]}"; do printf '%s %s\n' "$(rank "$s")" "$s"; done | + sort -k1,1n -k2,2 | cut -d' ' -f2 +) +if [ "$command" = down ]; then + mapfile -t selected < <(printf '%s\n' "${selected[@]}" | tac) +fi +[ "${#selected[@]}" -gt 0 ] || die "nothing to do" 1 + +case "$command" in + list) + for name in $(printf '%s\n' "${!group[@]}" | sort); do + printf '%-10s%s\n' "$name:" "${group[$name]}" + done + printf '%-10s%s\n' "stacks:" "${selected[*]}" + exit 0 + ;; + logs) + [ "${#selected[@]}" -eq 1 ] || die "logs follows exactly one stack" + cd "$root/${selected[0]}" + exec docker compose logs -f --tail=100 + ;; + ps) + for s in "${selected[@]}"; do + states="$(cd "$root/$s" && docker compose ps -a --format '{{.State}}')" + total="$(grep -c . <<<"$states" || true)" + running="$(grep -c '^running$' <<<"$states" || true)" + printf '%-24s %s/%s\n' "$s" "$running" "$total" + done + exit 0 + ;; +esac + +# Ask before stopping anything under "confirm:" — restarting OpenBao seals it. +if [ "$command" = down ] || [ "$command" = restart ]; then + for s in "${selected[@]}"; do + for c in "${confirm[@]}"; do + [ "$s" = "$c" ] || continue + $yes && continue + [ -t 0 ] || die "$command $s needs confirmation; rerun with -y" 1 + read -rp "$command $s? [y/N] " answer + [[ "$answer" == [yY]* ]] || die "not confirmed; nothing done" 1 + done + done +fi + +case "$command" in + up) args=(up -d) ;; + *) args=("$command") ;; +esac + +failed=() +for s in "${selected[@]}"; do + echo "== $command $s" + (cd "$root/$s" && docker compose "${args[@]}") || failed+=("$s") +done + +if [ "${#failed[@]}" -gt 0 ]; then + die "failed: ${failed[*]}" 1 +fi diff --git a/skills/adding-a-stack/SKILL.md b/skills/adding-a-stack/SKILL.md index 7d03250..aae18e9 100644 --- a/skills/adding-a-stack/SKILL.md +++ b/skills/adding-a-stack/SKILL.md @@ -48,9 +48,12 @@ files and one button. `AGENTS.md` holds the conventions and the full Delete whichever platforms you are not using rather than letting them rot. Leave a stack out only if it must never deploy from CI — see the table below. -5. **Verify statically.** `make check` from the repository root. It must pass; +5. **Group it, if it belongs in one.** Add it to a line in + `server//stack-groups` so `stackctl up ` on that host includes + it. Optional: `stackctl all` finds every stack regardless. +6. **Verify statically.** `make check` from the repository root. It must pass; this is the definition of done for the playbook. -6. **Deploy.** Run the check mode, read the diff, then deploy. Never run a +7. **Deploy.** Run the check mode, read the diff, then deploy. Never run a playbook against a host unless the person you are working with asked for it in that turn. @@ -100,6 +103,17 @@ working around it — that is information about the contract. - **A reload task with no `when: compose_stack_synced`.** Every run then reports a change, the playbook is never idempotent, and `changed=0` stops meaning anything. +- **Committing an upstream example compose file without reading it.** They are + written for a quick start, not for a host. Look for three things: a password + given as a default (`${POSTGRES_PASSWORD:-changeme}`), which is a committed + secret — make it `${VAR:?}` and use `stack_secrets`; database or cache ports + published on every interface, which nothing outside the stack needs; and + `KEY:value` entries in a list-form `environment:`, which set a variable + literally named `KEY:value` and nothing else — the list form needs `=`. +- **A stateful service with no bind mount.** An app that falls back to SQLite + inside its container loses everything the next time Compose recreates it. Find + where it writes and mount that directory; if the image runs as a fixed uid, + give the directory to it with `stack_dirs`. - **Putting the `stack_*` vars inside `roles:` for a two-phase playbook.** They are scoped to that invocation and the second `include_role` will not see them. Declare them at play level.