Add stackctl for running stacks by hand on a host
lint / yamllint + ansible-lint + syntax (push) Successful in 56s
lint / shellcheck (push) Successful in 7s
lint / secret scan (push) Successful in 2s

One shared script installed into /srv/stacks on every host, with per-host
groups in server/<host>/stack-groups. Tested in check.sh. Also two new
common mistakes in the adding-a-stack skill.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Lucas Winther
2026-10-03 17:49:12 +02:00
co-authored by Claude Opus 5.5
parent d28494d21f
commit 35b1378080
14 changed files with 442 additions and 4 deletions
+1
View File
@@ -33,6 +33,7 @@ on:
- site
- reverse-proxy
- banner
- stackctl
- metrics
- openbao
- static-site
+1
View File
@@ -25,6 +25,7 @@ on:
- site
- reverse-proxy
- banner
- stackctl
- metrics
- openbao
- static-site
+10
View File
@@ -209,6 +209,16 @@ deploy:banner:
variables:
STACK_FIXED: banner
check:stackctl:
extends: .check
variables:
STACK_FIXED: stackctl
deploy:stackctl:
extends: .deploy
variables:
STACK_FIXED: stackctl
check:metrics:
extends: .check
variables:
+2 -1
View File
@@ -110,7 +110,8 @@ Worked examples for each: `static-site.yml` (the minimum), `reverse-proxy.yml`
`openbao.yml` (owner-only escalation; a notice rather than a restart on
`compose_stack_synced`), `webapp-staging.yml`
(`stack_src` + `stack_dest` + `stack_env`), `banner.yml` (not a stack at all),
`bootstrap.yml` (preparing a host; see `docs/bootstrap.md`).
`stackctl.yml` (not a stack either: one shared file and one per-host file into
`stack_root`), `bootstrap.yml` (preparing a host; see `docs/bootstrap.md`).
#### `stack_dirs`
+29
View File
@@ -65,9 +65,12 @@ ansible/
└── playbooks/
├── site.yml # every stack, in dependency order
├── bootstrap.yml # prepares a host; run once, by an admin
├── stackctl.yml # installs stackctl and each host's stack-groups
└── <stack>.yml # one per stack, runnable on its own
server/<host>/<stack>/ # the payload: compose files and config, copied verbatim
server/shared/ # payloads used by more than one host
server/<host>/stack-groups # that host's groups for stackctl
scripts/test-stackctl.sh # tests for stackctl, run by check.sh
scripts/load-ssh-key.sh # produces the deploy key path (file / CI / vault)
scripts/new-deploy-key.sh # generates the deploy key pair
scripts/check.sh # the full static gate — what CI runs
@@ -220,6 +223,32 @@ Seven stacks, each chosen to demonstrate one thing. Read them in this order.
`server/` is data — see [`server/README.md`](server/README.md).
## On the host: stackctl
`/srv/stacks/stackctl` brings stacks up and down by hand, by name or by group,
without visiting each directory — for maintenance on a host, not for deploys,
which still go through Ansible.
```bash
/srv/stacks/stackctl up apps # a group from stack-groups
/srv/stacks/stackctl down monitoring webapp # groups and stacks mix
/srv/stacks/stackctl -y restart all # -y skips the openbao question
/srv/stacks/stackctl ps # running/total containers per stack
/srv/stacks/stackctl logs webapp # follows one stack
/srv/stacks/stackctl list # the groups, and every stack found
```
Each stack is plain `docker compose` in its own directory, so it uses the same
`.env` and project name a deploy does. A stack whose playbook passes other
compose files through `stack_files`, or other settings through `stack_env`, is
not started identically; the script's header says exactly what carries over.
Groups are per host, in `server/<host>/stack-groups`, along with the start order
(`down` reverses it) and the stacks that need confirming before `down` or
`restart`. The script is one file for every host, in
`server/shared/components/stackctl`. `playbooks/stackctl.yml`, which `site.yml`
runs, installs both; edit them here, not on the host.
## Secrets
Three separate things, handled three different ways. Full detail in
+2
View File
@@ -15,6 +15,8 @@
ansible.builtin.import_playbook: reverse-proxy.yml
- name: Default vhost banner
ansible.builtin.import_playbook: banner.yml
- name: Stackctl
ansible.builtin.import_playbook: stackctl.yml
- name: Metrics
ansible.builtin.import_playbook: metrics.yml
- name: OpenBao
+35
View File
@@ -0,0 +1,35 @@
---
# stackctl, and each host's groups for it, into the stack root. Not a Compose
# stack: two files and no `docker compose up`, so plain tasks rather than the
# role, as with banner.yml.
#
# The script is one file shared by every host, under server/shared/components/.
# The groups are per host, at server/<host>/stack-groups, because each host runs
# different stacks; a host without one still gets the script, and "all" and
# stack names work there without any groups. See "On the host: stackctl" in the
# README.
- name: Install stackctl
hosts:
- edge
- app
gather_facts: false
tags: [stackctl]
vars:
stackctl_src: "{{ repo_root }}/server/shared/components/stackctl"
stackctl_groups_src: "{{ repo_root }}/server/{{ inventory_hostname }}/stack-groups"
tasks:
- name: Install the stackctl script
ansible.builtin.copy:
src: "{{ stackctl_src }}"
dest: "{{ stack_root }}/stackctl"
mode: "0755"
# `is file` is evaluated on the control node, where the repository is.
- name: Install this host's stack groups
ansible.builtin.copy:
src: "{{ stackctl_groups_src }}"
dest: "{{ stack_root }}/stack-groups"
mode: "0644"
when: stackctl_groups_src is file
+3
View File
@@ -58,6 +58,9 @@ step() { printf '\n== %s ==\n' "$*"; }
# From the repository root, where .yamllint and its ignore list live. ansible-lint
# runs yamllint too, but only over files it recognises as Ansible content; this
# covers the workflows, the CI configuration and everything else.
step "stackctl tests"
scripts/test-stackctl.sh
step "yamllint"
"${yamllint_cmd[@]}" .
+115
View File
@@ -0,0 +1,115 @@
#!/usr/bin/env bash
#
# Tests for server/shared/components/stackctl, against a throwaway stack root and a stub
# `docker` that records what it was asked to do instead of doing it.
#
# scripts/test-stackctl.sh
#
# Run by scripts/check.sh. Needs nothing but bash.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
work="$(mktemp -d)"
trap 'rm -rf "$work"' EXIT
root="$work/stacks"
bin="$work/bin"
log="$work/calls"
mkdir -p "$root" "$bin"
cp "$repo_root/server/shared/components/stackctl" "$root/stackctl"
for s in reverse-proxy openbao webapp grafana prometheus static-site wiki; do
mkdir -p "$root/$s"
touch "$root/$s/docker-compose.yml"
done
mkdir -p "$root/not-a-stack"
cat > "$root/stack-groups" <<'EOF'
# a comment
order: reverse-proxy openbao
confirm: openbao
system: openbao webapp
monitoring: grafana prometheus
monitoring: ghost
sites: static-site
EOF
# Records "<stack> <args>" and fails for any stack named in $FAIL.
cat > "$bin/docker" <<'EOF'
#!/usr/bin/env bash
stack="$(basename "$PWD")"
echo "$stack $*" >> "$STACKCTL_TEST_LOG"
case " ${FAIL:-} " in *" $stack "*) exit 1 ;; esac
if [ "$2" = ps ]; then printf 'running\nexited\n'; fi
EOF
chmod +x "$bin/docker"
export PATH="$bin:$PATH" STACKCTL_TEST_LOG="$log"
failures=0
run() { : > "$log"; set +e; out="$("$root/stackctl" "$@" 2>&1 </dev/null)"; status=$?; set -e; }
calls() { cut -d' ' -f1-3 "$log" | paste -sd, -; }
check() {
local name="$1" want="$2" got="$3"
if [ "$want" = "$got" ]; then
printf 'ok %s\n' "$name"
else
printf 'FAIL %s\n want: %s\n got: %s\n' "$name" "$want" "$got"
failures=$((failures + 1))
fi
}
run up all
check "up all runs order first, then alphabetical" \
"reverse-proxy compose up,openbao compose up,grafana compose up,prometheus compose up,static-site compose up,webapp compose up,wiki compose up" \
"$(calls)"
check "up all succeeds" 0 "$status"
run -y down all
check "down reverses the order" \
"wiki compose down,webapp compose down,static-site compose down,prometheus compose down,grafana compose down,openbao compose down,reverse-proxy compose down" \
"$(calls)"
run up monitoring sites
check "groups expand, and repeated lines append" \
"grafana compose up,prometheus compose up,static-site compose up" "$(calls)"
check "a missing group member is warned about" yes \
"$(grep -q 'ghost' <<<"$out" && echo yes || echo no)"
run up grafana nope
check "an unknown target runs nothing" "" "$(calls)"
check "an unknown target fails" 2 "$status"
run up order
check "reserved names are not targets" 2 "$status"
run down system
check "down of a confirm stack without a TTY runs nothing" "" "$(calls)"
check "down of a confirm stack without a TTY fails" 1 "$status"
run -y down system
check "-y allows it" "webapp compose down,openbao compose down" "$(calls)"
run up openbao
check "up of a confirm stack needs no confirmation" "openbao compose up" "$(calls)"
FAIL=webapp run up system
check "a failing stack does not stop the rest" \
"openbao compose up,webapp compose up" "$(calls)"
check "a failing stack fails the run" 1 "$status"
check "a failing stack is named at the end" yes \
"$(tail -n1 <<<"$out" | grep -q webapp && echo yes || echo no)"
run logs grafana prometheus
check "logs takes exactly one stack" 2 "$status"
run ps grafana
check "ps counts running containers" yes \
"$(grep -Eq 'grafana +1/2' <<<"$out" && echo yes || echo no)"
run frobnicate all
check "an unknown command fails" 2 "$status"
[ "$failures" -eq 0 ] || { echo "$failures failed"; exit 1; }
echo "all stackctl tests passed"
+4 -1
View File
@@ -6,7 +6,9 @@ verbatim: no templating, no variable substitution by Ansible, no generation.
```
server/<host>/<stack>/ a stack that runs on exactly one host
server/shared/<stack>/ a payload used by more than one host or instance
server/shared/components/ single files that belong to another stack's directory
server/shared/components/ single files that belong to another stack's directory,
or to the stack root itself
server/<host>/stack-groups that host's stackctl groups; a file, not a stack
```
The directory names under `server/` must match the inventory host names, because
@@ -32,6 +34,7 @@ is the reusable part.
| `app/openbao/` | `openbao.yml` | The vault, on `127.0.0.1:8200` only. `config/openbao.hcl` is bind-mounted; a change needs a manual restart and unseal. |
| `shared/metrics/` | `metrics.yml` | Prometheus and Grafana, identical on both hosts, so it lives here rather than being duplicated per host. Its `.env` is placed on each host by hand. |
| `shared/components/banner.html` | `banner.yml` | A single file that belongs to the reverse proxy's directory on every host. |
| `shared/components/stackctl`, `<host>/stack-groups` | `stackctl.yml` | The by-hand stack runner, installed into `/srv/stacks` on every host, with each host's groups beside it. Not stacks. |
## Conventions worth copying
+12
View File
@@ -0,0 +1,12 @@
# Groups for stackctl on app. Format and reserved names: see the same file in
# server/edge/.
#
# OpenBao starts first because the stacks that use stack_secrets read it during a
# deploy, and it needs confirming before down or restart: restarting it seals it.
order: openbao
confirm: openbao
system: openbao
apps: static-site webapp webapp-staging
monitoring: metrics
+18
View File
@@ -0,0 +1,18 @@
# Groups for stackctl on edge, installed next to it in /srv/stacks. One group
# per line:
#
# name: stack stack ...
#
# A name on several lines collects all of them. A stack may be in any number of
# groups, or none; "all" always means every stack with a compose file. Each host
# has its own file, because each host has its own stacks.
#
# Two names are reserved:
# order: started first, in this order; everything else follows
# alphabetically, and down runs the whole list in reverse.
# confirm: down and restart ask before touching these, unless given -y.
order: reverse-proxy
proxy: reverse-proxy
monitoring: metrics
+194
View File
@@ -0,0 +1,194 @@
#!/usr/bin/env bash
#
# Start, stop and inspect the stacks under this directory by hand, one at a time
# or by group. Installed to /srv/stacks on every host by
# ansible/playbooks/stackctl.yml; each host's groups live next to it in
# stack-groups.
#
# stackctl up apps # every stack in the apps group
# stackctl -y down all # everything, without asking about openbao
# stackctl pull static-site && stackctl up static-site
# stackctl ps # running containers per stack
# stackctl logs webapp
#
# Each stack is plain `docker compose` in its own directory: the default compose
# file, the .env beside it, the directory name as the project name, and PWD set
# to the directory. That is what an Ansible deploy does for any stack that
# leaves stack_files and stack_env at their defaults, or sets only
# COMPOSE_PROJECT_NAME to the directory name. A stack that relies on anything
# else from those two is not started the same way here.
#
# Ansible remains how stacks are deployed; this is for working on the host.
set -euo pipefail
root="$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)"
groups_file="$root/stack-groups"
usage() {
cat >&2 <<EOF
usage: stackctl [-y] <command> [target...]
commands:
up docker compose up -d
down docker compose down
restart docker compose restart
pull docker compose pull
ps running containers per stack (default target: all)
logs follow the logs of exactly one stack
list the groups and every stack found
target: a stack, a group from $groups_file, or "all".
-y skips the confirmation before down or restart of a stack under "confirm:".
EOF
exit 2
}
die() { echo "stackctl: $*" >&2; exit "${2:-2}"; }
warn() { echo "stackctl: $*" >&2; }
yes=false
while getopts ':yh' opt; do
case "$opt" in
y) yes=true ;;
*) usage ;;
esac
done
shift $((OPTIND - 1))
[ $# -ge 1 ] || usage
command="$1"
shift
case "$command" in
up | down | restart | pull | logs) ;;
ps | list) [ $# -ge 1 ] || set -- all ;;
*) warn "unknown command: $command"; usage ;;
esac
[ $# -ge 1 ] || die "$command needs a target: a stack, a group, or all"
# Every directory holding a compose file is a stack.
stacks=()
for dir in "$root"/*/; do
[ -f "$dir/docker-compose.yml" ] && stacks+=("$(basename "$dir")")
done
is_stack() {
local s
for s in "${stacks[@]}"; do [ "$s" = "$1" ] && return 0; done
return 1
}
# stack-groups: "name: stack stack ...", one group per line. A name given on
# several lines collects all of them. "order" and "confirm" are reserved.
declare -A group
order=()
confirm=()
if [ -f "$groups_file" ]; then
while IFS= read -r line || [ -n "$line" ]; do
line="${line%%#*}"
[[ "$line" == *:* ]] || continue
name="$(echo "${line%%:*}" | tr -d '[:space:]')"
read -ra members <<<"${line#*:}"
case "$name" in
order) order+=("${members[@]}") ;;
confirm) confirm+=("${members[@]}") ;;
*) group[$name]="${group[$name]:-} ${members[*]}" ;;
esac
done <"$groups_file"
fi
# Resolve targets to stacks, without duplicates.
declare -A chosen
for target in "$@"; do
case "$target" in
all)
for s in "${stacks[@]}"; do chosen[$s]=1; done
;;
order | confirm)
die "\"$target\" is reserved in stack-groups, not a target"
;;
*)
if [ -n "${group[$target]+x}" ]; then
for s in ${group[$target]}; do
if is_stack "$s"; then
chosen[$s]=1
else
warn "group $target names $s, which has no compose file here; skipping"
fi
done
elif is_stack "$target"; then
chosen[$target]=1
else
die "no stack or group called $target (stackctl list shows both)"
fi
;;
esac
done
# Stacks under "order:" first, in that order, then the rest alphabetically.
rank() {
local i
for i in "${!order[@]}"; do
[ "${order[$i]}" = "$1" ] && { printf '%04d' "$i"; return; }
done
printf '9999'
}
mapfile -t selected < <(
for s in "${!chosen[@]}"; do printf '%s %s\n' "$(rank "$s")" "$s"; done |
sort -k1,1n -k2,2 | cut -d' ' -f2
)
if [ "$command" = down ]; then
mapfile -t selected < <(printf '%s\n' "${selected[@]}" | tac)
fi
[ "${#selected[@]}" -gt 0 ] || die "nothing to do" 1
case "$command" in
list)
for name in $(printf '%s\n' "${!group[@]}" | sort); do
printf '%-10s%s\n' "$name:" "${group[$name]}"
done
printf '%-10s%s\n' "stacks:" "${selected[*]}"
exit 0
;;
logs)
[ "${#selected[@]}" -eq 1 ] || die "logs follows exactly one stack"
cd "$root/${selected[0]}"
exec docker compose logs -f --tail=100
;;
ps)
for s in "${selected[@]}"; do
states="$(cd "$root/$s" && docker compose ps -a --format '{{.State}}')"
total="$(grep -c . <<<"$states" || true)"
running="$(grep -c '^running$' <<<"$states" || true)"
printf '%-24s %s/%s\n' "$s" "$running" "$total"
done
exit 0
;;
esac
# Ask before stopping anything under "confirm:" — restarting OpenBao seals it.
if [ "$command" = down ] || [ "$command" = restart ]; then
for s in "${selected[@]}"; do
for c in "${confirm[@]}"; do
[ "$s" = "$c" ] || continue
$yes && continue
[ -t 0 ] || die "$command $s needs confirmation; rerun with -y" 1
read -rp "$command $s? [y/N] " answer
[[ "$answer" == [yY]* ]] || die "not confirmed; nothing done" 1
done
done
fi
case "$command" in
up) args=(up -d) ;;
*) args=("$command") ;;
esac
failed=()
for s in "${selected[@]}"; do
echo "== $command $s"
(cd "$root/$s" && docker compose "${args[@]}") || failed+=("$s")
done
if [ "${#failed[@]}" -gt 0 ]; then
die "failed: ${failed[*]}" 1
fi
+16 -2
View File
@@ -48,9 +48,12 @@ files and one button. `AGENTS.md` holds the conventions and the full
Delete whichever platforms you are not using rather than letting them rot.
Leave a stack out only if it must never deploy from CI — see the table below.
5. **Verify statically.** `make check` from the repository root. It must pass;
5. **Group it, if it belongs in one.** Add it to a line in
`server/<host>/stack-groups` so `stackctl up <group>` on that host includes
it. Optional: `stackctl all` finds every stack regardless.
6. **Verify statically.** `make check` from the repository root. It must pass;
this is the definition of done for the playbook.
6. **Deploy.** Run the check mode, read the diff, then deploy. Never run a
7. **Deploy.** Run the check mode, read the diff, then deploy. Never run a
playbook against a host unless the person you are working with asked for it in
that turn.
@@ -100,6 +103,17 @@ working around it — that is information about the contract.
- **A reload task with no `when: compose_stack_synced`.** Every run then reports
a change, the playbook is never idempotent, and `changed=0` stops meaning
anything.
- **Committing an upstream example compose file without reading it.** They are
written for a quick start, not for a host. Look for three things: a password
given as a default (`${POSTGRES_PASSWORD:-changeme}`), which is a committed
secret — make it `${VAR:?}` and use `stack_secrets`; database or cache ports
published on every interface, which nothing outside the stack needs; and
`KEY:value` entries in a list-form `environment:`, which set a variable
literally named `KEY:value` and nothing else — the list form needs `=`.
- **A stateful service with no bind mount.** An app that falls back to SQLite
inside its container loses everything the next time Compose recreates it. Find
where it writes and mount that directory; if the image runs as a fixed uid,
give the directory to it with `stack_dirs`.
- **Putting the `stack_*` vars inside `roles:` for a two-phase playbook.** They
are scoped to that invocation and the second `include_role` will not see them.
Declare them at play level.