Cache the CI toolchain in a shared action

ci/toolchain installs the pinned tools, collections and rsync from cache for
both GitHub and Gitea; GitLab caches pip downloads. shellcheck now finds
scripts rather than globbing, which broke when a glob matched nothing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

.ansible-lint

@@ -6,6 +6,7 @@ profile: production
 exclude_paths:
   - .git/
   - .idea/
+  - .cache/
   # Stack payloads are data — third-party compose and config files copied to the
   # host verbatim. They are not Ansible content and must not be linted as if
   # they were. See server/README.md.

.gitea/workflows/connectivity.yml

@@ -36,13 +36,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
       - name: Install the pinned toolchain
-        run: |
-          set -eu
-          scripts/check.sh --print-specs | head -1 | xargs -d '\n' pip install --quiet
+        uses: ./ci/toolchain
       - name: Load the deploy key
         env:
           SSH_KEY_BACKEND: ci

.gitea/workflows/deploy.yml

@@ -88,10 +88,6 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-
       # STACK, MODE and PULL reach a shell command, and over the dispatch API they
       # arrive from another repository. They go through the environment, never
       # through ${{ }} interpolation into a run: block — an expression is
@@ -138,11 +134,13 @@ jobs:
             echo "PULL=$PULL"
           } >> "$GITHUB_ENV"
 
+      # rsync too: ansible.posix.synchronize shells out to it on the machine
+      # Ansible runs on, not only on the target.
       - name: Install the pinned toolchain
-        run: |
-          set -eu
-          scripts/check.sh --print-specs | head -1 | xargs -d '\n' pip install --quiet
-          ansible-galaxy install -r ansible/requirements.yml
+        uses: ./ci/toolchain
+        with:
+          collections: "true"
+          rsync: "true"
 
       # Two lines, not `export SSH_KEY_PATH="$(...)"`. export is a special builtin,
       # so the compound command reports *its* exit status and swallows the

.gitea/workflows/lint.yml

@@ -37,20 +37,15 @@ jobs:
       # mirror actions/checkout into the instance, or set the setting back.
       - uses: actions/checkout@v4
 
-      # Keep this even though the runner image ships a Python. Ubuntu 24.04 images
-      # are PEP 668 "externally managed", where a plain `pip install` refuses to
-      # touch the system interpreter; setup-python brings its own and sidesteps it.
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-
-      # scripts/check.sh is the single source of the version pins; it prints them
-      # so that this workflow installs exactly what a contributor runs locally.
-      # Nothing here can drift from `make check`.
+      # scripts/check.sh is the single source of the version pins; the action
+      # installs exactly what a contributor runs locally, so nothing here can
+      # drift from `make check`. The collections are cached with it, which
+      # leaves check.sh's own `ansible-galaxy install` nothing to fetch.
       - name: Install the pinned toolchain
-        run: |
-          set -eu
-          scripts/check.sh --print-specs | tee /dev/stderr | xargs -d '\n' pip install --quiet
+        uses: ./ci/toolchain
+        with:
+          specs: all
+          collections: "true"
 
       - name: Run the static gate
         run: RUNNER=installed scripts/check.sh
@@ -73,7 +68,13 @@ jobs:
             $sudo apt-get install -y --no-install-recommends shellcheck
           fi
       - name: Check the shell scripts
-        run: shellcheck scripts/*.sh server/*/*/app/*.sh
+        # find rather than a glob. Bash leaves an unmatched pattern alone instead
+        # of expanding it to nothing, so `server/*/*/app/*.sh` reaches
+        # shellcheck as a literal filename the moment the last payload script
+        # goes away, and the job fails with `openBinaryFile: does not exist`.
+        # `xargs -r` covers the other half: no scripts at all must be a no-op,
+        # not a shellcheck invocation with no arguments that sits reading stdin.
+        run: find scripts server -name '*.sh' -print0 | xargs -0 -r shellcheck
 
   secrets:
     name: secret scan

.github/workflows/connectivity.yml

@@ -37,13 +37,8 @@ jobs:
     environment: production
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
       - name: Install the pinned toolchain
-        run: |
-          set -eu
-          scripts/check.sh --print-specs | head -1 | xargs -d '\n' pip install --quiet
+        uses: ./ci/toolchain
       - name: Load the deploy key
         env:
           SSH_KEY_BACKEND: ci

.github/workflows/deploy.yml

@@ -80,10 +80,6 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-
       # STACK, MODE and PULL reach a shell command, and under repository_dispatch
       # they come from another repository. They go through the environment, never
       # through ${{ }} interpolation into a run: block — an expression is
@@ -122,11 +118,13 @@ jobs:
             echo "PULL=$PULL"
           } >> "$GITHUB_ENV"
 
+      # rsync too: ansible.posix.synchronize shells out to it on the machine
+      # Ansible runs on, not only on the target.
       - name: Install the pinned toolchain
-        run: |
-          set -eu
-          scripts/check.sh --print-specs | head -1 | xargs -d '\n' pip install --quiet
-          ansible-galaxy install -r ansible/requirements.yml
+        uses: ./ci/toolchain
+        with:
+          collections: "true"
+          rsync: "true"
 
       # Two lines, not `export SSH_KEY_PATH="$(...)"`. export is a special
       # builtin, so the compound command reports *its* exit status and swallows

.github/workflows/lint.yml

@@ -27,17 +27,15 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.12"
-
-      # scripts/check.sh is the single source of the version pins; it prints
-      # them so that this workflow can install exactly what a contributor runs
-      # locally. Nothing here can drift from the script.
+      # scripts/check.sh is the single source of the version pins; the action
+      # installs exactly what a contributor runs locally, so nothing here can
+      # drift from `make check`. The collections are cached with it, which
+      # leaves check.sh's own `ansible-galaxy install` nothing to fetch.
       - name: Install the pinned toolchain
-        run: |
-          set -eu
-          scripts/check.sh --print-specs | tee /dev/stderr | xargs -d '\n' pip install --quiet
+        uses: ./ci/toolchain
+        with:
+          specs: all
+          collections: "true"
 
       - name: Run the static gate
         run: RUNNER=installed scripts/check.sh
@@ -52,7 +50,13 @@ jobs:
       - name: Install shellcheck
         run: command -v shellcheck || sudo apt-get install -y --no-install-recommends shellcheck
       - name: Check the shell scripts
-        run: shellcheck scripts/*.sh server/*/*/app/*.sh
+        # find rather than a glob. Bash leaves an unmatched pattern alone instead
+        # of expanding it to nothing, so `server/*/*/app/*.sh` reaches
+        # shellcheck as a literal filename the moment the last payload script
+        # goes away, and the job fails with `openBinaryFile: does not exist`.
+        # `xargs -r` covers the other half: no scripts at all must be a no-op,
+        # not a shellcheck invocation with no arguments that sits reading stdin.
+        run: find scripts server -name '*.sh' -print0 | xargs -0 -r shellcheck
 
   secrets:
     name: secret scan

.gitignore

@@ -25,6 +25,7 @@ ansible/inventory/host_vars/*.local.yml
 
 # Python, for the linting toolchain
 __pycache__/
+.cache/
 *.pyc
 .venv/
 

.gitlab-ci.yml

@@ -37,6 +37,21 @@ variables:
   # discovery rule. Paths inside it then resolve relative to the file, so this
   # works from any working directory.
   ANSIBLE_CONFIG: $CI_PROJECT_DIR/ansible/ansible.cfg
+  # pip's download cache, kept between jobs by the default cache below. A fresh
+  # toolchain install can take longer than the deploy it serves. GitLab caches
+  # only paths inside the project directory, hence the location.
+  PIP_CACHE_DIR: $CI_PROJECT_DIR/.cache/pip
+
+# Keyed on the files that hold the pins, so raising one starts a fresh cache. A
+# runner with no cache configured just installs from the network, as before.
+default:
+  cache:
+    key:
+      files:
+        - scripts/check.sh
+        - ansible/requirements.yml
+    paths:
+      - .cache/pip
 
 lint:
   stage: test

.yamllint

@@ -4,6 +4,7 @@ extends: default
 ignore: |
   .git/
   .idea/
+  .cache/
   server/
 
 rules:

ci/toolchain/action.yml

@@ -0,0 +1,132 @@
+---
+# The pinned toolchain, cached. Shared by the deploy, lint and connectivity
+# workflows of both GitHub Actions and Gitea Actions, so that the six cannot
+# drift and so that the cache logic exists once. It lives outside .github/ and
+# .gitea/ for that reason: deleting the platform you do not use must not delete
+# it. Call it after actions/checkout, as `uses: ./ci/toolchain` — it reads the
+# pins from scripts/check.sh and is itself read from the checkout.
+#
+# A fresh install can take a couple of minutes against a deploy of seconds. On a
+# cache hit nothing here touches the network.
+#
+# The toolchain key covers which pins were asked for, the pins themselves
+# (check.sh, requirements.yml), the exact interpreter the venv is bound to, and
+# the month: the pins are ranges so that patch releases arrive, and a cache that
+# never expired would stop them. A new month, a raised pin or a new Python patch
+# builds it afresh. The hash is taken in shell, not with hashFiles(), because
+# Gitea does not implement it — see "Expressions are barely supported" in
+# docs/ci-gitea.md.
+#
+# A cache server the job cannot reach is a miss, not a failure: the steps fall
+# back to installing, exactly as they would with no cache at all.
+name: Pinned toolchain
+description: Python, the pinned Ansible tools and optionally rsync, from cache when possible
+
+inputs:
+  specs:
+    description: >-
+      core installs ansible-core only; all adds ansible-lint and yamllint
+    default: core
+  collections:
+    description: Install the Galaxy collections in ansible/requirements.yml
+    default: "false"
+  rsync:
+    description: Install rsync, which ansible.posix.synchronize needs locally
+    default: "false"
+
+runs:
+  using: composite
+  steps:
+    # Kept even though runner images ship a Python: Ubuntu 24.04 images are PEP
+    # 668 "externally managed", where a plain `pip install` refuses to touch the
+    # system interpreter. See "actions/setup-python stays" in docs/ci-gitea.md.
+    - uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+
+    - name: Compute the cache keys
+      id: keys
+      shell: bash
+      env:
+        SPECS: ${{ inputs.specs }}
+      run: |
+        set -eu
+        case "$SPECS" in
+          core|all) ;;
+          *) echo "toolchain: specs must be core or all, not '$SPECS'"; exit 1 ;;
+        esac
+        . /etc/os-release
+        py="$(python -c 'import platform; print(platform.python_version())')"
+        pins="$(cat scripts/check.sh ansible/requirements.yml | sha256sum | cut -c1-16)"
+        {
+          echo "os=${ID}-${VERSION_ID}"
+          echo "toolchain=toolchain-${SPECS}-${ID}-${VERSION_ID}-py${py}-$(date -u +%Y-%m)-${pins}"
+          echo "venv=$HOME/.venvs/toolchain"
+        } >> "$GITHUB_OUTPUT"
+
+    # ansible.posix.synchronize shells out to rsync on the machine Ansible runs
+    # on, not only on the target. GitHub's hosted images ship it; Gitea's
+    # act_runner images do not. The .debs are cached rather than the apt lists,
+    # which are what the slow `apt-get update` fetches. Jobs run as root in a
+    # container runner, where sudo is often not installed at all, so it is
+    # asked for only off uid 0.
+    - name: Restore the rsync packages
+      if: inputs.rsync == 'true'
+      uses: actions/cache@v4
+      with:
+        path: ~/.cache/toolchain-debs
+        key: rsync-debs-${{ steps.keys.outputs.os }}
+
+    - name: Install rsync
+      if: inputs.rsync == 'true'
+      shell: bash
+      run: |
+        set -eu
+        if ! command -v rsync >/dev/null 2>&1; then
+          sudo=""
+          [ "$(id -u)" -eq 0 ] || sudo=sudo
+          debs="$HOME/.cache/toolchain-debs"
+          if ! ls "$debs"/rsync_*.deb >/dev/null 2>&1; then
+            mkdir -p "$debs/partial"
+            $sudo apt-get update -qq
+            $sudo apt-get install -y --no-install-recommends --download-only \
+              -o Dir::Cache::archives="$debs" rsync
+          fi
+          $sudo dpkg -i "$debs"/*.deb
+        fi
+
+    - name: Restore the pinned toolchain
+      id: toolchain
+      uses: actions/cache@v4
+      with:
+        path: |
+          ${{ steps.keys.outputs.venv }}
+          ~/.ansible/collections
+        key: ${{ steps.keys.outputs.toolchain }}-collections-${{ inputs.collections }}
+
+    # A venv rather than the setup-python interpreter itself, so that the cache
+    # holds exactly what this step installed and nothing else. check.sh prints
+    # the core pin first, which is what `specs: core` takes.
+    - name: Install the pinned toolchain
+      if: steps.toolchain.outputs.cache-hit != 'true'
+      shell: bash
+      env:
+        VENV: ${{ steps.keys.outputs.venv }}
+        SPECS: ${{ inputs.specs }}
+        COLLECTIONS: ${{ inputs.collections }}
+      run: |
+        set -eu
+        python -m venv "$VENV"
+        filter=(cat)
+        [ "$SPECS" = "all" ] || filter=(head -n 1)
+        scripts/check.sh --print-specs | "${filter[@]}" | tee /dev/stderr \
+          | xargs -d '\n' "$VENV/bin/pip" install --quiet
+        if [ "$COLLECTIONS" = "true" ]; then
+          "$VENV/bin/ansible-galaxy" install -r ansible/requirements.yml
+        fi
+
+    - name: Put the toolchain on PATH
+      shell: bash
+      env:
+        VENV: ${{ steps.keys.outputs.venv }}
+      run: echo "$VENV/bin" >> "$GITHUB_PATH"

docs/ci-gitea.md

@@ -173,9 +173,20 @@ plain `pip install` refuses to touch it. `setup-python` brings its own
 interpreter and sidesteps that. If you drop it, you need
 `pip install --break-system-packages` and you are then installing into the image.
 
+**The toolchain is installed once, in `ci/toolchain/`.** All three workflows —
+and the three GitHub ones — call that composite action rather than repeating
+`setup-python` and `pip install`. It keeps a venv and the Galaxy collections in
+the runner's built-in cache server, keyed on the pins and the month, so a cache
+hit makes no network requests. A job that cannot reach the cache server installs
+from scratch, which is slower but still passes. It also installs rsync, which
+the act_runner images lack and `synchronize` needs on the runner. If your runner
+version offers `tool_cache_mode: shared` in its `config.yaml`, that saves
+`setup-python` downloading 3.12 on every job too — but only at `capacity: 1`,
+since two jobs unpacking the same tool at once corrupt it.
+
 **Every `uses:` is downloaded from github.com** unless your administrator set
-`DEFAULT_ACTIONS_URL = self`, in which case `actions/checkout` and
-`actions/setup-python` must be mirrored into the instance as repositories of
+`DEFAULT_ACTIONS_URL = self`, in which case `actions/checkout`,
+`actions/setup-python` and `actions/cache` must be mirrored into the instance as repositories of
 those names, or every workflow here fails on its first step. That setting takes
 only `github` or `self`; an arbitrary mirror URL has to be spelled out in full
 in the workflow.

docs/ci-github-actions.md

@@ -64,6 +64,15 @@ works where an inbound exception would need a policy change. Do not set the
 variable before a runner carries that label — a label no runner has leaves jobs
 queued forever rather than failing.
 
+A self-hosted runner needs Python's `venv` module and, for deploys, either rsync
+or `apt-get` with sudo: the toolchain action installs rsync when it is missing,
+because `synchronize` runs it on the runner as well as on the host.
+
+The toolchain itself comes from `ci/toolchain/`, a composite action shared with
+the Gitea workflows. It caches a venv and the Galaxy collections with
+`actions/cache`, keyed on the pins in `scripts/check.sh` and
+`ansible/requirements.yml` and on the month, so most runs install nothing.
+
 ## Deploying
 
 Actions → **deploy** → Run workflow. Three fields: