Lucas WintherandClaude Opus 5
e1f152ec3e
README for humans, AGENTS.md for agents and contributors, and the docs that outlive any one platform: architecture, secrets, connectivity, triggering a deploy from another repository. CLAUDE.md, GEMINI.md and .claude/skills/ are pointers rather than copies, so every agent and every human reads the same text. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
106 lines
4.9 KiB
Markdown
106 lines
4.9 KiB
Markdown
# When CI cannot reach a host
|
|
|
|
A deploy job that **hangs and then times out** is a different problem from one
|
|
that fails, and the distinction tells you where to look.
|
|
|
|
| What you see | What it means | Where the problem is |
|
|
| --- | --- | --- |
|
|
| `Permission denied (publickey)` | You reached sshd and it rejected the key | The key, or `authorized_keys` |
|
|
| `Connection refused` | You reached the host; nothing is listening on that port | sshd is down, or on another port |
|
|
| `Connection timed out` | The packets vanished in transit | The network path |
|
|
|
|
Only the third is a network problem. No amount of re-pasting the deploy key will
|
|
fix it, and it is worth being sure which one you have before spending an
|
|
afternoon on the wrong thing.
|
|
|
|
## The probe
|
|
|
|
`connectivity.yml` has two jobs.
|
|
|
|
**`ping`** uses the real deploy key and asks Ansible to talk to every host in the
|
|
inventory. Success looks like `SUCCESS => {"ping": "pong"}` for each one, which
|
|
proves the runner decoded the key, reached port 22, authenticated, and ran Python
|
|
on the far side. That is everything a deploy needs except the playbook. Run this
|
|
first, always.
|
|
|
|
**`probe`** uses no credentials at all and answers a narrower question: *where do
|
|
the packets stop?* Run it when `ping` times out. It reports:
|
|
|
|
| Section | Question |
|
|
| --- | --- |
|
|
| 0 | Does the runner have working DNS and outbound HTTPS at all? And what is its egress address? |
|
|
| 1 | Does each target host answer with an SSH banner? |
|
|
| 2 | Does SSH to a known-good third party work — is outbound 22 blocked wholesale? |
|
|
| 3 | For each host and port: open, refused, or filtered? |
|
|
|
|
## Reading the result
|
|
|
|
| What you see | What it means |
|
|
| --- | --- |
|
|
| Section 0 has no DNS or no external HTTPS | The runner has no usable egress at all. Talk to whoever runs it. |
|
|
| Section 2 gets `SSH-2.0-...` but section 1 does not | Outbound 22 works generally; your hosts are dropping this runner specifically. Take the egress address to your firewall or cloud security group. |
|
|
| Section 2 also gets no reply | Outbound 22 is blocked wholesale on the runner's network. |
|
|
| Every port but 443 `filtered` in section 3 | An egress allowlist. No alternative SSH port will help — see below. |
|
|
|
|
Two details in the probe are deliberate and worth keeping if you rewrite it.
|
|
|
|
**It asks for the SSH banner, not just a TCP connect.** A bare connect can be
|
|
answered by a transparent proxy that then says nothing, which looks like success
|
|
and is not. A real sshd greets first, so `SSH-2.0-...` is the only honest proof
|
|
you reached the actual server.
|
|
|
|
**It distinguishes `refused` from `filtered`.** "Refused" means you reached the
|
|
host and nothing was listening, so that port is *allowed out*. "Filtered" means
|
|
the packets vanished. That difference is what tells you whether you are looking
|
|
at a service problem or a network policy, and it is the one thing a plain
|
|
`timeout` in a deploy log never tells you.
|
|
|
|
## Fixing it
|
|
|
|
### Move the runner, not the port
|
|
|
|
If you find an egress allowlist, resist the urge to move sshd to a different
|
|
port. An allowlist permits a *set* of destination ports; it is not a block on 22,
|
|
so there is nowhere to move to. Test this rather than assuming it — the probe's
|
|
section 3 will tell you in a minute.
|
|
|
|
The reliable fix is a runner that sits somewhere with a working path to the
|
|
hosts. A CI runner only makes **outbound HTTPS** connections to its coordinator,
|
|
which is why this works where an inbound exception would need a policy change and
|
|
someone else's timeline.
|
|
|
|
```bash
|
|
# GitHub: Settings > Actions > Runners > New self-hosted runner
|
|
# GitLab:
|
|
sudo gitlab-runner register --url https://gitlab.example.com --token <token>
|
|
# Gitea:
|
|
act_runner register --instance https://gitea.example.com --token <token> \
|
|
--labels deploy-runner:docker://docker.gitea.com/runner-images:ubuntu-latest
|
|
```
|
|
|
|
Then point the workflows at it — `DEPLOY_RUNNER` on GitHub, the `tags:` block in
|
|
`.gitlab-ci.yml` on GitLab, and the `runs-on:` label in `.gitea/workflows/` on
|
|
Gitea, which has to be edited in the file because Gitea takes no expression
|
|
there. All three are described in the CI setup docs.
|
|
|
|
Two prerequisites, easy to forget: the host firewall or cloud security group must
|
|
allow SSH **from the runner's address**, and the deploy key must be authorised
|
|
for the deploy user on each host.
|
|
|
|
### Or ask for an exception
|
|
|
|
Ask whoever runs the network to allow the runner's egress address outbound to
|
|
your hosts on port 22. Less work on your side, but it depends on someone else's
|
|
policy and timeline, and it has to be renewed whenever the runner moves.
|
|
|
|
### In the meantime
|
|
|
|
Deploy from a workstation that can reach the hosts. The playbooks are identical
|
|
either way — that is the point of the `SSH_KEY_PATH` seam.
|
|
|
|
## A note on ICMP
|
|
|
|
Many cloud providers drop ICMP by default, so a failed `ping` to a host proves
|
|
nothing on its own. The probe does not test it for that reason. Judge
|
|
reachability by the SSH banner and the port states.
|