Add the documentation and the agent instructions
README for humans, AGENTS.md for agents and contributors, and the docs that outlive any one platform: architecture, secrets, connectivity, triggering a deploy from another repository. CLAUDE.md, GEMINI.md and .claude/skills/ are pointers rather than copies, so every agent and every human reads the same text. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 5
parent
971034e630
commit
e1f152ec3e
@@ -0,0 +1,105 @@
|
||||
# When CI cannot reach a host
|
||||
|
||||
A deploy job that **hangs and then times out** is a different problem from one
|
||||
that fails, and the distinction tells you where to look.
|
||||
|
||||
| What you see | What it means | Where the problem is |
|
||||
| --- | --- | --- |
|
||||
| `Permission denied (publickey)` | You reached sshd and it rejected the key | The key, or `authorized_keys` |
|
||||
| `Connection refused` | You reached the host; nothing is listening on that port | sshd is down, or on another port |
|
||||
| `Connection timed out` | The packets vanished in transit | The network path |
|
||||
|
||||
Only the third is a network problem. No amount of re-pasting the deploy key will
|
||||
fix it, and it is worth being sure which one you have before spending an
|
||||
afternoon on the wrong thing.
|
||||
|
||||
## The probe
|
||||
|
||||
`connectivity.yml` has two jobs.
|
||||
|
||||
**`ping`** uses the real deploy key and asks Ansible to talk to every host in the
|
||||
inventory. Success looks like `SUCCESS => {"ping": "pong"}` for each one, which
|
||||
proves the runner decoded the key, reached port 22, authenticated, and ran Python
|
||||
on the far side. That is everything a deploy needs except the playbook. Run this
|
||||
first, always.
|
||||
|
||||
**`probe`** uses no credentials at all and answers a narrower question: *where do
|
||||
the packets stop?* Run it when `ping` times out. It reports:
|
||||
|
||||
| Section | Question |
|
||||
| --- | --- |
|
||||
| 0 | Does the runner have working DNS and outbound HTTPS at all? And what is its egress address? |
|
||||
| 1 | Does each target host answer with an SSH banner? |
|
||||
| 2 | Does SSH to a known-good third party work — is outbound 22 blocked wholesale? |
|
||||
| 3 | For each host and port: open, refused, or filtered? |
|
||||
|
||||
## Reading the result
|
||||
|
||||
| What you see | What it means |
|
||||
| --- | --- |
|
||||
| Section 0 has no DNS or no external HTTPS | The runner has no usable egress at all. Talk to whoever runs it. |
|
||||
| Section 2 gets `SSH-2.0-...` but section 1 does not | Outbound 22 works generally; your hosts are dropping this runner specifically. Take the egress address to your firewall or cloud security group. |
|
||||
| Section 2 also gets no reply | Outbound 22 is blocked wholesale on the runner's network. |
|
||||
| Every port but 443 `filtered` in section 3 | An egress allowlist. No alternative SSH port will help — see below. |
|
||||
|
||||
Two details in the probe are deliberate and worth keeping if you rewrite it.
|
||||
|
||||
**It asks for the SSH banner, not just a TCP connect.** A bare connect can be
|
||||
answered by a transparent proxy that then says nothing, which looks like success
|
||||
and is not. A real sshd greets first, so `SSH-2.0-...` is the only honest proof
|
||||
you reached the actual server.
|
||||
|
||||
**It distinguishes `refused` from `filtered`.** "Refused" means you reached the
|
||||
host and nothing was listening, so that port is *allowed out*. "Filtered" means
|
||||
the packets vanished. That difference is what tells you whether you are looking
|
||||
at a service problem or a network policy, and it is the one thing a plain
|
||||
`timeout` in a deploy log never tells you.
|
||||
|
||||
## Fixing it
|
||||
|
||||
### Move the runner, not the port
|
||||
|
||||
If you find an egress allowlist, resist the urge to move sshd to a different
|
||||
port. An allowlist permits a *set* of destination ports; it is not a block on 22,
|
||||
so there is nowhere to move to. Test this rather than assuming it — the probe's
|
||||
section 3 will tell you in a minute.
|
||||
|
||||
The reliable fix is a runner that sits somewhere with a working path to the
|
||||
hosts. A CI runner only makes **outbound HTTPS** connections to its coordinator,
|
||||
which is why this works where an inbound exception would need a policy change and
|
||||
someone else's timeline.
|
||||
|
||||
```bash
|
||||
# GitHub: Settings > Actions > Runners > New self-hosted runner
|
||||
# GitLab:
|
||||
sudo gitlab-runner register --url https://gitlab.example.com --token <token>
|
||||
# Gitea:
|
||||
act_runner register --instance https://gitea.example.com --token <token> \
|
||||
--labels deploy-runner:docker://docker.gitea.com/runner-images:ubuntu-latest
|
||||
```
|
||||
|
||||
Then point the workflows at it — `DEPLOY_RUNNER` on GitHub, the `tags:` block in
|
||||
`.gitlab-ci.yml` on GitLab, and the `runs-on:` label in `.gitea/workflows/` on
|
||||
Gitea, which has to be edited in the file because Gitea takes no expression
|
||||
there. All three are described in the CI setup docs.
|
||||
|
||||
Two prerequisites, easy to forget: the host firewall or cloud security group must
|
||||
allow SSH **from the runner's address**, and the deploy key must be authorised
|
||||
for the deploy user on each host.
|
||||
|
||||
### Or ask for an exception
|
||||
|
||||
Ask whoever runs the network to allow the runner's egress address outbound to
|
||||
your hosts on port 22. Less work on your side, but it depends on someone else's
|
||||
policy and timeline, and it has to be renewed whenever the runner moves.
|
||||
|
||||
### In the meantime
|
||||
|
||||
Deploy from a workstation that can reach the hosts. The playbooks are identical
|
||||
either way — that is the point of the `SSH_KEY_PATH` seam.
|
||||
|
||||
## A note on ICMP
|
||||
|
||||
Many cloud providers drop ICMP by default, so a failed `ping` to a host proves
|
||||
nothing on its own. The probe does not test it for that reason. Judge
|
||||
reachability by the SSH banner and the port states.
|
||||
Reference in New Issue
Block a user