protondrive: run the wider suites against the fake server in fork CI

Add the TestProtonDriveFake init.d script, which builds and starts
protonfake on port 28642 and passes on the session it prints, and a
test_all entry for it, so fs/operations, fs/sync, vfs and cmd/bisync
run against the fake server. Five tests that need a second Fs of the
same account to see what the first wrote are ignored, as each Fs
caches separately, and cmd/gitannex is skipped as for every init.d
remote; both are in docs/TODO.md.

Add a Gitea Actions workflow that builds, vets, lints, race-tests,
fuzzes and vulnerability-checks the backend, and re-verify the agent
permission table on the host toolchain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed 2026-11-12 09:23:26 +01:00
1 parent 65840bdc57
commit 5afbe9e4fa
10 files changed
+232 -45

No files matched your search

+2 -1
View File
@@ -5,7 +5,8 @@
"run_shell_command(git show)", "run_shell_command(git branch --list)",
"run_shell_command(git add)", "run_shell_command(git commit)",
"run_shell_command(go mod download)", "run_shell_command(go build)",
"run_shell_command(go test)", "run_shell_command(go vet)", "run_shell_command(gofmt -l)"
"run_shell_command(go test)", "run_shell_command(go vet)", "run_shell_command(gofmt -l)",
"run_shell_command(golangci-lint run)"
],
"exclude": [
"run_shell_command(git push --force)", "run_shell_command(git push -f)",
+51
View File
@@ -0,0 +1,51 @@
# Fork CI for the Proton Drive backend. Every step is hermetic: the
# tests run against the fake server in backend/protondrive/protontest.
name: ci
on:
push:
pull_request:
env:
# Never pick up a config holding a real Proton remote.
RCLONE_CONFIG: /notfound
jobs:
protondrive:
runs-on: ubuntu-latest
# The race suite takes about 6 minutes on a fast machine.
timeout-minutes: 45
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
# Caches the module and build caches, keyed on go.sum.
cache: true
- name: Build
run: go build ./...
- name: Vet
run: go vet ./backend/protondrive/...
- name: Lint
uses: golangci/golangci-lint-action@v8
with:
version: v2.15.0
args: ./backend/protondrive/...
- name: Test
run: go test -race ./backend/protondrive/...
- name: Fuzz
run: |
for target in $(go test -list '^Fuzz' ./backend/protondrive/ | grep '^Fuzz'); do
go test -run '^$' -fuzz "^${target}\$" -fuzztime 20s ./backend/protondrive/
done
- name: Vulnerability check
run: go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/...
+4
View File
@@ -63,6 +63,10 @@ A native client of the Proton Drive API, following Proton's Drive SDK
- `fake_test.go` runs the full `fstests` contract suite against
`protontest` through the hidden `endpoint` option, on every
`go test`; `protondrive_test.go` is the live suite (`TestProtonDrive:`).
- `fstest/testserver/init.d/TestProtonDriveFake` builds and starts
`protontest/cmd/protonfake` for `test_all`, which runs the wider suites
against the `TestProtonDriveFake:` remote. Fork CI is
`.gitea/workflows/ci.yml`.
## Known rough edges
+26
View File
@@ -5,6 +5,32 @@ it with a new one that links back. One entry per decision a future maintainer
would otherwise have to reverse-engineer.
No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry.
## 2026-11-12 - test_all shares one protonfake session across test processes
**Status:** accepted
**Context:** `fstest/testserver/init.d/TestProtonDriveFake` starts one
`protonfake` per `test_all` run (reference counted by `run.bash`), and
every test process gets the same session through
`RCLONE_CONFIG_TESTPROTONDRIVEFAKE_*` variables. Refresh tokens are
single use and a process can't see another's refresh, so one process
refreshing would leave the rest with dead tokens.
**Decision:** Share the session; change nothing in `protontest`,
`protonfake` or the client. The client refreshes only after an HTTP 401
(`session.call`); the fake answers 401 only for a session expired by
`Server.ExpireAccessToken` or ended by logout or `RevokeSession`, and
`protonfake` calls none of them. The only logouts are `Fs.Disconnect`
and re-running `rclone config`, which no test in `fs/operations`,
`fs/sync`, `vfs`, `cmd/bisync`, `cmd/gitannex` or `fstests` calls.
**Alternatives:** A session per test process (`protonfake` would need a
sign-in route for `test_all` to call, and the init.d protocol has one
output per server); non-expiring or logout-tolerant tokens in
standalone mode (behaviour no test needs, and a fake that differs from
Proton).
**Consequences:** A future test that calls `Disconnect`, or a fake that
expires tokens on a timer, breaks every process sharing the run: the
symptom is "not signed in" or refresh failures across all suites at
once. Revisit then.
## 2026-11-11 - Proton Drive rewrite paused mid-plan and merged to master
- **Decision:** at the owner's request, work on plan 001 stopped after Task
+20 -32
View File
@@ -8,55 +8,43 @@ every tool file in the same commit. Review changes to these files and to
## Verified commands
Update 2026-11-11: later the same day Go 1.27.4 (`~/.local/opt/go`) and
golangci-lint 2.15.0 (`~/.local/bin`) were installed on the host at the
owner's request, and all Proton Drive work was built, tested and linted
with them directly. The table below predates that and has not been re-run
on the host (plan Task 18, step 3).
Run on 2026-11-11. The development host had **no Go toolchain** at the time, so every `go`
row was run inside the `golang:1.26` container (go1.26.11) with the repository
mounted, using this wrapper:
```sh
docker run --rm -u "$(id -u):$(id -g)" -e HOME=/cache -e GOCACHE=/cache/build \
-e GOMODCACHE=/cache/mod -e GOFLAGS=-buildvcs=false -e RCLONE_CONFIG=/notfound \
-v "$HOME/.cache/rclone-go:/cache" -v "$PWD:/src" -w /src golang:1.26 <command>
```
Run on the host on 2026-11-12 with Go 1.27.4 (`~/.local/opt/go`) and
golangci-lint 2.15.0 (`~/.local/bin`), from the repository root, with
`RCLONE_CONFIG=/notfound` set so no test can reach a real remote. The rows
marked (container) are from 2026-11-11, before Go was on the host, run in
the `golang:1.26` container and not repeated.
Only `worked` rows may appear in a tier, a tool file, or `AGENTS.md`.
| Command | Result |
|---|---|
| `go mod download` | worked (container) |
| `go build` | worked (container) - binary reports `rclone v1.77.0-DEV` |
| `go test ./backend/memory/` | worked (container) |
| `go test -run 'TestIntegration/FsMkdir/FsPutFiles/FsCopy' ./backend/memory/` | worked (container) |
| `go test ./backend/protondrive/` | worked (container) - no `TestProtonDrive:` remote, so the integration test skips |
| `go vet ./backend/protondrive/` | worked (container) |
| `gofmt -l backend/protondrive` | worked (container) - no output |
| `RCLONE_CONFIG=/notfound go test ./...` | failed (container) - 178 packages ok, 7 fail: `backend/smb`, `cmd/gitannex`, `cmd/mount`, `cmd/mount2`, `cmd/serve/docker`, `cmd/serve/nfs`, `fs/logger`. `fs/logger` fails because the container has no `rsync`; the other six were not diagnosed |
| `make`, `make quicktest`, `make racequicktest` | not run - need Go on the host |
| `golangci-lint run ./...` | not run - not installed on the host or in the container |
| `go run ./fstest/test_all -backends protondrive` | not run - needs a configured `TestProtonDrive:` remote |
TODO(maintainer): confirm every row above that is not `worked`.
| `go mod download` | worked |
| `go build` | worked - binary reports `rclone v1.77.0-DEV` |
| `go test ./backend/memory/` | worked |
| `go test -run 'TestIntegration/FsMkdir/FsPutFiles/FsCopy' ./backend/memory/` | worked |
| `go test ./backend/protondrive/...` | worked - the contract suite runs against the fake server; the live `TestProtonDrive:` suite skips without a config |
| `go vet ./backend/protondrive/...` | worked |
| `gofmt -l backend/protondrive` | worked - no output |
| `golangci-lint run ./backend/protondrive/...` | worked - 0 issues |
| `golangci-lint run ./...` | worked - 0 issues |
| `go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4` | worked - starts the fake server through `fstest/testserver/init.d/TestProtonDriveFake` |
| `RCLONE_CONFIG=/notfound go test ./...` | failed - only `cmd/serve/s3`, undiagnosed (2026-11-11, `docs/TODO.md` "Known issues"); in the container 7 packages failed: `backend/smb`, `cmd/gitannex`, `cmd/mount`, `cmd/mount2`, `cmd/serve/docker`, `cmd/serve/nfs`, `fs/logger` (container) |
| `make`, `make quicktest`, `make racequicktest` | not run - `make` is in Ask |
## Tiers
| Tier | Commands | Why |
|---|---|---|
| Allow | `git status`, `git diff`, `git log`, `git show`, `git branch --list`, `git rev-parse`, `git add`, `git commit`, `go mod download`, `go build`, `go test`, `go vet`, `gofmt -l` | Inspection and the verified dev loop; no prompt needed. |
| Allow | `git status`, `git diff`, `git log`, `git show`, `git branch --list`, `git rev-parse`, `git add`, `git commit`, `go mod download`, `go build`, `go test`, `go vet`, `gofmt -l`, `golangci-lint run` | Inspection and the verified dev loop; no prompt needed. |
| Ask | `git push`, `git merge`, `git rebase`, `git switch`, `git restore`, `git commit --amend`, `rm` (any), `go get`, `go mod tidy`, `go install`, `make` (any target), `docker` (any) | Leaves the machine, rewrites work, changes dependencies, or is unverified. |
| Deny | `git push --force` / `-f` / `--force-with-lease`, `git reset --hard`, `git clean` (any), `git add -f` / `--force`, `git commit --no-verify` / `-n`, `rm -r` / `-R` / `-rf` / `-fr` / `-Rf` / `-fR` / `--recursive`, `sudo`; reading `.env`, `.env.*`, `secrets/**` | Destroys work, skips checks, or exposes secrets. |
## Known gaps
- `.claude/settings.json` does not yet allow `golangci-lint run`; the owner
adds it (`docs/TODO.md`), as an agent may not widen its own permissions.
- Allowlists save prompts; they are not a security boundary. Use the tool's
sandbox for isolation.
- While the host has no Go, the allowed `go` commands only work through the
`docker run` wrapper above, and `docker` is in Ask - so each one prompts.
Installing Go on the host removes this.
- Prefix rules are bypassable: `git -C dir push`, flags placed later
(`git push origin main --force`), `+refspec` pushes. Claude Code's `:*`
matches at a word boundary, so each flag cluster must be listed - which is
+32 -8
View File
@@ -6,10 +6,9 @@ Small, specific, independently actionable. Remove only when actually done.
## Now
- [ ] protondrive: the backend rewrite (plan 001) is paused after Task 17 of
21. Start from `docs/plans/001-protondrive-tier1-HANDOFF.md`. Remaining:
Task 18 (wider suites against the fake, fork CI), 19 (remove the old
libraries), 20 (user docs - `docs/content/protondrive.md` still describes
- [ ] protondrive: the backend rewrite (plan 001) has Tasks 1-18 of 21
done. Start from `docs/plans/001-protondrive-tier1-HANDOFF.md`.
Remaining: Task 19 (remove the old libraries), 20 (user docs - `docs/content/protondrive.md` still describes
the old backend), 21 (live verification, paid account never signed in to),
then the whole-branch review; later plans for M5 (events) and M6
(persistent cache, Metadata, small-file upload).
@@ -118,7 +117,35 @@ Small, specific, independently actionable. Remove only when actually done.
- [ ] protondrive: several `Fs` of one account in one process each have
their own node cache; only Move/DirMove sources are invalidated across
them. Events (spec 6.2) fix this.
them. A name missing from one `Fs`'s cached listing stays missing after
another `Fs` creates it, so `--compare-dest`, `--copy-dest` (with
`--immutable`) and `checksum --download` decide on stale listings when
the other `Fs` writes during the run, and an `Fs` made on a path that
doesn't exist yet never finds it once it is made elsewhere. Five
`test_all` tests fail for this and are in the `ignore` list of
`TestProtonDriveFake:` (`fstest/test_all/config.yaml`):
`TestCheckSumDownload`, `TestCopyFileCompareDest`,
`TestCopyFileCopyDestImmutable`, `TestSyncCompareDest`,
`TestSyncCopyDestImmutable`; all pass with `enable_caching = false`.
Events polled every 30 s (spec 6.2) will not close a window of
seconds. Re-listing on every cache miss fixes them but breaks
`TestBudgetUpload` and `TestBudgetMkdir` (a folder re-listed per new
file). A cache shared by the `Fs` of one account would fix them, but
is package state, which the spec rules out: owner decision.
- [ ] protondrive: `cmd/gitannex` is in `ignoretests` for
`TestProtonDriveFake:` because its remote name check
(`cmd/gitannex/configparse.go`, `config.GetRemoteNames`) sees remotes
set by environment variables lower cased, as for every other init.d
remote; it says nothing about the backend. Run it against a remote in
a config file to cover it.
- [ ] `.claude/settings.json` lacks `"Bash(golangci-lint run:*)"` in
`allow`, which `docs/PERMISSIONS.md` lists in the Allow tier since plan
Task 18 (`.gemini/settings.json` has it; Codex writes no allow rules).
An agent may not widen its own permissions, so the owner adds it.
- [ ] `.gitea/workflows/ci.yml` could not be run locally. The Gitea has
an Actions runner (slow, so the workflow is kept light: 20 s per fuzz
target, a 45 minute job timeout, no `test_all`); the first push must
confirm the runner picks the workflow up.
- [ ] protondrive: `go mod tidy` would drop `github.com/rclone/Proton-API-Bridge`
from `go.mod` now (no Go file imports it; the differential tests import
only `go-proton-api`). Left in place for Task 19, which removes both.
@@ -128,9 +155,6 @@ Small, specific, independently actionable. Remove only when actually done.
- [ ] Propose a CI check for this fork: `AGENTS.md`, `CLAUDE.md`, `GEMINI.md`
and the `docs/` agent files exist, and `AGENTS.md` is at most 150 lines.
- [ ] Upstream CI (`.github/workflows/build.yml`) is gated on
`github.repository == 'rclone/rclone'` and the remote is Gitea, so this fork
has no CI at all. Decide whether to add a Gitea Actions workflow.
## Someday
+4 -4
View File
@@ -1264,13 +1264,13 @@ allocated in `PORTS.md`, echo its `key=value` lines and
> text leaves out how the script obtains `protonfake`'s output and manages
> its process, and the commands below need `RCLONE_CONFIG=/notfound`.
- [ ] **Step 1:** Add the script and config entry. Run
- [x] **Step 1:** Add the script and config entry. Run
`go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4`
and fix backend failures in `fs/operations`, `fs/sync`, `vfs` until the
report is clean. `cmd/bisync` and `cmd/gitannex`: run them; if they fail
for reasons that need M5/M6 work, add `ignoretests` for those two on the
fake remote and list the failures in `docs/TODO.md`.
- [ ] **Step 2:** `.gitea/workflows/ci.yml` on push and pull request:
- [x] **Step 2:** `.gitea/workflows/ci.yml` on push and pull request:
`actions/setup-go` with `go-version-file: go.mod`; steps `go build ./...`;
`go vet ./backend/protondrive/...`;
`golangci-lint run ./backend/protondrive/...`;
@@ -1278,10 +1278,10 @@ allocated in `PORTS.md`, echo its `key=value` lines and
`go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/...`.
It cannot be verified locally: say so in `docs/TODO.md` ("first push must
confirm the Gitea runner picks it up").
- [ ] **Step 3:** Update `docs/PERMISSIONS.md` verified-commands table for
- [x] **Step 3:** Update `docs/PERMISSIONS.md` verified-commands table for
the host toolchain (Go 1.27.4, golangci-lint 2.15.0) and add
`golangci-lint run` to the Allow tier in it and in the three tool files.
- [ ] **Step 4: Commit** `protondrive: run the wider suites against the fake server in fork CI`.
- [x] **Step 4: Commit** `protondrive: run the wider suites against the fake server in fork CI`.
### Task 19: Remove the old libraries
+15
View File
@@ -613,6 +613,21 @@ backends:
ignoretests:
- cmd/bisync
- cmd/gitannex
- backend: "protondrive"
remote: "TestProtonDriveFake:"
fastlist: false
ignore:
# Each Fs caches listings of its own, so a second Fs of the same
# account doesn't see what the first wrote - see docs/TODO.md.
- TestCheckSumDownload
- TestCopyFileCompareDest
- TestCopyFileCopyDestImmutable
- TestSyncCompareDest
- TestSyncCopyDestImmutable
ignoretests:
# cmd/gitannex checks the remote name against config.GetRemoteNames,
# which lists remotes set by environment variables lower cased.
- cmd/gitannex
- backend: "putio"
remote: "TestPutio:"
fastlist: false
+1
View File
@@ -34,6 +34,7 @@ They should be bound to localhost so they are not accessible externally.
| 28639 | TestWebdavInfiniteScale |
| 28640 | TestSia |
| 28641 | TestWebdavOwncloud |
| 28642 | TestProtonDriveFake |
## Non localhost tests
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
# Start the fake Proton Drive server (backend/protondrive/protontest).
#
# protonfake mints a signed in session when it starts and prints it as
# key=value lines on stdout, so the script builds it, captures that
# output and passes it on. Every user of the server shares the session.
set -e
NAME=TestProtonDriveFake
IP=127.0.0.1
PORT=28642
# This script lives in fstest/testserver/init.d; it is run from each
# test package's directory, so find the repository from here.
REPO_ROOT=$(cd "$(dirname "$0")/../../.." && pwd)
start() {
local bin="${RUN_STATE}/protonfake"
local out="${RUN_STATE}/protonfake.out"
# A built binary, not "go run", so the pid is the server's own.
(cd "${REPO_ROOT}" && go build -o "${bin}" ./backend/protondrive/protontest/cmd/protonfake) >&2
: > "${out}"
nohup "${bin}" -addr "${IP}:${PORT}" > "${out}" 2>> "${RUN_STATE}/protonfake.log" </dev/null &
local pid=$!
echo "${pid}" > "${RUN_STATE}/pid"
disown "${pid}"
# protonfake prints six lines, type= last, once it is listening.
local i
for i in $(seq 1 100); do
if grep -q '^type=' "${out}"; then
cat "${out}"
echo "_connect=${IP}:${PORT}"
return 0
fi
if ! kill -0 "${pid}" 2>/dev/null; then
echo "protonfake exited: $(cat "${RUN_STATE}/protonfake.log")" >&2
return 1
fi
sleep 0.1
done
echo "protonfake printed no config within 10s" >&2
stop
return 1
}
stop() {
if status; then
local pid
pid=$(cat "${RUN_STATE}/pid")
kill "${pid}"
# protonfake finishes its requests on SIGTERM; wait so the port
# is free for the next start.
local i
for i in $(seq 1 100); do
kill -0 "${pid}" 2>/dev/null || break
sleep 0.1
done
rm -f "${RUN_STATE}/pid"
echo "${NAME} stopped"
fi
}
status() {
if [ -e "${RUN_STATE}/pid" ]; then
if kill -0 "$(cat "${RUN_STATE}/pid")" >/dev/null 2>&1; then
return 0
fi
rm -f "${RUN_STATE}/pid"
fi
return 1
}
# shellcheck disable=SC1091
. "$(dirname "$0")/run.bash"