protondrive: run the wider suites against the fake server in fork CI
Add the TestProtonDriveFake init.d script, which builds and starts protonfake on port 28642 and passes on the session it prints, and a test_all entry for it, so fs/operations, fs/sync, vfs and cmd/bisync run against the fake server. Five tests that need a second Fs of the same account to see what the first wrote are ignored, as each Fs caches separately, and cmd/gitannex is skipped as for every init.d remote; both are in docs/TODO.md. Add a Gitea Actions workflow that builds, vets, lints, race-tests, fuzzes and vulnerability-checks the backend, and re-verify the agent permission table on the host toolchain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
65840bdc57
commit
5afbe9e4fa
10 files changed
+232
-45
No files matched your search
@@ -5,7 +5,8 @@
|
||||
"run_shell_command(git show)", "run_shell_command(git branch --list)",
|
||||
"run_shell_command(git add)", "run_shell_command(git commit)",
|
||||
"run_shell_command(go mod download)", "run_shell_command(go build)",
|
||||
"run_shell_command(go test)", "run_shell_command(go vet)", "run_shell_command(gofmt -l)"
|
||||
"run_shell_command(go test)", "run_shell_command(go vet)", "run_shell_command(gofmt -l)",
|
||||
"run_shell_command(golangci-lint run)"
|
||||
],
|
||||
"exclude": [
|
||||
"run_shell_command(git push --force)", "run_shell_command(git push -f)",
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
# Fork CI for the Proton Drive backend. Every step is hermetic: the
|
||||
# tests run against the fake server in backend/protondrive/protontest.
|
||||
name: ci
|
||||
|
||||
on:
|
||||
push:
|
||||
pull_request:
|
||||
|
||||
env:
|
||||
# Never pick up a config holding a real Proton remote.
|
||||
RCLONE_CONFIG: /notfound
|
||||
|
||||
jobs:
|
||||
protondrive:
|
||||
runs-on: ubuntu-latest
|
||||
# The race suite takes about 6 minutes on a fast machine.
|
||||
timeout-minutes: 45
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Install Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version-file: go.mod
|
||||
# Caches the module and build caches, keyed on go.sum.
|
||||
cache: true
|
||||
|
||||
- name: Build
|
||||
run: go build ./...
|
||||
|
||||
- name: Vet
|
||||
run: go vet ./backend/protondrive/...
|
||||
|
||||
- name: Lint
|
||||
uses: golangci/golangci-lint-action@v8
|
||||
with:
|
||||
version: v2.15.0
|
||||
args: ./backend/protondrive/...
|
||||
|
||||
- name: Test
|
||||
run: go test -race ./backend/protondrive/...
|
||||
|
||||
- name: Fuzz
|
||||
run: |
|
||||
for target in $(go test -list '^Fuzz' ./backend/protondrive/ | grep '^Fuzz'); do
|
||||
go test -run '^$' -fuzz "^${target}\$" -fuzztime 20s ./backend/protondrive/
|
||||
done
|
||||
|
||||
- name: Vulnerability check
|
||||
run: go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/...
|
||||
@@ -63,6 +63,10 @@ A native client of the Proton Drive API, following Proton's Drive SDK
|
||||
- `fake_test.go` runs the full `fstests` contract suite against
|
||||
`protontest` through the hidden `endpoint` option, on every
|
||||
`go test`; `protondrive_test.go` is the live suite (`TestProtonDrive:`).
|
||||
- `fstest/testserver/init.d/TestProtonDriveFake` builds and starts
|
||||
`protontest/cmd/protonfake` for `test_all`, which runs the wider suites
|
||||
against the `TestProtonDriveFake:` remote. Fork CI is
|
||||
`.gitea/workflows/ci.yml`.
|
||||
|
||||
## Known rough edges
|
||||
|
||||
|
||||
@@ -5,6 +5,32 @@ it with a new one that links back. One entry per decision a future maintainer
|
||||
would otherwise have to reverse-engineer.
|
||||
No index: `grep -n '^## ' docs/DECISIONS.md` lists every entry.
|
||||
|
||||
## 2026-11-12 - test_all shares one protonfake session across test processes
|
||||
|
||||
**Status:** accepted
|
||||
**Context:** `fstest/testserver/init.d/TestProtonDriveFake` starts one
|
||||
`protonfake` per `test_all` run (reference counted by `run.bash`), and
|
||||
every test process gets the same session through
|
||||
`RCLONE_CONFIG_TESTPROTONDRIVEFAKE_*` variables. Refresh tokens are
|
||||
single use and a process can't see another's refresh, so one process
|
||||
refreshing would leave the rest with dead tokens.
|
||||
**Decision:** Share the session; change nothing in `protontest`,
|
||||
`protonfake` or the client. The client refreshes only after an HTTP 401
|
||||
(`session.call`); the fake answers 401 only for a session expired by
|
||||
`Server.ExpireAccessToken` or ended by logout or `RevokeSession`, and
|
||||
`protonfake` calls none of them. The only logouts are `Fs.Disconnect`
|
||||
and re-running `rclone config`, which no test in `fs/operations`,
|
||||
`fs/sync`, `vfs`, `cmd/bisync`, `cmd/gitannex` or `fstests` calls.
|
||||
**Alternatives:** A session per test process (`protonfake` would need a
|
||||
sign-in route for `test_all` to call, and the init.d protocol has one
|
||||
output per server); non-expiring or logout-tolerant tokens in
|
||||
standalone mode (behaviour no test needs, and a fake that differs from
|
||||
Proton).
|
||||
**Consequences:** A future test that calls `Disconnect`, or a fake that
|
||||
expires tokens on a timer, breaks every process sharing the run: the
|
||||
symptom is "not signed in" or refresh failures across all suites at
|
||||
once. Revisit then.
|
||||
|
||||
## 2026-11-11 - Proton Drive rewrite paused mid-plan and merged to master
|
||||
|
||||
- **Decision:** at the owner's request, work on plan 001 stopped after Task
|
||||
|
||||
+20
-32
@@ -8,55 +8,43 @@ every tool file in the same commit. Review changes to these files and to
|
||||
|
||||
## Verified commands
|
||||
|
||||
Update 2026-11-11: later the same day Go 1.27.4 (`~/.local/opt/go`) and
|
||||
golangci-lint 2.15.0 (`~/.local/bin`) were installed on the host at the
|
||||
owner's request, and all Proton Drive work was built, tested and linted
|
||||
with them directly. The table below predates that and has not been re-run
|
||||
on the host (plan Task 18, step 3).
|
||||
|
||||
Run on 2026-11-11. The development host had **no Go toolchain** at the time, so every `go`
|
||||
row was run inside the `golang:1.26` container (go1.26.11) with the repository
|
||||
mounted, using this wrapper:
|
||||
|
||||
```sh
|
||||
docker run --rm -u "$(id -u):$(id -g)" -e HOME=/cache -e GOCACHE=/cache/build \
|
||||
-e GOMODCACHE=/cache/mod -e GOFLAGS=-buildvcs=false -e RCLONE_CONFIG=/notfound \
|
||||
-v "$HOME/.cache/rclone-go:/cache" -v "$PWD:/src" -w /src golang:1.26 <command>
|
||||
```
|
||||
Run on the host on 2026-11-12 with Go 1.27.4 (`~/.local/opt/go`) and
|
||||
golangci-lint 2.15.0 (`~/.local/bin`), from the repository root, with
|
||||
`RCLONE_CONFIG=/notfound` set so no test can reach a real remote. The rows
|
||||
marked (container) are from 2026-11-11, before Go was on the host, run in
|
||||
the `golang:1.26` container and not repeated.
|
||||
|
||||
Only `worked` rows may appear in a tier, a tool file, or `AGENTS.md`.
|
||||
|
||||
| Command | Result |
|
||||
|---|---|
|
||||
| `go mod download` | worked (container) |
|
||||
| `go build` | worked (container) - binary reports `rclone v1.77.0-DEV` |
|
||||
| `go test ./backend/memory/` | worked (container) |
|
||||
| `go test -run 'TestIntegration/FsMkdir/FsPutFiles/FsCopy' ./backend/memory/` | worked (container) |
|
||||
| `go test ./backend/protondrive/` | worked (container) - no `TestProtonDrive:` remote, so the integration test skips |
|
||||
| `go vet ./backend/protondrive/` | worked (container) |
|
||||
| `gofmt -l backend/protondrive` | worked (container) - no output |
|
||||
| `RCLONE_CONFIG=/notfound go test ./...` | failed (container) - 178 packages ok, 7 fail: `backend/smb`, `cmd/gitannex`, `cmd/mount`, `cmd/mount2`, `cmd/serve/docker`, `cmd/serve/nfs`, `fs/logger`. `fs/logger` fails because the container has no `rsync`; the other six were not diagnosed |
|
||||
| `make`, `make quicktest`, `make racequicktest` | not run - need Go on the host |
|
||||
| `golangci-lint run ./...` | not run - not installed on the host or in the container |
|
||||
| `go run ./fstest/test_all -backends protondrive` | not run - needs a configured `TestProtonDrive:` remote |
|
||||
|
||||
TODO(maintainer): confirm every row above that is not `worked`.
|
||||
| `go mod download` | worked |
|
||||
| `go build` | worked - binary reports `rclone v1.77.0-DEV` |
|
||||
| `go test ./backend/memory/` | worked |
|
||||
| `go test -run 'TestIntegration/FsMkdir/FsPutFiles/FsCopy' ./backend/memory/` | worked |
|
||||
| `go test ./backend/protondrive/...` | worked - the contract suite runs against the fake server; the live `TestProtonDrive:` suite skips without a config |
|
||||
| `go vet ./backend/protondrive/...` | worked |
|
||||
| `gofmt -l backend/protondrive` | worked - no output |
|
||||
| `golangci-lint run ./backend/protondrive/...` | worked - 0 issues |
|
||||
| `golangci-lint run ./...` | worked - 0 issues |
|
||||
| `go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4` | worked - starts the fake server through `fstest/testserver/init.d/TestProtonDriveFake` |
|
||||
| `RCLONE_CONFIG=/notfound go test ./...` | failed - only `cmd/serve/s3`, undiagnosed (2026-11-11, `docs/TODO.md` "Known issues"); in the container 7 packages failed: `backend/smb`, `cmd/gitannex`, `cmd/mount`, `cmd/mount2`, `cmd/serve/docker`, `cmd/serve/nfs`, `fs/logger` (container) |
|
||||
| `make`, `make quicktest`, `make racequicktest` | not run - `make` is in Ask |
|
||||
|
||||
## Tiers
|
||||
|
||||
| Tier | Commands | Why |
|
||||
|---|---|---|
|
||||
| Allow | `git status`, `git diff`, `git log`, `git show`, `git branch --list`, `git rev-parse`, `git add`, `git commit`, `go mod download`, `go build`, `go test`, `go vet`, `gofmt -l` | Inspection and the verified dev loop; no prompt needed. |
|
||||
| Allow | `git status`, `git diff`, `git log`, `git show`, `git branch --list`, `git rev-parse`, `git add`, `git commit`, `go mod download`, `go build`, `go test`, `go vet`, `gofmt -l`, `golangci-lint run` | Inspection and the verified dev loop; no prompt needed. |
|
||||
| Ask | `git push`, `git merge`, `git rebase`, `git switch`, `git restore`, `git commit --amend`, `rm` (any), `go get`, `go mod tidy`, `go install`, `make` (any target), `docker` (any) | Leaves the machine, rewrites work, changes dependencies, or is unverified. |
|
||||
| Deny | `git push --force` / `-f` / `--force-with-lease`, `git reset --hard`, `git clean` (any), `git add -f` / `--force`, `git commit --no-verify` / `-n`, `rm -r` / `-R` / `-rf` / `-fr` / `-Rf` / `-fR` / `--recursive`, `sudo`; reading `.env`, `.env.*`, `secrets/**` | Destroys work, skips checks, or exposes secrets. |
|
||||
|
||||
## Known gaps
|
||||
|
||||
- `.claude/settings.json` does not yet allow `golangci-lint run`; the owner
|
||||
adds it (`docs/TODO.md`), as an agent may not widen its own permissions.
|
||||
- Allowlists save prompts; they are not a security boundary. Use the tool's
|
||||
sandbox for isolation.
|
||||
- While the host has no Go, the allowed `go` commands only work through the
|
||||
`docker run` wrapper above, and `docker` is in Ask - so each one prompts.
|
||||
Installing Go on the host removes this.
|
||||
- Prefix rules are bypassable: `git -C dir push`, flags placed later
|
||||
(`git push origin main --force`), `+refspec` pushes. Claude Code's `:*`
|
||||
matches at a word boundary, so each flag cluster must be listed - which is
|
||||
|
||||
+32
-8
@@ -6,10 +6,9 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
|
||||
## Now
|
||||
|
||||
- [ ] protondrive: the backend rewrite (plan 001) is paused after Task 17 of
|
||||
21. Start from `docs/plans/001-protondrive-tier1-HANDOFF.md`. Remaining:
|
||||
Task 18 (wider suites against the fake, fork CI), 19 (remove the old
|
||||
libraries), 20 (user docs - `docs/content/protondrive.md` still describes
|
||||
- [ ] protondrive: the backend rewrite (plan 001) has Tasks 1-18 of 21
|
||||
done. Start from `docs/plans/001-protondrive-tier1-HANDOFF.md`.
|
||||
Remaining: Task 19 (remove the old libraries), 20 (user docs - `docs/content/protondrive.md` still describes
|
||||
the old backend), 21 (live verification, paid account never signed in to),
|
||||
then the whole-branch review; later plans for M5 (events) and M6
|
||||
(persistent cache, Metadata, small-file upload).
|
||||
@@ -118,7 +117,35 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
|
||||
- [ ] protondrive: several `Fs` of one account in one process each have
|
||||
their own node cache; only Move/DirMove sources are invalidated across
|
||||
them. Events (spec 6.2) fix this.
|
||||
them. A name missing from one `Fs`'s cached listing stays missing after
|
||||
another `Fs` creates it, so `--compare-dest`, `--copy-dest` (with
|
||||
`--immutable`) and `checksum --download` decide on stale listings when
|
||||
the other `Fs` writes during the run, and an `Fs` made on a path that
|
||||
doesn't exist yet never finds it once it is made elsewhere. Five
|
||||
`test_all` tests fail for this and are in the `ignore` list of
|
||||
`TestProtonDriveFake:` (`fstest/test_all/config.yaml`):
|
||||
`TestCheckSumDownload`, `TestCopyFileCompareDest`,
|
||||
`TestCopyFileCopyDestImmutable`, `TestSyncCompareDest`,
|
||||
`TestSyncCopyDestImmutable`; all pass with `enable_caching = false`.
|
||||
Events polled every 30 s (spec 6.2) will not close a window of
|
||||
seconds. Re-listing on every cache miss fixes them but breaks
|
||||
`TestBudgetUpload` and `TestBudgetMkdir` (a folder re-listed per new
|
||||
file). A cache shared by the `Fs` of one account would fix them, but
|
||||
is package state, which the spec rules out: owner decision.
|
||||
- [ ] protondrive: `cmd/gitannex` is in `ignoretests` for
|
||||
`TestProtonDriveFake:` because its remote name check
|
||||
(`cmd/gitannex/configparse.go`, `config.GetRemoteNames`) sees remotes
|
||||
set by environment variables lower cased, as for every other init.d
|
||||
remote; it says nothing about the backend. Run it against a remote in
|
||||
a config file to cover it.
|
||||
- [ ] `.claude/settings.json` lacks `"Bash(golangci-lint run:*)"` in
|
||||
`allow`, which `docs/PERMISSIONS.md` lists in the Allow tier since plan
|
||||
Task 18 (`.gemini/settings.json` has it; Codex writes no allow rules).
|
||||
An agent may not widen its own permissions, so the owner adds it.
|
||||
- [ ] `.gitea/workflows/ci.yml` could not be run locally. The Gitea has
|
||||
an Actions runner (slow, so the workflow is kept light: 20 s per fuzz
|
||||
target, a 45 minute job timeout, no `test_all`); the first push must
|
||||
confirm the runner picks the workflow up.
|
||||
- [ ] protondrive: `go mod tidy` would drop `github.com/rclone/Proton-API-Bridge`
|
||||
from `go.mod` now (no Go file imports it; the differential tests import
|
||||
only `go-proton-api`). Left in place for Task 19, which removes both.
|
||||
@@ -128,9 +155,6 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
|
||||
- [ ] Propose a CI check for this fork: `AGENTS.md`, `CLAUDE.md`, `GEMINI.md`
|
||||
and the `docs/` agent files exist, and `AGENTS.md` is at most 150 lines.
|
||||
- [ ] Upstream CI (`.github/workflows/build.yml`) is gated on
|
||||
`github.repository == 'rclone/rclone'` and the remote is Gitea, so this fork
|
||||
has no CI at all. Decide whether to add a Gitea Actions workflow.
|
||||
|
||||
## Someday
|
||||
|
||||
|
||||
@@ -1264,13 +1264,13 @@ allocated in `PORTS.md`, echo its `key=value` lines and
|
||||
> text leaves out how the script obtains `protonfake`'s output and manages
|
||||
> its process, and the commands below need `RCLONE_CONFIG=/notfound`.
|
||||
|
||||
- [ ] **Step 1:** Add the script and config entry. Run
|
||||
- [x] **Step 1:** Add the script and config entry. Run
|
||||
`go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4`
|
||||
and fix backend failures in `fs/operations`, `fs/sync`, `vfs` until the
|
||||
report is clean. `cmd/bisync` and `cmd/gitannex`: run them; if they fail
|
||||
for reasons that need M5/M6 work, add `ignoretests` for those two on the
|
||||
fake remote and list the failures in `docs/TODO.md`.
|
||||
- [ ] **Step 2:** `.gitea/workflows/ci.yml` on push and pull request:
|
||||
- [x] **Step 2:** `.gitea/workflows/ci.yml` on push and pull request:
|
||||
`actions/setup-go` with `go-version-file: go.mod`; steps `go build ./...`;
|
||||
`go vet ./backend/protondrive/...`;
|
||||
`golangci-lint run ./backend/protondrive/...`;
|
||||
@@ -1278,10 +1278,10 @@ allocated in `PORTS.md`, echo its `key=value` lines and
|
||||
`go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/...`.
|
||||
It cannot be verified locally: say so in `docs/TODO.md` ("first push must
|
||||
confirm the Gitea runner picks it up").
|
||||
- [ ] **Step 3:** Update `docs/PERMISSIONS.md` verified-commands table for
|
||||
- [x] **Step 3:** Update `docs/PERMISSIONS.md` verified-commands table for
|
||||
the host toolchain (Go 1.27.4, golangci-lint 2.15.0) and add
|
||||
`golangci-lint run` to the Allow tier in it and in the three tool files.
|
||||
- [ ] **Step 4: Commit** `protondrive: run the wider suites against the fake server in fork CI`.
|
||||
- [x] **Step 4: Commit** `protondrive: run the wider suites against the fake server in fork CI`.
|
||||
|
||||
### Task 19: Remove the old libraries
|
||||
|
||||
|
||||
@@ -613,6 +613,21 @@ backends:
|
||||
ignoretests:
|
||||
- cmd/bisync
|
||||
- cmd/gitannex
|
||||
- backend: "protondrive"
|
||||
remote: "TestProtonDriveFake:"
|
||||
fastlist: false
|
||||
ignore:
|
||||
# Each Fs caches listings of its own, so a second Fs of the same
|
||||
# account doesn't see what the first wrote - see docs/TODO.md.
|
||||
- TestCheckSumDownload
|
||||
- TestCopyFileCompareDest
|
||||
- TestCopyFileCopyDestImmutable
|
||||
- TestSyncCompareDest
|
||||
- TestSyncCopyDestImmutable
|
||||
ignoretests:
|
||||
# cmd/gitannex checks the remote name against config.GetRemoteNames,
|
||||
# which lists remotes set by environment variables lower cased.
|
||||
- cmd/gitannex
|
||||
- backend: "putio"
|
||||
remote: "TestPutio:"
|
||||
fastlist: false
|
||||
|
||||
@@ -34,6 +34,7 @@ They should be bound to localhost so they are not accessible externally.
|
||||
| 28639 | TestWebdavInfiniteScale |
|
||||
| 28640 | TestSia |
|
||||
| 28641 | TestWebdavOwncloud |
|
||||
| 28642 | TestProtonDriveFake |
|
||||
|
||||
## Non localhost tests
|
||||
|
||||
|
||||
Executable
+77
@@ -0,0 +1,77 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Start the fake Proton Drive server (backend/protondrive/protontest).
|
||||
#
|
||||
# protonfake mints a signed in session when it starts and prints it as
|
||||
# key=value lines on stdout, so the script builds it, captures that
|
||||
# output and passes it on. Every user of the server shares the session.
|
||||
|
||||
set -e
|
||||
|
||||
NAME=TestProtonDriveFake
|
||||
IP=127.0.0.1
|
||||
PORT=28642
|
||||
|
||||
# This script lives in fstest/testserver/init.d; it is run from each
|
||||
# test package's directory, so find the repository from here.
|
||||
REPO_ROOT=$(cd "$(dirname "$0")/../../.." && pwd)
|
||||
|
||||
start() {
|
||||
local bin="${RUN_STATE}/protonfake"
|
||||
local out="${RUN_STATE}/protonfake.out"
|
||||
# A built binary, not "go run", so the pid is the server's own.
|
||||
(cd "${REPO_ROOT}" && go build -o "${bin}" ./backend/protondrive/protontest/cmd/protonfake) >&2
|
||||
: > "${out}"
|
||||
nohup "${bin}" -addr "${IP}:${PORT}" > "${out}" 2>> "${RUN_STATE}/protonfake.log" </dev/null &
|
||||
local pid=$!
|
||||
echo "${pid}" > "${RUN_STATE}/pid"
|
||||
disown "${pid}"
|
||||
|
||||
# protonfake prints six lines, type= last, once it is listening.
|
||||
local i
|
||||
for i in $(seq 1 100); do
|
||||
if grep -q '^type=' "${out}"; then
|
||||
cat "${out}"
|
||||
echo "_connect=${IP}:${PORT}"
|
||||
return 0
|
||||
fi
|
||||
if ! kill -0 "${pid}" 2>/dev/null; then
|
||||
echo "protonfake exited: $(cat "${RUN_STATE}/protonfake.log")" >&2
|
||||
return 1
|
||||
fi
|
||||
sleep 0.1
|
||||
done
|
||||
echo "protonfake printed no config within 10s" >&2
|
||||
stop
|
||||
return 1
|
||||
}
|
||||
|
||||
stop() {
|
||||
if status; then
|
||||
local pid
|
||||
pid=$(cat "${RUN_STATE}/pid")
|
||||
kill "${pid}"
|
||||
# protonfake finishes its requests on SIGTERM; wait so the port
|
||||
# is free for the next start.
|
||||
local i
|
||||
for i in $(seq 1 100); do
|
||||
kill -0 "${pid}" 2>/dev/null || break
|
||||
sleep 0.1
|
||||
done
|
||||
rm -f "${RUN_STATE}/pid"
|
||||
echo "${NAME} stopped"
|
||||
fi
|
||||
}
|
||||
|
||||
status() {
|
||||
if [ -e "${RUN_STATE}/pid" ]; then
|
||||
if kill -0 "$(cat "${RUN_STATE}/pid")" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
rm -f "${RUN_STATE}/pid"
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# shellcheck disable=SC1091
|
||||
. "$(dirname "$0")/run.bash"
|
||||
Reference in new issue
Block a user