rclone/docs/plans/001-protondrive-tier1-HANDOFF.md

Handoff: Proton Drive backend rewrite (plan 001)

Written 2026-11-11 when development was paused at the owner's request. Everything a new agent needs is in this repository; nothing depends on the conversation that produced it. Fork-only document - never include it in a PR to rclone/rclone.

1. What this is

The owner's fork of rclone is rewriting backend/protondrive as a native Go client for Proton Drive's current API (volume-based drive/v2 routes), replacing the abandoned Proton-API-Bridge / go-proton-api libraries. The goal is a working, maintainable, secure backend that rclone's maintainers could one day accept.

Read, in this order:

1. AGENTS.md - code, comment and commit rules; the "Fork Workflow" section.
2. docs/specs/001-protondrive-tier1.md - the approved spec (binding).
3. docs/plans/001-protondrive-tier1.md - the 21-task plan. Checkboxes show what is done.
4. This file - state, rulings, and how to resume.
5. docs/plans/001-protondrive-tier1-workspace/progress.md - the full execution ledger (every ruling, every deferred minor, every live probe).
6. docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/TODO.md.

2. Owner's standing instructions

• Nothing is posted upstream (no issue, PR or forum post) for now. The upstream path in spec section 13 is deferred.
• The owner is not planning permanent maintenance of the backend.
• The fake server (backend/protondrive/protontest) stays in the fork even though it is about a third of the code; it would not go upstream.
• Implementers are Opus 5.5 subagents, one task at a time; each task gets an independent review (Opus for auth, crypto, concurrency and the swap; Sonnet for small diffs and scoped re-reviews).
• "Once I approve a design, carry it through spec, plan and execution with no extra review stops. Report the decisions you made at the end."
• Permission rules: docs/PERMISSIONS.md (no sudo, no force-push, no git reset --hard, no git clean, no recursive rm). Ask before adding or upgrading a dependency.
• The push of this work to master on 2026-11-11 was an explicit one-off instruction from the owner. The standing rule in AGENTS.md (never push or commit directly to master) still applies to future work: branch first and ask before pushing.

3. State at pause

Branch protondrive-tier1, fast-forwarded into master and pushed at the pause. Last code commit: 9c90998f7. Plan base (for a whole-branch diff): 06700aaf2.

Tasks	State
1-18	Implemented, each independently reviewed, all findings of Critical or Important severity fixed and re-reviewed
19-21	Not started (resumed 2026-11-12; see the ledger for current state)
Final whole-branch review	Not run

What exists now:

• backend/protondrive/protondrive.go is rewritten on the native client (session.go, account.go, crypto.go, nodes.go, cache.go, download.go, upload.go). The old bridge libraries left go.mod in Task 19, with the differential tests that used them.
• backend/protondrive/protontest is the fake Proton server (fork-only), with protontest/cmd/protonfake as a standalone binary.
• rclone's backend contract suite (fstests.Run) passes against the fake as TestIntegrationFake: 126 subtests, 97 pass, 29 skipped by the suite's own feature checks, no opt-outs.
• Test files: *_internal_test.go per client file, fake_test.go (harness), regression_test.go (16 user-reported failures), security_test.go (27 hostile-server cases, secret hygiene), budget_test.go (request budgets), fuzz_internal_test.go (5 targets), frozen old-library fixtures in testdata/oldlib/ (TestFrozenFixtures, TestFrozenFileFixture).
• Verified at 9c90998f7 on 2026-11-11: go build ./..., go vet ./backend/protondrive/..., golangci-lint run ./backend/protondrive/... (0 issues) and RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/... all pass. The race suite takes about 5 to 6 minutes.

Not done, and easy to assume otherwise:

• The user documentation (docs/content/protondrive.md) still describes the old backend (Task 20).
• The wider suites (fs/operations, fs/sync, vfs via fstest/test_all) have never been run against the new backend (Task 18).
• No fork CI exists (Task 18).
• ChangeNotify, the persistent cache, Metadata and the small-file upload path are not implemented (M5, M6).
• Ranged reads (mount, --multi-thread-streams, serve) check each block's hash but not the signed manifest (docs/TODO.md).

4. How to resume

Development used the superpowers:subagent-driven-development skill (obra's superpowers plugin, version 7.0.0; install with /plugin marketplace add obra/superpowers-marketplace then /plugin install superpowers@superpowers-marketplace). The skill is a convenience, not a requirement: the per-task loop below can be followed by hand. Its workspace lives at .superpowers/sdd/001-protondrive-tier1/, which .gitignore excludes; a copy of every Markdown file from it (ledger, per-task briefs, implementer reports, reviews, the shared implementer/reviewer rules and the global constraints) is committed at docs/plans/001-protondrive-tier1-workspace/. To resume with the skill:

mkdir -p .superpowers/sdd/001-protondrive-tier1
cp docs/plans/001-protondrive-tier1-workspace/*.md .superpowers/sdd/001-protondrive-tier1/
git switch -c protondrive-tier1-resume   # never work on master

The ledger's first line names the plan; tasks with a Task N: complete line are done. Paths inside implementer-common.md and reviewer-common.md assume the checkout is /home/user/rclone and the branch protondrive-tier1 - adjust both if they differ. One rule in implementer-common.md is obsolete: "the old protondrive.go must keep compiling until Task 14" (Task 14 is done; there is no old implementation left in the package).

Per task: extract the brief (task-N-brief.md holds the plan text for task N; later ones are not extracted yet - copy the task's section from the plan), record BASE=$(git rev-parse HEAD), dispatch one implementer with the brief plus implementer-common.md, then a reviewer with the brief, the report and git diff -U10 BASE..HEAD, fix until clean, and record it in the ledger. After Task 21: one whole-branch review on the most capable model (merge base 06700aaf2), one fix wave, then git rm the plan and this workspace copy before any upstream-bound merge.

Remaining tasks (plan sections of the same number)

• 18 - Wider hermetic suites and fork CI. init.d script starting protonfake, test_all config entry, .gitea/workflows/ci.yml, docs/PERMISSIONS.md update. Expect real backend failures from fs/operations, fs/sync and vfs; fix them in the backend. Note -run TestIntegration matches both the live and the fake test.
• 19 - Remove the old libraries. Move the frozen-fixture tests, delete the differential test file, go mod tidy, compare binary size, cross-compile.
• 20 - Documentation. Rewrite docs/content/protondrive.md; write docs/specs/001-conformance-checklist.md. Must cover what this pause leaves undocumented for users: the ignore_signature_errors behaviour and its consequences, the ranged-read caveat, --dump headers showing pm-storage-token and x-pm-uid, the whole-revision rollback limit.
• 21 - Live verification. The plan text says it is blocked until a test account exists and that the owner configures it by browser sign-in. That is out of date: both accounts exist and are configured with username and password (section 5). First things to check live: the root-of-trust rule on a drive created by Proton's own apps (the paid account); how many folders fail key verification on a real account; whether the server lists drafts and trashed items as children; browser sign-in; TestIntegration and test_all against the real service.

Notes for Tasks 18 to 21

The plan text for these tasks predates the work that changed things under it. Known corrections, found by having a fresh reader walk the plan against the code:

Task 18

• Every command needs RCLONE_CONFIG=/notfound except where it must read the TestProtonDriveFake: definition - test_all gets its remote from the init.d script's output, not from the config file, so RCLONE_CONFIG=/notfound go run ./fstest/test_all ... is right. Check that no step can reach the live TestProtonDrive: remote.
• -run TestIntegration matches both the live TestIntegration and TestIntegrationFake. TestIntegrationFake skips itself when -remote is given; keep that guard.
• The init.d script cannot simply copy TestWebdavRclone. protonfake prints freshly minted credentials on stdout (endpoint=, client_uid=, client_access_token=, client_refresh_token=, client_salted_key_pass=, type=protondrive), while the run helper in rclone-serve.bash redirects stdout to a log file. The script has to start the binary, wait for the six lines, and echo them. Use a built binary, not go run (killing the go parent leaves the server running), and find the repository root rather than assuming the working directory, because scripts are run from each test package's directory. Nobody has designed this yet.
• No port has been allocated in PORTS.md; pick the next free one there.
• All test processes started by test_all would share one fake session through those values. Refresh tokens are single-use, and a process whose credentials come from the environment cannot see another process's refresh (section 8, "two processes sharing tokens through environment variables"). If anything makes a process refresh - check whether the standalone fake ever expires an access token, and what tests that call Disconnect do to the shared session - the others would be left with dead tokens. Check this before writing the script; if it is a problem, have protonfake mint a session per run or issue non-expiring tokens in standalone mode.
• Whether the owner's Gitea has an Actions runner is unknown. Ask the owner; the workflow cannot be verified locally either way.
• Expect real failures from fs/operations, fs/sync and vfs: they have never been run against this backend.

Task 19

• Step 1 says to build master as the "old" binary. master now contains the new backend. Build the plan's base commit 06700aaf2 instead (it has the old backend and the same rclone version).
• Step 2: TestFrozenFileFixture is already in download_internal_test.go with no old-library import. TestFrozenFixtures still depends on helpers inside crypto_diff_internal_test.go and on an -update regeneration path. The accurate instructions are the docs/TODO.md entry beginning "when go-proton-api and Proton-API-Bridge leave go.mod", which also records that multi-block coverage of old-library files is lost at that point.
• go mod tidy is part of the approved plan for this task, so it does not need asking again, but accept only the removal of the two old libraries and what they alone pulled in (and relvacode/iso8601 returning to indirect or leaving). Anything else: stop and ask.
• "No new failures in go test ./...": the baseline on the host is in docs/TODO.md under "Known issues" (only cmd/serve/s3 fails).

Task 20

• Must also document what was decided after the plan was written: section 7 rulings 18 to 26 as user-visible behaviour, the ranged-read caveat, the --dump headers exposure, and both protocol limits.

Task 21

• The preconditions in the plan are out of date (see the note there and section 5). The accounts use password sign-in.
• "Interop with Proton's official CLI": the reference SDK checkout must never be built or run (it is untrusted reference data), and no official CLI is installed. Ask the owner whether to install a released Proton client for this, or to limit interop to the web app as done so far.
• Golden-fixture capture stores real account material; captured nodes must come only from the throwaway test accounts and be reviewed for anything identifying before they are committed.
• Do not delete rclone-live-probe-20261111-130916 on the free account.

Later milestones, each needing its own plan: M5 (events and ChangeNotify), M6 (persistent metadata cache, rclone Metadata support, small-file single-request upload), M7 (upstream path, deferred by the owner).

Open questions for the owner

Answered by the owner on 2026-11-12:

1. Usernames in git history: remove them. Done: commit 901d7fd5b was rewritten without the usernames (as 31f6aafd9), the later commits replayed unchanged, and master on Gitea force-pushed once (85f6b2f0e -> 4282540e0). A one-off; the no-force-push rule stands. Old hashes in this file and the ledger written before that date (901d7fd5b, 85f6b2f0e, d211b45e6, 28f1aaf8a) now map to 31f6aafd9, 4282540e0, 65840bdc5, 5afbe9e4f.
2. Gitea Actions: a runner exists but is slow and limited; keep .gitea/workflows/ci.yml light (no test_all in CI).
3. Official client interop (Task 21): allowed - install a released Proton client and use the web app, preferably inside a Docker container, signed in only to the test accounts.

Still open:

4. Rulings. Section 7 lists 28 decisions made on the owner's behalf; they were reported in summary and not confirmed one by one.
5. Cross-Fs cache staleness (Task 18). Five test_all tests are ignored because each Fs has its own node cache; a mount never sees other clients' changes. Fix options (shared per-account cache = package state the spec forbids; listing TTL; events in M5) are in docs/TODO.md. Current ruling: leave for M5.

5. Test accounts and machine setup

Machine-local, not in the repository. A new machine needs them recreated.

• Go 1.27.4 is in ~/.local/opt/go (symlinked into ~/.local/bin), golangci-lint 2.15.0 in ~/.local/bin.
• Two Proton accounts the owner created for testing, configured as rclone remotes in ~/.config/rclone/rclone.conf (mode 600, passwords obscured):
  • TestProtonDrive: - free account; password file ~/proton_rclone_test_acc_password.
  • TestProtonDrivePaid: - paid account; password file ~/proton_paid_acc_password. Never signed in to yet. Where to find the credentials on this machine (never copy either value into the repository, a commit message, a log or a subagent prompt):
  • Username: the username key of each remote in ~/.config/rclone/rclone.conf; rclone config show TestProtonDrive and rclone config show TestProtonDrivePaid print it. That output also contains the obscured password and, for the free account, session tokens, so do not paste it anywhere.
  • Password: plain text in the two files named above, one per account. The same password is stored obscured in the rclone config, so rclone itself needs nothing more; read the files only if a test must supply the password directly, and then pass it by file or environment, never on a command line that gets logged. Both remotes are set up for password sign-in, which is what every live probe used; browser sign-in has not been exercised against the real service. On a different machine, ask the owner for the accounts.
• Every go test must set RCLONE_CONFIG=/notfound, e.g. RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/.... Without it the live TestIntegration signs in to the real service on every run. Only Task 21 talks to Proton deliberately.
• Live probes were run from a scratch git worktree so nothing temporary could be committed. Proton's guidelines for third-party clients apply: honest x-pm-appversion (external-drive-rclone@<semver>-<channel>), official endpoints only, no hammering.
• The reference checkout of Proton's official SDK used during development was ~/.cache/protondrive-sdk-ref (github.com/ProtonDriveApps/sdk at commit 05c9115, tag js/v0.23.0). Read-only; never built or run.
• Live state on the free account: folder rclone-live-probe-20261111-130916 (files empty.bin, five-mib.bin, hello.txt with two revisions, and my-little-folder/Readme.txt made by the owner in Proton's web app). Leave it: it is the interop evidence.

6. What has been proven live (free account)

• Password (SRP) sign-in with the new code; a fresh account's Drive volume is created by the client.
• Uploads of 0 B, 25 B, 5 MiB+3 and 9 MB files and a second revision are accepted by Proton and read back byte-identical with signature and manifest verification.
• Proton's web app lists the files, opens them, and shows "Digital signature verified", the right size, modification time and SHA1.
• A folder and a file created in Proton's web app are listed, downloaded and verified by the new code.
• With the rewritten backend (binary built at c00162ed2): lsd, lsl, cat, sha1sum, about, sync up, check (0 differences), new revision, server-side moveto and copyto, sync back byte-identical, rmdir of a non-empty folder refused, purge.

Not yet proven: the paid account; browser (session-fork) sign-in against the real service; 2FA; the official CLI or desktop clients reading rclone's files; fstests/test_all against the real service; whether the real server lists drafts or trashed items as children (affects Rmdir); the items marked "confirm on a real account" in docs/TODO.md.

The security changes made after c00162ed2 were re-checked live on the free account without ignore_signature_errors: at e7e43095d (root of trust, subtree distrust, manifest-first reads: listing, a verified full read of the web-app file, an upload, a Range: bytes=0- read, purge) and at 9c90998f7, the last code commit (sync up, check with 0 differences, a 4-stream multi-thread download byte-identical, verified read of the web-app file, server-side move, purge). That drive's volume was created by this client, so the root-of-trust rule is still unchecked on a drive made by Proton's own apps. The hostile cases themselves can only be exercised against the fake.

7. Rulings made during execution

Decisions the controller made where the plan or spec was silent or wrong. Where this list and the plan or spec disagree, this list is what the code does. Most are in the ledger as Ruling: lines with their reason and what it costs if wrong; 16, 17 and 24 to 28 came out of review fix rounds and their reasoning is in task-14-review.md, task-16-review.md, task-16-rereview.md, task-16-rereview2.md and task-17-review.md in the workspace copy, and in docs/DECISIONS.md. The owner was told of them in summary but has not confirmed them one by one.

1. No separate worktree: work happened on branch protondrive-tier1 in the main checkout.
2. Task 8's retry test is TestClientShouldRetry (name collision).
3. The fake keeps Proton's published app-version grammar (channel optional); the client always sends a channel.
4. Two sessions sharing one config: after a rejected refresh the session re-reads the credential store for up to 3 s to adopt the other's new tokens. No cross-Fs lock (the spec allows no package state beyond unlockMu).
5. The key salt is chosen by the primary user key's ID (old library behaviour), first entry as fallback.
6. A fresh account has no Drive volume (my-files answers 422 code 2501): the client creates it, as the SDK does.
7. Written content follows the SDK (v4 node key advertising AEAD, v6 PKESK content key, SEIPDv2 blocks) even where the old library's comments disagree; confirmed accepted by the real server and web app.
8. The intended-recipient check on signatures is kept (stricter than the SDK may be).
9. The fake rejects AEAD-encrypted names, passphrases, hash keys and extended attributes written by the client.
10. An honoured Retry-After is capped at 60 s.
11. Upload commit outcome unknown: delete nothing, return errUploadOutcomeUnknown. A draft is deleted only on positive proof it never committed.
12. The fake reports a draft's owner in conflict details but does not gate routes by ClientUID.
13. Proton-API-Bridge stays in go.mod until Task 19 even though no Go file imports it any more.
14. In rclone config, browser sign-in is the default only when username and password are not both set.
15. Shutdown leaves the Fs usable (it only drops the node cache); keys are not cleared. This supersedes the plan's "Shutdown calls client.close() once". Disconnect logs out; later use returns a clear "not signed in" error naming the remote.
16. Rmdir asks the server, uncached, whether the folder has any child at all and refuses if so.
17. rclone config on a remote with a saved session asks before signing in again (default no) and logs the old session out when it does.
18. A file whose node key is not signed by its owner is refused unless ignore_signature_errors is set. A key verified against an unverified folder key is itself unverified, all the way down; creating, uploading, renaming, moving or copying into such a folder is refused without the flag. The flag never disables integrity checks.
19. A full read fetches the whole block list and verifies the signed manifest before releasing any byte. Ranged reads do not verify the manifest yet (docs/TODO.md). First-byte latency on very large files grows by about one request per 80 MiB. A re-fetched block list that changes a hash is an integrity error.
20. TestBudgetNewObject allows 8 requests, not the plan's 7 (four folder levels at two requests each; Proton has no lookup-by-name route).
21. Every node beneath a folder whose key is unverified is unverified, reads included, even when signed by the user's own address (the implementer went beyond the ruling; a reviewer judged it justified and it was kept). A genuine folder signed by a since-deleted address locks its subtree behind ignore_signature_errors.
22. Moving, copying or renaming a node whose key is unverified is refused without the flag. With the flag it proceeds, logs an ERROR once per node and operation, and the node is trusted from then on.
23. Root of trust: the My Files share's creator and the root folder's signer must be one of the account's own addresses, else the whole drive is unverified. Addresses are compared after lowercasing ASCII letters only; a match is always verified with the account's own keys and never looked up on the server.
24. A file may list at most 2^19 blocks (2 TiB at 4 MiB). This bounds the count, not the bytes: about 250-350 MiB of block metadata at the cap.
25. Parent chains deeper than 2048, or cyclic, are an integrity error.
26. A range covering the whole file takes the verified full-read path and returns only the bytes asked for.
27. A folder listing is cached only if the folder's children did not change while it was being read (per-folder change count).
28. Names that are not valid UTF-8 are refused on write.

Known protocol limits, documented rather than fixed: a server that serves a whole older, validly signed revision under the current revision ID cannot be detected (Proton's SDK cannot either); below the root, something signed under another user's email is verified against public keys the server itself supplies.

8. Deferred findings

Every review's minor findings were deferred to the final whole-branch review. They are listed per task in the ledger (progress.md, lines beginning Task N: minor (deferred) and "Deferred minors"). Only Tasks 14, 16 and 17 have review files in the workspace copy with file and line; for Tasks 1 to 13 and 15 the reviewer's text was not saved and the ledger's one-line entry (plus the implementer's report for that task) is all there is, so expect to re-locate those by reading the code. The whole-branch review will find most of them again. Items that need a decision or a live check are in docs/TODO.md. The ones most worth a new agent's attention:

• Ranged reads do not verify the signed manifest. Planned fix: a bounded per-revision cache of verified block hashes.
• With ignore_signature_errors set, Update of an unverified file logs "reading", not "updating" (upload.go, download.go contentKey).
• The depth-limit error is re-wrapped at every level and reaches about 88 KB; a hostile server can flood the log with it (nodes.go).
• TestListingRacingChange can hang until the package timeout if it fails before releasing its held request; protonfake's main_test.go leaks its run goroutine on a failed assertion.
• TestUploadShortSource failed its goroutine-leak check once under heavy parallel load and never again; TestListCancel uses the same goroutine-count approach.
• ModTime is returned with millisecond precision while Precision() says one second.
• Move, Copy and DirMove between two different accounts are not refused with fs.ErrorCant* (rclone only calls them for the same remote name).
• apiCall retries non-idempotent mutations after 5xx or network errors.
• An extended-attribute "Size": null is read as size 0.
• API responses have no body-size limit; API error messages are neither bounded nor sanitised before logging.
• Two processes sharing tokens through environment variables cannot see each other's refresh.
• The race test for volume creation waits 2 s for no benefit.
• Real-server behaviours the fake only assumes: error codes for a bad verifier, a missing block and a stale revision ID; whether drafts are hidden from children; what delete_multiple does to an active link.

9. Before any upstream work

The spec's upstream path (section 13) is deferred. If it is ever taken up: rclone's maintainers refuse cgo; Proton's SDK README says the SDK "is not yet ready for third-party production use" and allows personal, non-commercial projects under its guidelines; a Proton engineer wrote that Proton "does not prohibit the usage of rclone, but does not offer formal support for it either" (rclone issue #9252, rclone forum thread 53609). Fork-only paths (docs/specs/, docs/plans/, docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/TODO.md, docs/PERMISSIONS.md, GEMINI.md, .claude/, .codex/, .gemini/, the "Fork Workflow" section of AGENTS.md, and backend/protondrive/protontest) never go into an upstream PR.