rclone/docs/plans/001-protondrive-tier1-HANDOFF.md

Handoff: Proton Drive backend rewrite (plan 001)

Written 2026-11-11 when development was paused at the owner's request. Everything a new agent needs is in this repository; nothing depends on the conversation that produced it. Fork-only document - never include it in a PR to rclone/rclone.

1. What this is

The owner's fork of rclone is rewriting backend/protondrive as a native Go client for Proton Drive's current API (volume-based drive/v2 routes), replacing the abandoned Proton-API-Bridge / go-proton-api libraries. The goal is a working, maintainable, secure backend that rclone's maintainers could one day accept.

Read, in this order:

1. AGENTS.md - code, comment and commit rules; the "Fork Workflow" section.
2. docs/specs/001-protondrive-tier1.md - the approved spec (binding).
3. docs/plans/001-protondrive-tier1.md - the 21-task plan. Checkboxes show what is done.
4. This file - state, rulings, and how to resume.
5. docs/plans/001-protondrive-tier1-workspace/progress.md - the full execution ledger (every ruling, every deferred minor, every live probe).
6. docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/TODO.md.

2. Owner's standing instructions

• Nothing is posted upstream (no issue, PR or forum post) for now. The upstream path in spec section 13 is deferred.
• The owner is not planning permanent maintenance of the backend.
• The fake server (backend/protondrive/protontest) stays in the fork even though it is about a third of the code; it would not go upstream.
• Implementers are Opus 5.5 subagents, one task at a time; each task gets an independent review (Opus for auth, crypto, concurrency and the swap; Sonnet for small diffs and scoped re-reviews).
• "Once I approve a design, carry it through spec, plan and execution with no extra review stops. Report the decisions you made at the end."
• Permission rules: docs/PERMISSIONS.md (no sudo, no force-push, no git reset --hard, no git clean, no recursive rm). Ask before adding or upgrading a dependency.
• The push of this work to master on 2026-11-11 was an explicit one-off instruction from the owner. The standing rule in AGENTS.md (never push or commit directly to master) still applies to future work: branch first and ask before pushing.

3. State

Updated 2026-11-12. Branch protondrive-tier1; master on Gitea holds the work only up to the 2026-11-11 pause (4282540e0); everything after is local until the owner asks for a push. Plan base (for a whole-branch diff): 06700aaf2. Rewritten history: see "Open questions" item 1.

Work	State
Tasks 1-21	Implemented and independently reviewed; Task 21 as far as the test accounts allow (section 6)
Final whole-branch review	Run 2026-11-12 (four reviewers by area); one fix wave (e95317b52..944b36005); scoped re-review ALL CLOSED

What exists now:

• backend/protondrive is a native client of Proton's Drive API (session.go, account.go, crypto.go, nodes.go, cache.go, download.go, upload.go) behind the rclone Fs in protondrive.go. The old bridge libraries are gone from go.mod.
• backend/protondrive/protontest is the fake Proton server (fork-only), protontest/cmd/protonfake its standalone binary, started for fstest/test_all by fstest/testserver/init.d/TestProtonDriveFake.
• Tests: *_internal_test.go per client file, fake_test.go (TestIntegrationFake: rclone's contract suite against the fake), regression_test.go, security_test.go (hostile server), budget_test.go (request budgets), fuzz_internal_test.go, golden_internal_test.go (re-keyed files written by Proton's web app and official CLI, testdata/golden/), frozen old-library fixtures in testdata/oldlib/.
• Fork CI: .gitea/workflows/ci.yml (the owner's Gitea has a slow runner; the workflow has never run - the first push will show).
• User docs: docs/content/protondrive.md describes the new backend; its generated options block is regenerated only for upstream.
• Live: see section 6.

Not done, and easy to assume otherwise:

• ChangeNotify, the persistent cache, Metadata and the small-file upload path (M5, M6). Without events, a mount never sees other clients' changes to folders it has listed.
• Five test_all tests fail against both the fake and the real service because each Fs has its own listing cache (open question 5).
• Ranged reads check each block's hash and the content key's signature but not the signed manifest (docs/TODO.md).
• 2FA, CAPTCHA, Proton's desktop and mobile clients, and a drive created by Proton's own apps are unverified live.

4. How to resume

Development used the superpowers:subagent-driven-development skill (obra's superpowers plugin, version 7.0.0; install with /plugin marketplace add obra/superpowers-marketplace then /plugin install superpowers@superpowers-marketplace). The skill is a convenience, not a requirement: the per-task loop below can be followed by hand. Its workspace lives at .superpowers/sdd/001-protondrive-tier1/, which .gitignore excludes; a copy of every Markdown file from it (ledger, per-task briefs, implementer reports, reviews, the shared implementer/reviewer rules and the global constraints) is committed at docs/plans/001-protondrive-tier1-workspace/. To resume with the skill:

mkdir -p .superpowers/sdd/001-protondrive-tier1
cp docs/plans/001-protondrive-tier1-workspace/*.md .superpowers/sdd/001-protondrive-tier1/
git switch -c protondrive-tier1-resume   # never work on master

The ledger's first line names the plan; tasks with a Task N: complete line are done. Paths inside implementer-common.md and reviewer-common.md assume the checkout is /home/user/rclone and the branch protondrive-tier1 - adjust both if they differ. One rule in implementer-common.md is obsolete: "the old protondrive.go must keep compiling until Task 14" (Task 14 is done; there is no old implementation left in the package).

Per task: extract the brief (task-N-brief.md holds the plan text for task N; later ones are not extracted yet - copy the task's section from the plan), record BASE=$(git rev-parse HEAD), dispatch one implementer with the brief plus implementer-common.md, then a reviewer with the brief, the report and git diff -U10 BASE..HEAD, fix until clean, and record it in the ledger. After Task 21: one whole-branch review on the most capable model (merge base 06700aaf2), one fix wave, then git rm the plan and this workspace copy before any upstream-bound merge.

Remaining work

All 21 plan tasks are done. What is left:

• The owner's open questions (below), especially question 5.
• Later milestones, each needing its own plan: M5 (events and ChangeNotify), M6 (persistent metadata cache, rclone Metadata support, small-file single-request upload), M7 (upstream path, deferred by the owner; spec section 13 has the recipe).
• Live checks still open (docs/TODO.md): 2FA, CAPTCHA, desktop and mobile clients, a drive made by Proton's apps.

Notes for live work:

• Live probes run from a scratch git worktree with a binary built from it, never with go test in the main checkout without RCLONE_CONFIG=/notfound unless the live test is the point.
• Proton's official CLI 0.9.0 runs in Docker: image proton-drive-cli:0.9.0 (debian:trixie-slim, non-root, built from the checksum-verified release binary from https://proton.me/download/drive/cli/0.9.0/linux-x64/proton-drive), its free-account session in volume pdcli-free, e.g. docker run --rm -v pdcli-free:/home/drive proton-drive-cli:0.9.0 filesystem list /my-files. Its sign-in is browser-only (auth login prints a URL the owner opens) and gives up after about 10 minutes; a late sign-in shows "Invalid code".
• rclone config create prints the whole new remote, session secrets included, for every backend; never run it where the output is captured.
• Golden fixtures: TestGoldenCapture with -capture DIR -capture-label web|cli -capture-remote NAME:PATH (never -remote), into a scratch directory first; inspect for account data before copying into testdata/golden/ (the ledger records how it was done).
• Do not delete the evidence folders listed in section 6.

Open questions for the owner

Answered by the owner on 2026-11-12:

1. Usernames in git history: remove them. Done: commit 901d7fd5b was rewritten without the usernames (as 31f6aafd9), the later commits replayed unchanged, and master on Gitea force-pushed once (85f6b2f0e -> 4282540e0). A one-off; the no-force-push rule stands. Old hashes in this file and the ledger written before that date (901d7fd5b, 85f6b2f0e, d211b45e6, 28f1aaf8a) now map to 31f6aafd9, 4282540e0, 65840bdc5, 5afbe9e4f.
2. Gitea Actions: a runner exists but is slow and limited; keep .gitea/workflows/ci.yml light (no test_all in CI).
3. Official client interop (Task 21): allowed - install a released Proton client and use the web app, preferably inside a Docker container, signed in only to the test accounts.

Still open:

4. Rulings. Section 7 lists 33 decisions made on the owner's behalf; they were reported in summary and not confirmed one by one.
5. Cross-Fs cache staleness (Task 18). Five test_all tests are ignored because each Fs has its own node cache; a mount never sees other clients' changes. Fix options (shared per-account cache = package state the spec forbids; listing TTL; events in M5) are in docs/TODO.md. Current ruling: leave for M5.

5. Test accounts and machine setup

Machine-local, not in the repository. A new machine needs them recreated.

• Go 1.27.4 is in ~/.local/opt/go (symlinked into ~/.local/bin), golangci-lint 2.15.0 in ~/.local/bin.
• Two Proton accounts the owner created for testing, configured as rclone remotes in ~/.config/rclone/rclone.conf (mode 600, passwords obscured):
  • TestProtonDrive: - free account; password file ~/proton_rclone_test_acc_password.
  • TestProtonDrivePaid: - paid account; password file ~/proton_paid_acc_password. Never signed in to yet. Where to find the credentials on this machine (never copy either value into the repository, a commit message, a log or a subagent prompt):
  • Username: the username key of each remote in ~/.config/rclone/rclone.conf; rclone config show TestProtonDrive and rclone config show TestProtonDrivePaid print it. That output also contains the obscured password and, for the free account, session tokens, so do not paste it anywhere.
  • Password: plain text in the two files named above, one per account. The same password is stored obscured in the rclone config, so rclone itself needs nothing more; read the files only if a test must supply the password directly, and then pass it by file or environment, never on a command line that gets logged. Both remotes are set up for password sign-in, which is what every live probe used; browser sign-in has not been exercised against the real service. On a different machine, ask the owner for the accounts.
• Every go test must set RCLONE_CONFIG=/notfound, e.g. RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/.... Without it the live TestIntegration signs in to the real service on every run. Only Task 21 talks to Proton deliberately.
• Live probes were run from a scratch git worktree so nothing temporary could be committed. Proton's guidelines for third-party clients apply: honest x-pm-appversion (external-drive-rclone@<semver>-<channel>), official endpoints only, no hammering.
• The reference checkout of Proton's official SDK used during development was ~/.cache/protondrive-sdk-ref (github.com/ProtonDriveApps/sdk at commit 05c9115, tag js/v0.23.0). Read-only; never built or run.
• Live state on the free account: folder rclone-live-probe-20261111-130916 (files empty.bin, five-mib.bin, hello.txt with two revisions, and my-little-folder/Readme.txt made by the owner in Proton's web app). Leave it: it is the interop evidence.

6. What has been proven live

• Password (SRP) sign-in with the new code; a fresh account's Drive volume is created by the client.
• Uploads of 0 B, 25 B, 5 MiB+3 and 9 MB files and a second revision are accepted by Proton and read back byte-identical with signature and manifest verification.
• Proton's web app lists the files, opens them, and shows "Digital signature verified", the right size, modification time and SHA1.
• A folder and a file created in Proton's web app are listed, downloaded and verified by the new code.
• With the rewritten backend (binary built at c00162ed2): lsd, lsl, cat, sha1sum, about, sync up, check (0 differences), new revision, server-side moveto and copyto, sync back byte-identical, rmdir of a non-empty folder refused, purge.

Proven on 2026-11-12 (binaries at 502a32b5f and b440df2e1):

• TestIntegration (fstests) against TestProtonDrive:: 98 pass, 29 skipped by the suite's feature checks, 0 fail.
• test_all -n 1 against the real service: only the five cross-Fs cache tests fail (owner question 5); fs/operations, fs/sync and vfs otherwise pass. The first run found two real bugs (read-after- write lag after a commit), fixed in af6fdd789 and be0053607, and the re-run confirmed the fix.
• Browser (session-fork) sign-in with rclone config create against the real service, then listing, a verified read and config disconnect.
• The paid account: first password sign-in, about (64 GiB), mkdir and rmdir. Its drive was most likely created by rclone at that moment.
• Interop with Proton's official CLI 0.9.0 (run in Docker, signed in to the free account): each reads, verifies and modifies what the other wrote, both ways, including multi-block, empty and Unicode-named files.
• Upgrade/downgrade with released rclone v1.76.0: each reads what the other wrote, byte-identical.
• Re-keyed fixtures of files written by the web app and the CLI are in backend/protondrive/testdata/golden/ (TestGoldenFixtures).

Still not proven: 2FA; CAPTCHA; Proton's desktop and mobile clients; the root-of-trust rule on a drive made by Proton's own apps; the items marked "confirm" in docs/TODO.md.

Folders on the free account to leave in place (interop evidence): rclone-live-probe-20261111-130916, rclone-interop-20261112, rclone-golden-cli-20261112. The official CLI's session lives in the Docker volume pdcli-free (image proton-drive-cli:0.9.0, built from the checksum-verified release binary).

The security changes made after c00162ed2 were re-checked live on the free account without ignore_signature_errors: at e7e43095d (root of trust, subtree distrust, manifest-first reads: listing, a verified full read of the web-app file, an upload, a Range: bytes=0- read, purge) and at 9c90998f7, the last code commit (sync up, check with 0 differences, a 4-stream multi-thread download byte-identical, verified read of the web-app file, server-side move, purge). That drive's volume was created by this client, so the root-of-trust rule is still unchecked on a drive made by Proton's own apps. The hostile cases themselves can only be exercised against the fake.

7. Rulings made during execution

Decisions the controller made where the plan or spec was silent or wrong. Where this list and the plan or spec disagree, this list is what the code does. Most are in the ledger as Ruling: lines with their reason and what it costs if wrong; 16, 17 and 24 to 28 came out of review fix rounds and their reasoning is in task-14-review.md, task-16-review.md, task-16-rereview.md, task-16-rereview2.md and task-17-review.md in the workspace copy, and in docs/DECISIONS.md. The owner was told of them in summary but has not confirmed them one by one.

1. No separate worktree: work happened on branch protondrive-tier1 in the main checkout.

2. Task 8's retry test is TestClientShouldRetry (name collision).

3. The fake keeps Proton's published app-version grammar (channel optional); the client always sends a channel.

4. Two sessions sharing one config: after a rejected refresh the session re-reads the credential store for up to 3 s to adopt the other's new tokens. No cross-Fs lock (the spec allows no package state beyond unlockMu).

5. The key salt is chosen by the primary user key's ID (old library behaviour), first entry as fallback.

6. A fresh account has no Drive volume (my-files answers 422 code 2501): the client creates it, as the SDK does.

7. Written content follows the SDK (v4 node key advertising AEAD, v6 PKESK content key, SEIPDv2 blocks) even where the old library's comments disagree; confirmed accepted by the real server and web app.

8. The intended-recipient check on signatures is kept (stricter than the SDK may be).

9. The fake rejects AEAD-encrypted names, passphrases, hash keys and extended attributes written by the client.

10. An honoured Retry-After is capped at 60 s.

11. Upload commit outcome unknown: delete nothing, return errUploadOutcomeUnknown. A draft is deleted only when the server explicitly refused the commit (a 4xx with a Proton code, not 408) on a single attempt the pacer did not resend; no read-back decides deletion, as reads may lag (Task 21f fix round, superseding "positive proof" from a read).

12. The fake reports a draft's owner in conflict details but does not gate routes by ClientUID.

13. Proton-API-Bridge stays in go.mod until Task 19 even though no Go file imports it any more.

14. In rclone config, browser sign-in is the default only when username and password are not both set.

15. Shutdown leaves the Fs usable (it only drops the node cache); keys are not cleared. This supersedes the plan's "Shutdown calls client.close() once". Disconnect logs out; later use returns a clear "not signed in" error naming the remote.

16. Rmdir asks the server, uncached, whether the folder has any child at all and refuses if so.

17. rclone config on a remote with a saved session asks before signing in again (default no) and logs the old session out when it does.

18. A file whose node key is not signed by its owner is refused unless ignore_signature_errors is set. A key verified against an unverified folder key is itself unverified, all the way down; creating, uploading, renaming, moving or copying into such a folder is refused without the flag. The flag never disables integrity checks.

19. A full read fetches the whole block list and verifies the signed manifest before releasing any byte. Ranged reads do not verify the manifest yet (docs/TODO.md). First-byte latency on very large files grows by about one request per 80 MiB. A re-fetched block list that changes a hash is an integrity error.

20. TestBudgetNewObject allows 8 requests, not the plan's 7 (four folder levels at two requests each; Proton has no lookup-by-name route).

21. Every node beneath a folder whose key is unverified is unverified, reads included, even when signed by the user's own address (the implementer went beyond the ruling; a reviewer judged it justified and it was kept). A genuine folder signed by a since-deleted address locks its subtree behind ignore_signature_errors.

22. Moving, copying or renaming a node whose key is unverified is refused without the flag. With the flag it proceeds, logs an ERROR once per node and operation, and the node is trusted from then on.

23. Root of trust: the My Files share's creator and the root folder's signer must be one of the account's own addresses, else the whole drive is unverified. Addresses are compared after lowercasing ASCII letters only; a match is always verified with the account's own keys and never looked up on the server.

24. A file may list at most 2^19 blocks (2 TiB at 4 MiB). This bounds the count, not the bytes: about 250-350 MiB of block metadata at the cap.

25. Parent chains deeper than 2048, or cyclic, are an integrity error.

26. A range covering the whole file takes the verified full-read path and returns only the bytes asked for.

27. A folder listing is cached only if the folder's children did not change while it was being read (per-folder change count).

28. Names that are not valid UTF-8 are refused on write.

29. A file whose content key packet signature is missing or invalid is unverified (as 18 and 22): full reads, ranged reads and Update refused without ignore_signature_errors, ERROR once per node and operation with it. Stricter than Proton's SDK.

30. Proton codes 5001 and 5003 are the fatal "upgrade rclone" error; 2000 is a plain no-retry error keeping Proton's message; 2028 is retried through the pacer. Supersedes the spec 6.6 mapping of 2000 and 2028.

31. Keys from the server or another user whose Argon2 parameters exceed 4 passes, parallelism 8 or 128 MiB of memory are refused before the KDF runs (integrity error, not overridable).

32. A folder whose name hash key signature fails is unverified for writes (as 18); a rename or move gets a fresh name session key unless the old name verified; extended attributes whose signature fails are not used without the flag (size then comes from the verified block sizes, the modification time from the server, no SHA-1).

33. A remote whose config can't be saved (environment variables, connection strings) derives its draft client ID from the volume ID, so it is stable across runs; two machines using such a remote of the same name on one account share it.

Known protocol limits, documented rather than fixed: a server that serves a whole older, validly signed revision under the current revision ID cannot be detected (Proton's SDK cannot either); below the root, something signed under another user's email is verified against public keys the server itself supplies.

8. Deferred findings

The final whole-branch review (2026-11-12) left its Minor findings in docs/TODO.md under "final review minors (2026-11-12)"; its reports are final-review-*.md, final-fix-report.md and final-rereview.md in the workspace copy. The list below predates it and some items are now fixed.

Every review's minor findings were deferred to the final whole-branch review. They are listed per task in the ledger (progress.md, lines beginning Task N: minor (deferred) and "Deferred minors"). Only Tasks 14, 16 and 17 have review files in the workspace copy with file and line; for Tasks 1 to 13 and 15 the reviewer's text was not saved and the ledger's one-line entry (plus the implementer's report for that task) is all there is, so expect to re-locate those by reading the code. The whole-branch review will find most of them again. Items that need a decision or a live check are in docs/TODO.md. The ones most worth a new agent's attention:

• Ranged reads do not verify the signed manifest. Planned fix: a bounded per-revision cache of verified block hashes.
• With ignore_signature_errors set, Update of an unverified file logs "reading", not "updating" (upload.go, download.go contentKey).
• The depth-limit error is re-wrapped at every level and reaches about 88 KB; a hostile server can flood the log with it (nodes.go).
• TestListingRacingChange can hang until the package timeout if it fails before releasing its held request; protonfake's main_test.go leaks its run goroutine on a failed assertion.
• TestUploadShortSource failed its goroutine-leak check once under heavy parallel load and never again; TestListCancel uses the same goroutine-count approach.
• ModTime is returned with millisecond precision while Precision() says one second.
• Move, Copy and DirMove between two different accounts are not refused with fs.ErrorCant* (rclone only calls them for the same remote name).
• apiCall retries non-idempotent mutations after 5xx or network errors.
• An extended-attribute "Size": null is read as size 0.
• API responses have no body-size limit; API error messages are neither bounded nor sanitised before logging.
• Two processes sharing tokens through environment variables cannot see each other's refresh.
• The race test for volume creation waits 2 s for no benefit.
• Real-server behaviours the fake only assumes: error codes for a bad verifier, a missing block and a stale revision ID; whether drafts are hidden from children; what delete_multiple does to an active link.

9. Before any upstream work

The spec's upstream path (section 13) is deferred. If it is ever taken up: rclone's maintainers refuse cgo; Proton's SDK README says the SDK "is not yet ready for third-party production use" and allows personal, non-commercial projects under its guidelines; a Proton engineer wrote that Proton "does not prohibit the usage of rclone, but does not offer formal support for it either" (rclone issue #9252, rclone forum thread 53609). Fork-only paths (docs/specs/, docs/plans/, docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/TODO.md, docs/PERMISSIONS.md, GEMINI.md, .claude/, .codex/, .gemini/, the "Fork Workflow" section of AGENTS.md, and backend/protondrive/protontest) never go into an upstream PR.