rclone/docs/plans/001-protondrive-tier1-workspace/task-20-report.md

Task 20 report - documentation

Status: DONE_WITH_CONCERNS

Files and what they now say

docs/content/protondrive.md (hand-written parts only)

The autogenerated options block (between its markers) is byte-identical to before; checked with diff of the marker range before replacing the file. Sections:

• Intro: native client following Proton's Drive SDK; the third-party notice verbatim ("rclone is a third-party application not officially supported by Proton." - copied from thirdPartyNotice, protondrive.go:49). "Beta" note, old-library and henrybear327 text, and the "keys must have been generated in the browser" note removed (the client creates a missing Drive volume, nodes.go createVolume).
• Configuration: both sign-in methods; when browser sign-in is the default (no username+password set: configure, state method); the CAPTCHA -> reconnect path (errCaptcha handling in connect and configure); saved session in the config, advice to encrypt it. Transcript (replacing the stale user>/pass one) was captured from the real binary: rclone config run inside unshare -rn (no network at all) against a locally started protonfake, so every prompt text is what the code prints; the payload in the URL is replaced by XXXXXXXX, and the end summary and the long 2fa/otp help are [snip]ped. Password flow: the config_2fa prompt (fs.ConfigInput("password_2fa", "config_2fa", ...)), later re-sign-in using otp_secret_key (passwordSignIn) or a fatal "run rclone config reconnect" error (connect); reconnect on a signed-in remote asks first and signs the old session out (reauth state, signOut); 10 minute wait (forkPollTimeout); sign-in link holds a secret (showSignInURL comment, forkStart payload).
• Modification times and hashes: stored per upload, 1 s precision (Precision), SetModTime -> ErrorCantSetModTime; folder time set at creation (CreateDir passes time.Now()); SHA1 is the uploader's recorded value, empty when absent (node.sha1).
• Integrity and signatures: per-block SHA-256 check with three attempts (blockAttempts, errBlockHash); manifest verified before the first byte of a full read (openRead full path, verifyManifest); ~1 request per 80 MiB first-byte cost (blocksPerPage 20 x 4 MiB); node/content key signatures (contentKey). Upload: everything signed, checkBlock before sending, size/SHA1 checked before commit (uploadResult.check). Ranged-read caveat (open doc comment; whole-file range takes openWhole, Object.Open); files without block sizes may fail ranged reads not from the start (error at download.go:616). Root of trust (nodes.go:353-372), subtree distrust (decryptNode parent.unverified), refusals for reads, writes into unverified folders (targetFolder) and rename/move/copy of unverified nodes (nodes.go:1129-1139); with the flag an ERROR per node and operation (reportUnverified/reportOnce) and the node re-signed; block hashes never disabled. Name signature failures only logged (decryptNode verifyErrs, fs.Errorf). Both protocol limits (whole older revision; foreign signers checked against server-supplied keys).
• Restricted filename characters: 255 UTF-16 code units, not empty, ., .., no / or NUL (validateName); default encoding table derived from encoder.Base|InvalidUtf8|LeftSpace|RightSpace (Base = Zero, Slash, Dot); non-UTF-8 refused if the encoding allows it through. The old claim that spaces are "removed" was wrong (they are encoded).
• Duplicated files: name taken by a file -> new revision; own drafts replaced, other clients' drafts need replace_existing_draft (resolveConflict, error text at upload.go:291); "upload outcome unknown: it may have succeeded" quoted from errUploadOutcomeUnknown, not retried at once (NoLowLevelRetryError in translateError).
• Deleting files: Remove/Purge/Rmdir trash (trash), cleanup empties the whole volume trash (emptyTrash, DELETE volume trash). rmdir asks the server (hasChildren) so hidden children block it.
• Items left out of listings: Docs/Sheets with one notice (listable), undecryptable items with an ERROR (decryptNode unreadable path).
• Mailbox password, Caching (in-memory, no external changes seen, enable_caching=false advice), Logs and --dump: own logs carry no secrets (spec 7, security_test.go); --dump headers shows x-pm-uid (session.do) and pm-storage-token (download.go:273, upload.go:828); --dump bodies shows tokens (per docs/TODO.md entry from TestNoSecretsInLogs).
• Limitations (after the generated block, replacing the old library text): only My Files (bootstrap uses drive/v2/shares/my-files only); no change events/ChangeNotify; no --metadata; 2^19-block cap (maxFileBlocks), depth 2048; about is account-wide (/core/v4/users UsedSpace/MaxSpace); 4 MiB blocks, upload_concurrency default 4 max 5 (defaultUploadConcurrency, maxUploadConcurrency, cap in NewFs), 10 up + 10 down per remote (blocksInFlight token dispensers).

Rulings 18-28 coverage: 18, 21, 22, 23 (Integrity section), 19 and 26 (full read / ranged caveat), 24 and 25 (Limitations), 28 (filename section). 20 (test budget) and 27 (listing cache generation) are not user-visible and are not documented. Earlier user-visible rulings also covered: 4 (implicitly, saved session refresh), 6, 10 is not mentioned (internal pacing detail), 11, 14, 15/Disconnect not mentioned specifically, 16, 17.

Not claimed: browser sign-in, 2FA and desktop/mobile interop are described but never said to be tested; nothing says integration tests pass; no mention of ChangeNotify, Metadata, small-file upload or thumbnails as features.

backend/protondrive/protondrive.go

ignore_signature_errors Help: rewritten: the old text said only "read files"; the code (rulings 18, 21, 22) also gates writes into, and rename/move/copy of, unverified nodes, and the subtree rule. New text names those and says block hashes are always checked. No other Go change.

docs/data/backends/protondrive.yaml

bin/manage_backends.py features was not run as is: it reads remote: from the yaml (TestProtonDrive:, the live remote) and runs the system rclone with the user's config, which would contact Proton. Instead its exact computation was reproduced offline: built rclone, started fstest/testserver/init.d/TestProtonDriveFake (local protonfake), checked the endpoint was http://127.0.0.1:28642 before running RCLONE_CONFIG=/notfound rclone backend features TestProtonDriveFake: (env-var remote), took the sorted true Features, Hashes, Precision. Result written by hand: added Copy, ListP, PutStream, Shutdown; removed NoMultiThreading; hashes sha1 and precision 1000000000 unchanged; features_score: 5. tier, maintainers, integration_tests untouched.

docs/specs/001-conformance-checklist.md (new)

One row per SDK module (apiService, shares, nodes, upload, download, crypto, account/auth, plus events as not implemented): SDK files (paths checked to exist in ~/.cache/protondrive-sdk-ref, read-only listing only), Go files, js/v0.23.0, deliberate differences drawn from the rulings and code (no small-file path, no events, no thumbnails, root of trust, subtree distrust, manifest-first, block cap, depth limit, UTF-8, outcome-unknown, draft client UID, Retry-After cap 60 s maxRetryAfter, fork client ID external-drive, 3 s adopt wait adoptWait).

docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/TODO.md

• ARCHITECTURE: protondrive section gains user-docs/checklist pointers and a "not implemented yet" line (M5, M6, ranged-read manifest). The "Known rough edges" heading was kept: its one line is about host tooling for the whole repo, not protondrive, so replacing it would lose that pointer.
• DECISIONS: new entry "Proton Drive is a native port of Proton's Drive SDK" (checklist as the conformance mechanism, docs rules).
• TODO: "Now" updated to Tasks 1-20 done; new entries for M5, M6, M7, one entry listing every live check not run (paid account, browser sign-in, 2FA/OTP, CAPTCHA, official non-web clients, live fstests/test_all, drafts/trashed as children, non-ASCII address case), and one for the stale generated options block; the --dump entry no longer says "Task 20's docs must carry the warning".
• Plan: Task 20 checkboxes ticked.

Verification

• go build ./... OK; go vet ./backend/protondrive/... OK; golangci-lint run ./backend/protondrive/... 0 issues.
• RCLONE_CONFIG=/notfound go test ./backend/protondrive/... all ok (protondrive 105 s).
• Nothing contacted Proton: the features run and the transcript capture used only a local protonfake (the transcript one in a network-less namespace). No config or password file read. No dependency changes.

Where code, docs and spec disagree

1. The generated options block in docs/content/protondrive.md is stale (old options, old help, no upload_concurrency or ignore_signature_errors). Left alone per AGENTS.md; logged in TODO.
2. Spec 6.7 says filename rules are "non-empty, at most 255 characters, no /"; code also refuses ., .., NUL and non-UTF-8, and counts UTF-16 code units. Docs follow the code.
3. enable_caching help still says "please disable this feature" for VFS mounts; true today (no events), so left.
4. The local SDK checkout at 05c9115 carries only the tag cs/v0.33.0; js/v0.23.0 is taken from the handoff and code comment.
5. The brief says Step 4 replaces the "known rough edges" line; kept it (see above).

Concerns

• The docs present browser sign-in as the primary method, as the spec says, but it has never been run against the real service (handoff section 6). The docs do not claim it is tested; Task 21 should check it before anything goes upstream.
• rclone about "whole account" relies on /core/v4/users UsedSpace being account-wide, which is Proton's documented model but was not checked live for this wording.

Fix round 1

1. Ranged reads of files without recorded block sizes (code fix)

Problem: openRead placed a range in a file without recorded block sizes by assuming 4 MiB blocks (blockAt(nil, ...)), and also bounded the last block that way. The size check in next (checkAssumed/ prevSize) fired only when a second block followed a mis-sized one, so single-block ranges returned wrong bytes, and ranges ending early returned short data, silently.

Test first: TestDownloadRangesUnrecordedBlockSizes (download_internal_test.go) seeds a 6 MiB+5 file in 1 MiB blocks with no BlockSizes in its extended attributes (new fake helper protontest.User.SeedFileInBlocks; seedSpec.blockSize, newRevision takes the block size) and reads 9 ranges.

RED, before the fix (RCLONE_CONFIG=/notfound go test -run TestDownloadRangesUnrecordedBlockSizes ./backend/protondrive/):

--- FAIL: TestDownloadRangesUnrecordedBlockSizes/0+3145728      got 1048576 bytes, want 3145728
--- FAIL: TestDownloadRangesUnrecordedBlockSizes/1048573+10     got 3 bytes, want 10
--- FAIL: TestDownloadRangesUnrecordedBlockSizes/4194314+100    got 100 bytes, want 100   (wrong bytes, silently)
--- FAIL: .../4194314+-1, 2097159+3145728, 6291460+-1, 6291461+-1   unexpected error ("read it from the start")

Fix (download.go): without recorded sizes the read starts at block 1 with skip = offset and no last block (the remaining count stops it); next now discards whole blocks while skip covers them, then seeks into the block holding the offset. Every block is still checked against its listed hash (and its decryption); the manifest is still not checked for a ranged read. With recorded sizes nothing changes (skip is always inside the first block). checkAssumed, prevSize, the nil branch of blockAt and the FIXME are gone; defaultBlockSize's comment now says it is the upload block size.

GREEN: the new test passes; go test -count=3 -run 'TestDownload|TestBudget' passes.

Test expectation changed: TestDownloadRanges' unrecorded-size column (renamed from "assumed") now expects reads from the start, so ranges past block 1 fetch every block (3 for the 3-block file) where they fetched 1 or 2. Its count is now a range (at least the blocks needed up to the range, at most all 3), since blocks past the range may be fetched ahead before the reader is closed. No budget test changed (budget_test.go doesn't read files without block sizes).

Docs: the page's ranged-read paragraph now says such reads download, check and discard every block before the range, so they are slower, and still don't check the manifest. The checklist's download row says the same.

2. SDK identified by commit

docs/specs/001-conformance-checklist.md identifies the SDK as commit 05c9115955589d9f6ab7392a36f8f0cb39327133 (2026-11-05), noting its client/js/CHANGELOG.md newest entry is js/v0.23.0 (2026-10-19) and that the only tag on it is cs/v0.33.0; re-checks are a diff of the listed files between commits; the column reads "05c9115 (js/v0.23.0)". The DECISIONS entry says the same. Left alone, as not written by me in this task: the package comment in protondrive.go:5 ("last checked against its tag js/v0.23.0") and older spec/handoff/TODO text.

3. enable_caching

The Help: drops the "please disable this feature" notice and says to disable the cache if other clients change the same files while rclone runs, as can happen with a mount; the page's Caching section uses the same wording.

Verification

• go build ./... OK; go vet ./backend/protondrive/... OK; golangci-lint run ./backend/protondrive/... 0 issues.
• RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/...: all ok (protondrive 313 s, protontest 38 s, protonfake 1.4 s).
• No Proton host contacted, no secrets files read, no dependency change.