rclone/docs/plans/001-protondrive-tier1-workspace/final-review-transfer.md

Final review: data transfer (upload.go, download.go, Object Open/Update/Put/PutStream)

Reviewer area: backend/protondrive/upload.go, backend/protondrive/download.go, protondrive.go Object.Open/Update, Fs.Put/PutStream, api/types.go as far as these use it. HEAD 29ad26944. go test -race -run 'TestUpload|TestDownload|TestAcquire|TestIntended' passes in a scratch worktree (40 s).

Counts: Critical 0, Important 2, Minor 9.

Data-safety walk (no defect found)

I traced every path that can delete something during an upload:

• upload -> deleteDraft runs only when uploadDraft reports committed=false. That happens before the commit (verification, blocks, check, manifest, xattrs) or when refusedCommit is true on attempts == 1 (upload.go:403). refusedCommit needs an *api.Error with a Proton code and a 4xx status other than 408, so a 200 response with an error envelope, an HTML 4xx and a 5xx all leave the draft. Any pacer resend (5xx, 408, 429, network) makes attempts > 1.
• The session.call 401 -> refresh -> resend happens inside one pacer attempt. The first send got a 401, so the server never processed it. If the refresh then fails with a coded 4xx, refusedCommit matches the refresh's error and the draft is deleted. That is safe, because the commit was never processed. Go's transport does not replay a PUT unless nothing was written.
• When ctx is cancelled, fserrors.ContextError in shouldRetry turns even a successful commit into a ctx error. The detached single read (upload.go:411-414) then still finds the committed revision.
• resolveConflict deletes a draft (with delete_multiple) in only two cases: the details show a pure draft node (ConflictRevisionID == "" and a draft revision) left by our UID, or replace_existing_draft is set. It never deletes on a retry. A file conflict becomes a new revision, and a folder, draft or trashed node is refused.
• The read back after a commit never decides deletion. Lag gets about 7 s of reads that stop when ctx is done. A commit that is not ours ends as Unconfirmed, Superseded or OutcomeUnknown, and in each case only the cache is forgotten.

The upload pipeline (readBlocks -> requestURLs -> workers) frees every block on every path: the worker frees after upload, requestURLs frees on a URL error or cancel, and readBlocks frees on cancel. Both channels are unbuffered, so no block is stranded. Slots and tokens are always given back, and an abandoned acquireToken waiter returns its token. The size and SHA1 checks, the early long-source failure, the empty file and the exact-multiple-of-4 MiB case are all correct.

On the download side: the full read lists every block and checks the manifest before any fetch. Per-block hash and recorded-size checks are bounded by maxBlocks and maxFileBlocks before r.sizes[b.Index-1] is indexed, so that index cannot go out of range. A range with sizes not recorded starts from block 1 with skip=offset. A whole-file range goes to openWhole after FixRangeOption. Close drains every slot.

Important

I-1 draft_client_uid is not stable for remotes from environment variables or connection strings, so a leftover draft blocks its name

protondrive.go:652-657, upload.go:292-297.

NewFs makes a random UID and calls m.Set. fs.setConfigFile.Set (fs/configmap.go:90-94) saves nothing unless the remote is a section of the config file. A remote set up with RCLONE_CONFIG_X_* variables (typical in Docker and CI) or a connection string therefore gets a new UID on every run, and on every NewFs in the same process. By design, drafts are now left after every ambiguous commit failure, and also when the process dies mid-upload. On the next run the UID differs, so resolveConflict sees a foreign draft and fails with "another client is uploading the file (set replace_existing_draft...)". That repeats on every run for that name until the user sets the option.

This breaks spec 6.3 step 1 ("a stable per-remote draft client ID") for a common way of running rclone.

Fix: when the value cannot be saved, derive it deterministically instead of randomly. For example, use a hash of the account's user ID (known after sign-in) plus the remote name, or make the random UID only when fs.ConfigFileHasSection(name) and derive it otherwise. At the least, document it and log a NOTICE when the UID isn't persisted.

I-2 Multi-thread copies of files that record a size but not block sizes download data quadratically

download.go:142-151, protondrive.go:1150-1173.

docs/TODO.md records that a ranged read of such a file "downloads from the first block every time" and is slow on a mount. The case below is worse than that entry says.

Size() is known (xattr Size), so rclone's multi-thread copy turns on by itself above --multi-thread-cutoff (256 MiB) when copying to local. Each 64 MiB chunk is a separate Open with a RangeOption, and each one downloads and decrypts from block 1 up to the end of its chunk. It also lists every block page to the end of the file, because last = -1 when sizes aren't recorded. Total transfer is about S²/(2·64 MiB): about 8.5 GiB for a 1 GiB file and about 800 GiB for 10 GiB.

The code supports these files on purpose ("old clients didn't all write 4 MiB blocks"). How many real files lack BlockSizes is unknown (every golden capture records them), which is the only reason this isn't higher.

Fix, cheapest first:

1. A small per-client cache of revisionID -> learnt block sizes, filled as a reader decrypts blocks, so later chunks start at the nearest known block. This is the fix the TODO plans, and it should be raised in priority.
2. Until then, a ranged read with sizes not recorded and offset > 0 could at least stop the listing once the range is covered (it can't know where, so this is weak).
3. Or document that multi-thread copies of such files should use --multi-thread-streams 0.

Minor

M-1 Two processes sharing one config can delete each other's in-progress draft (upload.go:292-303). The UID is per remote, not per process. If a mount and a cron rclone move, or two overlapping runs, upload the same new name, B deletes A's live draft and A's upload fails. If A commits within the RTT between B's 2500 response and B's delete_multiple, B sends delete_multiple against an active file. What the server does then is the open TODO question. With move, A has already deleted its source. The TODO records only the --retries path. Add this case there. A per-process suffix on ClientUID, with "own" meaning the remote-wide prefix plus not-this-process, doesn't help. The real mitigation is the TODO's server check, or re-reading the link and confirming it is still a draft just before deleting (this narrows the window but does not close it).

M-2 pm-storage-token is still sent on redirects to other hosts (download.go:264-270, upload.go:868; Task 10 minor). Go's client drops only Authorization, Cookie and WWW-Authenticate when a redirect leaves the host. Set CheckRedirect on the storage calls, or refuse redirects for them.

M-3 The upload godoc has a broken reflow (upload.go:104-109: "...is errUploadSuperseded,\n// and one the server..."). That is the 21f re-review's N2 and it is still present. Rewrap the paragraph.

M-4 errUploadMismatch is a NoRetryError (protondrive.go:359). A source that changes while it is being read (a growing log file, or a local file still being written) fails for the whole run, including the --retries passes, where the usual rclone outcome is a retry at the high level. NoLowLevelRetryError would avoid a pointless immediate re-read and still let a later pass succeed.

M-5 A range read of a file whose block sizes aren't recorded keeps fetching after the range ends (download.go:145-151, 338-346). last stays -1 when sizes aren't recorded, so up to downloadAhead blocks (16 MiB) past the end of the range are downloaded and decrypted before Close. This adds to I-2.

M-6 An own leftover draft revision on an active file is never deleted (upload.go:279-290; Task 11 minor, still present). The conflict path uploads a new revision and leaves our old ConflictDraftRevisionID taking quota. A DELETE of that draft revision when ConflictDraftClientUID is ours would be safe: it is a draft revision, not the active one.

M-7 Updating an unverified file logs "reading" (upload.go:312 -> download.go:196; Task 16 minor, still present).

M-8 The errUploadSuperseded message is misleading when the node reads back as draft or trashed (upload.go:437-438; Task 11 minor, still present). The message says "revision ... is active".

M-9 The createDraft retry can report its own draft as a foreign one (upload.go:237-259, 294-295). Sequence: own draft deleted, re-POST gets a 5xx, the pacer resends, and the lost first response leaves our new draft, so the resend sees a 2500 with our UID. Because retried is set, the client reports "a draft of the file came back after it was deleted". Nothing is lost and the next run replaces the draft. When ConflictDraftClientUID is ours on the retried pass, the error could say so instead.

Still present and already in docs/TODO.md (not re-graded): Close waits behind the context-free downloadTokens.Get; countContent reads the whole file again for every suffix-range open of a file with no recorded size; --retries can reach the own-draft delete_multiple; unbounded block-list metadata bytes; ranged reads don't check the manifest; no test pins the 408 exclusion in refusedCommit (N3; the 408 case in nodes_internal_test.go:428 covers only shouldRetry).

Closed since earlier reviews: bytes=0- and whole-file ranges now go through openWhole (ruling 26). The read back now stops when ctx is done. Cleanup no longer deletes after a read back, so Task 11's "restore deletes our superseded history entry" can no longer happen.