docs: bring the Proton Drive handoff up to date after the final review
ci / protondrive (push) Successful in 23m29s
ci / protondrive (push) Successful in 23m29s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed
1 parent
944b360054
commit
fbf2d565e8
29 files changed
+3502
-138
No files matched your search
@@ -12,6 +12,14 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
`docs/plans/001-protondrive-tier1-HANDOFF.md`. Remaining: the
|
||||
whole-branch review and its fix wave.
|
||||
|
||||
- [ ] protondrive: the free test account's derived key passphrase (the
|
||||
`client_salted_key_pass` of a browser sign-in on 2026-11-12) was shown
|
||||
in an agent's tool output when `rclone config create` printed the new
|
||||
remote; that session was logged out at once, but the passphrase only
|
||||
changes with the account password. Owner: change the free account's
|
||||
password if that matters, then update `~/proton_rclone_test_acc_password`
|
||||
and the remote's saved password.
|
||||
|
||||
## Next
|
||||
|
||||
- [ ] protondrive M5: volume events polled no more often than every 30 s,
|
||||
@@ -228,6 +236,16 @@ Small, specific, independently actionable. Remove only when actually done.
|
||||
|
||||
### final review minors (2026-11-12)
|
||||
|
||||
- [ ] protondrive (final re-review): ranged reads place their start from
|
||||
block sizes in extended attributes even when those are unsigned, so a
|
||||
forging server can shift the bytes a ranged read returns (full reads
|
||||
are unaffected).
|
||||
- [ ] protondrive (final re-review): `TestMain` applies `RCLONE_CONFIG`
|
||||
only when it is set; with it unset, tests may still find the user's
|
||||
config.
|
||||
- [ ] protondrive (final re-review): the draft client ID stays random per
|
||||
run when the remote has a config section that can't be saved.
|
||||
|
||||
Minors from the whole-branch review (`final-review-*.md` in the plan
|
||||
workspace) not fixed in its fix wave. Paths are under `backend/protondrive/`.
|
||||
|
||||
|
||||
@@ -44,52 +44,51 @@ Read, in this order:
|
||||
or commit directly to `master`) still applies to future work: branch
|
||||
first and ask before pushing.
|
||||
|
||||
## 3. State at pause
|
||||
## 3. State
|
||||
|
||||
Branch `protondrive-tier1`, fast-forwarded into `master` and pushed at the
|
||||
pause. Last code commit: `9c90998f7`. Plan base (for a whole-branch diff):
|
||||
`06700aaf2`.
|
||||
Updated 2026-11-12. Branch `protondrive-tier1`; `master` on Gitea holds
|
||||
the work only up to the 2026-11-11 pause (`4282540e0`); everything after
|
||||
is local until the owner asks for a push. Plan base (for a whole-branch
|
||||
diff): `06700aaf2`. Rewritten history: see "Open questions" item 1.
|
||||
|
||||
| Tasks | State |
|
||||
| Work | State |
|
||||
|---|---|
|
||||
| 1-20 | Implemented, each independently reviewed, all findings of Critical or Important severity fixed and re-reviewed |
|
||||
| 21 | Done as far as the test accounts allow (2026-11-12); whole-branch review next |
|
||||
| Final whole-branch review | Not run |
|
||||
| Tasks 1-21 | Implemented and independently reviewed; Task 21 as far as the test accounts allow (section 6) |
|
||||
| Final whole-branch review | Run 2026-11-12 (four reviewers by area); one fix wave (`e95317b52`..`944b36005`); scoped re-review ALL CLOSED |
|
||||
|
||||
What exists now:
|
||||
|
||||
- `backend/protondrive/protondrive.go` is rewritten on the native client
|
||||
- `backend/protondrive` is a native client of Proton's Drive API
|
||||
(`session.go`, `account.go`, `crypto.go`, `nodes.go`, `cache.go`,
|
||||
`download.go`, `upload.go`). The old bridge libraries left `go.mod` in
|
||||
Task 19, with the differential tests that used them.
|
||||
`download.go`, `upload.go`) behind the rclone `Fs` in `protondrive.go`.
|
||||
The old bridge libraries are gone from `go.mod`.
|
||||
- `backend/protondrive/protontest` is the fake Proton server (fork-only),
|
||||
with `protontest/cmd/protonfake` as a standalone binary.
|
||||
- rclone's backend contract suite (`fstests.Run`) passes against the fake
|
||||
as `TestIntegrationFake`: 126 subtests, 97 pass, 29 skipped by the
|
||||
suite's own feature checks, no opt-outs.
|
||||
- Test files: `*_internal_test.go` per client file, `fake_test.go`
|
||||
(harness), `regression_test.go` (16 user-reported failures),
|
||||
`security_test.go` (27 hostile-server cases, secret hygiene),
|
||||
`budget_test.go` (request budgets), `fuzz_internal_test.go` (5 targets),
|
||||
frozen old-library fixtures in `testdata/oldlib/` (`TestFrozenFixtures`,
|
||||
`TestFrozenFileFixture`).
|
||||
- Verified at `9c90998f7` on 2026-11-11: `go build ./...`,
|
||||
`go vet ./backend/protondrive/...`,
|
||||
`golangci-lint run ./backend/protondrive/...` (0 issues) and
|
||||
`RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/...` all
|
||||
pass. The race suite takes about 5 to 6 minutes.
|
||||
`protontest/cmd/protonfake` its standalone binary, started for
|
||||
`fstest/test_all` by `fstest/testserver/init.d/TestProtonDriveFake`.
|
||||
- Tests: `*_internal_test.go` per client file, `fake_test.go`
|
||||
(`TestIntegrationFake`: rclone's contract suite against the fake),
|
||||
`regression_test.go`, `security_test.go` (hostile server),
|
||||
`budget_test.go` (request budgets), `fuzz_internal_test.go`,
|
||||
`golden_internal_test.go` (re-keyed files written by Proton's web app
|
||||
and official CLI, `testdata/golden/`), frozen old-library fixtures in
|
||||
`testdata/oldlib/`.
|
||||
- Fork CI: `.gitea/workflows/ci.yml` (the owner's Gitea has a slow
|
||||
runner; the workflow has never run - the first push will show).
|
||||
- User docs: `docs/content/protondrive.md` describes the new backend; its
|
||||
generated options block is regenerated only for upstream.
|
||||
- Live: see section 6.
|
||||
|
||||
Not done, and easy to assume otherwise:
|
||||
|
||||
- The user documentation (`docs/content/protondrive.md`) still describes
|
||||
the old backend (Task 20).
|
||||
- The wider suites (`fs/operations`, `fs/sync`, `vfs` via
|
||||
`fstest/test_all`) have never been run against the new backend (Task 18).
|
||||
- No fork CI exists (Task 18).
|
||||
- `ChangeNotify`, the persistent cache, Metadata and the small-file upload
|
||||
path are not implemented (M5, M6).
|
||||
- Ranged reads (mount, `--multi-thread-streams`, `serve`) check each
|
||||
block's hash but not the signed manifest (`docs/TODO.md`).
|
||||
path (M5, M6). Without events, a mount never sees other clients'
|
||||
changes to folders it has listed.
|
||||
- Five `test_all` tests fail against both the fake and the real service
|
||||
because each `Fs` has its own listing cache (open question 5).
|
||||
- Ranged reads check each block's hash and the content key's signature
|
||||
but not the signed manifest (`docs/TODO.md`).
|
||||
- 2FA, CAPTCHA, Proton's desktop and mobile clients, and a drive created
|
||||
by Proton's own apps are unverified live.
|
||||
|
||||
## 4. How to resume
|
||||
|
||||
@@ -127,113 +126,39 @@ record it in the ledger. After Task 21: one whole-branch review on the
|
||||
most capable model (merge base `06700aaf2`), one fix wave, then
|
||||
`git rm` the plan and this workspace copy before any upstream-bound merge.
|
||||
|
||||
### Remaining tasks (plan sections of the same number)
|
||||
### Remaining work
|
||||
|
||||
- **18 - Wider hermetic suites and fork CI.** `init.d` script starting
|
||||
`protonfake`, `test_all` config entry, `.gitea/workflows/ci.yml`,
|
||||
`docs/PERMISSIONS.md` update. Expect real backend failures from
|
||||
`fs/operations`, `fs/sync` and `vfs`; fix them in the backend.
|
||||
Note `-run TestIntegration` matches both the live and the fake test.
|
||||
- **19 - Remove the old libraries.** Move the frozen-fixture tests, delete
|
||||
the differential test file, `go mod tidy`, compare binary size,
|
||||
cross-compile.
|
||||
- **20 - Documentation.** Rewrite `docs/content/protondrive.md`; write
|
||||
`docs/specs/001-conformance-checklist.md`. Must cover what this pause
|
||||
leaves undocumented for users: the `ignore_signature_errors` behaviour
|
||||
and its consequences, the ranged-read caveat, `--dump headers` showing
|
||||
`pm-storage-token` and `x-pm-uid`, the whole-revision rollback limit.
|
||||
- **21 - Live verification.** The plan text says it is blocked until a
|
||||
test account exists and that the owner configures it by browser
|
||||
sign-in. That is out of date: both accounts exist and are configured
|
||||
with username and password (section 5). First things to check live:
|
||||
the root-of-trust rule on a drive created by Proton's own apps (the paid
|
||||
account); how many folders fail key verification on a real account;
|
||||
whether the server lists drafts and trashed items as children; browser
|
||||
sign-in; `TestIntegration` and `test_all` against the real service.
|
||||
All 21 plan tasks are done. What is left:
|
||||
|
||||
### Notes for Tasks 18 to 21
|
||||
- The owner's open questions (below), especially question 5.
|
||||
- Later milestones, each needing its own plan: M5 (events and
|
||||
`ChangeNotify`), M6 (persistent metadata cache, rclone Metadata
|
||||
support, small-file single-request upload), M7 (upstream path,
|
||||
deferred by the owner; spec section 13 has the recipe).
|
||||
- Live checks still open (`docs/TODO.md`): 2FA, CAPTCHA, desktop and
|
||||
mobile clients, a drive made by Proton's apps.
|
||||
|
||||
The plan text for these tasks predates the work that changed things under
|
||||
it. Known corrections, found by having a fresh reader walk the plan against
|
||||
the code:
|
||||
Notes for live work:
|
||||
|
||||
**Task 18**
|
||||
|
||||
- Every command needs `RCLONE_CONFIG=/notfound` except where it must read
|
||||
the `TestProtonDriveFake:` definition - `test_all` gets its remote from
|
||||
the init.d script's output, not from the config file, so
|
||||
`RCLONE_CONFIG=/notfound go run ./fstest/test_all ...` is right. Check
|
||||
that no step can reach the live `TestProtonDrive:` remote.
|
||||
- `-run TestIntegration` matches both the live `TestIntegration` and
|
||||
`TestIntegrationFake`. `TestIntegrationFake` skips itself when `-remote`
|
||||
is given; keep that guard.
|
||||
- The init.d script cannot simply copy `TestWebdavRclone`. `protonfake`
|
||||
prints freshly minted credentials on stdout
|
||||
(`endpoint=`, `client_uid=`, `client_access_token=`,
|
||||
`client_refresh_token=`, `client_salted_key_pass=`, `type=protondrive`),
|
||||
while the `run` helper in `rclone-serve.bash` redirects stdout to a log
|
||||
file. The script has to start the binary, wait for the six lines, and
|
||||
echo them. Use a built binary, not `go run` (killing the `go` parent
|
||||
leaves the server running), and find the repository root rather than
|
||||
assuming the working directory, because scripts are run from each test
|
||||
package's directory. Nobody has designed this yet.
|
||||
- No port has been allocated in `PORTS.md`; pick the next free one there.
|
||||
- All test processes started by `test_all` would share one fake session
|
||||
through those values. Refresh tokens are single-use, and a process whose
|
||||
credentials come from the environment cannot see another process's
|
||||
refresh (section 8, "two processes sharing tokens through environment
|
||||
variables"). If anything makes a process refresh - check whether the
|
||||
standalone fake ever expires an access token, and what tests that call
|
||||
`Disconnect` do to the shared session - the others would be left with
|
||||
dead tokens. Check this before writing the script; if it is a problem,
|
||||
have `protonfake` mint a session per run or issue non-expiring tokens in
|
||||
standalone mode.
|
||||
- Whether the owner's Gitea has an Actions runner is unknown. Ask the
|
||||
owner; the workflow cannot be verified locally either way.
|
||||
- Expect real failures from `fs/operations`, `fs/sync` and `vfs`: they have
|
||||
never been run against this backend.
|
||||
|
||||
**Task 19**
|
||||
|
||||
- Step 1 says to build `master` as the "old" binary. `master` now contains
|
||||
the new backend. Build the plan's base commit `06700aaf2` instead (it
|
||||
has the old backend and the same rclone version).
|
||||
- Step 2: `TestFrozenFileFixture` is already in
|
||||
`download_internal_test.go` with no old-library import.
|
||||
`TestFrozenFixtures` still depends on helpers inside
|
||||
`crypto_diff_internal_test.go` and on an `-update` regeneration path.
|
||||
The accurate instructions are the `docs/TODO.md` entry beginning "when
|
||||
go-proton-api and Proton-API-Bridge leave `go.mod`", which also records
|
||||
that multi-block coverage of old-library files is lost at that point.
|
||||
- `go mod tidy` is part of the approved plan for this task, so it does not
|
||||
need asking again, but accept only the removal of the two old libraries
|
||||
and what they alone pulled in (and `relvacode/iso8601` returning to
|
||||
indirect or leaving). Anything else: stop and ask.
|
||||
- "No new failures in `go test ./...`": the baseline on the host is in
|
||||
`docs/TODO.md` under "Known issues" (only `cmd/serve/s3` fails).
|
||||
|
||||
**Task 20**
|
||||
|
||||
- Must also document what was decided after the plan was written: section
|
||||
7 rulings 18 to 26 as user-visible behaviour, the ranged-read caveat,
|
||||
the `--dump headers` exposure, and both protocol limits.
|
||||
|
||||
**Task 21**
|
||||
|
||||
- The preconditions in the plan are out of date (see the note there and
|
||||
section 5). The accounts use password sign-in.
|
||||
- "Interop with Proton's official CLI": the reference SDK checkout must
|
||||
never be built or run (it is untrusted reference data), and no official
|
||||
CLI is installed. Ask the owner whether to install a released Proton
|
||||
client for this, or to limit interop to the web app as done so far.
|
||||
- Golden-fixture capture stores real account material; captured nodes must
|
||||
come only from the throwaway test accounts and be reviewed for anything
|
||||
identifying before they are committed.
|
||||
- Do not delete `rclone-live-probe-20261111-130916` on the free account.
|
||||
|
||||
Later milestones, each needing its own plan: M5 (events and `ChangeNotify`),
|
||||
M6 (persistent metadata cache, rclone Metadata support, small-file
|
||||
single-request upload), M7 (upstream path, deferred by the owner).
|
||||
- Live probes run from a scratch `git worktree` with a binary built from
|
||||
it, never with `go test` in the main checkout without
|
||||
`RCLONE_CONFIG=/notfound` unless the live test is the point.
|
||||
- Proton's official CLI 0.9.0 runs in Docker: image
|
||||
`proton-drive-cli:0.9.0` (debian:trixie-slim, non-root, built from the
|
||||
checksum-verified release binary from
|
||||
`https://proton.me/download/drive/cli/0.9.0/linux-x64/proton-drive`),
|
||||
its free-account session in volume `pdcli-free`, e.g.
|
||||
`docker run --rm -v pdcli-free:/home/drive proton-drive-cli:0.9.0 filesystem list /my-files`.
|
||||
Its sign-in is browser-only (`auth login` prints a URL the owner opens)
|
||||
and gives up after about 10 minutes; a late sign-in shows "Invalid
|
||||
code".
|
||||
- `rclone config create` prints the whole new remote, session secrets
|
||||
included, for every backend; never run it where the output is captured.
|
||||
- Golden fixtures: `TestGoldenCapture` with `-capture DIR
|
||||
-capture-label web|cli -capture-remote NAME:PATH` (never `-remote`),
|
||||
into a scratch directory first; inspect for account data before
|
||||
copying into `testdata/golden/` (the ledger records how it was done).
|
||||
- Do not delete the evidence folders listed in section 6.
|
||||
|
||||
## Open questions for the owner
|
||||
|
||||
@@ -254,7 +179,7 @@ Answered by the owner on 2026-11-12:
|
||||
|
||||
Still open:
|
||||
|
||||
4. **Rulings.** Section 7 lists 28 decisions made on the owner's behalf;
|
||||
4. **Rulings.** Section 7 lists 33 decisions made on the owner's behalf;
|
||||
they were reported in summary and not confirmed one by one.
|
||||
5. **Cross-`Fs` cache staleness (Task 18).** Five `test_all` tests are
|
||||
ignored because each `Fs` has its own node cache; a mount never sees
|
||||
@@ -449,6 +374,26 @@ in summary but has not confirmed them one by one.
|
||||
change while it was being read (per-folder change count).
|
||||
28. Names that are not valid UTF-8 are refused on write.
|
||||
|
||||
29. A file whose content key packet signature is missing or invalid is
|
||||
unverified (as 18 and 22): full reads, ranged reads and `Update`
|
||||
refused without `ignore_signature_errors`, ERROR once per node and
|
||||
operation with it. Stricter than Proton's SDK.
|
||||
30. Proton codes 5001 and 5003 are the fatal "upgrade rclone" error; 2000
|
||||
is a plain no-retry error keeping Proton's message; 2028 is retried
|
||||
through the pacer. Supersedes the spec 6.6 mapping of 2000 and 2028.
|
||||
31. Keys from the server or another user whose Argon2 parameters exceed 4
|
||||
passes, parallelism 8 or 128 MiB of memory are refused before the KDF
|
||||
runs (integrity error, not overridable).
|
||||
32. A folder whose name hash key signature fails is unverified for writes
|
||||
(as 18); a rename or move gets a fresh name session key unless the old
|
||||
name verified; extended attributes whose signature fails are not used
|
||||
without the flag (size then comes from the verified block sizes, the
|
||||
modification time from the server, no SHA-1).
|
||||
33. A remote whose config can't be saved (environment variables,
|
||||
connection strings) derives its draft client ID from the volume ID, so
|
||||
it is stable across runs; two machines using such a remote of the same
|
||||
name on one account share it.
|
||||
|
||||
Known protocol limits, documented rather than fixed: a server that serves
|
||||
a whole older, validly signed revision under the current revision ID
|
||||
cannot be detected (Proton's SDK cannot either); below the root, something
|
||||
@@ -457,6 +402,11 @@ server itself supplies.
|
||||
|
||||
## 8. Deferred findings
|
||||
|
||||
The final whole-branch review (2026-11-12) left its Minor findings in
|
||||
`docs/TODO.md` under "final review minors (2026-11-12)"; its reports are
|
||||
`final-review-*.md`, `final-fix-report.md` and `final-rereview.md` in the
|
||||
workspace copy. The list below predates it and some items are now fixed.
|
||||
|
||||
Every review's minor findings were deferred to the final whole-branch
|
||||
review. They are listed per task in the ledger (`progress.md`, lines
|
||||
beginning `Task N: minor (deferred)` and "Deferred minors"). Only Tasks
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Final fix wave — brief (controller rulings)
|
||||
|
||||
Findings files (read the cited entries in full; file:line refers to HEAD 29ad26944):
|
||||
- final-review-crypto.md (C1, C2, I1, I2, I3)
|
||||
- final-review-transfer.md (I-1, I-2)
|
||||
- final-review-auth.md (I-1, I-2)
|
||||
- final-review-tests.md (I-1, I-2 only; I-3..I-5 are the controller's)
|
||||
All in /home/user/rclone/.superpowers/sdd/001-protondrive-tier1/.
|
||||
|
||||
Work order: Critical first, one commit per finding (or tightly related group), each
|
||||
with a regression test written first and seen failing (the reviewers' reproductions
|
||||
are described in the files; reuse them). Commit prefix `protondrive:` (CI: `ci:` is
|
||||
fine), messages user-readable per AGENTS.md.
|
||||
|
||||
## Rulings
|
||||
|
||||
**crypto C1 — unsigned/invalid content key packet signature.** Verify
|
||||
ContentKeyPacketSignature (as the SDK defines it; check the reference checkout
|
||||
~/.cache/protondrive-sdk-ref read-only) on every use of a file's content key: reads
|
||||
(full and ranged) and Update. A missing or invalid signature makes the file
|
||||
unverified, exactly like an unverified node key (ruling 18/22): reads and writes
|
||||
refused unless `ignore_signature_errors`; with the flag, proceed and log an ERROR
|
||||
once per node and operation. Integrity checks are never disabled. This is stricter
|
||||
than the SDK; record it in docs/DECISIONS.md and as handoff ruling 29.
|
||||
|
||||
**crypto C2 — Argon2 parameters from the server.** Before running any S2K/KDF on
|
||||
key material that came from the server or another user (node keys, share keys,
|
||||
anything not the account's own locally verified keys), check the S2K parameters
|
||||
against fixed caps and refuse (integrity error, not overridable by the flag) above
|
||||
them. Choose the caps from what Proton's clients actually generate (gopenpgp v3 and
|
||||
go-crypto defaults; the SDK) with modest headroom, and document them. Also bound
|
||||
iterated-salted S2K counts if the library does not. Add a fuzz/unit test.
|
||||
|
||||
**crypto I1 — unverified hash key used for writes.** A hash key whose signature is
|
||||
missing/invalid makes the folder unverified for writes (same as ruling 18: creating,
|
||||
uploading, renaming, moving into it refused without the flag; ERROR with the flag).
|
||||
|
||||
**crypto I2 — rename/move reusing a name session key whose signature failed.** Never
|
||||
reuse a session key from a name that did not verify; generate a fresh one (if the
|
||||
name verified, keep current behaviour).
|
||||
|
||||
**crypto I3 — unsigned xattrs trusted.** Follow spec 6.4: xattrs whose signature is
|
||||
missing/invalid are not used for modtime/size/SHA1 unless `ignore_signature_errors`
|
||||
(then used with an ERROR once per node). Without them, fall back as the code already
|
||||
does when xattrs are absent (check what that fallback is and that rclone behaves
|
||||
sanely, e.g. ModTime from the revision create time, no SHA1). Do not make the file
|
||||
unreadable just for this.
|
||||
|
||||
**transfer I-1 — draft_client_uid not persisted for env/connection-string remotes.**
|
||||
When the value cannot be persisted (or no value exists), derive a stable one
|
||||
deterministically instead of random per run, e.g. from the account's user ID and the
|
||||
remote name through a hash, so the same remote on the same account gets the same
|
||||
UID across runs; keep saving it when the config can store it. No package state.
|
||||
|
||||
**transfer I-2 — quadratic multi-thread reads of files without recorded block
|
||||
sizes.** Find a fix that keeps returned bytes always correct and every block
|
||||
verified. Preferred: compute each block's plaintext size from the block list
|
||||
without downloading, if and only if it can be done exactly for the packet formats
|
||||
involved (verify against the golden fixtures and the fake's shapes; if compression
|
||||
or variable framing makes it inexact, do not use it). Otherwise a bounded per-`Fs`
|
||||
cache (not package state) of plaintext block sizes learned per revision as blocks
|
||||
are decrypted, so later chunks seek directly. Measure with a test counting block
|
||||
fetches for an N-chunk multi-thread read. Update the docs page and TODO.
|
||||
|
||||
**auth I-1 — error-code mapping.** Map Proton's bad-app-version codes (5003, 5001;
|
||||
confirm in the SDK reference) to the fatal "upgrade rclone" error. Code 2000 becomes
|
||||
a plain `fserrors.NoRetryError` keeping Proton's message (not fatal). Code 2028:
|
||||
treat as Proton's documented meaning (check the SDK); if it can mean rate limiting,
|
||||
make it retryable via the pacer, not fatal. Update spec section 6.6 note via a
|
||||
DECISIONS entry and handoff ruling 30. Tests for each code.
|
||||
|
||||
**auth I-2 — x-pm-drive-requirements.** Log those response headers at debug level
|
||||
when present, as spec 6.6 says (once per value change, not per request).
|
||||
|
||||
**tests I-1 — CI race timeout.** Add `-timeout 45m` (or fitting the job timeout) to
|
||||
the CI `go test -race` step; consider `-p`/`-parallel` limits for the slow runner.
|
||||
|
||||
**tests I-2 — CI Go version / govulncheck.** Pin Go to the 1.27 line the code was
|
||||
verified with (go-version: '1.27.x' or exact 1.27.4) instead of go.mod's minimum,
|
||||
and pin govulncheck to a fixed version instead of @latest.
|
||||
|
||||
Minors from all four files: fix the ones that are cheap and local to code you are
|
||||
already touching; list every other Minor in docs/TODO.md under a single
|
||||
"final review minors (2026-11-12)" heading with file references, so nothing is lost.
|
||||
|
||||
## Rules
|
||||
|
||||
Read implementer-common.md (its rule about the old protondrive.go is obsolete) and
|
||||
global-constraints.md first. RCLONE_CONFIG=/notfound on every go test; never contact
|
||||
Proton; never read ~/.config/rclone/rclone.conf or ~/proton_*password; never pass
|
||||
-remote/-capture-remote. No dependency changes. Do not edit
|
||||
docs/plans/001-protondrive-tier1-HANDOFF.md, docs/plans/001-protondrive-tier1-workspace/,
|
||||
or docs/specs/001-protondrive-tier1.md section 13 (the controller edits those in
|
||||
parallel); you may append rulings text to the controller via your report instead.
|
||||
Done = go build ./..., go vet ./backend/protondrive/..., golangci-lint run
|
||||
./backend/protondrive/... (0 issues), full race suite once at the end, TestGoldenFixtures
|
||||
passing. No sudo, no force-push, no git reset --hard, no git clean, no recursive rm,
|
||||
never push, never switch branches. End commits with:
|
||||
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
||||
@@ -0,0 +1,267 @@
|
||||
# Final fix wave: implementer report
|
||||
|
||||
Branch `protondrive-tier1`, starting at 29ad26944. One commit per finding.
|
||||
The controller's `ea8d09ce4` landed between mine and is not part of this
|
||||
work. Every test ran with `RCLONE_CONFIG=/notfound`. No dependency
|
||||
changes, nothing pushed, and no edits to the handoff, the workspace copy
|
||||
or spec section 13.
|
||||
|
||||
## Commits
|
||||
|
||||
| Commit | Finding |
|
||||
|---|---|
|
||||
| e95317b52 | crypto C1: content key signature |
|
||||
| db7afafc0 | crypto C2: Argon2 caps on node and share keys |
|
||||
| b3f5685e6 | crypto I1: unverified hash key blocks writes |
|
||||
| 67e4cfeb7 | crypto I2: no session key reuse for an unverified name |
|
||||
| 94eedc942 | crypto I3: unsigned xattrs not used |
|
||||
| 674b538ca | transfer I-1: stable draft client ID when it can't be saved |
|
||||
| 788a711dc | transfer I-2: learnt block sizes make multi-thread reads linear |
|
||||
| 5dbf5b04c | auth I-1: error-code mapping (5001/5003 fatal, 2000 no-retry, 2028 retried) |
|
||||
| b4a23a31e | auth I-2: x-pm-drive-requirements logged |
|
||||
| b5da26169 | tests I-1 and I-2: CI timeout, Go 1.27.x, govulncheck pinned |
|
||||
| 634a2423e | (found while working) tests honour RCLONE_CONFIG whichever tests run |
|
||||
| 30cfcac41 | auth M-1 and M-2: refresh 5xx/429/408 retried |
|
||||
| d3aee2af1 | cheap minors: crypto M5 and M6, transfer M-3 and M-8, fake gofmt |
|
||||
| bfa9b7122 | test minors 1, 6, 8 and 9, tests minors 13 and 14 (docs), every other minor logged in TODO |
|
||||
| ad3552e2e | ARCHITECTURE line for the block size cache |
|
||||
| 944b36005 | fake test follows the renamed code constant (found by the final race run) |
|
||||
|
||||
## Per finding
|
||||
|
||||
### crypto C1: unsigned or invalid content key signature (e95317b52)
|
||||
- **Failing test first:** two new `hostileContents` cases in `TestHostileServer`
|
||||
("unsigned content key the server made", "content key signed by another
|
||||
key"), and "ranged reads and updates of a file with content-key-unsigned /
|
||||
content-key-signature". Before the fix the read of an unsigned key returned
|
||||
content (`security_test.go:204 An error is expected but got nil`), the Update
|
||||
went through (`:570`), and the per-operation ERROR line was missing.
|
||||
- **Fake:** the new corruptions `content-key-unsigned` and `content-key-signature`
|
||||
make a new content key and re-encrypt every block under it. The blocks and
|
||||
manifest are re-signed with the user's key, which is more than a real server
|
||||
can do, so only the content key signature can catch it.
|
||||
- **Change:** `decryptContentKey` always verifies, so a missing signature is
|
||||
`errSignatureMissing`. `contentKey(n, op)` takes "reading" or "updating". Without
|
||||
the flag it refuses with errIntegrity and an op-specific hint. With the flag it
|
||||
logs an ERROR once per node and operation (`reportOnce("content key "+op)`). The
|
||||
pre-2023 all-addresses fallback now applies only to a signature that is present
|
||||
but invalid, as in the SDK. This also fixes transfer M-7 (Update logged
|
||||
"reading").
|
||||
- **Existing tests changed:** `TestContentKeyRoundTrip/NoSignature` (now expects
|
||||
`errSignatureMissing`) and `TestDownloadContentKey` ("unsigned" is now refused,
|
||||
plus a new "unsigned before 2023" case).
|
||||
- **Deviation:** none. There is no date exemption for unsigned keys, because the
|
||||
date comes from the server. Every golden fixture has a signed content key
|
||||
(`TestGoldenFixtures` passes). Recorded in DECISIONS (ruling 29) and on the docs page.
|
||||
|
||||
### crypto C2: Argon2 parameters from the server (db7afafc0)
|
||||
- **Failing test first:** `TestNodeKeyS2KLimits`. The patched keys (memory 2^18,
|
||||
passes 255, parallelism 255, and memory 2^31 at the library limit) got no
|
||||
errIntegrity. With a stub check, the memory case ran a 256 MiB KDF in 0.2 s.
|
||||
- **Change:** `decryptNodeKeys` (used for node and share keys) calls
|
||||
`unlockForeignKey`. That function unarmors once, runs `checkKeyS2K` on every
|
||||
secret key and subkey packet before the KDF can run, then unlocks. Neither
|
||||
go-crypto nor gopenpgp exposes a key's S2K parameters, so `s2kSpecifier`
|
||||
finds them by hand. It parses the packet with go-crypto (cheap: no KDF), takes
|
||||
the public key's serialized body as the prefix, then walks the usage, count,
|
||||
cipher, AEAD and length octets the way go-crypto does.
|
||||
- **Caps:** Argon2 may use at most 4 passes, parallelism 8 and 2^17 KiB (128 MiB).
|
||||
The defaults of gopenpgp and OpenPGP.js are 3, 4 and 64 MiB, and Proton's node
|
||||
keys use iterated-and-salted S2K anyway. Going over a cap is errIntegrity, and
|
||||
the flag can't override it.
|
||||
- **Iterated-salted:** not capped further. go-crypto's one-byte count encoding
|
||||
already bounds it to 65011712 bytes hashed (4 times the default), and a lower
|
||||
cap could refuse genuine old keys.
|
||||
- **Fuzz:** `FuzzCheckKeyS2K` ran 20 s locally (1.3M execs, no panic).
|
||||
- **Deviation:** the account's own user keys are not checked. The ruling scopes
|
||||
this to foreign keys, and a genuine user key's parameters aren't known. This is
|
||||
logged in TODO. Address keys are safe, because their token's signature is checked
|
||||
before the unlock. DECISIONS entry added.
|
||||
|
||||
### crypto I1: unverified hash key used for writes (b3f5685e6)
|
||||
- **Failing test first:** "writing into a folder whose hash key the server made".
|
||||
The upload into the folder succeeded before the fix (`:272`).
|
||||
- **Change:** `node.hashKeyErr` is set when the hash key signature fails and isn't
|
||||
date-exempt. `targetFolder` refuses without the flag, which covers create,
|
||||
upload, rename (within the folder), move into and copy into. With the flag it
|
||||
logs an ERROR once per folder. Reads are not affected. A comment notes that the
|
||||
date exemptions use the server's CreateTime (crypto M4). Docs page updated.
|
||||
- **Deviation:** not folded into `keys.unverified`. Doing that would make every
|
||||
child unreadable, and the ruling covers writes only.
|
||||
|
||||
### crypto I2: rename/move reusing an unverified name's session key (67e4cfeb7)
|
||||
- **Failing test first:** "renaming or moving a node whose name the server
|
||||
encrypted". The new stored name's session key equalled the server's for both
|
||||
the rename and the move. The fake gained `User.NameSessionKey` and the
|
||||
`name-unsigned` corruption.
|
||||
- **Change:** `node.nameSigned` records whether the name signature actually
|
||||
verified, taken before the pre-2021 exemption so that a back-dated name doesn't
|
||||
count. `renameSessionKey` returns nil, which means a fresh key, unless the name
|
||||
is signed. The same test checks that a verified name keeps its key.
|
||||
- **Deviation:** a pre-2021 unsigned (exempt) name also gets a fresh key. This is
|
||||
the safe reading of "did not verify".
|
||||
|
||||
### crypto I3: unsigned xattrs trusted (94eedc942)
|
||||
- **Failing test first:** "unsigned extended attributes". The forged modtime
|
||||
(2099), the forged SHA-1 and the forged 1 TiB size were all used, and the
|
||||
folder's modtime too.
|
||||
- **Change:** without the flag, when the xattr signature fails, `n.X` becomes
|
||||
`blockSizesOnly()`. The ERROR line says the modification time, size and SHA-1
|
||||
are not used. With the flag the attributes are used, logged once per node by the
|
||||
existing verify report.
|
||||
- **Deviation:** I did not use the plain "xattrs absent" fallback. That fallback
|
||||
reports the *encrypted* size, so every `rclone copy` of the file would fail its
|
||||
size check, which in practice makes the file unreadable. The ruling forbids
|
||||
that. Instead I keep only the recorded block sizes, which every read checks
|
||||
block by block, and take Size from their sum. ModTime falls back to the server's
|
||||
ModifyTime and there is no SHA-1. If there are no block sizes, Size is the
|
||||
encrypted size as before. The test checks that full and ranged reads still work.
|
||||
Docs page updated.
|
||||
|
||||
### transfer I-1: draft_client_uid for env/connection-string remotes (674b538ca)
|
||||
- **Failing test first:** `TestDraftClientUIDStableWhenUnsaved`, using a mapper
|
||||
whose `Set` does nothing. Two runs of the same remote got different IDs.
|
||||
- **Change:** `NewFs` sets a random ID through `m.Set`, then reads it back with
|
||||
`m.Get`. If it didn't stick (env vars or a connection string), it uses
|
||||
`client.derivedDraftClientUID(name)` = hex(SHA-256("rclone draft client\0" +
|
||||
volumeID + "\0" + name))[:32].
|
||||
- **Deviation:** I used the account's volume ID rather than the user ID. The
|
||||
account struct doesn't keep the user ID, and the volume ID is just as stable
|
||||
per account. Config-file remotes still get a random saved ID, so two machines
|
||||
with their own config files don't collide. The docs say that two machines
|
||||
running an env-configured remote of the same name count as one remote.
|
||||
|
||||
### transfer I-2: quadratic multi-thread reads (788a711dc)
|
||||
- **Option chosen:** a bounded per-client cache, not computing sizes from the
|
||||
listing. The packet framing (partial body lengths, the literal packet header and
|
||||
any compression) depends on the writer, so plaintext size can't be computed
|
||||
exactly from encrypted size.
|
||||
- **Failing test first:** `TestDownloadChunksUnrecordedBlockSizes`, 32 blocks read
|
||||
as 8 sequential range opens. It fetched 145 blocks before the fix and 39 after.
|
||||
A range read afterwards fetches exactly its 3 blocks, and a block list changed
|
||||
since the sizes were learnt is errIntegrity.
|
||||
- **Change:** `blockSizeCache` lives on `client`, so it is per Fs and not package
|
||||
state. It holds at most 16 revisions and 65536 blocks (about 2.5 MiB). The
|
||||
reader learns sizes in order in `next()`, as a contiguous prefix with each
|
||||
block's hash. `openRead` uses `locate` to start at the right block, and to stop
|
||||
at the last block when the prefix covers the range, which partly fixes transfer
|
||||
M-5.
|
||||
- **Correctness:** every block is still hash-checked. If the start block's listed
|
||||
hash differs from the learnt hash, or a re-read block's hash or size differs,
|
||||
that is errIntegrity.
|
||||
- **Limit:** chunks that start together still each read from block 1. The docs
|
||||
page, the TODO entry and the conformance checklist are updated.
|
||||
- **Existing tests changed:** `TestDownloadRanges` (the unrecorded case now
|
||||
benefits from learnt sizes) and the always-3 nit in it.
|
||||
|
||||
### auth I-1: error codes (5dbf5b04c)
|
||||
- **Codes checked in the SDK reference:**
|
||||
- 5003 is "You need to update this app" (`coreTypes.ts`).
|
||||
- 5001 is from go-proton-api.
|
||||
- 2028 is "feature disabled, rate-limited or blocked because of abuse"
|
||||
(`driveTypes.ts`).
|
||||
- **Failing tests first:** `TestTranslateError` (5001 fatal, 2000 no-retry),
|
||||
`TestClientShouldRetry` (2028 retried, and with Retry-After honoured),
|
||||
`TestRegressionFeatureBlockedRetried`, `TestRegressionRenameResendNotFatal`
|
||||
(the reviewer's lost-rename reproduction, which before the fix was fatal with
|
||||
"upgrade rclone"). `TestRegressionAppVersionRejected` now uses 5003.
|
||||
- **Change:**
|
||||
- api: `CodeFeatureBlocked=2028`, `CodeAppVersionMissing=5001` and
|
||||
`CodeAppVersionBad=5003` (renamed; it used to mean 2028).
|
||||
- `translateError`: 5001/5003 are fatal with "upgrade rclone". 2000 is a
|
||||
`NoRetryError` that keeps Proton's message. 2028 falls through as a plain
|
||||
error.
|
||||
- `shouldRetry`: 2028 is retried, through `RetryAfterError` when a Retry-After
|
||||
header is present.
|
||||
- **Deviation:** 2064 is left unmapped, because the user can set `app_version`
|
||||
wrong themselves. DECISIONS entry added (ruling 30).
|
||||
|
||||
### auth I-2: x-pm-drive-requirements (b4a23a31e)
|
||||
- **Failing test first:** `TestDriveRequirementsLogged`. The fake gained
|
||||
`SetDriveRequirements`.
|
||||
- **Change:** `session.do` calls `observeRequirements`, which logs at debug, `%q`,
|
||||
truncated to 1 KiB, only when the joined value changes.
|
||||
|
||||
### tests I-1 and I-2: CI (b5da26169)
|
||||
- `go test -race -timeout 40m`, and `timeout-minutes` raised from 45 to 60 to make
|
||||
room for the fuzz step.
|
||||
- `go-version: '1.27.x'`.
|
||||
- `govulncheck@v1.9.0`, the latest release on proxy.golang.org at the time of
|
||||
writing (queried with `go list -m -versions`).
|
||||
- No `-p`/`-parallel` limits: the race suite is one big package, so `-p` doesn't
|
||||
help, and limiting `-parallel` would make it slower.
|
||||
|
||||
### Extra: tests read the real rclone.conf (634a2423e)
|
||||
- While working I saw "Using config file from ~/.config/rclone/rclone.conf" in a
|
||||
captured test log. `RCLONE_CONFIG` is only applied by `fstest.Initialise`, which
|
||||
only `TestIntegration` runs, so any filtered run (`-run X`) read the real config
|
||||
holding the live `TestProtonDrive:` remote. Nothing contacted Proton, since the
|
||||
tests use env-configured fake remotes, but the file was read.
|
||||
- `TestMain` now calls `config.SetConfigPath(RCLONE_CONFIG)` first; `/notfound`
|
||||
means memory-only. Earlier runs in this and previous sessions read that file.
|
||||
|
||||
## Minors
|
||||
|
||||
**Fixed:**
|
||||
- crypto M4: code comment, plus a TODO line.
|
||||
- crypto M5: `decryptBlock` moved to the tests; the `reencryptPassphrase` alias
|
||||
removed.
|
||||
- crypto M6: `truncateName` cuts on a rune boundary, with a test.
|
||||
- transfer M-3: godoc rewrapped.
|
||||
- transfer M-5: partly; the range end is known once sizes are learnt.
|
||||
- transfer M-7: covered by C1.
|
||||
- transfer M-8: the superseded message gives the state and revision.
|
||||
- auth M-1 and M-2: refresh 5xx/429/408 retried, and 408 no longer kills the
|
||||
session; `TestRefreshTransientErrorRetried` failed first for 503 and for 408.
|
||||
- tests 1: timing bounds raised to 3 s.
|
||||
- tests 6: the 502-once test counts the uploads.
|
||||
- tests 8: `TestRefusedCommit` pins the 408 exclusion.
|
||||
- tests 9: "assumed" renamed and the always-3 bound fixed.
|
||||
- tests 13: conformance checklist.
|
||||
- tests 14: the docs explain errUploadUnconfirmed.
|
||||
|
||||
**Logged** in docs/TODO.md under "final review minors (2026-11-12)":
|
||||
- crypto M1, M2, M3, M7 and M8.
|
||||
- transfer M-1, M-2, M-4, M-6 and M-9.
|
||||
- auth M-3 to M-11.
|
||||
- tests 2, 3, 4, 5, 7, 10, 11, 12, 15, 16 and 17.
|
||||
- tests 17 (.codex rules) is a permission-file change, which I left to the owner.
|
||||
|
||||
**Not mine:** tests 18 and 19 (handoff) and tests I-3 to I-5 (the controller's).
|
||||
|
||||
## Verification (at ad3552e2e)
|
||||
- `go build ./...`: OK.
|
||||
- `go vet ./backend/protondrive/...`: OK.
|
||||
- `golangci-lint run ./backend/protondrive/...`: 0 issues.
|
||||
- Full race suite at ad3552e2e: `RCLONE_CONFIG=/notfound go test -race -count=1 -timeout 40m ./backend/protondrive/...`
|
||||
- `backend/protondrive`: ok, 332.6 s.
|
||||
- `protonfake`: ok.
|
||||
- `protontest`: failed `TestFaultInjection/Skip`. It still used the
|
||||
`api.CodeAppVersionBad` name for 2028, and my rename had missed it. Fixed in
|
||||
944b36005; the `protontest` race run then passed (39.1 s), and lint was
|
||||
re-run with 0 issues.
|
||||
- `TestGoldenFixtures`: PASS (also run alone with -v).
|
||||
|
||||
## Suggested handoff ruling text
|
||||
|
||||
29. A file whose content key signature is missing or invalid is
|
||||
unverified, like an unverified node key (rulings 18, 22): reads, full
|
||||
and ranged, and updates are refused unless `ignore_signature_errors`
|
||||
is set, when an ERROR is logged once per file and operation. The
|
||||
pre-2023 fallback to the account's other address keys applies only to
|
||||
a signature that is present. Stricter than the SDK, which skips the
|
||||
check when the signature is absent. In the same wave: a folder whose
|
||||
hash key signature fails can't be written into without the flag;
|
||||
extended attributes whose signature fails aren't used without it
|
||||
(modtime from the server, no SHA-1, size from the recorded block sizes);
|
||||
a rename or move never reuses the session key of a name whose signature
|
||||
didn't verify; node and share keys with an Argon2 S2K beyond 4 passes,
|
||||
parallelism 8 or 128 MiB are an integrity error the flag can't override.
|
||||
|
||||
30. Error codes: 5001 and 5003 (missing or rejected app version) are a
|
||||
fatal error telling the user to upgrade rclone; 2000 (requirements not
|
||||
met, e.g. a stale `OriginalHash` after a resent rename) is a no-retry
|
||||
error keeping Proton's message; 2028 (feature disabled, rate-limited or
|
||||
blocked) is retried through the pacer, honouring `Retry-After`. This
|
||||
supersedes spec 6.6's 2000/2028 fatal mapping. `x-pm-drive-requirements`
|
||||
is logged at debug level when it changes.
|
||||
@@ -0,0 +1,99 @@
|
||||
# Final fix wave: scoped re-review (29ad26944..944b36005)
|
||||
|
||||
Verdict: **ALL CLOSED**. No open or new Critical or Important findings. There are 3 new Minors.
|
||||
|
||||
How I checked: I read the diff (I skipped ea8d09ce4, the controller's spec section 13 commit) and the code at HEAD where needed. I compared it with the SDK reference (`~/.cache/protondrive-sdk-ref`, read-only) and with gopenpgp v3.4.1 and go-crypto v1.5.2 in the module cache. I worked in a scratch worktree, `fr-rereview` at 944b36005, which is now removed with `git worktree remove` (no --force). Every `go test` ran with `RCLONE_CONFIG=/notfound`. I did not touch `/home/user/rclone`, apart from this file.
|
||||
|
||||
## Findings: closed?
|
||||
|
||||
| Finding | Status | Test that fails without the fix (mutation run here) |
|
||||
|---|---|---|
|
||||
| crypto C1, unsigned content key | Closed | I put back `if sig == "" { return sk, nil, nil }` in `decryptContentKey`. That fails `TestContentKeyRoundTrip/NoSignature`, `TestHostileServer/unsigned_content_key_the_server_made`, `.../ranged_reads_and_updates_of_a_file_with_content-key-unsigned` and `TestDownloadContentKey/unsigned{,_before_2023}`. Reads (full and ranged) and Update (`upload.go:319`, op "updating") go through `contentKey(n, op)`. It has no other callers. |
|
||||
| crypto C2, Argon2 from the server | Closed | Mutation 1: removed the `checkKeyS2K` call. `TestNodeKeyS2KLimits/{Memory,Passes,Parallelism}` fail, and the KDF runs. Mutation 2: shifted the specifier offsets by one. `.../Accepted` fails. `decryptNodeKeys` is the only unlock path for node and share keys (`nodes.go:367,530`). The offsets match go-crypto's `PrivateKey.parse` (`private_key.go:219-271`): usage, the v5/v6 count, cipher, AEAD for 253, the v6 length. |
|
||||
| crypto I1, unverified hash key used for writes | Closed | I made the `hashKeyErr` refusal unreachable. `TestHostileServer/writing_into_a_folder_whose_hash_key_the_server_made` fails. |
|
||||
| crypto I2, rename/move reusing the session key of an unverified name | Closed | I made `renameSessionKey` ignore `nameSigned`. `TestHostileServer/renaming_or_moving_a_node_whose_name_the_server_encrypted` fails. |
|
||||
| crypto I3, unsigned xattrs | Closed (accepted deviation) | I removed `n.X = n.X.blockSizesOnly()`. `TestHostileServer/unsigned_extended_attributes` fails. |
|
||||
| transfer I-1, draft_client_uid | Closed (accepted deviation) | I made the derived-UID branch a no-op. `TestDraftClientUIDStableWhenUnsaved` fails. `m.Get` after `m.Set` works as a test of whether the value was saved: `fs.setConfigFile.Set` saves only when the section exists (`fs/configmap.go:90`), and `getConfigFile` reads the value back. |
|
||||
| transfer I-2, quadratic multi-thread reads | Closed | I made `openRead` skip `learnt.locate`. `TestDownloadChunksUnrecordedBlockSizes` fails with 146 blocks fetched for 8 chunks (the limit is 64). |
|
||||
| auth I-1, error codes | Closed | I mapped 2000/2028 back to fatal. `TestTranslateError` and `TestRegressionRenameResendNotFatal` fail. 5001 and 5003 are confirmed fatal in the code and the tests. |
|
||||
| auth I-2, x-pm-drive-requirements | Closed | `TestDriveRequirementsLogged`. The header is logged once per change and truncated to 1 KiB. |
|
||||
| tests I-1, CI race timeout | Closed | `-timeout 40m`, inside the job's `timeout-minutes: 60`. |
|
||||
| tests I-2, CI Go version and govulncheck | Closed | `go-version: '1.27.x'` (local Go is 1.27.4). `govulncheck@v1.9.0`, which `go list -m -versions golang.org/x/vuln` shows is the latest. |
|
||||
|
||||
## Regression checks
|
||||
|
||||
**(a) Reading files that real Proton clients wrote.** `TestGoldenFixtures`, `TestGoldenRoundTrip` and `TestGoldenRefusesForeignKeys` pass at HEAD. Re-made signatures could hide a real-world mismatch, so I also checked two things:
|
||||
|
||||
- **The fixtures keep each signature's role and keep missing signatures missing.**
|
||||
- `goldenCapture.verified` (`golden_internal_test.go:864`) made the original capture fail on any signature failure the backend reported against the real account.
|
||||
- `resignDetached` keeps `""` as `""` and re-signs with the replacement for the same signer role.
|
||||
- The rekey notes record which key signed what in the captured web and CLI files:
|
||||
- `ContentKeyPacketSignature`: by the node key, 4 files.
|
||||
- `NodeHashKey`: by the node key, including both roots.
|
||||
- `XAttr`: by the address key.
|
||||
- The oldlib fixture also has a `ContentKeyPacketSignature`.
|
||||
- **rclone verifies with the same key sets as the SDK:**
|
||||
- The content key with node key plus signature-email keys, or the parent key for an anonymous node. That is `[key, ...keyVerificationKeys]`, `cryptoService.ts:637-644`.
|
||||
- The fallback to all of the account's addresses only applies to a signature that is present but invalid (`:652`).
|
||||
- The hash key with node key plus signature-email keys. That is `decryptNodeHashKey`, `driveCrypto.ts:534-537`.
|
||||
- Revision and folder xattrs with the signature-email keys, or the node key when there is no email (`:582-584`, `:177-179`).
|
||||
|
||||
So the new refusals hit only signatures that the SDK would also report as failed. The one place rclone is stricter than the SDK is an absent content key signature. That is the recorded ruling 29.
|
||||
|
||||
**(b) Argon2 caps.**
|
||||
- The defaults are 3 passes, parallelism 4 and memory exponent 16:
|
||||
- go-crypto `Argon2Config` (`s2k_config.go:98-115`).
|
||||
- gopenpgp v3.4.1's RFC9580 preset (`preset.go:72-78`), which uses an empty `Argon2Config`, so it gets those defaults.
|
||||
- OpenPGP.js writes the same.
|
||||
- The caps are 4 passes, parallelism 8 and exponent 17, so they leave headroom.
|
||||
- The SDK checkout doesn't mention S2K or Argon2 anywhere, so it sets none of its own.
|
||||
- Iterated-and-salted S2K is not checked. Its one-octet count encoding bounds it already.
|
||||
- `checkKeyS2K` parses packets without running the KDF.
|
||||
- I noted no new problem here.
|
||||
|
||||
**(c) Block-size cache.**
|
||||
- It is per Fs: a `blockSizeCache` value on `client`, with no package state.
|
||||
- It is bounded: at most 16 revisions and 65536 blocks.
|
||||
- The eviction loops leave the counters consistent.
|
||||
- A new revision can't be left in `order` without its entry in `revs`, as long as `learntBlocks > 0`.
|
||||
- It is safe for concurrent use: every access holds the mutex, and `locate` clones the start hash before unlocking.
|
||||
- The focused `-race` run of `TestDownloadChunksUnrecordedBlockSizes`, `TestDownloadRanges` and `TestDownloadRangesUnrecordedBlockSizes` passed.
|
||||
- I also wrote and ran a scratch `-race` test: 8 to 9 concurrent chunk readers per round, 3 rounds, at offsets that don't line up with blocks. All bytes were correct and no race was reported.
|
||||
- It never returns wrong bytes from a genuine server:
|
||||
- Sizes are learnt only from blocks that were decrypted and checked against their hash, and only as a contiguous prefix.
|
||||
- The start block's listed hash must equal the learnt one.
|
||||
- A re-read block must match in both hash and size.
|
||||
- If eviction happens between `locate` and `learn`, learning just stops; it never corrupts.
|
||||
- It is invalidated on a new revision, because entries are keyed by revision ID and a committed revision's blocks are immutable.
|
||||
|
||||
**(d) Error-code remap.**
|
||||
- 2028 is retried only by `shouldRetry` inside the pacer, so the low-level retry count bounds it. A `Retry-After` value is capped by `maxRetryAfter`. After the retries, the plain error goes back to rclone.
|
||||
- 5001 and 5003 are 4xx, so they are not retried, and are fatal.
|
||||
- 2000 is `NoRetryError`.
|
||||
- The refresh path now returns `nil, refreshErr`. A rejected refresh is still `errSessionExpired`, which is not retried. A 5xx, 408 or 429 on the refresh is retried within the pacer's limit.
|
||||
- Nothing loops without a bound.
|
||||
|
||||
**(e) RCLONE_CONFIG in tests.**
|
||||
- `TestMain` lives in `package protondrive_test`, and Go builds the internal and external test files into one binary, so it covers every test in `backend/protondrive`.
|
||||
- I proved it with a fake `HOME`/`XDG_CONFIG_HOME` holding a sentinel `rclone.conf`, so the user's file was never at risk. I ran a scratch test that calls `configfile.Install()` with `RCLONE_CONFIG=/notfound`:
|
||||
- At HEAD it got path `""` (memory only) and no sections.
|
||||
- With the `TestMain` lookup disabled, it got the sentinel path and `[sentinelremote]`.
|
||||
- `protontest` and `protonfake` don't import `fs/config`.
|
||||
|
||||
**(f) CI file.** YAML structure is fine. The fuzz step has no `-timeout`, but each target runs for only 20 s. `go vet` passes. `golangci-lint run ./backend/protondrive/...` gives 0 issues (run here).
|
||||
|
||||
## New Minors (fold into docs/TODO.md "final review minors")
|
||||
|
||||
- **M-a. Ranged reads use block sizes from unsigned xattrs to find their start** (`nodes.go` decryptNode → `blockSizesOnly`; `download.go:116-151`).
|
||||
- Each fetched block's size is checked, but the blocks before the range's first block are never fetched.
|
||||
- So a server that forges unsigned xattrs with block 1 at 4 MiB-100 (and true sizes after it) makes a read at offset 4 MiB-100 start at block 2 and silently return genuine bytes 100 bytes out of place. A full read still fails.
|
||||
- This is the same class as the recorded "ranged read can be fed swapped blocks" limit, and it is no worse than before the fix, when all unsigned xattrs were used. But the docs page says the sizes are ones "which reads check".
|
||||
- Cheapest fix: keep the summed Size, but read such a file as if it recorded no block sizes, so the checked block-size cache places ranges. Or log it in TODO next to the swapped-blocks entry.
|
||||
- **M-b. `TestMain` applies `RCLONE_CONFIG` only when it is set** (`budget_test.go:39-45`).
|
||||
- A developer who forgets the variable still reads the default config.
|
||||
- AGENTS.md and CI always set it, so this is a gap in a safety net, not a defect.
|
||||
- Fix: use memory-only config when the variable is unset.
|
||||
- **M-c. Draft UID when the config can't be written** (`protondrive.go` NewFs).
|
||||
- If the remote's section exists but saving the config file fails (for example a read-only file), `ConfigFileSet` still keeps the value in memory. `m.Get` then returns the random UID, so it stays random on every run, like the original I-1.
|
||||
- This is rare and logged by rclone as "Failed saving config".
|
||||
- Fix: always use the derived UID unless the value is already in the config.
|
||||
@@ -0,0 +1,193 @@
|
||||
# Final review: authentication, sessions, rclone-facing layer
|
||||
|
||||
Reviewer area: `session.go`, `account.go`, `protondrive.go`, `api/types.go`
|
||||
(as used by these). HEAD 29ad26944.
|
||||
|
||||
Counts: Critical 0, Important 2, Minor 11.
|
||||
|
||||
Scratch proof tests (worktree since removed) are saved at
|
||||
`/tmp/claude-1000/-home-user-rclone/8cc17642-4f33-4045-ad2c-177bf869b82a/scratchpad/zz_review_auth_test.go.txt`.
|
||||
|
||||
## Important
|
||||
|
||||
### I-1. Error-code table: generic code 2000 is fatal ("upgrade rclone"), and the real bad-app-version code 5003 is not mapped
|
||||
|
||||
`protondrive.go:396-397`, `api/types.go:17,37-39`.
|
||||
|
||||
`translateError` maps `CodeInvalidRequirements` (2000) and `CodeAppVersionBad`
|
||||
(2028) to `fserrors.FatalError("this version of rclone is no longer accepted
|
||||
by Proton: upgrade rclone")`. The spec (6.6) asks for this, so this re-opens
|
||||
a recorded ruling. The reason is a defect the ruling didn't foresee:
|
||||
|
||||
- 2000 is Proton's generic "requirements not met" code. The fake, which
|
||||
models the SDK, returns it for a stale `OriginalHash` on rename/move
|
||||
(`protontest/mutate.go:527,574`), a stale `CurrentRevisionID` on revision
|
||||
creation (`protontest/upload.go:365`), a folder moved into itself, and
|
||||
commit validation (`upload.go:562`). Each of these then aborts the whole
|
||||
rclone run with a false "upgrade rclone" message.
|
||||
- A realistic trigger is a lost response. `apiCall` resends PUT rename/move
|
||||
after a 5xx or network error (the recorded Task 9 deferral). The first
|
||||
request has already landed, so the resend fails its `OriginalHash` check.
|
||||
**Reproduced** (scratch test `TestReviewRenameResendFatal`): inject 502
|
||||
AfterHandler once on `PUT .../rename`. The rename lands on the server
|
||||
(`new.txt` present), but `c.rename` returns `OriginalHash doesn't match
|
||||
(Code=2000)`, and `translateError(err, opMove)` gives
|
||||
`IsFatalError == true` with "upgrade rclone". A `rclone move` therefore
|
||||
aborts the whole sync over a rename that succeeded. A second trigger:
|
||||
another client uploads a new revision between `activeNode` and
|
||||
`createRevision` in `Object.Update`.
|
||||
- 2028, by our own comment in `api/types.go:37-38`, covers "a feature is
|
||||
disabled, rate-limited or blocked for abuse". A rate-limited feature
|
||||
should not give a fatal "upgrade rclone".
|
||||
- The code Proton actually returns for a rejected app version is **5003**
|
||||
(`AppVersionBadCode`), and **5001** for a missing one. That is Proton's own
|
||||
go-proton-api v1.0.4, `response.go:24-25`, still in the module cache. Neither
|
||||
is mapped, so a blocked rclone version gives a plain retried error on every
|
||||
call, without the upgrade hint the spec intends. The fake answers a bad
|
||||
`x-pm-appversion` with 2064 (`CodeInvalidSection`, `protontest/server.go:319`),
|
||||
which the client doesn't map either.
|
||||
|
||||
Fix: map 5001/5003 (and arguably 2064) to the fatal upgrade error. Drop 2000
|
||||
from the fatal list, leaving it a plain (non-fatal) error, or
|
||||
`NoRetryError`. Reconsider 2028: NoRetry rather than Fatal, with Proton's
|
||||
message kept. Update spec 6.6 and the `TestTranslateError` code list, and
|
||||
the `AppVersionRejected` regression test (`task-15`) to use 5003.
|
||||
|
||||
### I-2. Spec 6.6: `x-pm-drive-requirements` response headers are never logged
|
||||
|
||||
Spec `docs/specs/001-protondrive-tier1.md:363`: "The `x-pm-drive-requirements`
|
||||
response headers are logged at debug level so a pending server-side
|
||||
enforcement is visible in bug reports." Nothing in `backend/protondrive`
|
||||
reads the header (grep: no match outside the spec). It is not in
|
||||
`docs/TODO.md`, `DECISIONS.md` or the handoff rulings.
|
||||
|
||||
Failure scenario: Proton announces an upcoming requirement, such as a minimum
|
||||
client version or crypto, through the header. Bug reports with `-vv` don't
|
||||
show it, which is exactly what the spec item exists to prevent.
|
||||
|
||||
Fix: in `session.do` (`session.go:253-255`, next to `observeDate`), log
|
||||
`resp.Header.Values("x-pm-drive-requirements")` at debug level when present,
|
||||
ideally once per distinct value. The alternative is to record a ruling
|
||||
dropping it.
|
||||
|
||||
## Minor
|
||||
|
||||
### M-1. A 5xx or 429 on the token refresh is never retried
|
||||
`session.go:214-216`. `call` returns the original 401 `resp` together with
|
||||
the refresh error. `shouldRetry` (`nodes.go:256`) then takes status 401 from
|
||||
`resp` and falls through to `fserrors.ShouldRetry` on an `*api.Error`, which
|
||||
is false. **Reproduced** (`TestReviewRefresh503NotRetried`): expire the access
|
||||
token and inject one 503 on `/auth/v4/refresh`. `about` fails after 1
|
||||
refresh request. A 429 with Retry-After on refresh is ignored the same way.
|
||||
Every operation in flight at token expiry can fail once (EIO on a mount).
|
||||
Fix: `return nil, refreshErr` so `shouldRetry` reads the status from the
|
||||
wrapped `api.Error` (5xx gets retried, 429 gets the default Retry-After).
|
||||
|
||||
### M-2. A refresh that answers 408 permanently kills the session in-process
|
||||
`session.go:331`. Every 4xx except 429 counts as "refresh token rejected"
|
||||
(`deadRefresh` set, errSessionExpired, fatal), and that includes 408 Request
|
||||
Timeout, a transient status that `shouldRetry` itself retries elsewhere.
|
||||
The SDK precedent covers rejections, not timeouts. Fix: exclude 408 as well.
|
||||
|
||||
### M-3. Two `Fs` in the same process with an env-configured remote can't see each other's refresh
|
||||
`protondrive.go:278-339`, `fs/configmap.go:90-94`. For a remote defined
|
||||
only by `RCLONE_CONFIG_<NAME>_*`, `Save` writes nowhere: the config file
|
||||
has no section. Each `configStore` falls back to its own `saved`, so a
|
||||
second `Fs` (for example `rclone sync remote:a remote:b`) still reads the
|
||||
spent env refresh token. It POSTs that token, is rejected, waits 3 s, and
|
||||
fails fatally with "session expired". `docs/TODO.md` records this for
|
||||
*two processes* only, and the same-process case is worse than recorded.
|
||||
Fix: document it, or share a per-remote-name in-memory overlay. That overlay
|
||||
would be package state, which needs a ruling, so documenting is likely the
|
||||
answer.
|
||||
|
||||
### M-4. Re-sign-in prompt defaults to "no", including when the user follows our own reconnect hint
|
||||
`protondrive.go:581-589` (ruling 17). An expired session's error says
|
||||
`run "rclone config reconnect remote:"`. Credentials are never removed from
|
||||
the store, so reconnect asks "Already signed in - sign in again?" with
|
||||
default No. The statement is false (the session is dead), and pressing Enter
|
||||
does nothing and exits successfully. Fix: reword the prompt ("A session is
|
||||
saved - sign in again?"). Optionally default to yes, as `oauthutil`'s
|
||||
"Token already configured - replace it?" does. That would need the ruling
|
||||
revisited.
|
||||
|
||||
### M-5. After a CAPTCHA in Config, the method question defaults back to password
|
||||
`protondrive.go:630-631`. `ConfigError("method", ...)` returns to `method`,
|
||||
whose default is `!hasPassword`, which is false. Pressing Enter goes through
|
||||
password sign-in again and hits the CAPTCHA again. Fix: go straight to
|
||||
`browser`, or make the default true after a CAPTCHA.
|
||||
|
||||
### M-6. Wrong key password on a saved session: misleading message, no reconnect hint (deferred Task 14, still present)
|
||||
`account.go:123` says "no user key unlocks, is the password right?". This is
|
||||
returned from `connect` unwrapped, without `s.reconnect`. A browser-signed-in
|
||||
remote has no password, and the fix is to reconnect (for example after a
|
||||
password change re-keys the salts). Fix: wrap with the reconnect hint in
|
||||
`connect` when `loadAccount` fails to unlock.
|
||||
|
||||
### M-7. Refresh not proven against the real server; request differs from go-proton-api
|
||||
`api/types.go:73-77`, `session.go:322-326`. The body has no `UID` and no
|
||||
`RedirectURI`, both of which go-proton-api sends
|
||||
(`manager_auth_types.go:93-100`). Handoff section 6 lists no live refresh,
|
||||
and a broken refresh would break every remote after one access-token
|
||||
lifetime. Fix: add "token refresh after expiry" to the live checks in
|
||||
`docs/TODO.md`. If it fails live, add `RedirectURI`.
|
||||
|
||||
### M-8. Deferred Task 2/3/4 session minors, re-checked, all still present and minor
|
||||
- An adopted stored token whose access token is stale fails that call once
|
||||
with a plain 401 (`session.go:308-311` then `:231`).
|
||||
- The refresh HTTP call uses the first waiter's ctx (`session.go:322`).
|
||||
- `serverTime` can step back with out-of-order responses (`session.go:280-289`).
|
||||
- The half-signed-in cleanup DELETE uses the caller's possibly cancelled ctx
|
||||
(`session.go:462-467`). An empty SRP salt is not rejected up front
|
||||
(`session.go:430`).
|
||||
- `publicKeys` waiters get the leader's ctx error (`account.go:288`).
|
||||
`GET /core/v4/users` is fetched in `loginSRP` and again in `loadAccount`.
|
||||
|
||||
### M-9. Own-address public key ring rebuilt on every call
|
||||
`account.go:278-280`. `publicKeyRing` copies every address key (`ToPublic`)
|
||||
on each call. `decryptNode` calls it 2-3 times per node through `verifyKeys`
|
||||
(`nodes.go:494,500,573`), so large listings pay thousands of key copies.
|
||||
Fix: build each address's public ring once in `loadAccount`.
|
||||
|
||||
### M-10. Server-side Copy onto an existing destination always falls back to download and re-upload
|
||||
`protondrive.go:916-918`. `copyNode` refuses an existing name (errExists maps
|
||||
to `fs.ErrorCantCopy`). protondrive has no `PartialUploads`, so rclone copies
|
||||
in place to the existing name. Every server-side copy over a changed file
|
||||
then streams the whole file through the client. It is correct but costly.
|
||||
Fix (later): on errExists, copy to a temporary name and move over, or upload
|
||||
a new revision from the source node. At minimum, record it in
|
||||
`docs/TODO.md`.
|
||||
|
||||
### M-11. Known items re-checked, still minor and recorded
|
||||
- `Precision()` is 1 s while `ModTime` carries milliseconds (handoff 8).
|
||||
- Cross-account Move/Copy/DirMove are not refused (handoff 8). The rclone
|
||||
name check makes it practically unreachable.
|
||||
- The on-the-fly remote hint names the bare backend.
|
||||
- `fs.Debugf(nil, "protondrive: ...")` is used throughout instead of
|
||||
passing the `Fs`. This is style only: maintainers usually prefer the `Fs`
|
||||
as the log object, and it is worth a sweep before any upstream PR.
|
||||
|
||||
## Checked and found sound
|
||||
- Secrets: the password is never sent (SRP) or logged. The key password,
|
||||
tokens and fork key are never in log lines or error strings. The fork URL
|
||||
holding the AES key is printed, not logged. `--dump` behaviour is as
|
||||
documented in `docs/TODO.md`.
|
||||
- Refresh: single-flight per session. It re-reads the store first, and after
|
||||
a rejection it polls up to 3 s for a token rotated elsewhere. It never
|
||||
wipes the store. `Save` writes access before refresh, so a cross-process
|
||||
reader that sees a partial write adopts only after the full set lands.
|
||||
Logout holds `refreshMu` so an in-flight refresh can't resurrect the
|
||||
session.
|
||||
- Config: browser is the default only without username+password (ruling 14).
|
||||
The third-party notice is shown. The old session is signed out only after
|
||||
the new sign-in succeeds (ruling 17). The 2FA prompt loops through
|
||||
`password_2fa`. otp codes use the server clock.
|
||||
- Disconnect/Shutdown match rulings 15 and 17. A call racing Disconnect gets
|
||||
`errNotSignedIn` (`session.go:226-229`), so the Task 14 deferral is closed.
|
||||
- Error-mapping contract: DirNotFound vs ObjectNotFound by op; Mkdir exists
|
||||
is nil; Purge refuses the root by path and by ID; Rmdir asks the server
|
||||
uncached (ruling 16); Remove of a gone file is nil; SHA1-only `Hashes`.
|
||||
- `x-pm-appversion` matches the fake's grammar for releases, betas (build
|
||||
metadata) and dev builds (alpha).
|
||||
- `unlockMu` is the only package-level mutable state. protondrive.go holds
|
||||
no crypto or routes.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Final whole-branch review — shared instructions
|
||||
|
||||
Repository: /home/user/rclone, branch protondrive-tier1, HEAD 29ad26944. Base of the
|
||||
work: 06700aaf2 (`git diff 06700aaf2..HEAD -- <paths>` shows the change; most backend
|
||||
files are new, so read them whole). This is a rewrite of rclone's Proton Drive backend
|
||||
(`backend/protondrive`) as a native Go client following Proton's official SDK, with a
|
||||
fake Proton server in `backend/protondrive/protontest` (fork-only).
|
||||
|
||||
Authorities, in order:
|
||||
1. `docs/specs/001-protondrive-tier1.md` — approved spec (binding).
|
||||
2. `docs/plans/001-protondrive-tier1-HANDOFF.md` section 7 — rulings that supersede
|
||||
the spec/plan where they disagree; section 6 — what has been proven live.
|
||||
3. `docs/DECISIONS.md` (newest first), `docs/TODO.md` (known, deliberately deferred).
|
||||
4. `AGENTS.md` — rclone code, comment and commit rules (comment style is enforced:
|
||||
godoc on every exported identifier, no narrating comments, "why" inline comments).
|
||||
|
||||
Every per-task review deferred its Minor findings to this review. They are in
|
||||
`.superpowers/sdd/001-protondrive-tier1/progress.md` (lines with "minor", "Minor",
|
||||
"Deferred minors") and in the `task-*-review.md` / `task-*-rereview*.md` files in the
|
||||
same directory. Re-check the ones in your area: still present? still minor?
|
||||
|
||||
Your job: find real defects across the whole of your area — correctness, security,
|
||||
data safety, concurrency, resource leaks, error handling, rclone contract violations,
|
||||
spec/ruling non-conformance, upstream-readiness (rclone maintainers' style, minimal
|
||||
cleverness, no package-level mutable state except `unlockMu`, seam rule: no crypto or
|
||||
API routes in protondrive.go, no fs.Fs/fs.Object below it). Do not re-litigate a
|
||||
recorded ruling unless you find it causes a defect the ruling didn't foresee; say so
|
||||
explicitly then. Do not report what docs/TODO.md already records unless it is worse
|
||||
than recorded.
|
||||
|
||||
Grade findings Critical (data loss, security break, crash on normal input) / Important
|
||||
(wrong behaviour a user will hit, contract violation, spec non-conformance, maintainers
|
||||
would reject) / Minor. Each with file:line, a concrete failure scenario, and a suggested
|
||||
fix. Verify before reporting: read the code path end to end; where cheap, prove it with
|
||||
a test in a scratch worktree (`git worktree add --detach
|
||||
/tmp/claude-1000/-home-user-rclone/8cc17642-4f33-4045-ad2c-177bf869b82a/scratchpad/fr-<you> HEAD`,
|
||||
remove with `git worktree remove <path>`, never --force).
|
||||
|
||||
SAFETY: every `go test` MUST set RCLONE_CONFIG=/notfound. Never read
|
||||
~/.config/rclone/rclone.conf or ~/proton_*password; never contact Proton hosts; never
|
||||
pass -remote or -capture-remote. Read-only on /home/user/rclone: do not modify, stage or
|
||||
commit there. Do not dispatch subagents.
|
||||
@@ -0,0 +1,232 @@
|
||||
# Final review: cryptography, node tree, caching (trust model)
|
||||
|
||||
Reviewer area: `backend/protondrive/crypto.go`, `nodes.go`, `cache.go`, the parts of
|
||||
`api/types.go` they use, and the parts of `download.go`, `upload.go` and `account.go`
|
||||
they feed into (contentKey, createRevision, unlockKey, publicKeys). HEAD 29ad26944.
|
||||
SDK reference: ~/.cache/protondrive-sdk-ref @ 05c9115 (read only).
|
||||
|
||||
All Critical and Important findings were reproduced against the fake server in a
|
||||
scratch worktree (now removed). The probes used small scratch-only helpers in
|
||||
`protontest` that model what a hostile server can do with only public keys:
|
||||
encrypt to a node's public key, and drop or replace signatures.
|
||||
|
||||
Counts: Critical 2, Important 3, Minor 8.
|
||||
|
||||
---
|
||||
|
||||
## Critical
|
||||
|
||||
### C1. An unsigned content key packet is accepted. Updates are then encrypted with a key the server chose, and ranged reads return content the server made up
|
||||
|
||||
- `crypto.go:614-620` (`decryptContentKey` checks the signature only
|
||||
`if contentKeyPacketSig != ""`), `download.go:191-227` (`contentKey`),
|
||||
`upload.go:312` (`createRevision` encrypts the new revision's blocks with
|
||||
`c.contentKey(n)`).
|
||||
- Anyone who has the node's public key can make a content key packet: generate a
|
||||
session key and encrypt it to that key. The server stores the armored node key, so it
|
||||
has the public key. If the server sends `ContentKeyPacketSignature: ""` (or null),
|
||||
`verifyErr` stays nil and nothing is logged, even without `ignore_signature_errors`.
|
||||
- **Confidentiality break (reproduced).** A file with a verified node key: the server
|
||||
swaps in its own content key packet and blanks the signature. `Object.Update` (every
|
||||
`sync` of a changed file) goes ahead without an error or a log line and encrypts
|
||||
the user's new content with the server's session key. The probe then decrypted the
|
||||
stored block of the new revision with the server's key and got
|
||||
`"the user's new secret content"` back.
|
||||
- **Integrity break (reproduced).** The server also re-encrypts block 1 under its key
|
||||
(the server lists the hash, and the size matches the recorded block size). A
|
||||
ranged read (`RangeOption{Start:1}`, as mounts and multi-thread copies make)
|
||||
returned `"ORGED BY SERVER"` with no error. A full read failed on the manifest.
|
||||
But a server that also renames the revision's `SignatureEmail` to a foreign
|
||||
address and serves keys for it (the known protocol limit) can sign a manifest
|
||||
over its own block hashes, so full reads can be forged too. docs/TODO.md says a
|
||||
ranged read can be fed "validly encrypted blocks of the same file swapped or from
|
||||
an older revision". This is worse: the content can be anything the server likes.
|
||||
- The SDK also skips verification when the signature is absent
|
||||
(`driveCrypto.decryptAndVerifySessionKey`, `cryptoService.ts:240-249`). But rulings
|
||||
18/19 and spec section 7 put the server in the threat model and are stricter than
|
||||
the SDK on the node key. A key the server can mint is the same hole one level down.
|
||||
- Fix: in `decryptContentKey`, return `errSignatureMissing` as `verifyErr` when the
|
||||
signature is empty, so `contentKey` refuses without the flag. If legacy unsigned
|
||||
content keys exist, allow the exemption only by a creation date, the way names and
|
||||
hash keys do, and say that the date is server-supplied. Add a hostile case to
|
||||
`hostileContents` and a write case ("Update of a file whose content key is
|
||||
unsigned is refused").
|
||||
|
||||
### C2. Node keys from the server are unlocked with whatever Argon2 parameters they carry, up to a 2 TiB allocation
|
||||
|
||||
- `crypto.go:431` (`decryptNodeKeys` → `account.go:204` `unlockKey` →
|
||||
`crypto.Key.Unlock`). go-crypto accepts an Argon2 S2K with memory exponent up to
|
||||
31 (`s2k.go:431`), which is 2^31 KiB, and runs the KDF before it can tell that the
|
||||
passphrase is wrong.
|
||||
- The armored `NodeKey` is server data. For nodes in a folder the user has shared
|
||||
for editing, or in a public upload link folder, it is written by other people's
|
||||
clients. A valid encrypted passphrase is easy to supply: copy any node's, or
|
||||
encrypt any string to the parent's public key. The passphrase signature is checked
|
||||
only after the unlock and only produces `verifyErr`. So listing the folder runs
|
||||
the KDF.
|
||||
- Reproduced (scratch internal test): a node key locked with Argon2 memory 2^19 KiB
|
||||
and `unlockKey` called with a wrong passphrase took 0.47 s and allocated 512 MiB
|
||||
before it returned "error in unlocking key". At exponent 24 to 31 that is 16 GiB
|
||||
to 2 TiB: an OOM kill or `fatal error: runtime: out of memory`, which nothing can
|
||||
recover. Every listing that includes the node crashes the process (rclone mount,
|
||||
sync). `passes` (up to 255) multiplies the CPU cost on top.
|
||||
- Not in docs/TODO.md. The decompression-bomb entries cover message sizes, not key
|
||||
unlocking. The same function unlocks address keys in `account.go`, which is outside
|
||||
this area but has the same cause.
|
||||
- Fix: before `Unlock`, walk the private key packets (`key.GetEntity().PrivateKey`
|
||||
and subkeys, `PrivateKey.S2KParams`/`s2k.Params.Mode()` or by parsing the packet).
|
||||
Refuse a node key whose S2K is not iterated-and-salted. `crypto.go:374-375`
|
||||
already notes that Argon2-locked node keys break the official web app, so no
|
||||
genuine node key uses Argon2. Make it an unreadable node. Optionally bound the
|
||||
RSA modulus size too.
|
||||
|
||||
---
|
||||
|
||||
## Important
|
||||
|
||||
### I1. A hash key whose signature fails or is missing is accepted and used for writes, so the server can test guesses at the names written
|
||||
|
||||
- `nodes.go:550-556` only logs `hashKeyVerifyErr`. `targetFolder` (`nodes.go:964-988`)
|
||||
checks `hashKey != nil` and the key's `unverified`, not the hash key's signature.
|
||||
`createFolder`, upload, rename, move and copy then compute
|
||||
`nameHash(name, parent.hashKey)`.
|
||||
- Reproduced: the server replaced folder `dir`'s hash key with an unsigned one of
|
||||
its own, encrypted to the folder's public key. rclone logged one ERROR, uploaded
|
||||
`dir/tax-return-2026.pdf`, and the stored `Hash` equalled
|
||||
HMAC(server key, "tax-return-2026.pdf"). The server can now test names against a
|
||||
dictionary, or confirm guesses, for everything rclone writes into that folder.
|
||||
Name hashes exist so that it can't.
|
||||
- The SDK folds hash-key and folder-xattr author failures into the node's
|
||||
`keyAuthor` (`cryptoService.ts:265-281`, `finalKeyAuthor`). Under rulings 18/21,
|
||||
rclone's equivalent is `keys.unverified`, but rclone folds in only the passphrase
|
||||
signature. This is a defect the rulings didn't foresee, not a re-litigation of
|
||||
them.
|
||||
- The `hashKeySignedSince` exemption (`nodes.go:551`) is decided by the
|
||||
server-supplied `CreateTime`. A server that back-dates the folder silences even the
|
||||
ERROR line.
|
||||
- Fix: in `targetFolder`, refuse a folder whose hash key signature failed unless
|
||||
`ignore_signature_errors` is set, using the same message shape as an unverified
|
||||
key. The smallest way is to keep `hashKeyVerifyErr` on the node, or fold it into
|
||||
`keys.unverified` for folders. Document that the date exemption depends on a
|
||||
server value.
|
||||
|
||||
### I2. Renaming or moving a node whose name signature failed reuses a session key the server can know, so the server reads the new name
|
||||
|
||||
- `nodes.go:1149-1157` (`rename`) and `nodes.go:1218-1224` (`encryptForNewParent`)
|
||||
reuse `nameSessionKey(n.link.Link.Name, ...)`. The name signature failure is only
|
||||
logged (`nodes.go:505-516`, docs/TODO.md "a name whose signature fails is listed").
|
||||
- Reproduced: the server replaced file `real.txt`'s encrypted name with
|
||||
`"shown.txt"` under a session key it generated, unsigned. rclone listed
|
||||
`shown.txt` with one ERROR. `operations.Move` to `secret-new-name.txt` succeeded,
|
||||
and the server decrypted the new stored name with its own session key:
|
||||
`"secret-new-name.txt"`. Back-dating `CreateTime` before 2021 removes the ERROR
|
||||
line too (`nameSignedSince`).
|
||||
- docs/TODO.md records only that names can be swapped and shown, not that they leak
|
||||
on rename or move. This is a confidentiality loss, not just mislabelling.
|
||||
- Fix: reuse the old name's session key only when the name's signature verified.
|
||||
Otherwise use a new session key (`encryptName(newName, nil, ...)`), or refuse
|
||||
without the flag. Keep `nameVerifyErr` on the node to decide.
|
||||
|
||||
### I3. Extended attributes whose signature is missing or invalid are used without `ignore_signature_errors`
|
||||
|
||||
- `nodes.go:557-580`: the xattr `verifyErr` is only logged and `n.X` is kept. Spec
|
||||
6.4 says `ignore_signature_errors` "downgrades content and authorship signature
|
||||
failures to a logged ERROR", so without it they should fail. The SDK treats a
|
||||
revision's xattr signature as `contentAuthor`.
|
||||
- Anyone with the node's public key can write xattrs. Reproduced: unsigned xattrs
|
||||
claiming `ModificationTime` 2099-01-01. rclone logged one ERROR,
|
||||
`Object.ModTime` returned 2099, and the file read fine. Consequences: the server
|
||||
can set modtime, size and SHA1 at will. For example, `rclone sync --update` to
|
||||
the remote then skips every changed file the server has marked as newer, so a
|
||||
backup silently stops updating. Forged SHA1 or size gives false `check` results
|
||||
and failed copies. Integrity of the content itself holds: block sizes are checked
|
||||
on read.
|
||||
- Not in docs/TODO.md (only names are recorded).
|
||||
- Fix: when the xattr signature fails and `ignore_signature_errors` is not set,
|
||||
drop the attributes (`n.X = nil`, as for undecryptable ones). The file then shows
|
||||
the encrypted size and the server's modtime, and the log says why.
|
||||
|
||||
---
|
||||
|
||||
## Minor
|
||||
|
||||
### M1. A node fetched while it is being changed can overwrite the newer cached node and stay cached
|
||||
`nodes.go:724` `cache.put(n)` after `fetchLinks`. `cache.go:52-63` has no
|
||||
per-node generation, unlike the per-folder change count of ruling 27. Goroutine A
|
||||
starts `fetchNode(X)` (cache miss) and receives revision R1. Goroutine B commits
|
||||
R2 and calls `cache.add(X@R2)`. A then calls `put(X@R1)`, which wins. With no
|
||||
expiry, reads through the cache serve R1's (still stored, obsolete) content until
|
||||
a re-list, and the next Update fails on a stale `CurrentRevisionID`. The window is
|
||||
narrow: concurrent ops on one object in one process, e.g. a mount. Fix: give
|
||||
`put` a token from before the fetch (a global or per-ID change count, bumped by
|
||||
`invalidate` and `add`), as `startFetch`/`endFetch` do.
|
||||
|
||||
### M2. Unbounded per-client maps
|
||||
`client.folderKeys` (every folder ever seen, private keys and passphrases, kept
|
||||
whether or not caching is on; Task 8 minor, still present), the node cache (no
|
||||
size bound or expiry), `client.reported` (one entry per node ID that logged a
|
||||
problem), and `account.foreign` (one key ring per distinct foreign email a server
|
||||
names). A long mount or a hostile server grows them without limit. `close()`
|
||||
still doesn't clear the private keys of file nodes held by the cache (Task 8).
|
||||
|
||||
### M3. `listChildren` loops forever when the server keeps answering `More` with the same `AnchorID`
|
||||
`nodes.go:837-857`. No progress check: a buggy or hostile server keeps rclone
|
||||
requesting until ctx ends, with `seen` and `all` growing if the IDs change. Fix:
|
||||
stop with an error when a page adds no new ID or repeats the anchor.
|
||||
|
||||
### M4. Root-of-trust and signature date exemptions use server-supplied times
|
||||
`nameSignedSince`, `hashKeySignedSince` (`nodes.go:514`, `:551`) and
|
||||
`contentKeyFallbackUntil` (`download.go:204`) read `Link.CreateTime`, which the
|
||||
server controls. Only the first two waive a check. The SDK does the same. Worth a
|
||||
comment and a docs/TODO.md line, as it feeds I1 and I2.
|
||||
|
||||
### M5. `decryptBlock` is used only by tests
|
||||
`crypto.go:684-696`. Production uses `blockDecrypter`. `reencryptPassphrase`
|
||||
(`crypto.go:448-450`) is a one-line alias of `encryptPassphrase`. Maintainers will
|
||||
ask for both to go: move `decryptBlock` to a test file and call
|
||||
`encryptPassphrase` directly.
|
||||
|
||||
### M6. `truncateName` can cut a UTF-8 sequence
|
||||
`nodes.go:599-605` slices bytes at 64, so log lines can carry invalid UTF-8. Cut
|
||||
on a rune boundary.
|
||||
|
||||
### M7. Earlier deferred minors re-checked
|
||||
- Still present: the node passphrase is not checked as valid UTF-8 (Task 5,
|
||||
`crypto.go:429`). `publicKeys` waiters get the fetching caller's ctx error
|
||||
(Task 4, `account.go:288-303`). `setChildrenLocked` overwrites a cached node
|
||||
without the changed-parent rule `put` applies, so after an external move the
|
||||
old folder's cached listing can still return it (Task 9, `cache.go:100-107`;
|
||||
only matters with milestone M5 events or a second `Fs`). An xattr `"Size": null` reads
|
||||
as 0 and skips the block-size fallback (Task 17, `crypto.go:848`; recorded in
|
||||
the handoff). Move's anonymous detection uses `SignatureEmail == ""` (Task 9,
|
||||
`nodes.go:1231`; only reachable with the flag after `checkVerified`).
|
||||
- Closed: `cache.put` leaving a moved node in its old parent's listing (Task 8),
|
||||
and a negative xattr Size with block sizes (Task 17; the fallback now sums the
|
||||
block sizes). The root name not being validated (Task 8) has no effect: it is
|
||||
never used as a path.
|
||||
|
||||
### M8. `publicKeyRing` is rebuilt on every own-address lookup
|
||||
`account.go:278-279`. Each `decryptNode` of a node signed by the user calls
|
||||
`ToPublic` on every address key (up to three lookups per node). Cache the public
|
||||
ring on the address. Performance only.
|
||||
|
||||
---
|
||||
|
||||
## Checked and found sound
|
||||
- Transitive distrust: `decryptNode` inherits `parent.unverified` before its own
|
||||
check. `createFolder` copies the parent's. Uploads into an unverified folder read
|
||||
back through `decryptNode`. `checkVerified` gates rename, move and copy. Root of
|
||||
trust (`bootstrap`): share creator and root signer through `acct.owns`
|
||||
(ASCII-only lowering, `emailKey`), and own addresses are never looked up remotely.
|
||||
- `decryptMessage` decrypts with the session key alone, so holders of decKR can't
|
||||
pass as signers. The intended-recipient check is sound. Decompression is bounded
|
||||
for metadata (`maxSmallMessageSize`, `maxXAttrsSize`) and for blocks
|
||||
(`LimitReader(maxBlockSize+1)` in `fetchOnce`). Integrity checks (block hash,
|
||||
sizes, block count, re-listed hash) do not depend on the flag.
|
||||
- Name validation on read and write: empty, `.`, `..`, `/`, NUL, UTF-8, and 255
|
||||
UTF-16 units (`utf16.RuneLen` is never -1 on valid UTF-8).
|
||||
- Parent chains: cycle detection over `below`, the depth cap of 2048, and a
|
||||
parentless non-root refused ("not in the drive").
|
||||
- Listing cache (ruling 27): `startFetch`/`endFetch` with change counts. `add`,
|
||||
`invalidate` (including an unknown parent, which bumps every listing in
|
||||
progress), `move` (both parents) and `trash` bump correctly.
|
||||
@@ -0,0 +1,212 @@
|
||||
# Final whole-branch review: tests, fake, harness, CI, docs, hygiene
|
||||
|
||||
Reviewer area: everything that is not production backend code. HEAD 29ad26944,
|
||||
base 06700aaf2. Verified in a scratch worktree (removed afterwards).
|
||||
|
||||
## Checks run
|
||||
|
||||
- `RCLONE_CONFIG=/notfound go test -race -count=1 ./backend/protondrive/...` on
|
||||
16 cores: PASS. `backend/protondrive` 333.6 s, `protontest` 38.3 s,
|
||||
`protonfake` 1.4 s.
|
||||
- The same package pinned to 2 CPUs (`taskset -c 0-1`, `-timeout 60m`) to
|
||||
approximate the slow Gitea runner: PASS in 420.7 s (70% of Go's default 10 min test timeout), no failures, timing assertions included.
|
||||
- `golangci-lint run ./backend/protondrive/...`: 0 issues. `go vet`: clean.
|
||||
- `GOPROXY=off go mod tidy -diff`: no diff. go.mod/go.sum: Proton-API-Bridge,
|
||||
go-proton-api, gluon, resty, goquery/cascadia, go-message/vcard, juniper,
|
||||
logrus, relvacode/iso8601 all gone; go-srp and gopenpgp/v3 promoted to direct.
|
||||
Nothing unexpected.
|
||||
- `TestIntegrationFake -v`: 97 subtests pass, 29 skip (matches the handoff).
|
||||
- Test-account usernames (recovered from the pre-rewrite object 901d7fd5b,
|
||||
which is still in the local object store): 0 hits in HEAD's tracked files,
|
||||
text or binary (`git grep -a`). No tokens, UIDs or 32-hex IDs in committed
|
||||
docs/testdata. Largest tracked files outside upstream's MANUAL.* are the three
|
||||
golden CLI blocks (4 MiB, 4 MiB, 1 MiB), as intended.
|
||||
- shellcheck on `init.d/TestProtonDriveFake`: SC2034 only (known minor).
|
||||
- Live-reach audit: every `NewFs`/`configmap.Simple` in the tests carries the
|
||||
fake's `endpoint`, except unit tests of the config store that never connect.
|
||||
`TestIntegration` is gated only by the remote being configured (rclone's norm;
|
||||
hence the standing `RCLONE_CONFIG=/notfound` rule) and skips during a golden
|
||||
capture; `TestGoldenCapture` needs all three `-capture*` flags;
|
||||
`TestIntegrationFake` skips under `-remote`. No other test can reach Proton.
|
||||
|
||||
## Critical
|
||||
|
||||
None.
|
||||
|
||||
## Important
|
||||
|
||||
### I-1. CI: `go test -race` has no `-timeout`; the package will hit Go's default 10 min on the slow runner
|
||||
|
||||
`.gitea/workflows/ci.yml:43` runs `go test -race ./backend/protondrive/...`
|
||||
with Go's default per-binary `-timeout 10m`. The `backend/protondrive` test
|
||||
binary alone takes 333 s on 16 cores under `-race`; pinned to 2 of this host's fast cores it took 420.7 s, 70% of the 10 min budget, so a runner whose cores are ~1.5x slower (shared VM, older CPU, the build cache cold on top) crosses it. The
|
||||
job's `timeout-minutes: 45` never comes into play: the test binary panics with
|
||||
"test timed out after 10m0s" first, and the job fails with a goroutine dump
|
||||
rather than a result. The owner described the runner as "slow and not very
|
||||
capable".
|
||||
|
||||
Fix: `go test -race -timeout 40m ./backend/protondrive/...` (inside the 45 min
|
||||
job budget), or drop `-race` on CI and keep a non-race run (much faster) with a
|
||||
nightly race job.
|
||||
|
||||
### I-2. CI: `go-version-file: go.mod` installs Go 1.26.0, so `govulncheck` will very likely fail on standard-library advisories
|
||||
|
||||
`ci.yml:22-27` takes the Go version from `go.mod`'s `go 1.26.0` directive (no
|
||||
`toolchain` line). `actions/setup-go` treats that as an exact version and
|
||||
installs 1.26.0, a release nine months older than the 1.27.4 every local
|
||||
verification used. `ci.yml:51` then runs `govulncheck@latest` against it;
|
||||
govulncheck reports reachable standard-library vulnerabilities fixed in later
|
||||
patch releases (net/http, crypto/tls, crypto/x509 are reached by this backend)
|
||||
and exits non-zero. So the first push most likely goes red on a finding that is
|
||||
about the CI toolchain, not the code; it also means CI tests on a compiler no
|
||||
one has run the suite with. Could not be proven offline (needs vuln.go.dev).
|
||||
|
||||
Fix: `go-version: '~1.27.3'` (or `stable`), matching upstream's
|
||||
`.github/workflows/build.yml:290` lint job; pin govulncheck to a version
|
||||
instead of `@latest` so a new govulncheck release can't break the build.
|
||||
|
||||
### I-3. The committed workspace copy of the ledger stops at the 2026-11-11 pause; the handoff tells a new agent to restore from it
|
||||
|
||||
`docs/plans/001-protondrive-tier1-HANDOFF.md:96-108` says a copy of every
|
||||
workspace file "is committed at `docs/plans/001-protondrive-tier1-workspace/`"
|
||||
and to resume with `cp docs/plans/001-protondrive-tier1-workspace/*.md
|
||||
.superpowers/sdd/...`; line 25 calls `progress.md` "the full execution ledger
|
||||
(every ruling, every deferred minor, every live probe)". The committed
|
||||
`progress.md` is 175 lines and ends "PAUSED 2026-11-11 ... Tasks 18–21 not
|
||||
started"; the live ledger (gitignored `.superpowers/`) is 244 lines. Missing
|
||||
from git: all Task 18–21 ledger entries (rulings on the lag fix and on
|
||||
draft deletion, the live test_all results, the browser sign-in proof, the
|
||||
config-create token incident), and 21 files: task-18/19/20/21/21f/21g
|
||||
briefs, reports, reviews and re-reviews. On another machine, or after the
|
||||
`.superpowers/` directory is cleaned, a new agent loses the deferred minors of
|
||||
Tasks 18–21 and the reasoning behind rulings the handoff only summarises.
|
||||
|
||||
Fix: re-copy `.superpowers/sdd/001-protondrive-tier1/*.md` into the workspace
|
||||
directory (and do it again after this final review) and commit.
|
||||
|
||||
### I-4. Handoff section 3 still describes the pre-Task-18 state
|
||||
|
||||
A new agent reading top-down is told things that are false now:
|
||||
`HANDOFF.md:3` "Written 2026-11-11 when development was paused"; `:50` "Last
|
||||
code commit: `9c90998f7`" (it is be0053607 / f8e398fce); `:76` verification
|
||||
"at `9c90998f7`"; `:82-91` "Not done, and easy to assume otherwise": user docs
|
||||
"still describe the old backend (Task 20)", wider suites "have never been run
|
||||
against the new backend (Task 18)", "No fork CI exists (Task 18)" — all three
|
||||
are done; `:130` "Remaining tasks" lists 18–21 as remaining; section 8
|
||||
(`:463-466`) says only Tasks 14, 16 and 17 have review files. The table at
|
||||
`:55-57` and section 6 are current, so the document contradicts itself.
|
||||
|
||||
Fix: rewrite section 3's header, commit references and "Not done" list (the
|
||||
real not-done list is: whole-branch review fix wave, 2FA/CAPTCHA/desktop
|
||||
clients live, root of trust on a Proton-made drive, the five cross-Fs tests),
|
||||
and mark the "Remaining tasks"/"Notes for Tasks 18 to 21" subsections as
|
||||
historical or delete them.
|
||||
|
||||
### I-5. The upstream recipe is self-contradictory: the upstream branch would not compile its tests
|
||||
|
||||
Spec 13.4 (`docs/specs/001-protondrive-tier1.md:729-732`) says the upstream
|
||||
branch contains `backend/protondrive/` and the `fstest/test_all/config.yaml`
|
||||
change; spec 14.4, the handoff (`:2`, `:505-511`) and ARCHITECTURE say
|
||||
`backend/protondrive/protontest` never goes upstream. 13 of the 17 test files
|
||||
import `protontest` (all but `crypto_internal_test.go`, `cache_internal_test.go`,
|
||||
`export_test.go`, `protondrive_test.go`), so leaving it out makes
|
||||
`go test ./backend/protondrive/` fail to build upstream, and the
|
||||
`TestProtonDriveFake:` entry in `config.yaml:616-630` refers to an init.d
|
||||
script and port that are also fork-only. Upstream is deferred, so this isn't
|
||||
a defect today, but whoever cuts the branch per the written recipe produces a
|
||||
broken PR, and the handoff (section 9) lists no remedy. Ties to open question
|
||||
14.6.
|
||||
|
||||
Fix: record in the handoff section 9 and spec 13.4 that either protontest goes
|
||||
upstream with the tests (and question 14.6 must be answered first), or the
|
||||
upstream branch drops every protontest-dependent test file plus the
|
||||
`TestProtonDriveFake` config.yaml entry, PORTS.md row and init.d script, and
|
||||
keeps only `protondrive_test.go` and the two standalone internal tests.
|
||||
|
||||
## Minor
|
||||
|
||||
1. **Timing assertions in parallel tests are tight for a slow `-race`
|
||||
runner.** `nodes_internal_test.go:503` and `:523` assert a plain
|
||||
`about()` call finishes in under 500 ms inside `t.Parallel()` tests
|
||||
that run alongside ~120 other parallel tests doing multi-MiB crypto under
|
||||
the race detector; `:487-488` (`TestRetryAfterHonoured`) asserts a 2 s Retry-After call under 5 s.
|
||||
They passed on the 2-CPU run, so this is a margin concern, not an observed flake. Fix: 2-5 s bounds, which still catch a held pacer (the
|
||||
cases guard against a 999999999 s or 60 s hold).
|
||||
2. **Goroutine-count leak checks remain** (`download_internal_test.go:105-118`,
|
||||
`nodes_internal_test.go:582-596`, known). They run in sequential tests, so
|
||||
false failures need a stray goroutine started by something else; the
|
||||
`<= baseline` form can also pass vacuously when goroutines of an earlier
|
||||
test exit during the window. Recorded flake of `TestUploadShortSource` in the
|
||||
handoff. Consider `go.uber.org/goleak`-style filtering is not available
|
||||
without a dependency; keep, but don't expand the pattern.
|
||||
3. **`TestListingRacingChange` can hang to the package timeout**
|
||||
(`nodes_internal_test.go:1222-1228`, known, still present): `<-held` has no
|
||||
timeout and `release` is closed only after `require.NoError(test.change)`;
|
||||
a failure leaves the fake's handler blocked and `httptest.Server.Close`
|
||||
waiting. Fix: `t.Cleanup(func(){ close(release) })` guarded by sync.Once,
|
||||
and a `select` with `time.After` on `held`.
|
||||
4. **`protonfake` `TestRun` leaks the `run` goroutine on a failed require**
|
||||
(`protontest/cmd/protonfake/main_test.go:17-34`, known, still present).
|
||||
Fix: `t.Cleanup` closing `stop`.
|
||||
5. **`TestClientBootstrapCreateVolumeRace` waits a fixed 2 s** (`nodes_internal_test.go:128-147`,
|
||||
known, still present).
|
||||
6. **`TestUploadBlock502Once` doesn't check the fault fired**
|
||||
(`upload_internal_test.go:230-237`): a mistyped pattern would let it pass
|
||||
as a plain upload. Add `assert.Equal(t, 2, srv.Requests()[storage route])`
|
||||
or similar.
|
||||
7. **`TestAcquireTokenCancel` sleeps 20 ms to "let the wait start"**
|
||||
(`upload_internal_test.go:479`): on a slow runner the goroutine may not
|
||||
have started, so the cancelled-while-waiting path is not exercised (the
|
||||
test still passes via the early ctx check). Harmless, not a flake.
|
||||
8. **No test pins the 408 exclusion of `refusedCommit`** (21f re-review N3,
|
||||
still open).
|
||||
9. **Task 20 re-review nits still present**: `download_internal_test.go:621`
|
||||
names the unrecorded case "assumed"; `:178` `min(v.blocks, 1)*3` is always 3.
|
||||
10. **init.d `stop` has no SIGKILL escalation, SC2034 unused `i`**
|
||||
(`fstest/testserver/init.d/TestProtonDriveFake:51-62`, Task 18 minors,
|
||||
still present).
|
||||
11. **CI hygiene**: `on: push` plus `pull_request` with no `concurrency` group
|
||||
runs twice per PR push on a slow runner; `govulncheck@latest` is
|
||||
unpinned (see I-2).
|
||||
12. **The fake's read lag is never combined with the contract suite.**
|
||||
`Server.SetReadLag` (`protontest/drive.go:1749-1757`) models the live
|
||||
read-after-write lag that broke `TestPurge`/vfs on the real server, but only
|
||||
targeted upload tests turn it on; `TestIntegrationFake` and `test_all`
|
||||
against the fake run with no lag, so the next lag-intolerant path
|
||||
(`activeNode`, recorded in TODO) can't show up offline. Consider a second
|
||||
`fstests.Run` (or a `protonfake -read-lag` flag for `test_all`) with a lag
|
||||
of 1-2 reads.
|
||||
13. **Conformance checklist contradicts ruling 11**:
|
||||
`docs/specs/001-conformance-checklist.md:27` (upload row) says "a draft is
|
||||
deleted only on positive proof it never committed"; since be0053607 the
|
||||
rule is "only when the server explicitly refused the commit, on a single
|
||||
unresent attempt; no read-back decides deletion".
|
||||
14. **User docs: the second "unknown" error isn't documented.**
|
||||
`docs/content/protondrive.md:289-293` names only "upload outcome unknown";
|
||||
`errUploadUnconfirmed` ("upload committed, but the server doesn't show it
|
||||
yet", `upload.go:43`) gets the same no-retry treatment and is what a user
|
||||
will see under server lag.
|
||||
15. **User docs claim behaviour proven only against the fake**:
|
||||
CAPTCHA (`protondrive.md:35-37`), 2FA prompting (`:131-147`) and "`rclone
|
||||
about` reports the whole account" (`:548`) are unverified live, while
|
||||
`docs/TODO.md` says the docs "avoid claiming" 2FA and CAPTCHA. Either soften
|
||||
or fix the TODO wording.
|
||||
16. **Lost TODO from the live run**: the ledger's 2026-11-12 incident note
|
||||
("`rclone config create` printed the new remote's client_* tokens and
|
||||
salted key pass to stdout ... check whether client_* options are marked
|
||||
Sensitive") is not in `docs/TODO.md`. The options are `Sensitive`
|
||||
(`protondrive.go:113-142`), but `config create`/`config show` print
|
||||
unredacted (`fs/config/ui.go` `ShowRemote`) — generic rclone behaviour,
|
||||
yet worth a docs line ("the output of `rclone config create` contains the
|
||||
session; don't paste it") and a TODO entry, since the salted key password
|
||||
survives a logout.
|
||||
17. **`docs/PERMISSIONS.md` / `.codex/` out of step**: `golangci-lint run` was
|
||||
added to the Allow tier and to `.claude/` and `.gemini/`, but not to
|
||||
`.codex/rules/project.rules`, against AGENTS.md's "change them together".
|
||||
18. **Old username-bearing commit still in the local object store**
|
||||
(`901d7fd5b`, reachable from the reflog). Not tracked and not on the
|
||||
remote after the force-push; it disappears with reflog expiry/gc. Mention
|
||||
it in the handoff so a `git push --mirror` or a backup bundle of this
|
||||
checkout isn't made before then.
|
||||
19. **Handoff ruling 13** still reads as current ("stays in `go.mod` until
|
||||
Task 19"); mark it done (Task 19 review minor, still present).
|
||||
@@ -0,0 +1,160 @@
|
||||
# Final review: data transfer (upload.go, download.go, Object Open/Update/Put/PutStream)
|
||||
|
||||
Reviewer area: `backend/protondrive/upload.go`, `backend/protondrive/download.go`,
|
||||
`protondrive.go` Object.Open/Update, Fs.Put/PutStream, `api/types.go` as far as these use it.
|
||||
HEAD 29ad26944. `go test -race -run 'TestUpload|TestDownload|TestAcquire|TestIntended'`
|
||||
passes in a scratch worktree (40 s).
|
||||
|
||||
Counts: Critical 0, Important 2, Minor 9.
|
||||
|
||||
## Data-safety walk (no defect found)
|
||||
|
||||
I traced every path that can delete something during an upload:
|
||||
|
||||
- `upload` -> `deleteDraft` runs only when `uploadDraft` reports `committed=false`. That
|
||||
happens before the commit (verification, blocks, `check`, manifest, xattrs) or when
|
||||
`refusedCommit` is true on `attempts == 1` (upload.go:403). `refusedCommit` needs an
|
||||
`*api.Error` with a Proton code and a 4xx status other than 408, so a 200 response with an
|
||||
error envelope, an HTML 4xx and a 5xx all leave the draft. Any pacer resend (5xx, 408, 429,
|
||||
network) makes `attempts > 1`.
|
||||
- The `session.call` 401 -> refresh -> resend happens inside one pacer attempt. The first
|
||||
send got a 401, so the server never processed it. If the refresh then fails with a coded
|
||||
4xx, `refusedCommit` matches the refresh's error and the draft is deleted. That is safe,
|
||||
because the commit was never processed. Go's transport does not replay a PUT unless
|
||||
nothing was written.
|
||||
- When ctx is cancelled, `fserrors.ContextError` in `shouldRetry` turns even a successful
|
||||
commit into a ctx error. The detached single read (upload.go:411-414) then still finds the
|
||||
committed revision.
|
||||
- `resolveConflict` deletes a draft (with `delete_multiple`) in only two cases: the details
|
||||
show a pure draft node (`ConflictRevisionID == ""` and a draft revision) left by our UID,
|
||||
or `replace_existing_draft` is set. It never deletes on a retry. A file conflict becomes
|
||||
a new revision, and a folder, draft or trashed node is refused.
|
||||
- The read back after a commit never decides deletion. Lag gets about 7 s of reads that
|
||||
stop when ctx is done. A commit that is not ours ends as Unconfirmed, Superseded or
|
||||
OutcomeUnknown, and in each case only the cache is forgotten.
|
||||
|
||||
The upload pipeline (readBlocks -> requestURLs -> workers) frees every block on every
|
||||
path: the worker frees after upload, `requestURLs` frees on a URL error or cancel, and
|
||||
`readBlocks` frees on cancel. Both channels are unbuffered, so no block is stranded. Slots
|
||||
and tokens are always given back, and an abandoned `acquireToken` waiter returns its token.
|
||||
The size and SHA1 checks, the early long-source failure, the empty file and the
|
||||
exact-multiple-of-4 MiB case are all correct.
|
||||
|
||||
On the download side: the full read lists every block and checks the manifest before any
|
||||
fetch. Per-block hash and recorded-size checks are bounded by `maxBlocks` and
|
||||
`maxFileBlocks` before `r.sizes[b.Index-1]` is indexed, so that index cannot go out of
|
||||
range. A range with sizes not recorded starts from block 1 with `skip=offset`. A
|
||||
whole-file range goes to `openWhole` after `FixRangeOption`. `Close` drains every slot.
|
||||
|
||||
## Important
|
||||
|
||||
### I-1 `draft_client_uid` is not stable for remotes from environment variables or connection strings, so a leftover draft blocks its name
|
||||
|
||||
protondrive.go:652-657, upload.go:292-297.
|
||||
|
||||
`NewFs` makes a random UID and calls `m.Set`. `fs.setConfigFile.Set` (fs/configmap.go:90-94)
|
||||
saves nothing unless the remote is a section of the config file. A remote set up with
|
||||
`RCLONE_CONFIG_X_*` variables (typical in Docker and CI) or a connection string therefore
|
||||
gets a new UID on every run, and on every `NewFs` in the same process. By design, drafts
|
||||
are now left after every ambiguous commit failure, and also when the process dies
|
||||
mid-upload. On the next run the UID differs, so `resolveConflict` sees a foreign draft and
|
||||
fails with "another client is uploading the file (set replace_existing_draft...)". That
|
||||
repeats on every run for that name until the user sets the option.
|
||||
|
||||
This breaks spec 6.3 step 1 ("a stable per-remote draft client ID") for a common way of
|
||||
running rclone.
|
||||
|
||||
Fix: when the value cannot be saved, derive it deterministically instead of randomly. For
|
||||
example, use a hash of the account's user ID (known after sign-in) plus the remote name, or
|
||||
make the random UID only when `fs.ConfigFileHasSection(name)` and derive it otherwise. At
|
||||
the least, document it and log a NOTICE when the UID isn't persisted.
|
||||
|
||||
### I-2 Multi-thread copies of files that record a size but not block sizes download data quadratically
|
||||
|
||||
download.go:142-151, protondrive.go:1150-1173.
|
||||
|
||||
`docs/TODO.md` records that a ranged read of such a file "downloads from the first block
|
||||
every time" and is slow on a mount. The case below is worse than that entry says.
|
||||
|
||||
`Size()` is known (xattr `Size`), so rclone's multi-thread copy turns on by itself above
|
||||
`--multi-thread-cutoff` (256 MiB) when copying to local. Each 64 MiB chunk is a separate
|
||||
`Open` with a `RangeOption`, and each one downloads and decrypts from block 1 up to the end
|
||||
of its chunk. It also lists every block page to the end of the file, because `last = -1`
|
||||
when sizes aren't recorded. Total transfer is about S²/(2·64 MiB): about 8.5 GiB for a
|
||||
1 GiB file and about 800 GiB for 10 GiB.
|
||||
|
||||
The code supports these files on purpose ("old clients didn't all write 4 MiB blocks"). How
|
||||
many real files lack `BlockSizes` is unknown (every golden capture records them), which is
|
||||
the only reason this isn't higher.
|
||||
|
||||
Fix, cheapest first:
|
||||
1. A small per-client cache of revisionID -> learnt block sizes, filled as a reader decrypts
|
||||
blocks, so later chunks start at the nearest known block. This is the fix the TODO plans,
|
||||
and it should be raised in priority.
|
||||
2. Until then, a ranged read with sizes not recorded and offset > 0 could at least stop the
|
||||
listing once the range is covered (it can't know where, so this is weak).
|
||||
3. Or document that multi-thread copies of such files should use
|
||||
`--multi-thread-streams 0`.
|
||||
|
||||
## Minor
|
||||
|
||||
M-1 **Two processes sharing one config can delete each other's in-progress draft**
|
||||
(upload.go:292-303). The UID is per remote, not per process. If a mount and a cron
|
||||
`rclone move`, or two overlapping runs, upload the same new name, B deletes A's live draft
|
||||
and A's upload fails. If A commits within the RTT between B's 2500 response and B's
|
||||
`delete_multiple`, B sends `delete_multiple` against an active file. What the server does
|
||||
then is the open TODO question. With `move`, A has already deleted its source. The TODO
|
||||
records only the `--retries` path. Add this case there. A per-process suffix on ClientUID,
|
||||
with "own" meaning the remote-wide prefix plus not-this-process, doesn't help. The real
|
||||
mitigation is the TODO's server check, or re-reading the link and confirming it is still a
|
||||
draft just before deleting (this narrows the window but does not close it).
|
||||
|
||||
M-2 **`pm-storage-token` is still sent on redirects to other hosts** (download.go:264-270,
|
||||
upload.go:868; Task 10 minor). Go's client drops only Authorization, Cookie and
|
||||
WWW-Authenticate when a redirect leaves the host. Set `CheckRedirect` on the storage calls,
|
||||
or refuse redirects for them.
|
||||
|
||||
M-3 **The `upload` godoc has a broken reflow** (upload.go:104-109: "...is
|
||||
errUploadSuperseded,\n// and one the server..."). That is the 21f re-review's N2 and it is
|
||||
still present. Rewrap the paragraph.
|
||||
|
||||
M-4 **`errUploadMismatch` is a `NoRetryError`** (protondrive.go:359). A source that changes
|
||||
while it is being read (a growing log file, or a local file still being written) fails for
|
||||
the whole run, including the `--retries` passes, where the usual rclone outcome is a retry
|
||||
at the high level. `NoLowLevelRetryError` would avoid a pointless immediate re-read and still
|
||||
let a later pass succeed.
|
||||
|
||||
M-5 **A range read of a file whose block sizes aren't recorded keeps fetching after the
|
||||
range ends** (download.go:145-151, 338-346). `last` stays -1 when sizes aren't recorded, so
|
||||
up to `downloadAhead` blocks (16 MiB) past the end of the range are downloaded and
|
||||
decrypted before `Close`. This adds to I-2.
|
||||
|
||||
M-6 **An own leftover draft revision on an active file is never deleted** (upload.go:279-290;
|
||||
Task 11 minor, still present). The conflict path uploads a new revision and leaves our old
|
||||
`ConflictDraftRevisionID` taking quota. A `DELETE` of that draft revision when
|
||||
`ConflictDraftClientUID` is ours would be safe: it is a draft revision, not the active one.
|
||||
|
||||
M-7 **Updating an unverified file logs "reading"** (upload.go:312 -> download.go:196; Task
|
||||
16 minor, still present).
|
||||
|
||||
M-8 **The `errUploadSuperseded` message is misleading** when the node reads back as draft
|
||||
or trashed (upload.go:437-438; Task 11 minor, still present). The message says
|
||||
"revision ... is active".
|
||||
|
||||
M-9 **The `createDraft` retry can report its own draft as a foreign one** (upload.go:237-259,
|
||||
294-295). Sequence: own draft deleted, re-POST gets a 5xx, the pacer resends, and the lost
|
||||
first response leaves our new draft, so the resend sees a 2500 with our UID. Because
|
||||
`retried` is set, the client reports "a draft of the file came back after it was deleted".
|
||||
Nothing is lost and the next run replaces the draft. When `ConflictDraftClientUID` is ours
|
||||
on the retried pass, the error could say so instead.
|
||||
|
||||
Still present and already in docs/TODO.md (not re-graded): `Close` waits behind the
|
||||
context-free `downloadTokens.Get`; `countContent` reads the whole file again for every
|
||||
suffix-range open of a file with no recorded size; `--retries` can reach the own-draft
|
||||
`delete_multiple`; unbounded block-list metadata bytes; ranged reads don't check the
|
||||
manifest; no test pins the 408 exclusion in `refusedCommit` (N3; the 408 case in
|
||||
nodes_internal_test.go:428 covers only `shouldRetry`).
|
||||
|
||||
Closed since earlier reviews: `bytes=0-` and whole-file ranges now go through `openWhole`
|
||||
(ruling 26). The read back now stops when ctx is done. Cleanup no longer deletes after a
|
||||
read back, so Task 11's "restore deletes our superseded history entry" can no longer happen.
|
||||
@@ -173,3 +173,80 @@ Task 16: complete — 606580196, 59f0f31b3, db97ef7c8, f4105adfc, 88654eb04, 959
|
||||
|
||||
## PAUSED 2026-11-11 at 9c90998f7 (owner's instruction)
|
||||
Tasks 1–17 complete and reviewed. Tasks 18–21 not started. Final whole-branch review not run. Handoff: docs/plans/001-protondrive-tier1-HANDOFF.md. This ledger is copied to docs/plans/001-protondrive-tier1-workspace/progress.md.
|
||||
|
||||
## RESUMED 2026-11-12 (owner: "Continue work from HANDOFF") on branch protondrive-tier1 at d211b45e6
|
||||
- Ruling: Task 18 writes .gitea/workflows/ci.yml even though the runner is unknown (plan says so; unverifiable locally) — cost if wrong: an unused file.
|
||||
- Ruling: Task 18 implementer decides the shared-session design for protonfake under test_all after checking token expiry/Disconnect; any change stays inside protontest — cost if wrong: flaky wider suites, visible at once.
|
||||
- Task 18 dispatched (opus), BASE d211b45e6.
|
||||
- 2026-11-12: owner stopped the session to switch the controller to Opus. Task 18 implementer stopped mid-task, no commits, no report. Uncommitted work left in the tree (init.d/TestProtonDriveFake, .gitea/, PORTS.md, test_all config.yaml, docs/{ARCHITECTURE,DECISIONS,PERMISSIONS,TODO}.md); tracked-file diff saved as task-18-partial.diff. Unreviewed: next controller should inspect it, then resume Task 18 from it or discard it, and re-dispatch. A stray local protonfake process was killed.
|
||||
- 2026-11-12: controller now Opus 5.5. Task 18 implementer resumed (same agent, context intact) from its uncommitted work.
|
||||
- OWNER ANSWERS 2026-11-12: (1) Gitea HAS an Actions runner, but it is slow and not very capable — keep ci.yml light. (2) Task 21 may install a Proton desktop/CLI app and use the web app, preferably inside a Docker container (docker available on host). (3) YES to removing the test-account usernames from git history with a force push. Only 901d7fd5b adds them (85f6b2f0e removes them). Plan: after Task 18 commits land and the implementer is idle, rewrite from 901d7fd5b (redact usernames in that commit only, replay the rest), force-push master once, record in DECISIONS.md and handoff.
|
||||
- Task 18 implemented: 28f1aaf8a. DONE_WITH_CONCERNS: 5 compare-dest/copy-dest/checksum tests ignored (per-Fs listing cache; a second Fs on same account doesn't see the first's writes); gitannex ignored (test-harness env-remote limitation); .claude/settings.json not edited (owner must add golangci-lint allow — agent cannot widen own perms); shared session OK (no 401 in standalone fake). Review dispatched (sonnet).
|
||||
- Task 18 review (sonnet): spec ✅, quality approved, no Critical/Important. Cache diagnosis confirmed; init.d lifecycle checked (no orphans, build failure and busy port both rc=1). Minors (deferred): stop has no SIGKILL escalation; shellcheck SC2034 unused `i`; ci.yml lacks concurrency group; golangci-lint 2.15.0 build-Go vs go 1.26 may fail on first Gitea run. File: task-18-review.md.
|
||||
- Ruling: keep the 5 ignored cross-Fs-cache tests and leave the fix for M5 (events) — shared per-account cache is forbidden package state, re-list-on-miss breaks budgets, TTL doesn't fix the tests; added mount-staleness note to TODO; flagged as owner question 5 in handoff — cost if wrong: stale --compare-dest/--copy-dest only when the reference is written by another Fs in the same process; mounts never see external changes until M5.
|
||||
|
||||
Task 18: complete — 28f1aaf8a (rewritten as 5afbe9e4f). Review: approved, minors deferred.
|
||||
- HISTORY REWRITE 2026-11-12 (owner approved): 901d7fd5b→31f6aafd9 (usernames redacted), 85f6b2f0e→4282540e0, d211b45e6→65840bdc5, 28f1aaf8a→5afbe9e4f; final tree identical. Gitea master force-pushed 85f6b2f0e→4282540e0 with lease. Local master at 65840bdc5 (not pushed beyond what remote had). Recorded in DECISIONS.md and handoff.
|
||||
- Task 19 dispatched (opus), BASE b667a1ab0.
|
||||
- Owner added Bash(golangci-lint run:*) to .claude/settings.json allow list (2026-11-12). After Task 19 lands: remove that item from docs/TODO.md and the PERMISSIONS.md Known gaps, commit with the settings change.
|
||||
- Task 19 implemented: cf7577dc1. DONE_WITH_CONCERNS: binary 152,460,576 -> 148,995,400; tidy removals only (2 libs, iso8601, 8 indirect); 29 cross targets OK; whole repo 186 pass, cmd/bisync TestRcBackupDir fails pre-existing (also at 06700aaf2); differential coverage lost (multi-block/empty old-lib files, v1.76 reading new writes). Review dispatched (sonnet).
|
||||
- Committed owner's golangci-lint allow rule + doc cleanup. Found: deny rule 'git push --force-with-lease:*' does not match '--force-with-lease=...' — logged in PERMISSIONS Known gaps; proposed fix to owner.
|
||||
- Task 19 review (sonnet): spec ✅, quality approved, no Critical/Important. Minor: handoff table stale (fixed). File: task-19-review.md.
|
||||
- Ruling: do not generate multi-block/downgrade fixtures now — b667a1ab0 keeps the differential tests and libraries reachable in history, so it stays reproducible; noted in TODO — cost if wrong: a later agent spends an hour building fixtures from a worktree.
|
||||
|
||||
Task 19: complete — cf7577dc1. Review approved.
|
||||
- Task 20 dispatched (opus), BASE after docs commit.
|
||||
- Task 21 prep (controller): official Proton Drive CLI 0.9.0 exists (proton.me/download/drive/cli/0.9.0/linux-x64/proton-drive, SHA-512 3533025b...7e91ca8 on the download page). Login is browser-only (`proton-drive auth login` prints a URL) → owner must complete it once per account; run in Docker with PROTON_DRIVE_CREDENTIALS_STORE=unsafe_file in a throwaway volume. Commands: `filesystem list|upload|download --json`. Live rclone runs (TestIntegration, test_all -n 1) done by controller from a scratch worktree; v1.76 downgrade check uses a COPY of the config (refresh tokens single-use; new code falls back to password if the real config's token dies).
|
||||
- Task 20 implemented: 7be50f9e2. DONE_WITH_CONCERNS: generated options block stale (needs make backenddocs before upstream; logged); browser sign-in documented as primary but unproven live; docs follow code where spec differs (name rules); SDK ref tag question; ARCHITECTURE rough-edges line kept; about = account usage unverified live. Review dispatched (sonnet).
|
||||
- Task 21 prep: official CLI 0.9.0 downloaded, SHA-512 verified, image proton-drive-cli:0.9.0 built (debian:trixie-slim, non-root, unsafe_file store). Dockerfile in controller scratchpad proton-cli/. Next: owner completes 'auth login' URL for the free account.
|
||||
- Task 20 review (sonnet): spec ✅, quality needs fixes. Important: (1) docs say a ranged read of a file without recorded block sizes "may fail"; code FIXME (download.go:142-145) says it returns wrong bytes, and the check at download.go:~614 only fires on the 2nd block → single-block range silently wrong. (2) SDK version cited as tag js/v0.23.0 at 05c9115; cite commit hash + CHANGELOG entry. Minor: enable_caching Help vs page. File: task-20-review.md.
|
||||
- Ruling: fix (1) in code, not just docs — silent wrong bytes is a correctness bug for files from other clients. When block sizes are unrecorded and the range starts after block 0, read from the start of the file and discard (or another approach that is correct and checked), accepting the cost on such files; regression test with the fake writing non-4MiB blocks without sizes; docs then describe the cost. Cost if wrong: slower ranged reads (mount) on old-client files only.
|
||||
- Task 20 fix round 1 dispatched (resume implementer), BASE 7be50f9e2.
|
||||
- Task 20 fix round 1: 22b9c4fdc (ranged reads of unrecorded-size files read from block 0 and discard; regression TestDownloadRangesUnrecordedBlockSizes + fake SeedFileInBlocks; SDK cited by commit; enable_caching help aligned). Re-review dispatched (opus).
|
||||
- Task 20 re-review (opus): ALL CLOSED (mutation-checked: 7/9 subtests fail on old download.go). Minors: package comment tag wording (fixed by controller), spec 167-170 same tag wording (left: spec history), TestDownloadEndlessBlockList "assumed" name, TestDownloadRanges bound always 3, two unclear protontest/drive.go comments (deferred), perf TODO added by controller.
|
||||
|
||||
Task 20: complete — 7be50f9e2, 22b9c4fdc, + controller follow-up commit. Review approved after 1 fix round.
|
||||
- Task 21 live 2026-11-12 (free account, binary 502a32b5f, scratch worktree live-502a32b5f): TestIntegration -remote TestProtonDrive: PASS (98 pass, 29 skip, 0 fail, 120 s; only expected "directory not found" errors; no leftovers).
|
||||
- Task 21 downgrade check vs released rclone v1.76.0 (downloads.rclone.org, SHA256SUMS verified; run with a COPY of the config, deleted after): v1.76 downloaded every file the new backend wrote (incl. web-app file) byte-identical; new backend read v1.76's 9 MiB + small uploads byte-identical incl. a mid-file range. Test folder purged.
|
||||
- Official CLI sign-in #1 timed out (~19 min, owner not yet at it); restarted, new URL given to owner.
|
||||
- Task 21: test_all -n 1 -maxtries 1 against TestProtonDrive: running (bisync/gitannex ignoretests as configured upstream).
|
||||
- Official CLI 0.9.0 signed in (free account) by owner via browser fork URL on 3rd try (CLI polls only ~10 min; a late sign-in shows "Invalid code" in the browser).
|
||||
- Interop 2026-11-12 (free account, binary 502a32b5f, CLI in Docker): CLI downloaded rclone-written probe folder byte-identical, 0 failures; CLI `info` shows keyAuthor/nameAuthor/contentAuthor ok:true for rclone files. CLI uploaded 13 MiB random, small, empty, unicode-named nested file → rclone (no ignore flag) copied all byte-identical, mid-file range OK. rclone rcat new revision + moveto (unicode) + mkdir → CLI lists and downloads correct content, signatures ok. Folder rclone-interop-20261112 left on the account as evidence.
|
||||
- OWNER 2026-11-12: golden fixtures = option 2 (re-key captured nodes onto test-only keys; nothing from the account committed). Brief task-21g-brief.md written by controller. Ruling: implementer builds capture+rekey+TestGoldenFixtures against the fake only; controller runs the single live capture (web-app folder my-little-folder + CLI folder rclone-interop-20261112) and reviews output before committing fixtures. Task 21g dispatched (opus), BASE 502a32b5f.
|
||||
- Task 21 live test_all (free account, 502a32b5f, -n 1 -maxtries 1, 46 min): FAIL 8 = the 5 known cross-Fs-cache tests (confirmed live) + 3 live-only: operations TestPurge ("Failed to put A1/B1/C1/one: object not found"), vfs TestDirRemoveAll and TestZipDirsInRoot ("upload committed, but the file now has another revision (revision is active, not X)" — EMPTY active revision). Manual repro of nested put x3: no failure (timing). Logs: scratchpad/live-testall/2026-11-12-104408. backend suite inside test_all: OK.
|
||||
- Ruling: treat as one real-server read-after-write lag in the post-commit confirmation; fix = retry the confirmation briefly when node not found or active revision empty, distinguish from a genuine other revision; fake gets a lag fault to reproduce. Task 21f (fix) queued after 21g (same tree). Cost if wrong: diagnosis off, re-run live shows it.
|
||||
- Task 21g implemented: a2dc6f71f. DONE_WITH_CONCERNS: signed SEIPD (names/hash keys/xattrs) cannot stay byte-identical when inner signatures are re-made — decrypted, inner packets kept, sig/OPS re-made, re-encrypted with same session key/cipher/version/AEAD/chunk; unsigned messages and all blocks byte-identical; node passphrases new; block signatures, user key, thumbnailed files left out; untested real packet shapes fail loudly. Review dispatched (opus). Controller live capture into scratchpad (not tree) pending review.
|
||||
- Task 21f (post-commit read-after-write lag fix) dispatched (opus) in parallel with 21g review, BASE a2dc6f71f.
|
||||
- Task 21g review (opus): spec ❌. Important: I-1 foreign intended-recipient fingerprint written into Rekeyed note (golden_internal_test.go:1256-1259); I-2 identity() keeps user-ID name part and foreign emails inside armored keys (:898-908); I-3 capture reuses -remote → a bad -run would run TestIntegration (writes/deletes) live (:38-39,:108-128). Packet-level leak check of round trip by reviewer: clean. Capture is read-only on the account. File: task-21g-review.md. Fix round 1 dispatched (resume implementer). Live capture waits.
|
||||
- Task 21f implemented: af6fdd789. Diagnosis confirmed (both from the single post-commit read: draft/empty revision, or not found). Retry 100ms→2s, 8 reads (~7s). errUploadSuperseded only for non-empty revision that is neither ours nor the starting one. Ruling (accept implementer's choice over my lean): lag beyond retries after successful commit → errUploadUnconfirmed (deletes nothing, not retried at once) rather than success with a made-up node — cost if wrong: a rare spurious error where a later sync pass finds the file. Concerns: lag length unmeasured; activeNode not lag-tolerant; failed-commit stale read can still delete draft (ruling 11 risk now real); cancellation waits up to ~7s. Review dispatched (opus).
|
||||
- Task 21f review (opus): spec ❌. CRITICAL (pre-existing ruling-11 path, now likely under lag; reproduced in scratch): after a FAILED commit a single stale read (draft / old revision) is taken as proof it didn't land → delete_multiple on the now-active file or DELETE on the now-active revision (upload.go:401-403,425). Pacer resends commit on 5xx/408/network, ctx cancel also leads here. Minors: empty-revision lag state untested; 7s retry ignores Ctrl-C; --retries pass may add an identical revision; two tests not seen failing. File: task-21f-review.md.
|
||||
- Ruling (adopt reviewer's rule): delete a draft ONLY when the server explicitly refused the commit (4xx with a Proton code) on a single attempt that was not resent; no read-back decides deletion. Every other failure → errUploadOutcomeUnknown, delete nothing. Scratch repro becomes a regression test. Supersedes the "positive proof by read" half of ruling 11. Cost if wrong: orphan drafts accumulate on the server after ambiguous failures (cleaned by CleanUp / server expiry) — acceptable vs data loss.
|
||||
- Task 21f fix round 1 queued until 21g fix round 1 finishes (shared tree).
|
||||
- Task 21g fix round 1: 1b2fcbd21 (fail closed on foreign keys/recipients, placeholder user IDs, -capture-remote flag, packet-level leak check also in TestGoldenFixtures, tamper tests). Ruling: accept allowlist of generic node-key emails no-reply@proton.me / noreply@protonmail.com (C# SDK/CLI uses "Drive key <no-reply@proton.me>") — written keys still carry only the placeholder; cost if wrong: none (no account data). Re-review dispatched (opus).
|
||||
- Task 21f fix round 1 dispatched (resume implementer), BASE 1b2fcbd21.
|
||||
- Task 21g re-review (opus): NOT ALL CLOSED — I-3 open: TestIntegration ignores -capture/-capture-remote, so a mistyped -run with the real config runs the write/delete suite live. I-1, I-2 (allowlist proviso holds), M-1..M-5 closed; 4 new minors. File: task-21g-rereview.md.
|
||||
- Ruling: controller makes the trivial I-3 fix (TestIntegration skips when -capture or -capture-remote is set) after 21f lands, then live capture.
|
||||
- Task 21f fix round 1: be0053607 (delete draft only on single-attempt coded 4xx refusal, attempt counter via pacedCall; all else errUploadOutcomeUnknown; 4 regression tests failed before fix; Ctrl-C honoured post-commit; LagNoRevision tested). Re-review dispatched (opus).
|
||||
- Controller fixed 21g I-3: f8e398fce TestIntegration skips when -capture/-capture-remote set (verified skip + lint). Task 21g: complete — a2dc6f71f, 1b2fcbd21, f8e398fce. Live capture next.
|
||||
- Live capture 2026-11-12 by controller (free account, HEAD f8e398fce): web = my-little-folder (Readme.txt, 1 block), cli = rclone-golden-cli-20261112 (empty.bin no block sizes, nine-mib.bin 3 blocks, small.txt). Leak inspection: 11 secret needles (usernames, tokens, uid, salted key pass raw+decoded, both passwords) 0 hits; emails only golden@example.invalid; 0 Proton IDs; key UIDs only placeholder (gpg --show-keys); no 'proton.me'/'stereotypical' bytes anywhere. TestGoldenFixtures PASS offline. Committed 61733fce6 (~9.5 MB, mostly 3 random blocks). Plan Task 21 boxes ticked: capture, interop, upgrade/downgrade.
|
||||
- Task 21f re-review (opus): ALL CLOSED (resend guard mutation-checked). New minors: N1 --retries pass can reach own-draft delete_multiple in resolveConflict (logged in TODO, b440df2e1); N2 upload godoc wording; N3 no test pins 408 exclusion (deferred).
|
||||
|
||||
Task 21f: complete — af6fdd789, be0053607.
|
||||
- Live test_all re-run dispatched at b440df2e1 (scratch worktree live-b440df2e1).
|
||||
- Live test_all re-run (free account, b440df2e1, 37.5 min): FAIL only the 5 known cross-Fs-cache tests (TestCheckSumDownload, TestCopyFileCompareDest, TestCopyFileCopyDestImmutable, TestSyncCompareDest, TestSyncCopyDestImmutable). TestPurge, vfs TestDirRemoveAll/TestZipDirsInRoot now PASS — lag fix confirmed live. vfs fully green live. No leftovers. Logs scratchpad/live-testall-2/.
|
||||
- Paid account FIRST sign-in 2026-11-12 (b440df2e1): password/SRP OK, about = 64 GiB total, 1.152 MiB used; drive empty; mkdir/rmdir OK without ignore_signature_errors. Unknown whether the volume pre-existed (made by Proton apps) or was created now by the client (createVolume logs nothing) — asking owner.
|
||||
- rclone BROWSER sign-in proven live 2026-11-12 (b440df2e1, scratch config, free account, owner completed fork URL): session saved, listing + verified full read of the web-app file OK. Then `config disconnect` (logout) OK and later use gives the clear reconnect error; scratch config deleted.
|
||||
- Incident: `rclone config create` printed the new remote's config INCLUDING client_* tokens and salted key pass to stdout; the controller's redaction regex failed, so those values for that one session appeared in the controller's tool output (conversation transcript). Mitigation: that session was logged out immediately (tokens revoked server-side); the salted key pass is derived from the account password and is NOT revoked by logout — throwaway free account; owner informed. TODO: check whether client_* options are marked Sensitive so config create/show redact them.
|
||||
- Owner: paid account had (probably) never opened Drive before → its volume was most likely created by rclone; root-of-trust on a Proton-app-made drive still unverified (needs an account used only by Proton apps).
|
||||
|
||||
Task 21: complete as far as accounts allow — live TestIntegration, test_all (5 known cache fails), browser sign-in, paid sign-in, CLI interop, v1.76 up/downgrade, golden fixtures; commits a2dc6f71f 1b2fcbd21 f8e398fce 61733fce6 af6fdd789 be0053607 b440df2e1 29ad26944. Open: 2FA, CAPTCHA, desktop/mobile clients, Proton-app-made drive root of trust.
|
||||
- Final whole-branch review next (base 06700aaf2..29ad26944).
|
||||
- Final whole-branch review dispatched 2026-11-12: 4 parallel opus reviewers (auth/session/Fs, crypto/nodes/cache, upload/download, tests/fake/harness/docs/hygiene) over 06700aaf2..29ad26944; shared brief final-review-common.md; outputs final-review-{auth,crypto,transfer,tests}.md. Then ONE fix wave + scoped re-review.
|
||||
- Final review results: crypto 2C/3I/8m (C1 unsigned content key packet accepted → Update encrypts to server key, ranged reads forgeable; C2 server-supplied Argon2 params → OOM; I1 unverified hash key used on write; I2 rename reuses unverified name session key; I3 unsigned xattrs trusted); transfer 0C/2I/9m (draft_client_uid not persisted for env remotes; quadratic multi-thread reads of unrecorded-block-size files); auth 0C/2I/11m (2000/2028 fatal "upgrade", real 5003/5001 unmapped; x-pm-drive-requirements not logged); tests 0C/5I/19m (CI race timeout; CI Go 1.26.0 + govulncheck@latest; stale workspace copy; stale handoff §3; upstream recipe includes fork-only protontest dependents).
|
||||
- Rulings: see final-fix-brief.md (C1 = unverified file like ruling 18/22 → handoff ruling 29; error codes → ruling 30). Fix wave dispatched (opus, one implementer) for code+CI, BASE 29ad26944; controller fixes tests I-3/I-4/I-5 (handoff, workspace copy, spec 13 recipe) in parallel on disjoint files.
|
||||
- Final fix wave: e95317b52 (C1), db7afafc0 (C2), b3f5685e6 (I1), 67e4cfeb7 (I2), 94eedc942 (I3), 674b538ca (draft uid), 788a711dc (block-size cache 145→39 fetches), 5dbf5b04c (codes), b4a23a31e (requirements header), b5da26169 (CI), 634a2423e (tests honour RCLONE_CONFIG regardless of which tests run — earlier unit tests read the real config file; no Proton contact), 30cfcac41 (refresh retry), d3aee2af1 bfa9b7122 ad3552e2e 944b36005 (minors/docs).
|
||||
- Rulings: accept deviations — I3 size from verified block sizes (encrypted size would fail every copy); draft uid derived from volume ID (user ID not kept); C2 caps foreign keys only (own keys logged in TODO); 2064 unmapped. Scoped re-review dispatched (opus).
|
||||
- Live check of final fixes (944b36005, free account, no flag): all files in the 3 evidence folders (written by web app, official CLI, rclone, rclone v1.76) copy + check OK, bytes match original SHA1s; upload 9 MiB, mid-file range, moveto, full read OK; test folder purged. Stricter C1/I1/I3 checks accept real Proton-client files.
|
||||
- Final re-review (opus): ALL CLOSED; every fix mutation-checked; golden fixtures pass; Argon2 caps above gopenpgp/OpenPGP.js output; RCLONE_CONFIG fix proven with fake HOME. New minors M-a (ranged start from unsigned xattr block sizes), M-b (TestMain applies RCLONE_CONFIG only when set), M-c (draft uid random when section exists but unsaveable) → TODO.
|
||||
|
||||
Final whole-branch review: complete. Plan 001 execution complete as far as test accounts allow.
|
||||
@@ -0,0 +1,39 @@
|
||||
### Task 18: Wider hermetic suites and fork CI
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `fstest/testserver/init.d/TestProtonDriveFake`
|
||||
- Modify: `fstest/testserver/init.d/PORTS.md`
|
||||
- Modify: `fstest/test_all/config.yaml` (add backend `protondrive` remote
|
||||
`TestProtonDriveFake:` entry, `fastlist: false`)
|
||||
- Create: `.gitea/workflows/ci.yml`
|
||||
- Modify: `docs/TODO.md`, `docs/PERMISSIONS.md`
|
||||
|
||||
Model the init.d script on `fstest/testserver/init.d/TestWebdavRclone`
|
||||
(`rclone-serve.bash`/`run.bash` helpers): start `protonfake` on a port
|
||||
allocated in `PORTS.md`, echo its `key=value` lines and
|
||||
`_connect=127.0.0.1:PORT`.
|
||||
|
||||
> Read "Notes for Tasks 18 to 21" in the handoff before starting: this
|
||||
> text leaves out how the script obtains `protonfake`'s output and manages
|
||||
> its process, and the commands below need `RCLONE_CONFIG=/notfound`.
|
||||
|
||||
- [ ] **Step 1:** Add the script and config entry. Run
|
||||
`go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4`
|
||||
and fix backend failures in `fs/operations`, `fs/sync`, `vfs` until the
|
||||
report is clean. `cmd/bisync` and `cmd/gitannex`: run them; if they fail
|
||||
for reasons that need M5/M6 work, add `ignoretests` for those two on the
|
||||
fake remote and list the failures in `docs/TODO.md`.
|
||||
- [ ] **Step 2:** `.gitea/workflows/ci.yml` on push and pull request:
|
||||
`actions/setup-go` with `go-version-file: go.mod`; steps `go build ./...`;
|
||||
`go vet ./backend/protondrive/...`;
|
||||
`golangci-lint run ./backend/protondrive/...`;
|
||||
`go test -race ./backend/protondrive/...`; a 20s fuzz run per target;
|
||||
`go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/...`.
|
||||
It cannot be verified locally: say so in `docs/TODO.md` ("first push must
|
||||
confirm the Gitea runner picks it up").
|
||||
- [ ] **Step 3:** Update `docs/PERMISSIONS.md` verified-commands table for
|
||||
the host toolchain (Go 1.27.4, golangci-lint 2.15.0) and add
|
||||
`golangci-lint run` to the Allow tier in it and in the three tool files.
|
||||
- [ ] **Step 4: Commit** `protondrive: run the wider suites against the fake server in fork CI`.
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
# Task 18 report: wider hermetic suites and fork CI
|
||||
|
||||
## What was built
|
||||
|
||||
- `fstest/testserver/init.d/TestProtonDriveFake` (new, mode 755): init.d
|
||||
script for the fake server.
|
||||
- `fstest/testserver/init.d/PORTS.md`: port 28642 allocated (next free
|
||||
after 28641).
|
||||
- `fstest/test_all/config.yaml`: new entry `protondrive` /
|
||||
`TestProtonDriveFake:`, `fastlist: false`, with 5 `ignore` tests and
|
||||
`ignoretests: cmd/gitannex` (reasons below, each commented in the file).
|
||||
The existing live `TestProtonDrive:` entry is untouched.
|
||||
- `.gitea/workflows/ci.yml` (new): on push and pull_request, one job,
|
||||
`timeout-minutes: 45`; checkout, `actions/setup-go@v5` with
|
||||
`go-version-file: go.mod` and `cache: true`; `go build ./...`;
|
||||
`go vet ./backend/protondrive/...`; `golangci-lint-action@v8` pinned to
|
||||
v2.15.0 with `args: ./backend/protondrive/...` (the action caches);
|
||||
`go test -race ./backend/protondrive/...`; a 20 s fuzz run per target
|
||||
(targets discovered with `go test -list '^Fuzz'`, currently 5);
|
||||
`go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/...`.
|
||||
`RCLONE_CONFIG=/notfound` set workflow-wide. No test_all (per owner,
|
||||
relayed by the coordinator: runner exists but is slow).
|
||||
- Docs: `docs/DECISIONS.md` (shared-session decision), `docs/TODO.md`
|
||||
(cross-Fs cache entry expanded with the five failing tests, gitannex
|
||||
entry, CI entry, `.claude` permission entry, "Now" entry updated, old
|
||||
"decide whether to add Gitea CI" entry removed), `docs/PERMISSIONS.md`
|
||||
(verified table re-run on host, `golangci-lint run` in Allow, stale
|
||||
"host has no Go" gap removed, `.claude` gap noted),
|
||||
`docs/ARCHITECTURE.md` (one bullet on the init.d script and CI),
|
||||
`.gemini/settings.json` (`golangci-lint run` allowed), plan Task 18
|
||||
checkboxes ticked.
|
||||
- No backend code changed. `go.mod`/`go.sum` unchanged.
|
||||
|
||||
## Script design
|
||||
|
||||
- Sources `run.bash` (mandatory refcounting/locking), defines its own
|
||||
`start`/`stop`/`status`; state (pid, binary, captured stdout, log) lives
|
||||
in run.bash's `${RUN_STATE}` (`$XDG_RUNTIME_DIR/rclone-test-server/TestProtonDriveFake/state`),
|
||||
not `/tmp/<name>.pid`.
|
||||
- Repo root found from the script's own location
|
||||
(`$(dirname "$0")/../../..`), since testserver runs it from each test
|
||||
package's directory.
|
||||
- `start` builds `./backend/protondrive/protontest/cmd/protonfake` into
|
||||
the state dir with `go build` (fast with the build cache; rebuilt each
|
||||
first start so it never runs a stale binary), runs the binary itself
|
||||
with `nohup ... -addr 127.0.0.1:28642`, stdout to a file, stderr to a
|
||||
log, disowned. It polls up to 10 s for the `type=` line (the last of the
|
||||
six), failing early with the log if the process dies, then cats the six
|
||||
lines and echoes `_connect=127.0.0.1:28642`. run.bash caches that
|
||||
output, so every later `start` while the server lives returns the same
|
||||
session.
|
||||
- `stop` sends SIGTERM (protonfake handles it and closes cleanly), waits
|
||||
up to 10 s for exit so the port is free, removes the pid file.
|
||||
- Smoke-tested start/status/stop by hand; after both full test_all runs
|
||||
nothing listens on 28642 and no protonfake from the script remains.
|
||||
|
||||
## Shared-session finding
|
||||
|
||||
Not a problem; nothing changed in protontest/protonfake/client. Details in
|
||||
DECISIONS.md: the client refreshes only on HTTP 401 (`session.call`); the
|
||||
standalone fake returns 401 only after `ExpireAccessToken`,
|
||||
`RevokeSession` or logout, and protonfake calls none. Logout happens only
|
||||
in `Fs.Disconnect` and `rclone config` re-sign-in; no test in fs/operations,
|
||||
fs/sync, vfs, cmd/bisync, cmd/gitannex or fstests calls Disconnect (grep).
|
||||
The backend package's own tests that use `setupFakeRemote` override the
|
||||
env vars with `t.Setenv` and their own in-process server, so they never
|
||||
touch the shared session. Confirmed empirically: two full runs, 4 suites
|
||||
in parallel on one session, no auth failures.
|
||||
|
||||
## Wider-suite failures
|
||||
|
||||
First run (no ignores): vfs OK, backend OK, bisync OK; failures:
|
||||
|
||||
1. **cmd/gitannex** - all 19 `TestGitAnnexFstestBackendCases/*` fail with
|
||||
`INITREMOTE-FAILURE remote does not exist: TestProtonDriveFake:`.
|
||||
Cause: `cmd/gitannex/configparse.go` checks the name against
|
||||
`config.GetRemoteNames()`, which lists env-var remotes lower-cased
|
||||
(`fs/config/config.go` GetRemotes). Every other init.d remote in
|
||||
config.yaml ignores cmd/gitannex for the same reason. Core harness
|
||||
limitation, not backend: `ignoretests: cmd/gitannex` + TODO entry.
|
||||
2. **fs/operations** `TestCheckSumDownload`, `TestCopyFileCompareDest`,
|
||||
`TestCopyFileCopyDestImmutable`; **fs/sync** `TestSyncCompareDest`,
|
||||
`TestSyncCopyDestImmutable`. Cause: each `Fs` has its own node cache;
|
||||
these tests create a second `Fs` (`dataFs`, `fdst`, the compare/copy
|
||||
dest Fs) on the same account, list or miss through it, then write
|
||||
through `r.Fremote`; the second `Fs`'s cached listing still lacks the
|
||||
new names (or its root directory). Confirmed: all five pass with
|
||||
`RCLONE_CONFIG_TESTPROTONDRIVEFAKE_ENABLE_CACHING=false` and fail with
|
||||
caching on, run in isolation.
|
||||
Attempted fix: re-list a folder whenever `findChild` misses in a cached
|
||||
listing (RED test written, went GREEN), but it broke
|
||||
`TestBudgetUpload` and `TestBudgetMkdir` - it re-lists the folder once
|
||||
per new file or folder (O(n^2) for `--no-traverse`/`copyto` of many new
|
||||
files), which those budgets deliberately forbid. Reverted (no commit).
|
||||
Events (M5, 30 s polling) won't close a window of seconds; a cache
|
||||
shared by all `Fs` of an account would, but needs package state, which
|
||||
the spec/global constraints rule out. So: `ignore` for the five tests
|
||||
+ expanded TODO entry, flagged as an owner decision. Real-world impact
|
||||
is limited to one process where one Fs writes what another Fs of the
|
||||
same account then looks up (e.g. a compare/copy-dest written during the
|
||||
run).
|
||||
|
||||
No other failures. Nothing was fixed by touching the fake, core rclone or
|
||||
tests outside the backend.
|
||||
|
||||
## Final test_all report (run 3, with the config as committed)
|
||||
|
||||
`RCLONE_CONFIG=/notfound go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4 -output <scratch>`
|
||||
-> `PASS: All tests finished OK in 3m48s`, exit 0.
|
||||
|
||||
| Suite | Top-level PASS | FAIL (ignored) | SKIP | Result |
|
||||
|---|---|---|---|---|
|
||||
| backend/protondrive | 208 | 0 | 1 | OK |
|
||||
| fs/operations | 91 | 3 | 57 | OK (3 ignored) |
|
||||
| fs/sync | 96 | 2 | 12 | OK (2 ignored) |
|
||||
| vfs | 99 | 0 | 10 | OK |
|
||||
| cmd/bisync | 17 | 0 | 2 | OK |
|
||||
| cmd/gitannex | - | - | - | not run (ignoretests) |
|
||||
|
||||
(A second run was killed mid-way by the controller's pause; run 3 is the
|
||||
complete one.)
|
||||
|
||||
## Verification commands (all with RCLONE_CONFIG=/notfound)
|
||||
|
||||
- `go build ./...` - ok
|
||||
- `go vet ./backend/protondrive/...` - ok
|
||||
- `golangci-lint run ./backend/protondrive/...` - 0 issues
|
||||
- `golangci-lint run ./...` - 0 issues
|
||||
- `go test -race ./backend/protondrive/...` - all 3 packages ok (protondrive 322 s, 5m24s wall)
|
||||
- `go mod download`, `go test ./backend/memory/`,
|
||||
`go test -run 'TestIntegration/FsMkdir/FsPutFiles/FsCopy' ./backend/memory/`,
|
||||
`gofmt -l backend/protondrive` - all ok (PERMISSIONS.md table)
|
||||
- YAML of ci.yml and config.yaml parse (python yaml.safe_load).
|
||||
- `git diff go.mod go.sum` - empty.
|
||||
|
||||
## Concerns
|
||||
|
||||
- `.claude/settings.json` was NOT edited: adding an allow rule there
|
||||
widens my own tool permissions, which an agent message can't authorize.
|
||||
PERMISSIONS.md and `.gemini` have `golangci-lint run`; the owner needs
|
||||
to add `"Bash(golangci-lint run:*)"` to `.claude/settings.json` allow
|
||||
(TODO + PERMISSIONS Known gaps). Codex writes no allow rules by design.
|
||||
- Five wider-suite tests are ignored for cross-Fs cache staleness; the fix
|
||||
needs an owner decision (shared per-account cache vs. the no-package-
|
||||
state constraint).
|
||||
- ci.yml is unverified until the first push (action major versions v4/v5/
|
||||
v8 chosen as widely available; Gitea runner may need them mirrored).
|
||||
- The PERMISSIONS row for `go test ./...` keeps the 2026-11-11 host
|
||||
baseline; not re-run (long).
|
||||
- The handoff's open question 2 (Gitea runner) is now answered; I did not
|
||||
edit the handoff doc.
|
||||
- A protonfake (pid 2654616, started 2026-11-11 16:56 from an earlier
|
||||
session's scratchpad, port random) was not started by me; the
|
||||
coordinator killed it.
|
||||
|
||||
## Commit
|
||||
|
||||
- `28f1aaf8a` protondrive: run the wider suites against the fake server in fork CI
|
||||
@@ -0,0 +1,61 @@
|
||||
# Task 18 review (d211b45e6..28f1aaf8a)
|
||||
|
||||
## Spec Compliance
|
||||
|
||||
- ✅ Spec compliant, with the handoff corrections applied. Every listed file has a hunk: init.d script, PORTS.md (28642, next free), config.yaml entry (`fastlist: false`), `.gitea/workflows/ci.yml`, TODO.md, PERMISSIONS.md. The plan's step 1 allowed `ignoretests` only for bisync/gitannex; the extra `ignore` of five tests is a deviation the implementer flagged and the brief's "fix backend failures until clean" could not satisfy. I judge it acceptable (see cache analysis) but it is a decision for the owner.
|
||||
- ⚠️ Cannot verify: ci.yml on a real Gitea runner (label `ubuntu-latest`, `uses:` resolution of `actions/*` and `golangci/golangci-lint-action`, network for govulncheck). TODO.md records this, as required. `.claude/settings.json` not edited: correct, not counted.
|
||||
|
||||
## Strengths
|
||||
|
||||
- Init script lifecycle is sound. I ran it in a scratch `STATE_DIR` (RCLONE_CONFIG=/notfound, nothing but loopback):
|
||||
- start in under 1s, six key=value lines plus `_connect`; a second start reuses the server (refcount 2); first stop keeps it, second stop kills it; no protonfake and nothing on 28642 afterwards.
|
||||
- build failure (fake `go` that exits 1): `failed to start`, rc=1, no process left.
|
||||
- port already taken: protonfake exits, script reports its log and `failed to start`, rc=1, no orphan.
|
||||
- `shellcheck`: only SC2034 (unused loop var `i`), warning level.
|
||||
- Built binary instead of `go run`, so the pid is the server's. Output goes to a file and stderr to a log, stdin is /dev/null, so `CombinedOutput` in `testserver.run` does not hang on an inherited pipe.
|
||||
- Repo root derived from the script location (`dirname "$0"/../../..`); works from any package directory.
|
||||
- Cannot reach live remotes: the fake is a separate remote name `TestProtonDriveFake:` configured only through `RCLONE_CONFIG_TESTPROTONDRIVEFAKE_*` env (distinct prefix from `..._TESTPROTONDRIVE_...`), endpoint is 127.0.0.1, CI sets `RCLONE_CONFIG=/notfound`. The live entry is untouched.
|
||||
- DECISIONS.md entry for the shared session is accurate and records the revisit condition. TODO entries are specific and actionable. Stale "no Go on host" text removed from PERMISSIONS.md.
|
||||
|
||||
## Cache diagnosis (the five ignored tests)
|
||||
|
||||
**Diagnosis is right.** `nodeCache` (`backend/protondrive/cache.go:12-19`) is per `client`/`Fs` (`nodes.go:189`). `listChildren` (`nodes.go:753-757`) and `findChild` (`nodes.go:894-905`) trust a cached complete listing, and a miss in it returns `errNotFound` with no server check. `cache.add` (`cache.go:140-157`) appends only to this cache's listing. So Fs B holding a complete listing of folder X never learns of a file Fs A creates in X. There is no TTL or any expiry anywhere in `cache.go`; only `invalidate*`/`clear` and the Move/DirMove cross-Fs invalidation.
|
||||
|
||||
**Real user impact**
|
||||
- `--compare-dest` / `--copy-dest` (+`--immutable`) / `checksum --download`: only wrong when the reference Fs's folder is written through another Fs of the same account in the same process, after the reference Fs listed it. Normal use (reference dir is pre-existing and untouched) is fine. Overlapping `rclone sync remote:a remote:a/sub` style commands are the realistic hit. Low frequency.
|
||||
- Mount + a second `rclone` command: this is a separate, larger staleness, not the one the tests hit. Different processes share nothing, and the mount's cache has no TTL and no events (M5 absent), so the mount never sees external writes until restart; VFS `--dir-cache-time` refreshes call `List`, which is served from the backend cache. The option help (`protondrive.go:189-197`) and `docs/content/protondrive.md` warn about it, and `enable_caching` defaults to true. The implementer's TODO entry does not mention it; the user docs (Task 20) should keep the mount warning.
|
||||
- Wrong-answer severity: a stale miss makes `--compare-dest` re-copy a file (safe) but `--copy-dest --immutable` and Put-after-miss paths can attempt a create that the server rejects with 2500 (exists) rather than silently overwrite; I did not run this.
|
||||
|
||||
**Fix options (recommend, not implement)**
|
||||
1. Cheap and spec-compatible, helps mounts but not the five tests: a listing TTL (e.g. 1 to 5 minutes, only for listings, checked in `cache.children`). Budget tests run in seconds, so `TestBudgetUpload`/`TestBudgetMkdir`/`TestBudgetList` (warm listing = 0 requests) are unaffected. Needs a clock injected into `nodeCache` for tests. It does not cover the five tests, which finish in milliseconds.
|
||||
2. Re-list on miss, whatever flavour (always, age-limited, or only in read paths): breaks the budgets, because `operations.Copy` calls `NewObject` (a miss for new files) before every Put; any per-miss server check is a request per new file. A name-hash existence probe (`check_available_hashes`) is one request per miss: same budget break. An age gate ("only re-list if the listing is older than N ms") would be timing dependent and still fail tests that run within N.
|
||||
3. A cache shared by the Fs of one account fixes the tests properly but needs registry state keyed by account; the spec's no-package-state rule forbids it. The only spec-compatible shape is something owned by a shared object the Fs already receive, and none exists. Owner decision, as the implementer says.
|
||||
4. Test-side alternative to `ignore`: run those five against a second fake remote with `enable_caching=false` (they pass uncached per the report) so they still cover the backend's uncached path. Costs a second init.d script and port, so probably not worth it before M5.
|
||||
|
||||
Recommendation: keep `ignore`, keep the TODO, add the mount/no-TTL point to the TODO entry, and consider option 1 as a small separate task. Events (M5) are the right real fix.
|
||||
|
||||
## Issues
|
||||
|
||||
### Critical
|
||||
None.
|
||||
|
||||
### Important
|
||||
None.
|
||||
|
||||
### Minor
|
||||
|
||||
1. `fstest/testserver/init.d/TestProtonDriveFake:55-61` `stop` (lines 63-76) sends only SIGTERM and after 10 s removes the pid file whether or not the process died, so a hung protonfake would be orphaned and untracked (port stays bound; the next start then fails with "address already in use"). Add `kill -9` after the wait loop. Low likelihood (the fake exits on SIGTERM).
|
||||
2. `TestProtonDriveFake:57` unused loop variable `i` (shellcheck SC2034); use `for _ in $(seq ...)`.
|
||||
3. `TestProtonDriveFake:24` build output goes to stderr, which `testserver.run` merges with stdout via `CombinedOutput` and then parses with `^key=value`. A go build diagnostic that happened to look like `a=b` would be exported as an env var. Not realistic; redirecting build output to `${RUN_STATE}/build.log` and printing it only on failure would remove the risk.
|
||||
4. `TestProtonDriveFake:24` rebuilds on every first start with a cold `go build`, and `testserver.start` has no timeout. On a fresh slow runner with a cold build cache this could take minutes but works. Acceptable.
|
||||
5. `.gitea/workflows/ci.yml:5-7` `push:` and `pull_request:` with no filter and no `concurrency` group: a push to a PR branch runs the job twice, and superseded pushes keep running. On a slow runner this contradicts "light". Add `concurrency: {group: ci-${{ github.ref }}, cancel-in-progress: true}`, or limit `push` to `master`.
|
||||
6. `ci.yml:44-48` fuzz loop: `go test -list` compiles the test binary, and each of the 5 `-fuzz` runs links again (about 5 compiles plus 100 s of fuzzing). Fine but not cheap; running all targets as one `go test -fuzz` is impossible, so this is the minimum. `go-version-file: go.mod` gives Go 1.26.0, while golangci-lint v2.15.0 is a prebuilt binary; if its build Go is older than the module's `go 1.26.0` the lint step fails ("version lower than targeted"). Host lint passed with Go 1.27.4 only. Worth watching on first push.
|
||||
7. `ci.yml:26,51` `go run ...govulncheck@latest` is unpinned, so the job can break without a code change (brief mandated it). `go build ./...` builds all of rclone, the heaviest step on a slow runner (brief mandated it).
|
||||
8. `docs/TODO.md` "Now" entry: the line `Remaining: Task 19 (...` is a very long line, unlike the surrounding wrapped text (cosmetic).
|
||||
9. `fstest/test_all/config.yaml`: a bare `-backends protondrive` without `-remotes` runs both the fake and the live `TestProtonDrive:` entries. Pre-existing behaviour, but now that two entries exist, PERMISSIONS.md should say always pass `-remotes TestProtonDriveFake:` (the verified-commands row does).
|
||||
|
||||
## Assessment
|
||||
|
||||
**Task quality:** Approved
|
||||
|
||||
**Reasoning:** The script's start/stop/refcount/failure paths work as run, nothing can reach the live remotes, and CI plus docs match the brief; remaining items are minor hardening. The five ignored tests are a genuine consequence of per-Fs caches with no expiry, and the implementer's refusal to add package state or a per-miss re-list is correct given the budget tests.
|
||||
@@ -0,0 +1,30 @@
|
||||
### Task 19: Remove the old libraries
|
||||
|
||||
**Files:**
|
||||
|
||||
- Delete: `backend/protondrive/crypto_diff_internal_test.go`
|
||||
- Keep: `backend/protondrive/testdata/oldlib/` and move `TestFrozenFixtures`
|
||||
and `TestFrozenFileFixture` into `crypto_internal_test.go` /
|
||||
`download_internal_test.go` (they need no old-library import).
|
||||
- Modify: `go.mod`, `go.sum`
|
||||
|
||||
> Read "Notes for Tasks 18 to 21" in the handoff before starting: `master`
|
||||
> is no longer the old backend, `TestFrozenFileFixture` has already moved,
|
||||
> and `docs/TODO.md` has the accurate list of what to keep and delete.
|
||||
|
||||
- [ ] **Step 1:** Record the current binary size:
|
||||
`git stash -u` is not allowed - instead build `master` in a worktree:
|
||||
`git worktree add ../rclone-master master && (cd ../rclone-master && go build -o /tmp/rclone-old .)`,
|
||||
note `stat -c %s /tmp/rclone-old`, then `git worktree remove ../rclone-master`.
|
||||
- [ ] **Step 2:** Move the frozen-fixture tests, delete the differential
|
||||
file, `go mod tidy`.
|
||||
- [ ] **Step 3: Verify:**
|
||||
`grep -c 'Proton-API-Bridge\|rclone/go-proton-api\|replace' go.mod`
|
||||
prints 0; `go build -o /tmp/rclone-new . && stat -c %s /tmp/rclone-new`
|
||||
is not larger than the old size; `go test -race ./backend/protondrive/...`
|
||||
passes; `go run bin/cross-compile.go -compile-only` (see
|
||||
`Makefile` target `compile_all` for the exact flags) succeeds for all
|
||||
targets; `RCLONE_CONFIG=/notfound go test ./...` has no new failures
|
||||
compared with the list in `docs/TODO.md`.
|
||||
- [ ] **Step 4: Commit** `protondrive: drop Proton-API-Bridge and go-proton-api`.
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
# Task 19 report: remove the old libraries
|
||||
|
||||
**Status:** DONE_WITH_CONCERNS (the one concern is a bisync test failure that was already there before this work; details below)
|
||||
**Commit:** `cf7577dc1 protondrive: drop Proton-API-Bridge and go-proton-api` (base `b667a1ab0`)
|
||||
|
||||
## What was done
|
||||
|
||||
- Moved these from `crypto_diff_internal_test.go` into the end of
|
||||
`crypto_internal_test.go`: `oldLibDir`, `oldLibFixture`,
|
||||
`loadOldLibFixture`, `unlockedKeyRing`, `checkOldLibFixture` and
|
||||
`TestFrozenFixtures`. `TestFrozenFixtures` no longer has its `-update`
|
||||
branch, and the `updateOldLib` flag is gone. Added the `os` and
|
||||
`path/filepath` imports. The `oldLibDir` comment now names the library
|
||||
versions.
|
||||
- `TestFrozenFileFixture` was already in `download_internal_test.go` and
|
||||
is unchanged.
|
||||
- Deleted `crypto_diff_internal_test.go` with `git rm`.
|
||||
- Ran `go mod tidy`. The result is within the ruling (details below).
|
||||
- Docs, all in the same commit:
|
||||
- `docs/TODO.md`:
|
||||
- Closed (removed) the entry "when go-proton-api and Proton-API-Bridge
|
||||
leave `go.mod`" and the entry "`go mod tidy` would drop
|
||||
Proton-API-Bridge".
|
||||
- Changed the crypto-refresh entry to say `TestNewWriterOldReader` was
|
||||
removed.
|
||||
- Added an entry for the lost coverage.
|
||||
- Updated the "Now" entry to Tasks 1-19 done.
|
||||
- Added a Known issues entry for the `cmd/bisync` failure.
|
||||
- `docs/ARCHITECTURE.md`: removed the "Known rough edges" bullet about
|
||||
the old libraries.
|
||||
- `docs/DECISIONS.md`: new entry "Proton Drive keeps frozen old-library
|
||||
fixtures, not the old library".
|
||||
- `docs/plans/001-protondrive-tier1.md`: ticked Task 19's steps.
|
||||
- The handoff's current-state bullets no longer list the differential
|
||||
file and no longer say the libraries are still in `go.mod`.
|
||||
- Left alone: `.claude/settings.json` has an uncommitted change
|
||||
(adds `Bash(golangci-lint run:*)` to allow) that I did not make. It is
|
||||
not in the commit, and I did not edit `docs/PERMISSIONS.md` for it.
|
||||
|
||||
## go.mod / go.sum
|
||||
|
||||
The tidy only removed lines. Nothing was added, upgraded or downgraded,
|
||||
and there are no `replace` lines.
|
||||
|
||||
- Direct requirements removed:
|
||||
- `github.com/rclone/Proton-API-Bridge v1.0.5`
|
||||
- `github.com/rclone/go-proton-api v1.0.4`
|
||||
- `github.com/relvacode/iso8601 v1.7.0` (left `go.mod` entirely)
|
||||
- Indirect requirements removed:
|
||||
- `ProtonMail/gluon`
|
||||
- `PuerkitoBio/goquery`
|
||||
- `andybalholm/cascadia`
|
||||
- `bradenaw/juniper`
|
||||
- `emersion/go-message`
|
||||
- `emersion/go-vcard`
|
||||
- `go-resty/resty/v2`
|
||||
- `sirupsen/logrus`
|
||||
- `go.mod`: 11 deletions.
|
||||
- `go.sum`: 20 deletions, covering the modules above. `logrus` stays in
|
||||
`go.sum` because other modules' `go.mod` files refer to it.
|
||||
- `grep -c 'Proton-API-Bridge\|rclone/go-proton-api\|replace' go.mod` prints `0`.
|
||||
|
||||
## Binary size
|
||||
|
||||
Both binaries were built with plain `go build -o <path> .` and the same
|
||||
Go 1.27.4.
|
||||
|
||||
| Build | Bytes |
|
||||
|---|---|
|
||||
| Old: `06700aaf2` (old backend), built in a scratch worktree | 152,460,576 |
|
||||
| New: `cf7577dc1` content | 148,995,400 |
|
||||
|
||||
The new binary is 3,465,176 bytes smaller (about 3.3 MiB). The saving
|
||||
comes from the backend rewrite as a whole. The libraries were already
|
||||
test-only before this task, so removing them does not change the binary.
|
||||
|
||||
## Test coverage lost
|
||||
|
||||
These tests were deleted with the differential file:
|
||||
|
||||
- `TestNewWriterOldReader`: the old library reads what the new writer
|
||||
writes.
|
||||
- `TestOldWriterNewReader`
|
||||
- `TestKeyUnlockMatchesOldLibrary`
|
||||
- `TestUploadReadableByOldLibraryCrypto`
|
||||
- `TestOldLibraryFileReadable`: legacy and v6 formats at 0 B, 2500 B and
|
||||
5 MiB + 3 B.
|
||||
|
||||
What remains is `TestFrozenFixtures` (one folder and one file node) and
|
||||
`TestFrozenFileFixture` (legacy and v6 stored files). Both are single
|
||||
block.
|
||||
|
||||
Lost as a result:
|
||||
|
||||
- Old-library files with several blocks.
|
||||
- Empty old-library files.
|
||||
- Any check that released rclone (v1.76 and older) can read what the new
|
||||
backend writes. This matters for users who downgrade.
|
||||
|
||||
All of this is logged in `docs/TODO.md`. The fixtures can no longer be
|
||||
regenerated.
|
||||
|
||||
## Verification (all run in this session)
|
||||
|
||||
- `go build ./...`: ok
|
||||
- `go vet ./backend/protondrive/...`: ok
|
||||
- `golangci-lint run ./backend/protondrive/...`: `0 issues.` (re-run after
|
||||
the last doc edit too)
|
||||
- `gofmt -l backend/protondrive`: empty
|
||||
- `RCLONE_CONFIG=/notfound go test -count=1 -run TestFrozen -v ./backend/protondrive/`:
|
||||
`TestFrozenFixtures` and `TestFrozenFileFixture` pass.
|
||||
- `RCLONE_CONFIG=/notfound go test -race -count=1 ./backend/protondrive/...`
|
||||
(5m19s):
|
||||
- `backend/protondrive`: ok, 316s
|
||||
- `protontest`: ok, 39.5s
|
||||
- `protontest/cmd/protonfake`: ok, 1.4s
|
||||
- Cross-compile:
|
||||
- Command: `mkdir build && go run bin/cross-compile.go -no-clean -compile-only v1.77.0-beta.task19`.
|
||||
This is the Makefile `compile_all` command. `BUILD_FLAGS`, `BUILDTAGS`
|
||||
and `BUILD_ARGS` are empty here. I used `-no-clean` so the script would
|
||||
not run `rm -rf build` itself.
|
||||
- Result: "Compiled 29 arches in 10m33s", exit 0, no failures. These are
|
||||
all the OS/arch pairs in `osarches`: windows 386/amd64/arm64, darwin
|
||||
amd64/arm64, linux 386/amd64/arm/arm-v6/arm-v7/arm64/mips/mipsle,
|
||||
freebsd 386/amd64/arm/arm-v6/arm-v7, netbsd
|
||||
386/amd64/arm/arm-v6/arm-v7, openbsd 386/amd64, plan9 386/amd64,
|
||||
solaris amd64, aix ppc64. `js/wasm` is commented out upstream.
|
||||
- Clean-up: removed `build/` file by file with `rm` and then `rmdir`.
|
||||
- Whole repo: `RCLONE_CONFIG=/notfound go test ./...` (1m51s): 186
|
||||
packages ok, 1 FAIL, `cmd/bisync` (`TestRcBackupDir/backupDir1` and
|
||||
`/backupDir2`: "unable to find file .../backup/file0.txt").
|
||||
- It fails every time: 3 more package runs and `-count=5` all failed.
|
||||
- It fails the same way on unchanged HEAD `b667a1ab0` and on the
|
||||
pre-rewrite base `06700aaf2`, both run in a scratch worktree that I
|
||||
removed afterwards. So it was already failing on this host before my
|
||||
change, or the host changed, and it is not a regression from this
|
||||
work. It only touches the local backend.
|
||||
- The baseline failure `cmd/serve/s3` passed in this run.
|
||||
- Both facts are recorded under "Known issues" in `docs/TODO.md`.
|
||||
|
||||
## Commands run (in order, abridged)
|
||||
|
||||
```
|
||||
git worktree add <scratch>/rclone-old 06700aaf2 && (cd <scratch>/rclone-old && go build -o <scratch>/rclone-old-bin .)
|
||||
stat -c %s <scratch>/rclone-old-bin # 152460576
|
||||
python3 (move fixture code into crypto_internal_test.go); git rm backend/protondrive/crypto_diff_internal_test.go
|
||||
go mod tidy
|
||||
grep -c 'Proton-API-Bridge\|rclone/go-proton-api\|replace' go.mod # 0
|
||||
go build ./... && go vet ./backend/protondrive/... && go build -o <scratch>/rclone-new-bin . && stat -c %s ... # 148995400
|
||||
RCLONE_CONFIG=/notfound go test -count=1 -run TestFrozen -v ./backend/protondrive/
|
||||
RCLONE_CONFIG=/notfound go test -race -count=1 ./backend/protondrive/...
|
||||
golangci-lint run ./backend/protondrive/...
|
||||
git worktree remove <scratch>/rclone-old
|
||||
mkdir build && go run bin/cross-compile.go -no-clean -compile-only v1.77.0-beta.task19 ; rm build files ; rmdir build/*/ build
|
||||
RCLONE_CONFIG=/notfound go test ./...
|
||||
RCLONE_CONFIG=/notfound go test -count=1 ./cmd/bisync/ (x3, plus -count=5 -run TestRcBackupDir)
|
||||
git worktree add --detach <scratch>/rclone-head HEAD ; bisync tests ; git checkout --detach 06700aaf2 (in worktree) ; bisync tests ; git worktree remove <scratch>/rclone-head
|
||||
```
|
||||
|
||||
`<scratch>` = `/tmp/claude-1000/-home-user-rclone/8cc17642-4f33-4045-ad2c-177bf869b82a/scratchpad`
|
||||
|
||||
## Self-review
|
||||
|
||||
- The moved code is copied as it was. The only behaviour change is that
|
||||
`TestFrozenFixtures` has no `-update` branch.
|
||||
- `Comment` fields in the fixture JSON may still mention the old library.
|
||||
That is intended: they describe where the data came from.
|
||||
- No remaining references to the deleted tests outside the historical
|
||||
plan and workspace files.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 19 review (b667a1ab0..cf7577dc1)
|
||||
|
||||
### Spec Compliance
|
||||
- ✅ Spec compliant (with the handoff "Task 19" notes applied: base 06700aaf2 for old binary, TODO entry followed).
|
||||
- Verified: `git diff` of go.mod/go.sum is 11 + 20 deletions, no additions. Removed: Proton-API-Bridge, go-proton-api, relvacode/iso8601, ProtonMail/gluon, goquery, cascadia, juniper, go-message, go-vcard, resty/v2, logrus (plus their go.sum lines). Nothing else changed. `go mod tidy -diff` is clean, `go build ./...` passes, `go mod verify` ok, no .go file imports any removed module.
|
||||
- Not re-run (taken from report): size check, race suite, cross-compile, full `go test ./...`.
|
||||
- ⚠️ cmd/bisync TestRcBackupDir failure is claimed pre-existing (also at 06700aaf2); not independently verified. Handoff said only cmd/serve/s3 was the baseline, so controller may want to eyeball it.
|
||||
|
||||
### Strengths
|
||||
- crypto_diff_internal_test.go deleted; fixture helpers moved verbatim (crypto_internal_test.go:870-1010), `-update` branch and flag dropped cleanly, imports tidy, vet clean.
|
||||
- Frozen tests are not vacuous: `checkOldLibFixture` (crypto_internal_test.go ~930-1010) decrypts node keys, name, hash key, content key, block, block signature, manifest and xattrs, with require.NoError on each decrypt and assert.Equal against stored plaintexts; `n==2` guards against a deleted fixture. Ran `TestFrozenFixtures` (file, folder) and `TestFrozenFileFixture` (legacy, v6): PASS. Fixtures (folder.json, file.json, file/{legacy,v6}/*) still present.
|
||||
- Docs: TODO entries closed/rewritten accurately, lost coverage and downgrade risk logged, DECISIONS entry honest, ARCHITECTURE rough-edge removed.
|
||||
|
||||
### Issues
|
||||
#### Critical
|
||||
None.
|
||||
#### Important
|
||||
None.
|
||||
#### Minor
|
||||
- docs/plans/001-protondrive-tier1-HANDOFF.md:56 still says "19-21 Not started" and the table at :55 says 1-18 implemented; the bullet at :64 was updated to say Task 19 is done, so the file contradicts itself. Update the table to 1-19 / 20-21.
|
||||
- docs/plans/001-protondrive-tier1-HANDOFF.md:378 (ruling 13) says Proton-API-Bridge stays in go.mod until Task 19; now historical. Acceptable as a decision record, but could note "removed in Task 19".
|
||||
- Lost coverage (multi-block, empty old-library files, downgrade compat with released rclone) is admitted and logged. Cheap partial mitigation not taken: the fixtures could not be regenerated, so multi-block old-library fixtures are impossible without the library; however the downgrade direction (new writer -> old reader) could have been kept as a committed fixture only by writing it with the old library before removal (e.g. one new-writer file blob plus a note). This is now only possible by checking out 06700aaf2/b667a1ab0. Suggest, if downgrade matters, a one-off generation from b667a1ab0 of a multi-block old-library file (e.g. 2 blocks) and a new-writer output, committed under testdata/oldlib/. Not blocking.
|
||||
- crypto_internal_test.go oldLibFixture comment still says "one block of content"; accurate for current fixtures.
|
||||
|
||||
### Assessment
|
||||
**Task quality:** Approved
|
||||
|
||||
**Reasoning:** go.mod/go.sum changes are pure, tidy-clean removals of only the two libraries and their exclusive dependencies, build passes, and the retained frozen-fixture tests perform real assertions that pass. Only a stale handoff status table remains.
|
||||
@@ -0,0 +1,32 @@
|
||||
### Task 20: Documentation
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `docs/content/protondrive.md` (hand-written parts only; leave the
|
||||
autogenerated options block between its markers alone)
|
||||
- Modify: `docs/data/backends/protondrive.yaml`
|
||||
- Modify: `docs/ARCHITECTURE.md`, `docs/DECISIONS.md`, `docs/TODO.md`
|
||||
- Create: `docs/specs/001-conformance-checklist.md`
|
||||
|
||||
- [ ] **Step 1:** Rewrite the docs page per spec section 10: both sign-in
|
||||
methods (fix the stale `user>`/`pass` transcript), the third-party
|
||||
notice verbatim, what is verified on upload and download, the ranged-read
|
||||
caveat, the `--dump` warning from spec section 7, limits (name rules,
|
||||
no SetModTime, trash semantics), removal of the "Beta" note and the
|
||||
`henrybear327` library text. Do not change the tier.
|
||||
- [ ] **Step 2:** Run `python3 bin/manage_backends.py features protondrive`
|
||||
if it works offline with the fake remote; otherwise edit
|
||||
`protondrive.yaml` `features:` by hand to match the new feature set and
|
||||
set `features_score: 5`. Leave `tier`, `maintainers`,
|
||||
`integration_tests` untouched.
|
||||
- [ ] **Step 3:** `docs/specs/001-conformance-checklist.md`: one table row
|
||||
per SDK module (`apiService`, `shares`, `nodes`, `upload`, `download`,
|
||||
`crypto`, account/auth) with the SDK file, the Go file, the SDK tag last
|
||||
checked (`js/v0.23.0`), and known deliberate differences (no small-file
|
||||
path, no events yet, no thumbnails).
|
||||
- [ ] **Step 4:** `docs/ARCHITECTURE.md` (protondrive section replacing the
|
||||
"known rough edges" line), a `docs/DECISIONS.md` entry for the native
|
||||
port, `docs/TODO.md` entries for M5, M6, M7 and every live check not
|
||||
run.
|
||||
- [ ] **Step 5: Commit** `protondrive: update docs for the rewritten backend`.
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
# Task 20 report - documentation
|
||||
|
||||
Status: DONE_WITH_CONCERNS
|
||||
|
||||
## Files and what they now say
|
||||
|
||||
### `docs/content/protondrive.md` (hand-written parts only)
|
||||
|
||||
The autogenerated options block (between its markers) is byte-identical
|
||||
to before; checked with `diff` of the marker range before replacing the
|
||||
file. Sections:
|
||||
|
||||
- Intro: native client following Proton's Drive SDK; the third-party
|
||||
notice verbatim ("rclone is a third-party application not officially
|
||||
supported by Proton." - copied from `thirdPartyNotice`,
|
||||
`protondrive.go:49`). "Beta" note, old-library and `henrybear327` text,
|
||||
and the "keys must have been generated in the browser" note removed
|
||||
(the client creates a missing Drive volume, `nodes.go` `createVolume`).
|
||||
- Configuration: both sign-in methods; when browser sign-in is the
|
||||
default (no username+password set: `configure`, state `method`); the
|
||||
CAPTCHA -> reconnect path (`errCaptcha` handling in `connect` and
|
||||
`configure`); saved session in the config, advice to encrypt it.
|
||||
Transcript (replacing the stale `user>`/`pass` one) was captured from
|
||||
the real binary: `rclone config` run inside `unshare -rn` (no network
|
||||
at all) against a locally started `protonfake`, so every prompt text is
|
||||
what the code prints; the payload in the URL is replaced by `XXXXXXXX`,
|
||||
and the end summary and the long 2fa/otp help are `[snip]`ped. Password
|
||||
flow: the `config_2fa` prompt (`fs.ConfigInput("password_2fa",
|
||||
"config_2fa", ...)`), later re-sign-in using `otp_secret_key`
|
||||
(`passwordSignIn`) or a fatal "run rclone config reconnect" error
|
||||
(`connect`); reconnect on a signed-in remote asks first and signs the
|
||||
old session out (`reauth` state, `signOut`); 10 minute wait
|
||||
(`forkPollTimeout`); sign-in link holds a secret (`showSignInURL`
|
||||
comment, `forkStart` payload).
|
||||
- Modification times and hashes: stored per upload, 1 s precision
|
||||
(`Precision`), `SetModTime` -> `ErrorCantSetModTime`; folder time set at
|
||||
creation (`CreateDir` passes `time.Now()`); SHA1 is the uploader's
|
||||
recorded value, empty when absent (`node.sha1`).
|
||||
- Integrity and signatures: per-block SHA-256 check with three attempts
|
||||
(`blockAttempts`, `errBlockHash`); manifest verified before the first
|
||||
byte of a full read (`openRead` full path, `verifyManifest`); ~1
|
||||
request per 80 MiB first-byte cost (`blocksPerPage` 20 x 4 MiB);
|
||||
node/content key signatures (`contentKey`). Upload: everything signed,
|
||||
`checkBlock` before sending, size/SHA1 checked before commit
|
||||
(`uploadResult.check`). Ranged-read caveat (`open` doc comment;
|
||||
whole-file range takes `openWhole`, `Object.Open`); files without block
|
||||
sizes may fail ranged reads not from the start (error at
|
||||
`download.go:616`). Root of trust (`nodes.go:353-372`), subtree distrust
|
||||
(`decryptNode` `parent.unverified`), refusals for reads, writes into
|
||||
unverified folders (`targetFolder`) and rename/move/copy of unverified
|
||||
nodes (`nodes.go:1129-1139`); with the flag an ERROR per node and
|
||||
operation (`reportUnverified`/`reportOnce`) and the node re-signed;
|
||||
block hashes never disabled. Name signature failures only logged
|
||||
(`decryptNode` verifyErrs, `fs.Errorf`). Both protocol limits (whole
|
||||
older revision; foreign signers checked against server-supplied keys).
|
||||
- Restricted filename characters: 255 UTF-16 code units, not empty, `.`,
|
||||
`..`, no `/` or NUL (`validateName`); default encoding table derived
|
||||
from `encoder.Base|InvalidUtf8|LeftSpace|RightSpace` (Base = Zero,
|
||||
Slash, Dot); non-UTF-8 refused if the encoding allows it through.
|
||||
The old claim that spaces are "removed" was wrong (they are encoded).
|
||||
- Duplicated files: name taken by a file -> new revision; own drafts
|
||||
replaced, other clients' drafts need `replace_existing_draft`
|
||||
(`resolveConflict`, error text at `upload.go:291`); "upload outcome
|
||||
unknown: it may have succeeded" quoted from `errUploadOutcomeUnknown`,
|
||||
not retried at once (`NoLowLevelRetryError` in `translateError`).
|
||||
- Deleting files: Remove/Purge/Rmdir trash (`trash`), `cleanup` empties
|
||||
the whole volume trash (`emptyTrash`, DELETE volume trash). `rmdir`
|
||||
asks the server (`hasChildren`) so hidden children block it.
|
||||
- Items left out of listings: Docs/Sheets with one notice (`listable`),
|
||||
undecryptable items with an ERROR (`decryptNode` unreadable path).
|
||||
- Mailbox password, Caching (in-memory, no external changes seen,
|
||||
`enable_caching=false` advice), Logs and `--dump`: own logs carry no
|
||||
secrets (spec 7, `security_test.go`); `--dump headers` shows `x-pm-uid`
|
||||
(`session.do`) and `pm-storage-token` (`download.go:273`,
|
||||
`upload.go:828`); `--dump bodies` shows tokens (per `docs/TODO.md`
|
||||
entry from `TestNoSecretsInLogs`).
|
||||
- Limitations (after the generated block, replacing the old library
|
||||
text): only My Files (`bootstrap` uses `drive/v2/shares/my-files`
|
||||
only); no change events/ChangeNotify; no `--metadata`; 2^19-block cap
|
||||
(`maxFileBlocks`), depth 2048; `about` is account-wide
|
||||
(`/core/v4/users` `UsedSpace`/`MaxSpace`); 4 MiB blocks,
|
||||
`upload_concurrency` default 4 max 5 (`defaultUploadConcurrency`,
|
||||
`maxUploadConcurrency`, cap in `NewFs`), 10 up + 10 down per remote
|
||||
(`blocksInFlight` token dispensers).
|
||||
|
||||
Rulings 18-28 coverage: 18, 21, 22, 23 (Integrity section), 19 and 26
|
||||
(full read / ranged caveat), 24 and 25 (Limitations), 28 (filename
|
||||
section). 20 (test budget) and 27 (listing cache generation) are not
|
||||
user-visible and are not documented. Earlier user-visible rulings also
|
||||
covered: 4 (implicitly, saved session refresh), 6, 10 is not mentioned
|
||||
(internal pacing detail), 11, 14, 15/Disconnect not mentioned
|
||||
specifically, 16, 17.
|
||||
|
||||
Not claimed: browser sign-in, 2FA and desktop/mobile interop are
|
||||
described but never said to be tested; nothing says integration tests
|
||||
pass; no mention of ChangeNotify, Metadata, small-file upload or
|
||||
thumbnails as features.
|
||||
|
||||
### `backend/protondrive/protondrive.go`
|
||||
|
||||
`ignore_signature_errors` `Help:` rewritten: the old text said only
|
||||
"read files"; the code (rulings 18, 21, 22) also gates writes into, and
|
||||
rename/move/copy of, unverified nodes, and the subtree rule. New text
|
||||
names those and says block hashes are always checked. No other Go
|
||||
change.
|
||||
|
||||
### `docs/data/backends/protondrive.yaml`
|
||||
|
||||
`bin/manage_backends.py features` was not run as is: it reads `remote:`
|
||||
from the yaml (`TestProtonDrive:`, the live remote) and runs the system
|
||||
`rclone` with the user's config, which would contact Proton. Instead its
|
||||
exact computation was reproduced offline: built rclone, started
|
||||
`fstest/testserver/init.d/TestProtonDriveFake` (local `protonfake`),
|
||||
checked the endpoint was `http://127.0.0.1:28642` before running
|
||||
`RCLONE_CONFIG=/notfound rclone backend features TestProtonDriveFake:`
|
||||
(env-var remote), took the sorted true `Features`, `Hashes`, `Precision`.
|
||||
Result written by hand: added Copy, ListP, PutStream, Shutdown; removed
|
||||
NoMultiThreading; hashes `sha1` and precision 1000000000 unchanged;
|
||||
`features_score: 5`. `tier`, `maintainers`, `integration_tests`
|
||||
untouched.
|
||||
|
||||
### `docs/specs/001-conformance-checklist.md` (new)
|
||||
|
||||
One row per SDK module (`apiService`, `shares`, `nodes`, `upload`,
|
||||
`download`, `crypto`, account/auth, plus `events` as not implemented):
|
||||
SDK files (paths checked to exist in `~/.cache/protondrive-sdk-ref`,
|
||||
read-only listing only), Go files, `js/v0.23.0`, deliberate
|
||||
differences drawn from the rulings and code (no small-file path, no
|
||||
events, no thumbnails, root of trust, subtree distrust, manifest-first,
|
||||
block cap, depth limit, UTF-8, outcome-unknown, draft client UID,
|
||||
Retry-After cap 60 s `maxRetryAfter`, fork client ID `external-drive`,
|
||||
3 s adopt wait `adoptWait`).
|
||||
|
||||
### `docs/ARCHITECTURE.md`, `docs/DECISIONS.md`, `docs/TODO.md`
|
||||
|
||||
- ARCHITECTURE: protondrive section gains user-docs/checklist pointers
|
||||
and a "not implemented yet" line (M5, M6, ranged-read manifest). The
|
||||
"Known rough edges" heading was kept: its one line is about host
|
||||
tooling for the whole repo, not protondrive, so replacing it would
|
||||
lose that pointer.
|
||||
- DECISIONS: new entry "Proton Drive is a native port of Proton's Drive
|
||||
SDK" (checklist as the conformance mechanism, docs rules).
|
||||
- TODO: "Now" updated to Tasks 1-20 done; new entries for M5, M6, M7, one
|
||||
entry listing every live check not run (paid account, browser sign-in,
|
||||
2FA/OTP, CAPTCHA, official non-web clients, live fstests/test_all,
|
||||
drafts/trashed as children, non-ASCII address case), and one for the
|
||||
stale generated options block; the `--dump` entry no longer says
|
||||
"Task 20's docs must carry the warning".
|
||||
- Plan: Task 20 checkboxes ticked.
|
||||
|
||||
## Verification
|
||||
|
||||
- `go build ./...` OK; `go vet ./backend/protondrive/...` OK;
|
||||
`golangci-lint run ./backend/protondrive/...` 0 issues.
|
||||
- `RCLONE_CONFIG=/notfound go test ./backend/protondrive/...` all ok
|
||||
(protondrive 105 s).
|
||||
- Nothing contacted Proton: the features run and the transcript capture
|
||||
used only a local `protonfake` (the transcript one in a network-less
|
||||
namespace). No config or password file read. No dependency changes.
|
||||
|
||||
## Where code, docs and spec disagree
|
||||
|
||||
1. The generated options block in `docs/content/protondrive.md` is
|
||||
stale (old options, old help, no `upload_concurrency` or
|
||||
`ignore_signature_errors`). Left alone per AGENTS.md; logged in TODO.
|
||||
2. Spec 6.7 says filename rules are "non-empty, at most 255 characters,
|
||||
no `/`"; code also refuses `.`, `..`, NUL and non-UTF-8, and counts
|
||||
UTF-16 code units. Docs follow the code.
|
||||
3. `enable_caching` help still says "please disable this feature" for
|
||||
VFS mounts; true today (no events), so left.
|
||||
4. The local SDK checkout at `05c9115` carries only the tag
|
||||
`cs/v0.33.0`; `js/v0.23.0` is taken from the handoff and code comment.
|
||||
5. The brief says Step 4 replaces the "known rough edges" line; kept it
|
||||
(see above).
|
||||
|
||||
## Concerns
|
||||
|
||||
- The docs present browser sign-in as the primary method, as the spec
|
||||
says, but it has never been run against the real service (handoff
|
||||
section 6). The docs do not claim it is tested; Task 21 should check it
|
||||
before anything goes upstream.
|
||||
- `rclone about` "whole account" relies on `/core/v4/users` `UsedSpace`
|
||||
being account-wide, which is Proton's documented model but was not
|
||||
checked live for this wording.
|
||||
|
||||
## Fix round 1
|
||||
|
||||
### 1. Ranged reads of files without recorded block sizes (code fix)
|
||||
|
||||
Problem: `openRead` placed a range in a file without recorded block
|
||||
sizes by assuming 4 MiB blocks (`blockAt(nil, ...)`), and also bounded
|
||||
the last block that way. The size check in `next` (`checkAssumed`/
|
||||
`prevSize`) fired only when a second block followed a mis-sized one, so
|
||||
single-block ranges returned wrong bytes, and ranges ending early
|
||||
returned short data, silently.
|
||||
|
||||
Test first: `TestDownloadRangesUnrecordedBlockSizes`
|
||||
(`download_internal_test.go`) seeds a 6 MiB+5 file in 1 MiB blocks with
|
||||
no `BlockSizes` in its extended attributes (new fake helper
|
||||
`protontest.User.SeedFileInBlocks`; `seedSpec.blockSize`, `newRevision`
|
||||
takes the block size) and reads 9 ranges.
|
||||
|
||||
RED, before the fix (`RCLONE_CONFIG=/notfound go test -run
|
||||
TestDownloadRangesUnrecordedBlockSizes ./backend/protondrive/`):
|
||||
|
||||
```
|
||||
--- FAIL: TestDownloadRangesUnrecordedBlockSizes/0+3145728 got 1048576 bytes, want 3145728
|
||||
--- FAIL: TestDownloadRangesUnrecordedBlockSizes/1048573+10 got 3 bytes, want 10
|
||||
--- FAIL: TestDownloadRangesUnrecordedBlockSizes/4194314+100 got 100 bytes, want 100 (wrong bytes, silently)
|
||||
--- FAIL: .../4194314+-1, 2097159+3145728, 6291460+-1, 6291461+-1 unexpected error ("read it from the start")
|
||||
```
|
||||
|
||||
Fix (`download.go`): without recorded sizes the read starts at block 1
|
||||
with `skip = offset` and no last block (the remaining count stops it);
|
||||
`next` now discards whole blocks while `skip` covers them, then seeks
|
||||
into the block holding the offset. Every block is still checked against
|
||||
its listed hash (and its decryption); the manifest is still not checked
|
||||
for a ranged read. With recorded sizes nothing changes (skip is always
|
||||
inside the first block). `checkAssumed`, `prevSize`, the nil branch of
|
||||
`blockAt` and the FIXME are gone; `defaultBlockSize`'s comment now says
|
||||
it is the upload block size.
|
||||
|
||||
GREEN: the new test passes; `go test -count=3 -run
|
||||
'TestDownload|TestBudget'` passes.
|
||||
|
||||
Test expectation changed: `TestDownloadRanges`' unrecorded-size column
|
||||
(renamed from "assumed") now expects reads from the start, so ranges past
|
||||
block 1 fetch every block (3 for the 3-block file) where they fetched 1
|
||||
or 2. Its count is now a range (at least the blocks needed up to the
|
||||
range, at most all 3), since blocks past the range may be fetched ahead
|
||||
before the reader is closed. No budget test changed (`budget_test.go`
|
||||
doesn't read files without block sizes).
|
||||
|
||||
Docs: the page's ranged-read paragraph now says such reads download,
|
||||
check and discard every block before the range, so they are slower, and
|
||||
still don't check the manifest. The checklist's download row says the
|
||||
same.
|
||||
|
||||
### 2. SDK identified by commit
|
||||
|
||||
`docs/specs/001-conformance-checklist.md` identifies the SDK as commit
|
||||
`05c9115955589d9f6ab7392a36f8f0cb39327133` (2026-11-05), noting its
|
||||
`client/js/CHANGELOG.md` newest entry is `js/v0.23.0` (2026-10-19) and
|
||||
that the only tag on it is `cs/v0.33.0`; re-checks are a diff of the
|
||||
listed files between commits; the column reads "`05c9115`
|
||||
(js/v0.23.0)". The DECISIONS entry says the same. Left alone, as not
|
||||
written by me in this task: the package comment in `protondrive.go:5`
|
||||
("last checked against its tag js/v0.23.0") and older spec/handoff/TODO
|
||||
text.
|
||||
|
||||
### 3. enable_caching
|
||||
|
||||
The `Help:` drops the "please disable this feature" notice and says to
|
||||
disable the cache if other clients change the same files while rclone
|
||||
runs, as can happen with a mount; the page's Caching section uses the
|
||||
same wording.
|
||||
|
||||
### Verification
|
||||
|
||||
- `go build ./...` OK; `go vet ./backend/protondrive/...` OK;
|
||||
`golangci-lint run ./backend/protondrive/...` 0 issues.
|
||||
- `RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/...`: all
|
||||
ok (protondrive 313 s, protontest 38 s, protonfake 1.4 s).
|
||||
- No Proton host contacted, no secrets files read, no dependency change.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Task 20 re-review (fix round 1) - base 7be50f9e2, head 22b9c4fdc
|
||||
|
||||
Verdict: ALL CLOSED. No new Critical or Important findings.
|
||||
|
||||
## Findings from task-20-review.md
|
||||
|
||||
| Finding | Status | Evidence |
|
||||
|---|---|---|
|
||||
| Important 1: ranged read of a file without recorded block sizes could return wrong bytes silently | Closed (fixed in code, per controller ruling) | download.go:142-150 starts unrecorded-size reads at block 1 with `skip = offset`, no last block; download.go:592-625 `next` discards whole blocks while `skip >= size`, then seeks inside the block holding the offset. FIXME, `checkAssumed`, `prevSize` and the nil branch of `blockAt` are gone. |
|
||||
| Important 2: SDK cited by a tag not on the commit | Closed | docs/specs/001-conformance-checklist.md:7-13 cites full commit `05c9115955589d9f6ab7392a36f8f0cb39327133`, says `js/v0.23.0` is only the newest CHANGELOG entry and `cs/v0.33.0` is the only tag; re-check is a commit diff; column reads "`05c9115` (js/v0.23.0)". DECISIONS.md entry matches. |
|
||||
| Minor 2: enable_caching Help vs page | Closed | protondrive.go:~197-202 drops the "please disable" notice; Help and page Caching section now use the same "while rclone runs, as can happen with a mount" wording. |
|
||||
|
||||
## Code check: backend/protondrive/download.go
|
||||
|
||||
Arithmetic, unrecorded sizes (`first=1, skip=offset, last=-1`, `remaining=limit`):
|
||||
- Range inside block 0: block 1 has `skip < size`, seek to `skip`, read `remaining`. Correct.
|
||||
- Spanning blocks: after the first block is consumed, `next` returns later blocks with `skip == 0`; `Read` caps by `remaining`. Correct.
|
||||
- Exact block boundary (`skip == size`): `skip >= size` discards the block and carries `skip = 0` to the next one. Correct (test cases `1<<20-3+10` and the 4 MiB boundary in TestDownloadRanges cover neighbours; the 1 MiB-block file has no case exactly at 1 MiB, but the `>=` branch is hit by every multi-block skip).
|
||||
- Past EOF: every block is discarded, the end slot returns io.EOF, 0 bytes, no error (case `size,-1`). Correct; it downloads the whole file to find that out, which the docs' "slower" covers.
|
||||
- `limit -1`: `remaining` negative, never capped, ends at io.EOF. Correct.
|
||||
- Zero-length: `limit == 0` returns an empty reader before any request (download.go:139-141), unchanged.
|
||||
- Zero-size block: with `skip > 0` it is discarded (`skip >= 0`); with `skip == 0` it becomes `cur`, reads 0/EOF, and is closed. Correct.
|
||||
|
||||
Recorded sizes keep the direct-seek path: `first, skip = blockAt(sizes, offset)` and `last` as before (download.go:146-150). `blockAt` guarantees `skip < sizes[first-1]`, and fetchOnce (download.go:535-536) rejects any block whose size differs from the record, so the `skip >= size` discard branch is unreachable there and the old `Seek(min(skip,size))` clamp is not needed. Offsets at or past the recorded total return early with `limit = 0`.
|
||||
|
||||
Every block still verified: the unrecorded path lists from block 1 without `last`, and each block goes through `fetch` -> `fetchOnce` hash check (download.go:525-527) and decryption; `next` checks `res.err` before discarding, so an error in a discarded block still fails the read.
|
||||
|
||||
Early Close: unchanged. Close cancels before taking `mu`; a `next` blocked in its discard loop leaves via `ctx.Done()` or a ctx-aware fetch result; Close then waits `wg` and drains/frees queued slots (download.go:628-655). Discarded blocks are closed in the loop. No leak.
|
||||
|
||||
Security properties: `full` still requires offset 0 (download.go:117) and `open`/`openWhole` are unchanged, so full reads and whole-file ranges list every block and check the manifest before any byte; ranged reads still skip the manifest, as documented.
|
||||
|
||||
## Tests
|
||||
|
||||
- Mutation check run: scratch worktree at 22b9c4fdc with download.go reverted to 7be50f9e2, `RCLONE_CONFIG=/notfound go test -count=1 -run TestDownloadRangesUnrecordedBlockSizes ./backend/protondrive/`: 7 of 9 subtests FAIL (0+3145728, 1048573+10, 4194314+100, 4194314+-1, 2097159+3145728, 6291460+-1, 6291461+-1), matching the report's RED. With the fix, `-run 'TestDownloadRanges|TestDownloadLegacy'` passes. Worktree removed without --force; main working tree untouched.
|
||||
- The test does catch the silent wrong-bytes case (4194314+100 returns 100 wrong bytes before the fix) and the short-read case (0+3145728).
|
||||
- SeedFileInBlocks (protontest/drive.go:~640-650) is realistic: blocks are real encrypted 1 MiB blocks with correct per-block hashes and a signed manifest over them; `claimedBlockSizes: []int64{}` plus `BlockSizes ... omitempty` (drive.go:450) means the extended attributes omit BlockSizes, as an old client would, and the test asserts `n.X.BlockSizes == nil`. 1 MiB equals `minUnclaimedBlockSize`, the smallest the reader accepts, so the block-count bound is exercised at its edge.
|
||||
|
||||
## Docs
|
||||
|
||||
- docs/content/protondrive.md:207-211 accurately describes the new behaviour (downloads, checks and discards every block before the range; slower; manifest not checked).
|
||||
- Checklist download row adds the same sentence. Accurate.
|
||||
|
||||
## Package comment, protondrive.go:5
|
||||
|
||||
"last checked against its tag js/v0.23.0" should change: it repeats the inaccuracy closed in Important 2 (the checked commit is 05c9115, which `js/v0.23.0` does not point at), and unlike the checklist it is upstream-bound code. Suggest "last checked against commit 05c9115 (js/v0.23.0)". Minor, as it is a comment; fix before Task 21 / any upstream PR. docs/specs/001-protondrive-tier1.md:167-170 ("Reference pin ... the SDK tag ... re-checking it against new SDK tags") has the same wording; fork-only, Minor.
|
||||
|
||||
## Minor (new)
|
||||
|
||||
1. download_internal_test.go:621 TestDownloadEndlessBlockList still names its unrecorded case "assumed"; rename to "unrecorded" to match TestDownloadRanges.
|
||||
2. download_internal_test.go:636 `min(v.blocks, 1)*3` is always 3 for the table's rows (all `blocksUnrecorded >= 1`); a plain `3` (the file's block count) with a comment reads more clearly.
|
||||
3. protontest/drive.go:735 field comment "blockSize if zero" refers to the package constant of the same name as the field's own concept; "the package blockSize if zero" or naming the constant explicitly would avoid the ambiguity. The `newRevision` doc comment (drive.go:~736-739) now wraps oddly ("as signEmail, with its" on its own line).
|
||||
4. Performance note, not a defect: random-offset reads (mount without VFS cache, multi-thread downloads) of large unrecorded-size files download from the start each time. Ruled acceptable; consider a TODO.md entry if not already logged.
|
||||
@@ -0,0 +1,62 @@
|
||||
# Task 20 review (docs) - base c1464a39a, head 7be50f9e2
|
||||
|
||||
### Spec Compliance
|
||||
|
||||
- Verdict: spec compliant, with two Important wording fixes needed (below). Not a miss of any listed file: protondrive.md, protondrive.yaml, ARCHITECTURE/DECISIONS/TODO, conformance checklist all have hunks.
|
||||
- Cannot verify from the diff: that the transcript matches the real binary (report says captured offline; prompts spot-checked against protondrive.go only); the yaml features were produced offline per the report, but I confirmed them against the code (below).
|
||||
|
||||
### Strengths
|
||||
- Page is free of fork-only content (no fake server, plans, ruling numbers, accounts); no claim of live testing; browser sign-in never said to be tested.
|
||||
- Autogenerated block left untouched; Go `Help:` change limited to `ignore_signature_errors` and accurate.
|
||||
- Rulings 18-19, 21-26, 28, both protocol limits, `--dump` exposure, ranged-read caveat all present. 20 and 27 reasonably omitted as not user-visible.
|
||||
|
||||
### Claims spot-checked (page claim -> evidence)
|
||||
1. Third-party notice verbatim -> protondrive.go:49 (`thirdPartyNotice`), used in the config prompt at :596. OK.
|
||||
2. Browser sign-in default only when no password -> protondrive.go:~592 `fs.ConfigConfirm("choose", !hasPassword, ...)`. OK.
|
||||
3. CAPTCHA fails password sign-in, use reconnect -> protondrive.go:542, 633 (`errCaptcha`). OK.
|
||||
4. Re-sign-in uses otp_secret_key else asks reconnect -> protondrive.go:492-498, 535, 541. OK.
|
||||
5. 10 minute wait -> protondrive.go:54 `forkPollTimeout = 10*time.Minute`. OK.
|
||||
6. SetModTime unsupported; precision 1 s -> protondrive.go:1130 (`ErrorCantSetModTime`), :891-893. OK.
|
||||
7. Folder time set at creation -> protondrive.go:761 (`createFolder(..., time.Now())`). OK.
|
||||
8. SHA1 is the uploader's recorded value, empty if absent -> nodes.go:161-166; Hashes protondrive.go:1075. OK.
|
||||
9. Block hash check, 3 attempts -> download.go:54 `blockAttempts = 3`, :505. OK.
|
||||
10. ~1 request per 80 MiB -> download.go:46 `blocksPerPage = 20` x 4 MiB. OK.
|
||||
11. Manifest verified before first byte on full read; whole-file range takes the same path -> download.go:83-97, protondrive.go:1173 (`openWhole`). OK.
|
||||
12. Upload checks each block decrypts -> upload.go:652,668 (`checkBlock`). OK.
|
||||
13. Name rules (255 UTF-16 units, empty, `.`, `..`, `/`, NUL, UTF-8) -> crypto.go:500-519. OK. Default encoding Slash/Zero/Dot + spaces + InvalidUtf8 -> protondrive.go:~147-151 with `encoder.Base`. OK.
|
||||
14. Rmdir asks server, refuses non-empty; trash for Rmdir/Purge; CleanUp empties whole volume trash -> protondrive.go:867-885, 930-947, 949-951; nodes.go:1308. OK.
|
||||
15. `about` is account-wide -> nodes.go:1318-1323 (`/core/v4/users` UsedSpace/MaxSpace). OK (the report itself notes the "whole account" wording is not live-checked; code supports it).
|
||||
16. upload_concurrency default 4, max 5; 10 up/10 down per remote -> upload.go:44-45, nodes.go:50,187-188. OK.
|
||||
17. 2^19 blocks, depth 2048 -> download.go:71, nodes.go:42. OK.
|
||||
18. `--dump headers` exposes x-pm-uid, pm-storage-token -> session.go:248, download.go:273, upload.go:828. OK; "dump bodies" tokens rests on TODO.md:76-80 / TestNoSecretsInLogs only.
|
||||
19. Only My Files -> nodes.go:322 `drive/v2/shares/my-files`. OK.
|
||||
20. Root of trust / subtree unverified / refusals -> nodes.go:353-372, 939, 1139. OK.
|
||||
21. Draft replace wording -> upload.go:291, protondrive.go:172-183 Help. OK.
|
||||
22. "outcome unknown" text -> upload.go:35. OK.
|
||||
23. Docs/Sheets left out with one notice -> nodes.go:845, 884. OK.
|
||||
24. yaml features vs Fs: methods present About 1009, CleanUp 949, Copy 905, DirCacheFlush 1062, DirMove 1030, Disconnect 1054, ListP 802, Move 962, Purge 930, PutStream 852, Shutdown 1069; Features{} sets ReadMimeType, CanHaveEmptyDirectories (:674-677). No OpenWriterAt/ChangeNotify/Metadata/PublicLink methods exist. NoMultiThreading removal correct. Hashes sha1, precision 1e9 match. OK.
|
||||
25. All 40+ SDK paths in the checklist exist in the checkout (checked each with `[ -e ]`, none missing).
|
||||
|
||||
### Issues
|
||||
|
||||
#### Critical
|
||||
None.
|
||||
|
||||
#### Important
|
||||
1. docs/content/protondrive.md ("Ranged reads" paragraph) understates the failure mode. It says ranged reads of files without recorded block sizes "only work from the start ... one from elsewhere may fail with an error". The code's own FIXME at backend/protondrive/download.go:143-146 says such a read "returns the wrong bytes", and the error at download.go:615-617 only fires when a second block is read after a mis-sized one (`r.prevSize`), so a single-block ranged read starting past block 0 can return wrong data silently. Reword to "may fail with an error, or return the wrong bytes" (or fix the code). Overclaiming safety on a data-correctness path.
|
||||
2. docs/specs/001-conformance-checklist.md:9 and docs/DECISIONS.md (new entry) say "tag `js/v0.23.0` (commit `05c9115`)". The local checkout at 05c9115 carries only the tag `cs/v0.33.0` (a Kotlin client commit, "feat(client)!: create the Kotlin client..."); `js/v0.23.0` is not a git tag there. What does tie them: `client/js/CHANGELOG.md` in that checkout has `## js/v0.23.0 (2026-10-19)` as its newest entry, and `client/js/package.json` says version 0.0.1. Recommended wording: "SDK: github.com/ProtonDriveApps/sdk at commit 05c9115955589d9f6ab7392a36f8f0cb39327133 (2026-11-05; its `client/js/CHANGELOG.md` newest entry is `js/v0.23.0`, 2026-10-19). Local tags on that commit: `cs/v0.33.0` only." Use the commit hash as the identifier in the "Last checked" column ("05c9115 (js/v0.23.0)") and in DECISIONS.md, and say the re-check method is "diff the listed files between the two commits" rather than between tags (the file lists are JS paths; a commit diff is unambiguous).
|
||||
|
||||
#### Minor
|
||||
1. docs/content/protondrive.md autogenerated block is stale against the Go `Help:` fields (username/password still `Required: true`, old app_version/replace_existing_draft text including "Need to set to true for integration tests", no `upload_concurrency` or `ignore_signature_errors`), while the hand-written text refers to the missing options. Left alone per AGENTS.md and logged in TODO.md, which is acceptable, but nothing guarantees `make backenddocs` is run before any upstream PR; keep it in the Task 21 / pre-upstream checklist.
|
||||
2. protondrive.go:~182-190 `enable_caching` Help still opens with "Notice: ... please disable this feature" for VFS mounts, while the page now says to set it false only when other clients change the same files. Not wrong, slightly inconsistent; the report flagged it.
|
||||
3. docs/data/backends/protondrive.yaml: the yaml was hand-edited from an offline reproduction of `manage_backends.py` rather than the script; the result matches the code, so no action beyond noting it in the commit/handoff.
|
||||
4. "`--dump bodies` shows access, refresh and upload tokens" is supported only by a TODO.md note about a test, not by a code location; fine, but keep it phrased as it is (no stronger).
|
||||
5. Checklist "Last checked" repeats `js/v0.23.0` on every row although the review was one pass at one commit; harmless, but a single header line plus per-row blank-until-changed would age better.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Spec:** compliant (see Important items for wording corrections).
|
||||
|
||||
**Task quality:** Needs fixes (two Important wording items; both are small doc edits).
|
||||
|
||||
**Reasoning:** The page's behavioural claims match the code on all 25 spot-checks, no fork-only content, no live-testing overclaim, yaml matches Fs features. The ranged-read caveat under-describes a known wrong-bytes case, and the SDK version identifier in the checklist and DECISIONS entry should cite the commit hash.
|
||||
@@ -0,0 +1,37 @@
|
||||
### Task 21: Live verification (blocked until a test account exists)
|
||||
|
||||
> Out of date since 2026-11-11: two test accounts exist and are configured
|
||||
> as `TestProtonDrive:` (free) and `TestProtonDrivePaid:` (paid, never signed
|
||||
> in to) with username and password. See section 5 of the handoff for the
|
||||
> setup and section 4 for what to check first.
|
||||
|
||||
Needs the fork owner to create the account and configure it themselves as
|
||||
`TestProtonDrive:` in their rclone config (`rclone config`, browser
|
||||
sign-in). Credentials must never be pasted into an agent conversation or
|
||||
committed. If the remote is absent (`rclone listremotes` lacks
|
||||
`TestProtonDrive:`), **do not attempt this task**: leave every box
|
||||
unticked and report it as not run.
|
||||
|
||||
- [ ] `go test -v ./backend/protondrive/ -run '^TestIntegration$' -remote TestProtonDrive:`
|
||||
- [ ] `go run ./fstest/test_all -backends protondrive -remotes TestProtonDrive: -n 1`
|
||||
clean with no `ignore:` entries.
|
||||
- [ ] Capture golden fixtures from the account (nodes created by the web
|
||||
app and by Proton's official CLI) into `testdata/golden/` with a
|
||||
`-capture` test flag, and add `TestGoldenFixtures`.
|
||||
- [ ] Interop both ways with the official CLI and the web app (spec 9.7).
|
||||
- [ ] Upgrade/downgrade check against released rclone v1.76.
|
||||
- [ ] Record results in `docs/TODO.md`; only then is spec criterion 12
|
||||
(items 3, 6, 7) met.
|
||||
|
||||
---
|
||||
|
||||
## Self-review notes
|
||||
|
||||
- Spec coverage: sections 6.1 (Tasks 2-4, 14), 6.2 minus events and
|
||||
persistent cache (Task 8; those are M5/M6, next plan), 6.3 (11), 6.4
|
||||
(10), 6.5 minus Metadata (9, 14), 6.6 (8, 14), 6.7 (14), 7 (16), 9.1
|
||||
(5, 6, 12), 9.2 (1, 2, 3, 4, 7, 9, 11), 9.3 (13, 14, 18), 9.4 (15, 17),
|
||||
9.5 (17), 9.6 (16), 9.7 (21), 9.8 (18), 10 (20), 12 (19, 21).
|
||||
- Not in this plan by design: events and `ChangeNotify`, persistent cache,
|
||||
Metadata, small-file upload, the nightly live CI job, the weekly SDK
|
||||
reminder.
|
||||
@@ -0,0 +1,214 @@
|
||||
# Task 21f report: read-after-write lag after commit/create
|
||||
|
||||
## Diagnosis (verified, refined)
|
||||
|
||||
Both live failures come from the single read of the node in
|
||||
`uploadDraft` (upload.go) after the commit:
|
||||
|
||||
1. vfs `revision is active, not X`: the read returned the new file still
|
||||
as a **draft** (no active revision, so `RevisionID == ""`); the commit had
|
||||
returned success, so the code fell through to `errUploadSuperseded`.
|
||||
2. fs/operations TestPurge `object not found`: the read returned
|
||||
**not found**; `uploadDraft` returned `read the uploaded file: get node X:
|
||||
not found`, which wraps `errNotFound`, and `translateError` turns any
|
||||
wrapped `errNotFound` into the bare `fs.ErrorObjectNotFound`. That
|
||||
explains why the message had no `upload "one":` prefix. The dircache
|
||||
folder creates are not the cause with caching on (default): `createFolder`
|
||||
caches its own node, so `targetFolder` never reads it from the server.
|
||||
With `enable_caching = false` it does read it, and the same lag
|
||||
breaks `targetFolder`. Both reproduced exactly on the fake (see RED).
|
||||
|
||||
## What changed
|
||||
|
||||
- `protontest`: `Server.SetReadLag(reads int, missing bool)`. After a
|
||||
folder create, a copy or a commit, the next `reads` reads of that node
|
||||
through `POST /drive/v2/volumes/{id}/links` leave the node out (new node, or
|
||||
`missing`) or return its wire form from before the commit. Off by default.
|
||||
Listings are not lagged.
|
||||
- `nodes.go`: `client.readBack(ctx, id, shows)` reads with backoff (100 ms
|
||||
doubling to 2 s, `readBackAttempts = 8`, about 7 s), honouring ctx,
|
||||
while `shows` says the change isn't visible yet. `client.readBackDelay`
|
||||
can be lowered by tests. Used by:
|
||||
- `uploadDraft`, through `draft.lagging`: retry while not found, active
|
||||
with an empty revision, or (only when the commit returned success)
|
||||
still as before the commit (`notCommitted`: draft for a new file, base
|
||||
revision for a new revision). `errUploadSuperseded` now only means a
|
||||
non-empty revision that is neither ours nor the base.
|
||||
- `copyNode`: the read of the copy retries while not found.
|
||||
- `targetFolder`: retries while not found **only when the cache is
|
||||
disabled**. With caching on, a folder the client made is cached, so a
|
||||
miss means it is gone. A first version retried always, and that added
|
||||
7 s to `TestCreateFolder`/`TestMove` (writing into a missing folder).
|
||||
- Lag outlasting the retries after a successful commit gives the new
|
||||
`errUploadUnconfirmed`. Nothing is deleted (committed=true), and
|
||||
`translateError` maps it to `NoLowLevelRetryError` like
|
||||
`errUploadOutcomeUnknown`. Logged at DEBUG.
|
||||
- A failed commit (commitErr != nil) keeps ruling 11. Retrying is only
|
||||
added for not-found and empty-revision reads, which already gave
|
||||
"outcome unknown", so no new deletion path.
|
||||
|
||||
## Policy decision (recorded in docs/DECISIONS.md)
|
||||
|
||||
You leaned to returning success with an object built from what was
|
||||
committed. I didn't do that. The returned node is used directly by
|
||||
`Object.Open` (`openWhole`/`open` need `n.keys`, `n.link.File` content key
|
||||
packet, revision ID), and by Move/rename (link passphrase and name hash).
|
||||
The draft's node keyring is cleared when `upload` returns. A made-up node
|
||||
would therefore have to make up decrypted keys and server metadata, and it
|
||||
would be cached and acted on. So I return `errUploadUnconfirmed`: nothing
|
||||
is deleted, there is no immediate retry, and a later sync pass reads the
|
||||
file. If you prefer success, it is a contained change in the
|
||||
`commitErr == nil && d.lagging(...)` case of `uploadDraft`.
|
||||
|
||||
## TDD evidence
|
||||
|
||||
RED (`RCLONE_CONFIG=/notfound go test ./backend/protondrive/ -run 'ReadBackLag' -count=1`,
|
||||
with the tests and fake lag in place, before any client change):
|
||||
|
||||
```
|
||||
--- FAIL: TestUploadIntoNewFolderReadBackLag
|
||||
create folder "B1": get node 5d3b...: not found
|
||||
--- FAIL: TestCopyReadBackLag
|
||||
copy "f.txt" to "copy.txt" in 0bee...: get node 4f12...: not found
|
||||
--- FAIL: TestUploadReadBackLag/NewFileStale
|
||||
upload "f": upload committed, but the file now has another revision (revision is active, not 1e97...)
|
||||
--- FAIL: TestUploadReadBackLag/NewRevisionMissing
|
||||
upload "f": read the uploaded file: get node 1af9...: not found
|
||||
--- FAIL: TestUploadReadBackLag/NewFileMissing
|
||||
upload "f": read the uploaded file: get node 3420...: not found
|
||||
--- FAIL: TestUploadReadBackLag/NewRevisionStale
|
||||
upload "f": upload committed, but the file now has another revision (revision d623... is active, not 4e7b...)
|
||||
```
|
||||
`TestPutNestedReadBackLag` (Fs.Put of `A1/B1/C1/one`, caching on): `object not found`.
|
||||
NewFileStale is symptom 1 byte for byte, and TestPutNestedReadBackLag is symptom 2.
|
||||
|
||||
GREEN: same command → `ok github.com/rclone/rclone/backend/protondrive 1.757s`.
|
||||
I wrote `TestUploadReadBackLagOutlasted` (both stale and missing; asserts
|
||||
`errUploadUnconfirmed`, not superseded, exactly 8 link reads, no deletes,
|
||||
content on server) and the `errUploadUnconfirmed` row of
|
||||
`TestTranslateError` after the fix. I didn't watch those fail first.
|
||||
|
||||
## Verification
|
||||
|
||||
- `go build ./...`: ok
|
||||
- `go vet ./backend/protondrive/...`: ok
|
||||
- `golangci-lint run ./backend/protondrive/...`: 0 issues
|
||||
- `RCLONE_CONFIG=/notfound go test -race -count=1 ./backend/protondrive/...`: all ok
|
||||
(protondrive, protontest, protonfake); budget tests unchanged (lag is off
|
||||
by default, so no extra reads).
|
||||
|
||||
## Files
|
||||
|
||||
backend/protondrive/{nodes.go,upload.go,protondrive.go,upload_internal_test.go,
|
||||
nodes_internal_test.go,protondrive_internal_test.go},
|
||||
backend/protondrive/protontest/{server.go,drive.go,mutate.go,upload.go},
|
||||
docs/TODO.md, docs/DECISIONS.md.
|
||||
|
||||
## Concerns / not fixed (logged in docs/TODO.md)
|
||||
|
||||
- The real lag's length is unmeasured. 7 s is a guess, and a longer lag
|
||||
still fails the upload, now with a clear "committed" error.
|
||||
- `activeNode` (Update/Move/Copy/DirMove of an object just written) has no
|
||||
lag tolerance. It is the most likely next live symptom ("object not
|
||||
found" while the node is missing or still a draft). I left it alone
|
||||
because tolerating lag there would delay every genuine not-found by 7 s.
|
||||
- Move's read of the moved node may be stale (it can't be not found).
|
||||
- After a commit that failed, a stale read showing the draft or the old
|
||||
revision still counts as proof and the draft is deleted. That is the
|
||||
pre-existing ruling 11 risk, now known to be real under lag. Retrying
|
||||
there would slow every refused commit by 7 s.
|
||||
- The retry loop uses the upload's cancel-detached read ctx (30 s cap),
|
||||
so cancelling during lag after a successful commit still waits up to
|
||||
about 7 s.
|
||||
- Listing lag is neither modelled nor handled.
|
||||
|
||||
## Fix round 1
|
||||
|
||||
### Critical: deletion after an ambiguous commit failure (coordinator ruling)
|
||||
|
||||
- `uploadDraft` (upload.go) now sends the commit through
|
||||
`c.pacedCall(ctx, c.apiPacer, ...)` and counts attempts. A draft is
|
||||
deleted only when the commit failed on its **single** attempt with
|
||||
`refusedCommit`: an `*api.Error` with a 4xx status, not 408, and a
|
||||
Proton code. In that case the file is not read back at all.
|
||||
`session.call`'s own resend after a 401 refresh doesn't count, because
|
||||
a 401 attempt was never processed. 429 is retried by the pacer, so it
|
||||
ends up as "unknown".
|
||||
- Every other failure (5xx, network, timeout, cancelled ctx, resent
|
||||
attempts, a 4xx without a code) gives `errUploadOutcomeUnknown` and
|
||||
deletes nothing. The read after such a failure can only turn it into a
|
||||
success, when it shows our revision. `draft.notCommitted` is gone, and
|
||||
its states are now among those `lagging` keeps retrying through.
|
||||
- After a failed commit, the read goes through lag while the caller's ctx
|
||||
is live. If ctx is already done it reads once, detached, as before; this
|
||||
keeps `TestUploadCancelDuringCommit`'s "active commit is a success".
|
||||
|
||||
### Minors
|
||||
|
||||
- Empty-revision lag state: `SetReadLag(reads, view LagView)` now takes
|
||||
`LagBefore`, `LagMissing` or `LagNoRevision`. `LagNoRevision` serves the
|
||||
file as it is but with `ActiveRevision` nil. Tests NewFileNoRevision,
|
||||
NewRevisionNoRevision and the Outlasted `view=2` case cover it.
|
||||
- Ctrl-C: on the success path the read-back uses the caller's ctx, not a
|
||||
detached one. A done ctx stops the waits and returns
|
||||
`errUploadUnconfirmed`, with nothing deleted. Note that a ctx cancelled
|
||||
*while the commit response is in flight* still makes the pacer report
|
||||
the commit as `context.Canceled`, so that is the "unknown" path, which
|
||||
is also non-deleting.
|
||||
- `--retries` duplicate revision: added to docs/TODO.md. Orphan drafts
|
||||
after ambiguous failures, and CleanUp / server expiry: also in TODO.
|
||||
- DECISIONS: a new entry supersedes the read-back half of ruling 11.
|
||||
Handoff section 7 ruling 11 is rewritten.
|
||||
|
||||
### Changed existing tests (expectations follow the new rule)
|
||||
|
||||
- `TestUploadCancel/commit`: a commit cancelled in flight now leaves its
|
||||
draft (1 draft, no file). It used to expect the draft deleted.
|
||||
- `TestUploadCommitOutcomeUnknown`: the commit fault is now a 422 without
|
||||
a Proton code (ambiguous). A coded 422 on one attempt is now an explicit
|
||||
refusal and deletes, which `TestUploadCommitRefused` covers.
|
||||
|
||||
### TDD evidence
|
||||
|
||||
RED, on top of 1b2fcbd21 with the new tests and the fake change only:
|
||||
```
|
||||
--- FAIL: TestUploadAmbiguousCommitFailureUnderLag/ResentThenRefused/existing=false draft deletion sent
|
||||
--- FAIL: .../ResentThenRefused/existing=true revision deletion sent
|
||||
--- FAIL: .../ConnectionLost/existing=false draft deletion sent
|
||||
--- FAIL: .../ConnectionLost/existing=true revision deletion sent
|
||||
(fake log: "only drafts can be deleted" / "Draft revision not found":
|
||||
the deletes hit the committed file / active revision)
|
||||
--- FAIL: TestUploadCommittedThenCancelledUnderLag not errUploadUnconfirmed
|
||||
--- FAIL: TestUploadCommitRefusedNoReadBack/existing={false,true} file read back
|
||||
```
|
||||
`ResentThenRefused` is the reviewer's scenario: a 502 after the real
|
||||
handler commits, then the resent commit gets a 422 injected after the real
|
||||
handler ran, under `SetReadLag(LagBefore)`. `ConnectionLost` is the same
|
||||
but with a single attempt that ends in a dropped connection.
|
||||
|
||||
GREEN: `go test ./backend/protondrive/ -run 'Upload|Lag|Commit|Draft'` → ok.
|
||||
|
||||
Mutation checks, for the tests never seen failing:
|
||||
- Removing the `errUploadUnconfirmed` case fails TestUploadReadBackLagOutlasted (all 3 views).
|
||||
- Removing the empty-revision case of `lagging` fails NewFileNoRevision,
|
||||
NewRevisionNoRevision and Outlasted/view=2.
|
||||
- Removing `errUploadUnconfirmed` from `translateError` fails its TestTranslateError row.
|
||||
All mutations were reverted afterwards.
|
||||
|
||||
### Verification (fix round)
|
||||
|
||||
`go build ./...` ok; `go vet ./backend/protondrive/...` ok;
|
||||
`golangci-lint run ./backend/protondrive/...` 0 issues;
|
||||
`RCLONE_CONFIG=/notfound go test -race -count=1 ./backend/protondrive/...` — see below.
|
||||
Race suite result: `ok backend/protondrive 315.0s`, `ok protontest 36.4s`, `ok protonfake 1.4s`.
|
||||
(After it ran I changed only a comment, re-wrapping the `upload` godoc; build and lint were re-run on it.)
|
||||
|
||||
### Concerns (fix round)
|
||||
|
||||
- The real server's answer to a resent commit is unknown, which doesn't
|
||||
matter now because a resend is always "unknown". The real lag length
|
||||
is still unmeasured.
|
||||
- Ambiguous failures now leave orphan drafts. They are logged in TODO
|
||||
(CleanUp / server expiry).
|
||||
- A server that commits and then answers a coded 4xx on a single attempt
|
||||
would still get a delete. The rule takes that answer at face value.
|
||||
@@ -0,0 +1,117 @@
|
||||
# Task 21f re-review, fix round 1 (1b2fcbd21..be0053607)
|
||||
|
||||
### Verdict: ALL CLOSED (Critical 1 and Minors 1-4). No new Critical or Important.
|
||||
|
||||
### Critical 1: closed
|
||||
|
||||
- upload.go:403-406: the only `committed=false` return after the commit is
|
||||
sent is `commitErr != nil && attempts == 1 && refusedCommit(commitErr)`.
|
||||
Every later return (upload.go:418-445) gives `committed=true`, so no read
|
||||
result can lead to `deleteDraft`. `deleteDraft` is called only from
|
||||
`upload` when `!committed` (upload.go:126-130). The earlier
|
||||
`committed=false` returns (verificationData, uploadBlocks, check, xattrs)
|
||||
all happen before the commit is sent.
|
||||
- New file and update both go through the same `uploadDraft`, and
|
||||
`draft.newFile` only picks the delete call. PutStream and Update share
|
||||
`o.fs.c.upload` (protondrive.go:1236, the only caller).
|
||||
- Attempt counter: `attempts++` sits inside the paced fn, which the pacer
|
||||
calls once per resend, one after another, so there is no race. Network
|
||||
errors are resent through `fserrors.ShouldRetry` (nodes.go:278) and so
|
||||
count. When they aren't resent they are not `*api.Error`, so they give
|
||||
"unknown". `session.call`'s inner resend after a 401 (session.go:204-232)
|
||||
counts as one attempt, which is right: a 401 was not processed. If the
|
||||
refresh fails, the error may wrap a coded 4xx from the refresh endpoint
|
||||
and then delete. That is harmless, because the commit itself got a 401.
|
||||
Go's net/http does not silently replay a PUT (`isReplayable` covers only
|
||||
GET/HEAD/OPTIONS/TRACE), so there is no hidden resend below the counter.
|
||||
- `*api.Error` is built only from real responses (session.go:166, 267). A
|
||||
200 response with an error Code gives Status 200, which means "unknown".
|
||||
A proxy's 4xx without a Proton code also means "unknown". Both fail in
|
||||
the safe direction.
|
||||
- Cancellation at every stage: a ctx that is done makes `shouldRetry`
|
||||
return the ctx error (`fserrors.ContextError`, nodes.go:259). That is not
|
||||
`*api.Error`, so the result is "unknown", even if the server's answer was
|
||||
a refusal. If ctx ends during the read-back, `err != nil` gives
|
||||
"unknown" on the failed path, and on the success path
|
||||
`ctx.Err() != nil` gives `errUploadUnconfirmed`. Neither deletes.
|
||||
- 408: excluded correctly. `shouldRetry` resends it anyway. The exclusion
|
||||
only matters with `--low-level-retries 1`, and a 408 from a proxy may
|
||||
arrive after the request was forwarded.
|
||||
- 429: normally resent, which makes the result "unknown". On a single
|
||||
attempt (`--low-level-retries 1`), a coded 429 counts as a refusal. That
|
||||
is sensible, because rate limiting rejects a request before it is
|
||||
processed.
|
||||
- 409: a coded 409 on a single attempt means this request did not apply,
|
||||
for example because the base revision changed. Deleting our own draft
|
||||
revision is then correct. A commit-already-applied 409 can only follow
|
||||
an earlier attempt, and that path is "unknown".
|
||||
- Outer retry layers: `errUploadOutcomeUnknown` and `errUploadUnconfirmed`
|
||||
map to `NoLowLevelRetryError` (protondrive.go:366-370), so
|
||||
`operations.Copy` does not rerun the upload at once. `removeFailedPartialCopy`
|
||||
and `removeFailedCopy` are not reached on a failed Put, because protondrive
|
||||
sets no `PartialUploads` (so `inplace`; grep found no `PartialUploads` in
|
||||
backend/protondrive). See Minor N1 for the whole-sync `--retries` pass.
|
||||
- Mutation check (scratch worktree at be0053607, since removed): dropping
|
||||
`attempts == 1` makes TestUploadAmbiguousCommitFailureUnderLag fail
|
||||
("draft deletion sent" / "revision deletion sent"). The resend case is
|
||||
pinned.
|
||||
|
||||
### Changed existing tests: changed for the right reason
|
||||
|
||||
- TestUploadCancel/commit (upload_internal_test.go:375-407): a commit
|
||||
cancelled in flight is ambiguous under the ruling, so 1 draft left is
|
||||
the correct expectation. It still asserts DeadlineExceeded, the time
|
||||
bound, no file and no leaks.
|
||||
- TestUploadCommitOutcomeUnknown (:431-445): its intent (the read fails,
|
||||
so "unknown" and no deletes) is kept by making the commit fault uncoded.
|
||||
Under the new rule a coded 422 on one attempt is a refusal, and that is
|
||||
now covered by TestUploadCommitRefused and TestUploadCommitRefusedNoReadBack
|
||||
(which also asserts that no links read and exactly one delete happen).
|
||||
|
||||
### Minors from the original review
|
||||
|
||||
1. Empty-revision lag state: closed. `LagNoRevision` (protontest/drive.go:78-79,
|
||||
101-103), with the NoRevision rows and Outlasted view=2. A mutation was
|
||||
reported.
|
||||
2. Ctrl-C after a successful commit: closed. readCtx is derived from ctx
|
||||
(upload.go:409), and TestUploadCommittedThenCancelledUnderLag checks it
|
||||
in under 2 s.
|
||||
3. `--retries` duplicate revision: closed (docs/TODO.md entry).
|
||||
4. Tests never seen failing: closed by the reported mutations.
|
||||
|
||||
### New findings
|
||||
|
||||
#### Critical
|
||||
None.
|
||||
|
||||
#### Important
|
||||
None.
|
||||
|
||||
#### Minor
|
||||
- N1. backend/protondrive/upload.go:286-303 (`resolveConflict`, existing
|
||||
code): after `errUploadOutcomeUnknown`, a whole-sync `--retries` pass
|
||||
uploads the name again. If the create-draft conflict details say "our
|
||||
own draft" (`ConflictRevisionID == ""`), the client sends `delete_multiple`
|
||||
for that link. If the earlier commit did land, this is a deletion
|
||||
decided by server state that may lag. It is unlikely: the conflict
|
||||
details come from the write path, the failed-commit read-back has
|
||||
normally already waited through the lag window (about 7 s), and it needs
|
||||
the unconfirmed real `delete_multiple`-on-active behaviour
|
||||
(docs/TODO.md:145). It mirrors the SDK. Suggest adding a sentence to
|
||||
that TODO item, saying that a stale conflict report after an ambiguous
|
||||
commit would reach it.
|
||||
- N2. upload.go:100-108: the `upload` godoc says "a 4xx API error". It
|
||||
leaves out "with a Proton code, not 408" (`refusedCommit`'s doc is
|
||||
exact), and it has a ragged wrap ("errUploadSuperseded,\n// and one").
|
||||
- N3. No test pins the 408 exclusion or the `Code != 0` requirement on a
|
||||
single attempt. TestUploadCommitOutcomeUnknown covers the uncoded 422,
|
||||
so only 408 is unpinned.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Task quality:** Approved
|
||||
|
||||
**Reasoning:** A draft is now deleted only on a coded 4xx (not 408) from a
|
||||
single, un-resent commit attempt. Every other path returns committed=true,
|
||||
and the mutation check confirms the resend guard is tested. The changed
|
||||
tests follow the ruling and are no weaker.
|
||||
@@ -0,0 +1,135 @@
|
||||
# Task 21f review (a2dc6f71f..af6fdd789)
|
||||
|
||||
### Spec Compliance
|
||||
|
||||
- ❌ Issues found: the requirement "never delete anything when the commit
|
||||
outcome is unknown (ruling 11)" is not met under the lag this task
|
||||
establishes as real. After a commit that failed, one stale read showing the
|
||||
draft (new file) or the base revision (update) still counts as proof, and
|
||||
the draft is deleted (backend/protondrive/upload.go:401-403 stops retrying
|
||||
on such a read because `lagging(..., committed=false)` is false for it, and
|
||||
then upload.go:425 returns committed=false, which makes `upload` call
|
||||
`deleteDraft`). The implementer knows this and logged it in docs/TODO.md:165
|
||||
instead of fixing it. See Critical 1. Everything else asked for is there:
|
||||
retrying while the node is missing or has no active revision,
|
||||
errUploadSuperseded kept for a genuine other revision, TDD with a lag fault
|
||||
in the fake.
|
||||
- ⚠️ Cannot verify from the diff: what the real server does with
|
||||
`delete_multiple` on a link that is now active, and with `DELETE .../revisions/{id}`
|
||||
on a revision that is now active (the existing docs/TODO.md item says this
|
||||
is unconfirmed). Critical 1 is a data-loss bug if either call succeeds.
|
||||
|
||||
### Strengths
|
||||
|
||||
- The diagnosis is right and the RED evidence reproduces both live messages
|
||||
byte for byte (empty revision gives superseded, and not-found becomes a bare
|
||||
`fs.ErrorObjectNotFound` through `translateError`).
|
||||
- `readBack` (nodes.go:733) is small: it is bounded (8 reads, 100 ms doubling,
|
||||
capped at 2 s, about 7.1 s), honours ctx, and calls `fetchNode` once on the
|
||||
normal path, so the request budgets don't change. `TestUploadReadBackLagOutlasted`
|
||||
pins the bound at exactly `readBackAttempts` link reads and no deletes.
|
||||
- The success path cannot delete anything. Every non-`ours` branch with
|
||||
`commitErr == nil` returns committed=true (upload.go:418-423). A stale
|
||||
node is never left in the cache: on the `ours` path `cache.add` replaces it,
|
||||
and every other path calls `forget`.
|
||||
- The `targetFolder` retry is limited to `c.cache == nil` (nodes.go:967), so
|
||||
with caching on (the default) a genuine not-found costs nothing extra.
|
||||
Choosing `errUploadUnconfirmed` over an invented node is sound and recorded
|
||||
in DECISIONS.
|
||||
|
||||
### Issues
|
||||
|
||||
#### Critical (Must Fix)
|
||||
|
||||
1. **backend/protondrive/upload.go:401-403, 425 (with upload.go:126-130 and
|
||||
deleteDraft ~upload.go:927): a failed commit plus one stale read sends a
|
||||
delete against a file that is committed.** Checked in a scratch worktree at
|
||||
af6fdd789 (since removed) with a commit fault `{Status: 422, AfterHandler: true}`
|
||||
(the commit lands, the client sees an error) and `SetReadLag(1, false)`:
|
||||
- New file: the read shows the pre-commit draft, `notCommitted` is true, and
|
||||
`deleteDraft` sends `POST delete_multiple` for the link ID, which is now
|
||||
the **active file**. The fake refuses ("only drafts can be deleted"). The
|
||||
real server's behaviour is unconfirmed (docs/TODO.md), and the SDK sends
|
||||
`delete_multiple` for active links in its conflict handler. If Proton
|
||||
honours it, the whole file is **permanently deleted**, not trashed.
|
||||
- Update: the read shows the base revision, and the client sends
|
||||
`DELETE .../revisions/{ourRevision}`, the revision that is now
|
||||
**active**. The fake answers 2501. On the real server this could
|
||||
remove the active revision, leaving the file without content or rolled
|
||||
back.
|
||||
This path is likely to occur, not just possible. `shouldRetry` resends the
|
||||
commit PUT on 5xx/408/network errors (nodes.go:272-273). So "commit landed,
|
||||
502 or timeout, resent commit refused with 4xx", or simply a cancelled ctx
|
||||
during the commit, followed by a read in the lag window, reaches the
|
||||
deletion. In both cases the error returned is the plain commit error, which
|
||||
rclone retries at low level. On a new file the source still has the data,
|
||||
but a retry can race the deletion, and the update case has no such fallback.
|
||||
**Safe rule (recommended):** decide by the commit error, not by a read.
|
||||
- Delete only when the server definitively refused the commit on a single,
|
||||
un-retried attempt: an `*api.Error` with a 4xx status and a code. A
|
||||
refusal means it did not commit whatever any read shows, so no read and
|
||||
no 7 s wait are needed. This answers the report's worry that "retrying
|
||||
there would slow every refused commit by 7 s".
|
||||
- For any other failure (transport error, timeout, 5xx/408, a refusal after
|
||||
the pacer resent the PUT, a done ctx): never delete. Return
|
||||
errUploadOutcomeUnknown and leave the draft, which the next upload of the
|
||||
name replaces (the existing comment at upload.go ~429). Tracking "was
|
||||
retried" needs a flag set in the paced fn, or calling `c.s.call` for the
|
||||
commit with your own single retry.
|
||||
- If a read-based proof is still wanted for unknown failures, it has to be
|
||||
`readBackAttempts` consecutive reads that all show the draft or base
|
||||
revision over the whole window, and even that is a heuristic because the
|
||||
lag length is unmeasured (docs/TODO.md). The rule above is strictly safer.
|
||||
Add a test like the scratch one (`AfterHandler` 422 or 502 on the commit
|
||||
plus `SetReadLag(n, false)`, assert `requireNoDeletes`). It fails on
|
||||
af6fdd789 today.
|
||||
|
||||
#### Important (Should Fix)
|
||||
|
||||
None beyond Critical 1.
|
||||
|
||||
#### Minor (Nice to Have)
|
||||
|
||||
1. **upload.go:446**: the "active with an empty revision" lag state is not
|
||||
tested. The fake's stale view of a new file is the draft
|
||||
(protontest/drive.go:1710, `before` captured at commit), so deleting that
|
||||
`case` leaves every test passing. Either model it (a `SetReadLag` mode
|
||||
returning the link active with no active revision, which may be what the
|
||||
live "revision is active" was) or say it is speculative.
|
||||
2. **upload.go:399-403**: the success-path retry runs on the cancel-detached
|
||||
`readCtx`, so Ctrl-C after a successful commit can block about 7 s more. This
|
||||
is acceptable and logged. If wanted, return `errUploadUnconfirmed` as soon
|
||||
as the caller's ctx is done and commitErr == nil (nothing is deleted on that
|
||||
path anyway).
|
||||
3. **errUploadUnconfirmed / NoLowLevelRetryError (protondrive.go:366)**:
|
||||
`operations.Copy` won't retry immediately, which is correct. A whole-sync
|
||||
`--retries` pass relists, and since listing lag isn't modelled
|
||||
(docs/TODO.md) a stale listing re-uploads, creating an extra revision of
|
||||
identical content. That wastes work but loses no data. The DECISIONS
|
||||
entry could mention it.
|
||||
4. **TestUploadReadBackLagOutlasted (upload_internal_test.go:822)** and the
|
||||
`errUploadUnconfirmed` row of TestTranslateError were written after the fix
|
||||
and never seen failing (the report says so). By reasoning they do fail
|
||||
without it: before the fix the stale case gives superseded and the missing
|
||||
case gives a wrapped errNotFound.
|
||||
5. With caching off, a commit that failed and whose node is genuinely gone
|
||||
(deleted elsewhere mid-upload) now takes about 7 s to give outcome-unknown.
|
||||
This is rare and harmless.
|
||||
|
||||
Checks run: one scratch-worktree test (described in Critical 1), reasoning
|
||||
over `shouldRetry` (nodes.go:256-280), `deleteDraft`/`deleteDraftLink`
|
||||
(upload.go ~927-955), `newNodeCache(false) == nil` (cache.go:29), and the
|
||||
partial-upload removal in fs/operations/copy.go (not reached:
|
||||
protondrive doesn't set `PartialUploads`). Budget and normal path: readBack
|
||||
makes a single `fetchNode` when the first read shows the change, so it adds
|
||||
no request or sleep.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Task quality:** Needs fixes
|
||||
|
||||
**Reasoning:** The lag handling on the success path is correct, bounded,
|
||||
costs nothing on the normal path and is well tested. But the failed-commit
|
||||
path still treats one read, now known to be possibly stale, as proof, and
|
||||
sends a delete against a file that may be committed. That breaks ruling 11,
|
||||
which the task told it to keep.
|
||||
@@ -0,0 +1,73 @@
|
||||
### Task 21g: Golden fixtures from real Proton clients, re-keyed (owner chose option 2)
|
||||
|
||||
Part of plan Task 21 ("Capture golden fixtures from the account (nodes
|
||||
created by the web app and by Proton's official CLI) into
|
||||
`testdata/golden/` with a `-capture` test flag, and add
|
||||
`TestGoldenFixtures`"), with the owner's decision of 2026-11-12:
|
||||
|
||||
> Re-encrypt the captured files under keys generated only for the tests,
|
||||
> keeping the real format but carrying no secret from the account.
|
||||
|
||||
**Files (expected; adjust if the code says otherwise):**
|
||||
|
||||
- Create: `backend/protondrive/golden_internal_test.go` (capture + test)
|
||||
- Create: `backend/protondrive/testdata/golden/` (fixtures, written only by
|
||||
the controller's live capture run; the implementer commits the code and,
|
||||
for its own testing, fixtures captured from the FAKE into a temp dir -
|
||||
never commit fake-captured fixtures as golden ones)
|
||||
- Modify: `docs/TODO.md`, `docs/DECISIONS.md` (one entry: what is preserved
|
||||
and what is replaced)
|
||||
|
||||
**What the capture does** (`go test ./backend/protondrive/ -run TestGoldenCapture -capture DIR -remote NAME:PATH`):
|
||||
|
||||
1. Signs in through the normal backend code path (the remote's saved
|
||||
session), walks `PATH` (a folder made by another client), and for each
|
||||
folder and file node fetches what the client fetches: link/node JSON,
|
||||
node key and passphrase, name, hash key, xattrs, revision, block list
|
||||
and blocks (cap: 3 files, 16 MiB total per file; skip anything bigger).
|
||||
2. In memory, re-keys everything onto freshly generated test keys (a test
|
||||
user key, test address key, and new node keys of the same algorithm and
|
||||
packet version as the originals):
|
||||
- Keep every SEIPD/AEAD data packet byte-for-byte (block contents,
|
||||
encrypted names, xattrs, hash keys): only re-wrap their session keys
|
||||
(PKESK packets) to the corresponding test key.
|
||||
- Signatures made by the account's address key must be re-made by the
|
||||
test address key (same signature version, hash and algorithm, same
|
||||
signed data). Record in the fixture which signatures were re-made.
|
||||
- Manifest signature re-made over the unchanged block hashes.
|
||||
3. Sanitises: replace every ID (volume, share, link, revision, block,
|
||||
address, key) with stable placeholders, every email with
|
||||
`golden@example.invalid`, drop block URLs and tokens. Keep timestamps,
|
||||
sizes, flags, MIME types, block sizes and whether block sizes are
|
||||
recorded.
|
||||
4. Writes to `DIR`: one JSON fixture per node set plus the test keys'
|
||||
ARMORED PRIVATE keys and their passphrase (these are test-only keys, so
|
||||
committing them is fine). Never writes the account's keys, passphrases,
|
||||
salted key pass, session tokens or any plaintext beyond what the test
|
||||
needs (file contents are random test data chosen by the controller).
|
||||
5. Prints a summary listing each node, its original creator client (from
|
||||
the node's signature email domain is NOT enough - take it from a
|
||||
`-capture-label` flag), sizes, and which packets were re-wrapped or
|
||||
re-signed.
|
||||
|
||||
**`TestGoldenFixtures`:** loads every fixture in `testdata/golden/` into the
|
||||
in-process fake (`protontest`) as if served by Proton, then reads every
|
||||
file through the real backend (`Fs`/`Object`, not internals) with
|
||||
`ignore_signature_errors` false: full read, a ranged read starting in a
|
||||
later block, listing, names decrypted, sizes and SHA1 match the fixture's
|
||||
expected values. Skips (not fails) when the directory has no fixtures.
|
||||
|
||||
**Constraints:**
|
||||
|
||||
- The implementer NEVER talks to Proton. Develop and test capture against
|
||||
the fake: seed the fake with files written in other shapes (e.g.
|
||||
`SeedFileInBlocks`, unrecorded block sizes), capture into a temp dir,
|
||||
then load with `TestGoldenFixtures`' loader and read back. A test must
|
||||
prove the round trip (capture from fake -> fixture -> load -> read
|
||||
identical bytes) and that the output contains none of the original
|
||||
account's key material, IDs or emails (search the written files for the
|
||||
fake account's email, IDs, armored key blocks and salted key pass).
|
||||
- `-capture` without `-remote` must fail clearly; without `-capture` the
|
||||
capture test skips.
|
||||
- No new dependencies. Comment style per AGENTS.md.
|
||||
- Commit: `protondrive: test against re-keyed files written by Proton's own clients`.
|
||||
@@ -0,0 +1,307 @@
|
||||
# Task 21g report: golden fixtures from real Proton clients, re-keyed
|
||||
|
||||
## What I implemented
|
||||
|
||||
- `backend/protondrive/golden_internal_test.go` (new):
|
||||
- `TestGoldenCapture`: flags `-capture DIR`, `-capture-label NAME` and the
|
||||
existing `-remote NAME:PATH`. Skips without `-capture`; fails clearly if
|
||||
`-capture` is given without `-remote` or without `-capture-label` (or with
|
||||
a label that is not a plain name). Signs in through `fs.NewFs` (the
|
||||
remote's saved session), captures into `DIR/LABEL/` (refuses if it
|
||||
exists), then immediately loads that output into the fake and reads it
|
||||
back as `TestGoldenFixtures` does, so a bad capture fails at once.
|
||||
- `captureGolden`: walks the folder at the Fs root depth first in name
|
||||
order, all subfolders, at most 3 files (`goldenMaxFiles`), skipping files
|
||||
over 16 MiB encrypted (logged). Refuses to run with
|
||||
`ignore_signature_errors` on; refuses any node whose signatures don't
|
||||
verify with the account's keys (node key unverified or any problem the
|
||||
backend reported for that node); reads each file whole through
|
||||
`c.open` (which checks content key, block hashes and manifest), then
|
||||
downloads the raw blocks again through `c.revisionBlocks` /
|
||||
`c.downloadBlock`. Refuses files with thumbnails.
|
||||
- `rekeyer`: the re-keying (details below). Reuses `pgpFor`,
|
||||
`armorMessage`, `decryptContentKey`, `decryptBlock`, `generateHashKey`,
|
||||
`generatePassphrase`, `nameHash`; new crypto only where the backend has
|
||||
nothing (re-making signatures with chosen parameters, re-encrypting an
|
||||
SEIPD packet with a given session key).
|
||||
- `TestGoldenFixtures`: for each `testdata/golden/*/nodes.json`, seeds a
|
||||
fake user holding the fixture's test keys (`AddUserWithKeys`), stores
|
||||
every node as Proton served it (`SeedStoredFolder` / `SeedStoredFile`),
|
||||
and through `Fs`/`Object` only, with `ignore_signature_errors=false`:
|
||||
lists every folder (and the root) and compares names, then per file
|
||||
checks size, modification time, media type, SHA-1 (`Object.Hash`), a
|
||||
full read against the expected SHA-1, and a ranged read starting in the
|
||||
second block (or the middle of a one-block file) against the full read.
|
||||
Fails if the backend reported any problem (`c.reported`) or skipped any
|
||||
node. Skips when there are no fixtures.
|
||||
- `TestGoldenCaptureFlags`: the flag rules.
|
||||
- `TestGoldenRoundTrip`: seeds the fake (user with the known keys of the
|
||||
frozen `testdata/oldlib/file/legacy` fixture) with a captured folder and
|
||||
a subfolder holding: a 5 MiB+7 SDK-style file (v6 PKESK, SEIPDv2, two
|
||||
blocks, sizes recorded), a file in 1 MiB blocks with no recorded block
|
||||
sizes (`SeedFileInBlocks`), a legacy file (v3 PKESK, SEIPDv1, legacy
|
||||
time format), an anonymous file (signed by the parent folder's node key,
|
||||
no signature emails), and a fifth file beyond the cap (capture run with
|
||||
a cap of 4). Captures into `t.TempDir()`, loads, reads everything back
|
||||
through the backend and checks the bytes equal the seeded content, the
|
||||
blocks are byte-identical to what the fake stored, block-size shapes and
|
||||
anonymity survive, and the anonymous file's name signature is recorded
|
||||
as re-made by its parent's new node key. Then searches every written
|
||||
file (raw and JSON-escaped) for: the source email, the source address
|
||||
key and root key (armored) and passphrases, the root hash key, volume,
|
||||
share, root and every link ID, the salted key password (raw, base64,
|
||||
hex), every session and storage token the fake issued, every original
|
||||
armored node key, encrypted passphrase and passphrase signature, and
|
||||
every original node passphrase and content session key (raw, base64,
|
||||
hex).
|
||||
- `backend/protondrive/protontest/stored.go`: `StoredFolder` and
|
||||
`SeedStoredFolder`; `CreateTime`/`ModifyTime` on `StoredLink` and
|
||||
`CreateTime` on `StoredRevision` (zero means the time of seeding, so the
|
||||
existing frozen fixtures load unchanged); `StoredFile()` fills them.
|
||||
- `protontest/drive.go`: `insertLink` keeps a given modification time.
|
||||
- `docs/DECISIONS.md`: one entry, what is preserved, replaced and dropped.
|
||||
- `docs/TODO.md`: the pending live capture under "Now"; the thumbnail and
|
||||
key-algorithm limits of the capture under "Next".
|
||||
|
||||
Plan checkbox for the capture (Task 21) is NOT ticked: it needs the live
|
||||
capture and the committed fixtures, which are yours.
|
||||
|
||||
## The re-keying, per packet, and where the brief's plan could not be followed as written
|
||||
|
||||
Keys: a test address key (same algorithm, version, creation time and AEAD
|
||||
feature as the main share address's first key) replaces every key of every
|
||||
address of the account; a test root key replaces the captured folder's
|
||||
parent's node key (its node key, passphrase and a new hash key go in
|
||||
`keys.json` as `protontest.FixedKeys`); each node gets a new node key "like"
|
||||
its original (`generateKeyLike`: same public-key algorithm, RSA bit length,
|
||||
key version, creation time, SEIPDv2 feature; user ID copied with account
|
||||
emails replaced). Supported: RSA, EdDSA (legacy), Ed25519, Ed448; anything
|
||||
else fails the capture clearly.
|
||||
|
||||
| Field | What happens |
|
||||
|---|---|
|
||||
| blocks | byte for byte; hashes unchanged |
|
||||
| block signatures (`EncSignature`) | **dropped**: the backend never reads them (api.Block has no such field). Recorded per file. |
|
||||
| `ContentKeyPacket` | same session key re-wrapped to the new node key, same PKESK version (v3/v6) |
|
||||
| `ContentKeyPacketSignature`, `NodePassphraseSignature`, `ManifestSignature` (detached) | re-made by the key replacing the issuer, same signature type, hash, creation time, lifetime, notations; manifest over the unchanged block hashes |
|
||||
| `Name`, `NodeHashKey`, `XAttr` (file and folder) | **deviation**: the brief says keep their SEIPD packets byte for byte AND re-make the address/node-key signatures. That is impossible: those signatures are embedded inside the encrypted packet. Closest faithful option taken: decrypt the SEIPD packet, keep every inner packet (literal data packet verbatim: format byte, file name, date, content; compression algorithm), re-make only the one-pass-signature and signature packets, and encrypt the inner stream again with the SAME session key, cipher, SEIPD version, AEAD mode and chunk size. A message holding no signature keeps its SEIPD packet byte for byte (only the PKESK is re-wrapped). Each field records which was done. |
|
||||
| `NodePassphrase` | **deviation**: the passphrase is a secret of the account, so a new passphrase locks the new node key; it is encrypted in the original message's structure (same inner packets, new session key of the same algorithm and version). Its detached signature is re-made over the new passphrase. |
|
||||
| `NameHash` | kept for every node but the captured folder, whose parent changed; that one is recomputed with the test root hash key |
|
||||
| intended recipients in signatures | mapped to the replacement key's fingerprint; an unknown one is dropped and noted |
|
||||
| signer user ID subpacket | replaced by `golden@example.invalid` |
|
||||
| signature algorithm/version | follows the replacement key, which matches the original's; a mismatch would be noted |
|
||||
| emails | account emails -> `golden@example.invalid`; empty stays empty; an email not of the account fails the capture (its keys can't be replaced) |
|
||||
| IDs | placeholders `link-N`; volume/share/revision/block/address/key IDs, URLs and tokens are not written at all |
|
||||
| timestamps, sizes, media types, block sizes, whether sizes are recorded | kept (link CreateTime/ModifyTime and revision CreateTime are now served by the fake from the fixture) |
|
||||
| user key | **not written**: `AddUserWithKeys` seals the address key passphrase to a user key the fake generates, so a fixture user key would be unused |
|
||||
| thumbnails | a file with thumbnails fails the capture: the fake serves none and the manifest covers them |
|
||||
|
||||
Packet headers are re-serialised with fixed lengths (not partial lengths).
|
||||
Signature issuer subpackets are as go-crypto writes them (issuer key ID and
|
||||
issuer fingerprint).
|
||||
|
||||
## Fixture layout
|
||||
|
||||
`DIR/LABEL/keys.json` (`goldenKeys`: Comment, Email, FixedKeys - armored
|
||||
private test keys and their passphrases), `DIR/LABEL/nodes.json`
|
||||
(`goldenFixture`: Label, Nodes, each with placeholder ID/ParentID, the stored
|
||||
folder or file as `protontest.StoredFolder`/`StoredFile`, `Want` {Name,
|
||||
ModTime, MIMEType, Size, SHA1, BlockSizes (plaintext, measured),
|
||||
BlockSizesRecorded} and `Rekeyed` notes), and `link-N-block-M.bin`.
|
||||
|
||||
## Live capture: exact commands
|
||||
|
||||
Run from the repo root, with your real config (so WITHOUT
|
||||
`RCLONE_CONFIG=/notfound`), one label at a time. `go test` runs in the
|
||||
package directory, so give `-capture` an absolute path.
|
||||
|
||||
```sh
|
||||
cd /home/user/rclone
|
||||
go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
|
||||
-capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label web \
|
||||
-remote 'TestProtonDrive:rclone-live-probe-20261111-130916/my-little-folder'
|
||||
|
||||
go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
|
||||
-capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label cli \
|
||||
-remote 'TestProtonDrive:rclone-interop-20261112'
|
||||
```
|
||||
|
||||
Notes:
|
||||
- `-run '^TestGoldenCapture$'` matters: with `-remote` set, other tests in
|
||||
the package (e.g. `TestIntegration`) would run against the real service.
|
||||
- It reads everything, writes nothing to Proton.
|
||||
- It refuses an existing `testdata/golden/<label>`; delete it (not
|
||||
recursively from an agent) to re-run.
|
||||
- At most 3 files are captured, depth first in name order; files over
|
||||
16 MiB encrypted are skipped (logged). If `rclone-interop-20261112` also
|
||||
holds files rclone wrote (interop the other way), point `-remote` at a
|
||||
subfolder holding only CLI-written files, or the `cli` label will be
|
||||
wrong for them: the label is all the fixture knows about the writer.
|
||||
- The `-v` log lists each node, its label, size, block count, whether block
|
||||
sizes are recorded, and per field what was re-wrapped/re-made/kept.
|
||||
- Then, offline: `RCLONE_CONFIG=/notfound go test ./backend/protondrive/ -count=1 -v -run '^TestGoldenFixtures$'`.
|
||||
|
||||
## How to inspect the output for leaks
|
||||
|
||||
With `D=backend/protondrive/testdata/golden/web` (then `cli`):
|
||||
|
||||
1. Account secrets from the config, without printing them:
|
||||
```sh
|
||||
for k in username client_uid client_access_token client_refresh_token client_salted_key_pass; do
|
||||
v=$(rclone config show TestProtonDrive: | sed -n "s/^$k = //p")
|
||||
[ -n "$v" ] && grep -rqF -- "$v" "$D" && echo "LEAK: $k"
|
||||
done
|
||||
v=$(rclone config show TestProtonDrive: | sed -n 's/^client_salted_key_pass = //p' | base64 -d)
|
||||
[ -n "$v" ] && grep -rqF -- "$v" "$D" && echo "LEAK: decoded salted key pass"
|
||||
```
|
||||
(Tokens may have rotated during the run; the old ones were never written
|
||||
either.) Expect no output.
|
||||
2. Emails: `grep -rnoiE '[a-z0-9._%+-]+@[a-z0-9.-]+' "$D"/*.json | sort -u` -
|
||||
expect only `golden@example.invalid` (and possibly a non-account email in
|
||||
a node key user ID, if the client put one there; check the next step).
|
||||
3. Proton IDs (URL-safe base64, 88 characters ending `==`):
|
||||
`grep -rnE '[A-Za-z0-9_-]{86}==' "$D"/*.json` - expect nothing (armor lines
|
||||
are shorter and use `+/`).
|
||||
4. Keys: `grep -c 'BEGIN PGP PRIVATE KEY BLOCK' "$D"/keys.json` is 2 (test
|
||||
address and root keys); in `nodes.json` one per node, all new. To see
|
||||
their user IDs and creation times without importing anything:
|
||||
`jq -r '.Nodes[] | (.Folder // .File).Link.NodeKey' "$D/nodes.json" | gpg --show-keys`
|
||||
and `jq -r '.Keys.AddressKey, .Keys.RootKey' "$D/keys.json" | gpg --show-keys`.
|
||||
5. Plaintext: `jq '.Nodes[].Want' "$D/nodes.json"` is the only plaintext:
|
||||
names, modification times, media types, sizes, SHA-1s, block sizes.
|
||||
Check the names are ones you chose. Blocks (`*.bin`) are ciphertext.
|
||||
6. Notes: `jq -r '.Nodes[] | .ID + " " + .Want.Name, (.Rekeyed[] | " " + .)' "$D/nodes.json"`
|
||||
and look for `left out`, `key packets`, `made again as` (algorithm or
|
||||
version mismatch) or `claims SHA-1`: each marks something not faithful.
|
||||
7. Known by design: the session keys of names, hash keys, extended
|
||||
attributes and content are recoverable from the fixture with the test
|
||||
keys (the encrypted data is kept), so a folder's hash key and those
|
||||
names/contents are as public as the fixture. Node passphrases and every
|
||||
account key are not.
|
||||
|
||||
## Tests and results
|
||||
|
||||
All run in this session, all with `RCLONE_CONFIG=/notfound`, against the fake only:
|
||||
|
||||
- `go build ./...` - ok
|
||||
- `go vet ./backend/protondrive/...` - ok
|
||||
- `golangci-lint run ./backend/protondrive/...` - 0 issues
|
||||
- `go test -count=1 -run TestGolden -v ./backend/protondrive/` -
|
||||
TestGoldenCapture SKIP (no -capture), TestGoldenFixtures SKIP (no
|
||||
fixtures), TestGoldenCaptureFlags PASS, TestGoldenRoundTrip PASS
|
||||
- `go test -race -count=1 ./backend/protondrive/...` -
|
||||
`ok backend/protondrive 315.306s`, `ok protontest 38.192s`,
|
||||
`ok protontest/cmd/protonfake 1.400s`
|
||||
|
||||
Not TDD-first: the brief asked for a capture tool plus tests; the round-trip
|
||||
test was written with the code, and a mutation (wrong signed data) showed it
|
||||
fails as it should.
|
||||
|
||||
## Files changed
|
||||
|
||||
- backend/protondrive/golden_internal_test.go (new)
|
||||
- backend/protondrive/protontest/stored.go
|
||||
- backend/protondrive/protontest/drive.go
|
||||
- docs/DECISIONS.md
|
||||
- docs/TODO.md
|
||||
|
||||
## Self-review
|
||||
|
||||
- Mutation check: making `resignSignature` sign the wrong data makes
|
||||
`TestGoldenRoundTrip` fail (name, node key, hash key and xattr signature
|
||||
failures reported, read refused), so the read-back really verifies the
|
||||
re-made signatures.
|
||||
- The capture has only been run against the fake: real web-app/CLI data may
|
||||
hit paths the fake never produces (v6 signatures, text signatures,
|
||||
one-pass-signature nesting, compressed names, multiple PKESKs, RSA keys).
|
||||
These are handled in code but untested; failures are reported clearly
|
||||
rather than silently mis-captured.
|
||||
|
||||
## Concerns
|
||||
|
||||
- See the two deviations above (embedded signatures force re-encryption of
|
||||
names/hash keys/xattrs; passphrases are new), plus block signatures dropped
|
||||
and no user key written.
|
||||
- The live capture is untested against real data by construction (I never
|
||||
contact Proton).
|
||||
|
||||
## Commit
|
||||
|
||||
a2dc6f71f protondrive: test against re-keyed files written by Proton's own clients
|
||||
|
||||
## Fix round 1
|
||||
|
||||
Review: task-21g-review.md. Work is on top of af6fdd789.
|
||||
|
||||
### Important
|
||||
|
||||
- **I-1:** `resignSignature` now fails the capture when a signature is by, or intended for, a key that is not being replaced. The error names no key (`signature by a key which is not one being replaced` / `signature intended for a key which is not one being replaced`). Nothing about such a key reaches the notes or the fixture. Test: `TestGoldenRefusesForeignKeys` checks that a known recipient is mapped, that an unknown recipient errors, and that the fingerprint appears in neither the error nor the notes.
|
||||
- **I-2:** every key the capture writes (the address key, the root key and the node keys) now has the user ID `golden <golden@example.invalid>`. `identity()` is gone. `checkUserIDs` scans every user ID of each original node key for anything that looks like an email. If it finds one that is neither the account's nor one of the generic Proton key emails, the capture fails and the error names no email.
|
||||
- **Deviation:** I allowlisted the generic emails `no-reply@proton.me` and `noreply@protonmail.com`. The C# SDK, and so the official CLI 0.9.0, generates every node key as `Drive key <no-reply@proton.me>` (`client/cs/src/Proton.Drive.Sdk/Nodes/NodeOperations.cs:157`, `Cryptography/CryptoGenerator.cs:25`). The old Go library used `Drive key <noreply@protonmail.com>`, which is what the frozen fixture's root key carries. A strict rule would fail every CLI capture.
|
||||
- Tests: `TestGoldenRefusesForeignKeys` covers this, and the round trip's packet walk checks every written user ID equals the placeholder. As a mutation check, generating keys with a different user ID made that walk fail.
|
||||
- **I-3:** the capture now has its own flag, `-capture-remote NAME:PATH`. TestGoldenCapture needs both `-capture` and `-capture-remote`, plus `-capture-label`.
|
||||
- `-capture` or `-capture-remote` together with `-remote` fails before anything is contacted, as does either of them given without the other.
|
||||
- `-remote` on its own still means the integration tests, and TestGoldenCapture skips.
|
||||
- Test: `TestGoldenCaptureFlags`.
|
||||
|
||||
### Minor
|
||||
|
||||
- **M-1:** I added `checkGoldenPackets`. It decrypts the fixture down the key chain (root key, then each node's passphrase, then its node key) and walks every armored key, key packet, detached signature and the packets inside every encrypted message, compressed ones included. Every PKESK key ID or v6 fingerprint, signature issuer key ID and fingerprint, intended recipient, one-pass-signature key ID and signer user ID must belong to the test keys or be the placeholder. Signature emails must be empty or golden, and block signatures empty.
|
||||
- It runs in `TestGoldenRoundTrip` and also in `TestGoldenFixtures`, so the committed live fixtures get the same check.
|
||||
- The round trip's text search now also covers the source keys' key IDs and fingerprints, in hex (both cases) and base64.
|
||||
- **M-1b:** `TestGoldenRoundTrip/tampered` takes fresh copies of the captured fixture and changes one thing in each. Every case fails to read:
|
||||
- one byte of a block;
|
||||
- one byte of the manifest signature: the read fails with "manifest";
|
||||
- one byte of a name's encrypted data: the file can't be found.
|
||||
- **M-2:** a salt notation is written only when the original signature had one (with a new salt), via `NonDeterministicSignaturesViaNotation`. Other notations are copied.
|
||||
- **M-3:** I added a superseding entry to DECISIONS.md, since that file is append-only. It states what the committed fixtures expose: names, xattrs, literal file names and dates, folder hash keys and contents, all decryptable with the test keys, plus the address key's creation time. The inspection guide is corrected below.
|
||||
- **M-4:** `checkGolden` now lists every folder of the fixture, empty ones included, and the root. It checks each folder's modification time against `Want.ModTime`.
|
||||
- **M-5:** a message that mixes one-pass signatures with signatures that have none now fails with a clear error.
|
||||
|
||||
Refactors: `openMessage` (decrypt down to the inner packets) is shared by `reseal` and the packet check. `serveGolden` is split out of `checkGolden` so the tamper tests can reuse it.
|
||||
|
||||
### Final live-capture commands
|
||||
|
||||
Run from the repo root with your real config (no `RCLONE_CONFIG=/notfound`) and **without `-remote`**. Give `-capture` an absolute path, because `go test` runs in the package directory.
|
||||
|
||||
```sh
|
||||
cd /home/user/rclone
|
||||
go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
|
||||
-capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label web \
|
||||
-capture-remote 'TestProtonDrive:rclone-live-probe-20261111-130916/my-little-folder'
|
||||
|
||||
go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
|
||||
-capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label cli \
|
||||
-capture-remote 'TestProtonDrive:rclone-interop-20261112'
|
||||
|
||||
# then, offline (it also runs the packet-level leak walk):
|
||||
RCLONE_CONFIG=/notfound go test ./backend/protondrive/ -count=1 -v -run '^TestGoldenFixtures$'
|
||||
```
|
||||
|
||||
The other notes from the first round still apply:
|
||||
- at most 3 files are captured, depth first in name order;
|
||||
- files over 16 MiB encrypted are skipped;
|
||||
- an existing `<label>` directory is refused;
|
||||
- point at a subfolder if the interop folder also holds files rclone wrote.
|
||||
|
||||
### Inspection guide, corrected
|
||||
|
||||
Steps 1-4 and 6 above stand. Two changes:
|
||||
- Step 4: the node keys' user IDs are all `golden <golden@example.invalid>`.
|
||||
- Step 5 is replaced: `Want` is not the only plaintext. With the committed test keys, anyone can decrypt:
|
||||
- names;
|
||||
- extended attributes, exactly as the client wrote them;
|
||||
- the literal-packet file names and dates inside names, hash keys and xattrs;
|
||||
- folder hash keys (they let anyone test whether a guessed name exists in those real folders);
|
||||
- file contents.
|
||||
|
||||
The address key's creation time is also kept. Before committing, decrypt and read the xattrs, for example in a scratch test that calls `loadGolden` and then `decryptXAttrs` down the key chain the way `checkGoldenPackets` does. Or read them through the backend: `checkGolden` already decrypts them, and their `Common` and any other members show in a debugger or a temporary `t.Logf`. Look for camera, location or device fields, which web-app uploads of media can carry. The captured files should be random test data anyway.
|
||||
|
||||
### Verification (fix round 1)
|
||||
|
||||
All with `RCLONE_CONFIG=/notfound`, fake only:
|
||||
- `go build ./...` ok; `go vet ./backend/protondrive/...` ok; `golangci-lint run ./backend/protondrive/...` 0 issues
|
||||
- `go test -count=1 -run TestGolden -v ./backend/protondrive/`: Capture SKIP, Fixtures SKIP (no fixtures), CaptureFlags PASS, RoundTrip PASS (with tampered/block, tampered/manifest_signature, tampered/name), RefusesForeignKeys PASS
|
||||
- `go test -race -count=1 ./backend/protondrive/...`: ok protondrive 309.382s, ok protontest 36.647s, ok protonfake 1.359s
|
||||
|
||||
Commit: 1b2fcbd21 protondrive: make the golden capture fail closed on foreign keys
|
||||
@@ -0,0 +1,154 @@
|
||||
# Task 21g re-review, fix round 1 (af6fdd789..1b2fcbd21)
|
||||
|
||||
Line numbers are of `backend/protondrive/golden_internal_test.go` at 1b2fcbd21.
|
||||
|
||||
**Verdict:** NOT ALL CLOSED. I-1 and I-2 are closed. I-3 is still open: the
|
||||
writing integration suite can still run against the live account after a
|
||||
mistyped `-run`.
|
||||
|
||||
## Findings from the first review
|
||||
|
||||
- **I-1: closed.** In `resignSignature`, an unknown issuer and an unknown
|
||||
intended recipient now both fail the capture with an error that names no
|
||||
key. Nothing about them is written to the notes or the fixture.
|
||||
- `TestGoldenRefusesForeignKeys` checks that the stranger's fingerprint is
|
||||
in neither the error nor `rk.notes`.
|
||||
- The round trip's text search now also covers the source keys' key IDs and
|
||||
fingerprints, in hex (both cases) and base64.
|
||||
- **I-2: closed, and the controller's proviso holds.** `generateKeyLike`
|
||||
always writes the user ID `golden <golden@example.invalid>` and takes no
|
||||
name or email from the original key. That covers the address key, the root
|
||||
key and every node key.
|
||||
- `checkUserIDs` scans the full user-ID string (`Identity.Name` holds the
|
||||
name and the email) of each original node key and of the root key. Its
|
||||
error names no email.
|
||||
- The allowlist (`no-reply@proton.me`, `noreply@protonmail.com`) only lets
|
||||
a capture go ahead. No user-ID text from the original keys reaches a
|
||||
fixture.
|
||||
- `checkGoldenPackets` asserts the placeholder on every identity of every
|
||||
written key (`addKey`). My mutation test below confirms it.
|
||||
- **I-3: NOT closed (Important).** The capture flags were split from
|
||||
`-remote`, but that does not stop `TestIntegration` from running.
|
||||
- `TestIntegration` (protondrive_test.go:11-16) calls `fstests.Run` with the
|
||||
default `RemoteName: "TestProtonDrive:"`.
|
||||
- Without `-remote`, `fstests.Run` skips only when `RCLONE_CONFIG` ends in
|
||||
`/notfound` (fstest/fstests/fstests.go:409) or the remote is not configured
|
||||
(:413-416, :484).
|
||||
- A live capture has to run with the real config, and in that config
|
||||
`TestProtonDrive` is configured. It is the very remote named in
|
||||
`-capture-remote` in the report's commands.
|
||||
- So a missing or over-broad `-run`, for example `-run 'Capture|Integration'`,
|
||||
`-run Test`, or none at all, runs the full write/delete suite against the
|
||||
real account. Nothing in TestIntegration looks at `-capture` or
|
||||
`-capture-remote`.
|
||||
- Likewise, with `-capture … -remote X`, only TestGoldenCapture fails;
|
||||
TestIntegration in the same binary still runs against X.
|
||||
- `checkCaptureFlags` itself is correct. TestGoldenCapture calls it first
|
||||
(line 138), before `fstest.Initialise`/`fs.NewFs`, so a bad flag
|
||||
combination fails before any config or network access. TestMain
|
||||
(budget_test.go:34) reads no config.
|
||||
- **Fix:** make `TestIntegration` skip, or fail, while a capture is asked
|
||||
for. For example, in protondrive_test.go, check
|
||||
`flag.Lookup("capture").Value.String() != "" || flag.Lookup("capture-remote").Value.String() != ""`.
|
||||
Both test packages build into one binary, so the flags are registered.
|
||||
Add a line to `TestGoldenCaptureFlags` or a small test for it.
|
||||
- **M-1: closed.** `checkGoldenPackets` is strict (see the next section).
|
||||
- **M-1b: closed.** The tamper cases fail because of the tampering, which I
|
||||
confirmed against a control (see Checks).
|
||||
- **M-2: closed.** The salt notation is kept only when the original had one
|
||||
(`NonDeterministicSignaturesViaNotation: &salted`). Other notations are
|
||||
copied.
|
||||
- **M-3: closed.** The DECISIONS.md entry and the corrected inspection guide
|
||||
state what the fixture exposes.
|
||||
- **M-4: closed.** Every folder is listed, the root and empty folders
|
||||
included, and each folder's modification time is checked.
|
||||
- **M-5: closed.** A message that mixes signatures with and without one-pass
|
||||
signatures now fails with a clear error (line ~1234).
|
||||
|
||||
## Checks run
|
||||
|
||||
All runs were in a `git archive 1b2fcbd21` copy in the scratchpad, with
|
||||
`RCLONE_CONFIG=/notfound` and the fake server only. /home/user/rclone was not
|
||||
touched.
|
||||
|
||||
- **checkGoldenPackets strictness, by mutation.** I added subtests to
|
||||
`TestGoldenRoundTrip` that each corrupt a fresh `loadGolden` copy and call
|
||||
`checkGoldenPackets`. Every mutation failed, and the unmutated control
|
||||
passed.
|
||||
|
||||
| Mutation | Failure reported |
|
||||
|---|---|
|
||||
| Original `ContentKeyPacket` (a PKESK to a source key) | "key packet for a key which is not a test key" |
|
||||
| Original `ManifestSignature` (issuer subpacket and issuer fingerprint of the source address key) | "issuer … is not a test key", "issuer fingerprint is not a test key's" |
|
||||
| A signature made by the test address key with an intended recipient of the source root key's fingerprint | "intended recipient is not a test key" |
|
||||
| An address key with the user ID `Leak <leak@proton.me>` | `user ID` assert failed, plus issuer and one-pass failures for everything the old key signed |
|
||||
|
||||
- **checkGoldenPackets, by reading.**
|
||||
- A v4 PKESK with a wildcard (zero) key ID, and a v6 PKESK with no
|
||||
fingerprint, both fail. That is fail-closed.
|
||||
- For v6 signatures, go-crypto fills in `IssuerKeyId` from the fingerprint
|
||||
(go-crypto signature.go:698-702), so the `require.NotNil` does not fail
|
||||
falsely.
|
||||
- The checker walks every armored or base64 field of `StoredLink`,
|
||||
`StoredRevision`, `StoredFolder` and `StoredBlock`, and the root hash
|
||||
key.
|
||||
- Marker, padding and SKESK packets, user-ID packets and key packets inside
|
||||
a message all fail through the `default` case.
|
||||
- **A verbatim original key would be caught.** Every key in the fixture is
|
||||
trusted as a "test key", so the identity check is what catches an original
|
||||
key copied verbatim: every real Proton key carries an address or
|
||||
`Drive key <…>` user ID. The round trip's text search also covers it for
|
||||
the fake.
|
||||
- **Tamper tests.**
|
||||
- I added a control: reading the same untampered fixture succeeds.
|
||||
- The block case fails with "block 1: block hash mismatch".
|
||||
- The name case fails with "object not found", because the name no longer
|
||||
decrypts.
|
||||
- The manifest case fails on the manifest signature.
|
||||
- **Read-only on the account.** The diff changes only the flags and the
|
||||
`fs.NewFs` argument (`*goldenCaptureRemote`). No write call was added.
|
||||
- **New leak paths.**
|
||||
- The new errors name no key or email (`checkUserIDs`, `resignSignature`).
|
||||
- Notes now carry only the placeholder user ID.
|
||||
- `checkGoldenPackets` failure messages print unknown key IDs, and a
|
||||
`user ID` mismatch prints the user ID, to the terminal only. Those are
|
||||
public identifiers, not passphrases or tokens, so this is acceptable.
|
||||
- No passphrase, session key or token is printed anywhere.
|
||||
|
||||
## New findings
|
||||
|
||||
### Critical
|
||||
|
||||
None.
|
||||
|
||||
### Important
|
||||
|
||||
- **I-3 (still open).** See above. protondrive_test.go:11-16 together with
|
||||
fstest/fstests/fstests.go:409-416: TestIntegration ignores the capture
|
||||
flags, so a mistyped `-run` during a live capture runs the writing suite on
|
||||
the account.
|
||||
|
||||
### Minor
|
||||
|
||||
- **m-1, `checkPackets` (in checkGoldenPackets):** a
|
||||
`*packet.SymmetricallyEncrypted` packet is accepted silently. `checkPackets`
|
||||
never sees the outer data packet (`keyPackets` stops before it and `inner`
|
||||
is already decrypted). So this case is reachable only for an encrypted
|
||||
packet nested inside a message, and it would hide that packet's contents
|
||||
from the check. `resignPackets` already refuses such packets when writing,
|
||||
so there is no leak today. Make it an error.
|
||||
- **m-2:** `checkGoldenPackets` does not decrypt and walk the block data. The
|
||||
blocks are kept byte for byte from the client. A client that put an inline
|
||||
signature inside the block SEIPD would leave its issuer key ID unchecked.
|
||||
Proton's clients use the detached `EncSignature` instead, and the fixture
|
||||
requires that field to be empty, so the risk is low. A fix would decrypt
|
||||
each block with the content session key and walk it.
|
||||
- **m-3:** non-salt notations are copied verbatim from the original
|
||||
signatures, and neither the writer nor the checker looks at their values.
|
||||
Proton's clients are not known to set any. A fix would fail the capture on
|
||||
any notation other than the salt notation, or record each notation's name
|
||||
in `Rekeyed`.
|
||||
- **m-4:** no tamper case breaks a re-made signature that sits inside an
|
||||
intact SEIPD, such as a name's or an xattr's embedded signature. The block
|
||||
case is caught by the stored hash and the name case by decryption failure.
|
||||
The manifest case covers detached signatures only.
|
||||
@@ -0,0 +1,83 @@
|
||||
# Task 21g review (502a32b5f..a2dc6f71f)
|
||||
|
||||
Line numbers are of `backend/protondrive/golden_internal_test.go` at a2dc6f71f, not of the diff.
|
||||
|
||||
### Spec Compliance
|
||||
|
||||
- ❌ Issues found: sanitisation fails open in two places, which goes against the owner's rule that no account secret or ID reaches a fixture:
|
||||
- golden_internal_test.go:1258: an intended-recipient fingerprint that is not in the replacement map is written in hex into the fixture, inside `Rekeyed` in nodes.json (I-1).
|
||||
- golden_internal_test.go:898-908: the user-ID name of a node key is copied as it is, and an email that doesn't belong to the account is kept (I-2).
|
||||
|
||||
Everything else in the brief is done. Two deviations are justified and documented in the report and in DECISIONS.md:
|
||||
- Embedded signatures force the SEIPD packets of names, hash keys and xattrs to be encrypted again, with the same session key and parameters.
|
||||
- Node passphrases are replaced.
|
||||
|
||||
Also documented: block signatures are dropped and no user key is written.
|
||||
- ⚠️ Cannot verify from the diff: how the code behaves on real web-app and CLI data. That covers v6 signatures, text signatures, intended recipients, SignerUserId, RSA keys and several PKESKs; the fake produces none of them. The controller must run inspection steps 2, 4 and 6 of the report on the live output, and also look at the decrypted xattrs (see M-3).
|
||||
|
||||
### Checks run
|
||||
|
||||
- **Leak, packet level.** I ran a throwaway copy of the package (git archive into the scratchpad, with `RCLONE_CONFIG=/notfound`) that captures the round-trip seeds and parses every armored blob and ContentKeyPacket in the fixture. Every PKESK key ID, signature issuer key ID and issuer fingerprint belongs to a key in keys.json or nodes.json. None belongs to the source address, root or node keys, and no source key ID or fingerprint appears in nodes.json as text. The test PASSED. The fake writes no intended recipients and no SignerUserId, so those two branches were not exercised.
|
||||
- **Re-made signatures.** `resignSignature` (golden_internal_test.go:1232-1283) builds each signature from scratch. It copies only the type, hash, creation time, lifetime and notations. The issuer key ID is the new key's, and go-crypto `Signature.Sign` sets IssuerFingerprint from the signing key (go-crypto@v1.5.2 signature.go:978). Unhashed subpackets, embedded signatures and the original salt are not carried over.
|
||||
- **PKESKs.** `wrap` (golden_internal_test.go:926-941) makes one new PKESK to the test key. All original key packets are dropped (golden_internal_test.go:959-1007).
|
||||
- **NodePassphrase.** It is replaced, including inside a compressed packet, because `plain` recurses (golden_internal_test.go:1093). Its session key is new (golden_internal_test.go:1008-1015).
|
||||
- **Unexpected inner packets.** Any packet that is not compressed, literal, one-pass signature or signature fails the capture (golden_internal_test.go:1131-1132), so no stray key or user-ID packet can pass through.
|
||||
- **Emails.** `owns` is case-insensitive (account.go:261). An email not of the account fails the capture (golden_internal_test.go:893).
|
||||
- **IDs, URLs, tokens.** Only `link-N` placeholders are written. Volume, share, revision, block and address IDs, URLs and tokens are never put in the structs that get written (golden_internal_test.go:616-621, 722-750).
|
||||
- **User key.** No user key is written.
|
||||
- **Terminal output.** It holds node names, notes, and in error messages foreign emails or unknown key IDs. No passphrases, tokens or keys are printed.
|
||||
- **Read-only.** I checked the methods used: `RootID(ctx, false)`, `getNode`, `listChildren`, `open`, `revisionBlocks` and `downloadBlock` are all reads. `Fs.Shutdown` only clears the cache (protondrive.go:1066-1069); it does not log out.
|
||||
- Files are written only after the whole capture succeeds (golden_internal_test.go:474), and an existing directory is refused (golden_internal_test.go:429-431).
|
||||
- A capture with `ignore_signature_errors` on is refused (golden_internal_test.go:433-435).
|
||||
- `NewFs` would create a volume on an account that has none (nodes.go:410). That is normal backend behaviour and does not apply to an account the web app has used.
|
||||
- **protontest changes.** `insertLink`'s callers (drive.go:555, 719) never set `modifyTime`, so keeping a given modification time changes nothing for them.
|
||||
|
||||
### Strengths
|
||||
|
||||
- The re-keying is careful and close to the original format:
|
||||
- same PKESK version, SEIPD version, AEAD mode, chunk size and compression algorithm;
|
||||
- the literal packet kept verbatim;
|
||||
- SEIPD packets with no signature kept byte for byte.
|
||||
- Every node records what was done to it in `Rekeyed`.
|
||||
- Strict refusals: unverified nodes, any problem the backend reported, unknown signers, foreign emails, thumbnails, unsupported key algorithms.
|
||||
- `checkGolden` is a meaningful test:
|
||||
- it reads only through `Fs`/`Object`, with `ignore_signature_errors=false`;
|
||||
- it compares listings;
|
||||
- per file it checks size, modification time, MIME type, `Hash` and a full read against SHA-1;
|
||||
- the ranged read starts halfway into block 2 (golden_internal_test.go:412-421);
|
||||
- it fails if anything was reported or skipped (golden_internal_test.go:376-379).
|
||||
- The capture loads and checks its own output straight away (golden_internal_test.go:143-144).
|
||||
- The round trip covers SDK-style files, files with unrecorded block sizes, legacy files, anonymous files and the file cap. It checks the blocks are byte-identical, and its secret list is broad (golden_internal_test.go:266-288).
|
||||
|
||||
### Issues
|
||||
|
||||
#### Critical (Must Fix)
|
||||
|
||||
None.
|
||||
|
||||
#### Important (Should Fix)
|
||||
|
||||
- **I-1, golden_internal_test.go:1256-1259: leak.** When a signature's intended recipient is not in `rk.fingerprints`, the note `intended recipient %X, an unknown key, left out` writes the original fingerprint into nodes.json `Rekeyed`. That is a public identifier of a real account key: the account's user key, another member's key, or a subkey the map doesn't cover. It ties the fixture to the account, and the brief says every key ID is replaced. **Fix:** fail the capture, or write the note without the fingerprint. Add the hex and base64 fingerprints of the source keys to the round trip's secret list.
|
||||
- **I-2, golden_internal_test.go:898-908: leak path.** `identity` passes only the email part of a node key's user ID through `owns`. The name part is copied as it is, and a non-account email is kept. Proton address-style user IDs put the address in the name as well (`addr@proton.me <addr@proton.me>`), and a client could put a display name or another address there. The result sits inside armored keys, where the report's email grep (step 2) can't see it. **Fix:** treat the name with the same rule (use `nodeKeyUserID` when the name contains `@` or doesn't match the original's generic value), and fail on, or drop, non-account emails, as `rk.email` already does.
|
||||
- **I-3 (plan-mandated), golden_internal_test.go:38-39, 108-128: unsafe flag.** The capture reuses `-remote`. If the `-run` pattern is wrong or missing, `TestIntegration` in protondrive_test.go runs the full write/delete suite against the real account. `TestIntegrationFake` skips in that case (fake_test.go:66), and nothing in TestGoldenCapture can stop the other test. The brief specifies `-remote`, but a read-only capture should not be one regex away from a writing suite. **Fix:** give the capture its own flag (for example `-capture-remote`), so that `-remote` is never set during a capture.
|
||||
|
||||
#### Minor (Nice to Have)
|
||||
|
||||
- **M-1: leak test checks strings only.** The leak check in `TestGoldenRoundTrip` (golden_internal_test.go:266-288) searches raw and JSON-escaped strings. It does not parse packets, so a source key ID or fingerprint inside an armored signature or PKESK would not be found. Add a packet walk like the one I ran: every PKESK, issuer and recipient must belong to a test key.
|
||||
- **M-1b: no automated tamper test.** No test shows that `checkGolden` fails on a tampered fixture, for example one signature byte flipped. The mutation check exists only in the report.
|
||||
- **M-2: salt notation added.** At golden_internal_test.go:1248 and 1275, `Sign` with a nil config removes any `salt@notations.openpgpjs.org` notation and appends a fresh one (go-crypto signature.go:979-992). Every v4 signature that is made again therefore carries a salt notation, even when the original (for example from the Go SDK or CLI) had none. So "same notations" in DECISIONS.md and the report is not exact. Note it in `Rekeyed` when the original had no salt notation, or turn the salt notation off when the original had none.
|
||||
- **M-3: other data the fixture still holds.** The report's inspection step 5 says `Want` is the only plaintext. That is wrong: names, xattrs and folder hash keys can be decrypted with the committed test keys.
|
||||
- Xattrs are kept as written, so a media file from the web app could carry Camera or Location fields.
|
||||
- The literal-packet file name and date are kept (golden_internal_test.go:1114).
|
||||
- The real folders' hash keys are exposed. Anyone can then test whether a guessed name exists in those real folders.
|
||||
- The address key's creation time is kept (golden_internal_test.go:572). That is a weak account fingerprint.
|
||||
|
||||
Add "decrypt and read the xattrs" to the inspection steps, and state the exposure of hash keys and session keys in the DECISIONS.md entry.
|
||||
- **M-4: folder times unchecked.** golden_internal_test.go:357-366 records `Want.ModTime` for folders but never checks it.
|
||||
- **M-5: one-pass signature pairing.** At golden_internal_test.go:1139-1141 and 1156-1169, the nesting assumes every signature has a one-pass signature. A message mixing a prefixed signature with one-pass signatures would pair them wrongly. The read-back after capture would catch it, but the error would be confusing.
|
||||
|
||||
### Assessment
|
||||
|
||||
**Task quality:** Needs fixes
|
||||
|
||||
**Reasoning:** The re-keying is careful, and my packet-level check of the fake round trip found no source key IDs, fingerprints, emails or passphrases. Two paths still copy data from the real account into the fixture (an unmapped intended-recipient fingerprint in the notes, and a node key's user-ID name or a non-account email). Reusing `-remote` leaves a writing suite one `-run` slip away. All three should be fixed before the live capture.
|
||||
Reference in new issue
Block a user