rclone/docs/plans/001-protondrive-tier1-workspace/task-21g-report.md

Task 21g report: golden fixtures from real Proton clients, re-keyed

What I implemented

• backend/protondrive/golden_internal_test.go (new):
  • TestGoldenCapture: flags -capture DIR, -capture-label NAME and the existing -remote NAME:PATH. Skips without -capture; fails clearly if -capture is given without -remote or without -capture-label (or with a label that is not a plain name). Signs in through fs.NewFs (the remote's saved session), captures into DIR/LABEL/ (refuses if it exists), then immediately loads that output into the fake and reads it back as TestGoldenFixtures does, so a bad capture fails at once.
  • captureGolden: walks the folder at the Fs root depth first in name order, all subfolders, at most 3 files (goldenMaxFiles), skipping files over 16 MiB encrypted (logged). Refuses to run with ignore_signature_errors on; refuses any node whose signatures don't verify with the account's keys (node key unverified or any problem the backend reported for that node); reads each file whole through c.open (which checks content key, block hashes and manifest), then downloads the raw blocks again through c.revisionBlocks / c.downloadBlock. Refuses files with thumbnails.
  • rekeyer: the re-keying (details below). Reuses pgpFor, armorMessage, decryptContentKey, decryptBlock, generateHashKey, generatePassphrase, nameHash; new crypto only where the backend has nothing (re-making signatures with chosen parameters, re-encrypting an SEIPD packet with a given session key).
  • TestGoldenFixtures: for each testdata/golden/*/nodes.json, seeds a fake user holding the fixture's test keys (AddUserWithKeys), stores every node as Proton served it (SeedStoredFolder / SeedStoredFile), and through Fs/Object only, with ignore_signature_errors=false: lists every folder (and the root) and compares names, then per file checks size, modification time, media type, SHA-1 (Object.Hash), a full read against the expected SHA-1, and a ranged read starting in the second block (or the middle of a one-block file) against the full read. Fails if the backend reported any problem (c.reported) or skipped any node. Skips when there are no fixtures.
  • TestGoldenCaptureFlags: the flag rules.
  • TestGoldenRoundTrip: seeds the fake (user with the known keys of the frozen testdata/oldlib/file/legacy fixture) with a captured folder and a subfolder holding: a 5 MiB+7 SDK-style file (v6 PKESK, SEIPDv2, two blocks, sizes recorded), a file in 1 MiB blocks with no recorded block sizes (SeedFileInBlocks), a legacy file (v3 PKESK, SEIPDv1, legacy time format), an anonymous file (signed by the parent folder's node key, no signature emails), and a fifth file beyond the cap (capture run with a cap of 4). Captures into t.TempDir(), loads, reads everything back through the backend and checks the bytes equal the seeded content, the blocks are byte-identical to what the fake stored, block-size shapes and anonymity survive, and the anonymous file's name signature is recorded as re-made by its parent's new node key. Then searches every written file (raw and JSON-escaped) for: the source email, the source address key and root key (armored) and passphrases, the root hash key, volume, share, root and every link ID, the salted key password (raw, base64, hex), every session and storage token the fake issued, every original armored node key, encrypted passphrase and passphrase signature, and every original node passphrase and content session key (raw, base64, hex).
• backend/protondrive/protontest/stored.go: StoredFolder and SeedStoredFolder; CreateTime/ModifyTime on StoredLink and CreateTime on StoredRevision (zero means the time of seeding, so the existing frozen fixtures load unchanged); StoredFile() fills them.
• protontest/drive.go: insertLink keeps a given modification time.
• docs/DECISIONS.md: one entry, what is preserved, replaced and dropped.
• docs/TODO.md: the pending live capture under "Now"; the thumbnail and key-algorithm limits of the capture under "Next".

Plan checkbox for the capture (Task 21) is NOT ticked: it needs the live capture and the committed fixtures, which are yours.

The re-keying, per packet, and where the brief's plan could not be followed as written

Keys: a test address key (same algorithm, version, creation time and AEAD feature as the main share address's first key) replaces every key of every address of the account; a test root key replaces the captured folder's parent's node key (its node key, passphrase and a new hash key go in keys.json as protontest.FixedKeys); each node gets a new node key "like" its original (generateKeyLike: same public-key algorithm, RSA bit length, key version, creation time, SEIPDv2 feature; user ID copied with account emails replaced). Supported: RSA, EdDSA (legacy), Ed25519, Ed448; anything else fails the capture clearly.

Field	What happens
blocks	byte for byte; hashes unchanged
block signatures (EncSignature)	dropped: the backend never reads them (api.Block has no such field). Recorded per file.
ContentKeyPacket	same session key re-wrapped to the new node key, same PKESK version (v3/v6)
ContentKeyPacketSignature, NodePassphraseSignature, ManifestSignature (detached)	re-made by the key replacing the issuer, same signature type, hash, creation time, lifetime, notations; manifest over the unchanged block hashes
Name, NodeHashKey, XAttr (file and folder)	deviation: the brief says keep their SEIPD packets byte for byte AND re-make the address/node-key signatures. That is impossible: those signatures are embedded inside the encrypted packet. Closest faithful option taken: decrypt the SEIPD packet, keep every inner packet (literal data packet verbatim: format byte, file name, date, content; compression algorithm), re-make only the one-pass-signature and signature packets, and encrypt the inner stream again with the SAME session key, cipher, SEIPD version, AEAD mode and chunk size. A message holding no signature keeps its SEIPD packet byte for byte (only the PKESK is re-wrapped). Each field records which was done.
NodePassphrase	deviation: the passphrase is a secret of the account, so a new passphrase locks the new node key; it is encrypted in the original message's structure (same inner packets, new session key of the same algorithm and version). Its detached signature is re-made over the new passphrase.
NameHash	kept for every node but the captured folder, whose parent changed; that one is recomputed with the test root hash key
intended recipients in signatures	mapped to the replacement key's fingerprint; an unknown one is dropped and noted
signer user ID subpacket	replaced by golden@example.invalid
signature algorithm/version	follows the replacement key, which matches the original's; a mismatch would be noted
emails	account emails -> golden@example.invalid; empty stays empty; an email not of the account fails the capture (its keys can't be replaced)
IDs	placeholders link-N; volume/share/revision/block/address/key IDs, URLs and tokens are not written at all
timestamps, sizes, media types, block sizes, whether sizes are recorded	kept (link CreateTime/ModifyTime and revision CreateTime are now served by the fake from the fixture)
user key	not written: AddUserWithKeys seals the address key passphrase to a user key the fake generates, so a fixture user key would be unused
thumbnails	a file with thumbnails fails the capture: the fake serves none and the manifest covers them

Packet headers are re-serialised with fixed lengths (not partial lengths). Signature issuer subpackets are as go-crypto writes them (issuer key ID and issuer fingerprint).

Fixture layout

DIR/LABEL/keys.json (goldenKeys: Comment, Email, FixedKeys - armored private test keys and their passphrases), DIR/LABEL/nodes.json (goldenFixture: Label, Nodes, each with placeholder ID/ParentID, the stored folder or file as protontest.StoredFolder/StoredFile, Want {Name, ModTime, MIMEType, Size, SHA1, BlockSizes (plaintext, measured), BlockSizesRecorded} and Rekeyed notes), and link-N-block-M.bin.

Live capture: exact commands

Run from the repo root, with your real config (so WITHOUT RCLONE_CONFIG=/notfound), one label at a time. go test runs in the package directory, so give -capture an absolute path.

cd /home/user/rclone
go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
  -capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label web \
  -remote 'TestProtonDrive:rclone-live-probe-20261111-130916/my-little-folder'

go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
  -capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label cli \
  -remote 'TestProtonDrive:rclone-interop-20261112'

Notes:

• -run '^TestGoldenCapture$' matters: with -remote set, other tests in the package (e.g. TestIntegration) would run against the real service.
• It reads everything, writes nothing to Proton.
• It refuses an existing testdata/golden/<label>; delete it (not recursively from an agent) to re-run.
• At most 3 files are captured, depth first in name order; files over 16 MiB encrypted are skipped (logged). If rclone-interop-20261112 also holds files rclone wrote (interop the other way), point -remote at a subfolder holding only CLI-written files, or the cli label will be wrong for them: the label is all the fixture knows about the writer.
• The -v log lists each node, its label, size, block count, whether block sizes are recorded, and per field what was re-wrapped/re-made/kept.
• Then, offline: RCLONE_CONFIG=/notfound go test ./backend/protondrive/ -count=1 -v -run '^TestGoldenFixtures$'.

How to inspect the output for leaks

With D=backend/protondrive/testdata/golden/web (then cli):

1. Account secrets from the config, without printing them:

   for k in username client_uid client_access_token client_refresh_token client_salted_key_pass; do
     v=$(rclone config show TestProtonDrive: | sed -n "s/^$k = //p")
     [ -n "$v" ] && grep -rqF -- "$v" "$D" && echo "LEAK: $k"
   done
   v=$(rclone config show TestProtonDrive: | sed -n 's/^client_salted_key_pass = //p' | base64 -d)
   [ -n "$v" ] && grep -rqF -- "$v" "$D" && echo "LEAK: decoded salted key pass"
   

   (Tokens may have rotated during the run; the old ones were never written either.) Expect no output.
2. Emails: grep -rnoiE '[a-z0-9._%+-]+@[a-z0-9.-]+' "$D"/*.json | sort -u - expect only golden@example.invalid (and possibly a non-account email in a node key user ID, if the client put one there; check the next step).
3. Proton IDs (URL-safe base64, 88 characters ending ==): grep -rnE '[A-Za-z0-9_-]{86}==' "$D"/*.json - expect nothing (armor lines are shorter and use +/).
4. Keys: grep -c 'BEGIN PGP PRIVATE KEY BLOCK' "$D"/keys.json is 2 (test address and root keys); in nodes.json one per node, all new. To see their user IDs and creation times without importing anything: jq -r '.Nodes[] | (.Folder // .File).Link.NodeKey' "$D/nodes.json" | gpg --show-keys and jq -r '.Keys.AddressKey, .Keys.RootKey' "$D/keys.json" | gpg --show-keys.
5. Plaintext: jq '.Nodes[].Want' "$D/nodes.json" is the only plaintext: names, modification times, media types, sizes, SHA-1s, block sizes. Check the names are ones you chose. Blocks (*.bin) are ciphertext.
6. Notes: jq -r '.Nodes[] | .ID + " " + .Want.Name, (.Rekeyed[] | " " + .)' "$D/nodes.json" and look for left out, key packets, made again as (algorithm or version mismatch) or claims SHA-1: each marks something not faithful.
7. Known by design: the session keys of names, hash keys, extended attributes and content are recoverable from the fixture with the test keys (the encrypted data is kept), so a folder's hash key and those names/contents are as public as the fixture. Node passphrases and every account key are not.

Tests and results

All run in this session, all with RCLONE_CONFIG=/notfound, against the fake only:

• go build ./... - ok
• go vet ./backend/protondrive/... - ok
• golangci-lint run ./backend/protondrive/... - 0 issues
• go test -count=1 -run TestGolden -v ./backend/protondrive/ - TestGoldenCapture SKIP (no -capture), TestGoldenFixtures SKIP (no fixtures), TestGoldenCaptureFlags PASS, TestGoldenRoundTrip PASS
• go test -race -count=1 ./backend/protondrive/... - ok backend/protondrive 315.306s, ok protontest 38.192s, ok protontest/cmd/protonfake 1.400s

Not TDD-first: the brief asked for a capture tool plus tests; the round-trip test was written with the code, and a mutation (wrong signed data) showed it fails as it should.

Files changed

• backend/protondrive/golden_internal_test.go (new)
• backend/protondrive/protontest/stored.go
• backend/protondrive/protontest/drive.go
• docs/DECISIONS.md
• docs/TODO.md

Self-review

• Mutation check: making resignSignature sign the wrong data makes TestGoldenRoundTrip fail (name, node key, hash key and xattr signature failures reported, read refused), so the read-back really verifies the re-made signatures.
• The capture has only been run against the fake: real web-app/CLI data may hit paths the fake never produces (v6 signatures, text signatures, one-pass-signature nesting, compressed names, multiple PKESKs, RSA keys). These are handled in code but untested; failures are reported clearly rather than silently mis-captured.

Concerns

• See the two deviations above (embedded signatures force re-encryption of names/hash keys/xattrs; passphrases are new), plus block signatures dropped and no user key written.
• The live capture is untested against real data by construction (I never contact Proton).

Commit

a2dc6f71f protondrive: test against re-keyed files written by Proton's own clients

Fix round 1

Review: task-21g-review.md. Work is on top of af6fdd789.

Important

• I-1: resignSignature now fails the capture when a signature is by, or intended for, a key that is not being replaced. The error names no key (signature by a key which is not one being replaced / signature intended for a key which is not one being replaced). Nothing about such a key reaches the notes or the fixture. Test: TestGoldenRefusesForeignKeys checks that a known recipient is mapped, that an unknown recipient errors, and that the fingerprint appears in neither the error nor the notes.
• I-2: every key the capture writes (the address key, the root key and the node keys) now has the user ID golden <golden@example.invalid>. identity() is gone. checkUserIDs scans every user ID of each original node key for anything that looks like an email. If it finds one that is neither the account's nor one of the generic Proton key emails, the capture fails and the error names no email.
  • Deviation: I allowlisted the generic emails no-reply@proton.me and noreply@protonmail.com. The C# SDK, and so the official CLI 0.9.0, generates every node key as Drive key <no-reply@proton.me> (client/cs/src/Proton.Drive.Sdk/Nodes/NodeOperations.cs:157, Cryptography/CryptoGenerator.cs:25). The old Go library used Drive key <noreply@protonmail.com>, which is what the frozen fixture's root key carries. A strict rule would fail every CLI capture.
  • Tests: TestGoldenRefusesForeignKeys covers this, and the round trip's packet walk checks every written user ID equals the placeholder. As a mutation check, generating keys with a different user ID made that walk fail.
• I-3: the capture now has its own flag, -capture-remote NAME:PATH. TestGoldenCapture needs both -capture and -capture-remote, plus -capture-label.
  • -capture or -capture-remote together with -remote fails before anything is contacted, as does either of them given without the other.
  • -remote on its own still means the integration tests, and TestGoldenCapture skips.
  • Test: TestGoldenCaptureFlags.

Minor

• M-1: I added checkGoldenPackets. It decrypts the fixture down the key chain (root key, then each node's passphrase, then its node key) and walks every armored key, key packet, detached signature and the packets inside every encrypted message, compressed ones included. Every PKESK key ID or v6 fingerprint, signature issuer key ID and fingerprint, intended recipient, one-pass-signature key ID and signer user ID must belong to the test keys or be the placeholder. Signature emails must be empty or golden, and block signatures empty.
  • It runs in TestGoldenRoundTrip and also in TestGoldenFixtures, so the committed live fixtures get the same check.
  • The round trip's text search now also covers the source keys' key IDs and fingerprints, in hex (both cases) and base64.
• M-1b: TestGoldenRoundTrip/tampered takes fresh copies of the captured fixture and changes one thing in each. Every case fails to read:
  • one byte of a block;
  • one byte of the manifest signature: the read fails with "manifest";
  • one byte of a name's encrypted data: the file can't be found.
• M-2: a salt notation is written only when the original signature had one (with a new salt), via NonDeterministicSignaturesViaNotation. Other notations are copied.
• M-3: I added a superseding entry to DECISIONS.md, since that file is append-only. It states what the committed fixtures expose: names, xattrs, literal file names and dates, folder hash keys and contents, all decryptable with the test keys, plus the address key's creation time. The inspection guide is corrected below.
• M-4: checkGolden now lists every folder of the fixture, empty ones included, and the root. It checks each folder's modification time against Want.ModTime.
• M-5: a message that mixes one-pass signatures with signatures that have none now fails with a clear error.

Refactors: openMessage (decrypt down to the inner packets) is shared by reseal and the packet check. serveGolden is split out of checkGolden so the tamper tests can reuse it.

Final live-capture commands

Run from the repo root with your real config (no RCLONE_CONFIG=/notfound) and without -remote. Give -capture an absolute path, because go test runs in the package directory.

cd /home/user/rclone
go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
  -capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label web \
  -capture-remote 'TestProtonDrive:rclone-live-probe-20261111-130916/my-little-folder'

go test ./backend/protondrive/ -v -count=1 -timeout 30m -run '^TestGoldenCapture$' \
  -capture /home/user/rclone/backend/protondrive/testdata/golden -capture-label cli \
  -capture-remote 'TestProtonDrive:rclone-interop-20261112'

# then, offline (it also runs the packet-level leak walk):
RCLONE_CONFIG=/notfound go test ./backend/protondrive/ -count=1 -v -run '^TestGoldenFixtures$'

The other notes from the first round still apply:

• at most 3 files are captured, depth first in name order;
• files over 16 MiB encrypted are skipped;
• an existing <label> directory is refused;
• point at a subfolder if the interop folder also holds files rclone wrote.

Inspection guide, corrected

Steps 1-4 and 6 above stand. Two changes:

• Step 4: the node keys' user IDs are all golden <golden@example.invalid>.

• Step 5 is replaced: Want is not the only plaintext. With the committed test keys, anyone can decrypt:

  • names;
  • extended attributes, exactly as the client wrote them;
  • the literal-packet file names and dates inside names, hash keys and xattrs;
  • folder hash keys (they let anyone test whether a guessed name exists in those real folders);
  • file contents.

  The address key's creation time is also kept. Before committing, decrypt and read the xattrs, for example in a scratch test that calls loadGolden and then decryptXAttrs down the key chain the way checkGoldenPackets does. Or read them through the backend: checkGolden already decrypts them, and their Common and any other members show in a debugger or a temporary t.Logf. Look for camera, location or device fields, which web-app uploads of media can carry. The captured files should be random test data anyway.

Verification (fix round 1)

All with RCLONE_CONFIG=/notfound, fake only:

• go build ./... ok; go vet ./backend/protondrive/... ok; golangci-lint run ./backend/protondrive/... 0 issues
• go test -count=1 -run TestGolden -v ./backend/protondrive/: Capture SKIP, Fixtures SKIP (no fixtures), CaptureFlags PASS, RoundTrip PASS (with tampered/block, tampered/manifest_signature, tampered/name), RefusesForeignKeys PASS
• go test -race -count=1 ./backend/protondrive/...: ok protondrive 309.382s, ok protontest 36.647s, ok protonfake 1.359s

Commit: 1b2fcbd21 protondrive: make the golden capture fail closed on foreign keys