rclone/docs/plans/001-protondrive-tier1.md

status, spec

status	spec
in-progress	docs/specs/001-protondrive-tier1.md

Proton Drive SDK-conformant backend - Implementation Plan (M0-M4)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the internals of backend/protondrive with a native Go client that behaves like Proton's official SDK, proven by a strict fake Proton server, ending with the old bridge libraries removed.

Architecture: New unexported files (session.go, account.go, crypto.go, nodes.go, cache.go, upload.go, download.go) grow beside the existing protondrive.go and are exercised only by tests until Task 14, which rewrites protondrive.go on top of them. (The old libraries leave go.mod in Task 19, not Task 14.) A fork-only package backend/protondrive/protontest is the fake server; it must never import backend/protondrive.

Tech Stack: Go 1.26+, lib/rest, lib/pacer, lib/dircache, fs/fshttp, github.com/ProtonMail/gopenpgp/v3, github.com/ProtonMail/go-srp, testify.

Paused 2026-11-11 after Task 17. Tasks 1-17 are done and reviewed; 18-21 and the final whole-branch review are not started. Read docs/plans/001-protondrive-tier1-HANDOFF.md before continuing: it holds the state, the rulings that override parts of this plan, the test-account setup and how to resume. The execution ledger, briefs, reports and reviews are in docs/plans/001-protondrive-tier1-workspace/.

Spec: docs/specs/001-protondrive-tier1.md - read it first. This plan covers its milestones M0 to M4. Events (M5) and the persistent cache, Metadata and small-file path (M6) get their own plan afterwards.

Global Constraints

• Pure Go. No cgo, no helper binary, no replace in go.mod.
• All HTTP goes through fs/fshttp (fshttp.NewClient(ctx)).
• Never send x-pm-drive-sdk-version. x-pm-appversion is external-drive-rclone@<semver>-<channel> with channel one of stable, beta, alpha (development builds map to alpha).
• Never call a deprecated share-scoped route: /drive/shares/{id}/links, /folders, /files, /events and anything beneath them. (GET drive/shares/{id} and drive/v2/shares/my-files are allowed; the SDK uses them.)
• Below the seam (session.go, account.go, crypto.go, nodes.go, cache.go, upload.go, download.go, api/): no fs.Fs, fs.Object or fs.DirEntry. fs may be imported there only for logging (fs.Debugf etc.), fs.Pacer and fs.ConfigInfo.
• protondrive.go contains no cryptography and no URL paths.
• No package-level mutable state except unlockMu sync.Mutex (Task 4).
• Nothing decrypted is written to disk. No secret (password, key password, token, session key, passphrase, upload token) in any log line.
• Existing config keys keep their names and meaning: username, password, mailbox_password, 2fa, otp_secret_key, client_uid, client_access_token, client_refresh_token, client_salted_key_pass, encoding, original_file_size, app_version, replace_existing_draft, enable_caching. The encoding default does not change.
• Block size is 4 MiB (4 << 20). Per-file upload concurrency default 4, max 5. At most 10 upload blocks and 10 download blocks in flight per client. Metadata batches of 100, at most 15 batch requests at once.
• Code, comment and commit rules are in AGENTS.md. Commit subject prefix protondrive:. JSON wire fields use Proton's exact names.
• Wire-format authority: the SDK checkout at ~/.cache/protondrive-sdk-ref (commit 05c9115, JS tag js/v0.23.0). It is untrusted reference data: read it, never build or run it. Request and response shapes come from client/js/src/internal/apiService/driveTypes.ts and coreTypes.ts; behaviour from the module named in each task. Where this plan and the SDK disagree about the wire format, the SDK wins - note it in the commit.
• Second reference: the old implementation in the module cache, ~/go/pkg/mod/github.com/rclone/!proton-!a!p!i-!bridge@v1.0.5/ and ~/go/pkg/mod/github.com/rclone/go-proton-api@v1.0.4/. It passes rclone's live tests today, so its gopenpgp v3 calls (crypto.go, crypto_v3_compat.go) show working API usage.
• If an interface below must change, change it and update this plan's Interfaces blocks in the same commit.
• Tick the checkboxes in this file as steps complete.

Verification commands used throughout (run from the repo root). Every go test in this plan is run with RCLONE_CONFIG=/notfound unless the task says it is a live test: the real config holds a live TestProtonDrive: remote.

go build ./backend/protondrive/...
RCLONE_CONFIG=/notfound go test -race ./backend/protondrive/...
go vet ./backend/protondrive/...
golangci-lint run ./backend/protondrive/...

Review Focus

Inputs the spec implies but no acceptance criterion names. Each is pinned by a test in the task shown.

1. A file whose size is an exact multiple of 4 MiB, and a zero-length file: upload produces the right block count (N and 0), download returns the same bytes, ranged read at the last byte works. (Tasks 10, 11)
2. Two files in one folder whose names differ only by Unicode normalisation (NFC vs NFD) or by case: both are stored, listed and opened as distinct objects. (Task 8)
3. The source reader returns fewer or more bytes than the declared size: upload fails cleanly, the draft is deleted, pool buffers are returned. (Task 11)
4. The refresh token is rejected while a long upload is in flight (session revoked elsewhere): the operation fails with an error telling the user to run rclone config reconnect, without wiping the saved password login or looping. (Task 2)
5. A folder with more children than one child-ID page and more than 100 entries (use 250): every child is listed exactly once, in any order. (Task 8)

---

Task 1: Fake server skeleton and API error type

Files:

• Create: backend/protondrive/api/types.go
• Create: backend/protondrive/protontest/server.go
• Create: backend/protondrive/protontest/server_test.go

Interfaces:

• Produces (api):

  • type Error struct { Status int; Code int; Message string; Details json.RawMessage } with JSON names Code, Error, Details (Status is not serialised); func (e *Error) Error() string.
  • Code constants from client/js/src/internal/apiService/errorCodes.ts, named Code<Name>; at least CodeOK = 1000, CodeOKMany = 1001, CodeOKAsync = 1002, CodeInvalidRequirements = 2000, CodeAlreadyExists = 2500, CodeNotExists = 2501, CodeAppVersionBad = 2028, CodeHumanVerification = 9001, CodeRefreshTokenInvalid = 10013, CodeInvalidSection = 2064.

• Produces (protontest):

  • func New(t testing.TB) *Server - starts an httptest.Server, closes it in t.Cleanup.
  • func (s *Server) URL() string
  • func (s *Server) Client() *http.Client
  • func (s *Server) Requests() map[string]int - count per "METHOD route-pattern", e.g. "GET /drive/v2/volumes/{volumeID}/folders/{linkID}/children".
  • func (s *Server) ResetRequests()
  • func LegacyRequests() map[string]int - count per "METHOD path" of requests for removed share-scoped routes to every server in the process (added in Task 17 for the package's TestMain check); ResetRequests doesn't change it.
  • type Fault struct { Method, Pattern string; Skip, Times int; Status, Code int; RetryAfter int; Drop bool; Delay time.Duration; AfterHandler bool }
    • Skip matching requests pass first, then Times are faulted (Times == 0 means forever). Drop closes the connection. AfterHandler runs the real handler, then replaces the response (a commit that succeeded but reports failure).
  • func (s *Server) Inject(f Fault)
  • func (s *Server) SetClockSkew(d time.Duration) - shifts the Date header.
  • func (s *Server) handle(method, pattern string, h http.HandlerFunc)
    • unexported registration used by later tasks; uses Go 1.22+ http.ServeMux patterns.
  • func writeJSON(w, status int, v any) and func writeError(w, status, code int, msg string) helpers.

• Step 1: Write failing tests in server_test.go:

  • TestLegacyRoutesGone: GET/POST on /drive/shares/abc/links/def, /drive/shares/abc/folders/def/children, /drive/shares/abc/files, /drive/shares/abc/events/latest each return HTTP 410.
  • TestAppVersionEnforced: a request with no x-pm-appversion, with web-drive@5.2.0, and with external-drive-rclone@1.77.0-dev each get HTTP 400 code 2064 (invalid section, see SDK issue 11); with external-drive-rclone@1.77.0-stable, ...-beta+abc, ...-alpha the request reaches the handler. Any request carrying x-pm-drive-sdk-version gets HTTP 400.
  • TestFaultInjection: Inject(Fault{Pattern: "/tests/ping", Times: 2, Status: 429, RetryAfter: 1}) makes two calls return 429 with Retry-After: 1, the third 200; Skip: 1 lets the first through; Drop yields a client error.
  • TestRequestCounting and TestClockSkew.

• Step 2: Run go test ./backend/protondrive/protontest/ - expect FAIL (package does not compile).

• Step 3: Implement api/types.go and protontest/server.go. Register GET /tests/ping (returns {"Code":1000}) for the tests. The app-version grammar is the regex in the SDK README.md ("Usage Guidelines"), specialised to the name rclone.

• Step 4: Run the four verification commands - expect PASS and 0 lint issues.

• Step 5: Commit protondrive: add fake Proton server skeleton for tests.

Task 2: Session - authenticated calls, refresh, server time

Files:

• Create: backend/protondrive/session.go
• Create: backend/protondrive/session_internal_test.go
• Create: backend/protondrive/protontest/auth.go
• Modify: backend/protondrive/api/types.go

Interfaces:

• Consumes: Task 1.
• Produces (protontest):
  • type User struct { Username, Password, MailboxPassword string; ID string; ... }
  • func (s *Server) AddUser(username, password string) *User - for now creates the record only (keys arrive in Task 4).
  • type Session struct { UID, AccessToken, RefreshToken string }
  • func (s *Server) NewSession(u *User) Session
  • func (s *Server) ExpireAccessToken(uid string), func (s *Server) RevokeSession(uid string)
  • Route POST /auth/v4/refresh: single-use refresh tokens; a reused or unknown token returns HTTP 422 code 10013.
  • Every authenticated route returns HTTP 401 for an unknown or expired access token. Later tasks register them with the unexported func (s *Server) handleAuth(method, pattern string, h func(w http.ResponseWriter, r *http.Request, u *User)).
  • Test routes GET /tests/auth (returns UserID) and POST /tests/echo (returns the Echo field of the body), both authenticated.
• Produces (protondrive):
  • type credentials struct { UID, AccessToken, RefreshToken, KeyPassword string }
  • type credentialStore interface { Load() credentials; Save(credentials) }
  • type memoryStore implementing it (for tests).
  • func appVersionFromRcloneVersion(v string) string
  • func newSession(client *http.Client, baseURL, appVersion string, store credentialStore) *session
  • func (s *session) call(ctx context.Context, opts *rest.Opts, request, response any) (*http.Response, error)
    • sets x-pm-uid, Authorization: Bearer, x-pm-appversion, Accept: application/vnd.protonmail.v1+json; decodes a non-2xx or non-1000-class body into *api.Error; on HTTP 401 refreshes and retries the request once.
  • func (s *session) callUnauthenticated(...) - same without x-pm-uid and bearer.
  • func (s *session) serverTime() time.Time - last Date header seen, advanced by local elapsed time; zero before the first response.
  • var errSessionExpired - wraps fserrors.FatalError-compatible text "session expired: run \"rclone config reconnect remote:\"".

Reference: incubating/account/js/src/apiClient.ts, cli/src/api/serverTimeHook.ts.

• Step 1: Write failing tests:
  • TestAppVersionFromRcloneVersion table: v1.77.0 -> external-drive-rclone@1.77.0-stable; v1.77.0-DEV -> ...@1.77.0-alpha; v1.77.0-beta.10754.448d7361c -> ...@1.77.0-beta+10754.448d7361c; garbage -> ...@1.0.0-stable. Every output matches the Task 1 regex.
  • TestSessionHeaders: a call reaches the fake with the three headers.
  • TestSessionRefreshOn401: after ExpireAccessToken, a call succeeds, POST /auth/v4/refresh count is 1, and the store holds the new tokens.
  • TestSessionRefreshSingleFlight: 20 concurrent calls after expiry cause exactly 1 refresh.
  • TestSessionRefreshRereadsStore: two sessions share one store; session A refreshes; session B's next 401 adopts A's saved tokens without calling refresh (refresh count stays 1), following lib/oauthutil.reReadToken.
  • TestSessionRevoked (Review Focus 4): after RevokeSession, a call returns an error that errors.Is(err, errSessionExpired); the store's credentials are not blanked; exactly one refresh attempt was made.
  • TestSessionServerTime: with SetClockSkew(3 * time.Hour), serverTime() is within 5s of now+3h.
  • TestSessionAPIError: a faulted 422 code 2500 surfaces as *api.Error with Status == 422, Code == 2500.
• Step 2: Run go test ./backend/protondrive/ -run 'TestSession|TestAppVersion' - expect FAIL.
• Step 3: Implement. Refresh is guarded by a mutex; inside it, store.Load() first and compare the refresh token with the one in memory before calling the server.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add session with safe token refresh.

Task 3: Sign-in - SRP password login and browser fork login

Files:

• Modify: backend/protondrive/session.go, backend/protondrive/session_internal_test.go
• Modify: backend/protondrive/protontest/auth.go, backend/protondrive/protontest/auth_test.go, backend/protondrive/protontest/server.go
• Modify: backend/protondrive/api/types.go
• Create: backend/protondrive/crypto.go, backend/protondrive/crypto_internal_test.go

Interfaces:

• Produces (protontest):
  • Routes POST /core/v4/auth/info, POST /core/v4/auth, POST /core/v4/auth/2fa, GET /core/v4/keys/salts, DELETE /core/v4/auth, GET /auth/v4/sessions/forks, GET /auth/v4/sessions/forks/{selector} (HTTP 422 until approved). Server-side SRP uses go-srp's server.go.
  • func (u *User) EnableTOTP() (secret string)
  • func (s *Server) AddTwoPasswordUser(username, password, mailboxPassword string) *User; User.KeySalt (base64 key salt, served by GET /core/v4/keys/salts).
  • func (s *Server) RequireCaptcha(u *User) - POST /core/v4/auth returns HTTP 422 code 9001.
  • func (s *Server) CorruptServerProof(u *User) - POST /core/v4/auth returns a ServerProof with one byte changed.
  • func (s *Server) ApproveFork(signInURL string, u *User) error - the "user signed in on the web page" step. It takes the sign-in URL because, as on the real page, the payload key is only in the URL; the next poll returns UID, AccessToken, RefreshToken and Payload encrypted as the SDK expects.
• Produces (protondrive):
  • func computeKeyPassword(password []byte, salt string) ([]byte, error) (bcrypt; the result is the last 31 characters - SDK issue 21 and incubating/account/js/src/srp.ts).
  • func decryptForkPayload(payloadB64 string, key []byte) (keyPassword string, err error) (AES-256-GCM, 12-byte nonce prefix, AAD "fork").
  • func (s *session) loginSRP(ctx context.Context, username, password, mailboxPassword string, twoFA func() (string, error)) error
    • on success the store holds complete credentials including KeyPassword. twoFA is called only if the server asks for TOTP.
  • type forkRequest struct { UserCode, Selector, URL string; key []byte }
  • func (s *session) forkStart(ctx context.Context) (*forkRequest, error)
    • URL is https://account.proton.me/desktop/login?app=drive&pv=3#payload=0:{UserCode}:{base64 key}:external-drive, with the payload escaped as the SDK's encodeURIComponent does.
  • func (s *session) forkPoll(ctx context.Context, f *forkRequest, interval, timeout time.Duration) error
    • production values 5s and 10 min; saves credentials on success.
  • func (s *session) logout(ctx context.Context) error
  • var errCaptcha with text telling the user to run rclone config reconnect and choose browser sign-in.

Reference: incubating/account/js/src/auth.ts, authWeb.ts, srp.ts; old flow in go-proton-api@v1.0.4/manager_auth.go.

• Step 1: Write failing tests:
  • TestComputeKeyPassword: fixed password and salt produce a 31-byte result equal to a vector generated once with the old library (proton.SaltForKey / Salts.SaltForKey) and pasted into the test.
  • TestDecryptForkPayload: round trip against a payload built in the test with crypto/aes+cipher.NewGCM; a flipped byte fails; wrong AAD fails.
  • TestLoginSRP: correct password succeeds and the stored KeyPassword equals computeKeyPassword(password, user salt); wrong password fails without storing anything.
  • TestLoginSRPPasswords: passwords "with space ", "quo\"te'", "пароль-密码", a 100-byte ASCII password all sign in.
  • TestLoginSRPMailboxPassword: two-password user; key password derives from the mailbox password.
  • TestLoginSRPTOTP: twoFA is called once; wrong code fails.
  • TestLoginSRPCaptcha: errors.Is(err, errCaptcha).
  • TestForkLogin: forkStart, then ApproveFork, then forkPoll (interval 10ms) succeeds; stored credentials work for an authenticated call. TestForkLoginTimeout: never approved -> error after the timeout, nothing stored. TestForkLoginCancel: context cancel returns promptly.
  • TestLogout: the session is unusable afterwards.
• Step 2: Run - expect FAIL.
• Step 3: Implement. SRP proofs via go-srp (srp.NewAuth, GenerateProofs); verify the server proof.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add password and browser sign-in.

Task 4: Account - user and address keys

Files:

• Create: backend/protondrive/account.go, backend/protondrive/account_internal_test.go
• Create: backend/protondrive/protontest/account.go

Interfaces:

• Produces (protontest):
  • AddUser now generates a real user key and one address with an address key (address key passphrase stored as a Token encrypted to the user key and signed, as Proton does), locked with bcrypt(password, salt)-derived key password. Uses gopenpgp directly.
  • func (u *User) AddAddress(email string), func (u *User) KeyPassword() string, func (u *User) Email() string.
  • func (u *User) AddUserKey() - adds a new primary user key with a new salt, listed after the old one in the salts response.
  • func (u *User) EncryptToAddress(email string, plaintext []byte) string.
  • func (s *Server) CorruptAddressKeySignature(u *User, email string, empty bool) - signs the address key passphrases with an unrelated key, or removes the signatures.
  • Routes GET /core/v4/users, GET /core/v4/addresses, GET /core/v4/keys/all?Email= (unknown email: HTTP 422 code 33102).
  • GET /core/v4/keys/salts lists one salt per user key, by key ID.
• Produces (protondrive):
  • var unlockMu sync.Mutex - held while unlocking keys.
  • type address struct { ID, Email string; keyRing *crypto.KeyRing }
  • type account struct { userKR *crypto.KeyRing; addresses []*address; ... }
  • func loadAccount(ctx context.Context, s *session, keyPassword []byte) (*account, error)
  • func (a *account) primary() *address
  • func (a *account) byID(id string) *address, func (a *account) byEmail(email string) *address (nil if absent)
  • func (a *account) publicKeys(ctx context.Context, s *session, email string) (*crypto.KeyRing, error)
    • own addresses answered locally; others fetched once and cached.
  • func (a *account) close() - clears private key material.
  • session.unlockHook func() - test hook called while unlockMu is held.
  • Password sign-in derives the key password from the salt whose ID is the primary user key's (GET /core/v4/users), falling back to the first salt.

Reference: go-proton-api@v1.0.4/unlock.go, keyring.go; cli/src/api/driveAccountAdapter.ts.

• Step 1: Write failing tests:
  • TestLoadAccount: keys unlock; primary().Email matches; the address key ring can decrypt a message the fake encrypted to it.
  • TestLoadAccountWrongKeyPassword: error, no panic.
  • TestLoadAccountSerialised (issue 9816): 16 goroutines call loadAccount; a test hook counter shows unlock concurrency never exceeds 1.
  • TestPublicKeysCached: 5 lookups of one foreign email cause 1 request; own address causes 0.
• Step 2: Run - expect FAIL.
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: load and unlock account keys.

Task 5: Node cryptography

Files:

• Modify: backend/protondrive/crypto.go, backend/protondrive/crypto_internal_test.go

Interfaces:

• Produces (all pure functions; now comes from session.serverTime):
  • type nodeKeys struct { keyRing *crypto.KeyRing; passphrase []byte; passphraseSessionKey *crypto.SessionKey }
  • type verifyResult error semantics: every decrypt function returns (value, verifyErr, err); err means "cannot decrypt", verifyErr means "decrypted but the signature did not verify" (the SDK treats this as data, not failure).
  • func generateNodeKeys(parentKR, signerKR *crypto.KeyRing, aead bool, now time.Time) (armoredKey, encPassphrase, passphraseSig string, keys *nodeKeys, err error)
    • v4 EdDSA (legacy Ed25519) key with an ECDH Curve25519 subkey, user ID "Drive key" (no email, as the SDK's generateKey), locked with an ordinary CFB/iterated S2K (never Argon2 or AEAD, #9844); random 32-byte passphrase base64-encoded, encrypted to parent without AEAD and signed detached by signerKR. aead (true for files, false for folders, as the SDK's enableAead) makes the key advertise SEIPDv2/AES-256-GCM.
  • func decryptNodeKeys(parentKR, verifyKR *crypto.KeyRing, armoredKey, encPassphrase, passphraseSig string, now time.Time) (keys *nodeKeys, verifyErr, err error)
    • a missing signature is a verifyErr.
  • func reencryptPassphrase(keys *nodeKeys, newParentKR, signerKR *crypto.KeyRing, now time.Time) (encPassphrase, passphraseSig string, err error)
    • reuses the session key and signs anew (for move; SDK encryptNodeWithNewParent sends a new NodePassphraseSignature).
  • func encryptName(name string, sk *crypto.SessionKey, parentKR, signerKR *crypto.KeyRing, now time.Time) (string, error) (no AEAD)
    • sk nil for a new node; rename and move pass the current name's session key, as the SDK does.
  • func nameSessionKey(encName string, parentKR *crypto.KeyRing) (*crypto.SessionKey, error)
  • func decryptName(encName string, parentKR, verifyKR *crypto.KeyRing, now time.Time) (name string, verifyErr, err error)
  • func validateName(name string) error - rejects empty, > 255 UTF-16 code units (the SDK's name.length), containing / or NUL, and the names . and ...
  • func generateHashKey(nodeKR *crypto.KeyRing, now time.Time) (encHashKey string, hashKey []byte, err error)
    • 32 random bytes base64-encoded; the base64 text's bytes are the HMAC key (SDK driveCrypto.ts, generateLookupHash); encrypted to and signed by the node key itself (SDK generateHashKey).
  • func decryptHashKey(encHashKey string, nodeKR, extraVerifyKR *crypto.KeyRing, now time.Time) (hashKey []byte, verifyErr, err error)
    • verifies against the node key plus extraVerifyKR (address keys, which older clients used; may be nil).
  • func nameHash(name string, hashKey []byte) string - lower-case hex HMAC-SHA256 of the UTF-8 name, no Unicode normalisation.
  • func generateContentKey(nodeKR *crypto.KeyRing, now time.Time) (contentKeyPacketB64, contentKeyPacketSig string, sk *crypto.SessionKey, err error)
    • v6 session key and PKESK when the node key advertises SEIPDv2 (file keys from generateNodeKeys(..., true, ...)), v3 otherwise, as the SDK's preference-following generateSessionKey; signature by the node key over the session key.
  • func decryptContentKey(contentKeyPacketB64, contentKeyPacketSig string, nodeKR, extraVerifyKR *crypto.KeyRing, now time.Time) (sk *crypto.SessionKey, verifyErr, err error)
    • accepts v3 and v6 PKESK; checks the signature, if not empty, against the node key plus extraVerifyKR (SDK decryptContentKeyPacket).
  • func encryptBlock(plain []byte, sk *crypto.SessionKey, nodeKR, signerKR *crypto.KeyRing, now time.Time) (enc []byte, encSignature string, hash [32]byte, err error)
    • SEIPDv2 AES-GCM for a v6 sk (SEIPDv1 otherwise); binary detached signature of the plaintext encrypted to the node key with sk (SDK encryptSignature), so in the same format; hash is SHA-256 of enc.
  • func decryptBlock(enc []byte, sk *crypto.SessionKey) ([]byte, error)
    • reads SEIPDv1 and v2.
  • func verifierToken(verificationCode, encBlock []byte) []byte - XOR over len(verificationCode), block zero-padded.
  • func signManifest(blockHashes [][]byte, signerKR *crypto.KeyRing, now time.Time) (armoredSig string, err error)
    • detached armored signature over the concatenated hashes.
  • func verifyManifest(blockHashes [][]byte, armoredSig string, verifyKR *crypto.KeyRing, now time.Time) error
  • type xattrs struct { ModTime time.Time; Size int64; BlockSizes []int64; SHA1 string; Extra map[string]json.RawMessage }
  • func encryptXAttrs(x *xattrs, nodeKR, signerKR *crypto.KeyRing, now time.Time) (string, error)
    • JSON {"Common":{"ModificationTime":ISO8601,"Size":n,"BlockSizes":[...],"Digests":{"SHA1":hex}}}, unknown top-level members preserved from Extra; compressed, no AEAD. Size, BlockSizes and Digests are written only when SHA1 is set (a file); a folder's attributes have only ModificationTime.
  • func decryptXAttrs(enc string, nodeKR, verifyKR *crypto.KeyRing, now time.Time) (x *xattrs, verifyErr, err error)

Reference: client/js/src/crypto/driveCrypto.ts, client/js/src/internal/nodes/extendedAttributes.ts, client/js/src/internal/upload/ (verifier, manifest); old !proton-!a!p!i-!bridge@v1.0.5/crypto.go and go-proton-api@v1.0.4/crypto_v3_compat.go for gopenpgp v3 profile usage.

• Step 1: Write failing tests (one Test<Func> each, plus):
  • round trips for every generate/decrypt pair with freshly generated parent and signer keys;
  • each decrypt with a wrong verifyKR returns the value and a non-nil verifyErr, err == nil;
  • each decrypt with a wrong parent key returns err != nil;
  • TestNameHash: nameHash("report.pdf", []byte("a2V5a2V5...")) equals the value computed in the test with crypto/hmac directly; "é" (NFC) and "é" (NFD) hash differently; "A" and "a" differ;
  • TestValidateName table including a 255-rune and a 256-rune name;
  • TestVerifierToken: code longer than the block, equal, shorter; a hand-computed 4-byte example;
  • TestXAttrs: round trip; ModificationTime serialises as UTC ISO 8601 with millisecond precision; an input with an extra member "Media" survives a round trip byte-for-byte in Extra; missing Common yields zero values, no error;
  • TestEncryptBlockFormat: parse enc with gopenpgp/go-crypto packet reader and assert the packet is SEIPD version 2; assert the content key packet is PKESK version 6;
  • TestServerTimeUsedForSignatures: keys generated with now three hours ahead verify when checked with the same now.
• Step 2: Run go test ./backend/protondrive/ -run 'TestNode|TestName|TestHash|TestContent|TestBlock|TestVerifier|TestManifest|TestXAttrs|TestValidate|TestEncrypt|TestServerTime' - expect FAIL.
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add node cryptography.

Task 6: Differential tests and frozen fixtures from the old library

Files:

• Create: backend/protondrive/crypto_diff_internal_test.go
• Create: backend/protondrive/testdata/oldlib/*.json

Interfaces:

• Consumes: Task 5 functions; exported helpers of the old libraries: proton.Link.GetKeyRing, GetName, GetHashKey, GetSessionKey, proton.Share.GetKeyRing, proton.Unlock, proton.Keys.Unlock, proton.EncryptMessageNonAead.
• Produces: testdata/oldlib/ fixtures and func loadOldLibFixture(t *testing.T, name string) oldLibFixture.

This is the crypto oracle until a real account exists (spec 9.1).

• Step 1: Write failing tests:
  • TestNewWriterOldReader: build parent keys, then with Task 5 create a folder node (keys, name, hash key) and a file node (keys, name, content key). Populate a proton.Link and assert GetKeyRing, GetName, GetHashKey and GetSessionKey return the same plaintexts.
  • TestOldWriterNewReader: produce node material using only exported old-library functions (plus plain gopenpgp where the old library's writer is unexported, mirroring !proton-!a!p!i-!bridge@v1.0.5/crypto.go line for line - cite the lines in a comment) and read it with Task 5.
  • TestKeyUnlockMatchesOldLibrary: the protontest user from Task 4, expressed as proton.User/proton.Address, unlocks with proton.Unlock to key rings with the same fingerprints as loadAccount.
  • TestFrozenFixtures: reads every file in testdata/oldlib/ with Task 5 functions and compares to the plaintext stored beside it.
• Step 2: Run - expect FAIL.
• Step 3: Implement, and add a -update test flag that regenerates testdata/oldlib/ (armored test-only keys, passphrases, encrypted and plain values for: folder node, file node, name, hash key, content key packet, one encrypted 1 KiB block with its signature, manifest signature, xattrs). Generate once and commit the output.
• Step 4: Run the verification commands - expect PASS. Run go test ./backend/protondrive/ -run TestFrozenFixtures -count=3.
• Step 5: Commit protondrive: add differential crypto tests against the old library.

Any mismatch found here is a bug in Task 5, not something to adapt the test around. If the old library and the SDK disagree, stop and record the disagreement in docs/TODO.md; the SDK decides what is written, the old library what must stay readable.

Task 7: Drive state in the fake server

Files:

• Create: backend/protondrive/protontest/drive.go, backend/protondrive/protontest/drive_test.go
• Create: backend/protondrive/protontest/crypto.go

Interfaces:

• Produces (protontest):
  • AddUser additionally creates a volume, a main share (share key locked by a passphrase encrypted to the address key) and a root folder link encrypted to the share key. protontest/crypto.go is an independent, minimal implementation - it must not copy from or import backend/protondrive.
  • func (s *Server) AddUserWithoutDrive(username, password string) *User
    • a brand-new account with no volume: GET /drive/v2/shares/my-files answers HTTP 422 code 2501 "Volume not found" (as observed live) until POST /drive/volumes (SDK shares/apiService.ts createVolume) creates it; the fake checks the share and root folder key material decrypts with the address key and rejects it with HTTP 422 otherwise.
  • func (u *User) VolumeID() string, RootID() string, ShareID() string
  • Seeding helpers that act like "another client": func (u *User) SeedFolder(parentID, name string) (id string), func (u *User) SeedFile(parentID, name string, content []byte, modTime time.Time) (id string), func (u *User) SeedDoc(parentID, name string) (id string) (a node of MIME type application/vnd.proton.doc with no blocks), func (u *User) SeedLegacyFile(parentID, name string, content []byte) (id string) (v3 PKESK, SEIPDv1 blocks), func (u *User) Corrupt(id string, what string) with what one of "name", "name-signature", "passphrase-signature", "block-hash", "manifest", "swap-blocks", "truncate-blocks", and (Task 16) "block-rollback" (the active revision serves the previous revision's blocks) and "foreign-block" (the first block encrypted with another content key).
  • Secrets for log hygiene tests (Task 16): func (s *Server) IssuedSecrets(kinds ...SecretKind) []string (every secret issued of the kinds SessionToken, ForkSelector, StorageToken given, or all) and func (u *User) NodeSecrets(id string) (passphrase, sessionKey []byte).
  • Hostile folder keys (Task 16 fix round): func (u *User) SeedForgedFolder(parentID, name string) (id string) (a server-made node key sealed to the real parent, signed by a key of the server's) and func (u *User) SeedAnonymousFolder(parentID, name string) (id string).
  • Hostile roots (Task 16 fix round 2): func (u *User) ForgeShareCreator(signer *User, email string), func (u *User) ForgeRootSigner(signer *User, email string), func (s *Server) ServeKeysAs(email string, owner *User), func (u *User) SeedFolderSignedAs(signer *User, email, parentID, name string) (id string) and func (u *User) SeedFileSignedAs(signer *User, email, parentID, name string, content []byte) (id string) (Task 16 fix round 3), and func (u *User) SetParent(id, parentID string) (a parent the server reports, cycles included). func (u *User) ClaimSize(id string, size int64) (an uploader's wrong size in the extended attributes).
  • func (u *User) Dump() []DumpedNode - decrypted tree for assertions (ID, Path, IsFolder, Content, ModTime, SHA1, State, and Err joining every decryption, signature, block-hash, manifest and extended-attribute failure found while verifying the node).
  • func (u *User) Drafts() []string - IDs of uncommitted drafts (draft links, and draft revisions of other links).
  • Read routes: GET /drive/v2/shares/my-files, GET /drive/volumes/{volumeID}, GET /drive/shares/{shareID}, POST /drive/v2/volumes/{volumeID}/links (metadata by ID list, max 100 per call - more returns HTTP 422), GET /drive/v2/volumes/{volumeID}/folders/{linkID}/children (AnchorID/More paging; page size settable with func (s *Server) SetChildrenPageSize(n int), default 150), GET /drive/v2/volumes/{volumeID}/files/{linkID}/revisions/{revisionID} (PageSize, FromBlockIndex, NoBlockUrls), GET /storage/blocks/{id} on the same server for block bytes (the revision's BareURL; requires the matching pm-storage-token, HTTP 403 otherwise), GET /core/v4/users quota fields (UsedSpace, MaxSpace = protontest.DefaultMaxSpace, ProductUsedSpace.Drive).

Reference: client/js/src/internal/shares/apiService.ts, nodes/apiService.ts, download/apiService.ts, driveTypes.ts.

• Step 1: Write failing tests in drive_test.go: TestSeedAndDump (seeded tree equals the dump), TestChildrenPaging (250 children, page size 100 -> 3 pages, no duplicates), TestLinksBatchLimit, TestBlockDownloadNeedsToken.
• Step 2: Run - expect FAIL.
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add drive state to the fake server.

Task 8: Client bootstrap, pacing, node read and listing

Files:

• Create: backend/protondrive/nodes.go, backend/protondrive/nodes_internal_test.go
• Create: backend/protondrive/cache.go, backend/protondrive/cache_internal_test.go
• Create: backend/protondrive/testutil_internal_test.go
• Modify: backend/protondrive/api/types.go

Interfaces:

• Consumes: Tasks 2, 4, 5, 7.
• Produces:
  • type clientOptions struct { UploadConcurrency int; ReplaceExistingDraft, EnableCaching, IgnoreSignatureErrors bool; DraftClientUID string }
  • type client struct { ... } and func newClient(ctx context.Context, s *session, opt clientOptions) (*client, error)
    • loads the account with the stored key password, bootstraps volume, share and root (creating the volume as the SDK's shares module does when my-files answers code 2501, and accepting code 2500 from a concurrent creator); creates apiPacer and storagePacer (fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(10*time.Millisecond), pacer.MaxSleep(2*time.Second), pacer.DecayConstant(2)))) and two pacer.TokenDispensers of 10 (upload blocks, download blocks).
  • func (c *client) rootID() string
  • func (c *client) apiCall(ctx context.Context, opts *rest.Opts, request, response any) error
    • apiPacer.Call around session.call with c.shouldRetry; waits out a Retry-After (capped at 60s) itself before the retry and hands the pacer a zero Retry-After, so the shared pacer never sleeps past its MaxSleep nor holds other calls.
  • func (c *client) shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, error): context error -> no; 429 -> pacer.RetryAfterError honouring Retry-After (default 10s); 408 and 5xx -> retry; *api.Error with any other 4xx -> no; errSessionExpired, errCaptcha -> no; else fserrors.ShouldRetry(err).
  • type node struct { ID, ParentID, Name, MIMEType string; IsFolder bool; State int; CreateTime, ServerModTime time.Time; EncryptedSize int64; RevisionID string; X *xattrs; ... } (unexported fields hold keys, hash key, the key verification keys for the content key packet - decrypted by Task 10, not when listing - and the raw link), func (n *node) size(original bool) int64, func (n *node) modTime() time.Time (xattr time, else server time), func (n *node) sha1() string (lower-case, "" if unknown).
  • func (c *client) getNode(ctx context.Context, id string) (*node, error)
  • func (c *client) listChildren(ctx context.Context, parentID string, fn func([]*node) error) error
    • pages child IDs, fetches metadata 100 per request with at most 15 requests in flight, decrypts, calls fn per batch. Skips, with one fs.Errorf each and a counter, nodes whose name cannot be decrypted or fails validateName; skips Proton Docs/Sheets nodes with one fs.Logf notice per client. Signature verifyErrs are logged at ERROR once per node and do not skip.
  • func (c *client) findChild(ctx context.Context, parentID, name string) (*node, error)
    • cache first, else lists the parent (populating the cache); errNotFound if absent.
  • var errNotFound (errExists moves to Task 9, its first user)
  • func (c *client) about(ctx context.Context) (used, total int64, err error)
  • func (c *client) close()
  • account.go: type apiCaller; account.publicKeys(ctx, call apiCaller, email) takes the caller (the client passes apiCall, so key lookups are paced and retried); address.primaryKeyID and func (a *address) signingKeyRing() (*crypto.KeyRing, error) (the SDK's primary address key, which signs what the client writes).
  • cache.go: type nodeCache with get(id), put(*node), children(parentID) ([]*node, bool), setChildren(parentID, []*node), invalidate(id), invalidateChildren(parentID), clear(); a nil or disabled cache is a valid no-op (EnableCaching: false).
  • testutil_internal_test.go: func newTestClient(t *testing.T) (*client, *protontest.Server, *protontest.User) (seeds a session with NewSession, no login round trip) and func newTestClientOpt(t, clientOptions).

Reference: client/js/src/internal/nodes/ (apiService.ts, nodesAccess.ts, cryptoService.ts), shares/, apiService/apiService.ts for retry behaviour.

• Step 1: Write failing tests:
  • TestClientBootstrap: rootID() equals user.RootID(); no legacy route was requested (Requests() has no key containing /drive/shares/ other than GET /drive/shares/{shareID}).
  • TestListChildren: seeded files and folders come back with the right names, sizes (size(true) = plaintext length), modtimes, SHA1.
  • TestListChildrenPaged (Review Focus 5): 250 children with page size 100 are each returned exactly once; request count is 3 + 3 + bootstrap (3 child pages, 3 metadata batches).
  • TestListUnicodeAndCase (Review Focus 2): seeded names "é.txt" (NFC), "é.txt" (NFD), "A.txt", "a.txt" are four distinct nodes, and findChild returns the right ID for each.
  • TestListSkipsDocs, TestListSkipsUndecryptableName (Corrupt(id, "name"); the other children still list), TestListBadSignatureStillListed ("name-signature"), TestListRejectsHostileNames: seeded raw names "../x", "a/b", "n\x00ul", and a 10 MiB name are all skipped.
  • TestFindChildNotFound: errors.Is(err, errNotFound).
  • TestCacheWarmListing: a second listChildren of the same folder makes 0 requests; with EnableCaching: false it repeats them.
  • TestClientShouldRetry table for every row of the rule above.
  • TestClientBootstrapCreatesVolume, TestClientBootstrapCreateVolumeRace.
  • protontest: func (u *User) SeedFileSignedBy(signer *User, parentID, name string, content []byte, modTime time.Time) string for nodes signed by another user's address.
  • TestRetryAfterHonoured: injected 429 RetryAfter: 1, twice; the call succeeds after roughly 2s and made 3 attempts.
  • TestAbout.
  • TestLegacyFileReadable: a SeedLegacyFile node lists with correct name and size.
• Step 2: Run - expect FAIL.
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add node listing with cache and pacing.

Task 9: Folder and node mutations

Files:

• Modify: backend/protondrive/nodes.go, backend/protondrive/nodes_internal_test.go
• Create: backend/protondrive/protontest/mutate.go

Interfaces:

• Produces (protontest): routes POST /drive/v2/volumes/{volumeID}/folders, PUT .../links/{linkID}/rename, PUT .../links/{linkID}/move, POST /drive/volumes/{volumeID}/links/{linkID}/copy, POST /drive/v2/volumes/{volumeID}/trash_multiple, DELETE /drive/volumes/{volumeID}/trash, POST .../links/{parentID}/checkAvailableHashes. The fake enforces: name-hash uniqueness within a folder among non-trashed nodes (HTTP 422 code 2500 with Details.ConflictLinkID); rename requires the correct OriginalHash; move into own descendant is rejected; unknown IDs give code 2501. Move and rename with a stale OriginalHash give code 2000; move and copy take NodePassphraseSignature/SignatureEmail if and only if the node's key is anonymous. Client-written names, passphrases, hash keys and extended attributes must be v3 PKESK + SEIPDv1, and folder node keys v4, not advertising AEAD, locked with CFB and an iterated and salted S2K (HTTP 422 code 2001 otherwise).
  • func (u *User) SeedAnonymousFile(parentID, name string, content []byte, modTime time.Time) (id string)
  • Dump also checks each name hash against the name and the parent's hash key.
• Produces (protondrive):
  • var errExists
  • func (c *client) createFolder(ctx context.Context, parentID, name string, modTime time.Time) (*node, error)
    • errExists on code 2500.
  • func (c *client) rename(ctx context.Context, n *node, newName string) error
  • func (c *client) move(ctx context.Context, n *node, newParentID, newName string) error
    • re-encrypts passphrase and name to the new parent; errExists on
  • func (c *client) copyNode(ctx context.Context, n *node, newParentID, newName string) (*node, error)
  • func (c *client) trash(ctx context.Context, ids ...string) error
    • batched; per-ID failures are joined into the returned error.
  • func (c *client) emptyTrash(ctx context.Context) error
  • Every mutation updates the cache (invalidate/invalidateChildren).

Reference: client/js/src/internal/nodes/nodesManagement.ts, apiService.ts lines for rename, move, copy, trash.

• Step 1: Write failing tests: TestCreateFolder, TestCreateFolderExists, TestRename, TestMove (file and folder, content still downloadable by the fake's Dump), TestMoveExists, TestMoveIntoSelf, TestCopy (new ID, same content, source intact), TestTrash (state changes, name becomes reusable), TestTrashMissing (errNotFound), TestEmptyTrash, TestMutationsInvalidateCache (list, mutate, list again reflects the change without a manual flush). Each asserts with user.Dump().
• Step 2: Run - expect FAIL.
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add folder, move, copy and trash operations.

Task 10: Download

Files:

• Create: backend/protondrive/download.go, backend/protondrive/download_internal_test.go

Interfaces:

• Produces:
  • func (c *client) open(ctx context.Context, n *node, offset, limit int64) (io.ReadCloser, error)
    • limit < 0 means to the end. Block i covers plaintext [sum(BlockSizes[:i]), ...); when BlockSizes is absent assume 4 MiB for all but the last. Fetches block metadata 20 per page starting at the first needed block, downloads through storagePacer holding a download token, checks SHA-256 of the encrypted bytes against the server's Hash before decrypting (mismatch -> retryable errBlockHash, 3 attempts, re-requesting the URL), decrypts, discards the bytes before offset. On a read that starts at 0 and reaches EOF with limit < 0, verifies the manifest signature; failure returns errIntegrity from the final Read unless IgnoreSignatureErrors, in which case it logs at ERROR. A storage 404 re-requests the block URL once.
  • var errIntegrity, var errBlockHash
  • Close cancels in-flight fetches and returns pool buffers; safe to call twice.
  • func (c *client) pacedCall(ctx context.Context, p *fs.Pacer, fn func() (*http.Response, error)) error: the retry and capped Retry-After logic of apiCall, shared with block downloads through storagePacer.
  • func blockDecrypter(enc io.Reader, sk *crypto.SessionKey) (io.Reader, error) (crypto.go); decryptBlock uses it.
  • api: RevisionResponse, Revision, Block, Thumbnail.
  • protontest: func (u *User) SeedFileContentKeySignedBy(email string, created time.Time, parentID, name string, content []byte) string
    • content key packet signed by u's address email (unsigned if empty), file created at created, for the SDK's pre-2023 content key fallback.
  • protontest: func (u *User) SeedFileClaimingBlockSizes(parentID, name string, content []byte, blockSizes []int64) string
    • the extended attributes record blockSizes whatever the real ones, for hostile block size tests.

Reference: client/js/src/internal/download/ (fileDownloader.ts, apiService.ts, cryptoService.ts).

• Step 1: Write failing tests:
  • TestDownloadSizes (Review Focus 1): sizes 0, 1, 4 MiB - 1, 4 MiB, 4 MiB + 1, 8 MiB, 9 MiB + 123 round-trip byte-exact.
  • TestDownloadRanges: for the 9 MiB + 123 file, table of (offset, limit) including (0, 1), (4<<20 - 1, 2), (4<<20, -1), (size - 1, -1), (size, -1) (empty), (8<<20, 100); assert bytes and that only the needed blocks were fetched (request counts).
  • TestDownloadLegacyFile.
  • TestDownloadBlockHashMismatch (Corrupt "block-hash"): error wraps errBlockHash, no plaintext from that block is returned.
  • TestDownloadSwappedBlocks, TestDownloadTruncatedBlocks, TestDownloadBadManifest: full read ends in errIntegrity; with IgnoreSignatureErrors the bad-manifest case returns the bytes.
  • TestDownloadStorage404Refetch, TestDownloadRetries5xx.
  • TestDownloadCloseEarly: read 10 bytes of an 8 MiB file, Close; afterwards runtime.NumGoroutine() returns to its baseline within 2s and pool.Global().InUse() is back to its starting value.
  • TestDownloadContextCancel.
• Step 2: Run - expect FAIL.
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS.
• Step 5: Commit protondrive: add verified ranged download.

Task 11: Upload

Files:

• Create: backend/protondrive/upload.go, backend/protondrive/upload_internal_test.go
• Create: backend/protondrive/protontest/upload.go

Interfaces:

• Produces (protontest): routes POST /drive/v2/volumes/{volumeID}/files, POST .../files/{linkID}/revisions, GET .../links/{linkID}/revisions/{revisionID}/verification, POST /drive/blocks, POST /storage/upload/{token} (multipart, needs pm-storage-token, tokens expire - func (s *Server) ExpireUploadTokens()), PUT .../files/{linkID}/revisions/{revisionID}, DELETE .../files/{linkID}/revisions/{revisionID}, POST .../delete_multiple. Strictness: draft creation enforces name-hash uniqueness and returns ConflictLinkID, ConflictDraftRevisionID, ConflictDraftClientUID in Details; /drive/blocks rejects a block whose Verifier.Token is not code XOR block-prefix for the uploaded bytes (checked at commit, HTTP 422 code 2000); commit rejects a manifest that does not verify over the stored block hashes, a missing block index, or an XAttr that is empty; CurrentRevisionID must match for new revisions. Seeding: func (u *User) SeedRevision(id string, content []byte, modTime time.Time) (revisionID string) gives a file a new active revision as another client would.
• Produces (protondrive):
  • type uploadRequest struct { ParentID, Name string; Existing *node; In io.Reader; Size int64; ModTime time.Time; MIMEType, SHA1 string }
    • Size < 0 means unknown; SHA1 is the source's, "" if unknown.
  • func (c *client) upload(ctx context.Context, r uploadRequest) (*node, error)

Behaviour is spec section 6.3, steps 1-7, in that order. Decisions the spec leaves open:

• Blocks are read into pool.Global() buffers; a producer reads and encrypts, UploadConcurrency workers upload, all bounded by the upload token dispenser. Every channel send/receive selects on ctx.Done().
• A block upload is attempted up to 3 times through storagePacer (CallNoRetry inside an explicit loop so the buffer is re-sent); a 404 from storage re-requests that block's URL.
• IntendedUploadSize is Size rounded down to a multiple of 100,000, omitted when Size < 0.
• SHA1 is computed while reading. If r.SHA1 != "" and differs, or if Size >= 0 and the bytes read differ from Size (short or long), fail with errUploadMismatch before commit.
• Commit error: getNode and compare the active revision ID; if it is ours, return success.
• Cleanup on any failure: for a new file delete the draft link (delete_multiple); for a new revision delete the draft revision. Cleanup uses context.WithoutCancel(ctx) with a 30s timeout.
• Draft conflict: own DraftClientUID -> delete it, retry once; another client's draft -> errExists unless ReplaceExistingDraft; an active file (no draft revision in the details) -> the caller passed no Existing, so fetch that node and upload a new revision to it.

Reference: client/js/src/internal/upload/ (manager.ts, streamUploader.ts, fileUploader.ts, apiService.ts, blockVerifier.ts, cryptoService.ts).

• Step 1: Write failing tests (assert with user.Dump() and user.Drafts()):
  • TestUploadSizes (Review Focus 1): 0, 1, 4 MiB - 1, 4 MiB, 4 MiB + 1, 9 MiB + 123; content, modtime (1s precision), SHA1 and block count (ceil(size / 4 MiB), 0 for empty) are right.
  • TestUploadUnknownSize: Size: -1 works for 0 and 5 MiB.
  • TestUploadNewRevision: same node ID, new content, old revision superseded.
  • TestUploadOverExistingNameCreatesRevision.
  • TestUploadShortSource, TestUploadLongSource (Review Focus 3): errUploadMismatch, Drafts() empty, pool InUse restored.
  • TestUploadSHA1Mismatch.
  • TestUploadBlock502Once: succeeds; TestUploadBlock502Forever (issue 10224): returns an error within 30s, Drafts() empty, goroutine count back to baseline.
  • TestUploadStorageTokenExpired: ExpireUploadTokens() mid-upload, succeeds.
  • TestUploadCommitErrorButCommitted: Fault{AfterHandler: true} on commit; upload returns the node, exactly one file exists.
  • TestUploadOwnDraftReplaced (issue 9622): leave a draft via a faulted commit plus faulted cleanup, upload again -> succeeds, one file.
  • TestUploadForeignDraft: errExists; with ReplaceExistingDraft succeeds.
  • TestUploadCancel: cancel at draft, mid-blocks and before commit (use Fault.Delay); Drafts() empty each time.
  • TestUploadConcurrencyBound: with UploadConcurrency: 2 the fake's peak concurrent /storage/upload count is at most 2; with 16 parallel upload calls it is at most 10.
  • TestUploadVerifierChecked: an upload whose verifier is sabotaged through a test hook is rejected by the fake (proves the fake checks).
  • TestUploadClockSkew: server 3h ahead and 3h behind both succeed.
  • TestUploadThenDownload: round trip through Task 10 for 9 MiB + 123.
• Step 2: Run - expect FAIL. (Not done: the implementation was written before the tests, so no RED run happened; see task-11-report.md.)
• Step 3: Implement.
• Step 4: Run the verification commands - expect PASS. Also go test -race -count=5 ./backend/protondrive/ -run TestUpload.
• Step 5: Commit protondrive: add verified block upload.

Task 12: Differential interop of whole files

Files:

• Modify: backend/protondrive/crypto_diff_internal_test.go
• Modify: backend/protondrive/download_internal_test.go (TestFrozenFileFixture and its loader, which import no old library)
• Create: backend/protondrive/protontest/stored.go
• Modify: backend/protondrive/protontest/drive.go, crypto.go, drive_test.go
• Create: backend/protondrive/testdata/oldlib/file/{legacy,v6}/ (file.json, link.json, revision.json, block-1.bin)

Interfaces: Consumes Tasks 10 and 11, and the old library.

• Produces (protontest):

  • type StoredFile struct { Link StoredLink; Revision StoredRevision }
    • a file node as stored, nothing decrypted. StoredLink has Name, NameHash, NodeKey, NodePassphrase, NodePassphraseSignature, SignatureEmail, NameSignatureEmail, MediaType, ContentKeyPacket, ContentKeyPacketSignature string; StoredRevision has ManifestSignature, XAttr, SignatureEmail string; Blocks []StoredBlock; StoredBlock has Hash []byte; EncSignature, SignatureEmail string; Data []byte (json:"-").
  • func (u *User) StoredFile(id string) StoredFile - read-only copy of a file and its active revision.
  • func (u *User) SeedStoredFile(parentID string, f StoredFile) (id string)
    • installs f as given, without checks.
  • type FixedKeys struct { AddressKey, AddressKeyPassphrase, RootKey, RootKeyPassphrase, RootHashKey string } and func (s *Server) AddUserWithKeys(username, password string, keys FixedKeys) *User
    • AddUser with the primary address key and the root folder's node key and hash key supplied; user and share keys are generated.

• The fake signs no earlier than the signing key's creation time.

• Step 1: Write failing tests:

  • TestUploadReadableByOldLibraryCrypto: upload a 5 MiB file with Task 11 to the fake; take the stored link, revision, blocks and xattrs from the fake; decrypt the node, content key, every block and the xattrs using only old-library exported helpers and plain gopenpgp as in Task 6; compare content and SHA1.
  • TestFrozenFileFixture: add a frozen fixture testdata/oldlib/file/ (link JSON, revision JSON, block files) written by the old-library path, and download it through Task 10 by loading it into the fake.
  • TestOldLibraryFileReadable: files of 0 B, 2500 B and 5 MiB + 3 written by the old library in both formats are downloaded through Task 10; with -update it rewrites the frozen fixtures.

• Step 2: Run - expect FAIL, then implement helper code until PASS.

• Step 3: Commit protondrive: add whole-file differential tests.

Task 13: Hermetic contract-suite harness (old backend untouched)

Files:

• Create: backend/protondrive/protontest/cmd/protonfake/main.go
• Create: backend/protondrive/fake_test.go

Interfaces:

• Produces:

  • protonfake: go run ./backend/protondrive/protontest/cmd/protonfake -addr 127.0.0.1:0 prints endpoint=http://..., client_uid=..., client_access_token=..., client_refresh_token=..., client_salted_key_pass=..., type=protondrive one per line, then serves until killed. Needs a non-test constructor: func NewStandalone(addr string) (*Server, error).
  • fake_test.go (package protondrive_test): func setupFakeRemote(t *testing.T) (remoteName string, srv *protontest.Server, user *protontest.User)
    • sets RCLONE_CONFIG_TESTPROTONDRIVEFAKE_* environment variables (type, endpoint, the four client_* keys, draft_client_uid) with t.Setenv and returns "TestProtonDriveFake:".
  • TestIntegrationFake: fstests.Run with RemoteName: remoteName, NilObject: (*protondrive.Object)(nil), QuickTestOK: true. Guarded by t.Skip until Task 14 (the old backend has no endpoint option).

• Step 1: Write fake_test.go and protonfake; run go run ./backend/protondrive/protontest/cmd/protonfake -addr 127.0.0.1:0 and confirm the six lines print; go test ./backend/protondrive/ -run TestIntegrationFake -v shows SKIP.

• Step 2: Commit protondrive: add hermetic contract test harness.

Task 14: Swap - rewrite protondrive.go on the new client

Files:

• Rewrite: backend/protondrive/protondrive.go
• Rewrite: backend/protondrive/protondrive_internal_test.go
• Keep: backend/protondrive/protondrive_test.go (live TestProtonDrive:)
• Modify: backend/protondrive/fake_test.go (remove the skip)

Interfaces:

• Consumes: everything above.
• Produces: Fs, Object, Options, NewFs, as other backends; template for structure and method order is backend/box/box.go, as the current file already follows it.

Decisions:

• Options. Existing ones unchanged (names, defaults, help text may be improved; app_version help warns that a wrong value can get the client blocked). New, all Advanced: true: upload_concurrency (int, default 4, values above 5 are clamped with a log line), ignore_signature_errors (bool, default false), draft_client_uid (Hide: fs.OptionHideBoth, Sensitive: true; generated with random.String(32) and saved on first use), endpoint (Hide: fs.OptionHideBoth, default https://drive-api.proton.me). username and password become Required: false (browser sign-in needs neither).

• Config state machine (model: backend/internxt/internxt.go:139-209): state "" asks "Sign in with a web browser?" (default true) with the text rclone is a third-party application not officially supported by Proton. (as built: default true only when username and password are not both set, and a remote with a saved session is first asked whether to sign in again, default no - handoff rulings 14 and 17); "browser" calls forkStart, shows the URL, then forkPoll; "password" runs loginSRP, asking for a 2FA code through fs.ConfigInput only when needed. Both end by saving the four client_* keys.

• NewFs. credentialStore is a per-Fs type over the configmap.Mapper (no package globals). If the saved session is missing or refresh fails with errSessionExpired and username+password are configured, sign in with loginSRP (using 2fa, else otp_secret_key via totp.GenerateCode); otherwise return the reconnect error. Then newClient, dircache rooted at client.rootID(), the box-style "root is a file" fallback.

• Features: ReadMimeType: true, CanHaveEmptyDirectories: true; no NoMultiThreading. Implement and assert: fs.Purger, fs.Copier, fs.Mover, fs.DirMover, fs.PutStreamer, fs.ListPer, fs.CleanUpper, fs.Abouter, fs.Disconnecter, fs.Shutdowner, fs.DirCacheFlusher, fs.MimeTyper, fs.IDer.

• Error mapping (one function, translateError): errNotFound -> fs.ErrorObjectNotFound or fs.ErrorDirNotFound by call site; errExists on Mkdir -> nil; on Move/DirMove -> fs.ErrorCantMove / fs.ErrorDirExists; API codes 200001-200004, 200300, 200301 -> fserrors.NoRetryError keeping Proton's message; 2000 and 2028 -> fserrors.FatalError("... upgrade rclone"); errIntegrity and errUploadMismatch -> fserrors.NoRetryError.

• Object.Open handles fs.RangeOption and fs.SeekOption by computing offset, limit and calling client.open; no io.NopCloser.

• Object.Update passes the source's SHA1 when src.Fs().Hashes().Contains(hash.SHA1), and the MIME type from fs.MimeType(ctx, src).

• Object.SetModTime returns fs.ErrorCantSetModTime. Precision is time.Second. Hashes is hash.Set(hash.SHA1).

• Rmdir lists the directory first and returns fs.ErrorDirectoryNotEmpty if it has children (as built: it asks the server, uncached, whether the folder has any child at all - handoff ruling 16); Purge of the root returns the existing errCanNotPurgeRootDirectory.

• DirMove to itself or an existing destination returns fs.ErrorDirExists.

• Shutdown calls client.close() once (guard with sync.Once). (Superseded: Shutdown only drops the node cache and leaves the Fs usable - handoff ruling 15.)

• go.mod: go get github.com/ProtonMail/gopenpgp/v3 github.com/ProtonMail/go-srp as direct requirements, remove the bridge imports, go mod tidy. The differential tests still import the old libraries, so they remain in go.mod after this task; their removal is Task 19.

• Step 1: Write failing tests in protondrive_internal_test.go: TestTranslateError table for every mapping above; TestOptionsBackwardsCompatible: a configmap.Simple holding exactly the keys of a v1.76 config (list in Global Constraints) builds an Fs against the fake without any auth request (Requests() has no /core/v4/auth key); TestNewFsPasswordFallback: revoked session + username/password signs in again and saves new client_* values; TestNewFsNoCredentials: error mentions rclone config reconnect; TestTwoRemotesIsolated: two Fs for two users in one process never see each other's tokens in their mappers; TestConfigBrowser and TestConfigPassword: drive Config through its states with fs.ConfigIn results; the browser state's text contains the third-party notice.

• Step 2: Run - expect FAIL.

• Step 3: Rewrite protondrive.go.

• Step 4: Run go test -race ./backend/protondrive/ -run TestIntegrationFake -v 2>&1 | tail -40. Fix failures in the new code (never by weakening the fake or skipping a test) until the suite passes. Then the four verification commands, and go build ./....

• Step 5: Commit protondrive: rewrite the backend on a native client following Proton's SDK.

Task 15: Regression tests for known failures

Files:

• Create: backend/protondrive/regression_test.go (package protondrive_test, through fs.Fs and the fake)

Interfaces: Consumes Task 13 setupFakeRemote.

• Step 1: Write the tests (spec 9.4, rows not marked M5/M6). Each is named TestRegression<Issue or Topic>:
  • TwoFsRefreshRace (9880, 8135): two Fs from one config; expire the token; both List concurrently; both succeed; the mapper holds valid credentials.
  • TwoProcessRefreshRace: two Fs on two configmap.Simple copies that are synced through a shared backing map with a delay; the loser adopts the winner's token, credentials never blank.
  • ManyNewFs (9816): 16 concurrent NewFs; unlock concurrency 1.
  • FileOverDirectory and DirectoryOverFile (9438): operations.Copy reports a clean error, nothing is destroyed.
  • DocsNodeSkipped (7959): a sync from a folder holding a Docs node completes with 0 errors.
  • BadSignatureFile (8003): lists; Open fails with a clear message; with ignore_signature_errors=true it succeeds.
  • SDKCreatedFolderResolvable (9622): a SeedFolder/SeedFile tree is reachable by NewObject("dir/sub/file") with a cold cache.
  • MoveIsServerSide, CopyIsServerSide: no block upload requests.
  • MultiThreadDownload: operations.Copy of a 20 MiB object to local with --multi-thread-streams 4 matches SHA1.
  • PutStreamUnknownSize.
  • Captcha: NewFs with password fallback under RequireCaptcha returns the browser sign-in hint.
  • AppVersionRejected: fault code 2028 -> fatal error mentioning upgrading rclone.
  • QuotaExceeded: fault code 200001 on draft creation -> not retried (request count 1), message preserved.
  • RateLimited: 5 consecutive 429s on listing -> succeeds, waited at least the sum of Retry-After.
• Step 2: Run go test -race ./backend/protondrive/ -run TestRegression -v - fix the backend until PASS.
• Step 3: Commit protondrive: add regression tests for known failures.

Task 16: Security tests

Files:

• Create: backend/protondrive/security_test.go

• Step 1: Write the tests (spec 9.6):

  • TestHostileServer subtests: wrong block hash, swapped blocks, truncated block list, older revision content under the current revision ID, attacker-signed passphrase, names "../x", "a/b", NUL, 10 MiB - each yields an error or a skipped entry, never file data or a path outside the listed directory.
  • TestNoSecretsInLogs: capture rclone's log output at fs.LogLevelDebug (as fstest log capture helpers do) across password sign-in, a 5 MiB upload, a download and Disconnect; assert the log contains none of: the password, mailbox password, key password, access token, refresh token, the node passphrase and session key (exposed by the fake for the test), any pm-storage-token value.
  • TestConfigHoldsNoPasswordAfterBrowserSignIn, and TestPasswordStoredObscured (the stored value differs from the plaintext and obscure.Reveal restores it).

• Step 2: Run - fix until PASS.

• Step 3: Commit protondrive: add hostile-server and secret-hygiene tests.

Task 17: Request budgets and fuzzing

Files:

• Create: backend/protondrive/budget_test.go

• Create: backend/protondrive/fuzz_internal_test.go

• Step 1: Write budget tests, using Requests() deltas after NewFs (exclude bootstrap by ResetRequests()):

  • TestBudgetList: folder of N in {0, 1, 100, 101, 250} files with page size 150 costs at most P + ceil(N/100) + 1 requests (P = child pages).
  • TestBudgetWarmList: second List costs 0.
  • TestBudgetNoopSync: tree of 5 folders x 40 files, already in sync; sync.Sync cold costs at most 2*D + ceil(N/100) + 5 requests for D = 6, N = 200, and uploads/downloads 0 blocks.
  • TestBudgetNewObject: NewObject three levels deep, cold: at most 2 requests per level + 1. (As built: 8, four folder levels at two requests each - handoff ruling 20.)
  • TestBudgetNoLegacyRoutes: after the whole fstests run, assert in a TestMain hook that no counted route matches the deprecated patterns. Record the measured values in a comment next to each bound.

• Step 2: Write fuzz targets seeded from testdata/oldlib: FuzzDecryptXAttrs, FuzzDecodeAPIError, FuzzDecryptForkPayload, FuzzDecodeLink (link JSON -> node), FuzzValidateName. Each must not panic and must not return success with an invalid name.

• Step 3: Run go test ./backend/protondrive/ -run 'TestBudget' and go test ./backend/protondrive/ -fuzz FuzzDecryptXAttrs -fuzztime 30s (repeat per target) - fix findings.

• Step 4: Commit protondrive: add request budget tests and fuzz targets.

Task 18: Wider hermetic suites and fork CI

Files:

• Create: fstest/testserver/init.d/TestProtonDriveFake
• Modify: fstest/testserver/init.d/PORTS.md
• Modify: fstest/test_all/config.yaml (add backend protondrive remote TestProtonDriveFake: entry, fastlist: false)
• Create: .gitea/workflows/ci.yml
• Modify: docs/TODO.md, docs/PERMISSIONS.md

Model the init.d script on fstest/testserver/init.d/TestWebdavRclone (rclone-serve.bash/run.bash helpers): start protonfake on a port allocated in PORTS.md, echo its key=value lines and _connect=127.0.0.1:PORT.

Read "Notes for Tasks 18 to 21" in the handoff before starting: this text leaves out how the script obtains protonfake's output and manages its process, and the commands below need RCLONE_CONFIG=/notfound.

• Step 1: Add the script and config entry. Run go run ./fstest/test_all -backends protondrive -remotes TestProtonDriveFake: -maxtries 1 -n 4 and fix backend failures in fs/operations, fs/sync, vfs until the report is clean. cmd/bisync and cmd/gitannex: run them; if they fail for reasons that need M5/M6 work, add ignoretests for those two on the fake remote and list the failures in docs/TODO.md.
• Step 2: .gitea/workflows/ci.yml on push and pull request: actions/setup-go with go-version-file: go.mod; steps go build ./...; go vet ./backend/protondrive/...; golangci-lint run ./backend/protondrive/...; go test -race ./backend/protondrive/...; a 20s fuzz run per target; go run golang.org/x/vuln/cmd/govulncheck@latest ./backend/protondrive/.... It cannot be verified locally: say so in docs/TODO.md ("first push must confirm the Gitea runner picks it up").
• Step 3: Update docs/PERMISSIONS.md verified-commands table for the host toolchain (Go 1.27.4, golangci-lint 2.15.0) and add golangci-lint run to the Allow tier in it and in the three tool files.
• Step 4: Commit protondrive: run the wider suites against the fake server in fork CI.

Task 19: Remove the old libraries

Files:

• Delete: backend/protondrive/crypto_diff_internal_test.go
• Keep: backend/protondrive/testdata/oldlib/ and move TestFrozenFixtures and TestFrozenFileFixture into crypto_internal_test.go / download_internal_test.go (they need no old-library import).
• Modify: go.mod, go.sum

Read "Notes for Tasks 18 to 21" in the handoff before starting: master is no longer the old backend, TestFrozenFileFixture has already moved, and docs/TODO.md has the accurate list of what to keep and delete.

• Step 1: Record the current binary size: git stash -u is not allowed - instead build master in a worktree: git worktree add ../rclone-master master && (cd ../rclone-master && go build -o /tmp/rclone-old .), note stat -c %s /tmp/rclone-old, then git worktree remove ../rclone-master.
• Step 2: Move the frozen-fixture tests, delete the differential file, go mod tidy.
• Step 3: Verify: grep -c 'Proton-API-Bridge\|rclone/go-proton-api\|replace' go.mod prints 0; go build -o /tmp/rclone-new . && stat -c %s /tmp/rclone-new is not larger than the old size; go test -race ./backend/protondrive/... passes; go run bin/cross-compile.go -compile-only (see Makefile target compile_all for the exact flags) succeeds for all targets; RCLONE_CONFIG=/notfound go test ./... has no new failures compared with the list in docs/TODO.md.
• Step 4: Commit protondrive: drop Proton-API-Bridge and go-proton-api.

Task 20: Documentation

Files:

• Modify: docs/content/protondrive.md (hand-written parts only; leave the autogenerated options block between its markers alone)

• Modify: docs/data/backends/protondrive.yaml

• Modify: docs/ARCHITECTURE.md, docs/DECISIONS.md, docs/TODO.md

• Create: docs/specs/001-conformance-checklist.md

• Step 1: Rewrite the docs page per spec section 10: both sign-in methods (fix the stale user>/pass transcript), the third-party notice verbatim, what is verified on upload and download, the ranged-read caveat, the --dump warning from spec section 7, limits (name rules, no SetModTime, trash semantics), removal of the "Beta" note and the henrybear327 library text. Do not change the tier.

• Step 2: Run python3 bin/manage_backends.py features protondrive if it works offline with the fake remote; otherwise edit protondrive.yaml features: by hand to match the new feature set and set features_score: 5. Leave tier, maintainers, integration_tests untouched.

• Step 3: docs/specs/001-conformance-checklist.md: one table row per SDK module (apiService, shares, nodes, upload, download, crypto, account/auth) with the SDK file, the Go file, the SDK tag last checked (js/v0.23.0), and known deliberate differences (no small-file path, no events yet, no thumbnails).

• Step 4: docs/ARCHITECTURE.md (protondrive section replacing the "known rough edges" line), a docs/DECISIONS.md entry for the native port, docs/TODO.md entries for M5, M6, M7 and every live check not run.

• Step 5: Commit protondrive: update docs for the rewritten backend.

Task 21: Live verification (blocked until a test account exists)

Out of date since 2026-11-11: two test accounts exist and are configured as TestProtonDrive: (free) and TestProtonDrivePaid: (paid, never signed in to) with username and password. See section 5 of the handoff for the setup and section 4 for what to check first.

Needs the fork owner to create the account and configure it themselves as TestProtonDrive: in their rclone config (rclone config, browser sign-in). Credentials must never be pasted into an agent conversation or committed. If the remote is absent (rclone listremotes lacks TestProtonDrive:), do not attempt this task: leave every box unticked and report it as not run.

• go test -v ./backend/protondrive/ -run '^TestIntegration$' -remote TestProtonDrive:
• go run ./fstest/test_all -backends protondrive -remotes TestProtonDrive: -n 1 clean with no ignore: entries.
• Capture golden fixtures from the account (nodes created by the web app and by Proton's official CLI) into testdata/golden/ with a -capture test flag, and add TestGoldenFixtures.
• Interop both ways with the official CLI and the web app (spec 9.7).
• Upgrade/downgrade check against released rclone v1.76.
• Record results in docs/TODO.md; only then is spec criterion 12 (items 3, 6, 7) met.

---

Self-review notes

• Spec coverage: sections 6.1 (Tasks 2-4, 14), 6.2 minus events and persistent cache (Task 8; those are M5/M6, next plan), 6.3 (11), 6.4 (10), 6.5 minus Metadata (9, 14), 6.6 (8, 14), 6.7 (14), 7 (16), 9.1 (5, 6, 12), 9.2 (1, 2, 3, 4, 7, 9, 11), 9.3 (13, 14, 18), 9.4 (15, 17), 9.5 (17), 9.6 (16), 9.7 (21), 9.8 (18), 10 (20), 12 (19, 21).
• Not in this plan by design: events and ChangeNotify, persistent cache, Metadata, small-file upload, the nightly live CI job, the weekly SDK reminder.