Compare commits

..
285 Commits
Author SHA1 Message Date
Nick Craig-Wood be8d6044ea Version v1.75.2 2026-10-09 10:34:46 +01:00
Nick Craig-Wood bd07f481e2 build: fix lint errors after golang.org/x/net update
golang.org/x/net v0.60.0 deprecates its HTTP/2 server and transport in
favour of net/http.

Make the h2c test use http.Transport with unencrypted HTTP/2. serve
restic --stdio still needs http2.Server to serve HTTP/2 on a single
connection as http.Server has no way of doing that.

(cherry picked from commit d3db33c6b7)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 56059c47c8 build: update golang.org/x/net to v0.60.0 to fix multiple CVEs
Upgrade to v0.60.0 of golang.org/x/net in order to address:

- CVE-2026-78659: net/http: HTTP/2 server memory exhaustion due to Trailer headers
- CVE-2026-78669: net/http: excessive CPU consumption from repeated initial window changes
- CVE-2026-78663: net/http: double flow control refund on HTTP/2 server streams
- CVE-2026-97032: net/http: HTTP/2 server crash due to HPACK encoder race

This also updates golang.org/x/crypto, sync, sys, term and text to the
versions golang.org/x/net v0.60.0 requires.

(cherry picked from commit e4a3adb94c)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 35ab21109b rest: fix TestStripHeadersOnCrossHostRedirectFn failing on go1.26
Before go1.27 net/http compares hosts case sensitively when deciding
whether to keep Authorization on a redirect, so it dropped the header
itself on the localhost -> LOCALHOST redirect in the
SameHostDifferentCase test, before StripHeadersOnCrossHostRedirectFn
was consulted. The other headers still show the function treats the
hosts as the same.

(cherry picked from commit f440573c3e)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood fc7a57b447 serve s3: apply --metadata-max-objects to each user separately
The metadata of the objects of all users was kept in one store, so with
--auth-proxy or several --auth-key credentials one user storing the
metadata of many objects could make the server forget that of every
other user.

Each user (each access key) now has a store of their own.

(cherry picked from commit c2ce9a0bfb)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 028f2b4568 serve s3: apply the multipart upload limits to each user separately
--multipart-max-uploads and --multipart-streaming-buffer-total limited
the uploads of all users together, so with --auth-proxy or several
--auth-key credentials one user could use up the limits and stop every
other user uploading.

They now limit the uploads of each user (each access key) separately.

(cherry picked from commit b1657092d3)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 62e770eb54 serve s3: fix multipart upload corruption if the backend write of an early part fails
A multipart part which arrives before the parts ahead of it is buffered
and acknowledged to the client, then written to the backend when its
turn comes. If that write failed the part was dropped, possibly with
some of it already written, but the upload carried on as if nothing
had happened, so it could be completed with the part missing, or with
a resend of the part appended to the partial data.

Such an upload is now failed, so the client starts it again.

(cherry picked from commit 5379bd5409)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 652bbb0908 serve s3: fix presigned URL being turned into a copy of any object GHSA-w3h7-jxxq-vfrp CVE-PENDING
The SigV4 verification in gofakes3 only checked the headers the
client listed as signed and never looked at the other headers which
arrived. A presigned PUT URL signs only the host header, so a holder
of one could add an unsigned x-amz-copy-source header and have the
server copy any object it can reach - in any bucket, and in any
remote when serving a cloud backend - into the object the URL names,
then read it back with the same URL. This affected both --auth-key
and --auth-proxy. AWS refuses such a request with AccessDenied, as
it requires every x-amz-* header to be signed.

This updates gofakes3 to a version which refuses requests carrying
an x-amz-* header not in their signed headers list with AccessDenied,
and adds tests that a presigned PUT URL can't be turned into a copy
in either auth mode. Real S3 clients always sign x-amz-* headers so
they are unaffected.

(cherry picked from commit c3b582ccdf)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 4df376e2db serve s3: test multipart buffer limits can't be bypassed by overflowing part sizes GHSA-qjw8-8hmp-c3m6
In v1.75 the multipart reorder buffer admitted the next part the stream
needed however big, adding its client-declared length to the running
total unchecked. Two concurrent requests for the next part declaring
huge lengths overflowed the total negative, after which out-of-order
parts were admitted past --multipart-streaming-buffer-limit.

The next part is now streamed straight to the backend without being
charged to the buffer, and a buffered part is only admitted if it fits
in what is left of the limits, so this can no longer happen. Add a
test of the attack against both the per-upload and server-wide limits.

(cherry picked from commit d120b5fe0b)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 68e43e5b88 serve s3: document memory use and bring the docs up to date
Add a section on the memory serve s3 uses and how to limit it, and
update the docs for the recent changes to multipart uploads, listings
and metadata.

(cherry picked from commit da71a52714)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 1dcc246e03 serve s3: fix stale metadata being returned for objects changed elsewhere
serve s3 keeps the metadata of objects uploaded through it in memory.
If the object was then changed some other way - by another --auth-proxy
user sharing the backend, another server or directly on the backend -
serve s3 went on returning the old metadata, including the old
modification time, for the new object.

The size and modification time of the object are now stored with its
metadata, which is ignored if they no longer match.

(cherry picked from commit 221398e2d0)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 4cc69c2a62 serve s3: fix uploads failing when the auth proxy expires the user's backend
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. A request didn't count as using it, so a
long upload or download could have its backend shut down under it and
fail. A multipart upload used the backend it started on throughout, so
failed if the client paused for more than 5 minutes between parts, or
the auth proxy changed the user's secret.

Each request and multipart upload now holds the backend it uses until
it is finished with it. In-progress multipart uploads are aborted when
the server is shut down so their backends are released.

(cherry picked from commit 0edd92134e)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 8d97a50c4a serve s3: stop auth proxy users listing or using each other's multipart uploads GHSA-35g6-9fjv-rmx5 CVE-PENDING
With --auth-proxy each access key ID may be mapped to a different
backend, but gofakes3 kept a single record of multipart uploads for
everyone. Any user could list the uploads of every other user with
ListMultipartUploads (including their upload IDs) and the parts of
them with ListParts.

With --disable-multipart-streaming the uploads are held by gofakes3,
whose upload IDs are sequential so easy to guess. Any user could
upload parts to, abort, or complete another user's upload - which
wrote it to the backend of the user completing it.

This updates gofakes3 to a version which can keep each user's uploads
private and uses the access key ID as the owner.

(cherry picked from commit 32b08f0237)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 7c2405c035 serve s3: stop auth proxy users using each other's multipart uploads and metadata GHSA-35g6-9fjv-rmx5 CVE-PENDING
With --auth-proxy each access key ID may be mapped to a different
backend, but serve s3 found a streaming multipart upload by its upload
ID alone and stored object metadata by path alone.

A user who learnt another user's upload ID (for example from
ListMultipartUploads on a bucket of the same name) could upload parts
to, complete or abort that upload on the other user's backend. A user
with an object at the same bucket and key as another user was returned
the other user's metadata by HeadObject and GetObject and could
overwrite it.

Keep the uploads and metadata of each access key ID apart.

(cherry picked from commit a16a901de7)
2026-10-09 00:01:36 +01:00
Nick Craig-Wood 10335538d1 serve s3: keep the state remembered for each user in one place
Refactor the state serve s3 keeps between requests - in-flight
multipart uploads and object metadata - so that it is reached through
a tenant struct for the user making the request. This makes it
possible to keep users' state apart when they are serving different
backends.

There is only one tenant for now, so there is no change in behaviour.

(cherry picked from commit 2385e27cfd)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood a7b1599204 serve s3: add --metadata-max-objects to limit the metadata kept in memory
serve s3 keeps the metadata of the objects uploaded to it in memory,
with no limit on the number of objects, so memory use grew with every
object uploaded, and a client could use a lot by uploading many empty
objects with large metadata.

The new --metadata-max-objects (default 100000, 0 for no limit) limits
the number of objects whose metadata is kept, forgetting the metadata of
the least recently used first. As the metadata is only kept in memory
it is already lost when the server restarts; the modification time is
stored on the object itself so is never lost.

(cherry picked from commit 77ed639805)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 518f43dee9 serve s3: forget expired multipart uploads completely
When an abandoned multipart upload expired after --multipart-expiry,
gofakes3's record of it was left behind, so it stayed in memory and in
ListMultipartUploads until the server was restarted.

That record is now removed too.

(cherry picked from commit 0ce74ffaac)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 43d3374465 serve s3: update gofakes3 to limit the memory requests can use
This picks up these fixes to serve s3 from gofakes3:

- XML request bodies (CompleteMultipartUpload, DeleteObjects) bigger
  than 10 MiB are rejected rather than read into memory.
- DeleteObjects requests with more than 1000 keys are rejected.
- CompleteMultipartUpload checks the upload exists before reading the
  request body.
- With --disable-multipart-streaming, parts are no longer allocated at
  their declared size before the data arrives.
- Browser POST uploads hold at most 1 MiB in memory and are limited to
  5 GB.

It also brings a SlowDown error, so the SlowDown replies serve s3 sends
when a part waits too long for buffer space, or too many multipart
uploads are in progress, now have status 503 as S3 does rather than
500.

(cherry picked from commit f4c1268f33)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 10875eec70 serve s3: add --multipart-max-uploads to limit uploads in progress
There was no limit on the number of multipart uploads in progress, and
each one holds server resources until it completes or expires - once it
has data, a VFS upload and the buffers of the remote's own upload - so a
client could use unbounded memory by starting many uploads.

The new --multipart-max-uploads (default 1000, 0 for no limit) limits
the streamed multipart uploads in progress. Starting another fails with
SlowDown, which S3 clients retry, before anything is created on the
remote.

With --multipart-expiry 0 nothing ever cleans up abandoned uploads, so
they go on counting towards the limit until no new upload can be
started. A warning is logged at startup when both are set.

(cherry picked from commit d24568e3e6)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 40eaa114c1 serve s3: start the upload to the backend on the first multipart part
Starting a multipart upload opened its VFS upload straight away, which
allocates a 100 KiB buffer (--streaming-upload-cutoff) and starts a
goroutine and a transfer, all held until the upload finished or was
expired after --multipart-expiry (default 24h). A client could make the
server hold about 100 KiB per tiny CreateMultipartUpload request.

The VFS upload is now only opened when the first part is written to it,
so an upload with no parts holds none of these.

(cherry picked from commit 7d2c16269d)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood a34b606abf serve s3: add --multipart-streaming-buffer-total to limit memory across uploads
--multipart-streaming-buffer-limit only limits the memory buffered for
parts arriving out of order in each upload, so a client could use as
much memory as it liked by starting more uploads.

The new --multipart-streaming-buffer-total (default 1G, 0 for no limit)
limits the memory buffered by all uploads together. A part which would
take either over its limit waits as before. Since the part the stream
needs next is never buffered, every upload can still make progress.

(cherry picked from commit 87d81fcb2e)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 37aa0cb629 serve s3: stream in-order multipart parts without buffering them
Every multipart part was buffered in memory in full before being written
to the backend, even the part the stream needed next. There is no limit
on part size, and several concurrent requests for the current part were
each admitted, so a client could make the server hold as much memory as
it sent, whatever --multipart-streaming-buffer-limit was set to.

The part the stream needs next is now written straight through to the
backend as it arrives, so only parts which arrive ahead of their turn
are buffered, and those always within the limit - a part bigger than
the limit waits until it is the next part rather than being admitted
when the buffer is empty. Further requests for the part being streamed
wait, holding no memory, and are then checked against it.

Since what reaches the backend can't be taken back, a part whose body
fails part way records how far it got, and the client's retry of the
part, which must have the same content, carries on from there. Such an
upload can't be completed without the rest of the part.

The body is read to its end, so that an error it only reports there -
as the Content-MD5 check does - fails the part rather than being
missed, and a body longer than declared is rejected. A part rejected
once it has been streamed leaves data which can be neither completed
nor replaced, so the whole upload is failed and cleaned up and the
client starts again.

(cherry picked from commit 7773bb6487)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood a7f0259e55 serve s3: time out multipart parts waiting for buffer space
A part waiting for room in the out-of-order buffer waited forever,
ignoring the request being cancelled, and a waiting part stopped its
upload being expired, so a client could hold requests, goroutines and
uploads open indefinitely.

The wait now ends when the request is cancelled, and after a minute the
part fails with SlowDown so the client retries it later.

(cherry picked from commit 603e6238eb)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 323b0b4559 serve s3: don't buffer re-uploads of multipart parts already streamed
A part uploaded again after it had been streamed to the backend (a
client retrying after a timeout) was buffered in memory in full just to
compare it with the original, and the buffer limit didn't apply to it.

Such a re-upload is now hashed as it is received and compared with the
original without being buffered.

(cherry picked from commit 122b4e8df2)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 26175553a7 serve s3: count buffered multipart parts in whole memory pages
The out-of-order multipart buffer was charged the declared length of
each part, but a buffered part occupies whole 1 MiB pool pages however
small it is. A client could therefore send up to 10000 one-byte parts
ahead of the part the stream needed and pin about 10 GiB of memory in a
single upload while --multipart-streaming-buffer-limit saw only 10 KB.

Each buffered part is now charged the whole pages it occupies.

(cherry picked from commit 4aa3c41aa1)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 4a944dff0f webdav: fix headers option leaking to other hosts on redirect GHSA-3rqf-7h69-gg5v CVE-PENDING
The headers set with the `headers` option were sent to any host the
server redirected to, as net/http only strips Authorization and Cookie
on a cross domain redirect and copies every other header onto each
hop. A server which could be made to redirect to another host would
therefore receive any credential carried in a custom header such as
X-Auth-Token.

Strip the configured headers from a redirect hop once the chain has
left the configured host (a different hostname, subdomain or port)
using rest.StripHeadersOnCrossHostRedirectFn. This logs at debug level
when it happens so a 401 from the redirect target can be diagnosed.
The `auth_redirect` option is unchanged as it is an explicit opt-in to
keep credentials across redirects.

The Authorization header set from `user`, `pass` and `bearer_token` is
deliberately left to net/http's policy, which strips it on a change of
domain but keeps it for a subdomain or a different port, as servers
which redirect to a subdomain rely on that today.

(cherry picked from commit 224a515d68)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood e9b9b48a68 docs: fix release verification instructions trusting unsigned hashes
The manual release verification steps checked the signature with gpg
--verify and then checked the binaries against the whole SHA256SUMS
file. gpg only verifies the clearsigned part, so an unsigned hash line
added outside it would be used by sha256sum -c or by rclone hashsum -C.

The instructions now extract the signed hashes with gpg --decrypt and
check the binaries against those only.

See: GHSA-w33h-3qgq-8mv8
(cherry picked from commit 204adfeedc)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 3a14789eb4 selfupdate: fix hash check trusting unsigned data in the signed SHA256SUMS GHSA-w33h-3qgq-8mv8 CVE-PENDING
rclone selfupdate checked the PGP signature on the clearsigned block
of SHA256SUMS but then looked up the archive hash in the whole
downloaded file. Unsigned data can appear in that file without
invalidating the signature. A hash line placed before or after the
signed text was accepted, allowing whoever controlled the download to
install an arbitrary unsigned archive.

The hash is now only looked up in the signed text.

Affects stable (non --beta) selfupdate since v1.55.0. --beta updates
are not signature checked.

(cherry picked from commit 672bbf75ec)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 47027efd12 fshttp: drop --header values over HTTP after HTTPS downgrade GHSA-rrwf-2qr6-p832 CVE-PENDING
The transport strips the headers set with --header once a redirect
chain leaves the originally requested host, but the host comparison
ignores the URL scheme. A server which redirects https://host:8443/
to http://host:8443/ keeps the same host and port, so the headers
were added to the plaintext request and could be read by anyone on
the network path.

The redirect chain walk now also treats a hop which downgrades the
original HTTPS request to HTTP as leaving the host, so the headers are
removed from that hop and every one after it.

The transport also logs at debug level when it removes the headers,
for either reason, so that a resulting 401 from the redirect target
can be diagnosed.

(cherry picked from commit bc1050bc8a)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 2f43b99abe vfs: fix serve nfs path traversal by refusing unsafe file names GHSA-44pm-2q5p-fm63 CVE-PENDING
Dir.Create, Dir.Mkdir and Dir.Rename joined whatever name they were
given onto the directory's path. A name of "..", "." or one containing
a "/" isn't a child of the directory, and at the VFS root
path.Join("", "..") preserves the ".." so the new node's path pointed
outside the root. Backends which join the Fs root with the remote then
resolved it to a location above the directory the VFS was created on.

serve nfs was exploitable through this. The go-nfs library refuses a
name of exactly "." or ".." in MKDIR only, doesn't check the name in
CREATE, RENAME or SYMLINK, and never refuses a name containing a "/".

serve webdav was exploitable in the same way and has its own fix. The
other serve front-ends rely on their own or their protocol library's
path cleaning to keep such names out of the VFS.

Reject such names with EINVAL in the VFS itself, so that every
front-end and mount is protected regardless of what the layer above it
checks. This is the fix for serve nfs and defence in depth for the
others. The serve webdav check is kept as the front-end is the trust
boundary and gives the client a clear error. The source name of a
rename is only used to look the node up so it isn't checked.

VFS.Mkdir of the root, which previously went through Dir.Mkdir with an
empty name and added a nameless child node to the root, now makes the
root directory on the backend directly, still honouring --read-only.
serve webdav and serve ftp made directories through Dir.Mkdir rather
than VFS.Mkdir, so they are switched to VFS.Mkdir to keep "MKCOL /" and
"MKD /" working.

(cherry picked from commit 3739ad9c35)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 133ced103a serve webdav: fix path traversal above the served directory GHSA-44pm-2q5p-fm63 CVE-PENDING
A request path with a ".." element, such as "MKCOL /..", was passed
straight through to the VFS. At the VFS root path.Join("", "..")
preserves the "..", so a directory node with the path ".." was added
under the root and every path built beneath it started "../". Backends
which join the Fs root with the remote before encoding it - such as
webdav, ftp, sftp, http and memory - resolved that "../" away, letting
an unauthenticated client create, overwrite, copy, move and delete
objects and directories outside the directory rclone was told to serve.
Moving a directory or file from outside the served tree into it also
made it readable.

Reject any request with a "." or ".." element in the request path, or
in the Destination header of COPY and MOVE, with 400 Bad Request. The
check is made once in ServeHTTP rather than in each webdav.FileSystem
method so that the client gets a consistent error. Rejected requests
are logged at INFO level like other requests.

The webdav handler strips --baseurl from the Destination as a string
prefix, so "/base../x" reaches the FileSystem as "../x". The Destination
is therefore checked after stripping the prefix in the same way.

(cherry picked from commit c266a63ced)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 317c885857 netstorage: stop a listing resume token moving signed requests to another host GHSA-xqmq-v4wj-rw5m CVE-PENDING
The resume start value returned by the NetStorage list API is joined
onto the endpoint URL as a URL reference to build the next listing
request. A value beginning with `//` is a network-path reference, so a
server (or an on-path attacker when `protocol = http`) which returns
`//other.example/path` could make rclone send the next request, signed
with the account secret, to a host other than the configured one.

Use rest.URLJoinRoot so the resume token can only change the path.

(cherry picked from commit c8dfd98b0c)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood a34f1f153a netstorage: stop a path starting with // moving the remote to another host GHSA-xqmq-v4wj-rw5m CVE-PENDING
The netstorage backend joins the path after `remote:` onto the
configured host URL according to the URL standard. A path beginning
with `//` is a network-path reference under that standard, so a path
such as `//other.example/dir/` replaced the configured host (and port
and userinfo) rather than being treated as a path. Every request,
including the initial stat, was then signed with the account secret and
sent to the substituted host. The signature covers the request path and
action but not the host, so the substituted host received signatures
valid for replay against the real NetStorage endpoint for the duration
of their validity.

Use rest.URLJoinRoot, which refuses any root that changes the scheme,
host or userinfo of the configured URL.

(cherry picked from commit 4f28f4492c)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood e7d4d425bc http: stop a path starting with // moving the remote to another host GHSA-xqmq-v4wj-rw5m CVE-PENDING
The http backend resolves the path after `remote:` against the
configured url according to the URL standard. A path beginning with
`//` is a network-path reference under that standard, so a path such as
`//other.example/dir/` replaced the configured host (and port and
userinfo) rather than being treated as a path. The configured headers
were then sent to the substituted host on the initial HEAD request and
every request after it.

Use rest.URLJoinRoot, which refuses any root carrying a scheme, host
or userinfo of its own, to join the path onto the configured url.

(cherry picked from commit ecd821243f)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 01a9d67e06 fstests: check a root starting with // can't move the remote to another host GHSA-xqmq-v4wj-rw5m CVE-PENDING
Add an integration test which starts a listener on localhost, makes
the remote with roots naming that listener and checks that nothing
connects to it. The listener replies with junk so a backend which does
connect fails at once rather than retrying a closed connection.

(cherry picked from commit f33fa3872f)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood aece53963a rest: add URLJoinRoot to join a path onto a URL without changing its host GHSA-xqmq-v4wj-rw5m CVE-PENDING
rest.URLJoin resolves its path as a URL reference according to the URL
standard, so a path beginning with `//` is a network-path reference
which replaces the host (and port and userinfo) of the base URL. That
is often the wrong behaviour.

Add rest.URLJoinRoot which joins an unescaped root onto a base URL and
refuses any root which parses with a scheme, host or userinfo of its own.

(cherry picked from commit 28562c8a79)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 291016ea75 lib/http: require authentication for OPTIONS requests GHSA-gj73-fh6v-92fj CVE-PENDING
Before this change the basic, htpasswd and auth proxy authentication
middlewares let every OPTIONS request through without checking
credentials so that browser CORS preflights, which can't carry
credentials, would succeed.

However in serve webdav the OPTIONS request is answered by the WebDAV
handler whose Allow header depends on whether the path is an existing
file, an existing directory or doesn't exist, so an unauthenticated
client could discover the existence and type of any path on the served
remote even with --user/--pass or --htpasswd set.

After this change he authentication middlewares no longer special-case
OPTIONS. Instead the CORS middleware, which runs before authentication
and owns the --allow-origin setting, answers genuine CORS preflights
itself with an empty 200 carrying the CORS headers only when
--allow-origin is set.

All other OPTIONS requests need credentials like any other method, so
clients which probe with OPTIONS before authenticating will get a 401
challenge and must retry with credentials.

(cherry picked from commit 25707f2dc6)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 7e57142de1 pcloud: only accept pCloud hosts from the hostname in the OAuth callback
When pCloud redirects the browser back to rclone's local OAuth
callback server it adds a `hostname` parameter saying which regional
API host (api.pcloud.com or eapi.pcloud.com) the account lives on.
rclone used this for the token exchange and saved it in the config as
the host for all of the remote's later traffic, without checking it.

This only accepts a well formed hostname within pCloud's own domain,
lower cased, failing the config with an error otherwise, and adds a
unit test for the check. This is defence in depth for GHSA-hvpr-p4pv-4f46

(cherry picked from commit 3760671b36)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 722c14d806 pcloud: fix OAuth config accepting callbacks without state GHSA-hvpr-p4pv-4f46 CVE-PENDING
The local web server which receives the OAuth callback during
`rclone config` and `rclone authorize` checks that the `state`
parameter matches the random value rclone generated when starting the
flow. The pcloud backend opted out of this check with the
`StateBlankOK` option in lib/oauthutil, which made the local server
accept a callback with no state at all.

The callback URL (http://127.0.0.1:53682/) is fixed and well known,
so while a pcloud OAuth flow was in progress any web page open in the
user's browser could complete it with a single cross-site request
carrying an attacker's authorization code. This is a CSRF
(CWE-352) on the auth callback.

This commit removes the `StateBlankOK` option so that every callback
must carry the correct state and adds a test that the local auth
server rejects callbacks with a blank, missing or wrong state.

The option was added in May 2020 (#4210) because pCloud briefly
omitted the state parameter from its redirect. The callbacks recorded
five weeks later when EU region support was added show both
api.pcloud.com and eapi.pcloud.com returning `state=` again, so the
exemption has been unnecessary since then.

See: GHSA-hvpr-p4pv-4f46
(cherry picked from commit a53fdac5ec)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 89b8a59e8b vfs: stop --links objects called "...rclonelink" appearing as ".." in directory listings
When reading a directory with --links the VFS skipped entries called "."
and ".." before removing the ".rclonelink" suffix, so an object called
"...rclonelink" (or "..rclonelink" or ".rclonelink") produced a directory
entry called ".." (or "." or "") whose path resolved to the parent
directory. The same happened for names added back from the VFS cache.

The check is now done after the suffix is removed.

See GHSA-55qp-jrwr-x827

(cherry picked from commit 93ba2d40f0)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 1b500594d3 local: fix .rclonelink names escaping the root with --links GHSA-55qp-jrwr-x827 CVE-PENDING
In --links mode the local backend checked that a source name stayed inside
the destination root and only then removed the ".rclonelink" suffix from the
resulting OS path. A source entry called "...rclonelink" passed the check and
was then trimmed to "..", so the object addressed the parent of the
destination the user chose.

Writing the content of such an entry was already refused, as it is done
through an os.Root, but metadata was not: with --metadata rclone rewrote the
timestamps, ownership and extended attributes of the directory above the
destination from values supplied by the source.

The suffix is now removed from the remote name before it is joined to the
root, so the containment check applies to the name that is actually used. A
translated link must also leave a real native file name once the suffix is
removed.

A regular file whose name is exactly ".rclonelink" (or "..rclonelink" or
"...rclonelink") is not a valid translated link and is now skipped in
directory listings.

(cherry picked from commit 4d1bd4c71d)
2026-10-09 00:01:35 +01:00
Dhevenddra 9b495f2717 test: skip the symlink tests when the platform won't allow symlinks
Nine tests fail on an ordinary Windows machine, eight in backend/local and
TestEnvironmentVariables in cmdtest, all with

    symlink file.txt \?\C:\Users\...\symlink.txt: A required privilege is not
    held by the client.

Windows grants SeCreateSymbolicLinkPrivilege only to an elevated process or one
running with Developer Mode enabled, and a default install gives an ordinary
user neither. CI does not see this because the windows-latest runner is
elevated, so the failures only show up on a contributor's own machine, where
AGENTS.md asks for make quicktest to pass before opening a pull request.

cmdtest already recognised the situation and attached a note to the failure
saying the test could safely be ignored. If it is safe to ignore then the test
knows it cannot run, so skip it and say why instead.

backend/local gains a helper that tries a symlink in t.TempDir() and skips if it
cannot make one, called from the six tests that need the privilege. Where a
platform can create symlinks the probe succeeds and nothing is skipped, so other
platforms are unchanged.

TestMetadata is skipped whole because it creates its symlink before anything
else and the object built from it is used throughout.
TestSymlinkEscapeConcurrent is left alone: it goes through putLink and ignores
the error, so it never needed the privilege.

(cherry picked from commit 5fc1cc3ca1)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood f6290108df local: fix directories named *.rclonelink being treated as links with --links
In --links mode the local backend removed the ".rclonelink" suffix from
directory names as well as file names when building the OS path. Only files
are ever translated into symlinks, so a directory called "sub.rclonelink"
addressed the sibling "sub" instead: listing it logged "Failed to read link
size", its metadata was read from the wrong place, and setting metadata on
it with --metadata failed or landed on the sibling.

Directories now use their name as is, whatever suffix it has, in the same
way DirSetModTime already does.

(cherry picked from commit a7ef4c9064)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood ed9f1d836e b2: fix account token and SSE-C key leaking on redirects from --b2-download-url GHSA-gpw7-77cq-9f72 CVE-PENDING
Every file download sends the live account authorization token and,
when configured, the SSE-C customer key headers to the download host.
With --b2-download-url that host is a CDN or proxy the user does not
operate, and the http client followed its redirects with Go's default
policy:

- Go only strips Authorization when the redirect target is not the same
  host or a subdomain, and never looks at the scheme, so a same-host
  HTTPS to HTTP redirect replayed the full-capability B2 token in
  cleartext.
- Go has no idea that the X-Bz-Server-Side-Encryption-Customer-* headers
  carry the customer encryption key, so it forwarded them to any
  redirect target including a different host.

The B2 client now uses rest.StripHeadersOnCrossHostRedirectFn, so it
refuses to follow a redirect which downgrades HTTPS to HTTP and strips
the Authorization and SSE-C headers from any redirect which leaves the
host and port of the original request, matching the protection already
given to the S3, WebDAV and HTTP backends.

Fixes GHSA-gpw7-77cq-9f72

(cherry picked from commit 30009599e4)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 59ea591bcb rest: add StripHeadersOnCrossHostRedirectFn and use it in the s3 and http backends
The s3 and http backends each carried their own copy of the same
CheckRedirect policy. This moves it into lib/rest as
StripHeadersOnCrossHostRedirectFn, so that any backend which sends
credentials in headers to a host it does not control can use the same
policy. It logs at debug level when it strips headers so that a
resulting 401 from the redirect target can be diagnosed. A header
starting with "*" is matched without canonicalising, as that is how
SetHeader and Opts.ExtraHeaders send it.

This changes the s3 policy in two edge cases:

- It compared the raw URL host string, so a redirect to the same host
  in different case or with an explicit default port stripped the
  credentials. It now uses SameHost like the http backend, which treats
  those as the same host.
- The downgrade check compared each hop with the previous one. It now
  compares with the original request like the other backends, so a
  chain which starts on plaintext HTTP may return to it.

The behaviour tests move to lib/rest. The backend tests which check the
policy is wired into each client are kept.

(cherry picked from commit cae10713e8)
2026-10-09 00:01:35 +01:00
Nick Craig-Wood 2cc9088374 build: fix multiple CVEs by upgrading to go1.26.9
- CVE-2026-97032: net/http: HTTP/2 server crash due to HPACK encoder race
- CVE-2026-78659: net/http: HTTP/2 server memory exhaustion due to Trailer headers
- CVE-2026-97031: crypto/tls: reject malformed ECH outer extension references
- CVE-2026-94444: cmd/go: checksum bypass for golang.org/fips140
- CVE-2026-94447: cmd/go: checksum database bypass for golang.org/toolchain
- CVE-2026-94448: html/template: reset context tracking on consecutive template expressions
- CVE-2026-97030: html/template: recognize |yield| as regexp preceder keyword
- CVE-2026-94440: net/textproto, mime/multipart: memory limit bypass when parsing MIME headers
- CVE-2026-56866: net/http: HTTP/1 client connection desynchronization after CONNECT rejection
- CVE-2026-94439: net/http: HTTP/1 server connection desynchronization after 2xx CONNECT response
- CVE-2026-78669: net/http: excessive CPU consumption from repeated initial window changes
- CVE-2026-78660: net/http: HTTP/2 transport accepts malformed framing-related headers
- CVE-2026-56857: os: Root.Mkdir(All) can follow junctions out of the root on Windows
- CVE-2026-78667: net/http: lack of limit on size of parsed Range headers
- CVE-2026-78663: net/http: double flow control refund on HTTP/2 server streams

This also updates the go1.26 test job to go1.26.9.
2026-10-09 00:01:35 +01:00
Nick Craig-Wood d8dbeb9de4 operations: remove --copy-dest --immutable test for single file copy
TestCopyFileCopyDestImmutable relies on --immutable being enforced for
copyto, moveto and single file copies, which v1.75 doesn't do, so it
failed wherever the remote supports server-side copy, such as the
local backend on macOS.

The --copy-dest --immutable fix it was picked with still applies to
sync and copy and is tested by TestSyncCopyDestImmutable.
2026-10-09 00:01:30 +01:00
James Truitt 9cf1af177a docs: fix formatting error in Google Drive docs
(cherry picked from commit 2debe0ae6a)
2026-10-08 17:17:50 +01:00
kaii9 bb42b076af serve restic: fix repositories named data - fixes #6086
(cherry picked from commit 9823661bb8)
2026-10-08 17:17:50 +01:00
kaii9 0992024791 accounting: keep speed averaging active when transfers overlap
(cherry picked from commit aa308e3d7c)
2026-10-08 17:17:50 +01:00
somaz 48ac01c2b8 mount: fix --daemon failing when --rc is enabled - fixes #9196
Before this change the parent process kept the remote control and
metrics servers it started in initConfig listening, so the daemon child
could not bind the same addresses and exited.

After this change the parent shuts those servers down just before
daemonizing, and the child starts them as usual.

(cherry picked from commit 4532a88838)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood b928e8ddd0 serve: fix --auth-proxy backends never being shut down after simultaneous logins
With --auth-proxy, when several logins with the same credentials
arrived together and none was in the cache yet, as when an FTP client
opens several connections at once, each one ran the auth proxy program
and took its own reference to the backend. Only one of them was
remembered, so the others were never given back and the backend was
never shut down.

Logins which arrive together now share one run of the auth proxy
program and one reference to the backend.

(cherry picked from commit 2e9c06c84d)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 9ac7855da9 serve: fix --auth-proxy stopping VFS cache uploads after 5 minutes
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. With --vfs-cache-mode writes or full, files
are uploaded to the backend after the transfer which wrote them has
finished, which didn't count as a use. So an upload which hadn't
finished 5 minutes after the user's last command (or disconnection for
serve sftp) was stopped, leaving the file in the VFS cache but not on
the backend.

The backend is now kept until it has no files open for write and
nothing in the VFS cache waiting to be uploaded.

This was introduced in v1.75.1 by

f425f8d46 serve: refactor VFS and proxy handling into Provider

(cherry picked from commit 0caf24af50)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 5d20bd2832 vfs: add Busy to tell if a VFS still has data to write
Busy returns true if the VFS has files open for write or files in the
VFS cache which are open or waiting to be uploaded. It is for code
which wants to shut a VFS down only once that wouldn't lose any work.

(cherry picked from commit 4fa2a5b02c)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 2f84013eec lib/cache: add SetCanExpire to keep entries which are still busy
An entry is expired when it hasn't been used for a while, but the user
of the cache may know it is still doing something in the background.
SetCanExpire sets a function which is asked before expiring an entry
and can keep it in the cache until it is ready to go.

The function is called without the cache locked as it may take a
while, so an entry which is used in the meantime isn't expired.

(cherry picked from commit 49d8566ec5)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 675f9c3684 serve webdav: fix transfers failing after 5 minutes with --auth-proxy
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so an upload or download lasting longer than that had its backend
shut down under it and failed.

Each request now holds the backend it uses until it has been served.

This was introduced in v1.75.1 by

f425f8d46 serve: refactor VFS and proxy handling into Provider

(cherry picked from commit 7e6d6fa838)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood b8a697888c serve http: fix downloads failing after 5 minutes with --auth-proxy
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so a download lasting longer than that had its backend shut down
under it and failed.

Each request now holds the backend it uses until it has been served.

This was introduced in v1.75.1 by

f425f8d46 serve: refactor VFS and proxy handling into Provider

(cherry picked from commit 8457da3c8a)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood dddf808b47 serve ftp: fix transfers failing after 5 minutes with --auth-proxy
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each FTP command counted
as a use, so an upload or download lasting longer than that had its
backend shut down under it and failed with "context canceled".

Each FTP command now holds the backend it uses until it has finished,
which for a download is when the file is closed.

This was introduced in v1.75.1 by

f425f8d46 serve: refactor VFS and proxy handling into Provider

(cherry picked from commit ee0a70f875)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 451346f01c sftp: fix failed uploads leaving disk space in use on the server - fixes #10033
When an upload failed during the transfer, the partial file was
removed but its handle was never closed. The server kept the deleted
file open, so its space stayed allocated until the pooled connection
was closed, which in rcd, mount or serve could be until rclone exited.

This closes the handle before removing the partial file.

(cherry picked from commit d545615c16)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 6fadaa46aa filescom: document that names with / or \ in can't be stored and ignore their tests
Files.com now treats the replacement characters rclone uses for / and
\ in file names (/ and \) as if they were the originals, and the
names . and .. as if they were . and .., and rejects them all with
"Invalid path". There is no other way of encoding these names, so
document the restriction and ignore the integration tests for them.

(cherry picked from commit ed92e4446e)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 563919b0c0 webdav: fix "XML syntax error" on paths which don't exist with ownCloud 10.16
ownCloud 10.16 answers a PROPFIND for a path which doesn't exist with
a 207 Multi-Status response, rather than a 404, with a body which
starts a multistatus document then appends a Sabre NotFound error to
it. This isn't valid XML so rclone failed with

    read metadata failed: XML syntax error on line 2: expected attribute name in element

whenever it was pointed at a directory which didn't exist yet.

Detect the NotFound error in the response and treat it as a 404.

(cherry picked from commit f217c8a2aa)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood 1ecdeb4466 chunker: fix panic listing a directory after an interrupted upload with meta_format none
With meta_format = none, listing a directory which contained the
temporary chunks of an interrupted upload, but no completed chunks for
that file, panicked with "invalid chunked object".

The listing made a placeholder object for the file on seeing a
temporary chunk, which then had no chunks in it when it was validated.
Only make the placeholder on seeing a data chunk, so files which have
nothing but temporary chunks are ignored as they are when metadata is
in use.

(cherry picked from commit 475997b6de)
2026-10-08 17:17:50 +01:00
Nick Craig-Wood b85a681721 azureblob: use the released Azure SDK for Apache Arrow listing
The Blob Listing with Apache Arrow feature is now public and shipped
in azblob v1.8.1, so the temporary backend/azureblob/arrowlist package
which implemented it on top of the previous SDK has been removed and
the backend now uses the SDK's own listing pager with ResponseFormat
set to Arrow.

As the SDK client handles every credential type this also makes Arrow
and parallel listing work with connection_string auth, and the
use_arrow_list and list_parallelism options are no longer hidden.

(cherry picked from commit 3b9f0a05b5)
2026-10-08 17:17:49 +01:00
GhostCoder6969 61b8a90db8 docs: fix typo in speed test help text
Co-authored-by: bounty-agent <bounty@example.com>
(cherry picked from commit 7ebef790b4)
2026-10-08 17:17:12 +01:00
Nick Craig-Wood 419c5f9844 archive: fix a squashfs path to a single file listing its whole directory
Pointing the archive backend at a file inside a squashfs image, for
example `rclone cat :archive:image.sqfs/dir/file.txt`, listed every
file in the directory containing it instead of just that file. The zip
archiver already behaved correctly.

The squashfs archiver now remembers the file the root points at and
only exposes that one, as zip does. Its Root() includes the file so
that the fs cache does not hand back the Fs for the whole directory in
its place.

(cherry picked from commit 67afe55a83)
2026-10-08 17:17:12 +01:00
Neil Cawse 50d2430b4a vfs: stop setting a file's modtime from discarding its cached data
Setting the modtime of a closed file refreshed the cache item's
fingerprint before the new modtime had been applied to the remote
object. The next open then saw a fingerprint mismatch, logged "removed
cache file as stale (remote is different)" and downloaded the whole
file again, even though only the modtime had changed.

Refresh the cache fingerprint again once the remote modtime has been
set.

(cherry picked from commit 9380daed0d)
2026-10-08 17:17:12 +01:00
Neil Cawse cd6941ed93 vfs: fix reads returning zeros after a file changes during handle caching
When a file was reopened within the --vfs-handle-caching grace period
(default 5s) after its remote fingerprint changed - for example after a
touch or any other modtime change - _checkObject removed the stale
cache file and open recreated an empty one, but nothing sized it. The
next GetSize stat'd the empty file and set the item size to 0, so
ReadAt's _ensure clamped the request to nothing and skipped the
download, then the size check zero-extended the file and returned the
zeros. Reads through a mount returned the correct length but zeros for
the first read (128 KiB through FUSE).

This is easy to hit with git on a --vfs-cache-mode full mount: git
freshens pack files with utime, and a concurrent reader then sees
"not a GIT packfile".

Size the recreated cache file from the object before opening it, as a
normal open does.

(cherry picked from commit e0fb01084d)
2026-10-08 17:17:11 +01:00
TastyHeadphones 8290a478c8 oauthutil: fix Renew.Shutdown nil timer panic and race
Shutdown was calling expiryTimer.Stop without holding ts.mu, so it could
panic if OnExpiry had not created the timer yet, and raced the write in
OnExpiry. Hold the lock and skip Stop when the timer is still nil.

Fixes #9980

(cherry picked from commit 80e462d48c)
2026-10-08 17:17:11 +01:00
Rohit Behera d89f30a136 docs: fix the expanded --bwlimit timetable example
It was missing a space between two entries, so it didn't parse.

(cherry picked from commit e497464b60)
2026-10-08 17:17:11 +01:00
Rohit Behera f8899ca982 fs: fix --bwlimit timetables written out of order using the wrong limit
Until the first time slot of the week, the timetable used its last entry
as the limit carried over from the week before. That is only the latest
slot of the week when the entries are in order, so a timetable written
weekend first, like "Sat-00:00,off Mon-00:00,1M", limited Sunday to 1M
instead of leaving it unlimited.

Carry over the latest time slot of the week instead.

(cherry picked from commit 124f3bd830)
2026-10-08 17:17:11 +01:00
Dirk Petersen d77c1bf5c6 s3: fix storage class missing from --s3-versions listings
When listing with --s3-versions or --s3-version-at the storage class
of each object was dropped, so it read as STANDARD unless the object's
metadata was fetched separately. This meant backend restore skipped
objects in GLACIER or DEEP_ARCHIVE with "Not GLACIER or DEEP_ARCHIVE
or INTELLIGENT_TIERING storage class", and lsf --format T showed the
wrong tier.

This happened because ObjectVersion.StorageClass has a different type
from Object.StorageClass so the generated setFrom helper does not copy
it. Convert it explicitly after the setFrom call.

(cherry picked from commit e5eaf414f0)
2026-10-08 17:17:11 +01:00
hsdfat 19bf14aea4 docker: fix files uploaded with the wrong mime type - fixes #6384
The Docker image had no /etc/mime.types, so MIME types came only from
Go's built-in table plus rclone's small extra list, and many
extensions uploaded as application/octet-stream.

This installs Alpine's mailcap package to fix the problem.

(cherry picked from commit 3cdf855547)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 353167fd38 selfupdate: fix --version X.Y depending on ./ in the download site links
To find the latest patch release of a minor version, for example
`rclone selfupdate --version 1.75`, selfupdate searched the listing of
downloads.rclone.org for href="./vX.Y.Z/". The leading ./ is a detail
of how Caddy's file server writes its links. A listing which linked to
the same directories as "vX.Y.Z/", which is an equally valid relative
URL, made selfupdate fail with "could not find the minor release".

This makes the ./ optional in the pattern, so selfupdate no longer
depends on which program wrote the listing, and takes the version from
a capture group rather than from fixed offsets into the match.

The listings written by rclone index now include the ./ for the
benefit of rclone versions without this fix, so this is to remove the
dependency for the future.

(cherry picked from commit 8cd43746a3)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 09e1967c73 serve s3: fix TestEtagHashAuto on Windows
The temporary directory contains a drive letter colon so it needs
quoting in the crypt connection string, otherwise the password
parameter is dropped.

(cherry picked from commit 9dc8b71ae9)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 1d1eb94c34 build: fix lint errors from golangci-lint v2.14.0
The newer revive flags an exported function returning an unexported
type and a redundant type in a var declaration.

(cherry picked from commit 881cedd348)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood fd3b9bd383 build: fix TestS3Minio by switching to pgsty/minio as quay.io image is gone
MinIO no longer publishes images on quay.io (nor Docker Hub or ghcr.io)
so pulling quay.io/minio/minio returns "unauthorized". pgsty/minio is a
maintained community fork with the same entrypoint layout.

(cherry picked from commit 220c65f81d)
2026-10-08 17:17:11 +01:00
nielash 6676f2673e fs: retry "network is unreachable" and "network is down" errors
ENETUNREACH and ENETDOWN were not in the list of retriable errors on
non-Windows platforms, so a single failed connection aborted the
transfer instead of being retried as a low level retry. The Windows
list already includes the equivalent WSAENETUNREACH and WSAENETDOWN.

This is easy to hit on a host with IPv6 enabled but no IPv6 route: if
the A lookup fails transiently while the AAAA lookup succeeds, the only
address to dial is IPv6 and the connect fails with ENETUNREACH. A retry
resolves again and normally succeeds.

(cherry picked from commit 94e319d2fa)
2026-10-08 17:17:11 +01:00
Roland f6af51420f fs: fix bandwidth limits below 1 KiB being multiplied by 1024 - fixes #9958
BwPair.String printed a sub-KiB bandwidth as a bare number, and Set
reads a bare number as KiB, so a rate under 1 KiB grew by a factor of
1024 whenever it went through a string. rc core/bwlimit reports the
current rate in that format and accepts it back, so reading the limit
and setting it again raised it.

63c4fef27 fixed the same corruption for config values by suffixing bare
numbers with B inside Option.String. Move that into a SizeSuffix method
and use it from BwPair.String as well.

(cherry picked from commit 60bdc6d454)
2026-10-08 17:17:11 +01:00
Harsh Raj Singhania a755455609 serve webdav: escape filename in zip download Content-Disposition header
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.

Fixes #9962

(cherry picked from commit 14a1359c96)
2026-10-08 17:17:11 +01:00
Harsh Raj Singhania 0e6ed13131 serve http: escape filename in zip download Content-Disposition header
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.

Fixes #9962

(cherry picked from commit 661484f36c)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 96d2633836 serve s3: fix CopyObject of a missing object not returning NoSuchKey
Copying an object onto itself, as clients do to replace its metadata,
reported success when the object didn't exist, and copying a missing
object to a different key failed with an internal error. Both now fail
with NoSuchKey as S3 does.

(cherry picked from commit ce5351c52e)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood f625ac98a6 serve s3: fix --etag-hash auto crashing or using the wrong hash with --auth-proxy
With --etag-hash auto the hash for the ETags was chosen once from the
remote serve s3 was started with. With --auth-proxy there is no such
remote so serve s3 crashed on startup, and when started by the rc it
was the wrong remote, so users whose backends lacked that hash got no
ETags.

The hash is now chosen from the backend of the user making each
request.

(cherry picked from commit 746eac73ef)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 19068b6608 serve s3: forget the metadata of deleted objects
The metadata of every object uploaded was kept in memory for as long as
the server ran, even after the object was deleted, so it grew without
limit and the metadata of a deleted object reappeared on any object
later created at the same key other than through serve s3.

Deleting an object now forgets its metadata.

(cherry picked from commit c350f8ebbf)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood ee40507d38 serve s3: list objects lazily so paging a deep hierarchy is fast - fixes #9855
Listing a prefix with no delimiter walked the entire subtree below it into
memory and only then sliced out the requested page, so every page of a
listing cost a full traversal of the tree.

Walk the tree lazily instead, stopping as soon as the page is full and
skipping the subtrees an earlier page already returned. A page now costs a
number of directory reads proportional to the keys it returns rather than to
the size of the subtree, and ETags are computed only for the objects that
are actually returned.

Entries are emitted in the order their keys have in a flat keyspace, with a
directory sorting as if it carried its trailing slash, so that "a.txt" comes
before "a/b" as it does in a real S3 bucket. Without this a resumed listing
would silently skip keys across a page boundary.

(cherry picked from commit 04697cc02a)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood b4cb0426a3 vfs: fix a VFS being reused while it is being shut down
When the last user of a VFS shut it down at the same time as a new
user of the same remote asked for one, the new user could be given the
VFS being shut down, with its cache and background tasks stopped.

Only a VFS which is still in use is now reused, otherwise a new one is
made.

(cherry picked from commit c62aa2adc9)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 5977137b3b vfs: add Hold to keep a VFS from being shut down while in use
A VFS is shut down when the last of the references to it from New is
given back with Shutdown. Hold takes another reference, for code using
a VFS it didn't create itself, which may otherwise be shut down under
it.

(cherry picked from commit 1dab0f3abd)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 512f6193a2 operations: fix TestMultithreadCopy leaving files behind when multipart upload is skipped
When a backend rejected the multipart upload as too small, the subtest
skipped without removing its local source file, so the following
upload subtests failed their local listing checks.

(cherry picked from commit ee9c228012)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 46cf167c22 internxt: fix server-side directory move failing after a gateway timeout
Moving a directory can take longer than the API gateway allows, so the
request fails with a 520 or 502 error even though the move completes.
The retry then failed with "Folder ... was already moved to that
location (status 409)".

Treat that error as success.

(cherry picked from commit 0c52b183d0)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 2d24951658 internxt: fix "directory not empty" and stale directories after moves and deletes
The Internxt API serves listings from read replicas which lag behind
writes, so for a short while after files or directories are moved or
deleted they can still be listed in their old location.

This caused removing a directory which had just been emptied to fail
with "directory not empty" (eg when moving a directory without server
side directory moves or purging a directory) and a directory which had
just been moved to be found in its old location.

Remember the directories this process has moved or deleted and ignore
directory and file entries which contradict that when listing, finding
directories and checking a directory is empty before removing it.

(cherry picked from commit 7ff5da8517)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood ad9ad335f4 internxt: fix server-side moves failing with "Not Found" or "already exists"
Moving a file into a directory which had just been created failed with
"Not Found (status 404)", and moving a file over one which had just been
deleted (as sync does with --backup-dir and --suffix) failed with "A file
with the same name already exists in destination folder (status 409)".

The API checks moves against read replicas which lag behind writes, so
retry these errors until the replicas catch up.

(cherry picked from commit ee26daecd8)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 7d3e148212 internxt: fix sync with --backup-dir or --suffix deleting the backed up file
The Internxt API serves lookups and listings from read replicas which
lag behind writes, so for a short while after a file is moved it can
still be returned from its old location.

When sync moved a file into the backup location and then uploaded its
replacement, the upload could find the moved file under its old name
and overwrite it. Overwriting renames the existing file by UUID and
deletes it once the upload succeeds, so the file that had just been
moved into the backup location was deleted.

Remember the files this process has moved or deleted for a minute and
ignore lookups and listing entries which contradict that.

(cherry picked from commit f068abd607)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 2ff47e8274 drime: fix deleted files and directories still being listed with hard_delete
With hard_delete set, deleted files and directories carried on being
listed for about a second afterwards because the Drime server doesn't
invalidate its cache of the parent folder listing when entries are
deleted forever. This made removing a directory straight after
emptying it fail with "directory not empty".

Moving an entry to the trash does invalidate the cache, so with
hard_delete set rclone now moves the entry to the trash first and then
deletes it forever.

(cherry picked from commit a2e2b73725)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood f3bec5b1aa drime: fix server-side copy over an existing file leaving a "name (1)" copy
When server-side copying to a destination which already existed, the
Drime server gave the copy the name "name (1)" and rclone only renamed
it if the source and destination leaf names differed. The existing file
was then deleted leaving the copy under the wrong name.

The server refuses to rename an entry to a name which is already in
use, so this removes the existing file straight after the copy, then
renames the copy whenever its name differs from the destination name.

(cherry picked from commit eb10c48a17)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood d418a5a882 premiumizeme: fix uploading files with ";" in their names
The premiumize.me upload server has started truncating the multipart
file name at the first ";", so uploading "a;b.txt" created a file
called "a" and the upload then failed with "object not found".

Directory creation and renames still accept ";", so files whose names
contain ";" are now uploaded under a temporary name and renamed into
place. The encoding is left unchanged so existing files and
directories containing ";" remain accessible.

(cherry picked from commit 7c18e1eb86)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood db96e30c5e vfs: fix TestDirMetadataExtension on remotes which can't upload empty files
(cherry picked from commit 034ac6275f)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 111fda1287 onedrive: document that shared with me shortcuts may fail to list on personal
Shortcuts to shared items (and Personal Vault) are stored as remote
items pointing at another drive and listing them can fail with
"The provided drive id appears to be malformed". This aborts a
--fast-list listing of the whole drive.

(cherry picked from commit 8c38d3068a)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 10d999e3ee rest: limit the size of HTTP response bodies read into memory
rest.ReadBody read the whole response body into memory with no limit.
It is used by the default error handler and by many backends' error
handlers and small API calls, so a server which answered with an error
status and then streamed an endless body could make rclone allocate
memory until it was killed.

ReadBody now reads at most 10 MiB (the same limit drainAndClose
already uses to discard unread bodies) and returns an error if the
body is bigger than that. Every caller reads small API responses -
error bodies, status documents and upload tokens - so no legitimate
response is affected.

Reported by @manus-pi

(cherry picked from commit 90e67915c8)
2026-10-08 17:17:11 +01:00
jxj 6a7dcf12c3 s3: fix version-at listings with URL encoded keys
When S3 returns URL-encoded keys from ListObjectVersions, URL encoding
can change their lexical order. This could make mergeDeleteMarkers
place a delete marker after older versions of the same key, so
--s3-version-at reported deleted objects as live.

Compare decoded keys while merging, while preserving the encoded keys
for the existing listing decode path.

Fixes #9948

(cherry picked from commit cfb90e3ebe)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood fecd1d578a onedrive: update docs for versions, links and time precision on personal accounts
Testing against OneDrive personal (free) and OneDrive for Business shows

- personal accounts now create versions on setting the modification
  time and can delete them, so --onedrive-no-versions and rclone
  cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS

See #9917

(cherry picked from commit ff02636fe4)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood af92ea943d vfs: fix crash reading the metadata of a file which is being written
With --vfs-metadata-extension set, looking up the metadata file of a
file which was open for write and not yet uploaded caused a nil pointer
panic. Such a file has no object to read the modification time from.

Use the modification time of the VFS node instead, which is valid
whether or not the file has been uploaded.

(cherry picked from commit 7a2d7c766d)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood b46bb7647b operations: fix TestDeleteFatalError, TestDirMoveMoveError and TestDirMoveContext on remotes
TestDeleteFatalError set --max-delete before writing its files, so on
chunker, which deletes while uploading, the setup failed.

TestDirMoveMoveError and TestDirMoveContext test the core DirMove logic
with a wrapping Fs. This fails on remotes without Move (eg s3, memory)
and on those whose objects don't belong to the wrapped Fs (eg archive),
so they now only run on local.

(cherry picked from commit ff958c999f)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 3f6b38c9d9 iclouddrive: fix potential crash looking up items
findItem read the response status code when the lookup failed, so a
failure with no HTTP response panicked.

Thanks to @manus-pi for finding this problem.

(cherry picked from commit 1e92520076)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood a1fba2e8c1 shade: fix potential crash in directory move
DirMove discarded the error from the destination check and read the
response status code, so a failure with no HTTP response panicked.
Other errors were reported as the destination existing; they are now
returned.

Thanks to @manus-pi for finding this problem.

(cherry picked from commit 35abcadfc7)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood dca3902fef imagekit: fix potential crash in rmdir and purge
Rmdir and Purge read the response status code before checking for an
error, so a failed DeleteFolder call with no HTTP response (a network
error, or purging the root which fails validation) panicked.

Thanks to @manus-pi for finding this problem.

(cherry picked from commit e2cd9a5dfb)
2026-10-08 17:17:11 +01:00
phatlc 06387f878a serve dlna: log unescaped paths - fixes #7370
The request log printed the escaped URL, so non-ASCII file names showed
up as long runs of %XX escapes. Log the unescaped URL path instead, as
the other serve commands do.

(cherry picked from commit 556940ea44)
2026-10-08 17:17:11 +01:00
phatlc 13ae25b561 vfs: fix AddVirtual ignoring isDir
VFS.AddVirtual always added a file entry to the directory cache, even
when called with isDir set, so a virtual directory showed up as a file
and nothing could be added inside it.

See #9310

(cherry picked from commit ebb1cc1221)
2026-10-08 17:17:11 +01:00
phatlc 5ea1851d52 smb: save the user name in the config even if it matches the current user - fixes #9356
The default for --smb-user was the name of the user running rclone
config, and rclone does not write defaults to the config file, so
entering your own user name left it out. A remote made that way then
logged in as whoever ran it later, e.g. root under systemd.

Make the default blank and look up the current user when the remote is
used, as the ftp backend does. Existing configs work as before.

(cherry picked from commit 3c51b96774)
2026-10-08 17:17:11 +01:00
phatlc 2b65691945 operations: stop --copy-dest replacing files with --immutable
When the source matched a file in --copy-dest, copyDest server-side
copied it over an existing destination which differed, so sync, copy
and copyto with --copy-dest could modify a file --immutable should have
protected.

Leave such a destination for the caller, which rejects it with
ErrorImmutableModified as for any other modified file.

(cherry picked from commit dc98c216f2)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 5de8a01717 rc: reject out of range integer parameters instead of truncating them
This was spotted by CodeQL after this change was merged:

976d05e1d rc: fix rc API accepting an out of range number and overflowing 64 bits

(cherry picked from commit 7723091be9)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 4d50f6d878 docs: remove broken SecureBuild links
(cherry picked from commit 1dc2516f08)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 34ad19a1eb box, serve s3: fix log messages with bad format strings
The box backend logged an int64 with %q which printed
%!q(int64=123) instead of the sequence ID.

serve s3 passed the message from gofakes3 as the format string, so
any % in it was interpreted as a formatting directive and the message
was mangled.

(cherry picked from commit 1c3e432c3f)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 5bc26e57b4 rc: make job/status and job/list require authentication
job/status returns the complete output of the job so it is as
sensitive as the call which started the job, e.g. config/dump started
with _async returns the config file including any secrets. It was
possible to read this without authentication where an authenticated
and an unauthenticated rc server share the same process, e.g. rclone
gui --rc.

An unauthenticated client can't start jobs on a server which needs
authentication, other than with the calls which don't need it, so
this shouldn't affect existing users.

(cherry picked from commit 939f82d908)
2026-10-08 17:17:11 +01:00
Nick Craig-Wood 8d15b2bb75 log: fix race when adding a log output while logging
Handle read the list of extra outputs without holding the mutex, so
calling AddOutput while logging caused a data race.

This reads the extra outputs once under the mutex.

(cherry picked from commit 4cd6359ae9)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood c062ec1b12 jobs: fix tests failing and racing when run with -count > 1
TestRcJobList listed the jobs left running in the global job list by
the previous run of the tests, so reset the global job list first.

(cherry picked from commit dd07bb33ba)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood 7b647c729b operations: don't sleep for a Retry-After error when the transfer is cancelled
The wait to obey a Retry-After error from the server used
time.Sleep() so it carried on sleeping when the context was
cancelled, e.g. by job/stop or --max-duration, and it also slept
after the last try when there was nothing left to retry.

Copying a file also logged the low level retry number counting from
0 rather than 1 as everything else does.

(cherry picked from commit 88bd49fab8)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood 4347404119 operations: fix hang and lost context when moving a directory file by file
When the backend has no DirMove method, operations.DirMove moves the
objects one by one. If one of the moves failed the movers stopped
without receiving the rest of the objects, so DirMove blocked forever
once the channel filled up. This could be seen renaming a directory on
a mount of a backend without DirMove, such as s3.

The moves were also done with context.Background() instead of the
caller's context, so they didn't have the caller's config, couldn't
be cancelled and weren't counted in the caller's stats.

(cherry picked from commit b82a024de0)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood 366b188a5a operations: fix hang when deleting files and a fatal error occurs
When a fatal error such as --max-delete being reached stopped the
deletions, the deleters returned without receiving the rest of the
objects, so whatever was sending them blocked forever once the
channel filled up.

For example "rclone delete --max-delete 1" on a directory with more
files than --checkers hung instead of returning the error.

The deleters now keep receiving objects without deleting them after a
fatal error.

(cherry picked from commit caa341878f)
2026-10-08 17:17:10 +01:00
Jeremy Schoemaker 0dd95c2840 rc: fix large numeric parameters being rejected on 32 bit builds
Params.GetInt64 parsed string parameters with strconv.ParseInt(x, 10, 0).
A bitSize of 0 means int, which is 32 bits on 386, arm, mips and mipsle,
so any value outside the int32 range was rejected as out of range even
though it fits in the int64 the method returns.

Parameters reach the rc API as strings on the paths that matter here: the
rc server puts every URL query and form value into Params as a string, and
rclone rc turns each key=value argument into a string. So on a 32 bit build
operations/getfile could not be given an offset, count, head or tail beyond
2 GiB, and debug/set-soft-memory-limit could not be given a limit beyond
2 GiB, while the same commands worked on a 64 bit build.

Parse with a bitSize of 64 to match the declared return type. This is the
string half of the range checking that was added to the float64 branch of
the same function in 976d05e1.

(cherry picked from commit b5f83bdbf2)
2026-10-08 17:17:10 +01:00
Wang Chencheng efba10c065 operations: fix ignored error in rcat probe reads
Return the first non-EOF input error encountered while Rcat probes whether an
upload is small. Previously that error was ignored and the full probe buffer,
including bytes that were never read, could be passed to Put or PutStream.

(cherry picked from commit b93b73bcb0)
2026-10-08 17:17:10 +01:00
Vasek Sraier cc53a4cdac serve restic: prevent concurrent append-only overwrites
- Reject uploads when object lookup fails for reasons other than "not found".
- Serialize uploads to the same object to prevent races between the
  existence check and the write.

(cherry picked from commit d632f8bb51)
2026-10-08 17:17:10 +01:00
Acts1631 89c6cb0911 compress: fix crash on ranged reads when gzip metadata is corrupted
Gzip metadata is read from the wrapped remote and could contain an
invalid block size or incomplete block index. A range read could then
panic in the seekable gzip reader.

Validate the gzip sidecar invariants before constructing a reader so
malformed remote metadata returns an error instead of crashing rclone.

(cherry picked from commit c8d60a67fc)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood a5f9653807 fstest: remove the TestS3MinioEdge test server
The minio/minio:edge image it ran has gone from Docker Hub and there
is no equivalent tag on quay.io, so this test remote can no longer be
started.

(cherry picked from commit a401763cf6)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood 620ce60d00 fstest: make test server start fail fast when docker run fails
The start function was run inside an if condition, where bash ignores
errexit, so a failed docker run was not noticed. The script then
printed its connection details anyway and the test spent 100 seconds
trying to connect before failing with a message that hid the real
error.

Run start in a subshell with errexit on and check its status
explicitly so the failure is reported immediately with docker's
error message.

(cherry picked from commit 4cf1da0d4f)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood fa75e2b178 build: fix TestS3Minio by pulling minio from quay.io as it has gone from Docker Hub
The minio/minio repository has been removed from Docker Hub so the
TestS3Minio test server could not be started and every Linux CI run
failed. quay.io/minio/minio still serves the final release so use
that instead.

(cherry picked from commit e0d846f29e)
2026-10-08 17:17:10 +01:00
KBS b5055efecf rc: fix rc API accepting an out of range number and overflowing 64 bits
float64(math.MaxInt64) rounds up to 2^63, so x > math.MaxInt64 in
GetInt64 lets 2^63 through to int64(x), which is out of range.

(cherry picked from commit 976d05e1dd)
2026-10-08 17:17:10 +01:00
foecmke ab8a00e159 docs: update --onedrive-hard-delete to mention personal account support
(cherry picked from commit b1a10fe17e)
2026-10-08 17:17:10 +01:00
Kunpeng Xie 70700de356 accounting: stop averaging when the last check finishes
Release the averaging goroutine when checks outlast transfers, with a regression
test for the completion order.

Assisted-by: OpenAI Codex
(cherry picked from commit 4f0148db0c)
2026-10-08 17:17:10 +01:00
tomaszni 277d3ea146 oracleobjectstorage: upload empty streams without multipart
OCI rejects a multipart completion request with no parts. Probe unknown-size
streams through a buffered reader and use a regular upload when the stream is
empty. Peek preserves a non-empty stream for the selected upload path.

(cherry picked from commit e85836d4c7)
2026-10-08 17:17:10 +01:00
Eugene fc65a1b9cb cmount: ignore com.apple.* xattrs on the macOS FSKit backend
(cherry picked from commit 0cb6ddada9)
2026-10-08 17:17:10 +01:00
jzunigax2 b1677718cd internxt: fix lookups of files starting with a dot and dropped uploads
Internxt stores a file as a (plainName, type) pair and never derives the
split itself, so it is a convention shared between clients. This backend
split at the final dot, storing and looking up ".bashrc" as an empty name
of type "bashrc", where the web, desktop, Linux and macOS clients all keep
the leading dot in plainName. List rebuilt the full name so such files
appeared, but NewObject looked them up by the split and missed them.

(cherry picked from commit 77ec281072)
2026-10-08 17:17:10 +01:00
jzunigax2 4dde74364b refactor: streamline file existence checks and metadata retrieval
- Replaced the preUploadCheck function with findFile for better clarity and efficiency in checking file existence.
- Introduced splitNameExt to encapsulate name and extension parsing logic.
- Updated NewObject and Update methods to utilize findFile for improved file metadata handling.
- Enhanced error handling and reduced redundant code in file checks.

(cherry picked from commit e441773d24)
2026-10-08 17:17:10 +01:00
n4n5 bb7b312b19 docs: extend librclone instructions for gomobile
(cherry picked from commit 8fafc08dd1)
2026-10-08 17:17:10 +01:00
Vladimir Babin 43162d20e0 archive: fix crash when creating archive to stdout - fixes #9910
When rclone archive create is run without a destination the archive
should be written to stdout, but ArchiveCreate called
CheckValidDestination on the nil dst and panicked with a nil pointer
dereference. Skip the destination check when there is no destination.

Also remove a leftover debug Printf that wrote to stdout before the
archive data.

Signed-off-by: Vladimir Babin <vovababin@gmail.com>
(cherry picked from commit 3342e34f58)
2026-10-08 17:17:10 +01:00
maximilize ba3a6c17ce docs: clarify --password-command quoting for a path with spaces
(cherry picked from commit 0e19ed565f)
2026-10-08 17:17:10 +01:00
enkvadrat 7c1a79e071 docs: add padding to footer card
This is mostly visible in dark mode, as a side effect off adding the class,
the background of the card also changed to be dark-gray.

(cherry picked from commit c2a5884ad9)
2026-10-08 17:17:10 +01:00
ZRHann 0d582bfffe webdav: fix duplicated listing entries after retried PROPFIND
(cherry picked from commit 812e693fd7)
2026-10-08 17:17:10 +01:00
shaurya 459501873b docs: fix broken --check-filename self-link in bisync docs
The link pointed at a bare relative path (--check-filename) instead of
the in-page anchor for the ### --check-filename heading further down the
page, so it 404s on the rendered docs site. Point it at #check-filename
(Hugo strips leading dashes when generating header anchors).

---------

Co-authored-by: no-hup <19599684+no-hup@users.noreply.github.com>
(cherry picked from commit c3ba184611)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood f1590144a4 docs: update sponsors
(cherry picked from commit 7b3a4e5144)
2026-10-08 17:17:10 +01:00
subomi 9d25ce63a3 docs: lshelp: explain that a directory on a bucket-based remote is a key prefix
Listing a bucket without -R shows each prefix as one directory entry, so the
objects under it are not in the output. This is easy to read as a truncated
listing rather than as one level of a hierarchy.

Fixes #9797

(cherry picked from commit 8c32435bec)
2026-10-08 17:17:10 +01:00
dependabot[bot] 18c39dc283 build: update google.golang.org/grpc to 1.85.0-dev.0.20260825072537-93e31b48545e to fix CVE-2026-84445
A vulnerability exists in gRPC-Go servers configured with
xds.NewGRPCServer() where a crafted request missing both :authority
and Host headers can cause a server panic, resulting in a Denial of
Service (DoS).

This update fixes the problem.

(cherry picked from commit ef6968730f)
2026-10-08 17:17:10 +01:00
youdie006 79ccf3d26b fs: fix about showing a negative total when a quota reaches the int64 maximum
NewUsageValue exists to clip an oversized quota to the maximum value of an
int64, which is what dc95f36bc added it for when Box raised the Enterprise
space_amount to 1e+18 and started returning it as a float.

For the float64 instantiation the guard misses its own boundary.
float64(math.MaxInt64) is not 2**63-1, it rounds up to 2**63, so a quota of
exactly 2**63 fails the comparison and falls through to the int64 conversion,
which the spec leaves implementation dependent for an unrepresentable value.
On linux/amd64 it wraps:

    Before: rclone about -> Total=-9223372036854775808
    After:  rclone about -> Total=9223372036854775807

A negative total is not just a wrong number. vfs.Statfs documents -1 as "not
known", vfs.fillInMissingSizes branches on total < 0, and serve sftp only
computes its usage percentage when total > 0, so the value is read back as a
missing quota.

The int64 and uint64 instantiations are unaffected, since for them
T(int64(math.MaxInt64)) is exact and clipping MaxInt64 to MaxInt64 is a no-op.

(cherry picked from commit 52ac7e0e18)
2026-10-08 17:17:10 +01:00
youdie006 d659d10afd fs: make BwTimetable.Set replace the timetable instead of appending to it
Set built the timetable with *x = append(*x, ts), so setting a bandwidth
timetable on a value that already held one kept both schedules. The single-value
branch of the same function has always done *x = BwTimetable{ts}, and the other
multi-token Set methods in this package build into a local and assign at the end.

The visible effect is through the rc API. The "main" options block registered in
fs.RegisterGlobalOptions is the live globalConfig, and options/set reshapes JSON
straight into it, so

  rclone rc options/set --json '{"main": {"BwLimit": "Mon-10:00,1Mi"}}'

added to the running daemon's timetable rather than replacing it, and the older
slot kept winning: LimitAt for a Sunday returned the previous 10Mi. The same
applies to a _config override on a single call, since AddConfig shallow-copies
the global.

Building into a local also stops a failed parse from leaving the previous
timetable partly overwritten, which the existing error cases already expect.

(cherry picked from commit 5bbc5d5545)
2026-10-08 17:17:10 +01:00
Aditya b8431d4f50 s3: disable signing Accept-Encoding for Ceph and Linode - fixes #8206
Ceph RGW (and Linode Object Storage, which is Ceph-backed) can break
SigV4 when Accept-Encoding is included in the signature, especially
when a reverse proxy rewrites that header. GCS already sets this quirk;
apply the same default for Ceph and Linode as suggested in #8206.

(cherry picked from commit ea589de941)
2026-10-08 17:17:10 +01:00
tomaszni 6c566ca014 oracleobjectstorage: fix SSE-C server-side copies
Set the OCI source SSE-C request headers when using a customer key.
Server-side copies need these headers to decrypt the source object, in
addition to the existing headers that encrypt the destination.

(cherry picked from commit 3eee2c0dd2)
2026-10-08 17:17:10 +01:00
Murat Topcu cd7cef72db selfupdate: fix TestInstallOnLinux panicking when the build is the latest beta
The test asks InstallUpdate to install the latest beta into an
unwritable file and expects an error. When the binary under test
reports exactly the latest beta version (as make quicktest does right
after a beta is published from the same commit), InstallUpdate
correctly decides there is nothing to do and returns nil, and the test
then dereferences the nil error and panics.

Pin fs.Version to a fixed old value for the duration of the test so an
update is always attempted, and use require.Error so a missing error
fails the test instead of crashing it.

(cherry picked from commit 89fc14059e)
2026-10-08 17:17:10 +01:00
phatlc 33ee8c0ef8 serve docker: fix volume path being lost when the plugin restarts
applyOptions consumes the "path" option into vol.Path rather than leaving
it in vol.Options, but restoreState rebuilt the options with only fs and
type. The explicit path was therefore dropped when the plugin restarted,
and since fsString is rebuilt from those options the volume was remounted
at the root of the remote instead of at its subpath.

Before this change a volume created with type + path lost its path
completely, and one created with remote + path silently fell back to the
path of the connection string. With a backend whose credentials are
scoped to the subpath the restored mount then failed every operation
rather than serving the wrong directory.

Feed the persisted path back like fs and type, so applyOptions applies
the same precedence on restore that it applies when the volume is
first created.

Fixes #9853

(cherry picked from commit 9ac29e3b35)
2026-10-08 17:17:10 +01:00
phatlc bb0633ee43 dropbox: match shared-folder and received-file names case-insensitively - fixes #9706
The Dropbox backend advertises CaseInsensitive: true, but the two
shared-mode lookup helpers compared names with an exact, case-sensitive
==, so a shared folder or received file named "Project" could not be
found when requested as "project". Use strings.EqualFold in both
findSharedFolder and findSharedFile to honour the advertised
case-insensitivity.

Fixes #9706

(cherry picked from commit b549554c31)
2026-10-08 17:17:10 +01:00
phatlc 347738843f dropbox: fix shared folder mount for roots nested more than one level deep
In shared_folders mode NewFs derived the shared folder name with
path.Dir(f.root), which returns the parent path rather than the first
path component. For a root like "SharedFolder/subdir/deeper" this yielded
"SharedFolder/subdir", which findSharedFolder cannot match, so NewFs
failed with ErrorDirNotFound. Use the first path component of the root,
as the shared_folders option documents, so deeply nested roots mount.

Fixes #9705

(cherry picked from commit ac7cfcc848)
2026-10-08 17:17:10 +01:00
ferrumclaudepilgrim 4d547c4b7b vfs/vfscache: fix hang when the cache cleaner is disabled
KickCleaner sets the out of space flag, kicks the cleaner and then waits for
that flag to clear. Only the cleaner clears it, and the cleaner returns
immediately when the cache poll interval is not positive, so when it is
disabled nothing ever reads the kick or clears the flag and the caller waits
forever.

It now returns straight away in that case, under the same condition the
cleaner itself uses to decide it is disabled. Callers already retry a bounded
number of times and then report the error, which is the right outcome when
nothing is going to free space.

(cherry picked from commit e724790620)
2026-10-08 17:17:10 +01:00
ferrumclaudepilgrim ab95b42328 fserrors: fix out of space detection on Windows - fixes #8011
IsErrNoSpace compared against syscall.ENOSPC. Go defines that constant on
Windows as a value in its application reserved range which no Windows API
returns, so the comparison could never be true there. A full disk on Windows
reports ERROR_DISK_FULL or ERROR_HANDLE_DISK_FULL instead.

Preallocation failures were still caught, because those return a separate
sentinel, but a disk that is already full fails at the directory creation or
at the open long before preallocation is reached. That is the case reported.

The errors are now held in a list which platform specific files add to in
their init, which is the shape retriable_errors already uses in this package,
and the comparison itself is unchanged. Windows appends the two codes that
lib/file already recognises when preallocation fails. Every other platform
keeps exactly the behaviour it had.

This also reaches the VFS cache, which uses the same helper and has no
preallocation path of its own, so its out of space handling has been inert
on Windows.

(cherry picked from commit ca41db095b)
2026-10-08 17:17:10 +01:00
Sanjay Kanth A 409ba3b1f4 docs: drive: document Branding step needed to publish own client_id
Google now requires an app homepage URL and privacy policy URL to be
set on the OAuth consent screen's "Branding" page before the "PUBLISH
APP" button becomes clickable, even for a personal single-user app.
The existing instructions jumped straight to publishing in step 9
without mentioning this, leaving the button greyed out with no
explanation of why.

Fixes #9854

(cherry picked from commit c875d89033)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood 274342326c dedupe: fix rename mode giving up after 100 names and make it faster - fixes #9860
Before this change `rclone dedupe --dedupe-mode rename` probed the
backend for each candidate `name-N.ext` in turn and gave up when it
had tried 100 names for a given object. With daily runs against the
same duplicated filename this ceiling was eventually reached and
rclone logged "Could not find an available new name". Each probe was
also a backend lookup, so a run against 99 existing names took
minutes on Google Drive.

The rename now uses the listing dedupe has already made to skip names
known to be taken without asking the backend, and only confirms the
final candidate with NewObject (the listing may be incomplete because
of filters). The suffix counter is shared between the objects being
renamed so no name is checked twice. The safety limit is raised to
10000 which, thanks to the listing, no longer costs a lookup per name.

(cherry picked from commit bc4a208e7e)
2026-10-08 17:17:10 +01:00
Nick Craig-Wood 97eee1fecf serve ftp: fix VFS leak when the server fails to start
The deferred cleanup in the constructor checked a local error variable
rather than the error being returned, so failures after the VFS was
created (such as an invalid --passive-port) never shut it down. Name
the error return so the cleanup sees the returned error.

(cherry picked from commit 9b9fd3f493)
2026-10-08 17:17:09 +01:00
Hakan İSMAİL 98274d294a serve, mountlib: test VFS release on shutdown and mount failure
(cherry picked from commit 30e79a017b - without the cmd/mountlib test as the mountlib fix isn't in this branch)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood 34aeb94efa serve webdav,http: fix crash when the server fails to start - fixes #9882
When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.

The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.

(cherry picked from commit f2a390b2d4)
2026-10-08 17:17:09 +01:00
Dhevenddra K G de09fecf1c docs: fix duplicated words in vfs and backend documentation
(cherry picked from commit 3d7b101c7f)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood b7b89b0708 serve docker: fix tests leaving unkillable processes and stale FUSE mounts
Writing to the mount with os.WriteFile made the Go runtime register
the file with its poller so the kernel then polled the file from
epoll_ctl and epoll_wait, sending POLL requests to the FUSE server
running in this same process. A thread waiting inside epoll cannot be
preempted by the runtime, so a garbage collection starting while such
a POLL was outstanding stopped the world for good - the test binary
could not be killed even with SIGKILL and the mount was left behind,
wedging anything that touched it.

Now we write through the mount with a descriptor straight from
open(2), which os.NewFile keeps out of the poller, check that it
really is out of the poller with SetDeadline, and check at the end of
the test that the mountpoint is unmounted.

In this commit we fixed the same problem for mount by running in a
subprocess however changing one write file routine here was much
easier than re-arranging the tests.

4a382c09ec mount: run tests in a subprocess to fix deadlock - #3259

Note that go-fuse (and hence mount2) works around this problem it by
forcing an early POLL it can answer with ENOSYS.

See: https://github.com/golang/go/issues/21014
(cherry picked from commit e855d2ed36)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood 4079b6f493 archive: fix listing entries with a leading slash as if they were in the root
The check that an entry returned by an archiver is a direct child of
the directory being listed normalised a parent of "/" to the root, so
an entry named "/x" passed as a child of the root while "x/" and
"dir//x" were rejected.

Decide by stripping the directory prefix and checking what is left
with sanitize.Leaf, which rejects an empty name, ".", ".." and any
name containing a "/". This also covers the leading slash case.

(cherry picked from commit b88e237e8c)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood bc482e79ff archive: fix zip file entries named for a directory causing confusion
A file entry in a zip whose name refers to a directory, such as ".",
"/", "" or "sub/.", was only skipped when it named the root of an
archive which was itself the root of the remote. When the archive was
found by listing its parent directory the entry appeared as a file
with the same name as the archive alongside the directory for it, and
copying the archive tried to write both. When the entry named a
subdirectory it appeared as a file alongside that directory, and with
that subdirectory mounted as the archive root the entry was taken to
be the single file the root points at, hiding every real entry.

Skip any file entry whose last path component is "", "." or "..",
checked on the raw name before it is cleaned or joined on the prefix.

(cherry picked from commit da352a2a1b)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood 7ef04886b9 archive: fix corrupt listings when listing a zip directory more than once
The zip archiver handed out its cached directory tree directly. Any
caller which filters a listing in place (as the core listing code
does) altered the cache, so later listings of the same directory could
be corrupted.

Return a copy of the cached listing instead.

(cherry picked from commit 68eab60564)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood b70ebaba7b seafile: fix corrupted uploads after a retried upload error
When an upload failed with a 500 error the upload was retried with a
new upload link but the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.

This fixes it by returning a RetryError instead so the caller retries
the upload with a fresh stream, which will fetch a new upload link.

(cherry picked from commit 6cdd0ea761)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood b03a9f8ea3 filescom: fix corrupted uploads after a retried upload error
When an upload failed with a retryable error the pacer retried the
whole upload with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.

This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.

It also makes 5xx errors from the upload storage servers retryable.
These come back from the SDK as a different error type to API errors
so were not being retried at all.

(cherry picked from commit fa43f10af2)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood 8a2bed2f71 pixeldrain: fix corrupted uploads after a retried upload error
When an upload failed with a retryable error the pacer retried the
whole PUT with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.

This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.

(cherry picked from commit f4cd80a535)
2026-10-08 17:17:09 +01:00
Shane McCarron df2a2d1127 fs/fshttp: fix TestCertificates leaking client cert/key onto global config
This was fixed in this commit in an inelegant way

399bc6a6a6 fshttp: don't send --header values to other hosts on redirect

The current commit fixes it properly with AddConfig.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1b4dea8dac)
2026-10-08 17:17:09 +01:00
PSR94 2e18447255 docs: update Huawei Drive client ID setup
(cherry picked from commit b4c598dbe4)
2026-10-08 17:17:09 +01:00
Nick Craig-Wood ef6bae24c5 Start v1.75.2-DEV development 2026-09-04 18:21:52 +01:00
Nick Craig-Wood 687d264b68 Version v1.75.1 2026-09-04 17:03:20 +01:00
Nick Craig-Wood 4158f63d2f Start v1.75.1-DEV development 2026-09-04 16:50:38 +01:00
Hakan İSMAİL c47832ad34 serve: fix VFS instance leaks on server startup failures and shutdown
(cherry picked from commit 216d2a8c76)
2026-09-04 16:39:12 +01:00
Nick Craig-Wood 60e96416df rc: silence deprecation lint warning in the old web gui plugin proxy
Raising the minimum Go version to 1.26 made staticcheck flag the
deprecated ReverseProxy.Director. This code is unmaintained and has
been removed on the development branch, so suppress the warning rather
than rewrite it.
2026-09-04 16:34:03 +01:00
Nick Craig-Wood c841c3bba3 build: disable staticcheck SA4023 to fix lint job timeout
The dataflow analysis behind SA4023, new in the staticcheck 0.8.0
bundled with golangci-lint v2.13.0, makes linting large packages more
than 10x slower (89s vs 7s for backend/s3 alone) which took the CI
lint job past its 30 minute limit. golangci-lint no longer enforces
its run timeout during analysis so the job ran until cancelled, and
the cancellation meant the lint cache was never saved, making every
subsequent run cold and guaranteeing the timeout repeated.

The check also produces false positives (eg claiming operations.Delete
never returns nil).

(cherry picked from commit 357c2a2b44)
2026-09-04 16:33:21 +01:00
Nick Craig-Wood 79fbc0842f fshttp: don't send --header values to other hosts on redirect GHSA-486v-q2wf-fp2r CVE-PENDING
The headers set with --header and --header-download are added to
every request by the rclone transport, including redirect hops which
net/http makes to other hosts, so a credential passed with --header
for one host could be sent to any host that server chose to redirect
to.

The transport now walks the redirect chain net/http records on each
redirected request and, once the chain has visited a host other than
the one originally requested, removes the headers rather than adding
them.

Also restore the global --client-cert and --client-key config after
TestCertificates so its temporary files are not used by later tests.

(cherry picked from commit 101c3e342c86ae65c7c743474d7319330f0eb0ca)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 22859b7e69 http: don't leak configured headers to other hosts or over plaintext on redirect GHSA-486v-q2wf-fp2r CVE-PENDING
The headers set with --http-headers are documented for passing
credentials such as Authorization or Cookie. The backend used the
default net/http redirect policy which copies all but a handful of
well known headers to any redirect target, so a redirect from the
configured server to another host would send those credentials to
that host, and a redirect from https to http would send them in
plaintext.

When headers are configured this installs a CheckRedirect function
which:

- removes the configured headers from every hop once the redirect
  chain has left the originally requested host
- refuses a redirect from https to http with an error

(cherry picked from commit 0adc0082dff7ef6ed1597418f85ac691478a6fa5)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 925fb4fb21 rest: add SameHost and check HTTPS downgrades against the original request GHSA-486v-q2wf-fp2r CVE-PENDING
SameHost compares two URLs by host name (case insensitively) and port
(treating the scheme's default port as no port) so redirect policies
can tell a real change of host from a server spelling its own host
differently, e.g. redirecting "https://example.com/" to
"https://EXAMPLE.com:443/".

The HTTPS downgrade check now compares the redirect target against
the original request rather than the previous hop, so a chain which
started on plaintext http, passed through an https server and came
back to http is no longer refused - nothing is being downgraded
relative to what the user asked for. A chain which started on https
and reaches http via any number of hops is still refused.

(cherry picked from commit fc7a64f61c2e4b364b5789f6ea17ad0485d8246e)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 96f298bdec archive: hide any archive entry which escapes the directory being listed GHSA-66hp-wgxq-6f5q
Whether an archive entry name can escape the archive's namespace was
left entirely to each archiver. Enforce it in the archive backend too.

List only passes on direct children of the directory listed and
NewObject only returns the object asked for, so a future archiver
which forgets to validate names cannot expose a traversal to fs/sync
and fs/operations.

(cherry picked from commit a6a7d95e081b91231b49c4004e8a2bff16da4cd8)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 60fb55dc6a archive: fix "directory not found" for archive paths containing "./" or "//" GHSA-66hp-wgxq-6f5q
The path inside the archive was compared against the cleaned entry
names without being cleaned itself, so `archive.zip/sub/./dir` or
`archive.zip/sub//dir` failed to list even though `archive.zip/sub/dir`
worked.

(cherry picked from commit 34636f34079585f3a8f883038483ece58d39b084)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 181ef190ce archive: fix zip entry named "." hiding every other file GHSA-66hp-wgxq-6f5q
A zip containing a file entry whose name refers to the archive's own
root (".", "/" or "") was presented as a single file called "." and
all its other entries disappeared. A file at the root can only be the
archive member the backend was pointed at, so with no root such an
entry is skipped like any other unsafe name.

(cherry picked from commit c9fac578aae7707faf340b58cf45465a5f698cbd)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood ad8cd41c49 archive: reject unsafe entry names when mounting squashfs images GHSA-66hp-wgxq-6f5q
Entry names read from a squashfs directory are not sanitized by
go-diskfs. The squashfs backend joined each leaf name onto its
directory to form the object's remote, so a crafted image could escape
its directory.

Use sanitize.Leaf to skip unsafe entries in List. A "\" is an
ordinary character in a file name on the systems squashfs images are
made on and in an rclone remote path, so it is deliberately not
rejected; making it safe for the destination is the destination
backend's job.

Skipped entries are logged at DEBUG with a single NOTICE count per
listing so a crafted image under a mount cannot flood the log.

(cherry picked from commit 63385c66175141b63bb53b3e42b50908649f8437)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 5dae3adbf5 archive: fix zip subdirectory root matching sibling directories GHSA-66hp-wgxq-6f5q
When a zip archive was mounted at a subdirectory root, readZip used a bare
strings.HasPrefix to decide which entries fell inside the root. This
matched on a raw string prefix rather than a path boundary, so mounting
root "foo" also exposed sibling entries such as "foobar/..." with their
names left uncorrected.

Require a path boundary when filtering by root.

(cherry picked from commit b444227264cee2a9307f03ba90c2a411e7eeb693)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 6507e13d5a archive: fix zip slip path traversal in untrusted zip files GHSA-66hp-wgxq-6f5q CVE-PENDING
The zip backend mounts a zip file as a browsable Fs. Go's archive/zip
does not sanitize entry names, and readZip applied path.Clean but did
not reject a cleaned name that still pointed outside the archive. A
crafted zip could make rclone copy/sync attempt writes outside the
intended destination.

Sanitize entry names with sanitize.Path - the same check used by
rclone archive extract - skipping any entry with a ".." path
component, whether separated by "/" or "\". A backslash is otherwise
kept as an ordinary character in the name, as archive extract does. It
is up to the destination backend to make names safe for its storage.

Skipped entries are logged as a single count per archive so a crafted
archive with many escaping entries cannot flood the log.

(cherry picked from commit 224fe97ac13105094651264a9e16ee3cf41ccf77)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood e4913c9e91 lib/sanitize: factor untrusted path sanitization out of archive extract
Move the archive entry name validation added for CVE-2026-59732 from
cmd/archive/extract into a new lib/sanitize package as sanitize.Path,
so the same check can be shared with the archive backend which mounts
archives as a filesystem.

sanitize.Path keeps the extract semantics - reject any name with a
".." path component, treating both "/" and "\" as separators - and
additionally cleans the name with path.Clean. This corrects two edge
cases in extract: a repeated "./" prefix ("././file.txt") is now fully
stripped rather than only the first, and a bare "." entry is now
treated as the archive root and skipped.

Add sanitize.Leaf, which rejects a name that is empty, ".", ".." or
contains a "/", for checking a single directory entry name read from
an archive.

The names handled are rclone remote paths, in which "/" is the only
separator and "\" an ordinary character, so Leaf does not reject a
backslash: making a name safe for its storage is the destination
backend's job (the local backend encodes "\" on Windows and refuses
paths which escape its root). Path's rejection of ".." between
backslashes is kept as defence in depth for extract.

(cherry picked from commit 4aab7cd450127e902509b4c8e3a1e4f5c12056df)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 28bf49d66f local: fix panic on Range request past the end of a symlink GHSA-p6m2-r3w9-mpxw CVE-PENDING
With --links/-l, a symlink is served as a .rclonelink object whose
content is the target path. A Range request with a start offset beyond
the target length (e.g. "Range: bytes=99999999999-") reached
openTranslatedLink and sliced the target string at that offset, panicking
with "slice bounds out of range".

Clamp the offset to the target length so an out-of-range start reads
empty, matching how a real file read past EOF behaves.

(cherry picked from commit 3fa32192c20f0b4cfd4f4b07636dc3445026041a)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood 17b0c03338 local: fix btime escaping the root via a planted symlink GHSA-f8g7-2xjc-7mfh CVE-PENDING
The birth-time (btime) write in writeMetadataToFile followed symlinks for
any object that was not a translated link, so under -l/--links a symlink
planted by an untrusted source at the destination path could redirect the
btime write to a target outside the backup destination on OSes where
birth time is settable (Windows).

Use the NOFOLLOW birth-time write whenever translating symlinks, not only
for translated links. It is a no-op on a real file or directory and stops
a planted symlink from being followed out of the destination.

(cherry picked from commit bd86336faac7cfc4e9057add38ec5fd12d762d02)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood a7ab39d3d1 local: fix dir metadata escaping the root through a planted symlink GHSA-f8g7-2xjc-7mfh CVE-PENDING
With -l/--links the local backend faithfully recreates a source ".rclonelink" as
a real symlink at the destination. Directory metadata (chmod/chown/chtimes),
however, was applied with the raw following syscalls
os.Chmod/os.Chown/os.Chtimes rather than through the os.Root sandbox used for
content writes. A Directory is never a translatedLink, so when the destination
path already existed as a symlink planted by an untrusted source, the metadata
was applied through it to a target outside the backup destination.

Route directory metadata through os.Root when translating symlinks, so a planted
symlink can no longer redirect chmod/chown/chtimes out of the destination, while
legitimate in-tree directories are unaffected.

(cherry picked from commit b764ccbd23582f29c611862f11f86dde3544035a)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood 7a594b09c6 serve docker: reject volume names resolving to the base directory itself GHSA-p6vx-hf7p-98j6
An empty or "." volume name joined onto the base directory resolves to the
base directory itself. newVolume does not call validate, so such a name
would mount a remote over the base directory and shadow every other
volume's mountpoint.

Require the resolved mountpoint to be a strict descendant of the base
directory so these degenerate names are refused.

(cherry picked from commit 4765ab020802ebf8ffbc2f2e529a6b23e640bf6f)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood f376c64784 serve docker: re-derive volume mountpoint from name when restoring state GHSA-p6vx-hf7p-98j6
When the plugin restarts it reads its persisted state file and used the
stored mountpoint verbatim. A state file written by an older rclone that
allowed escaping volume names, or one that was tampered with, could point
the mountpoint outside the base directory, so upgrading did not remediate
an already-escaped volume.

Re-derive the mountpoint from the base directory and the volume name on
restore, confined to the base directory, rather than trusting the stored
path.

(cherry picked from commit b4069bc49676e55b9c8843de9a1919f940a585ac)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood afeb26ae74 serve docker: reject volume names that escape the base directory GHSA-p6vx-hf7p-98j6
A Docker VolumeDriver.Create request carries a raw volume name that was
joined onto the base directory with filepath.Join and used verbatim as the
mountpoint. filepath.Join collapses ".." components, so a crafted name such
as "../../../etc/foo" resolved to a host path outside the base directory,
where the plugin then created a directory and mounted the remote.

Confine the mountpoint to the base directory and refuse any name that
resolves outside it.

(cherry picked from commit d9ed70376eb500d23d4d7ed0d6b3db2f3c7242e8)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood c6af0b57c2 serve ftp: fix auth-proxy sessions sharing credentials by username GHSA-c476-6w5q-jw77 CVE-PENDING
When serving FTP with --auth-proxy, the obscured password was cached in a
driver-global map keyed only by the username. Two sessions that logged in
with the same username but different credentials shared one map entry, so a
later login overwrote it and every subsequent operation on the earlier,
still-authenticated session was re-authorized with the later session's
credential and executed against the later session's backend.

Bind the credential to the FTP session by storing the obscured password in
the per-session goftp Session.Data map instead, so each session always
resolves the backend it authenticated for.

(cherry picked from commit 0bc745328dbf4669561d2abd9c81c596fbb0787d)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood 90595f34f2 serve s3: fix auth proxy accepting any request signed with an empty secret GHSA-xwwr-4h3p-r22c CVE-PENDING
With --auth-proxy set and --auth-key unset, serve s3 registered every client
supplied access key ID with an empty secret and verified the SigV4 signature
against that, so anyone could sign a request for an arbitrary access key ID with
an empty secret and be let in. The proxy program was only ever given the access
key ID (as both user and pass) so it had nothing with which to authenticate the
client either.

An S3 client never sends its secret, only a signature made with it, so the
server has to know the secret to check the request. The auth proxy protocol as
been changed to handle this. For serve s3 the proxy program is given just the
access key ID as the user (no pass or public_key) and must return the matching
secret as _secret_access_key in its output. rclone verifies the request's
signature against that secret, refusing the request if the proxy rejects the
access key ID, doesn't return a secret or returns an empty one, or the signature
doesn't match. The secret is only used for this server's own verification and is
never registered with gofakes3, so other serve s3 instances in the same process
don't honour it.

The proxy's answers are cached. If a signature fails against a cached secret the
proxy is consulted again so a rotated secret takes effect immediately - but only
for a signature mismatch, and at most once every 10 seconds per access key ID
and client IP, so a stream of bad signatures can't make the proxy program run
for every request. A rotation never shuts down the cached backend under requests
still using it. A cached answer is checked with the proxy again once it is 5
minutes old even if in constant use, so revoking an access key ID takes effect
within 5 minutes.

This means --auth-key is no longer needed with --auth-proxy: it is ignored and a
warning is given at startup if both are set. The proxy is the source of truth
for both the credentials and the backend they map to. Presigned URLs (credential
in the query string) are now recognised by the proxy middleware too. The auth
proxy docs are added to serve s3.

Note that the serve s3 auth proxy protocol has changed. The proxy program is now
given the access key ID as "user" (it was previously given an MD5 hash of it,
with the access key ID as "pass") and must return the matching secret as
"_secret_access_key".

This needs gofakes3 v0.0.9 for signature.V4SignVerifyWithSecret.

(cherry picked from commit 7306f0668125ecbd1cb9d704f570fbb492a76fb8)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood b3cf0444a2 serve s3: fix each server accepting the --auth-key credentials of all the others
gofakes3 kept the keys given with --auth-key in a store global to the process,
so when more than one serve s3 was running in one rclone (eg started via the rc)
each accepted the others' credentials and a client with the key for one server
could read and write the backend of another.

This updates gofakes3 to v0.0.9 which keeps auth keys per instance and adds a
test that two servers only accept their own keys.

(cherry picked from commit b81c9c892f855095924b89e623f0e4c4e7149168)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 11ff6e49bb serve sftp: fix auth proxy configured via rc being silently ignored GHSA-p569-5gjg-9cmj CVE-PENDING
From v1.70.0, an SFTP server started through the rc serve/start API with
a per-server proxyOpt.AuthProxy decided whether to enable proxy
authentication by checking the process-global proxy.Opt.AuthProxy
instead of the supplied proxyOpt.AuthProxy. In the normal rc case the
global is empty, so the auth proxy was silently ignored: the server
either failed to start with "no authorization found" or authenticated
against the local authorized_keys file instead of routing each login
through the proxy the operator configured.

The serve Provider refactor (f425f8d46) fixed the constructor by building the
provider from the supplied proxyOpt, but the authorized-keys handling in
configure() still consulted the global option. Make it depend on whether
proxy mode is actually active, and add a regression test for the
per-server option.

(cherry picked from commit 929933f5c303d6e7d5ab0cdcbae843862cb15e73)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood bcef98db3f serve s3: fix misleading anonymous access log and add test for auth proxy via rc GHSA-p569-5gjg-9cmj CVE-PENDING
From v1.70.0 until the serve Provider refactor (f425f8d46), an S3 server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server served the fixed
filesystem supplied to serve/start rather than routing each access key
to the backend chosen by the proxy, bypassing the operator's intended
per-key authorization.

The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again, and only
logs "allowing anonymous access" when neither an auth key nor an auth
proxy is configured so the log reflects the effective mode.

(cherry picked from commit 1e8134e83bbdcededfdc3c48f00e52544a775342)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood f18aacf2c7 serve ftp: add test for auth proxy configured via rc GHSA-p569-5gjg-9cmj CVE-PENDING
From v1.70.0 until the serve Provider refactor (f425f8d46), an FTP server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server fell back to its
fixed-backend mode, whose default account accepts user "anonymous" with
any password - a complete authentication bypass.

The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again.

(cherry picked from commit f89737278f27dc233f4dae2cb126a5e8b1980f70)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood ab1f458013 serve s3: reject bogus multipart part sizes in the reorder buffer GHSA-2p48-j3qc-rx9f
The multipart reorder-buffer admission trusted the client-declared part length.
A negative length was accepted, and `buffered + size` could overflow int64 for
a huge declared length, wrapping the running total negative and admitting
further parts past --multipart-streaming-buffer-limit.

Reject a negative length and use the overflow-safe comparison `size <=
bufferLimit - buffered` so an untrusted Content-Length can neither poison nor
overflow the budget.

(cherry picked from commit 337ab7f76ba6ba3731fba04b088bd831511bca04)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 7c1dfd99f3 serve s3: fix memory exhaustion from client-declared multipart part size GHSA-2p48-j3qc-rx9f CVE-PENDING
Streamed multipart UploadPart called Reserve(contentLength) before reading any
body bytes, so the pool immediately allocated one 1 MiB page per MiB of the
client-declared Content-Length (or X-Amz-Decoded-Content-Length). An client
could declare a huge part size, send no body, and force an arbitrarily large
allocation without paying the bandwidth cost of the declared body.

Drop the Reserve so the pool-backed buffer grows a page at a time as the body
is actually read: memory now tracks the bytes received, not the unverified
header.

(cherry picked from commit 8f2ad09941b0d69b67366101d5c2c350ca2a12df)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 57842c5ee4 fs: confine directory listing entries that escape the root GHSA-3vxh-3pcx-9m8q GHSA-38xv-hf3p-h7mq CVE-PENDING
The rclone core does not sanitise ".." in an object's Remote(). Such a name can
arrive from a malicious or buggy backend - an object store permits keys
containing ".." or a leading "/" - and, if acted on, lets a listing or transfer
escape the configured root. A source object named "../../other/x" is copied to
"other/x" outside the destination root, and a crafted listing name surfaces
outside the directory being listed.

Add list.RemoteEscapesRoot, which reports whether a Remote climbs above the
root when joined onto it, and list.RemoveEscaping, which drops and logs such
entries.

Apply RemoveEscaping unconditionally - independent of the include/exclude
filters - at the three per-entry filtering points every listing passes through:
filterDir, walk.listR and walk.walkRDirTree (recursive ListR).
operations.StatJSON calls List and NewObject directly, bypassing those, so it
rejects an escaping remote up front.

This confines every backend at once, so no per-backend change is needed.

(cherry picked from commit 3530367fcdefeb718d9cac14a5147836b448bc73)
2026-09-04 16:18:18 +01:00
Hakan İSMAİL 739403963a serve: refactor VFS and proxy handling into Provider
(cherry picked from commit f425f8d466)
2026-09-04 16:18:18 +01:00
am-at-enrollvb 1bec2133c5 serve: pass the client IP address to the auth proxy - fixes #4499
The auth proxy was only given the user and their password or public
key, so a proxy program had no way to restrict logins to particular
networks, or to record where an authentication attempt came from.

The JSON sent to the program now has a client_ip key holding the bare
IP the client connected from, with the port stripped so IPv6 arrives
as 2001:db8::1 rather than [2001:db8::1]:52344. An IPv4-mapped IPv6
address is reported as plain IPv4 so that a client arriving over a
dual-stack listener still matches IPv4 networks. The key is omitted
when the client has no IP address.

The IP is also mixed into the backend cache key. That is needed as the
program is only run on a cache miss, so a client from a
non-allowlisted address presenting valid credentials within the 5
minute cache lifetime would get a cache hit and be let in without the
program being consulted at all.

(cherry picked from commit 5dd34275dc)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 64de81e6a0 build: make go1.26 the minimum required version
golang.org/x/crypto v0.56.0, which fixes CVE-2026-78662 and
CVE-2026-56855 in its ssh package, requires go1.26, so rclone can no
longer be built with go1.25.
2026-09-04 14:07:42 +01:00
Nick Craig-Wood 0b187f3d7a build: update golang.org/x/crypto to v0.56.0 to fix CVE-2026-78662 and CVE-2026-56855
CVE-2026-78662: a malicious peer could flood an undecided channel's
incoming requests, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6354)

CVE-2026-56855: a malicious peer could send crafted messages on an
established channel, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6355)

(cherry picked from commit f550317590)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 0e5100203c bisync: fix failed transfers of empty files being recorded as synced
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.

(cherry picked from commit 7e17e1b90d)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood ba3d34b9f8 docs: describe how backends should allocate memory
(cherry picked from commit 220fe76192)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 77e6390f5b quatrix: fix chunk upload retries and fix memory leak
Each upload chunk is buffered in a pool.RW from the global memory pool
but was never closed, so its pages were never returned to the pool.

Close the buffer after each chunk is uploaded and on the read error
path.

A chunk that failed with a retryable error was also retried without
rewinding the buffer, so the retry sent an empty body with the original
Content-Length and Content-Range and failed.

Seek the chunk back to the start inside the pacer closure so each
attempt re-sends it in full.

The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.

(cherry picked from commit 2f0228029e)
2026-09-04 14:07:19 +01:00
SillyZir f190b34d95 onedrive: fall back to manual drive ID entry when drive listing fails
When both /me/drives and /me/drive fail during config (for example an
account-level 403 serviceReadOnly "Database Is Read Only"), send the
config state machine to the existing manual drive ID entry state
instead of dead-ending at choose_type with the raw error. The drive
itself remains usable when only the enumeration API is blocked.

Fixes #9794

(cherry picked from commit 03fe2ef794)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 49e7016d12 build: update golang.org/x/crypto to v0.55.0 to fix CVE-2026-56854
CVE-2026-56854: source-address critical option not enforced for
non-public-key auth callbacks in golang.org/x/crypto/ssh (GO-2026-6303)

(cherry picked from commit e5e1ee3e96)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 6240cbb694 crypt: warn about directories with legacy version-like encrypted names
Directory names which look like they have a --b2-versions version
string are now encrypted in full, so directories created by older
rclone (which left the version string in plain text) no longer
decrypt and vanished silently from listings.

DecryptDirName now falls back to the old form for such names so the
directory is listed, and logs the name it needs to be renamed to on
the underlying remote to make it accessible again. Document this in
the crypt docs.

(cherry picked from commit 1583cce1e2)
2026-09-04 14:07:19 +01:00
CAOShurong 66bc465d1a docs: fix dead links in sia and storj backends
(cherry picked from commit 413138f56b)
2026-09-04 14:07:19 +01:00
0rangeSeaW0lf b576cbdf40 internxt: persist rotated token returned by the user info call
The refresh endpoint returns a rotated token with a fresh expiry on
every successful call, but getUserInfo discarded it, so routine use
never extended the stored token's life. Once the stored token aged
out, accounts with 2FA enabled could not recover non-interactively
and required a manual reconnect.

Carry the rotated token out of getUserInfo and persist it in NewFs
via the same jwtToOAuth2Token + oauthutil.PutToken path that
refreshJWTToken uses, keeping f.cfg.Token in sync (same pattern as
refreshOrReLogin).

Fixes #9584

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 66761670da)
2026-09-04 14:07:19 +01:00
TowyTowy 66dba8239f crypt: fix directory names which look like versioned file names
The --b2-versions support added in 3fe2aaf96 strips a version string
from the last segment of a path before encrypting it, so that the
plain text version suffixes which the underlying backend appends to
encrypted file leaf names can be handled. EncryptDirName and
DecryptDirName share that code, so the last segment of a *directory*
name was version stripped too. Only file leaf names are ever given a
version string by the backend - a directory gets a
version-string-like name from the user, and such a name is encrypted
verbatim when it appears as the parent of a file name, so the same
directory ended up with two different encryptions.

Before this change, with a directory whose name matches rclone's
version format, eg dir-v2001-02-03-040506-123:

    rclone copy file.txt crypt:dir-v2001-02-03-040506-123/
    rclone ls crypt:dir-v2001-02-03-040506-123
    # => "directory not found" - the file is invisible to listings
    rclone mkdir crypt:dir-v2001-02-03-040506-123
    # => creates a second directory with the same decrypted name

After this change EncryptDirName and DecryptDirName encrypt directory
names verbatim, so a directory encrypts the same way whether it is
named on its own or as the parent of a file. Version strings are only
added to file names by the underlying backend, so --b2-versions is
unaffected and the existing version tests are untouched.

A directory which was created by the old EncryptDirName will no longer
decrypt and will be reported as undecryptable in listings. Such
directories were already unusable - anything copied into one was
written to a different encrypted directory - so nothing which worked
before is broken by this.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 67b184d6e7)
2026-09-04 14:07:19 +01:00
Anatoly Tarnavsky 92ef010fd2 s3: fix server side copy failing with --s3-no-head-object - fixes #9629
With no_head_object set, NewObject does not read any metadata, so the
destination object returned from a server side copy had a size of 0.
The size check in operations.Copy then failed with "corrupted on
transfer: sizes differ N vs 0" and deleted the newly copied object.
This also broke Move and hence renames through rclone mount.

Populate the destination object's size and MD5 from the source object
when no_head_object is set, as a server side copy produces an object
with identical content.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6df7b8aba1)
2026-09-04 14:07:19 +01:00
Loi Nguyen feb0664b64 dropbox: fix ChangeNotify when the root's case differs from Dropbox's - fixes #9692
Dropbox is case insensitive and the path_display it returns in
change notifications may not match the case of the configured root.
Before this change the root was trimmed with a case sensitive prefix
match, so when the cases differed the full path was passed to the
ChangeNotify callback and the notification was ignored.

This trims the root case insensitively while preserving the display
case of the remaining path.

(cherry picked from commit 4af64270cc)
2026-09-04 14:07:19 +01:00
Sune Mølgaard 61427e9729 serve http: prevent scrolling to the top on page reload - fixes #9771
(cherry picked from commit bdeb95ae01)
2026-09-04 14:07:19 +01:00
Vijay Misal 3d1fd51f61 vfscache: fix log message growing without bound on repeated write errors
Write() overwrote a successful write's nil error with the stale
lastErr returned by kickWaiters() once the downloader had recorded
too many errors. download() then wrapped that stale error again and
stored it back as the new lastErr, so every subsequent write added
another "vfs reader: failed to write to cache file:" prefix - fixes #4998

(cherry picked from commit efa5e8fcc1)
2026-09-04 14:07:19 +01:00
Rayan Salhab 6c44400bf7 accounting: fix bwlimit burst overflow - fixes #9820
Co-authored-by: cyphercodes <cyphercodes@users.noreply.github.com>
(cherry picked from commit 468eccb122)
2026-09-04 14:07:19 +01:00
water 041b766428 fix: do not retry multipart upload chunk on 404 (upload session not found)
(cherry picked from commit 5d1feea7e8)
2026-09-04 14:07:19 +01:00
shaurya 3c505b1e99 docs: fix broken links and wrong s3 directory bucket flag name
Several documentation links pointed at anchors or paths that no longer
resolve, and the S3 directory buckets section named the config option
and flag in the plural, which does not match the backend.

Co-authored-by: shaurya <19599684+no-hup@users.noreply.github.com>
Co-authored-by: no-hup <shauryaj.finance@gmail.com>
(cherry picked from commit 9dbfd9d852)
2026-09-04 14:07:19 +01:00
CAOShurong 6317faccc0 s3: treat UploadPart success without ETag as retryable error
A successful UploadPart whose response carries no ETag header made
WriteChunk panic dereferencing uout.ETag in a debug log line. The part
ETag is required by CompleteMultipartUpload, so an ETag-less 200 is
unusable: return a retryable error from inside the pacer callback so
the chunk is retried instead of crashing the transfer or completing
the upload with a broken part list.

Fixes #9822

Co-authored-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
(cherry picked from commit 660144d311)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 5a490a31c5 docs: update sponsors
(cherry picked from commit c140d36a1f)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood b8470e4cac test_all: pikpak: ignore TestRcatSizeChecksum/Corrupted
Pikpak never returns MD5 for uploads which causes this test to fail.

Perhaps Pikpak should not declare MD5 but that is a bigger decision
being discussed in #9826

(cherry picked from commit 4369d16a1c)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood e7ae39f42d webdav: fix SetModTime failing and hashes missing on Nextcloud
Nextcloud only stores a checksum which is supplied in the OC-Checksum
header of an upload, and discards it again when the modification time
is set with PROPPATCH. Re-sending the checksum in the PROPPATCH (as is
done for ownCloud) is rejected by Nextcloud with 403 Forbidden which
made the whole PROPPATCH fail, so SetModTime returned an error on any
object which had a hash. Uploads from sources without hashes, eg
streamed uploads with `rclone rcat`, were stored with no hash at all.

Use the Nextcloud PATCH extension with the X-Recalculate-Hash header
to have the server calculate and store the SHA1 of an object after a
streamed upload and after setting the modification time. This gives
a server side hash of the stored data which also lets rclone verify
streamed uploads.

(cherry picked from commit f7c510af49)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 43f107e049 pikpak: fix truncated single part uploads reported as ok when source ends early
If the source supplied fewer bytes than its declared size, the single
part upload path accepted the short body and stored a truncated file
recorded with the declared size, reporting a successful upload. The
multipart path already checks for this.

Count the bytes actually read and fail the upload if they do not match
the declared size, which cancels the partially created file.

This was found by the FsPutShortEOF integration test.

(cherry picked from commit 8e744de5e6)
2026-09-04 14:07:19 +01:00
machsix 748eaf9b28 onedrive: fix 403 Forbidden for configuration personal onedrive
(cherry picked from commit 8869a848f2)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 11dd394670 azureblob: fix test which didn't compile
We accidentally merged this commit with non compiling tests.

bee45bccfd azureblob: fix spurious vfs cache corruption errors during chunked reads #9782

(cherry picked from commit d3a71eea36)
2026-09-04 14:07:19 +01:00
Sanjay Kanth A 21483522a6 dropbox: decode received shared-file names - fixes #9707
listSharedFolders already decoded shared-folder names with
f.opt.Enc.ToStandardName, but listReceivedFiles stored the raw name
returned by the Dropbox API unchanged. Names that require encoding
(e.g. a trailing space, which Dropbox itself rejects, so rclone
stores it as "name␠" via EncodeRightSpace) were therefore shown under
their raw, encoded form for received files instead of being decoded
back to the standard name, and findSharedFile could not resolve such
a file by its standard name.

Apply the same ToStandardName conversion listSharedFolders uses.

(cherry picked from commit f3a7aaf635)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood f92262d48c azureblob: fix spurious vfs cache corruption errors during chunked reads - fixes #9782
On a ranged download the metadata decoder stored the response's
Content-Length (the length of the range, not the blob) in the object's
size and only corrected it from the Content-Range total afterwards.
Object.Size() is read concurrently by the VFS cache and chunked reader
while a download is in progress, so with --vfs-read-chunk-size a reader
could observe the chunk length (e.g. 67108864 for 64M chunks) as the
object size. The VFS cache then logged

    vfs cache: cached file (N) is unexpectedly larger than the remote
    object (67108864). The cached file is likely corrupted after an
    unclean shutdown; recovering ...

and truncated the read request against the bogus size, breaking
sequential reads of large blobs with --vfs-cache-mode full.

This applies the Content-Range correction before the size is stored so
the range length is never published as the object size.

(cherry picked from commit bee45bccfd)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood ad87bf6c93 lib/rest: make ParseContentRange public
(cherry picked from commit 2a8d8afd0f)
2026-09-04 14:07:19 +01:00
kingston125 f1c06a047e filelu: fix duplicate root path during multipart folder creation
(cherry picked from commit 6ee1d851ec)
2026-09-04 14:07:19 +01:00
Rohit Behera d6fb148d9b huaweidrive: fix truncated files being uploaded successfully when the source ends early
The multipart upload copied the source into the request buffer without
checking how many bytes it had read, so a source that supplied fewer
bytes than its declared size was accepted by the server and reported as
a success with a truncated file stored.

Count the bytes actually read and fail the upload if they do not match
the declared size.

Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
(cherry picked from commit 83b143103c)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 930de88f69 protondrive: fix files uploaded with v1.75.0 not being readable in the Proton apps
rclone v1.75.0 started creating files in Proton Drive's new
crypto-refresh encryption format, following guidance from Proton that
new file node keys should use the v6/AEAD profile. It turns out the
official Proton web app cannot decrypt files whose node key is a v6
key (but the Android app can), so every file uploaded with v1.75.0 (or
a beta after 2026-07-13) shows 'Item cannot be decrypted' in the web
app, even though rclone itself reads the files fine. Inspecting a file
created by the web app shows Proton itself still creates v4 node keys,
using the new format only for the file content.

New files are now created with the same fully pre-crypto-refresh
format as v1.74.4 (v4 node key, v3 PKESK content key, v1 SEIPD
blocks), which every Proton client can read. Reading files in the new
format still works, new revisions of files which already use the new
content format keep it, and the auxiliary fields (name, node
passphrase, extended attributes, block signatures) are pinned to the
old format regardless of the recipient key's preferences, as Proton
requires.

Files already uploaded with v1.75.0 cannot be repaired in place -
uploading a new revision does not change the file's node key. To make
such a file readable by the Proton apps again, delete it from the
remote and upload it again with a fixed version of rclone.

This updates Proton-API-Bridge to v1.0.5 and go-proton-api to v1.0.4.

See: https://forum.rclone.org/t/proton-drive-unable-to-decrypt/54087
(cherry picked from commit d9aa903358)
2026-09-04 14:07:19 +01:00
Rohit Behera fad0359e6b box: fix truncated files being uploaded successfully when the source ends early
The single-shot upload path sent the source straight to Box as a multipart
body with no Content-Length, so a source that supplied fewer bytes than its
declared size produced a short request that Box accepted and stored, and the
upload was reported as a success.

Count the bytes actually read and fail the upload if they do not match the
declared size. The multipart path already reads each chunk with io.ReadFull
and so already fails in this case.

(cherry picked from commit 64ab1ac322)
2026-09-04 14:07:18 +01:00
Rahman Yilmaz c14e507609 walk: stop directory traversal when the context is cancelled - fixes #9788
The concurrent walker created by walk() only stopped when the callback
returned an error or the whole tree had been listed. Cancelling the
context (for example via the rc job/stop endpoint for an async
operations/size or recursive operations/list call) was therefore
ignored: the checkers kept pulling list jobs from the channel and kept
listing the entire tree, burning CPU and making job cancellation
useless for every backend without a native ListR implementation.

Make every checker select on ctx.Done() so a cancelled walk shuts down
promptly through the existing quit/drain path and reports the context
error. Also check the context between directory read chunks in the
local backend so a single huge directory does not block cancellation.

(cherry picked from commit 5eb5c01e36)
2026-09-04 14:07:18 +01:00
Rohit Behera 0720a194cf yandex: fix truncated files being uploaded successfully when the source ends early
Update already wrapped the source in a counting reader but never looked at the
count, so a source that supplied fewer bytes than its declared size was
uploaded as a chunked request, accepted by the server and reported as a
success with a truncated file stored.

Compare the bytes actually read against the declared size.

(cherry picked from commit 1128693468)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 194d22ce71 build: untap aws/tap to silence homebrew tap trust warnings on macOS
(cherry picked from commit be7f9b38b0)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 7995d87cd5 s3: Mega: update endpoints
(cherry picked from commit ec3a95c279)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 2d261879dd docs: add assigned CVE numbers to the v1.75.0 security advisories in the changelog
Five of the advisories released with v1.75.0 now have CVEs assigned:

- GHSA-45pq-889g-fcgh serve restic path traversal: CVE-2026-71309
- GHSA-xhf4-832v-7xcr lib/proxy CONNECT header OOM: CVE-2026-71310
- GHSA-8c48-q9wj-3w37 ftp command injection: CVE-2026-71311
- GHSA-2m8m-jhrm-w6j2 sftp PowerShell command injection: CVE-2026-71312
- GHSA-7p4m-qxvv-g567 local file name escape: CVE-2026-71313

GHSA-6jcg-q3wp-x2f4 (squashfs) loses its CVE-PENDING marker as GitHub
declined to issue a CVE from the rclone repository - the vulnerable code
is in go-diskfs so any CVE must come from an advisory there.

GHSA-mfvx-7rcj-9m5g (pprof) keeps its CVE-PENDING marker as the CVE
request is still awaiting allocation.

(cherry picked from commit 2c1174af0d)
2026-09-04 14:07:18 +01:00
VXNCXNX 2e0718167f lib/transform: fix panic in truncate_keep_extension
Return error when extension is longer than truncation limit.

(cherry picked from commit c667e53638)
2026-09-04 14:07:18 +01:00
Shantanav Mukherjee 9718ed3fc2 docs: clarify VFS cache age semantics
(cherry picked from commit 4f22d62c66)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 645e1ea5be build: fix multiple CVEs by upgrading to go1.26.6
- CVE-2026-56860: net/url: quadratic complexity in resolvePath
- CVE-2026-56858: html/template: JavaScript regexp context tracking
- CVE-2026-56862: crypto/tls: limit handshake messages accepted post-handshake
- CVE-2026-56853: net/http: apply ReadHeaderTimeout to unencrypted HTTP/2 check
- CVE-2026-56859: encoding/xml: recursion depth guard during decode
- CVE-2026-33818: encoding/asn1: enforce maximum recursion depth
- CVE-2026-46600: net: panic parsing an invalid SVCB or HTTPS RR in dnsmessage
- CVE-2026-39821: net/http: reject ASCII-only Punycode-encoded labels in idna

This also updates the go1.25 test job to go1.25.13 which contains the
same fixes.

(cherry picked from commit f0b210a886)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 40dab7ecdd build: update golang.org/x/image to v0.45.0 to fix CVE-2026-46603
CVE-2026-46603: excessive memory allocation during VP8L decoding

This also updates golang.org/x/text to v0.41.0 as a dependency.

(cherry picked from commit 00593a96fe)
2026-09-04 14:07:18 +01:00
Dave 43a2a92950 vfs/vfscache: fix reader deadlock when the item size drops below the read offset
_dispatchWaiters decided whether a waiter was satisfied by clipping its
range against dls.src.Size(), the size of the fs.Object snapshot taken
when the Downloaders was created. _ensureDownloader decided whether to
start a downloader from Item.FindMissing, which clips against
item.info.Size instead.

When item.info.Size dropped below the offset a waiter was parked on while
the source object still reported the full size, the two disagreed.
_ensureDownloader found nothing missing so it started no downloader, and
_dispatchWaiters found the range absent so it never released the waiter.
Nothing was downloaded and no error was produced, so the error count never
reached maxErrorCount and the waiter was never woken. The reader blocked
forever with nothing logged at any level.

Wake a waiter when FindMissing reports nothing left to download for it as
well as when its data has arrived. Since _ensureDownloader starts a
downloader only when FindMissing is non empty, a waiter with nothing
missing has nothing that could ever wake it.

Fixes #9769

(cherry picked from commit 6e0c71bd27)
2026-09-04 14:07:18 +01:00
Morax 62f8f944e2 lib/rest: validate ranged responses
Add response validation for calls made with Range open options. Verify
Content-Range, Content-Length, response status, and the complete
representation size before a backend accepts the response body.

Return a shared sentinel when a server ignores a partial range so callers
can avoid retrying the same unsupported request.

(cherry picked from commit 69e5aff2a9)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 841ff2d418 crypt: fix hash mismatches with no_data_encryption on backends which check upload hashes
Before this change, when no_data_encryption was set, uploads from
local disk advertised the hash of the encrypted data even though the
data was uploaded unencrypted.

On backends which check upload hashes (eg b2) this made uploads of
small files fail with errors like "Checksum did not match data
received", and made chunked uploads store an incorrect hash so the
files failed their checksum on download with "corrupted on transfer:
SHA1 hashes differ".

See: https://forum.rclone.org/t/sha1-mismatches-on-b2-with-no-data-encryption-true/54121
(cherry picked from commit 5a0b7d6746)
2026-09-04 14:07:18 +01:00
Pastalikek65 33ab81ce6d config: redact env var config values in logs
Before this change the environment variable getters in fs/configmap.go
logged the option value with %q, so a password set via
RCLONE_CONFIG_remote_pass (or RCLONE_remote_pass) was printed in full
to the debug log. Values from the config file were already redacted,
which made the leak easy to miss.

This change routes both getters through fs.RedactOptionValue, which
looks up the option in the backend's option list: options marked
IsPassword or Sensitive log as XXX, unknown options are conservatively
redacted, and --dump auth still shows the value for debugging.

Fixes #5794

(cherry picked from commit adc7f2ebfa)
2026-09-04 14:07:18 +01:00
Dean Chen 632ff74375 docs/mount: mention nfsmount for macOS NFS mounts
The NFS section under Mounting on macOS talked about serve nfs without
pointing at rclone nfsmount, which is the command that actually does the
NFS-based mount on macOS.

Fixes #7869

Signed-off-by: Dean Chen <862469039@qq.com>
(cherry picked from commit c785ff90d7)
2026-09-04 14:07:18 +01:00
Recoordinate 39b926a2ce docs: fix doubled words
(cherry picked from commit cfdc9d0558)
2026-09-04 14:07:18 +01:00
Rodrigo Rodrigues 31f4c78c5e docs: fix typo in drive client_id section
(cherry picked from commit 8b42a38e9d)
2026-09-04 14:07:18 +01:00
Christian De Santis fcb317e226 iclouddrive: fix uploads into an app container failing with 412 - fixes #9729
Writing any file into a third-party app container - the Obsidian, Pages or
Shortcuts folders that iCloud Drive shows alongside your own - failed with

    HTTP error 412 (412 Precondition Failed) returned body:
    "{ ... \"error_code\" : \"VALIDATING_REFERENCE_ERROR\", \"reason\" :
    \"Request has out of order children to be chained but the parents were
    missing\" }"

Reading from those paths worked, and so did creating directories in them, so
the failure looked like a missing parent when the parent was plainly there.

Items in an app container live in a different zone from ordinary iCloud Drive
folders: a folder under Documents has a drive ID like
FOLDER::com.apple.CloudDocs::<uuid>, while the Obsidian container has
FOLDER::iCloud.md.obsidian::documents#o2v. DownloadFile already accounts for
this - it deconstructs the item's own ID and addresses the zone it finds - but
CreateUpload and UpdateFile hardcoded defaultZone, and UpdateFile built the
resulting Drivewsid with a hardcoded com.apple.CloudDocs as well.

So rclone asked Apple to chain the new document to a parent in
com.apple.CloudDocs while the parent lived in iCloud.md.obsidian. The parent
really was missing from the zone being addressed, which is what the error said.

Take the zone from the parent's drive ID instead, the same way the download
path does, and build the new item's ID with ConstructDriveID. Uploads outside
an app container are unaffected: their parents are in com.apple.CloudDocs, so
the derived zone is the value that was previously hardcoded.

Verified against a real remote: files now upload into an Obsidian vault inside
the container and read back correctly with an unpatched binary afterwards.

(cherry picked from commit aba403fa79)
2026-09-04 14:07:18 +01:00
nielash 82a4a63b92 accounting: fix memory leak from stats groups on long-running rcd
Before this change, `NewStats` stored the context it was created from on the
`StatsInfo`. Stats groups are never freed -- they are only evicted once there are
`--max-stats-groups` of them -- so each one kept its context, and everything reachable
from it, alive for the life of the process. As the rc creates a group per call,
that included the call's filters and their compiled regexps.

The context was only ever used to get `ci.StatsFileNameLength` from the config.
`StatsInfo` already stores that same `*fs.ConfigInfo`, read from the same
context in `NewStats`.

This change fixes the issue by passing the stored ci to `transferMap.String` and
dropping the context from `StatsInfo`.

(cherry picked from commit abae66ee1a)
2026-09-04 14:07:18 +01:00
nielash d407018995 accounting: fix memory leak on long-running rcd
Before this change, Transfer.Done closed the account of a completed transfer but
never released it, because it assigned nil to only a local copy of the pointer.
As a result, every completed transfer continued to reference its account.

An Account holds the transfer context and the source reader, and the stats keep
completed transfers around up to `MaxCompletedTransfers` per group, with groups
discarded only at --max-stats-groups. On a long-running `rclone rcd`, this adds up.

It was noticeable when running bisync repeatedly via the rc, where the transfer
context carries `b.WriteResults` (bisync's `LoggerFn`). A `*bisyncRun` holds
Path1 and Path2 listings, which can be quite large, and are not supposed to be
retained between runs. (Naturally they aren't, when running bisync on the
command line -- which is probably why we didn't notice this issue sooner.)

This change fixes the issue by releasing `tr.acc` in Done, instead of the local
copy. `tr.Snapshot` reads the byte counts off the account, so the progress is
recorded on the transfer first. That read happens before taking `tr.mu`, because
`acc.progress()` locks `acc.values.mu`, `checkReadBefore` holds that lock while
calling `StatsInfo.GetBytes`, and `StatsInfo` locks back into `Transfer` in
`Transferred` and `_removeTransfer`.

(cherry picked from commit 71a0932126)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood b5f0ea9b7b serve s3: clean up abandoned multipart uploads after --multipart-expiry
A client which started a multipart upload and vanished without either
completing or aborting it used to hold on to its resources forever.

Incomplete multipart uploads which have had no activity for
--multipart-expiry (default 24h) are now aborted and cleaned up
exactly as if the client had called AbortMultipartUpload, with a
NOTICE logged.

An upload with a part still being received is never expired, and each
completed part restarts the clock. Late operations on an expired
upload fail with NoSuchUpload, as they do on real S3 when a lifecycle
rule has aborted the upload.

Set --multipart-expiry 0 to keep incomplete uploads forever.

(cherry picked from commit 1947e4217c)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood d166eaf635 serve s3: upload all multipart uploads via the VFS
Multipart uploads used to be streamed directly to the remote with their
own PutStream machinery, bypassing the VFS, a design left over from
before the VFS could abandon a streaming write.

They are now written through the VFS exactly like plain object PUTs in
every cache mode. The parts are written, in part-number order, to a
temporary object which is renamed into place server-side on
completion.

With the default --vfs-cache-mode off the parts stream through the VFS
to the remote as they arrive. With --vfs-cache-mode writes or above
they are buffered in the VFS cache and uploaded by its write-back.

User visible changes:

- Multipart uploads now show in rclone's transfer stats and obey
  --bwlimit (previously they bypassed both).
- Remotes without streaming upload support now spool the upload to a
  temporary file on local disk instead of buffering it in memory.
- Multipart uploads are never buffered in memory because of missing
  remote capabilities - only --disable-multipart-streaming does that.
- Remotes that upload atomically now also write to a temporary object
  renamed into place, so an in-progress multipart upload is no longer
  briefly visible under its final key.
- On the few remotes with no server-side move or copy the parts are
  written straight to the final object in all cache modes.
- With --vfs-cache-mode writes, plain PUTs and multipart uploads to the
  same key go through the same cache entry, so an earlier PUT still in
  the write-back window can no longer be written back over a newer
  multipart upload.
- Failed write-backs are retried by the VFS without the client having
  to restart the upload, and completed objects are served from the
  cache for read-after-write.

(cherry picked from commit b4db289d0a)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 62a1df7271 serve s3: reserve the .rclone_temp_ prefix for temporary objects
The temporary objects that uploads are written to before being renamed
into place are now named .rclone_temp_put_* and .rclone_temp_multipart_*,
and the whole .rclone_temp_ prefix is reserved: any object whose name
starts with it is hidden from S3 listings. This gives a single pattern
for cleaning up leftovers from killed servers:

    rclone delete --min-age 24h --include ".rclone_temp_*" remote:path

The .rclone_multipart_upload_* objects rclone v1.75 used are still
hidden from listings so leftovers from an older server stay invisible
to S3 clients.

(cherry picked from commit 0aa90200bd)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood e171f996d6 serve s3: fix modtime not being set when only mtime metadata is supplied on PUT
The mtime metadata fallback was nested inside the X-Amz-Meta-Mtime
branch, so it only ran when X-Amz-Meta-Mtime was present but invalid -
and then set the modtime from the invalid value's failed parse rather
than parsing mtime. An object PUT with only mtime metadata kept the
upload time as its modtime.

Now the two keys are checked independently, as TouchObject already
does.

(cherry picked from commit a3489456de)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 2bf9e0d20e serve s3: fix crash when a multipart upload is aborted while a part is uploading
streamPart did not check whether the upload had been torn down, but
AbortMultipartUpload sets the reorder buffer map to nil, so an abort
arriving while a part body was still being received panicked with an
assignment to a nil map once the part was buffered.

Now a part whose upload has been aborted or completed under it is
rejected with NoSuchUpload.

(cherry picked from commit 531d873bd7)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood afb1daa8fa serve s3: fix failed uploads deleting or corrupting the object at the key - fixes #9718
A PUT which failed part way through removed the object at the
destination key. As well as differing from real S3 (where a failed PUT
never affects the stored object), this raced with the client's
automatic retry of the same PUT: the retry stored the object and
returned 200 OK, then the failed first attempt's cleanup deleted it,
silently losing an acknowledged upload. The interrupted upload could
also be committed as a truncated object, since closing the write handle
gave the streaming upload a clean end of stream.

Now a failed or interrupted PUT never disturbs the object at the key:

- The object at the key is never removed on error.
- On backends where a partial upload is visible at its final name
  (PartialUploads), and when the VFS cache mode is writes or above, the
  upload is written to a temporary object which is renamed into place
  on success and removed on failure, as streamed multipart uploads
  already do. Backends which upload atomically are still streamed
  straight to the destination.
- An interrupted or short body fails the upload via
  WriteFileHandle.CloseWithError instead of committing truncated data,
  and a body which ends cleanly short of its declared size is rejected
  with IncompleteBody.

(cherry picked from commit 84298fc090)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 5775f0430b vfs: add WriteFileHandle.CloseWithError to abandon streaming writes
Closing a streaming write handle sends a clean EOF to the backend
upload, so a writer which knows its data source failed part way through
had no way to stop the truncated file being stored as if it were
complete.

CloseWithError closes the handle failing the upload with the given
reason instead. EOF-like reasons are rewritten so the layers reading
the pipe can't mistake them for a clean end of stream and commit the
partial file.

Needed by serve s3 to abort interrupted PUTs - see #9718

(cherry picked from commit 2f0657c35b)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood d0f00a22d5 internetarchive: fix corrupted files being created when the source ends early
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.

Count the bytes actually read from the source and if they do not match
the declared size return an error.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit 7357fb82a9)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood ad7e0c64af sia: fix corrupted files being created when the source ends early
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.

Count the bytes actually read from the source and if they do not match
the declared size, remove the partially uploaded file and return an
error.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit 5d057549b9)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 674d7a8a70 pikpak: fix truncated files being created when the source ends early
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated file and
reporting a successful upload.

Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit a2baa978db)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 1274301521 multipart: fix chunked uploads storing truncated objects when the source ends early
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated object and
reporting a successful upload.

Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.

This affects all backends which use lib/multipart for chunked uploads:
azureblob, b2, drime, internxt, oracleobjectstorage, s3 and shade.

This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.

(cherry picked from commit 5b7cc097e3)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 840755543f filelu: fix truncated files being uploaded successfully when the source ends early
If the source supplied fewer bytes than its declared size, the
truncated file was stored and the upload reported success with the
object claiming the declared size.

Count the bytes actually read from the source and if they do not match
the declared size, remove the truncated file and return an error.

This was found by the new TestRcatSizeShortEOF integration test.

(cherry picked from commit e1bf9405e2)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 2472718b7c azurefiles: fix zero padded files being created when the source ends early
The file is created at the declared size and the data then written
with ranged writes, so if the source supplied fewer bytes than
declared, the remainder of the file was left as zeroes and the upload
reported success.

Count the bytes actually read from the source and if they do not match
the declared size, delete the partially uploaded file (if newly
created) and return an error.

This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.

(cherry picked from commit 884b28c203)
2026-09-04 14:07:18 +01:00
nielash a4f1c1558e bisync: fix memory leak when running via the rc
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.

For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.

This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.

(cherry picked from commit 387b4b5e11)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 76d5c49ae6 docs: update sponsor links
(cherry picked from commit 1aa9efef17)
2026-09-04 14:07:18 +01:00
Edward Klesel 9fec073ab7 azureblob: fix Entra ID server-side copy source authentication
When using Microsoft Entra ID credentials, Azure Blob server-side copy
uses a user delegation SAS URL for the private copy source.

The SAS start time was set to the current local time. Azure Storage
validates the copy source from the service side, and small clock
differences can make that SAS appear not yet valid. The service then
returns 403 CannotVerifyCopySource with AuthenticationFailed.

Start the copy-source SAS 15 minutes in the past, matching Microsoft SAS
guidance for clock skew.

(cherry picked from commit e4c7aca6bd)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood a9c4548d92 overview: fix "internal error: no overview data found" on 32 bit architectures - fixes #9723
The precision field in the backend overview YAML files can hold
fs.ModTimeNotSupported (100 years in nanoseconds) which overflows int
on 32 bit platforms, making the YAML for those backends fail to parse
and causing rclone to log 18 internal errors on every invocation.

Use int64 for the precision field and add a test that parses every
embedded backend YAML file so this is caught on 32 bit test runs.

(cherry picked from commit 5629f2668c)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood d038b8b1c9 protondrive: fix corrupted uploads after a retried upload error - fixes #9722
When an upload failed part way through with a retryable error (eg a
502 from the block storage servers) the pacer retried the whole upload
call with the same input stream. The stream had already been partially
consumed, so the retry re-created the upload draft and committed just
the remainder of the stream as a complete file, silently truncating
it. With restic over serve restic this corrupted the repository as the
truncated pack was reported as successfully uploaded.

This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.

(cherry picked from commit a06df7a2de)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 3ceea42329 operations: fix silent truncation of streaming uploads whose source ends early
Uploads through RcatSize with a known size - used by rcat --size, the
rc operations/uploadfile and the serve backends, eg serve restic - did
not check the size of the uploaded object. If the source stream ended
before the declared size worth of data had been read, the truncated
object was reported as a successful upload. This could corrupt data
for callers which trust the result, eg a restic repository accessed
via serve restic (see #9722).

This adds the same size check operations.Copy performs after a copy,
respecting --ignore-size and backends which do not report sizes.

(cherry picked from commit 9b13247ba8)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 6beb4a5bd2 docs: fix width of sponsor images on very big screens
(cherry picked from commit 1318962a96)
2026-09-04 14:07:18 +01:00
Anton Karpov 7d70921bf5 docs: use the --dump form for the obsolete --dump-* flags
The SFTP page said `--dump-auth`, which no longer exists: it became a
value of `--dump`, so the docs asked for a flag rclone would reject.

The same line, and a line in the Swift troubleshooting section, also
used `--dump-headers` and `--dump-bodies`. Those still parse, but
SetFlags logs "--dump-headers is obsolete - please use --dump headers
instead", so the docs were steering readers onto a deprecated form.

The generated flag listings in docs/content/flags.md and
docs/content/commands/rclone.md are left alone: those flags do still
exist, so `--help` output should keep showing them.

(cherry picked from commit 6cb4732cc3)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 76c6992260 compress: fix corrupted objects being created when the source ends early
When the source supplied fewer bytes than its declared size, the
compressed data file was stored under a name containing the declared
size while the metadata recorded the actual number of bytes read.
NewObject looks the data file up by the size in the metadata, so the
resulting object could never be read again, and the upload reported
success.

Check that the number of bytes read matches the declared size after
uploading the data and before writing the metadata, and remove the
data file and return an error if it does not.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit 18fa445ffc)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood f1297d6298 dropbox: don't retry chunked upload requests when the upload has been cancelled
The append loop retries everything once the upload session has
started, so a cancelled context error was retried through all the low
level retries with exponential backoff before the upload gave up.

(cherry picked from commit e0701daea0)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 9a112f40c1 dropbox: fix chunked uploads of truncated files never finishing - fixes #9704
A source which returned EOF before supplying as many bytes as it
declared would either commit a truncated file (if the shortfall was
within the final chunk) or loop forever appending empty chunks to the
upload session. Return an error wrapping io.ErrUnexpectedEOF instead.

Note that all dropbox uploads use the chunked upload path with the
default batch_mode of sync, so this affected uploads of every size.

(cherry picked from commit bff17664ad)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood a3dcf7f0ad fstests: check uploads terminate when the source ends before its declared size
A source reader which returns EOF before supplying as many bytes as it
declared should not cause a backend Put to hang, panic or create an
object claiming the declared size.

The layers above detect a truncated transfer by comparing sizes after
the upload, so a backend may either return an error or create a
shorter object.

See #9704

(cherry picked from commit 03f8582a55)
2026-09-04 14:07:18 +01:00
alliasgher 3084177049 smb: reuse the upload connection for SetModTime - fixes #9675
Object.Update held its connection until the deferred putConnection ran at
function exit, so the SetModTime it does at the end of every upload had to take
a second connection from the pool, dialling a whole new SMB session when the
pool was empty. With N transfers in flight the pool grew to roughly 2N sessions
for no reason.

Return the connection as soon as the file is closed. At that point the upload
has succeeded and remove() can no longer be reached, so nothing else needs it,
and SetModTime picks the same connection straight back out of the pool.
putConnection nils the pointer, so the deferred putConnection becomes a no-op
and the connection is not returned twice.

(cherry picked from commit ea9a64c751)
2026-09-04 14:07:18 +01:00
alliasgher e4b778b5e3 drive: don't list trashed files when removing a directory into the trash - fixes #9681
purgeCheck listed a directory's children with includeAll set, so the Drive API
returned trashed children as well as live ones. The only thing that came of
that was the trashedFiles flag, which is used solely to decide whether to trash
the directory or delete it permanently, and when use_trash is on the directory
gets trashed either way.

That made the common case pay for nothing: after trashing N files in a
directory, removing the now empty directory paged through all N trashed entries,
about one API call and pacer delay per 1000, instead of running a single query
that returns nothing.

Ask the server to filter out trashed children when the directory is going to be
trashed anyway. Hard deletes still enumerate them, since there the distinction
decides whether the directory can be removed permanently (#1040), and
--drive-trashed-only is left alone because it deliberately lists trashed items.

(cherry picked from commit 065af89635)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood ee363b1826 build: fix intermittent CI failures by raising the test timeout to 20m
The cmd/gitannex end to end tests routinely take 230-530s and on slow
CI runners the package exceeds the go test default timeout of 10m.

Raise the per-package timeout to 20m in quicktest and racequicktest to
give slow runners headroom.

(cherry picked from commit 01d32dbde5)
2026-09-04 14:07:18 +01:00
Loi Nguyen e7f9c7bc10 smb: test closing failed dial connections
Followup to:

74f9f182aa smb: fix TCP connection leak when connection setup fails - #9678

(cherry picked from commit 23f1dbfe50)
2026-09-04 14:07:18 +01:00
debaditya 10eb7b5b10 dropbox: fix context cancellation (e.g. --max-duration limit) not stopping in-flight requests
This switches the calls to the dropbox SDK over to use the context taking
variants. This allows context cancellation to propagate properly.

Fixes #9688

(cherry picked from commit c97ab80ca1)
2026-09-04 14:07:18 +01:00
Loi Nguyen e0faeeb5d6 vfs: make poll shutdown lifecycle deterministic
(cherry picked from commit 1d03a77172)
2026-09-04 14:07:17 +01:00
Loi Nguyen dcf15b5f9c vfs: test poll interval update lifecycle
(cherry picked from commit d5275c4ebb)
2026-09-04 14:07:17 +01:00
Loi Nguyen c93cc1e1ae vfs: synchronize poll updates with shutdown - fixes #9689
(cherry picked from commit f132aef295)
2026-09-04 14:07:17 +01:00
Loi Nguyen 21c98ff5ff dropbox: preserve Paper export paths on lookup - fixes #9691
Direct lookups of exported Dropbox Paper files retained the
caller-visible extension before export metadata processing appended it
again. Track when metadata was resolved through an export path so the
object keeps the requested remote name while listing behavior remains
unchanged.

(cherry picked from commit 8c8062da78)
2026-09-04 14:07:17 +01:00
Loi Nguyen 5fbc677ed4 lib/batcher: prevent commits racing shutdown - fixes #9687
(cherry picked from commit 7f6207fae2)
2026-09-04 14:07:17 +01:00
SillyZir 00347ecf36 serve nfs: fix EIO when creating symlinks with --vfs-links
The macOS NFS client sends SETATTR after SYMLINK, which arrives as
Chmod/Chown on the link path. These opened the target with vfs.Open,
which follows symlinks - a freshly created symlink usually dangles, so
the open failed with ENOENT, surfaced to the client as NFS3ERR_IO even
though the link was created.

Add path-based VFS.Chmod and VFS.Chown mirroring VFS.Chtimes. They do
not follow symlinks (lstat semantics, matching VFS.Stat) and return
ENOSYS when the node exists, since the VFS stores neither permissions
nor ownership; serve nfs calls them and masks ENOSYS as before.

Fixes #9627

(cherry picked from commit 7804c1b315)
2026-09-04 14:07:17 +01:00
769 changed files with 68820 additions and 90060 deletions

No files matched your search

-25
View File
@@ -1,25 +0,0 @@
{
"permissions": {
"allow": [
"Bash(git status:*)", "Bash(git diff:*)", "Bash(git log:*)", "Bash(git show:*)",
"Bash(git rev-parse:*)", "Bash(git add:*)", "Bash(git commit:*)",
"Bash(go mod download:*)", "Bash(go build:*)", "Bash(go test:*)", "Bash(go vet:*)",
"Bash(gofmt -l:*)",
"Bash(golangci-lint run:*)"
],
"ask": [
"Bash(git push:*)", "Bash(git merge:*)", "Bash(git rebase:*)", "Bash(git switch:*)",
"Bash(git restore:*)", "Bash(git commit --amend:*)", "Bash(rm:*)",
"Bash(go get:*)", "Bash(go mod tidy:*)", "Bash(go install:*)",
"Bash(make:*)", "Bash(docker:*)"
],
"deny": [
"Bash(git push --force:*)", "Bash(git push -f:*)", "Bash(git push --force-with-lease:*)",
"Bash(git reset --hard:*)", "Bash(git clean:*)", "Bash(git add -f:*)", "Bash(git add --force:*)",
"Bash(git commit --no-verify:*)", "Bash(git commit -n:*)",
"Bash(rm -r:*)", "Bash(rm -R:*)", "Bash(rm -rf:*)", "Bash(rm -fr:*)", "Bash(rm -Rf:*)",
"Bash(rm -fR:*)", "Bash(rm --recursive:*)", "Bash(sudo:*)",
"Read(./.env)", "Read(./.env.*)", "Read(./secrets/**)"
]
}
}
-2
View File
@@ -1,2 +0,0 @@
sandbox_mode = "workspace-write"
approval_policy = "on-request"
-20
View File
@@ -1,20 +0,0 @@
# Rendered from docs/PERMISSIONS.md - change both in the same commit.
# Deny
prefix_rule(pattern = ["git", "push", ["--force", "-f", "--force-with-lease"]], decision = "forbidden",
justification = "Force-push rewrites shared history.")
prefix_rule(pattern = ["git", "reset", "--hard"], decision = "forbidden", justification = "Discards work.")
prefix_rule(pattern = ["git", "clean"], decision = "forbidden", justification = "Deletes untracked files.")
prefix_rule(pattern = ["git", "add", ["-f", "--force"]], decision = "forbidden",
justification = "Stages ignored files, e.g. secrets.")
prefix_rule(pattern = ["git", "commit", ["--no-verify", "-n"]], decision = "forbidden", justification = "Skips hooks.")
prefix_rule(pattern = ["rm", ["-r", "-R", "-rf", "-fr", "-Rf", "-fR", "--recursive"]], decision = "forbidden",
justification = "Recursive delete.")
prefix_rule(pattern = ["sudo"], decision = "forbidden", justification = "No privilege escalation.")
# Ask
prefix_rule(pattern = ["git", ["push", "merge", "rebase", "switch", "restore"]], decision = "prompt")
prefix_rule(pattern = ["git", "commit", "--amend"], decision = "prompt")
prefix_rule(pattern = ["rm"], decision = "prompt")
prefix_rule(pattern = ["go", ["get", "install"]], decision = "prompt")
prefix_rule(pattern = ["go", "mod", "tidy"], decision = "prompt")
prefix_rule(pattern = ["make"], decision = "prompt")
prefix_rule(pattern = ["docker"], decision = "prompt")
-20
View File
@@ -1,20 +0,0 @@
{
"tools": {
"allowed": [
"run_shell_command(git status)", "run_shell_command(git diff)", "run_shell_command(git log)",
"run_shell_command(git show)", "run_shell_command(git branch --list)",
"run_shell_command(git add)", "run_shell_command(git commit)",
"run_shell_command(go mod download)", "run_shell_command(go build)",
"run_shell_command(go test)", "run_shell_command(go vet)", "run_shell_command(gofmt -l)",
"run_shell_command(golangci-lint run)"
],
"exclude": [
"run_shell_command(git push --force)", "run_shell_command(git push -f)",
"run_shell_command(git push --force-with-lease)", "run_shell_command(git reset --hard)",
"run_shell_command(git clean)", "run_shell_command(git add -f)", "run_shell_command(git add --force)",
"run_shell_command(git commit --no-verify)", "run_shell_command(git commit -n)",
"run_shell_command(rm -r)", "run_shell_command(rm -R)", "run_shell_command(rm -rf)",
"run_shell_command(rm -fr)", "run_shell_command(rm -Rf)", "run_shell_command(sudo)"
]
}
}
-4
View File
@@ -2,13 +2,9 @@
go.mod text eol=lf
go.sum text eol=lf
# Embedded templates must render the same on every OS
lib/http/templates/** text eol=lf
# Ignore generated files in GitHub language statistics and diffs
/MANUAL.* linguist-generated=true
/rclone.1 linguist-generated=true
cmd/bisync/testdata/*/golden/** linguist-generated=true
# Don't fiddle with the line endings of test data
**/testdata/** -text
-56
View File
@@ -1,56 +0,0 @@
# Fork CI for the Proton Drive backend. Every step is hermetic: the
# tests run against the fake server in backend/protondrive/protontest.
name: ci
on:
push:
pull_request:
env:
# Never pick up a config holding a real Proton remote.
RCLONE_CONFIG: /notfound
jobs:
protondrive:
runs-on: ubuntu-latest
# The race suite takes about 6 minutes on a fast machine and 7 on
# two cores; the slow runner may take several times that.
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v5
with:
# The line the code was verified with, not go.mod's minimum,
# so govulncheck sees a patched standard library.
go-version: '1.27.x'
# Caches the module and build caches, keyed on go.sum.
cache: true
- name: Build
run: go build ./...
- name: Vet
run: go vet ./backend/protondrive/...
- name: Lint
uses: golangci/golangci-lint-action@v8
with:
version: v2.15.0
args: ./backend/protondrive/...
- name: Test
# Go's default 10 minute limit per test binary is too short
# for the race suite on a slow runner.
run: go test -race -timeout 40m ./backend/protondrive/...
- name: Fuzz
run: |
for target in $(go test -list '^Fuzz' ./backend/protondrive/ | grep '^Fuzz'); do
go test -run '^$' -fuzz "^${target}\$" -fuzztime 20s ./backend/protondrive/
done
- name: Vulnerability check
run: go run golang.org/x/vuln/cmd/govulncheck@v1.9.0 ./backend/protondrive/...
+12 -15
View File
@@ -37,7 +37,7 @@ jobs:
include:
- job_name: linux
os: ubuntu-latest
go: '~1.27.3'
go: '~1.26.9'
gotags: cmount
build_flags: '-include "^linux/"'
check: true
@@ -48,14 +48,14 @@ jobs:
- job_name: linux_386
os: ubuntu-latest
go: '~1.27.3'
go: '~1.26.9'
goarch: 386
gotags: cmount
quicktest: true
- job_name: mac_amd64
os: macos-latest
go: '~1.27.3'
go: '~1.26.9'
gotags: 'cmount'
build_flags: '-include "^darwin/amd64" -cgo'
quicktest: true
@@ -64,14 +64,14 @@ jobs:
- job_name: mac_arm64
os: macos-latest
go: '~1.27.3'
go: '~1.26.9'
gotags: 'cmount'
build_flags: '-include "^darwin/arm64" -cgo -macos-arch arm64 -cgo-cflags=-I/usr/local/include -cgo-ldflags=-L/usr/local/lib'
deploy: true
- job_name: windows
os: windows-latest
go: '~1.27.3'
go: '~1.26.9'
gotags: cmount
cgo: '0'
build_flags: '-include "^windows/"'
@@ -81,14 +81,14 @@ jobs:
- job_name: other_os
os: ubuntu-latest
go: '~1.27.3'
go: '~1.26.9'
build_flags: '-exclude "^(windows/|darwin/|linux/)"'
compile_all: true
deploy: true
- job_name: go1.26
os: ubuntu-latest
go: '~1.26.10'
go: '~1.26.9'
quicktest: true
racequicktest: true
@@ -177,12 +177,9 @@ jobs:
# https://github.com/Homebrew/brew/issues/15621#issuecomment-1619266788
# https://github.com/orgs/Homebrew/discussions/4612#discussioncomment-6319008
unset HOMEBREW_NO_INSTALL_FROM_API
# brew untap fails if the tap isn't installed, which varies between runner images
for tap in homebrew/core homebrew/cask aws/tap; do
if brew tap | grep -qx "$tap"; then
brew untap --force "$tap"
fi
done
brew untap --force homebrew/core
brew untap --force homebrew/cask
brew untap --force aws/tap
brew update
brew install --cask macfuse
brew install git-annex git-annex-remote-rclone
@@ -287,7 +284,7 @@ jobs:
id: setup-go
uses: actions/setup-go@v7
with:
go-version: '~1.27.3'
go-version: '~1.26.9'
check-latest: true
cache: false
@@ -402,7 +399,7 @@ jobs:
id: setup-go
uses: actions/setup-go@v7
with:
go-version: '~1.27.3'
go-version: '~1.26.9'
# Caching is handled explicitly below to share the module cache
# with the other jobs - see the build job for the rationale.
cache: false
-8
View File
@@ -19,12 +19,4 @@ Thumbs.db
__pycache__
.DS_Store
resource_windows_*.syso
/librclone.h
.devcontainer
/cmd/bisync/testdata/bisync_vscode_debuggers_launch.json
.env
.env.*
CLAUDE.local.md
.claude/settings.local.json
.worktrees/
.superpowers/
-15
View File
@@ -20,8 +20,6 @@ The core parts of rclone under `fs` and `vfs` need to work with all backends and
Make sure added tests **actually test the code you have written** and test the intention behind the change. If you are fixing a problem, write the tests first to reproduce the problem before starting on the fix.
Tests in `fs/operations`, `fs/sync`, `cmd/bisync`, `cmd/gitannex` and `vfs` (the definitive list is in `fstest/test_all/config.yaml`) are run on the integration test machine against **all** the backends, so they **must be prepared to run on backends with limited functionality**. Ideally tests, or parts of tests, should be skipped using the `Fs.Features()` flags on backends which can't support them.
## Build and Test Commands
```bash
@@ -134,16 +132,3 @@ Uses golangci-lint v2 with config in `.golangci.yml`. Enabled linters: errcheck,
- Command docs are in the command source code (e.g., `cmd/ls/ls.go`).
- Don't commit autogenerated doc changes from `make backenddocs` or `make commanddocs`.
- Website docs are in `docs/content/` as markdown, built with Hugo (`make serve` to preview).
## Fork Workflow
This fork carries agent tooling that upstream does not. Never include `docs/specs/`, `docs/plans/`, `docs/{ARCHITECTURE,DECISIONS,TODO,PERMISSIONS}.md`, `GEMINI.md`, `.claude/`, `.codex/`, `.gemini/` or this section in a PR to rclone/rclone.
- Required skill: `superpowers` - check at session start, install if missing. There is no product UI here, so `impeccable` is not needed.
- Specs go in `docs/specs/NNN-slug.md` (permanent). Plans go in `docs/plans/NNN-slug.md` and are removed with `git rm` before merge. This overrides the superpowers default paths.
- Paused or in-flight work is listed under "Now" in `docs/TODO.md` with the handoff to read first.
- Non-trivial work: brainstorm → spec → plan → TDD → review. For trivial fixes say "trivial, skipping spec" and proceed.
- Log things noticed but deliberately not fixed in `docs/TODO.md`, record decisions in `docs/DECISIONS.md`, and keep `docs/ARCHITECTURE.md` in step with structural changes - in the same commit as the code.
- `docs/PERMISSIONS.md` is the source for the `.claude/`, `.codex/` and `.gemini/` permission files - change them together.
- Never push, force-push or commit directly to `master`. Ask before adding or upgrading a dependency (`go get`, `go mod tidy`).
- Done means run in this session and seen passing: `go build`, `go test` on the touched packages, and `golangci-lint run` on them.
+1 -4
View File
@@ -35,9 +35,6 @@ RUN echo "**** Print Version Binary ****" && \
# Begin final image
FROM alpine:latest
ARG RCLONE_UID=1009
ARG RCLONE_GID=1009
RUN echo "**** Install Dependencies ****" && \
apk add --no-cache \
ca-certificates \
@@ -49,7 +46,7 @@ RUN echo "**** Install Dependencies ****" && \
COPY --from=builder /go/src/github.com/rclone/rclone/rclone /usr/local/bin/
RUN addgroup -g "${RCLONE_GID}" rclone && adduser -u "${RCLONE_UID}" -Ds /bin/sh -G rclone rclone
RUN addgroup -g 1009 rclone && adduser -u 1009 -Ds /bin/sh -G rclone rclone
ENTRYPOINT [ "rclone" ]
-8
View File
@@ -1,8 +0,0 @@
# GEMINI.md
**`AGENTS.md` is the single source of truth for this repository.** Read
`./AGENTS.md` before doing anything else and follow it exactly.
This file exists only so Gemini CLI finds the rules. It contains no project
rules of its own - add those to `AGENTS.md`. If anything here ever conflicts
with `AGENTS.md`, `AGENTS.md` wins.
Generated
+2503 -5994
View File
File diff suppressed because it is too large. Load diff
Generated
+442 -3287
View File
File diff suppressed because it is too large. Load diff
Generated
+482 -3557
View File
File diff suppressed because it is too large. Load diff
+14 -25
View File
@@ -38,9 +38,6 @@ BETA_PATH := $(BRANCH_PATH)$(TAG)$(BETA_SUBDIR)
BETA_URL := https://beta.rclone.org/$(BETA_PATH)/
BETA_UPLOAD_ROOT := beta.rclone.org:
BETA_UPLOAD := $(BETA_UPLOAD_ROOT)/$(BETA_PATH)
# Update the beta site's directory listings after an upload - add
# --changed for the uploaded directories
BETA_INDEX := --config bin/ci.rclone.conf -v index $(BETA_UPLOAD_ROOT) --use-server-modtime --checkers 16 --output index.html=html --output index.json=caddy --header-upload "Cache-Control: public, max-age=60" --exclude /favicon.ico
# Pass in GOTAGS=xyz on the make command line to set build tags
ifdef GOTAGS
BUILDTAGS=-tags "$(GOTAGS)"
@@ -237,7 +234,6 @@ check_sign:
upload:
rclone -P copy build/ downloads.rclone.org:/$(TAG)
rclone lsf build --files-only --include '*.{zip,deb,rpm}' --include version.txt | xargs -i bash -c 'i={}; j="$$i"; [[ $$i =~ (.*)(-v[0-9\.]+-)(.*) ]] && j=$${BASH_REMATCH[1]}-current-$${BASH_REMATCH[3]}; rclone copyto -v "downloads.rclone.org:/$(TAG)/$$i" "downloads.rclone.org:/$$j"'
rclone -P index downloads.rclone.org: --changed /$(TAG)/ --checkers 16 --output index.html=html --output index.json=caddy --header-upload "Cache-Control: public, max-age=60" --exclude /favicon.ico
upload_github:
./bin/upload-github $(TAG)
@@ -270,7 +266,6 @@ ci_upload:
ifeq ($(or $(BRANCH_PATH),$(RELEASE_TAG)),)
./rclone --no-check-dest --config bin/ci.rclone.conf -v copy build/ $(BETA_UPLOAD_ROOT)/test/testbuilds-latest
endif
./rclone $(BETA_INDEX) --changed $(BETA_PATH)/testbuilds/ $(if $(or $(BRANCH_PATH),$(RELEASE_TAG)),,--changed test/testbuilds-latest/)
@echo Beta release ready at $(BETA_URL)/testbuilds
ci_beta:
@@ -280,12 +275,11 @@ ci_beta:
ifeq ($(or $(BRANCH_PATH),$(RELEASE_TAG)),)
rclone --no-check-dest --config bin/ci.rclone.conf -v copy --include '*beta-latest*' --include version.txt build/ $(BETA_UPLOAD_ROOT)$(BETA_SUBDIR)
endif
rclone $(BETA_INDEX) --changed $(BETA_PATH)/ $(if $(or $(BRANCH_PATH),$(RELEASE_TAG)),,$(if $(BETA_SUBDIR),--changed $(BETA_SUBDIR)/))
@echo Beta release ready at $(BETA_URL)
# Fetch the binary builds from GitHub actions
fetch_binaries:
rclone -P sync --exclude "/testbuilds/**" --exclude "/index.{html,json}" --delete-excluded $(BETA_UPLOAD) build/
rclone -P sync --exclude "/testbuilds/**" --delete-excluded $(BETA_UPLOAD) build/
serve: website
cd docs && hugo server --logLevel info -w --disableFastRender --ignoreCache
@@ -302,26 +296,21 @@ retag:
@echo "Version is $(VERSION)"
git tag -f -s -m "Version $(VERSION)" $(VERSION)
# Start development of the next version after a release: the next
# minor version on master or the next patch version on a stable branch.
startdev:
@branch=$$(git rev-parse --abbrev-ref HEAD); \
case "$$branch" in \
master) next="$(NEXT_VERSION)" ;; \
v*-stable) next="$(NEXT_PATCH_VERSION)" ;; \
*) echo "Branch $$branch isn't master or a stable branch" >&2; exit 1 ;; \
esac; \
if ! git rev-parse -q --verify "refs/tags/$(VERSION)" >/dev/null; then \
echo "$(VERSION) hasn't been tagged yet - has development already been started?" >&2; exit 1; \
fi; \
echo "Version is $(VERSION)"; \
echo "Next version is $$next"; \
echo -e "package fs\n\n// VersionTag of rclone\nvar VersionTag = \"$$next\"\n" | gofmt > fs/versiontag.go; \
echo -n "$$next" > docs/layouts/partials/version.html; \
echo "$$next" > VERSION; \
git commit -m "Start $$next-DEV development" fs/versiontag.go VERSION docs/layouts/partials/version.html
@echo "Version is $(VERSION)"
@echo "Next version is $(NEXT_VERSION)"
echo -e "package fs\n\n// VersionTag of rclone\nvar VersionTag = \"$(NEXT_VERSION)\"\n" | gofmt > fs/versiontag.go
echo -n "$(NEXT_VERSION)" > docs/layouts/partials/version.html
echo "$(NEXT_VERSION)" > VERSION
git commit -m "Start $(NEXT_VERSION)-DEV development" fs/versiontag.go VERSION docs/layouts/partials/version.html
startstable: startdev
startstable:
@echo "Version is $(VERSION)"
@echo "Next stable version is $(NEXT_PATCH_VERSION)"
echo -e "package fs\n\n// VersionTag of rclone\nvar VersionTag = \"$(NEXT_PATCH_VERSION)\"\n" | gofmt > fs/versiontag.go
echo -n "$(NEXT_PATCH_VERSION)" > docs/layouts/partials/version.html
echo "$(NEXT_PATCH_VERSION)" > VERSION
git commit -m "Start $(NEXT_PATCH_VERSION)-DEV development" fs/versiontag.go VERSION docs/layouts/partials/version.html
winzip:
zip -9 rclone-$(TAG).zip rclone.exe
-9
View File
@@ -24,25 +24,21 @@ directories to and from different cloud storage providers.
## Storage providers
- 1Fichier [:page_facing_up:](https://rclone.org/fichier/)
- Abrha Storage [:page_facing_up:](https://rclone.org/s3/#abrhastorage)
- Akamai Netstorage [:page_facing_up:](https://rclone.org/netstorage/)
- Alibaba Cloud (Aliyun) Object Storage System (OSS) [:page_facing_up:](https://rclone.org/s3/#alibaba-oss)
- Amazon S3 [:page_facing_up:](https://rclone.org/s3/)
- ArvanCloud Object Storage (AOS) [:page_facing_up:](https://rclone.org/s3/#arvan-cloud-object-storage-aos)
- Autonomys Auto Drive [:page_facing_up:](https://rclone.org/s3/#autonomys)
- Bizfly Cloud Simple Storage [:page_facing_up:](https://rclone.org/s3/#bizflycloud)
- Backblaze B2 [:page_facing_up:](https://rclone.org/b2/)
- Box [:page_facing_up:](https://rclone.org/box/)
- Ceph [:page_facing_up:](https://rclone.org/s3/#ceph)
- China Mobile Ecloud Elastic Object Storage (EOS) [:page_facing_up:](https://rclone.org/s3/#china-mobile-ecloud-eos)
- Citrix ShareFile [:page_facing_up:](https://rclone.org/sharefile/)
- Clever Cloud Cellar [:page_facing_up:](https://rclone.org/s3/#clever-cloud-cellar)
- Cloudflare R2 [:page_facing_up:](https://rclone.org/s3/#cloudflare-r2)
- Cloudinary [:page_facing_up:](https://rclone.org/cloudinary/)
- Cubbit DS3 [:page_facing_up:](https://rclone.org/s3/#Cubbit)
- DigitalOcean Spaces [:page_facing_up:](https://rclone.org/s3/#digitalocean-spaces)
- Digi Storage [:page_facing_up:](https://rclone.org/koofr/#digi-storage)
- dosya.dev [:page_facing_up:](https://rclone.org/dosya/)
- Dreamhost [:page_facing_up:](https://rclone.org/s3/#dreamhost)
- Drime [:page_facing_up:](https://rclone.org/s3/#drime)
- Dropbox [:page_facing_up:](https://rclone.org/dropbox/)
@@ -74,11 +70,8 @@ directories to and from different cloud storage providers.
- Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
- IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
- Impossible Cloud [:page_facing_up:](https://rclone.org/s3/#impossible-cloud)
- Infomaniak Swiss Backup (S3) [:page_facing_up:](https://rclone.org/s3/#infomaniak)
- Infomaniak Swiss Backup (Swift) [:page_facing_up:](https://rclone.org/swift/)
- Intercolo Object Storage [:page_facing_up:](https://rclone.org/s3/#intercolo)
- IONOS Cloud [:page_facing_up:](https://rclone.org/s3/#ionos)
- kDrive [:page_facing_up:](https://rclone.org/kdrive/)
- Koofr [:page_facing_up:](https://rclone.org/koofr/)
- Leviia Object Storage [:page_facing_up:](https://rclone.org/s3/#leviia)
- Liara Object Storage [:page_facing_up:](https://rclone.org/s3/#liara-object-storage)
@@ -94,7 +87,6 @@ directories to and from different cloud storage providers.
- Microsoft Azure Files Storage [:page_facing_up:](https://rclone.org/azurefiles/)
- Microsoft OneDrive [:page_facing_up:](https://rclone.org/onedrive/)
- Minio [:page_facing_up:](https://rclone.org/s3/#minio)
- Neon Object Storage [:page_facing_up:](https://rclone.org/s3/#neon)
- Nextcloud [:page_facing_up:](https://rclone.org/webdav/#nextcloud)
- Blomp Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
- OpenDrive [:page_facing_up:](https://rclone.org/opendrive/)
@@ -118,7 +110,6 @@ directories to and from different cloud storage providers.
- Quatrix [:page_facing_up:](https://rclone.org/quatrix/)
- Rackspace Cloud Files [:page_facing_up:](https://rclone.org/swift/)
- RackCorp Object Storage [:page_facing_up:](https://rclone.org/s3/#RackCorp)
- RelAix.Storage [:page_facing_up:](https://rclone.org/s3/#relaix)
- rsync.net [:page_facing_up:](https://rclone.org/sftp/#rsync-net)
- Scaleway [:page_facing_up:](https://rclone.org/s3/#scaleway)
- Scality (RING / ARTESCA) [:page_facing_up:](https://rclone.org/s3/#scality)
+22 -81
View File
@@ -6,41 +6,27 @@ This file describes how to make the various kinds of releases
- [gh the github cli](https://github.com/cli/cli) for uploading packages
- pandoc for making the html and man pages
- hugo for building the website
- golangci-lint and docker for `make check` (markdownlint runs in docker)
- a gpg key for signing the tag and the checksums
- rclone remotes `beta.rclone.org:`, `downloads.rclone.org:`,
`www.rclone.org:` and `test-rclone-org:`
- the sponsor logos - see [Sponsor logos](#sponsor-logos)
## Making a release
- Enable the [release freeze rule](https://github.com/rclone/rclone/settings/rules/24751862)
- git checkout master # see below for stable branch
- git pull # IMPORTANT
- make fetch-gui-and-commit # bump the embedded GUI if rclone-web has a newer release
- git status - make sure everything is checked in
- Check GitHub actions build for master is Green
- Check the integration tests at <https://integration.rclone.org>
- cat VERSION # check this is the version being released
- make tag # signed tag on HEAD + new changelog entry + docs
- edit docs/content/changelog.md # entries already in point releases are
removed automatically
- make test # see integration test server or run locally
- make tag
- edit docs/content/changelog.md # duplicate logs from point rels removed automatically
releases
- make tidy
- make doc # again to pick up the changelog edits
- git status - to check for new command docs in docs/content/commands - git add them
- make doc
- git status - to check for new man pages - git add them
- git commit -a -v -m "Version v1.XX.0"
- make check # if this fails, fix, git commit --amend and run it again
- make retag # moves the tag onto the Version commit
- git push origin # without --follow-tags so it doesn't push the tag yet
- Wait for the GitHub actions build of master to complete and check it is
green - this is the first time CI sees any commits which were only local,
such as security fixes, so it must pass before the tag is pushed
- if it fails: fix, git commit, make retag, git push origin and wait again
- make check
- make retag
- git push origin # without --follow-tags so it doesn't push the tag if it fails
- git push --follow-tags origin
- \# Wait for the GitHub builds of the tag to complete then...
- \# Don't commit or push anything until after make upload_github as these steps
use the tag on HEAD to find the version
- \# Wait for the GitHub builds to complete then...
- make fetch_binaries
- make tarball
- make vendorball
@@ -49,31 +35,19 @@ This file describes how to make the various kinds of releases
- make upload
- make upload_test_website # check sponsors are correct/unchanged
- make upload_website
- make upload_github # this triggers the docker plugin and winget workflows
- Check the docker image, docker plugin and winget GitHub actions succeeded
- make startdev # works out the next version for master or a stable branch
- git push
- Disable the release freeze rule
- Release any security advisories
- request a CVE for each advisory if not done already
- set the patched version on each advisory and publish it
- when the CVEs are assigned replace `CVE-PENDING` in the changelog on
master and the stable branch then `make upload_website`
- \# announce with forum post, twitter post, and post to announcements list
- make upload_github
- make startdev # make startstable for stable branch
- \# announce with forum post, twitter post, and post to annoucements list
## Update dependencies
Early in the next release cycle update the dependencies.
- Active Pins
- `github.com/coreos/go-systemd/v22` v22.6.0 - v22.7.0 fails to compile on netbsd
- `github.com/hanwen/go-fuse/v2` v2.11.0 - v2.12.0 fails to compile on 32
bit linux, awaiting fix for
[go-fuse/issues/628](https://github.com/hanwen/go-fuse/issues/628)
- `github.com/gdamore/tcell/v2` v2.9.0 - awaiting fix for [tcell/issues/1124](https://github.com/gdamore/tcell/issues/1124)
- See: [#9539](https://github.com/rclone/rclone/issues/9539)
- Review Active Pins in go.mod and remove if possible
- `make updatedirect`
- The update ignores the pins so put them back, eg
`go get github.com/coreos/go-systemd/v22@v22.6.0` then `go mod tidy`
- `make GOTAGS=cmount`
- `make compiletest`
- Fix anything which doesn't compile at this point and commit changes here
@@ -216,53 +190,20 @@ First make the release branch. If this is a second point release then
this will be done already.
- git co -b ${BASE_TAG}-stable ${BASE_TAG}.0
- make startdev # sets the version to ${BASE_TAG}.1
- git push -u origin ${BASE_TAG}-stable
- make startstable
Now
- Enable release freeze rule
- git co master
- git pull --rebase
- make fetch-gui-and-commit # update the GUI on master first
- git push
- git co ${BASE_TAG}-stable
- git pull
- cat VERSION # check this is ${NEW_TAG} - if not run make startdev
- `git cherry-pick -x` any fixes, including the GUI update if there was one
- for security releases consider updating the Go patch version in
`.github/workflows/build.yml` and dependencies with CVE fixes
- fixes may depend on commits which weren't picked so push to CI
and check it is green before tagging
- Do the steps in "Making a release" above, except
- stay on ${BASE_TAG}-stable - don't checkout master or run
make fetch-gui-and-commit
- check the GitHub actions build for ${BASE_TAG}-stable not master
- run the integration tests on ${BASE_TAG}-stable as described below
- the release freeze rule is already enabled - leave it until the end
- `git cherry-pick -x` any fixes
- make startstable
- Do the steps as above
- git co master
- git pull --rebase
- `#` cherry pick the changes to the changelog - check the diff to make sure it
is correct
- git checkout ${BASE_TAG}-stable docs/content/changelog.md
- git diff --cached # check only the ${NEW_TAG} entry has changed
- git commit -v -m "Changelog updates from Version ${NEW_TAG}" docs/content/changelog.md
- git commit -a -v -m "Changelog updates from Version ${NEW_TAG}"
- git push
- Disable release freeze rule
Point releases are only made for the latest release. Once v1.(XX+1).0
is out the ${BASE_TAG}-stable branch is abandoned.
### Running the integration tests on the stable branch
The integration tests at <https://integration.rclone.org> test master.
To test the stable branch push it to the testing server and run the
tests there in `screen` as they take a while.
```console
git push -f rclone@rclone-testing:private-rclone.git ${BASE_TAG}-stable
ssh rclone@rclone-testing
screen
cd ~/integration-test && go run integration-test.go -repo /home/rclone/private-rclone.git -branch v1.XX-stable
```
## Sponsor logos
+1 -1
View File
@@ -1 +1 @@
v1.77.0
v1.75.2
-2
View File
@@ -16,7 +16,6 @@ import (
_ "github.com/rclone/rclone/backend/compress"
_ "github.com/rclone/rclone/backend/crypt"
_ "github.com/rclone/rclone/backend/doi"
_ "github.com/rclone/rclone/backend/dosya"
_ "github.com/rclone/rclone/backend/drime"
_ "github.com/rclone/rclone/backend/drive"
_ "github.com/rclone/rclone/backend/dropbox"
@@ -39,7 +38,6 @@ import (
_ "github.com/rclone/rclone/backend/internetarchive"
_ "github.com/rclone/rclone/backend/internxt"
_ "github.com/rclone/rclone/backend/jottacloud"
_ "github.com/rclone/rclone/backend/kdrive"
_ "github.com/rclone/rclone/backend/koofr"
_ "github.com/rclone/rclone/backend/linkbox"
_ "github.com/rclone/rclone/backend/local"
+2 -6
View File
@@ -194,12 +194,8 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (outFs fs
}
// Path to the archive
archiveRemote := remote[:len(foundArchive.remote)]
// Remote is archive leaf name. This must understand remote
// names as the archive may be in the root, e.g. "remote:file.zip"
_, foundArchive.remote, err = fspath.Split(archiveRemote)
if err != nil {
return nil, fmt.Errorf("failed to parse archive path %q: %w", archiveRemote, err)
}
// Remote is archive leaf name
foundArchive.remote = path.Base(archiveRemote)
foundArchive.prefix = ""
// Point remote to archive file
remote = archiveRemote
-147
View File
@@ -12,19 +12,16 @@ import (
"os/exec"
"path"
"path/filepath"
"sort"
"strconv"
"strings"
"testing"
_ "github.com/rclone/rclone/backend/alias"
"github.com/rclone/rclone/backend/archive/archiver"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/cache"
"github.com/rclone/rclone/fs/filter"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/sync"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/stretchr/testify/assert"
@@ -324,50 +321,6 @@ func TestArchiveUncleanRoot(t *testing.T) {
}
}
// TestArchiveInRemoteRoot checks that an archive in the root of a
// remote, with no "/" between the remote name and the archive name, can
// be read, as can a path within it.
func TestArchiveInRemoteRoot(t *testing.T) {
fstest.Initialise()
ctx := context.Background()
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
w, err := zw.Create("sub/a.txt")
require.NoError(t, err)
_, err = w.Write([]byte("data"))
require.NoError(t, err)
require.NoError(t, zw.Close())
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "test.zip"), buf.Bytes(), 0600))
t.Chdir(dir)
for _, test := range []struct {
name string
remote string
}{
{name: "plain", remote: ":local:test.zip"},
{name: "slash in remote name", remote: ":alias,remote='" + dir + "':test.zip"},
} {
for _, sub := range []struct {
path string
want string
}{
{path: "", want: "sub"},
{path: "/sub", want: "a.txt"},
} {
t.Run(test.name+sub.path, func(t *testing.T) {
f, err := cache.Get(ctx, ":archive:"+test.remote+sub.path)
require.NoError(t, err)
entries, err := f.List(ctx, "")
require.NoError(t, err)
require.Len(t, entries, 1)
assert.Equal(t, sub.want, entries[0].Remote())
})
}
}
}
// escapingObject is an object whose remote is not where it was asked for.
type escapingObject struct {
fs.Object
@@ -462,103 +415,3 @@ func TestIsDirectChild(t *testing.T) {
assert.Equal(t, test.want, isDirectChild(test.dir, test.remote), "dir=%q remote=%q", test.dir, test.remote)
}
}
// A symlink in a squashfs image is skipped unless -l/--links is in use,
// as it is on the other backends. With the flag it is exposed under its
// name with the link suffix and reads back as its target.
//
// Note that this uses mksquashfs as an external binary.
func TestArchiveSquashfsSymlink(t *testing.T) {
fstest.Initialise()
skipIfNoExe(t, "mksquashfs")
ctx := context.Background()
dir := t.TempDir()
input := filepath.Join(dir, "input")
require.NoError(t, os.Mkdir(input, 0777))
require.NoError(t, os.WriteFile(filepath.Join(input, "file.txt"), []byte("hello"), 0600))
require.NoError(t, os.Symlink("file.txt", filepath.Join(input, "link.txt")))
// a regular file which happens to carry the link suffix must stay
// readable under the flag, not be taken for a symlink
require.NoError(t, os.WriteFile(filepath.Join(input, "note"+fs.LinkSuffix), []byte("plain"), 0600))
image := filepath.Join(dir, "test.sqfs")
run(t, "mksquashfs", input, image)
remotes := func(t *testing.T, f fs.Fs) []string {
entries, err := f.List(ctx, "")
require.NoError(t, err)
names := make([]string, 0, len(entries))
for _, entry := range entries {
names = append(names, entry.Remote())
}
sort.Strings(names)
return names
}
read := func(t *testing.T, f fs.Fs, remote string) string {
o, err := f.NewObject(ctx, remote)
require.NoError(t, err)
rc, err := o.Open(ctx)
require.NoError(t, err)
contents, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, o.Size(), int64(len(contents)))
return string(contents)
}
f, err := fs.NewFs(ctx, ":archive:"+image)
require.NoError(t, err)
assert.Equal(t, []string{"file.txt", "note" + fs.LinkSuffix}, remotes(t, f))
ctx, ci := fs.AddConfig(ctx)
ci.Links = true
f, err = fs.NewFs(ctx, ":archive:"+image)
require.NoError(t, err)
assert.Equal(t, []string{"file.txt", "link.txt" + fs.LinkSuffix, "note" + fs.LinkSuffix}, remotes(t, f))
assert.Equal(t, "file.txt", read(t, f, "link.txt"+fs.LinkSuffix))
assert.Equal(t, "plain", read(t, f, "note"+fs.LinkSuffix))
}
// A hostile archive must not be able to write outside the destination.
// A symlink pointing above the root with a member inside it is the
// classic way to try it, and the refusal comes from the local backend,
// so pin it from end to end here rather than trusting it stays.
func TestArchiveZipSymlinkEscape(t *testing.T) {
fstest.Initialise()
ctx := context.Background()
ctx, ci := fs.AddConfig(ctx)
ci.Links = true
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
hdr := &zip.FileHeader{Name: "a"}
hdr.SetMode(os.ModeSymlink | 0777)
w, err := zw.CreateHeader(hdr)
require.NoError(t, err)
_, err = w.Write([]byte("../outside"))
require.NoError(t, err)
w, err = zw.Create("a/pwned.txt")
require.NoError(t, err)
_, err = w.Write([]byte("pwned"))
require.NoError(t, err)
require.NoError(t, zw.Close())
dir := t.TempDir()
zipPath := filepath.Join(dir, "evil.zip")
require.NoError(t, os.WriteFile(zipPath, buf.Bytes(), 0600))
dst := filepath.Join(dir, "dst")
require.NoError(t, os.Mkdir(dst, 0777))
fsrc, err := fs.NewFs(ctx, ":archive:"+zipPath)
require.NoError(t, err)
fdst, err := fs.NewFs(ctx, dst)
require.NoError(t, err)
assert.Error(t, sync.CopyDir(ctx, fdst, fsrc, false))
_, err = os.Stat(filepath.Join(dir, "outside", "pwned.txt"))
assert.True(t, os.IsNotExist(err), "the archive wrote outside the destination")
}
+1 -76
View File
@@ -43,7 +43,6 @@ type Fs struct {
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
file string // remote of the single file root points at, if any
links bool // read symlinks as .rclonelink files
}
// recoverParsePanic converts a panic raised while parsing a squashfs image
@@ -100,7 +99,6 @@ func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (_ f
c: c,
remote: remote,
root: strings.Trim(root, "/"),
links: fs.GetConfig(ctx).Links,
prefix: prefix,
prefixSlash: prefix + "/",
}
@@ -222,30 +220,6 @@ func (f *Fs) objectFromFileInfo(nativeDir string, item os.FileInfo) *Object {
}
}
// Convert a symlink entry into an Object holding its target
func (f *Fs) objectFromLink(nativeDir string, item os.FileInfo, target string) *Object {
return &Object{
fs: f,
remote: f.fromNative(nativeDir, item.Name()) + fs.LinkSuffix,
size: int64(len(target)),
modTime: item.ModTime(),
isLink: true,
link: target,
}
}
// readlink returns the target of the symlink the info describes.
//
// A symlink may carry an empty target, so the caller decides what is a
// symlink from the mode rather than from the target being set.
func readlink(info os.FileInfo) (string, error) {
st, ok := info.Sys().(*squashfs.StatT)
if !ok {
return "", fmt.Errorf("squashfs entry %q has no stat information", info.Name())
}
return st.LinkTarget, nil
}
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
@@ -275,7 +249,6 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
entries = make(fs.DirEntries, 0, len(items))
skipped := 0
skippedLinks := 0
for _, item := range items {
// fs.Debugf(item.Name(), "entry = %#v", item)
// Skip entry names that aren't safe
@@ -298,20 +271,6 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, fmt.Errorf("error reading item %q: %w", item.Name(), err)
}
entry = f.objectFromFileInfo(nativeDir, info)
} else if item.Type()&os.ModeSymlink != 0 {
if !f.links {
skippedLinks++
continue
}
info, err := item.Info()
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", item.Name(), err)
}
target, err := readlink(info)
if err != nil {
return nil, err
}
entry = f.objectFromLink(nativeDir, info, target)
} else {
fs.Debugf(item.Name(), "FIXME Not regular file - skipping")
continue
@@ -327,9 +286,6 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
if skipped > 0 {
fs.Logf(f, "Skipped %d squashfs entries in %q whose names escape the archive", skipped, dir)
}
if skippedLinks > 0 {
fs.Logf(f, "Skipped %d symlinks in %q - use -l/--links to read them", skippedLinks, dir)
}
// fs.Debugf(f, "dir=%q, entries=%v", dir, entries)
return entries, nil
@@ -351,17 +307,7 @@ func (f *Fs) newObjectNative(nativePath string) (o fs.Object, err error) {
}
for _, fi := range fis {
// Name the entry as List does, so that the two agree on
// which names exist
isLink := fi.Type()&os.ModeSymlink != 0
if isLink && !f.links {
continue
}
name := fi.Name()
if isLink {
name += fs.LinkSuffix
}
if name == leaf {
if fi.Name() == leaf {
if fi.IsDir() {
return nil, fs.ErrorNotAFile
}
@@ -369,14 +315,6 @@ func (f *Fs) newObjectNative(nativePath string) (o fs.Object, err error) {
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", fi.Name(), err)
}
if isLink {
target, err := readlink(info)
if err != nil {
return nil, err
}
o = f.objectFromLink(dir, info, target)
break
}
o = f.objectFromFileInfo(dir, info)
break
}
@@ -455,8 +393,6 @@ type Object struct {
remote string
size int64
modTime time.Time
isLink bool // whether this is a symlink
link string // target of the symlink
}
// Fs returns read only access to the Fs that this object is part of
@@ -528,17 +464,6 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.Read
}
}
if o.isLink {
// Clamp offset into range to avoid panic
if offset < 0 {
offset = 0
}
if offset > int64(len(o.link)) {
offset = int64(len(o.link))
}
return readers.NewLimitedReadCloser(io.NopCloser(strings.NewReader(o.link[offset:])), limit), nil
}
remote, err := o.fs.toNative(o.remote)
if err != nil {
return nil, err
-15
View File
@@ -43,7 +43,6 @@ type Fs struct {
prefix string // position for objects
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
links bool // read symlinks as .rclonelink files
dt dirtree.DirTree // read from zipfile
}
@@ -68,7 +67,6 @@ func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (fs.
node: node,
remote: remote,
root: root,
links: fs.GetConfig(ctx).Links,
prefix: prefix,
prefixSlash: prefix + "/",
}
@@ -140,7 +138,6 @@ func (f *Fs) readZip() (singleObject bool, err error) {
}
dt := dirtree.New()
skipped := 0
skippedLinks := 0
for _, file := range zr.File {
// Skip entries whose name escapes the archive's own namespace
remote, err := sanitize.Path(file.Name)
@@ -155,15 +152,6 @@ func (f *Fs) readZip() (singleObject bool, err error) {
skipped++
continue
}
if !isDir && file.Mode()&os.ModeSymlink != 0 {
if !f.links {
skippedLinks++
continue
}
// The member holds the link target, so it reads back
// like a .rclonelink file
remote += fs.LinkSuffix
}
remote = path.Join(f.prefix, remote)
if f.root != "" {
// Ignore all files outside the root, requiring a path
@@ -203,9 +191,6 @@ func (f *Fs) readZip() (singleObject bool, err error) {
if skipped > 0 {
fs.Logf(f, "Skipped %d zip entries which escape the archive", skipped)
}
if skippedLinks > 0 {
fs.Logf(f, "Skipped %d symlinks - use -l/--links to read them", skippedLinks)
}
dt.CheckParents("")
dt.Sort()
f.dt = dt
-55
View File
@@ -5,7 +5,6 @@ import (
"bytes"
"context"
"fmt"
"io"
"os"
"path/filepath"
"sort"
@@ -37,26 +36,6 @@ func writeZip(t *testing.T, dir, name string, names ...string) string {
return name
}
// writeZipSymlink builds a zip holding file and a symlink entry link
// pointing at target, and returns its leaf name.
func writeZipSymlink(t *testing.T, dir, name, file, link, target string) string {
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
w, err := zw.Create(file)
require.NoError(t, err)
_, err = w.Write([]byte("data for " + file))
require.NoError(t, err)
hdr := &zip.FileHeader{Name: link}
hdr.SetMode(os.ModeSymlink | 0777)
w, err = zw.CreateHeader(hdr)
require.NoError(t, err)
_, err = w.Write([]byte(target))
require.NoError(t, err)
require.NoError(t, zw.Close())
require.NoError(t, os.WriteFile(filepath.Join(dir, name), buf.Bytes(), 0600))
return name
}
// allRemotes returns every Object remote in the mounted archive's dirtree.
func allRemotes(t *testing.T, f fs.Fs) []string {
zf, ok := f.(*Fs)
@@ -191,37 +170,3 @@ func TestListCacheNotAliased(t *testing.T) {
}
assert.Equal(t, []string{"a.txt", "b.txt", "c.txt"}, remotes)
}
// A symlink stored in the archive is skipped unless -l/--links is in
// use, as it is on the other backends. With the flag it is exposed
// under its name with the link suffix and reads back as its target.
func TestReadZipSymlink(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
name := writeZipSymlink(t, dir, "links.zip", "file.txt", "link.txt", "file.txt")
localFs, err := cache.Get(ctx, dir)
require.NoError(t, err)
f, err := New(ctx, localFs, name, "", "")
require.NoError(t, err)
assert.Equal(t, []string{"file.txt"}, allRemotes(t, f))
ctx, ci := fs.AddConfig(ctx)
ci.Links = true
f, err = New(ctx, localFs, name, "", "")
require.NoError(t, err)
assert.Equal(t, []string{"file.txt", "link.txt" + fs.LinkSuffix}, allRemotes(t, f))
o, err := f.NewObject(ctx, "link.txt"+fs.LinkSuffix)
require.NoError(t, err)
assert.Equal(t, int64(len("file.txt")), o.Size())
rc, err := o.Open(ctx)
require.NoError(t, err)
target, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, "file.txt", string(target))
}
+4 -4
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
// Parallel listing support for azureblob, built on the Apache Arrow
// listing format.
@@ -62,7 +62,7 @@ func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, pr
// Schedule every shard; SetLimit bounds concurrency, so more shards than
// workers gives free load balancing as workers pick up the next shard.
var total atomic.Int64
var total int64
g, gCtx := errgroup.WithContext(ctx)
g.SetLimit(f.opt.ListParallelism)
for _, sh := range shards {
@@ -85,12 +85,12 @@ func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, pr
shardOpts.EndBefore = nil
}
n, err := f.listBlobsPager(gCtx, containerName, directory, prefix, addContainer, &shardOpts, delimiter, safeFn)
total.Add(int64(n))
atomic.AddInt64(&total, int64(n))
return err
})
}
err := g.Wait()
found := int(total.Load())
found := int(atomic.LoadInt64(&total))
if isEndBeforeUnsupported(err) {
// The account can't honour the shards' endBefore bounds, so parallel
// listing can't be used on it.
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
package azureblob
+38 -171
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
// Package azureblob provides an interface to the Microsoft Azure blob object storage system
package azureblob
@@ -19,7 +19,6 @@ import (
"path"
"slices"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
@@ -365,22 +364,7 @@ creates an empty object ending with "/", to persist the folder.
This object also has the metadata "` + dirMetaKey + ` = ` + dirMetaValue + `" to conform to
the Microsoft standard.
This is not needed on storage accounts with a hierarchical namespace
(see the hns option) as these always support empty directories.
`,
}, {
Name: "hns",
Default: fs.Tristate{},
Advanced: true,
Help: `Set if the storage account has a hierarchical namespace.
Storage accounts with hierarchical namespace enabled (Azure Data Lake
Storage Gen2) have real directories which can be empty.
If this is unset then rclone will read it from the storage account
and save it in the config file.
`,
}, {
Name: "no_check_container",
Help: `If set, don't attempt to check the container exists or create it.
@@ -448,7 +432,6 @@ type Options struct {
Enc encoder.MultiEncoder `config:"encoding"`
PublicAccess string `config:"public_access"`
DirectoryMarkers bool `config:"directory_markers"`
HNS fs.Tristate `config:"hns"`
NoCheckContainer bool `config:"no_check_container"`
NoHeadObject bool `config:"no_head_object"`
DeleteSnapshots string `config:"delete_snapshots"`
@@ -472,7 +455,6 @@ type Fs struct {
rootContainer string // container part of root (if any)
rootDirectory string // directory part of root (if any)
isLimited bool // if limited to one container
hns bool // if the storage account has a hierarchical namespace
cache *bucket.Cache // cache for container creation status
pacer *fs.Pacer // To pace and retry the API calls
uploadToken *pacer.TokenDispenser // control concurrency
@@ -539,7 +521,7 @@ func parsePath(path string) (root string) {
// relative to f.root
func (f *Fs) split(rootRelativePath string) (containerName, containerPath string) {
containerName, containerPath = bucket.Split(bucket.Join(f.root, rootRelativePath))
if f.dirMarkers() && strings.HasSuffix(containerPath, "//") {
if f.opt.DirectoryMarkers && strings.HasSuffix(containerPath, "//") {
containerPath = containerPath[:len(containerPath)-1]
}
return f.opt.Enc.FromStandardName(containerName), f.opt.Enc.FromStandardPath(containerPath)
@@ -587,7 +569,8 @@ func (f *Fs) shouldRetry(ctx context.Context, err error) (bool, error) {
if fserrors.ContextError(ctx, &err) {
return false, err
}
if storageErr, ok := errors.AsType[*azcore.ResponseError](err); ok {
var storageErr *azcore.ResponseError
if errors.As(err, &storageErr) {
// General errors from:
// https://learn.microsoft.com/en-us/rest/api/storageservices/common-rest-api-error-codes
// Blob specific errors from:
@@ -706,6 +689,10 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
ServerSideAcrossConfigs: true,
DoubleSlash: true,
}).Fill(ctx, f)
if opt.DirectoryMarkers {
f.features.CanHaveEmptyDirectories = true
fs.Debugf(f, "Using directory markers")
}
conf := auth.NewClientOpts[service.Client, service.ClientOptions, service.SharedKeyCredential]{
DefaultBaseURL: storageDefaultBaseURL,
@@ -738,28 +725,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
f.isLimited = true
}
if opt.HNS.Valid {
f.hns = opt.HNS.Value
} else if !f.anonymous {
containerName := f.rootContainer
if containerName == "" {
containerName = res.Container
}
f.hns, err = f.readHNS(ctx, containerName)
if err != nil {
fs.Debugf(f, "Failed to read whether storage account has a hierarchical namespace - assuming not: %v", err)
} else {
m.Set("hns", strconv.FormatBool(f.hns))
}
}
if f.hns {
fs.Debugf(f, "Storage account has a hierarchical namespace")
}
if f.dirMarkers() {
f.features.CanHaveEmptyDirectories = true
fs.Debugf(f, "Using directory markers")
}
if f.rootContainer != "" && f.rootDirectory != "" {
// Check to see if the (container,directory) is actually an existing file
oldRoot := f.root
@@ -780,39 +745,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
return f, nil
}
// readHNS reads whether the storage account has a hierarchical namespace
//
// If containerName is set then the container client is used which works
// with credentials limited to that container.
func (f *Fs) readHNS(ctx context.Context, containerName string) (hns bool, err error) {
var isHNS *bool
err = f.pacer.Call(func() (bool, error) {
var err error
if containerName != "" {
var resp container.GetAccountInfoResponse
resp, err = f.cntSVC(containerName).GetAccountInfo(ctx, nil)
isHNS = resp.IsHierarchicalNamespaceEnabled
} else {
var resp service.GetAccountInfoResponse
resp, err = f.svc.GetAccountInfo(ctx, nil)
isHNS = resp.IsHierarchicalNamespaceEnabled
}
return f.shouldRetry(ctx, err)
})
if err != nil {
return false, err
}
if isHNS == nil {
return false, errors.New("not returned by server")
}
return *isHNS, nil
}
// dirMarkers returns true if directories are persisted with directory markers
func (f *Fs) dirMarkers() bool {
return f.opt.DirectoryMarkers || f.hns
}
// return the container client for the container passed in
func (f *Fs) cntSVC(containerName string) (containerClient *container.Client) {
f.cntSVCcacheMu.Lock()
@@ -850,12 +782,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *contain
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
o, err := f.newObjectWithInfo(ctx, remote, nil)
if err == fs.ErrorNotAFile {
// The blob is a directory
return nil, fs.ErrorObjectNotFound
}
return o, err
return f.newObjectWithInfo(ctx, remote, nil)
}
// getBlobSVC creates a blob client
@@ -924,15 +851,15 @@ func mapMetadataToAzure(meta map[string]string, logf func(string, ...any)) (head
lowerKey := strings.ToLower(k)
switch lowerKey {
case "cache-control":
headers.BlobCacheControl = new(v)
headers.BlobCacheControl = pString(v)
case "content-disposition":
headers.BlobContentDisposition = new(v)
headers.BlobContentDisposition = pString(v)
case "content-encoding":
headers.BlobContentEncoding = new(v)
headers.BlobContentEncoding = pString(v)
case "content-language":
headers.BlobContentLanguage = new(v)
headers.BlobContentLanguage = pString(v)
case "content-type":
headers.BlobContentType = new(v)
headers.BlobContentType = pString(v)
case "x-ms-tags":
parsed, perr := parseXMsTags(v)
if perr != nil {
@@ -1169,33 +1096,18 @@ func isDirectoryMarker(size int64, metadata map[string]*string, remote string) b
if endsWithSlash || remote == "" {
return true
}
return hasDirMetadata(metadata)
}
return false
}
// Returns whether the metadata marks the blob as a directory
func hasDirMetadata(metadata map[string]*string) bool {
// Note that metadata with hdi_isfolder = true seems to be a
// defacto standard for marking blobs as directories.
// Note also that the metadata hasn't been normalised to lower case yet
for k, v := range metadata {
if v != nil && strings.EqualFold(k, dirMetaKey) && *v == dirMetaValue {
return true
// Note that metadata with hdi_isfolder = true seems to be a
// defacto standard for marking blobs as directories.
// Note also that the metadata hasn't been normalised to lower case yet
for k, v := range metadata {
if v != nil && strings.EqualFold(k, dirMetaKey) && *v == dirMetaValue {
return true
}
}
}
return false
}
// Returns whether props are for a directory on a storage account with
// a hierarchical namespace
//
// These ignore a trailing "/" on the name so a file can be returned
// when looking for a directory.
func isHNSDirectory(props *blob.GetPropertiesResponse) bool {
return props.ContentLength != nil && *props.ContentLength == 0 && hasDirMetadata(props.Metadata)
}
// listFn is called from list to handle an object
type listFn func(remote string, object *container.BlobItem, isDirectory bool) error
@@ -1254,18 +1166,15 @@ func (f *Fs) list(ctx context.Context, containerName, directory, prefix string,
if err != nil {
return err
}
if f.dirMarkers() && foundItems == 0 && directory != "" {
if f.opt.DirectoryMarkers && foundItems == 0 && directory != "" {
// Determine whether the directory exists or not by whether it has a marker
props, err := f.readMetaData(ctx, containerName, directory)
_, err := f.readMetaData(ctx, containerName, directory)
if err != nil {
if err == fs.ErrorObjectNotFound {
return fs.ErrorDirNotFound
}
return err
}
if f.hns && !isHNSDirectory(props) {
return fs.ErrorDirNotFound
}
}
return nil
}
@@ -1318,13 +1227,6 @@ func (f *Fs) listBlobsPager(ctx context.Context, containerName, directory, prefi
fs.Debugf(f, "Nil name received")
continue
}
// Storage accounts with a hierarchical namespace return
// the file "dir" when listing "dir/"
if !strings.HasPrefix(*file.Name, directory) {
fs.Debugf(f, "Ignoring %q not in directory %q", *file.Name, directory)
foundItems--
continue
}
remote := f.opt.Enc.ToStandardPath(*file.Name)
if !strings.HasPrefix(remote, prefix) {
fs.Debugf(f, "Odd name received %q", remote)
@@ -1586,7 +1488,7 @@ type listContainerFn func(Name string, LastModified time.Time) error
func (f *Fs) listContainersToFn(fn listContainerFn) error {
max := int32(f.opt.ListChunkSize)
pager := f.svc.NewListContainersPager(&service.ListContainersOptions{
Include: service.ListContainersInclude{Metadata: true},
Include: service.ListContainersInclude{Metadata: true, Deleted: true},
MaxResults: &max,
})
ctx := context.Background()
@@ -1636,7 +1538,7 @@ func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, opt
// Create directory marker file and parents
func (f *Fs) createDirectoryMarker(ctx context.Context, container, dir string) error {
if !f.dirMarkers() || container == "" {
if !f.opt.DirectoryMarkers || container == "" {
return nil
}
@@ -1693,11 +1595,6 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
// mkdirParent creates the parent bucket/directory if it doesn't exist
func (f *Fs) mkdirParent(ctx context.Context, remote string) error {
if f.hns {
// Parent directories are created by the upload
container, _ := f.split(remote)
return f.makeContainer(ctx, container)
}
remote, _ = strings.CutSuffix(remote, "/")
dir := path.Dir(remote)
if dir == "/" || dir == "." {
@@ -1803,9 +1700,6 @@ func (f *Fs) deleteContainer(ctx context.Context, containerName string) error {
// Returns an error if it isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
container, directory := f.split(dir)
if f.hns && container != "" && directory != "" {
return f.rmdirHNS(ctx, container, directory)
}
// Remove directory marker file
if f.opt.DirectoryMarkers && container != "" && directory != "" {
o := &Object{
@@ -1828,39 +1722,6 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return f.deleteContainer(ctx, container)
}
// rmdirHNS removes the directory on a storage account with a
// hierarchical namespace
//
// The directory must be deleted without a trailing "/" and the server
// refuses to delete it if it isn't empty.
func (f *Fs) rmdirHNS(ctx context.Context, container, directory string) error {
directory, _ = strings.CutSuffix(directory, "/")
// Check it is a directory as deleting a file would succeed
props, err := f.readMetaData(ctx, container, directory)
if err == fs.ErrorObjectNotFound {
return fs.ErrorDirNotFound
} else if err != nil {
return err
}
if !isHNSDirectory(props) {
return fs.ErrorIsFile
}
blb := f.getBlobSVC(container, directory)
err = f.pacer.Call(func() (bool, error) {
_, err := blb.Delete(ctx, nil)
return f.shouldRetry(ctx, err)
})
if storageErr, ok := errors.AsType[*azcore.ResponseError](err); ok {
if storageErr.ErrorCode == "DirectoryIsNotEmpty" {
return fs.ErrorDirectoryNotEmpty
}
if storageErr.StatusCode == http.StatusNotFound {
return fs.ErrorDirNotFound
}
}
return err
}
// Precision of the remote
func (f *Fs) Precision() time.Duration {
return time.Nanosecond
@@ -2699,6 +2560,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
return downloadResponse.Body, nil
}
// Converts a string into a pointer to a string
func pString(s string) *string {
return &s
}
// readSeekCloser joins an io.Reader and an io.Seeker and provides a no-op io.Closer
type readSeekCloser struct {
io.Reader
@@ -2853,7 +2719,8 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
// isInvalidBlockOrBlob looks for the InvalidBlockOrBlob error in err
// returning true if it is found
func isInvalidBlockOrBlob(err error) bool {
if storageErr, ok := errors.AsType[*azcore.ResponseError](err); ok {
var storageErr *azcore.ResponseError
if errors.As(err, &storageErr) {
return storageErr.ErrorCode == string(bloberror.InvalidBlobOrBlock)
}
return false
@@ -3225,7 +3092,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
// Start with default content-type based on source
ui.httpHeaders = blob.HTTPHeaders{
BlobContentType: new(fs.MimeType(ctx, src)),
BlobContentType: pString(fs.MimeType(ctx, src)),
}
// Apply mapped metadata/headers/tags if requested
@@ -3271,15 +3138,15 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
o.tags[parts[0]] = parts[1]
}
case "cache-control":
ui.httpHeaders.BlobCacheControl = new(value)
ui.httpHeaders.BlobCacheControl = pString(value)
case "content-disposition":
ui.httpHeaders.BlobContentDisposition = new(value)
ui.httpHeaders.BlobContentDisposition = pString(value)
case "content-encoding":
ui.httpHeaders.BlobContentEncoding = new(value)
ui.httpHeaders.BlobContentEncoding = pString(value)
case "content-language":
ui.httpHeaders.BlobContentLanguage = new(value)
ui.httpHeaders.BlobContentLanguage = pString(value)
case "content-type":
ui.httpHeaders.BlobContentType = new(value)
ui.httpHeaders.BlobContentType = pString(value)
}
}
+1 -13
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
package azureblob
@@ -325,14 +325,6 @@ func assertHeadersAndMetadata(t *testing.T, props *blob.GetPropertiesResponse, w
}
// helper to read blob tags for an object
// skipIfHNS skips the test if the storage account has a hierarchical
// namespace as these don't support blob tags
func skipIfHNS(t *testing.T, f *Fs) {
if f.hns {
t.Skip("blob tags not supported with hierarchical namespace")
}
}
func getTagsMap(ctx context.Context, t *testing.T, o fs.Object) map[string]string {
ao := o.(*Object)
blb := ao.getBlobSVC()
@@ -418,7 +410,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Singlepart upload
t.Run("PutSinglepartTags", func(t *testing.T) {
skipIfHNS(t, f)
contents := random.String(int(f.opt.ChunkSize / 2))
item := fstest.NewItem("tags-single.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
@@ -434,7 +425,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Multipart upload
t.Run("PutMultipartTags", func(t *testing.T) {
skipIfHNS(t, f)
contents := random.String(int(f.opt.ChunkSize + 2048))
item := fstest.NewItem("tags-multipart.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
@@ -622,7 +612,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Singlepart copy
t.Run("CopySinglepartTags", func(t *testing.T) {
skipIfHNS(t, f)
// create small source
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("tags-copy-single-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
@@ -648,7 +637,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Multipart copy
t.Run("CopyMultipartTags", func(t *testing.T) {
skipIfHNS(t, f)
// create large source to force multipart
contents := random.String(int(f.opt.CopyCutoff + 4096))
srcItem := fstest.NewItem("tags-copy-multi-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
+1 -1
View File
@@ -1,6 +1,6 @@
// Test AzureBlob filesystem interface
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
package azureblob
+1 -1
View File
@@ -1,7 +1,7 @@
// Build for azureblob for unsupported platforms to stop go complaining
// about "no buildable Go source files "
//go:build plan9 || js
//go:build plan9 || solaris || js
// Package azureblob provides an interface to the Microsoft Azure blob object storage system
package azureblob
+13 -8
View File
@@ -79,7 +79,7 @@ This is required and is the name of the share to access.
Help: `Upload chunk size.
Note that this is stored in memory and there may be up to
"--transfers" * "--azurefiles-upload-concurrency" chunks stored at once
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.`,
Default: defaultChunkSize,
Advanced: true,
@@ -95,7 +95,7 @@ links and these uploads do not fully utilize your bandwidth, then
increasing this may help to speed up the transfers.
Note that chunks are stored in memory and there may be up to
"--transfers" * "--azurefiles-upload-concurrency" chunks stored at once
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.`,
Default: 16,
Advanced: true,
@@ -650,6 +650,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
return resp.Body, nil
}
// Returns a pointer to t - useful for returning pointers to constants
func ptr[T any](t T) *T {
return &t
}
var warnStreamUpload sync.Once
// Update the object with the contents of the io.Reader, modTime, size and MD5 hash
@@ -813,8 +818,8 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
return nil, fmt.Errorf("Move: mkParentDir failed: %w", err)
}
opt := file.RenameOptions{
IgnoreReadOnly: new(true),
ReplaceIfExists: new(true),
IgnoreReadOnly: ptr(true),
ReplaceIfExists: ptr(true),
}
dstAbsPath := f.absPath(remote)
fc := srcObj.fileClient()
@@ -859,8 +864,8 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
}
opt := directory.RenameOptions{
IgnoreReadOnly: new(false),
ReplaceIfExists: new(false),
IgnoreReadOnly: ptr(false),
ReplaceIfExists: ptr(false),
}
dstAbsPath := dstFs.absPath(dstRemote)
dirClient := srcFs.dirClient(srcRemote)
@@ -899,8 +904,8 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
ChangeTime: file.SourceCopyFileChangeTime{},
CreationTime: file.SourceCopyFileCreationTime{},
LastWriteTime: file.SourceCopyFileLastWriteTime{},
PermissionCopyMode: new(file.PermissionCopyModeTypeSource),
IgnoreReadOnly: new(true),
PermissionCopyMode: ptr(file.PermissionCopyModeTypeSource),
IgnoreReadOnly: ptr(true),
},
}
srcURL := srcObj.fileClient().URL()
+2 -2
View File
@@ -277,7 +277,7 @@ See: [rclone backend lifecycle](#lifecycle) for setting lifecycles after bucket
Name: "sse_customer_key",
Help: `To use SSE-C, you may provide the secret encryption key encoded in a UTF-8 compatible string to encrypt/decrypt your data
Alternatively you can provide --b2-sse-customer-key-base64.`,
Alternatively you can provide --sse-customer-key-base64.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -288,7 +288,7 @@ Alternatively you can provide --b2-sse-customer-key-base64.`,
Name: "sse_customer_key_base64",
Help: `To use SSE-C, you may provide the secret encryption key encoded in Base64 format to encrypt/decrypt your data
Alternatively you can provide --b2-sse-customer-key.`,
Alternatively you can provide --sse-customer-key.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
+9 -10
View File
@@ -27,7 +27,7 @@ import (
"sync/atomic"
"time"
"github.com/golang-jwt/jwt/v5"
"github.com/golang-jwt/jwt/v4"
"github.com/rclone/rclone/backend/box/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
@@ -77,10 +77,7 @@ var (
)
type boxCustomClaims struct {
jwt.RegisteredClaims
// Audience overrides RegisteredClaims.Audience so it is sent as
// a single string rather than an array of strings
Audience string `json:"aud,omitempty"`
jwt.StandardClaims
BoxSubType string `json:"box_sub_type,omitempty"`
}
@@ -226,7 +223,7 @@ func getBoxConfig(m configmap.Mapper) (boxConfig *api.ConfigJSON, err error) {
if configFileCredentials == "" {
configFile, _ := m.Get("box_config_file")
configFileBytes, err = os.ReadFile(env.ShellExpand(configFile))
configFileBytes, err = os.ReadFile(configFile)
if err != nil {
return nil, fmt.Errorf("box: failed to read Box config: %w", err)
}
@@ -246,13 +243,15 @@ func getClaims(boxConfig *api.ConfigJSON, boxSubType string) (claims *boxCustomC
}
claims = &boxCustomClaims{
RegisteredClaims: jwt.RegisteredClaims{
ID: val,
//lint:ignore SA1019 since we need to use jwt.StandardClaims even if deprecated in jwt-go v4 until a more permanent solution is ready in time before jwt-go v5 where it is removed entirely
//nolint:staticcheck // Don't include staticcheck when running golangci-lint to avoid SA1019
StandardClaims: jwt.StandardClaims{
Id: val,
Issuer: boxConfig.BoxAppSettings.ClientID,
Subject: boxConfig.EnterpriseID,
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Second * 45)),
Audience: tokenURL,
ExpiresAt: time.Now().Add(time.Second * 45).Unix(),
},
Audience: tokenURL,
BoxSubType: boxSubType,
}
return claims, nil
-118
View File
@@ -1,118 +0,0 @@
package box
import (
"crypto"
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/rclone/rclone/backend/box/api"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/lib/jwtutil"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// decodeJWTPart decodes one base64url encoded part of a JWT into a map
func decodeJWTPart(t *testing.T, part string) map[string]any {
data, err := base64.RawURLEncoding.DecodeString(part)
require.NoError(t, err)
var result map[string]any
require.NoError(t, json.Unmarshal(data, &result))
return result
}
// Test the JWT assertion sent to the token endpoint is in the format Box expects
func TestJWTAssertion(t *testing.T) {
privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
boxConfig := &api.ConfigJSON{
BoxAppSettings: api.AppSettings{
ClientID: "client-id",
ClientSecret: "client-secret",
AppAuth: api.AppAuth{
PublicKeyID: "public-key-id",
},
},
EnterpriseID: "enterprise-id",
}
var form map[string][]string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "POST", r.Method)
assert.NoError(t, r.ParseForm())
form = r.PostForm
_, _ = w.Write([]byte(`{"access_token":"access-token","token_type":"bearer","expires_in":3600}`))
}))
defer ts.Close()
claims, err := getClaims(boxConfig, "enterprise")
require.NoError(t, err)
m := configmap.Simple{}
start := time.Now()
err = jwtutil.Config("box", "test", ts.URL, *claims, getSigningHeaders(boxConfig), getQueryParams(boxConfig), privateKey, m, ts.Client())
require.NoError(t, err)
// Check the token was stored
assert.Contains(t, m["token"], `"access_token":"access-token"`)
// Check the form parameters
assert.Equal(t, []string{"urn:ietf:params:oauth:grant-type:jwt-bearer"}, form["grant_type"])
assert.Equal(t, []string{"client-id"}, form["client_id"])
assert.Equal(t, []string{"client-secret"}, form["client_secret"])
require.Len(t, form["assertion"], 1)
parts := strings.Split(form["assertion"][0], ".")
require.Len(t, parts, 3)
// Check the signature independently of the JWT library
signature, err := base64.RawURLEncoding.DecodeString(parts[2])
require.NoError(t, err)
hash := sha256.Sum256([]byte(parts[0] + "." + parts[1]))
require.NoError(t, rsa.VerifyPKCS1v15(&privateKey.PublicKey, crypto.SHA256, hash[:], signature))
// Check the header
assert.Equal(t, map[string]any{
"alg": "RS256",
"typ": "JWT",
"kid": "public-key-id",
}, decodeJWTPart(t, parts[0]))
// Check the claims - aud must be a string, not an array
payload := decodeJWTPart(t, parts[1])
exp, ok := payload["exp"].(float64)
require.True(t, ok, "exp should be a number")
assert.InDelta(t, float64(start.Add(45*time.Second).Unix()), exp, 5)
jti, ok := payload["jti"].(string)
require.True(t, ok, "jti should be a string")
assert.Len(t, jti, 40)
delete(payload, "exp")
delete(payload, "jti")
assert.Equal(t, map[string]any{
"aud": tokenURL,
"iss": "client-id",
"sub": "enterprise-id",
"box_sub_type": "enterprise",
}, payload)
}
// Test box_config_file has ~ and environment variables expanded
func TestGetBoxConfigShellExpand(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"enterpriseID":"enterprise-id"}`), 0600))
t.Setenv("RCLONE_TEST_BOX_CONFIG_DIR", dir)
m := configmap.Simple{
"box_config_file": "$RCLONE_TEST_BOX_CONFIG_DIR/config.json",
}
boxConfig, err := getBoxConfig(m)
require.NoError(t, err)
assert.Equal(t, "enterprise-id", boxConfig.EnterpriseID)
}
+15 -23
View File
@@ -3,6 +3,7 @@
package box
import (
"bytes"
"context"
"crypto/sha1"
"encoding/base64"
@@ -19,7 +20,6 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/lib/atexit"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/rest"
)
@@ -55,15 +55,9 @@ func sha1Digest(digest []byte) string {
}
// uploadPart uploads a part in an upload session
//
// chunk must contain exactly the part's data and is read from the start
// on each attempt.
func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, totalSize int64, chunk io.ReadSeeker, wrap accounting.WrapFn, options ...fs.OpenOption) (response *api.UploadPartResponse, err error) {
hasher := sha1.New()
chunkSize, err := io.Copy(hasher, chunk)
if err != nil {
return nil, err
}
func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, totalSize int64, chunk []byte, wrap accounting.WrapFn, options ...fs.OpenOption) (response *api.UploadPartResponse, err error) {
chunkSize := int64(len(chunk))
sha1sum := sha1.Sum(chunk)
opts := rest.Opts{
Method: "PUT",
Path: "/files/upload_sessions/" + SessionID,
@@ -73,16 +67,12 @@ func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, total
ContentRange: fmt.Sprintf("bytes %d-%d/%d", offset, offset+chunkSize-1, totalSize),
Options: options,
ExtraHeaders: map[string]string{
"Digest": sha1Digest(hasher.Sum(nil)),
"Digest": sha1Digest(sha1sum[:]),
},
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
_, err = chunk.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
opts.Body = wrap(chunk)
opts.Body = wrap(bytes.NewReader(chunk))
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &response)
return shouldRetry(ctx, resp, err)
})
@@ -223,25 +213,27 @@ outer:
reqSize := min(remaining, chunkSize)
// Read the chunk into memory from the global pool, hashing
// it for the whole file hash (must be done sequentially)
rw := multipart.NewRW().Reserve(reqSize)
_, err = io.CopyN(rw, io.TeeReader(in, hash), reqSize)
// Make a block of memory
buf := make([]byte, reqSize)
// Read the chunk
_, err = io.ReadFull(in, buf)
if err != nil {
_ = rw.Close()
err = fmt.Errorf("multipart upload failed to read source: %w", err)
break outer
}
// Make the global hash (must be done sequentially)
_, _ = hash.Write(buf)
// Transfer the chunk
wg.Add(1)
o.fs.uploadToken.Get()
go func(part int, position int64) {
defer wg.Done()
defer o.fs.uploadToken.Put()
defer func() { _ = rw.Close() }()
fs.Debugf(o, "Uploading part %d/%d offset %v/%v part size %v", part+1, session.TotalParts, fs.SizeSuffix(position), fs.SizeSuffix(size), fs.SizeSuffix(chunkSize))
partResponse, err := o.uploadPart(ctx, session.ID, position, size, rw, wrap, options...)
partResponse, err := o.uploadPart(ctx, session.ID, position, size, buf, wrap, options...)
if err != nil {
err = fmt.Errorf("multipart upload failed to upload part: %w", err)
select {
+5 -26
View File
@@ -1502,21 +1502,11 @@ func (c *chunkingReader) rollback(ctx context.Context, metaObject fs.Object) {
func (f *Fs) removeOldChunks(ctx context.Context, remote string) {
oldFsObject, err := f.NewObject(ctx, remote)
if err == nil {
removeStaleChunks(ctx, oldFsObject.(*Object), nil)
}
}
func removeStaleChunks(ctx context.Context, oldObject *Object, currentChunks []fs.Object) {
currentRemotes := make(map[string]struct{}, len(currentChunks))
for _, chunk := range currentChunks {
currentRemotes[chunk.Remote()] = struct{}{}
}
for _, chunk := range oldObject.chunks {
if _, found := currentRemotes[chunk.Remote()]; found {
continue
}
if err := chunk.Remove(ctx); err != nil {
fs.Errorf(chunk, "Failed to remove old chunk: %v", err)
oldObject := oldFsObject.(*Object)
for _, chunk := range oldObject.chunks {
if err := chunk.Remove(ctx); err != nil {
fs.Errorf(chunk, "Failed to remove old chunk: %v", err)
}
}
}
}
@@ -1697,20 +1687,12 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
// metadata format which might involve unsupported chunk types.
return nil, fmt.Errorf("can't %s this file: %w", opName, err)
}
// Snapshot destination chunks before its metadata is replaced.
var oldObject *Object
if oldFsObject, err := f.NewObject(ctx, remote); err == nil {
oldObject = oldFsObject.(*Object)
}
if !o.isComposite() {
fs.Debugf(o, "%s non-chunked object...", opName)
oResult, err := do(ctx, o.mainChunk(), remote) // chain operation to a single wrapped chunk
if err != nil {
return nil, err
}
if oldObject != nil {
removeStaleChunks(ctx, oldObject, nil)
}
return f.newObject("", oResult, nil), nil
}
@@ -1776,9 +1758,6 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
silentlyRemove(ctx, newObj)
return nil, err
}
if oldObject != nil {
removeStaleChunks(ctx, oldObject, newChunks)
}
return newObj, nil
}
+1 -103
View File
@@ -11,7 +11,6 @@ import (
"strings"
"testing"
_ "github.com/rclone/rclone/backend/memory"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/fspath"
@@ -44,7 +43,7 @@ func testPutLarge(t *testing.T, f *Fs, kilobytes int) {
type settings map[string]any
func deriveFs(ctx context.Context, t *testing.T, f fs.Fs, path string, opts settings) fs.Fs {
fsName, _, _ := strings.Cut(f.Name(), "{") // strip off hash
fsName := strings.Split(f.Name(), "{")[0] // strip off hash
configMap := configmap.Simple{}
for key, val := range opts {
configMap[key] = fmt.Sprintf("%v", val)
@@ -878,104 +877,6 @@ func testChunkerServerSideMove(t *testing.T, f *Fs) {
_ = operations.Purge(ctx, f.base, dir)
}
func testChunkerServerSideOverwrite(t *testing.T, moveFs *Fs) {
ctx := context.Background()
const dir = "serveroverwrite"
remote := fmt.Sprintf(":chunker,remote=':memory:%s',chunk_size=3B:", dir)
fsResult, err := fs.NewFs(ctx, remote)
require.NoError(t, err)
copyFs, ok := fsResult.(*Fs)
require.True(t, ok)
oldMoveChunkSize := moveFs.opt.ChunkSize
moveFs.opt.ChunkSize = fs.SizeSuffix(3)
defer func() {
moveFs.opt.ChunkSize = oldMoveChunkSize
_ = operations.Purge(ctx, copyFs.base, "")
_ = operations.Purge(ctx, moveFs.base, dir)
}()
type operation struct {
name string
fs *Fs
enabled bool
do copyMoveFn
}
operations := []operation{
{name: "Copy", fs: copyFs, enabled: copyFs.Features().Copy != nil, do: copyFs.Copy},
{name: "Move", fs: moveFs, enabled: moveFs.Features().Move != nil, do: moveFs.Move},
}
testCases := []struct {
name string
sourceContents string
targetContents string
}{
{name: "SmallOverComposite", sourceContents: "hi", targetContents: "abcdef"},
{name: "FewerChunks", sourceContents: "abcd", targetContents: "abcdefghi"},
}
transactionModes := []struct {
name string
useNoRename bool
}{
{name: "Rename"},
{name: "NoRename", useNoRename: true},
}
for _, operation := range operations {
t.Run(operation.name, func(t *testing.T) {
if !operation.enabled {
t.Skipf("wrapped remote does not support server-side %s", strings.ToLower(operation.name))
}
for _, transactionMode := range transactionModes {
t.Run(transactionMode.name, func(t *testing.T) {
f := operation.fs
if transactionMode.useNoRename && !f.useMeta {
t.Skip("Can't test norename transactions without metadata")
}
oldUseNoRename := f.useNoRename
f.useNoRename = transactionMode.useNoRename
defer func() { f.useNoRename = oldUseNoRename }()
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
prefix := path.Join(dir, strings.ToLower(operation.name), strings.ToLower(transactionMode.name), strings.ToLower(testCase.name))
source := testPutFile(ctx, t, f, prefix+"-source", testCase.sourceContents, "source", true)
target := testPutFile(ctx, t, f, prefix+"-target", testCase.targetContents, "target", true)
targetObject := target.(*Object)
require.NotEmpty(t, targetObject.chunks)
oldChunks := append([]fs.Object(nil), targetObject.chunks...)
result, err := operation.do(ctx, source, target.Remote())
require.NoError(t, err)
reader, err := result.Open(ctx)
require.NoError(t, err)
contents, err := io.ReadAll(reader)
require.NoError(t, err)
require.NoError(t, reader.Close())
assert.Equal(t, testCase.sourceContents, string(contents))
resultObject := result.(*Object)
currentChunks := make(map[string]struct{}, len(resultObject.chunks))
for _, chunk := range resultObject.chunks {
currentChunks[chunk.Remote()] = struct{}{}
}
removed := 0
for _, chunk := range oldChunks {
if _, found := currentChunks[chunk.Remote()]; found {
continue
}
_, err := f.base.NewObject(ctx, chunk.Remote())
assert.Equal(t, fs.ErrorObjectNotFound, err)
removed++
}
assert.Positive(t, removed)
})
}
})
}
})
}
}
// Test that md5all creates metadata even for small files
func testMD5AllSlow(t *testing.T, f *Fs) {
ctx := context.Background()
@@ -1049,9 +950,6 @@ func (f *Fs) InternalTest(t *testing.T) {
t.Run("ChunkerServerSideMove", func(t *testing.T) {
testChunkerServerSideMove(t, f)
})
t.Run("ChunkerServerSideOverwrite", func(t *testing.T) {
testChunkerServerSideOverwrite(t, f)
})
t.Run("MD5AllSlow", func(t *testing.T) {
testMD5AllSlow(t, f)
})
+2 -2
View File
@@ -472,8 +472,8 @@ func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options .
if updateOptions, ok := option.(*api.UpdateOptions); ok {
if updateOptions.PublicID != "" {
updateObject = true
params.Overwrite = new(true)
params.Invalidate = new(true)
params.Overwrite = SDKApi.Bool(true)
params.Invalidate = SDKApi.Bool(true)
params.PublicID = updateOptions.PublicID
params.ResourceType = updateOptions.ResourceType
params.Type = SDKApi.DeliveryType(updateOptions.DeliveryType)
+5 -19
View File
@@ -31,7 +31,6 @@ import (
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/lib/multipart"
)
// Globals
@@ -564,19 +563,15 @@ type compressionResult[T sgzip.GzipMetadata | SzstdMetadata] struct {
func (f *Fs) rcat(ctx context.Context, dstFileName string, in io.ReadCloser, modTime time.Time, options []fs.OpenOption) (o fs.Object, err error) {
// cache small files in memory and do normal upload
rw := multipart.NewRW()
defer fs.CheckClose(rw, &err)
_, err = io.CopyN(rw, in, int64(f.opt.RAMCacheLimit))
if err == io.EOF {
src := object.NewStaticObjectInfo(dstFileName, modTime, rw.Size(), false, nil, f.Fs)
return f.Fs.Put(ctx, rw, src, options...)
} else if err != nil {
return nil, err
buf := make([]byte, f.opt.RAMCacheLimit)
if n, err := io.ReadFull(in, buf); err == io.EOF || err == io.ErrUnexpectedEOF {
src := object.NewStaticObjectInfo(dstFileName, modTime, int64(len(buf[:n])), false, nil, f.Fs)
return f.Fs.Put(ctx, bytes.NewBuffer(buf[:n]), src, options...)
}
// Need to include what we already read
in = &ReadCloserWrapper{
Reader: io.MultiReader(rw, in),
Reader: io.MultiReader(bytes.NewReader(buf), in),
Closer: in,
}
@@ -1157,15 +1152,6 @@ func (o *Object) Remove(ctx context.Context) error {
return objErr
}
// countingDiscard is an io.Writer which discards its input and counts
// the bytes written
type countingDiscard int64
func (c *countingDiscard) Write(p []byte) (int, error) {
*c += countingDiscard(len(p))
return len(p), nil
}
// ReadCloserWrapper combines a Reader and a Closer to a ReadCloser
type ReadCloserWrapper struct {
io.Reader
+3 -2
View File
@@ -2,6 +2,7 @@ package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
@@ -22,7 +23,7 @@ type gzipModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (g *gzipModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
var b countingDiscard
var b bytes.Buffer
var n int64
w, err := sgzip.NewWriterLevel(&b, sgzip.DefaultCompression)
if err != nil {
@@ -36,7 +37,7 @@ func (g *gzipModeHandler) isCompressible(r io.Reader, compressionMode int) (bool
if err != nil {
return false, err
}
ratio := float64(n) / float64(b)
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
-137
View File
@@ -1,137 +0,0 @@
package compress
import (
"bytes"
"context"
"crypto/rand"
"io"
"testing"
"time"
_ "github.com/rclone/rclone/backend/memory"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// putRecorder wraps an fs.Fs and records what rcat hands to Put and
// PutStream. It reads each body twice when it is seekable to model a
// backend retrying an upload.
type putRecorder struct {
fs.Fs
t *testing.T
stream bool // expose PutStream from the wrapped Fs
size int64
seekable bool
body []byte
streamed bool
}
func (r *putRecorder) Features() *fs.Features {
features := *r.Fs.Features()
features.PutStream = nil
if r.stream {
features.PutStream = r.PutStream
}
return &features
}
func (r *putRecorder) record(in io.Reader, src fs.ObjectInfo) []byte {
r.size = src.Size()
body, err := io.ReadAll(in)
require.NoError(r.t, err)
if seeker, ok := in.(io.Seeker); ok {
r.seekable = true
_, err = seeker.Seek(0, io.SeekStart)
require.NoError(r.t, err)
again, err := io.ReadAll(in)
require.NoError(r.t, err)
assert.Equal(r.t, body, again, "retried body should be identical")
}
r.body = body
return body
}
func (r *putRecorder) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
body := r.record(in, src)
return r.Fs.Put(ctx, bytes.NewReader(body), object.NewStaticObjectInfo(src.Remote(), src.ModTime(ctx), int64(len(body)), true, nil, r.Fs), options...)
}
func (r *putRecorder) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
r.streamed = true
return r.Put(ctx, in, src, options...)
}
// TestRcat checks the three upload paths rcat takes: small files buffered
// in memory and uploaded with a known size and a seekable body, large
// files streamed when the wrapped remote supports it, and large files
// spooled to a temporary file when it doesn't. Pool pages taken for the
// buffer must be returned on every path.
func TestRcat(t *testing.T) {
ctx := context.Background()
const limit = 1024
for _, test := range []struct {
name string
size int
stream bool
wantSize int64
}{
{name: "Small", size: limit / 2, stream: true, wantSize: limit / 2},
{name: "Empty", size: 0, stream: true, wantSize: 0},
{name: "ExactLimit", size: limit, stream: true, wantSize: -1},
{name: "LargeStream", size: 3 * limit, stream: true, wantSize: -1},
{name: "LargeSpool", size: 3 * limit, stream: false, wantSize: 3 * limit},
} {
t.Run(test.name, func(t *testing.T) {
remote, err := fs.NewFs(ctx, ":memory:rcat-"+test.name)
require.NoError(t, err)
rec := &putRecorder{Fs: remote, t: t, stream: test.stream}
f := &Fs{Fs: rec}
f.opt.RAMCacheLimit = limit
content := []byte(random.String(test.size))
inUse := pool.Global().InUse()
o, err := f.rcat(ctx, "file", io.NopCloser(bytes.NewReader(content)), time.Now(), nil)
require.NoError(t, err)
assert.Equal(t, inUse, pool.Global().InUse(), "pool pages not returned")
assert.Equal(t, test.wantSize, rec.size)
assert.Equal(t, content, rec.body)
assert.Equal(t, test.stream && test.size >= limit, rec.streamed)
if test.size < limit {
assert.True(t, rec.seekable, "small file body should be seekable")
}
rc, err := o.Open(ctx)
require.NoError(t, err)
got, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, content, got)
})
}
}
// TestIsCompressible checks the compressibility heuristic on data at
// both ends of the scale for every compression mode which has one.
func TestIsCompressible(t *testing.T) {
compressible := bytes.Repeat([]byte("compress me "), 4096)
incompressible := make([]byte, len(compressible))
_, err := io.ReadFull(rand.Reader, incompressible)
require.NoError(t, err)
for name, handler := range map[string]compressionModeHandler{
"gzip": &gzipModeHandler{},
"zstd": &zstdModeHandler{},
} {
t.Run(name, func(t *testing.T) {
ok, err := handler.isCompressible(bytes.NewReader(compressible), 0)
require.NoError(t, err)
assert.True(t, ok)
ok, err = handler.isCompressible(bytes.NewReader(incompressible), 0)
require.NoError(t, err)
assert.False(t, ok)
})
}
}
+3 -2
View File
@@ -2,6 +2,7 @@ package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
@@ -22,7 +23,7 @@ type zstdModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (z *zstdModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
var b countingDiscard
var b bytes.Buffer
var n int64
w, err := NewWriterSzstd(&b, zstd.WithEncoderLevel(zstd.SpeedDefault))
if err != nil {
@@ -36,7 +37,7 @@ func (z *zstdModeHandler) isCompressible(r io.Reader, compressionMode int) (bool
if err != nil {
return false, err
}
ratio := float64(n) / float64(b)
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
+1 -1
View File
@@ -1197,7 +1197,7 @@ func (o *ObjectInfo) Hash(ctx context.Context, hash hash.Type) (string, error) {
return "", nil
}
// if this is wrapping a local object then we work out the hash
if srcObj != nil && srcObj.Fs().Features().IsLocal {
if srcObj.Fs().Features().IsLocal {
// Read the data and encrypt it to calculate the hash
fs.Debugf(o, "Computing %v hash of encrypted source", hash)
return o.f.computeHashWithNonce(ctx, o.nonce, srcObj, hash)
-10
View File
@@ -93,16 +93,6 @@ func testObjectInfo(t *testing.T, f *Fs, wrap bool) {
assert.Equal(t, fmt.Sprintf("%x", wantHash), gotHash)
}
// Test ObjectInfo.Hash doesn't panic when the source is an
// fs.OverrideRemote which wraps an fs.ObjectInfo that isn't an fs.Object
func TestObjectInfoHashOverrideRemoteNotObject(t *testing.T) {
src := object.NewStaticObjectInfo("file", time.Now(), 3, true, nil, nil)
oi := (&Fs{}).newObjectInfo(fs.NewOverrideRemote(src, "new_remote"), nonce{})
gotHash, err := oi.Hash(context.Background(), hash.MD5)
require.NoError(t, err)
assert.Equal(t, "", gotHash)
}
func testComputeHash(t *testing.T, f *Fs) {
var (
contents = random.String(100)
-259
View File
@@ -1,259 +0,0 @@
// Package api has type definitions for dosya.dev
package api
import "fmt"
// Response is the base response wrapper
type Response struct {
OK bool `json:"ok"`
Error string `json:"error,omitempty"`
}
// Error is an error response from the API.
//
// StatusCode and Status are filled in from the HTTP response. The API
// sets ErrorCode on the errors a client is expected to act on rather
// than just report, e.g. "concurrent_upload_limit".
type Error struct {
StatusCode int `json:"-"`
Status string `json:"-"`
OK bool `json:"ok"`
Message string `json:"error"`
ErrorCode string `json:"error_code,omitempty"`
MaxConcurrentUploads int `json:"max_concurrent_uploads,omitempty"`
}
// Error satisfies the error interface
func (e *Error) Error() string {
out := fmt.Sprintf("HTTP error %d (%s)", e.StatusCode, e.Status)
if e.Message != "" {
out += ": " + e.Message
}
if e.ErrorCode != "" {
out += " (" + e.ErrorCode + ")"
}
return out
}
// FileItem represents a file from the dosya API
type FileItem struct {
ID string `json:"id"`
Name string `json:"name"`
SizeBytes int64 `json:"size_bytes"`
MimeType string `json:"mime_type"`
Extension string `json:"extension"`
Region string `json:"region"`
CreatedAt float64 `json:"created_at"`
UpdatedAt float64 `json:"updated_at"`
FolderID *string `json:"folder_id"`
IsSynced int `json:"is_synced"`
// ContentHash is the lowercase-hex SHA-256 of the file's bytes, or "" when
// the server has none (e.g. a file uploaded as an R2 multipart, whose parts
// are never re-read to hash the whole object - the same limitation S3 has).
ContentHash string `json:"content_hash"`
}
// FolderItem represents a folder from the dosya API
type FolderItem struct {
ID string `json:"id"`
Name string `json:"name"`
ParentID *string `json:"parent_id"`
CreatedAt float64 `json:"created_at"`
FileCount int `json:"file_count"`
IsSynced int `json:"is_synced"`
}
// ListResponse is the response from GET /api/files
type ListResponse struct {
Response
Files []FileItem `json:"files"`
Folders []FolderItem `json:"folders"`
Pagination Pagination `json:"pagination"`
}
// Pagination describes which page of files a ListResponse holds
type Pagination struct {
Page int `json:"page"`
PerPage int `json:"per_page"`
TotalFiles int `json:"total_files"`
TotalPages int `json:"total_pages"`
}
// CreateFolderRequest is the request body for POST /api/folders
type CreateFolderRequest struct {
WorkspaceID string `json:"workspace_id"`
ParentID *string `json:"parent_id"`
Name string `json:"name"`
}
// CreateFolderResponse is the response from POST /api/folders
type CreateFolderResponse struct {
Response
Folder struct {
ID string `json:"id"`
Name string `json:"name"`
ParentID *string `json:"parent_id"`
Workspace string `json:"workspace_id"`
} `json:"folder"`
}
// DeleteFolderResponse is the response from DELETE /api/folders/:id
type DeleteFolderResponse struct {
Response
Permanent bool `json:"permanent"`
Complete bool `json:"complete"`
Remaining int `json:"remaining"`
FilesAffected int `json:"files_affected"`
FoldersRemoved int `json:"folders_removed"`
}
// RenameFolderRequest is the request body for PUT /api/folders/:id/rename
type RenameFolderRequest struct {
Name string `json:"name"`
}
// MoveFolderRequest is the request body for PUT /api/folders/:id/move
type MoveFolderRequest struct {
ParentID *string `json:"parent_id"`
Name string `json:"name,omitempty"`
}
// MoveFolderResponse is the response from PUT /api/folders/:id/move
type MoveFolderResponse struct {
Response
Name string `json:"name"`
}
// UploadInitRequest is the request body for POST /api/upload/init
type UploadInitRequest struct {
WorkspaceID string `json:"workspace_id"`
FileName string `json:"file_name"`
FileSize int64 `json:"file_size"`
MimeType string `json:"mime_type"`
Region string `json:"region,omitempty"`
FolderID *string `json:"folder_id"`
FileID *string `json:"file_id,omitempty"`
}
// UploadInitResponse is the response from POST /api/upload/init
type UploadInitResponse struct {
Response
SessionID string `json:"session_id"`
UploadURL string `json:"upload_url"`
WorkspaceID string `json:"workspace_id"`
FileName string `json:"file_name"`
FileSize int64 `json:"file_size"`
MimeType string `json:"mime_type"`
Extension string `json:"extension"`
Region string `json:"region"`
Resumable *struct {
PartSize int64 `json:"part_size"`
TotalParts int `json:"total_parts"`
PartUploadURL string `json:"part_upload_url"`
CompleteURL string `json:"complete_url"`
StatusURL string `json:"status_url"`
} `json:"resumable"`
}
// UploadCompleteResponse is the response from PUT /api/upload/:sessionId or POST /api/upload/:sessionId/complete
type UploadCompleteResponse struct {
Response
File struct {
ID string `json:"id"`
Name string `json:"name"`
SizeBytes int64 `json:"size_bytes"`
MimeType string `json:"mime_type"`
Extension string `json:"extension"`
Region string `json:"region"`
Version int `json:"version"`
CreatedAt float64 `json:"created_at"`
// SHA-256 of the uploaded bytes for a single-shot (small-file) upload;
// "" for a multipart upload, which the server does not hash.
ContentHash string `json:"content_hash"`
} `json:"file"`
}
// UploadPartResponse is the response from PUT /api/upload/:sessionId/part/:partNumber
type UploadPartResponse struct {
Response
PartNumber int `json:"part_number"`
ETag string `json:"etag"`
BytesUploaded int64 `json:"bytes_uploaded"`
PartsCompleted int `json:"parts_completed"`
TotalParts int `json:"total_parts"`
}
// DeleteFileResponse is the response from DELETE /api/files/:id
type DeleteFileResponse struct {
Response
Permanent bool `json:"permanent"`
}
// RenameFileRequest is the request body for PUT /api/files/:id/rename
type RenameFileRequest struct {
Name string `json:"name"`
}
// RenameFileResponse is the response from PUT /api/files/:id/rename
type RenameFileResponse struct {
Response
Name string `json:"name"`
}
// MoveFileRequest is the request body for PUT /api/files/:id/move
type MoveFileRequest struct {
FolderID *string `json:"folder_id"`
Name *string `json:"name,omitempty"`
}
// CopyFileRequest is the request body for POST /api/files/:id/copy
type CopyFileRequest struct {
FolderID *string `json:"folder_id"`
Name string `json:"name,omitempty"`
}
// CopyFileResponse is the response from POST /api/files/:id/copy
type CopyFileResponse struct {
Response
FileID string `json:"file_id"`
Name string `json:"name"`
}
// ShareLinkRequest is the request body for POST /api/files/:id/share
type ShareLinkRequest struct {
ExpiresInDays *int `json:"expires_in_days,omitempty"`
}
// ShareLinkResponse is the response from POST /api/files/:id/share
type ShareLinkResponse struct {
Response
Link struct {
ID string `json:"id"`
Token string `json:"token"`
URL string `json:"url"`
ExpiresAt *int64 `json:"expires_at"`
CreatedAt float64 `json:"created_at"`
} `json:"link"`
}
// Workspace is one entry of the workspace list
type Workspace struct {
ID string `json:"id"`
Name string `json:"name"`
}
// WorkspaceListResponse is the response from GET /api/workspaces
type WorkspaceListResponse struct {
Response
Workspaces []Workspace `json:"workspaces"`
}
// WorkspaceInfoResponse is the response from GET /api/workspaces/:id
type WorkspaceInfoResponse struct {
Response
Storage struct {
Used int64 `json:"used"`
Total int64 `json:"total"`
Free int64 `json:"free"`
} `json:"storage"`
}
-772
View File
@@ -1,772 +0,0 @@
// Package dosya provides an interface to the dosya.dev cloud storage system.
package dosya
import (
"context"
"errors"
"fmt"
"io"
"strings"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/random"
"github.com/rclone/rclone/lib/rest"
)
const (
rootID = "root"
defaultBaseURL = "https://api.dosya.dev"
minSleep = 100 * time.Millisecond
maxSleep = 2 * time.Second
decayConstant = 2
attackConstant = 0
)
func init() {
fs.Register(&fs.RegInfo{
Name: "dosya",
Description: "dosya.dev",
NewFs: NewFs,
Options: []fs.Option{{
Help: "Your API Key, get it from https://dosya.dev/profile under \"API keys\".",
Name: "api_key",
Sensitive: true,
Required: true,
}, {
Name: "workspace_id",
Help: `Your workspace ID.
Leave this blank to use the one workspace the API key can reach, which
is what a key restricted to a single workspace does. If the key can
reach several, rclone will refuse and list their IDs for you to choose
from.`,
}, {
Help: "API base URL.",
Name: "api_url",
Default: defaultBaseURL,
Advanced: true,
}, {
Name: "upload_concurrency",
Help: `Concurrency for multipart uploads.
This is the number of parts of the same file that are uploaded
concurrently. The first part is always uploaded on its own, because
the server creates the upload when it receives it.
Each part in flight is held in memory, so memory use grows with
this value times the part size the server chooses.
If you are uploading large files over high-speed links and these
uploads do not fully utilize your bandwidth, then increasing this
may help to speed up the transfers.`,
Default: 4,
Advanced: true,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
Advanced: true,
Default: (encoder.Display |
encoder.EncodeBackSlash |
encoder.EncodeSingleQuote |
encoder.EncodeDoubleQuote |
encoder.EncodeLtGt |
encoder.EncodeCtl |
encoder.EncodeLeftSpace |
encoder.EncodeRightSpace |
encoder.EncodeInvalidUtf8),
}},
})
}
// Options defines the configuration for this backend
type Options struct {
APIKey string `config:"api_key"`
WorkspaceID string `config:"workspace_id"`
APIURL string `config:"api_url"`
UploadConcurrency int `config:"upload_concurrency"`
Enc encoder.MultiEncoder `config:"encoding"`
}
// Fs represents a remote dosya.dev filesystem
type Fs struct {
name string
root string
opt Options
features *fs.Features
rest *rest.Client
pacer *fs.Pacer
dirCache *dircache.DirCache
}
// Object represents a dosya.dev file
type Object struct {
fs *Fs
remote string
file api.FileItem
}
// findWorkspace fills in the workspace ID when the config leaves it blank.
//
// The ID is not shown anywhere in the dosya.dev web interface, so requiring
// it meant asking people to find it in a developer console. A key restricted
// to one workspace can only mean that one, and an account with a single
// workspace has only one possible answer; anything else genuinely has to be
// chosen, so list the choices in the error.
func (f *Fs) findWorkspace(ctx context.Context) error {
workspaces, err := f.listWorkspaces(ctx)
if err != nil {
return err
}
switch len(workspaces) {
case 0:
return errors.New("no workspaces found for this API key")
case 1:
f.opt.WorkspaceID = workspaces[0].ID
fs.Debugf(f, "Using workspace %q (%s)", workspaces[0].Name, workspaces[0].ID)
return nil
}
var choices strings.Builder
for _, w := range workspaces {
_, _ = fmt.Fprintf(&choices, "\n %s (%s)", w.ID, w.Name)
}
return fmt.Errorf("this API key can reach %d workspaces, so workspace_id must be set to one of:%s",
len(workspaces), choices.String())
}
// FindLeaf finds a directory of name leaf in the folder with ID pathID
func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
listing, err := f.listFilesAndFolders(ctx, pathID)
if err != nil {
return "", false, err
}
for _, folder := range listing.Folders {
if f.opt.Enc.ToStandardName(folder.Name) == leaf {
return folder.ID, true, nil
}
}
return "", false, nil
}
// CreateDir makes a directory with pathID as parent and name leaf
func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string, err error) {
name := f.opt.Enc.FromStandardName(leaf)
resp, err := f.createFolder(ctx, name, pathID)
if err != nil {
return "", err
}
return resp.Folder.ID, nil
}
// Name of the remote (as passed into NewFs)
func (f *Fs) Name() string {
return f.name
}
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
return f.root
}
// String returns a description of the FS
func (f *Fs) String() string {
return fmt.Sprintf("dosya.dev root '%s'", f.root)
}
// Precision of the ModTimes in this Fs
func (f *Fs) Precision() time.Duration {
// Modification times are stored and returned to the second (the API
// keeps them as Unix seconds, sent on upload via the source-mtime header).
return time.Second
}
// Hashes returns the supported hash types of the filesystem
func (f *Fs) Hashes() hash.Set {
// The server computes and stores the SHA-256 of a single-shot upload's
// bytes; multipart uploads have none (see Object.Hash).
return hash.NewHashSet(hash.SHA256)
}
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
// NewFs makes a new Fs object from the path
func NewFs(ctx context.Context, name string, root string, config configmap.Mapper) (fs.Fs, error) {
opt := new(Options)
err := configstruct.Set(config, opt)
if err != nil {
return nil, err
}
root = strings.Trim(root, "/")
f := &Fs{
name: name,
root: root,
opt: *opt,
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant), pacer.AttackConstant(attackConstant))),
}
// No DuplicateFiles: an upload that names no existing file but whose
// (workspace, folder, name) is already taken becomes a new VERSION of
// that file rather than a second row, so a name is unique within a
// folder and the same upload twice returns the same file id.
f.features = (&fs.Features{
CaseInsensitive: false,
CanHaveEmptyDirectories: true,
ReadMimeType: true,
}).Fill(ctx, f)
client := fshttp.NewClient(ctx)
f.rest = rest.NewClient(client).SetRoot(strings.TrimSuffix(opt.APIURL, "/")).SetErrorHandler(errorHandler)
f.rest.SetHeader("Authorization", "Bearer "+f.opt.APIKey)
if f.opt.WorkspaceID == "" {
err = f.findWorkspace(ctx)
if err != nil {
return nil, err
}
}
f.dirCache = dircache.New(root, rootID, f)
// Find the current root
err = f.dirCache.FindRoot(ctx, false)
if err != nil {
// Assume it is a file
newRoot, remote := dircache.SplitPath(root)
tempF := *f
tempF.dirCache = dircache.New(newRoot, rootID, &tempF)
tempF.root = newRoot
// Make new Fs which is the parent
err = tempF.dirCache.FindRoot(ctx, false)
if err != nil {
// No root so return old f
return f, nil
}
_, err := tempF.NewObject(ctx, remote)
if err != nil {
if err == fs.ErrorObjectNotFound {
// File doesn't exist so return old f
return f, nil
}
return nil, err
}
f.features.Fill(ctx, &tempF)
f.dirCache = tempF.dirCache
f.root = tempF.root
return f, fs.ErrorIsFile
}
return f, nil
}
// List the objects and directories in dir into entries
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return nil, err
}
listing, err := f.listFilesAndFolders(ctx, directoryID)
if err != nil {
return nil, err
}
for _, folder := range listing.Folders {
folderName := f.opt.Enc.ToStandardName(folder.Name)
fullPath := getRemote(dir, folderName)
modTime := time.Unix(int64(folder.CreatedAt), 0)
d := fs.NewDir(fullPath, modTime).SetID(folder.ID)
entries = append(entries, d)
f.dirCache.Put(fullPath, folder.ID)
}
for _, file := range listing.Files {
fileName := f.opt.Enc.ToStandardName(file.Name)
remote := getRemote(dir, fileName)
o := &Object{
fs: f,
remote: remote,
file: file,
}
entries = append(entries, o)
}
return entries, nil
}
// NewObject finds the Object at remote
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
leaf, directoryID, err := f.dirCache.FindPath(ctx, remote, false)
if err != nil {
if err == fs.ErrorDirNotFound {
return nil, fs.ErrorObjectNotFound
}
return nil, err
}
listing, err := f.listFilesAndFolders(ctx, directoryID)
if err != nil {
return nil, err
}
for _, file := range listing.Files {
if f.opt.Enc.ToStandardName(file.Name) == leaf {
return &Object{
fs: f,
remote: remote,
file: file,
}, nil
}
}
return nil, fs.ErrorObjectNotFound
}
// Put uploads a new file
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
existingObj, err := f.NewObject(ctx, src.Remote())
switch err {
case nil:
return existingObj, existingObj.Update(ctx, in, src, options...)
case fs.ErrorObjectNotFound:
return f.put(ctx, in, src, options...)
default:
return nil, err
}
}
// put uploads src as a new file
func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
remote := src.Remote()
_, directoryID, err := f.dirCache.FindPath(ctx, remote, true)
if err != nil {
return nil, err
}
modTime := src.ModTime(ctx)
resp, err := f.uploadFile(ctx, in, remote, src.Size(), modTime, fs.MimeType(ctx, src), directoryID, nil)
if err != nil {
return nil, err
}
return &Object{
fs: f,
remote: remote,
file: fileFromUpload(resp, modTime),
}, nil
}
// fileFromUpload builds the FileItem for the object an upload just created.
// The single-shot upload response omits updated_at, so the modification time
// is taken from what was sent (modTime) rather than defaulting to the create
// time - it is the time the server stored via the source-mtime header. The
// SHA-256 comes straight from the response (empty for a multipart upload).
func fileFromUpload(resp *api.UploadCompleteResponse, modTime time.Time) api.FileItem {
updatedAt := resp.File.CreatedAt
if !modTime.IsZero() {
updatedAt = float64(modTime.Unix())
}
return api.FileItem{
ID: resp.File.ID,
Name: resp.File.Name,
SizeBytes: resp.File.SizeBytes,
MimeType: resp.File.MimeType,
Extension: resp.File.Extension,
Region: resp.File.Region,
CreatedAt: resp.File.CreatedAt,
UpdatedAt: updatedAt,
ContentHash: resp.File.ContentHash,
}
}
// Mkdir makes the directory (container, bucket)
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
_, err := f.dirCache.FindDir(ctx, dir, true)
return err
}
// Rmdir removes the directory (container, bucket) if empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return err
}
// The server checks emptiness itself (?if_empty=1) and answers
// fs.ErrorDirectoryNotEmpty's 409 without a separate listing call.
err = f.removeFolder(ctx, directoryID, true)
if err != nil {
return err
}
f.dirCache.FlushDir(dir)
return nil
}
// Move src to this remote using server-side move operations
func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
srcObj, ok := src.(*Object)
if !ok {
fs.Debugf(src, "Can't move - not same remote type")
return nil, fs.ErrorCantMove
}
srcLeaf := f.opt.Enc.ToStandardName(srcObj.file.Name)
_, srcDirID, err := srcObj.fs.dirCache.FindPath(ctx, srcObj.remote, false)
if err != nil {
return nil, err
}
// Find destination directory
dstLeaf, dstDirID, err := f.dirCache.FindPath(ctx, remote, true)
if err != nil {
return nil, err
}
needsMove := srcDirID != dstDirID
needsRename := srcLeaf != dstLeaf
// Moving and renaming in the same request (rather than a moveFile call
// followed by a separate renameFile call) matters when both are needed:
// a two-step move-then-rename can transiently place the file's CURRENT
// name into a destination folder that already holds an unrelated file
// with that name, even though the FINAL name is free - the server only
// allows one active name per folder, so that intermediate state 409s.
if needsMove || needsRename {
newName := ""
if needsRename {
newName = f.opt.Enc.FromStandardName(dstLeaf)
}
err = f.moveFile(ctx, srcObj.file.ID, dstDirID, newName)
if err != nil {
return nil, err
}
}
// Return new object. file.Name must reflect the destination leaf, not
// the source's - a caller that reuses this object as the src of a
// SECOND Move (e.g. operations.RemoveExisting's restore-on-error path)
// recomputes srcLeaf from file.Name; a stale name there makes that
// second move see srcLeaf == dstLeaf and skip the rename entirely,
// silently leaving the object under the wrong name.
newFile := srcObj.file
newFile.Name = f.opt.Enc.FromStandardName(dstLeaf)
return &Object{
fs: f,
remote: remote,
file: newFile,
}, nil
}
// DirMove moves src, srcRemote to this remote at dstRemote
func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error {
srcFs, ok := src.(*Fs)
if !ok {
fs.Debugf(srcFs, "Can't move directory - not same remote type")
return fs.ErrorCantDirMove
}
srcID, srcDirectoryID, _, dstDirectoryID, dstLeaf, err := f.dirCache.DirMove(ctx, srcFs.dirCache, srcFs.root, srcRemote, f.root, dstRemote)
if err != nil {
return err
}
srcLeaf := srcRemote
if idx := strings.LastIndex(srcRemote, "/"); idx >= 0 {
srcLeaf = srcRemote[idx+1:]
}
needsRename := srcLeaf != dstLeaf
newName := ""
if needsRename {
newName = f.opt.Enc.FromStandardName(dstLeaf)
}
// Move folder to new parent only if parent changed. The new name goes
// in the same request, as for files in Move: a move followed by a
// separate rename can collide with an unrelated folder sharing the
// CURRENT name in the destination even though the final name is free.
if srcDirectoryID != dstDirectoryID {
renamed, err := f.moveFolder(ctx, srcID, dstDirectoryID, newName)
if err != nil {
return fmt.Errorf("couldn't move directory: %w", err)
}
if renamed {
needsRename = false
}
}
if needsRename {
err = f.renameFolder(ctx, srcID, newName)
if err != nil {
return fmt.Errorf("couldn't rename directory: %w", err)
}
}
srcFs.dirCache.FlushDir(srcRemote)
return nil
}
// Copy src to this remote using server-side copy operations
func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
srcObj, ok := src.(*Object)
if !ok {
fs.Debugf(src, "Can't copy - not same remote type")
return nil, fs.ErrorCantCopy
}
dstLeaf, dstDirID, err := f.dirCache.FindPath(ctx, remote, true)
if err != nil {
return nil, err
}
// The copy API creates a new row named after the source file when
// copying across folders, which collides with one already at the
// destination. Copy under a temporary name, then delete the old
// destination, then move the copy onto the name - so a copy that fails
// leaves the destination as it was. Deleting it up front lost it
// (prior art: backend/box).
existingObj, err := f.NewObject(ctx, remote)
switch err {
case nil:
tempRemote := remote + "-rclone-copy-" + random.String(8)
fs.Debugf(remote, "dst already exists, copying to temp name %v", tempRemote)
tempObj, err := f.Copy(ctx, src, tempRemote)
if err != nil {
return nil, err
}
fs.Debugf(tempRemote, "moving to real name %v", remote)
if err := existingObj.Remove(ctx); err != nil {
return nil, fmt.Errorf("couldn't remove existing destination file: %w", err)
}
return f.Move(ctx, tempObj, remote)
case fs.ErrorObjectNotFound:
// nothing at the destination, so copy straight onto the name
default:
return nil, err
}
dstName := f.opt.Enc.FromStandardName(dstLeaf)
resp, err := f.copyFile(ctx, srcObj.file.ID, dstDirID, dstName)
if err != nil {
return nil, fmt.Errorf("couldn't copy file: %w", err)
}
// Servers that predate the name field ignore it and name the copy after
// the source, or "Copy of <original>" in the same folder. Rename to the
// requested destination name if it differs.
if resp.Name != dstName {
err = f.renameFile(ctx, resp.FileID, dstName)
if err != nil {
return nil, fmt.Errorf("couldn't rename copied file: %w", err)
}
resp.Name = dstName
}
// A server-side copy is byte-identical to the source and the API
// preserves the source's modification time and content hash onto the
// copy, so the returned object must report the SOURCE's modtime and hash,
// not the copy's wall-clock time. Reading "now" here made the object
// rclone holds in memory disagree with the one read back from the remote
// (its Fingerprint check compares the two exactly).
return &Object{
fs: f,
remote: remote,
file: api.FileItem{
ID: resp.FileID,
Name: resp.Name,
SizeBytes: srcObj.file.SizeBytes,
MimeType: srcObj.file.MimeType,
Extension: srcObj.file.Extension,
Region: srcObj.file.Region,
CreatedAt: srcObj.file.CreatedAt,
UpdatedAt: srcObj.file.UpdatedAt,
ContentHash: srcObj.file.ContentHash,
},
}, nil
}
// Purge deletes all the files and the container
func (f *Fs) Purge(ctx context.Context, dir string) error {
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return err
}
// DELETE /api/folders/:id?permanent=1 recursively deletes all contents
err = f.removeFolder(ctx, directoryID, false)
if err != nil {
return err
}
f.dirCache.FlushDir(dir)
return nil
}
// PublicLink generates a public link to the remote path
func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration, unlink bool) (string, error) {
o, err := f.NewObject(ctx, remote)
if err != nil {
return "", err
}
obj := o.(*Object)
resp, err := f.createShareLink(ctx, obj.file.ID, expire)
if err != nil {
return "", err
}
return resp.Link.URL, nil
}
// DirCacheFlush resets the directory cache - used in testing as an
// optional interface
func (f *Fs) DirCacheFlush() {
f.dirCache.ResetRoot()
}
// About gets quota information from the Fs
func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
info, err := f.getWorkspaceInfo(ctx)
if err != nil {
return nil, err
}
usage := &fs.Usage{
Used: fs.NewUsageValue(info.Storage.Used),
Total: fs.NewUsageValue(info.Storage.Total),
Free: fs.NewUsageValue(info.Storage.Free),
}
return usage, nil
}
// ------------------------------------------------------------
// Object methods
// ------------------------------------------------------------
// String returns a description of the Object
func (o *Object) String() string {
return o.remote
}
// Remote returns the remote path
func (o *Object) Remote() string {
return o.remote
}
// ModTime returns the modification date of the file
func (o *Object) ModTime(ctx context.Context) time.Time {
if o.file.UpdatedAt > 0 {
return time.Unix(int64(o.file.UpdatedAt), 0)
}
return time.Unix(int64(o.file.CreatedAt), 0)
}
// Size returns the size of the file
func (o *Object) Size() int64 {
return o.file.SizeBytes
}
// Fs returns read only access to the Fs that this object is part of
func (o *Object) Fs() fs.Info {
return o.fs
}
// Hash returns the selected checksum of the file
func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
if t != hash.SHA256 {
return "", hash.ErrUnsupported
}
// May be "" - a multipart-uploaded object carries no server-side hash,
// which rclone treats as "hash unavailable for this object" (as it does
// for S3 multipart objects).
return o.file.ContentHash, nil
}
// Storable says whether this object can be stored
func (o *Object) Storable() bool {
return true
}
// SetModTime sets the metadata on the object to set the modification date
func (o *Object) SetModTime(ctx context.Context, t time.Time) error {
return fs.ErrorCantSetModTime
}
// Open opens the file for read
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadCloser, error) {
fs.FixRangeOption(options, o.file.SizeBytes)
return o.fs.downloadFile(ctx, o.file.ID, options)
}
// Update replaces the file content
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
// Find the directory ID for this file
_, directoryID, err := o.fs.dirCache.FindPath(ctx, o.remote, false)
if err != nil {
return err
}
// Upload as new version of existing file
fileID := o.file.ID
modTime := src.ModTime(ctx)
resp, err := o.fs.uploadFile(ctx, in, o.remote, src.Size(), modTime, fs.MimeType(ctx, src), directoryID, &fileID)
if err != nil {
return err
}
// Update object metadata
o.file = fileFromUpload(resp, modTime)
return nil
}
// Remove removes this object
func (o *Object) Remove(ctx context.Context) error {
return o.fs.deleteFile(ctx, o.file.ID)
}
// MimeType of an Object if known, "" otherwise
func (o *Object) MimeType(ctx context.Context) string {
return o.file.MimeType
}
// ID returns the ID of the Object if known, or "" if not
func (o *Object) ID() string {
return o.file.ID
}
func getRemote(dir, fileName string) string {
if dir == "" {
return fileName
}
return dir + "/" + fileName
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
_ fs.Mover = (*Fs)(nil)
_ fs.DirMover = (*Fs)(nil)
_ fs.Copier = (*Fs)(nil)
_ fs.Purger = (*Fs)(nil)
_ fs.PublicLinker = (*Fs)(nil)
_ fs.Abouter = (*Fs)(nil)
_ fs.DirCacheFlusher = (*Fs)(nil)
_ dircache.DirCacher = (*Fs)(nil)
_ fs.Object = (*Object)(nil)
_ fs.MimeTyper = (*Object)(nil)
_ fs.IDer = (*Object)(nil)
)
-880
View File
@@ -1,880 +0,0 @@
package dosya
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"path"
"strconv"
"strings"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/sync/errgroup"
)
// sourceMTimeHeader is the request header the upload endpoints read to set a
// file's modification time (Unix seconds); see lib/upload/source-timestamps.ts.
const sourceMTimeHeader = "x-dosya-source-mtime"
// modTimeHeaders returns the ExtraHeaders that carry a source modification time
// on an upload, or nil when there is nothing plausible to send. The server
// clamps implausible values itself, but a zero time is never worth a header.
func modTimeHeaders(modTime time.Time) map[string]string {
if modTime.IsZero() {
return nil
}
return map[string]string{sourceMTimeHeader: strconv.FormatInt(modTime.Unix(), 10)}
}
// workspaceHintHeader names the workspace a by-id request is about. The API
// maps an id to its workspace through an index the workspace fills about a
// second after a write, so an id this client was handed moments ago can still
// miss; the hint lets the server look for it in the named workspace instead.
// It only locates: every route still checks membership against the workspace
// it resolved, and a workspace-pinned key's hint is ignored unless it matches.
const workspaceHintHeader = "X-Dosya-Workspace"
// byIDHeaders returns the ExtraHeaders every /api/files/:id and
// /api/folders/:id call carries, or nil when no workspace is configured.
func (f *Fs) byIDHeaders() map[string]string {
if f.opt.WorkspaceID == "" {
return nil
}
return map[string]string{workspaceHintHeader: f.opt.WorkspaceID}
}
// retryErrorCodes is a slice of error codes that we will retry
var retryErrorCodes = []int{
429, // Too Many Requests
500, // Internal Server Error
502, // Bad Gateway
503, // Service Unavailable
504, // Gateway Timeout
}
// shouldRetry returns a boolean as to whether this resp and err
// deserve to be retried
func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, error) {
if fserrors.ContextError(ctx, &err) {
return false, err
}
retry := fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes)
if retry {
if d, ok := retryAfter(resp); ok {
// The API answers 503 + Retry-After while a workspace is being
// moved between storage homes, and Cloudflare sends it with 429.
// The pacer's own backoff tops out at maxSleep, far short of that.
fs.Debugf(nil, "Server asked to retry after %v", d)
err = pacer.RetryAfterError(err, d)
}
}
return retry, err
}
// maxRetryAfter caps how long a single Retry-After is honoured for
const maxRetryAfter = 5 * time.Minute
// retryAfter reads a 429 or 503 response's Retry-After header, which the API
// sends as whole seconds
func retryAfter(resp *http.Response) (time.Duration, bool) {
if resp == nil || (resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode != http.StatusServiceUnavailable) {
return 0, false
}
seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After")))
if err != nil || seconds <= 0 {
return 0, false
}
return min(time.Duration(seconds)*time.Second, maxRetryAfter), true
}
// errorHandler decodes a non-2xx response into an *api.Error so callers can
// act on the machine-readable error_code rather than the message text.
func errorHandler(resp *http.Response) error {
body, err := rest.ReadBody(resp)
if err != nil {
return fmt.Errorf("error reading error out of body: %w", err)
}
apiErr := &api.Error{
StatusCode: resp.StatusCode,
Status: resp.Status,
}
if json.Unmarshal(body, apiErr) != nil || apiErr.Message == "" {
// Not the API's JSON shape (e.g. an edge response that never
// reached the API) - keep the body text so it isn't lost.
apiErr.ErrorCode = ""
apiErr.Message = strings.TrimSpace(string(body))
}
return apiErr
}
// The workspace caps the number of in-flight uploads per user. The API
// refuses an upload/init past that cap with a 400 carrying
// error_code "concurrent_upload_limit", and documents it as an error to
// wait on rather than give up on: slots free as uploads finish, and the
// server marks sessions idle for 10 minutes as failed, so waiting up to
// that long guarantees progress unless the workspace really is saturated.
// A 400 is deliberately outside the pacer's retry codes, so the wait
// lives in initUpload instead of shouldRetry.
var (
concurrentUploadRetryDelay = 5 * time.Second
concurrentUploadWaitMax = 10 * time.Minute
)
// abortUploadTimeout bounds the abort of a session the client is giving up on,
// which runs outside the transfer's context so a cancelled transfer still
// sends it
const abortUploadTimeout = 10 * time.Second
// isConcurrentUploadLimit reports whether err is the API refusing an
// upload because the workspace's concurrent upload limit is reached
func isConcurrentUploadLimit(err error) bool {
var apiErr *api.Error
return errors.As(err, &apiErr) && apiErr.ErrorCode == "concurrent_upload_limit"
}
// listPageSize is the largest page GET /api/files will serve
const listPageSize = 500
// listFilesAndFolders lists files and folders in a directory
//
// The API pages files (at most listPageSize per page) but returns every
// folder on each page, so the files from all pages are gathered and the
// folders are taken from the first page only. Pages are ordered by name,
// which the API tie-breaks on id, so offsets are stable between requests.
func (f *Fs) listFilesAndFolders(ctx context.Context, folderID string) (*api.ListResponse, error) {
var listing *api.ListResponse
for page := 1; ; page++ {
opts := rest.Opts{
Method: "GET",
Path: "/api/files",
Parameters: map[string][]string{
"workspace_id": {f.opt.WorkspaceID},
"per_page": {strconv.Itoa(listPageSize)},
"page": {strconv.Itoa(page)},
"sort": {"name"},
"dir": {"asc"},
},
}
if folderID != "" && folderID != rootID {
opts.Parameters["folder_id"] = []string{folderID}
}
var result api.ListResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't list files: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
if listing == nil {
listing = &result
} else {
listing.Files = append(listing.Files, result.Files...)
}
if page >= result.Pagination.TotalPages || len(result.Files) == 0 {
return listing, nil
}
}
}
// createFolder creates a folder
func (f *Fs) createFolder(ctx context.Context, name string, parentID string) (*api.CreateFolderResponse, error) {
request := api.CreateFolderRequest{
WorkspaceID: f.opt.WorkspaceID,
Name: name,
}
if parentID != "" && parentID != rootID {
request.ParentID = &parentID
}
opts := rest.Opts{
Method: "POST",
Path: "/api/folders",
}
var result api.CreateFolderResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't create folder: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// removeFolder deletes a folder
//
// ?permanent=1 trashes and purges a live folder in one request on today's
// API. ifEmpty also sends ?if_empty=1, which asks the server to refuse with
// HTTP 409 "Folder is not empty" instead of purging a folder that still
// holds a live file or a live child folder; that 409 is turned into
// fs.ErrorDirectoryNotEmpty here rather than treated as a generic API
// error, and it is never retried (409 is not one of retryErrorCodes). A big
// subtree is purged in bounded passes (202, complete=false), so keep
// calling until the purge is complete. An older server that ignores
// permanent=1 only trashes on the first call (answering permanent=false),
// so the loop's second pass sends the plain DELETE that purges it.
func (f *Fs) removeFolder(ctx context.Context, folderID string, ifEmpty bool) error {
parameters := map[string][]string{"permanent": {"1"}}
if ifEmpty {
parameters["if_empty"] = []string{"1"}
}
opts := rest.Opts{
Method: "DELETE",
Path: "/api/folders/" + folderID,
Parameters: parameters,
ExtraHeaders: f.byIDHeaders(),
}
for pass := 0; ; pass++ {
var result api.DeleteFolderResponse
var statusCode int
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
if resp != nil {
statusCode = resp.StatusCode
}
return shouldRetry(ctx, resp, err)
})
if err != nil {
if ifEmpty && statusCode == http.StatusConflict {
return fs.ErrorDirectoryNotEmpty
}
if pass == 0 {
return fmt.Errorf("couldn't remove folder: %w", err)
}
return fmt.Errorf("couldn't permanently remove folder: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
if result.Permanent && result.Complete {
return nil
}
if pass >= maxFolderPurgePasses {
return fmt.Errorf("couldn't permanently remove folder: purge still incomplete after %d passes", pass)
}
}
}
// maxFolderPurgePasses bounds the DELETE calls spent purging one folder
const maxFolderPurgePasses = 1000
// renameFolder renames a folder
func (f *Fs) renameFolder(ctx context.Context, folderID string, newName string) error {
request := api.RenameFolderRequest{Name: newName}
opts := rest.Opts{
Method: "PUT",
Path: "/api/folders/" + folderID + "/rename",
ExtraHeaders: f.byIDHeaders(),
}
var result api.Response
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't rename folder: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
return nil
}
// moveFolder moves a folder to a new parent
func (f *Fs) moveFolder(ctx context.Context, folderID string, newParentID string, newName string) (renamed bool, err error) {
request := api.MoveFolderRequest{Name: newName}
if newParentID != "" && newParentID != rootID {
request.ParentID = &newParentID
}
opts := rest.Opts{
Method: "PUT",
Path: "/api/folders/" + folderID + "/move",
ExtraHeaders: f.byIDHeaders(),
}
var result api.MoveFolderResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return false, fmt.Errorf("couldn't move folder: %w", err)
}
if !result.OK {
return false, fmt.Errorf("API error: %s", result.Error)
}
// Servers that predate the name field ignore it and answer without a
// name, leaving the caller to rename separately
return newName != "" && result.Name == newName, nil
}
// deleteFile deletes a file
//
// ?permanent=1 trashes and purges it in one request on today's API. An
// older server that ignores the flag only soft-deletes it, answering
// permanent=false, so a second DELETE on the now soft-deleted file purges
// it from the database and R2, as it always has.
func (f *Fs) deleteFile(ctx context.Context, fileID string) error {
opts := rest.Opts{
Method: "DELETE",
Path: "/api/files/" + fileID,
Parameters: map[string][]string{"permanent": {"1"}},
ExtraHeaders: f.byIDHeaders(),
}
var result api.DeleteFileResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't delete file: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
// Answered into its own value so a field the second response omits
// can't be read off the first.
if !result.Permanent {
var permanent api.DeleteFileResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &permanent)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't permanently delete file: %w", err)
}
if !permanent.OK {
return fmt.Errorf("API error: %s", permanent.Error)
}
}
return nil
}
// renameFile renames a file
func (f *Fs) renameFile(ctx context.Context, fileID string, newName string) error {
request := api.RenameFileRequest{Name: newName}
opts := rest.Opts{
Method: "PUT",
Path: "/api/files/" + fileID + "/rename",
ExtraHeaders: f.byIDHeaders(),
}
var result api.RenameFileResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't rename file: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
return nil
}
// moveFile moves a file to a new folder and, if newName is non-empty,
// renames it in the same request
func (f *Fs) moveFile(ctx context.Context, fileID string, folderID string, newName string) error {
request := api.MoveFileRequest{}
if folderID != "" && folderID != rootID {
request.FolderID = &folderID
}
if newName != "" {
request.Name = &newName
}
opts := rest.Opts{
Method: "PUT",
Path: "/api/files/" + fileID + "/move",
ExtraHeaders: f.byIDHeaders(),
}
var result api.Response
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't move file: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
return nil
}
// copyFile copies a file to a folder
func (f *Fs) copyFile(ctx context.Context, fileID string, folderID string, name string) (*api.CopyFileResponse, error) {
request := api.CopyFileRequest{Name: name}
if folderID != "" && folderID != rootID {
request.FolderID = &folderID
}
opts := rest.Opts{
Method: "POST",
Path: "/api/files/" + fileID + "/copy",
ExtraHeaders: f.byIDHeaders(),
}
var result api.CopyFileResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't copy file: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// downloadFile downloads a file by its ID
//
// The download endpoint answers 302 with a presigned R2 URL in Location, so
// this is two calls. Both go through the rest client, which is built on
// fshttp, so --timeout, --contimeout, --ca-cert, --no-check-certificate,
// --user-agent, --bind, --dump and the proxy settings apply to downloads as
// they do to every other call, and errorHandler and shouldRetry decide the
// errors and the retries uniformly.
func (f *Fs) downloadFile(ctx context.Context, fileID string, options []fs.OpenOption) (io.ReadCloser, error) {
// NoRedirect stops the client following the 302 with the API's
// Authorization header attached. A 302 is not a 2xx, so Call reports it
// as an error (having read and closed the body); the Location header
// is what actually matters, and finding it clears the error, as in
// backend/linkbox.
opts := rest.Opts{
Method: "GET",
Path: "/api/files/" + fileID + "/download",
ExtraHeaders: f.byIDHeaders(),
NoRedirect: true,
}
var downloadURL string
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.Call(ctx, &opts)
if resp != nil {
if location := resp.Header.Get("Location"); location != "" {
downloadURL, err = location, nil
}
}
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't get download URL: %w", err)
}
if downloadURL == "" {
return nil, errors.New("no download URL in redirect response")
}
// Fetch the object from the presigned URL. It carries its own
// credentials and R2 refuses a request that also has an Authorization
// header, so the client-level bearer token is removed by setting the
// header empty: the rest client skips a header with an empty value.
opts = rest.Opts{
Method: "GET",
RootURL: downloadURL,
Options: options,
ExtraHeaders: map[string]string{"Authorization": ""},
}
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
var err error
resp, err = f.rest.Call(ctx, &opts)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't download file: %w", err)
}
return resp.Body, nil
}
// uploadSmallFile uploads a file smaller than 10MB in a single request
func (f *Fs) uploadSmallFile(ctx context.Context, in io.Reader, sessionID string, size int64, modTime time.Time) (*api.UploadCompleteResponse, error) {
opts := rest.Opts{
Method: "PUT",
Path: "/api/upload/" + sessionID,
Body: in,
ContentLength: &size,
ContentType: "application/octet-stream",
ExtraHeaders: modTimeHeaders(modTime),
}
// The body is a stream which can't be rewound, so a low level retry
// would resend whatever is left of it - nothing - and the server
// would store that as an empty file. Don't retry here; rclone's high
// level retries redo the whole transfer instead.
var result api.UploadCompleteResponse
err := f.pacer.CallNoRetry(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't upload file: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// uploadPart uploads a single part of a multipart upload
func (f *Fs) uploadPart(ctx context.Context, part []byte, sessionID string, partNumber int) (*api.UploadPartResponse, error) {
size := int64(len(part))
opts := rest.Opts{
Method: "PUT",
Path: "/api/upload/" + sessionID + "/part/" + strconv.Itoa(partNumber),
ContentLength: &size,
ContentType: "application/octet-stream",
}
var result api.UploadPartResponse
err := f.pacer.Call(func() (bool, error) {
// A fresh reader per attempt so a retry resends the whole part
opts.Body = bytes.NewReader(part)
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't upload part %d: %w", partNumber, err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// completeUpload finalizes a multipart upload
func (f *Fs) completeUpload(ctx context.Context, sessionID string, modTime time.Time) (*api.UploadCompleteResponse, error) {
opts := rest.Opts{
Method: "POST",
Path: "/api/upload/" + sessionID + "/complete",
ExtraHeaders: modTimeHeaders(modTime),
}
var result api.UploadCompleteResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't complete upload: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// initUpload initializes a new upload session
func (f *Fs) initUpload(ctx context.Context, fileName string, fileSize int64, mimeType string, folderID string, fileID *string) (*api.UploadInitResponse, error) {
request := api.UploadInitRequest{
WorkspaceID: f.opt.WorkspaceID,
FileName: fileName,
FileSize: fileSize,
MimeType: mimeType,
FileID: fileID,
}
if folderID != "" && folderID != rootID {
request.FolderID = &folderID
}
opts := rest.Opts{
Method: "POST",
Path: "/api/upload/init",
}
var result api.UploadInitResponse
deadline := time.Now().Add(concurrentUploadWaitMax)
for {
result = api.UploadInitResponse{}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err == nil {
break
}
if !isConcurrentUploadLimit(err) || time.Now().After(deadline) {
return nil, fmt.Errorf("couldn't init upload: %w", err)
}
fs.Debugf(f, "Upload slots in use, waiting %v before retrying %q: %v", concurrentUploadRetryDelay, fileName, err)
select {
case <-ctx.Done():
return nil, fmt.Errorf("couldn't init upload: %w", ctx.Err())
case <-time.After(concurrentUploadRetryDelay):
}
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// uploadParts reads the source part by part and uploads the parts.
//
// Part 1 goes on its own: the server creates the R2 multipart upload (and a
// new file's id and key) when it receives the first part, so parts sent
// before that has finished would each start an upload of their own. The
// rest go out opt.UploadConcurrency at a time. A part is only read from the
// source once a buffer is free, so at most that many parts are held in
// memory, and a short source is still found by the reader before the part
// it could not fill is sent.
func (f *Fs) uploadParts(ctx context.Context, in *readers.CountingReader, sessionID string, size, partSize int64, totalParts int) error {
readPart := func(buf []byte, partNum int) ([]byte, error) {
currentPartSize := partSize
if partNum == totalParts {
currentPartSize = size - partSize*int64(totalParts-1)
}
part := buf[:currentPartSize]
if _, err := io.ReadFull(in, part); err != nil {
if err == io.EOF || err == io.ErrUnexpectedEOF {
return nil, fmt.Errorf("expected %d bytes in input, but got %d: %w", size, in.BytesRead(), io.ErrUnexpectedEOF)
}
return nil, fmt.Errorf("failed to read part %d: %w", partNum, err)
}
return part, nil
}
concurrency := max(f.opt.UploadConcurrency, 1)
buffers := make(chan []byte, concurrency)
buffers <- make([]byte, partSize)
allocated := 1
first := <-buffers
part, err := readPart(first, 1)
if err != nil {
return err
}
if _, err := f.uploadPart(ctx, part, sessionID, 1); err != nil {
return err
}
buffers <- first
// A failed read has to take the parts already in flight down with it,
// or they upload to completion while the error waits on Wait.
ctx, cancel := context.WithCancel(ctx)
defer cancel()
g, gCtx := errgroup.WithContext(ctx)
var readErr error
for partNum := 2; partNum <= totalParts; partNum++ {
// Take a free buffer, allocating one while under the limit.
var buf []byte
select {
case buf = <-buffers:
default:
if allocated < concurrency {
buf = make([]byte, partSize)
allocated++
} else {
select {
case buf = <-buffers:
case <-gCtx.Done():
}
}
}
if buf == nil {
break // a part failed; its error comes back from Wait
}
part, err := readPart(buf, partNum)
if err != nil {
readErr = err
cancel()
break
}
g.Go(func() error {
defer func() { buffers <- buf }()
_, err := f.uploadPart(gCtx, part, sessionID, partNum)
return err
})
}
err = g.Wait()
if readErr != nil {
return readErr
}
return err
}
// abortUpload retires an upload session the client is giving up on.
//
// Best effort: the server reclaims an abandoned session itself, so a failure
// here - including an older server with no such route - costs the slot the
// wait it would have cost anyway and is not worth failing an upload over.
// Sent on a context of its own so a cancelled transfer still gives its slot
// back rather than leaving it to the sweep.
func (f *Fs) abortUpload(ctx context.Context, sessionID string) {
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), abortUploadTimeout)
defer cancel()
opts := rest.Opts{
Method: "DELETE",
Path: "/api/upload/" + sessionID,
NoResponse: true,
}
err := f.pacer.CallNoRetry(func() (bool, error) {
resp, err := f.rest.Call(ctx, &opts)
return shouldRetry(ctx, resp, err)
})
if err != nil {
fs.Debugf(f, "Couldn't abort upload session %q: %v", sessionID, err)
}
}
// uploadFile handles the full upload flow (small or multipart)
func (f *Fs) uploadFile(ctx context.Context, in io.Reader, remote string, size int64, modTime time.Time, mimeType string, folderID string, fileID *string) (resp *api.UploadCompleteResponse, err error) {
// Both refusals live here so a new file and a new version of an existing
// one answer the same way: upload/init needs a byte count up front, and
// the server stores no zero-length object.
if size < 0 {
return nil, errors.New("can't upload files with unknown size")
}
if size == 0 {
return nil, fs.ErrorCantUploadEmptyFiles
}
leaf := f.opt.Enc.FromStandardName(path.Base(remote))
initResp, err := f.initUpload(ctx, leaf, size, mimeType, folderID, fileID)
if err != nil {
return nil, err
}
// A session abandoned here holds one of the workspace's upload slots
// until the server's sweep retires it ten minutes later, and rclone's
// high-level retry opens a fresh session rather than resuming this one -
// so without this, a few failed attempts use up every slot.
defer func() {
if err != nil {
f.abortUpload(ctx, initResp.SessionID)
}
}()
// Count what the source actually supplies, so a source which ends
// early is reported as an error rather than stored as a truncated
// file.
counter := readers.NewCountingReader(in)
// Small file upload (single PUT, no resumable)
if initResp.Resumable == nil {
resp, err = f.uploadSmallFile(ctx, counter, initResp.SessionID, size, modTime)
if err != nil {
return nil, err
}
if got := int64(counter.BytesRead()); got != size {
return nil, fmt.Errorf("expected %d bytes in input, but got %d: %w", size, got, io.ErrUnexpectedEOF)
}
return resp, nil
}
// Multipart upload. Each part is read fully into memory before it is
// sent, so a short source is caught before the part goes out and a
// part which has to be retried is resent with all of its bytes.
partSize := initResp.Resumable.PartSize
totalParts := initResp.Resumable.TotalParts
// Both bounds matter. partSize*(totalParts-1) >= size means a part would
// have nothing to carry; size > partSize*totalParts means the last part
// is bigger than partSize, and readPart would slice past its buffer and
// panic.
if partSize <= 0 || totalParts <= 0 || partSize*int64(totalParts-1) >= size || size > partSize*int64(totalParts) {
return nil, fmt.Errorf("invalid multipart layout from server: %d parts of %d bytes for %d bytes", totalParts, partSize, size)
}
if err = f.uploadParts(ctx, counter, initResp.SessionID, size, partSize, totalParts); err != nil {
return nil, err
}
return f.completeUpload(ctx, initResp.SessionID, modTime)
}
// createShareLink creates a public share link for a file
func (f *Fs) createShareLink(ctx context.Context, fileID string, expire fs.Duration) (*api.ShareLinkResponse, error) {
request := api.ShareLinkRequest{}
if expire < fs.DurationOff {
days := int(time.Duration(expire).Hours() / 24)
if days < 1 {
days = 1
}
request.ExpiresInDays = &days
}
opts := rest.Opts{
Method: "POST",
Path: "/api/files/" + fileID + "/share",
ExtraHeaders: f.byIDHeaders(),
}
var result api.ShareLinkResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't create share link: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// getWorkspaceInfo fetches workspace storage info
// listWorkspaces lists the workspaces this credential can reach. An API key
// pinned to a workspace sees exactly one: its own.
func (f *Fs) listWorkspaces(ctx context.Context) ([]api.Workspace, error) {
opts := rest.Opts{
Method: "GET",
Path: "/api/workspaces",
}
var result api.WorkspaceListResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't list workspaces: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return result.Workspaces, nil
}
func (f *Fs) getWorkspaceInfo(ctx context.Context) (*api.WorkspaceInfoResponse, error) {
opts := rest.Opts{
Method: "GET",
Path: "/api/workspaces/" + f.opt.WorkspaceID,
}
var result api.WorkspaceInfoResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't get workspace info: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
-235
View File
@@ -1,235 +0,0 @@
package dosya
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"testing"
"time"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/pacer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestShouldRetryHonoursRetryAfter(t *testing.T) {
ctx := context.Background()
for _, tc := range []struct {
name string
status int
header string
want time.Duration
ok bool
}{
{"503 during a workspace move", http.StatusServiceUnavailable, "30", 30 * time.Second, true},
{"429 from the edge", http.StatusTooManyRequests, "7", 7 * time.Second, true},
{"capped", http.StatusServiceUnavailable, "86400", maxRetryAfter, true},
{"missing header", http.StatusServiceUnavailable, "", 0, false},
{"HTTP date is not seconds", http.StatusServiceUnavailable, "Wed, 21 Oct 2015 07:28:00 GMT", 0, false},
{"502 ignores the header", http.StatusBadGateway, "30", 0, false},
} {
t.Run(tc.name, func(t *testing.T) {
resp := jsonResp(tc.status, `{"ok":false,"error":"busy"}`)
if tc.header != "" {
resp.Header.Set("Retry-After", tc.header)
}
retry, err := shouldRetry(ctx, resp, errorHandler(resp))
assert.True(t, retry)
d, ok := pacer.IsRetryAfter(err)
assert.Equal(t, tc.ok, ok)
if tc.ok {
assert.Equal(t, tc.want, d)
}
})
}
}
func TestShouldRetryDoesNotRetryConflict(t *testing.T) {
resp := jsonResp(http.StatusConflict, `{"ok":false,"error":"A folder with this name already exists"}`)
resp.Header.Set("Retry-After", "30")
retry, err := shouldRetry(context.Background(), resp, errorHandler(resp))
assert.False(t, retry)
_, ok := pacer.IsRetryAfter(err)
assert.False(t, ok)
}
func TestRemoveFolderPurgesPermanently(t *testing.T) {
// a big subtree purges in bounded passes: 202 complete=false repeats
// the same request until complete=true
answers := []struct {
status int
body string
}{
{http.StatusAccepted, `{"ok":true,"permanent":true,"complete":false,"remaining":1}`},
{http.StatusOK, `{"ok":true,"permanent":true,"complete":true,"remaining":0}`},
}
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
require.Equal(t, "DELETE", r.Method)
require.Equal(t, "/api/folders/fld_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "", r.URL.Query().Get("if_empty"))
require.Less(t, calls, len(answers), "DELETE sent after the purge completed")
a := answers[calls]
calls++
return jsonResp(a.status, a.body), nil
}))
require.NoError(t, f.removeFolder(context.Background(), "fld_1", false))
assert.Equal(t, len(answers), calls)
}
func TestRemoveFolderStopsOnPurgeError(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
return jsonResp(http.StatusForbidden, `{"ok":false,"error":"This folder contains a locked file and cannot be purged"}`), nil
}))
err := f.removeFolder(context.Background(), "fld_1", false)
require.Error(t, err)
assert.Contains(t, err.Error(), "locked file")
assert.Equal(t, 1, calls)
}
func TestRemoveFolderIfEmptyRejectsNonEmptyWithoutRetry(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "/api/folders/fld_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "1", r.URL.Query().Get("if_empty"))
return jsonResp(http.StatusConflict, `{"ok":false,"error":"Folder is not empty"}`), nil
}))
err := f.removeFolder(context.Background(), "fld_1", true)
assert.ErrorIs(t, err, fs.ErrorDirectoryNotEmpty)
assert.Equal(t, 1, calls, "a 409 must not be retried by the pacer")
}
func TestRemoveFolderIfEmptyPurgesEmptyFolderInOneCall(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "1", r.URL.Query().Get("if_empty"))
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true,"complete":true}`), nil
}))
require.NoError(t, f.removeFolder(context.Background(), "fld_1", true))
assert.Equal(t, 1, calls)
}
func TestRmdirMakesOneRequest(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "DELETE", r.Method)
assert.Equal(t, "/api/folders/fld_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "1", r.URL.Query().Get("if_empty"))
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true,"complete":true}`), nil
}))
f.dirCache = dircache.New("", rootID, f)
require.NoError(t, f.dirCache.FindRoot(context.Background(), false))
f.dirCache.Put("sub", "fld_1")
require.NoError(t, f.Rmdir(context.Background(), "sub"))
assert.Equal(t, 1, calls, "Rmdir must not list the directory before removing it")
}
func TestRmdirMapsNonEmptyConflict(t *testing.T) {
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
return jsonResp(http.StatusConflict, `{"ok":false,"error":"Folder is not empty"}`), nil
}))
f.dirCache = dircache.New("", rootID, f)
require.NoError(t, f.dirCache.FindRoot(context.Background(), false))
f.dirCache.Put("sub", "fld_1")
err := f.Rmdir(context.Background(), "sub")
assert.ErrorIs(t, err, fs.ErrorDirectoryNotEmpty)
}
func TestDeleteFileSendsPermanentFlag(t *testing.T) {
for _, tc := range []struct {
name string
permanent bool
wantCalls int
}{
{"today's API purges in one call", true, 1},
{"an older server ignores the flag and still gets a second DELETE", false, 2},
} {
t.Run(tc.name, func(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "DELETE", r.Method)
assert.Equal(t, "/api/files/fil_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
return jsonResp(http.StatusOK, fmt.Sprintf(`{"ok":true,"permanent":%t}`, tc.permanent)), nil
}))
require.NoError(t, f.deleteFile(context.Background(), "fil_1"))
assert.Equal(t, tc.wantCalls, calls)
})
}
}
// decodeBody reads a JSON request body into a map
func decodeBody(t *testing.T, r *http.Request) map[string]any {
b, err := io.ReadAll(r.Body)
require.NoError(t, err)
var m map[string]any
require.NoError(t, json.Unmarshal(b, &m))
return m
}
func TestMoveFolderSendsName(t *testing.T) {
for _, tc := range []struct {
name string
newName string
respBody string
wantRenamed bool
}{
{"server honours name", "dst", `{"ok":true,"name":"dst"}`, true},
{"older server ignores name", "dst", `{"ok":true}`, false},
{"no rename requested", "", `{"ok":true,"name":"src"}`, false},
} {
t.Run(tc.name, func(t *testing.T) {
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
assert.Equal(t, "/api/folders/fld_1/move", r.URL.Path)
body := decodeBody(t, r)
assert.Equal(t, "fld_parent", body["parent_id"])
if tc.newName == "" {
assert.NotContains(t, body, "name")
} else {
assert.Equal(t, tc.newName, body["name"])
}
return jsonResp(http.StatusOK, tc.respBody), nil
}))
renamed, err := f.moveFolder(context.Background(), "fld_1", "fld_parent", tc.newName)
require.NoError(t, err)
assert.Equal(t, tc.wantRenamed, renamed)
})
}
}
func TestCopyFileSendsName(t *testing.T) {
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
assert.Equal(t, "/api/files/fil_1/copy", r.URL.Path)
body := decodeBody(t, r)
assert.Equal(t, "fld_dst", body["folder_id"])
assert.Equal(t, "final.txt", body["name"])
return jsonResp(http.StatusCreated, `{"ok":true,"file_id":"fil_2","name":"final.txt"}`), nil
}))
resp, err := f.copyFile(context.Background(), "fil_1", "fld_dst", "final.txt")
require.NoError(t, err)
assert.Equal(t, "fil_2", resp.FileID)
assert.Equal(t, "final.txt", resp.Name)
}
-171
View File
@@ -1,171 +0,0 @@
package dosya
import (
"context"
"io"
"net/http"
"strconv"
"strings"
"testing"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestFsAdvertisesSHA256AndSecondPrecision(t *testing.T) {
f := &Fs{}
assert.Equal(t, time.Second, f.Precision())
assert.True(t, f.Hashes().Contains(hash.SHA256), "SHA-256 must be advertised")
// Exactly SHA-256, nothing else.
assert.Equal(t, hash.NewHashSet(hash.SHA256), f.Hashes())
}
func TestObjectHashReturnsContentHash(t *testing.T) {
o := &Object{file: api.FileItem{ContentHash: "abc123"}}
got, err := o.Hash(context.Background(), hash.SHA256)
require.NoError(t, err)
assert.Equal(t, "abc123", got)
// A multipart object with no server-side hash yields "" and no error,
// exactly as an S3 multipart object does.
empty := &Object{file: api.FileItem{ContentHash: ""}}
got, err = empty.Hash(context.Background(), hash.SHA256)
require.NoError(t, err)
assert.Equal(t, "", got)
// Any other algorithm is unsupported.
_, err = o.Hash(context.Background(), hash.MD5)
assert.ErrorIs(t, err, hash.ErrUnsupported)
}
func TestObjectModTimeUsesUpdatedAt(t *testing.T) {
mt := time.Date(2003, 2, 3, 4, 5, 6, 0, time.UTC)
o := &Object{file: api.FileItem{UpdatedAt: float64(mt.Unix()), CreatedAt: 1}}
assert.True(t, mt.Equal(o.ModTime(context.Background())))
}
func TestModTimeHeaders(t *testing.T) {
assert.Nil(t, modTimeHeaders(time.Time{}), "a zero time sends no header")
mt := time.Date(2003, 2, 3, 4, 5, 6, 499, time.UTC)
h := modTimeHeaders(mt)
require.NotNil(t, h)
assert.Equal(t, strconv.FormatInt(mt.Unix(), 10), h[sourceMTimeHeader])
}
func TestFileFromUploadKeepsModTimeAndHash(t *testing.T) {
resp := &api.UploadCompleteResponse{}
resp.File.ID = "fil_1"
resp.File.Name = "x.txt"
resp.File.SizeBytes = 10
resp.File.CreatedAt = 1_000
resp.File.ContentHash = "deadbeef"
mt := time.Date(2004, 3, 3, 4, 5, 6, 0, time.UTC)
item := fileFromUpload(resp, mt)
assert.Equal(t, float64(mt.Unix()), item.UpdatedAt, "modtime is what we sent, not the create time")
assert.Equal(t, "deadbeef", item.ContentHash)
assert.Equal(t, float64(1_000), item.CreatedAt)
// A zero modtime falls back to the create time.
item = fileFromUpload(resp, time.Time{})
assert.Equal(t, float64(1_000), item.UpdatedAt)
}
// TestUploadSendsSourceMTimeHeader drives a real single-PUT upload through a
// fake transport and asserts the source-mtime header actually rides on the PUT.
func TestUploadSendsSourceMTimeHeader(t *testing.T) {
var putMTime string
rt := rtFunc(func(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_, _ = io.ReadAll(r.Body) // drain so the upload's counting reader advances
}
switch {
case r.Method == "POST" && r.URL.Path == "/api/upload/init":
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1"}`), nil
case r.Method == "PUT" && r.URL.Path == "/api/upload/upl_1":
putMTime = r.Header.Get(sourceMTimeHeader)
return jsonResp(http.StatusCreated, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":5,"content_hash":"h"}}`), nil
}
t.Fatalf("unexpected %s %s", r.Method, r.URL.Path)
return nil, nil
})
f := newFakeFs(rt)
mt := time.Date(2003, 2, 3, 4, 5, 6, 0, time.UTC)
resp, err := f.uploadFile(context.Background(), strings.NewReader("hello"), "file.txt", 5, mt, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, "h", resp.File.ContentHash)
assert.Equal(t, strconv.FormatInt(mt.Unix(), 10), putMTime, "the PUT must carry the source-mtime header")
}
// TestMultipartCompleteSendsSourceMTimeHeader asserts the header also rides on
// the completion request for a multipart upload (the server stores the modtime
// there, not on the individual parts).
func TestMultipartCompleteSendsSourceMTimeHeader(t *testing.T) {
var completeMTime string
rt := rtFunc(func(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_, _ = io.ReadAll(r.Body)
}
switch {
case r.URL.Path == "/api/upload/init":
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1","resumable":{"part_size":4,"total_parts":3}}`), nil
case strings.HasPrefix(r.URL.Path, "/api/upload/upl_1/part/"):
return jsonResp(http.StatusOK, `{"ok":true}`), nil
case r.URL.Path == "/api/upload/upl_1/complete":
completeMTime = r.Header.Get(sourceMTimeHeader)
return jsonResp(http.StatusOK, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":10}}`), nil
}
t.Fatalf("unexpected %s %s", r.Method, r.URL.Path)
return nil, nil
})
f := newFakeFs(rt)
mt := time.Date(2003, 2, 3, 4, 5, 6, 0, time.UTC)
_, err := f.uploadFile(context.Background(), strings.NewReader("0123456789"), "file.txt", 10, mt, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, strconv.FormatInt(mt.Unix(), 10), completeMTime, "the complete call must carry the source-mtime header")
}
// TestCopyReturnsSourceModTimeAndHash guards the fingerprint of the object a
// server-side Copy returns: it must report the SOURCE's modtime and hash (the
// copy is byte-identical and the API preserves both), not the copy's wall
// clock. Reading "now" here made rclone's in-memory object disagree with the
// one read back from the remote and failed FsCopy's fingerprint check.
func TestCopyReturnsSourceModTimeAndHash(t *testing.T) {
rt := rtFunc(func(r *http.Request) (*http.Response, error) {
switch {
case r.Method == "GET" && r.URL.Path == "/api/files":
// NewObject's existence probe on the destination: nothing there.
return jsonResp(http.StatusOK, `{"ok":true,"files":[],"folders":[]}`), nil
case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/copy"):
return jsonResp(http.StatusOK, `{"ok":true,"file_id":"fil_copy","name":"f-copy.txt"}`), nil
}
t.Fatalf("unexpected %s %s", r.Method, r.URL.Path)
return nil, nil
})
f := newFakeFs(rt)
f.dirCache = dircache.New("", rootID, f)
mt := time.Date(2001, 2, 3, 4, 5, 10, 0, time.UTC)
ct := time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC)
src := &Object{fs: f, remote: "orig.txt", file: api.FileItem{
ID: "fil_src", Name: "orig.txt", SizeBytes: 100, MimeType: "text/plain",
CreatedAt: float64(ct.Unix()), UpdatedAt: float64(mt.Unix()), ContentHash: "srchash",
}}
dst, err := f.Copy(context.Background(), src, "f-copy.txt")
require.NoError(t, err)
assert.True(t, mt.Equal(dst.ModTime(context.Background())), "copy keeps the source modtime")
assert.Equal(t, int64(100), dst.Size())
got, err := dst.Hash(context.Background(), hash.SHA256)
require.NoError(t, err)
assert.Equal(t, "srchash", got, "copy keeps the source hash")
}
-156
View File
@@ -1,156 +0,0 @@
package dosya
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const concurrentLimitBody = `{"ok":false,"error":"You have 5 uploads in progress. Wait for them to finish before starting more.","error_code":"concurrent_upload_limit","max_concurrent_uploads":5}`
func TestErrorHandlerDecodesAPIError(t *testing.T) {
resp := &http.Response{
StatusCode: http.StatusBadRequest,
Status: "400 Bad Request",
Body: io.NopCloser(strings.NewReader(concurrentLimitBody)),
}
err := errorHandler(resp)
var apiErr *api.Error
require.ErrorAs(t, err, &apiErr)
assert.Equal(t, http.StatusBadRequest, apiErr.StatusCode)
assert.Equal(t, "concurrent_upload_limit", apiErr.ErrorCode)
assert.Equal(t, 5, apiErr.MaxConcurrentUploads)
assert.Contains(t, err.Error(), "uploads in progress")
assert.Contains(t, err.Error(), "concurrent_upload_limit")
}
func TestErrorHandlerNonJSONBody(t *testing.T) {
// e.g. an edge (Cloudflare) response that never reached the API
resp := &http.Response{
StatusCode: http.StatusTooManyRequests,
Status: "429 Too Many Requests",
Body: io.NopCloser(strings.NewReader("error code: 1015\n")),
}
err := errorHandler(resp)
var apiErr *api.Error
require.ErrorAs(t, err, &apiErr)
assert.Equal(t, http.StatusTooManyRequests, apiErr.StatusCode)
assert.Equal(t, "", apiErr.ErrorCode)
assert.Contains(t, err.Error(), "429")
assert.Contains(t, err.Error(), "error code: 1015")
}
func TestShouldRetryAPIErrors(t *testing.T) {
ctx := context.Background()
// The concurrent upload limit is handled by initUpload's slot wait,
// not by the pacer's low level retries.
limitErr := &api.Error{StatusCode: http.StatusBadRequest, ErrorCode: "concurrent_upload_limit"}
retry, err := shouldRetry(ctx, &http.Response{StatusCode: http.StatusBadRequest}, limitErr)
assert.False(t, retry)
assert.Equal(t, limitErr, err)
retry, _ = shouldRetry(ctx, &http.Response{StatusCode: http.StatusTooManyRequests}, &api.Error{StatusCode: http.StatusTooManyRequests})
assert.True(t, retry)
retry, _ = shouldRetry(ctx, &http.Response{StatusCode: http.StatusBadRequest}, &api.Error{StatusCode: http.StatusBadRequest, ErrorCode: "something_else"})
assert.False(t, retry)
}
// swapWaitTimings shortens the slot wait timings for a test and returns a
// function which restores them
func swapWaitTimings(delay, max time.Duration) func() {
oldDelay, oldMax := concurrentUploadRetryDelay, concurrentUploadWaitMax
concurrentUploadRetryDelay, concurrentUploadWaitMax = delay, max
return func() { concurrentUploadRetryDelay, concurrentUploadWaitMax = oldDelay, oldMax }
}
// newTestFs makes an Fs which talks to srv
func newTestFs(srv *httptest.Server) *Fs {
ctx := context.Background()
f := &Fs{
name: "test",
opt: Options{WorkspaceID: "ws_test"},
pacer: fs.NewPacer(ctx, pacer.NewDefault(
pacer.MinSleep(time.Millisecond),
pacer.MaxSleep(time.Millisecond),
pacer.DecayConstant(decayConstant),
pacer.AttackConstant(attackConstant),
)),
}
f.rest = rest.NewClient(srv.Client()).SetRoot(srv.URL).SetErrorHandler(errorHandler)
return f
}
// limitServer answers the first n init calls with the concurrent upload
// limit error and the rest with success
func limitServer(t *testing.T, n int32, calls *atomic.Int32) *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "/api/upload/init", r.URL.Path)
w.Header().Set("Content-Type", "application/json")
if calls.Add(1) <= n {
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(concurrentLimitBody))
return
}
w.WriteHeader(http.StatusCreated)
_, _ = w.Write([]byte(`{"ok":true}`))
}))
}
func TestInitUploadWaitsForFreeSlot(t *testing.T) {
defer swapWaitTimings(5*time.Millisecond, 10*time.Second)()
var calls atomic.Int32
srv := limitServer(t, 2, &calls)
defer srv.Close()
f := newTestFs(srv)
resp, err := f.initUpload(context.Background(), "file.txt", 10, "text/plain", "", nil)
require.NoError(t, err)
assert.True(t, resp.OK)
assert.Equal(t, int32(3), calls.Load())
}
func TestInitUploadGivesUpWhenSlotsStayFull(t *testing.T) {
defer swapWaitTimings(5*time.Millisecond, 40*time.Millisecond)()
var calls atomic.Int32
srv := limitServer(t, 1<<30, &calls)
defer srv.Close()
f := newTestFs(srv)
_, err := f.initUpload(context.Background(), "file.txt", 10, "text/plain", "", nil)
require.Error(t, err)
var apiErr *api.Error
require.ErrorAs(t, err, &apiErr)
assert.Equal(t, "concurrent_upload_limit", apiErr.ErrorCode)
assert.Contains(t, err.Error(), "couldn't init upload")
assert.GreaterOrEqual(t, calls.Load(), int32(2), "should have retried at least once")
}
func TestInitUploadStopsWaitingWhenContextCancelled(t *testing.T) {
defer swapWaitTimings(time.Minute, time.Hour)()
var calls atomic.Int32
srv := limitServer(t, 1<<30, &calls)
defer srv.Close()
f := newTestFs(srv)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
_, err := f.initUpload(ctx, "file.txt", 10, "text/plain", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, context.DeadlineExceeded)
assert.Less(t, time.Since(start), 5*time.Second)
}
-68
View File
@@ -1,68 +0,0 @@
package dosya
import (
"context"
"fmt"
"net/http"
"strconv"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeListServer serves GET /api/files the way the API does: files paged at
// per_page, every folder repeated on each page
func fakeListServer(t *testing.T, totalFiles int, pages *[]int) rtFunc {
return func(r *http.Request) (*http.Response, error) {
require.Equal(t, "/api/files", r.URL.Path)
q := r.URL.Query()
assert.Equal(t, "name", q.Get("sort"))
assert.Equal(t, "asc", q.Get("dir"))
page, _ := strconv.Atoi(q.Get("page"))
perPage, _ := strconv.Atoi(q.Get("per_page"))
require.Positive(t, page)
require.Positive(t, perPage)
*pages = append(*pages, page)
var files []string
for i := (page - 1) * perPage; i < page*perPage && i < totalFiles; i++ {
files = append(files, fmt.Sprintf(`{"id":"fil_%d","name":"f%05d.txt","size":1}`, i, i))
}
totalPages := max(1, (totalFiles+perPage-1)/perPage)
body := fmt.Sprintf(`{"ok":true,"files":[%s],"folders":[{"id":"fld_1","name":"sub"}],`+
`"pagination":{"page":%d,"per_page":%d,"total_files":%d,"total_pages":%d}}`,
strings.Join(files, ","), page, perPage, totalFiles, totalPages)
return jsonResp(http.StatusOK, body), nil
}
}
func TestListFilesAndFoldersFollowsPages(t *testing.T) {
var pages []int
f := newFakeFs(fakeListServer(t, 2*listPageSize+7, &pages))
listing, err := f.listFilesAndFolders(context.Background(), "fld_parent")
require.NoError(t, err)
assert.Equal(t, []int{1, 2, 3}, pages)
require.Len(t, listing.Files, 2*listPageSize+7)
assert.Equal(t, "f01006.txt", listing.Files[len(listing.Files)-1].Name)
// folders repeat on every page and must not be duplicated
assert.Len(t, listing.Folders, 1)
}
func TestListFilesAndFoldersSinglePage(t *testing.T) {
for _, total := range []int{0, 1, listPageSize} {
t.Run(strconv.Itoa(total), func(t *testing.T) {
var pages []int
f := newFakeFs(fakeListServer(t, total, &pages))
listing, err := f.listFilesAndFolders(context.Background(), rootID)
require.NoError(t, err)
assert.Equal(t, []int{1}, pages)
assert.Len(t, listing.Files, total)
})
}
}
-153
View File
@@ -1,153 +0,0 @@
package dosya
import (
"context"
"fmt"
"io"
"net/http"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// parallelUploadServer answers the multipart endpoints without holding a
// lock across a request, so parts really do overlap, and records what the
// ordering rules below are about: how many parts were in flight at once, and
// whether any part started before part 1 had finished.
//
// Part 1 matters because the API creates the R2 multipart upload (and, for a
// new file, its id and key) on the first part it sees. Parts that race ahead
// of part 1 would each create their own upload, and only one of those would
// be completed.
type parallelUploadServer struct {
t *testing.T
resumable string
partDelay time.Duration
failPart int // answer this part with a 400, 0 for none
mu sync.Mutex
inFlight int
maxFlight int
part1Done bool
earlyParts []int // parts that started before part 1 finished
parts map[int]string
completes int
aborts int
}
func (s *parallelUploadServer) roundTrip(r *http.Request) (*http.Response, error) {
var body []byte
if r.Body != nil {
var err error
body, err = io.ReadAll(r.Body)
require.NoError(s.t, err)
}
path := r.URL.Path
switch {
case r.Method == "POST" && path == "/api/upload/init":
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1","resumable":`+s.resumable+`}`), nil
case r.Method == "PUT" && strings.HasPrefix(path, "/api/upload/upl_1/part/"):
var n int
_, err := fmt.Sscanf(strings.TrimPrefix(path, "/api/upload/upl_1/part/"), "%d", &n)
require.NoError(s.t, err)
s.mu.Lock()
if n != 1 && !s.part1Done {
s.earlyParts = append(s.earlyParts, n)
}
s.inFlight++
s.maxFlight = max(s.maxFlight, s.inFlight)
s.mu.Unlock()
time.Sleep(s.partDelay)
s.mu.Lock()
s.inFlight--
if n == 1 {
s.part1Done = true
}
s.parts[n] = string(body)
fail := n == s.failPart
s.mu.Unlock()
if fail {
return jsonResp(http.StatusBadRequest, `{"ok":false,"error":"part refused"}`), nil
}
return jsonResp(http.StatusCreated, fmt.Sprintf(`{"ok":true,"part_number":%d,"etag":"e%d"}`, n, n)), nil
case r.Method == "POST" && path == "/api/upload/upl_1/complete":
s.mu.Lock()
s.completes++
s.mu.Unlock()
return jsonResp(http.StatusOK, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":20}}`), nil
case r.Method == "DELETE" && path == "/api/upload/upl_1":
s.mu.Lock()
s.aborts++
s.mu.Unlock()
return jsonResp(http.StatusOK, `{"ok":true}`), nil
}
s.t.Fatalf("unexpected request %s %s", r.Method, path)
return nil, nil
}
func newParallelServer(t *testing.T, resumable string) *parallelUploadServer {
return &parallelUploadServer{t: t, resumable: resumable, partDelay: 50 * time.Millisecond, parts: map[int]string{}}
}
const tenParts = `{"part_size":2,"total_parts":10}`
func TestUploadMultipartSendsPartsConcurrently(t *testing.T) {
srv := newParallelServer(t, tenParts)
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 4
content := "aabbccddeeffgghhiijj"
_, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, 4, srv.maxFlight, "parts after the first should use the whole concurrency")
assert.Empty(t, srv.earlyParts, "no part may start before part 1 has finished")
assert.Equal(t, 1, srv.completes)
for i := 1; i <= 10; i++ {
assert.Equal(t, content[2*(i-1):2*i], srv.parts[i], "part %d carries its own bytes", i)
}
}
func TestUploadMultipartConcurrencyOneIsSequential(t *testing.T) {
srv := newParallelServer(t, tenParts)
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 1
_, err := f.uploadFile(context.Background(), strings.NewReader("aabbccddeeffgghhiijj"), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, 1, srv.maxFlight)
assert.Equal(t, 1, srv.completes)
}
func TestUploadMultipartFailedPartIsNotCompleted(t *testing.T) {
srv := newParallelServer(t, tenParts)
srv.failPart = 6
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 4
_, err := f.uploadFile(context.Background(), strings.NewReader("aabbccddeeffgghhiijj"), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.Contains(t, err.Error(), "part 6")
assert.Equal(t, 0, srv.completes, "an upload with a failed part must not be completed")
}
func TestUploadMultipartParallelRejectsShortSource(t *testing.T) {
// 20 bytes declared, 15 supplied: the short read is found by the
// reader, whichever parts are already in flight, and nothing completes.
srv := newParallelServer(t, tenParts)
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 4
_, err := f.uploadFile(context.Background(), strings.NewReader("aabbccddeeffggh"), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
assert.Equal(t, 0, srv.completes)
assert.Empty(t, srv.parts[8], "a part the source never filled must not be sent")
}
-15
View File
@@ -1,15 +0,0 @@
// Test Dosya filesystem interface
package dosya
import (
"testing"
"github.com/rclone/rclone/fstest/fstests"
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestDosya:",
})
}
-247
View File
@@ -1,247 +0,0 @@
package dosya
import (
"context"
"fmt"
"io"
"net/http"
"strings"
"sync"
"testing"
"time"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// rtFunc adapts a function into an http.RoundTripper so the upload path
// can be driven without a network. Unlike a real server it always reads
// the whole request body and answers, which is exactly the misbehaviour
// (accepting a short body as a complete upload) these tests guard against.
type rtFunc func(*http.Request) (*http.Response, error)
func (f rtFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
func jsonResp(status int, body string) *http.Response {
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{"Content-Type": {"application/json"}},
Body: io.NopCloser(strings.NewReader(body)),
ContentLength: int64(len(body)),
}
}
func newFakeFs(rt http.RoundTripper) *Fs {
ctx := context.Background()
f := &Fs{
name: "test",
opt: Options{WorkspaceID: "ws_test"},
pacer: fs.NewPacer(ctx, pacer.NewDefault(
pacer.MinSleep(time.Millisecond),
pacer.MaxSleep(time.Millisecond),
pacer.DecayConstant(decayConstant),
pacer.AttackConstant(attackConstant),
)),
}
f.rest = rest.NewClient(&http.Client{Transport: rt}).SetRoot("https://fake.invalid").SetErrorHandler(errorHandler)
return f
}
// fakeUploadServer records what an upload sends and lets a test inject
// failures per request
type fakeUploadServer struct {
t *testing.T
mu sync.Mutex
resumable string // JSON for the init "resumable" field, "" for a single PUT
puts [][]byte // bodies of PUT /api/upload/<id>
parts map[int][][]byte // bodies of PUT /api/upload/<id>/part/<n>, per part, per attempt
completes int
aborts int // DELETE /api/upload/<id>
failFirst map[string]bool // path -> answer the first request with a 502
seenBefore map[string]bool
}
func newFakeUploadServer(t *testing.T, resumable string) *fakeUploadServer {
return &fakeUploadServer{
t: t,
resumable: resumable,
parts: map[int][][]byte{},
failFirst: map[string]bool{},
seenBefore: map[string]bool{},
}
}
func (s *fakeUploadServer) roundTrip(r *http.Request) (*http.Response, error) {
s.mu.Lock()
defer s.mu.Unlock()
var body []byte
if r.Body != nil {
var err error
body, err = io.ReadAll(r.Body)
require.NoError(s.t, err)
}
path := r.URL.Path
// Bodies are recorded for every attempt, including the ones answered
// with a 502, so tests can see exactly what each attempt carried.
fail := s.failFirst[path] && !s.seenBefore[path]
s.seenBefore[path] = true
bad := jsonResp(http.StatusBadGateway, `{"ok":false,"error":"bad gateway"}`)
switch {
case r.Method == "POST" && path == "/api/upload/init":
if s.resumable != "" {
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1","resumable":`+s.resumable+`}`), nil
}
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1"}`), nil
case r.Method == "PUT" && path == "/api/upload/upl_1":
s.puts = append(s.puts, body)
if fail {
return bad, nil
}
return jsonResp(http.StatusCreated, fmt.Sprintf(`{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":%d}}`, len(body))), nil
case r.Method == "PUT" && strings.HasPrefix(path, "/api/upload/upl_1/part/"):
var n int
_, err := fmt.Sscanf(strings.TrimPrefix(path, "/api/upload/upl_1/part/"), "%d", &n)
require.NoError(s.t, err)
s.parts[n] = append(s.parts[n], body)
if fail {
return bad, nil
}
return jsonResp(http.StatusOK, fmt.Sprintf(`{"ok":true,"part_number":%d,"bytes_uploaded":%d}`, n, len(body))), nil
case r.Method == "POST" && path == "/api/upload/upl_1/complete":
s.completes++
return jsonResp(http.StatusOK, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":10}}`), nil
case r.Method == "DELETE" && path == "/api/upload/upl_1":
s.aborts++
return jsonResp(http.StatusOK, `{"ok":true}`), nil
}
s.t.Fatalf("unexpected request %s %s", r.Method, path)
return nil, nil
}
func TestUploadSmallSendsWholeBody(t *testing.T) {
srv := newFakeUploadServer(t, "")
f := newFakeFs(rtFunc(srv.roundTrip))
content := strings.Repeat("a", 100)
resp, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 100, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, "fil_1", resp.File.ID)
require.Len(t, srv.puts, 1)
assert.Equal(t, content, string(srv.puts[0]))
}
func TestUploadSmallRejectsShortSource(t *testing.T) {
// The source declares 100 bytes but only supplies 60. A server which
// accepts that as a complete upload must not turn into a success.
srv := newFakeUploadServer(t, "")
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader(strings.Repeat("a", 60)), "file.txt", 100, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
assert.Contains(t, err.Error(), "expected 100 bytes in input, but got 60")
}
func TestUploadSmallDoesNotResendDrainedBody(t *testing.T) {
// A retried PUT would resend an already-consumed reader, i.e. an
// empty body, which the server would store as an empty file. The
// single PUT therefore must not be retried at the low level.
srv := newFakeUploadServer(t, "")
srv.failFirst["/api/upload/upl_1"] = true
f := newFakeFs(rtFunc(srv.roundTrip))
content := strings.Repeat("a", 100)
_, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 100, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
require.Len(t, srv.puts, 1, "the PUT must not be retried with a drained body")
assert.Equal(t, content, string(srv.puts[0]))
}
const threeParts = `{"part_size":4,"total_parts":3}`
func TestUploadMultipartAssemblesAllBytes(t *testing.T) {
srv := newFakeUploadServer(t, threeParts)
f := newFakeFs(rtFunc(srv.roundTrip))
content := "0123456789"
resp, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, "fil_1", resp.File.ID)
assert.Equal(t, 1, srv.completes)
require.Len(t, srv.parts, 3)
assert.Equal(t, "0123", string(srv.parts[1][0]))
assert.Equal(t, "4567", string(srv.parts[2][0]))
assert.Equal(t, "89", string(srv.parts[3][0]))
}
func TestUploadMultipartRejectsShortSource(t *testing.T) {
// 10 bytes declared, 8 supplied: the last part must never be sent
// and the upload must not be completed.
srv := newFakeUploadServer(t, threeParts)
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader("01234567"), "file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
assert.Contains(t, err.Error(), "expected 10 bytes in input, but got 8")
assert.Empty(t, srv.parts[3], "short final part must not be uploaded")
assert.Equal(t, 0, srv.completes, "upload must not be completed")
}
func TestUploadMultipartRetriesPartWithFullBody(t *testing.T) {
// A part which fails transiently must be resent with all its bytes,
// not with whatever is left of a consumed reader.
srv := newFakeUploadServer(t, threeParts)
srv.failFirst["/api/upload/upl_1/part/2"] = true
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader("0123456789"), "file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
require.Len(t, srv.parts[2], 2, "part 2 should have been sent twice")
assert.Equal(t, "4567", string(srv.parts[2][0]))
assert.Equal(t, "4567", string(srv.parts[2][1]))
assert.Equal(t, "89", string(srv.parts[3][0]))
assert.Equal(t, 1, srv.completes)
}
// An upload that fails after init has to hand its session back: the session
// holds one of the workspace's upload slots for ten minutes otherwise, and
// rclone's high-level retry opens a new one rather than resuming it.
func TestUploadAbortsTheSessionOnFailure(t *testing.T) {
for _, tc := range []struct {
name string
resumable string
content string
size int64
}{
{"small file short source", "", "0123", 10},
{"multipart short source", `{"part_size":5,"total_parts":4}`, "aabbccddeeffggh", 20},
} {
t.Run(tc.name, func(t *testing.T) {
srv := newFakeUploadServer(t, tc.resumable)
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader(tc.content),
"file.txt", tc.size, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.Equal(t, 1, srv.aborts)
})
}
}
// A successful upload must not abort anything
func TestUploadDoesNotAbortOnSuccess(t *testing.T) {
srv := newFakeUploadServer(t, "")
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader(strings.Repeat("a", 10)),
"file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, 0, srv.aborts)
}
@@ -1,73 +0,0 @@
package dosya
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Every by-id call names the workspace, so the API can find an id it minted
// moments ago before its file_home index row lands.
func TestByIDCallsSendWorkspaceHint(t *testing.T) {
seen := map[string]string{}
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
seen[r.Method+" "+r.URL.Path] = r.Header.Get("X-Dosya-Workspace")
switch r.URL.Path {
case "/api/folders/fld_1":
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true,"complete":true}`), nil
case "/api/files/fil_1":
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true}`), nil
case "/api/folders/fld_1/move":
return jsonResp(http.StatusOK, `{"ok":true,"name":"n"}`), nil
default:
return jsonResp(http.StatusOK, `{"ok":true}`), nil
}
}))
ctx := context.Background()
require.NoError(t, f.removeFolder(ctx, "fld_1", false))
require.NoError(t, f.renameFolder(ctx, "fld_1", "n"))
_, err := f.moveFolder(ctx, "fld_1", "fld_2", "n")
require.NoError(t, err)
require.NoError(t, f.deleteFile(ctx, "fil_1"))
require.NoError(t, f.renameFile(ctx, "fil_1", "n"))
require.NoError(t, f.moveFile(ctx, "fil_1", "fld_2", "n"))
_, err = f.copyFile(ctx, "fil_1", "fld_2", "n")
require.NoError(t, err)
_, err = f.createShareLink(ctx, "fil_1", 0)
require.NoError(t, err)
for call, header := range seen {
assert.Equal(t, "ws_test", header, call)
}
assert.Len(t, seen, 8)
}
// The download's API request carries the hint; the presigned R2 fetch must not.
func TestDownloadSendsWorkspaceHintOnlyToTheAPI(t *testing.T) {
var apiHint, blobHint string
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/files/fil_1/download" {
apiHint = r.Header.Get("X-Dosya-Workspace")
http.Redirect(w, r, srv.URL+"/blob", http.StatusFound)
return
}
blobHint = r.Header.Get("X-Dosya-Workspace")
_, _ = w.Write([]byte("bytes"))
}))
defer srv.Close()
f := newTestFs(srv)
f.opt.APIURL = srv.URL
rc, err := f.downloadFile(context.Background(), "fil_1", nil)
require.NoError(t, err)
_ = rc.Close()
assert.Equal(t, "ws_test", apiHint)
assert.Equal(t, "", blobHint)
}
-52
View File
@@ -1,52 +0,0 @@
package dosya
import (
"context"
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeWorkspaceServer answers GET /api/workspaces with the given body
func fakeWorkspaceServer(t *testing.T, body string, calls *int) rtFunc {
return func(r *http.Request) (*http.Response, error) {
require.Equal(t, "/api/workspaces", r.URL.Path)
*calls++
return jsonResp(http.StatusOK, body), nil
}
}
func TestFindWorkspaceAdoptsTheOnlyOne(t *testing.T) {
calls := 0
f := newFakeFs(fakeWorkspaceServer(t, `{"ok":true,"workspaces":[{"id":"ws_only","name":"Mine"}]}`, &calls))
f.opt.WorkspaceID = ""
require.NoError(t, f.findWorkspace(context.Background()))
assert.Equal(t, "ws_only", f.opt.WorkspaceID)
assert.Equal(t, 1, calls)
}
func TestFindWorkspaceListsTheChoices(t *testing.T) {
calls := 0
f := newFakeFs(fakeWorkspaceServer(t, `{"ok":true,"workspaces":[`+
`{"id":"ws_one","name":"One"},{"id":"ws_two","name":"Two"}]}`, &calls))
f.opt.WorkspaceID = ""
err := f.findWorkspace(context.Background())
require.Error(t, err)
// the ids have to be in the message: they are the only place the user
// can read them, which is the whole point of listing them
assert.Contains(t, err.Error(), "ws_one (One)")
assert.Contains(t, err.Error(), "ws_two (Two)")
assert.Empty(t, f.opt.WorkspaceID)
}
func TestFindWorkspaceWithNone(t *testing.T) {
calls := 0
f := newFakeFs(fakeWorkspaceServer(t, `{"ok":true,"workspaces":[]}`, &calls))
f.opt.WorkspaceID = ""
require.ErrorContains(t, f.findWorkspace(context.Background()), "no workspaces found")
}
+1 -9
View File
@@ -321,13 +321,10 @@ func (f *Fs) getItem(ctx context.Context, id string, dirID string, leaf string)
}
return false
})
if err != nil {
return nil, err
}
if !found {
return nil, fs.ErrorObjectNotFound
}
return info, nil
return info, err
}
// errorHandler parses a non 2xx error response into an error
@@ -549,11 +546,6 @@ OUTER:
if err != nil {
return found, fmt.Errorf("couldn't list files: %w", err)
}
// Drime won't page past a fixed number of entries (20,000 for
// rclone) and serves the last page it allows again instead
if result.CurrentPage != page {
return found, errors.New("couldn't list files: directory has more entries than drime will list")
}
for _, item := range result.Data {
if item.Type == api.ItemTypeFolder {
if filesOnly {
-158
View File
@@ -1,158 +0,0 @@
package drime
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
"sync/atomic"
"testing"
"time"
"github.com/rclone/rclone/backend/drime/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const (
testPerPage = 5 // entries per page the test server returns
testLimit = 20 // entries the test server will page through
)
// listingServer serves /drive/file-entries from entries the way Drime
// does.
//
// It returns testPerPage entries per page and won't page past testLimit
// entries. Asking for a later page returns the last page it allows
// again, with last_page one beyond it.
//
// It counts the requests made in requests.
func listingServer(t *testing.T, entries []api.Item, requests *atomic.Int64) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
assert.Equal(t, "/drive/file-entries", r.URL.Path)
params := r.URL.Query()
page, err := strconv.Atoi(params.Get("page"))
require.NoError(t, err)
maxPage := testLimit / testPerPage
currentPage := min(page, maxPage)
start := min((currentPage-1)*testPerPage, len(entries))
end := min(start+testPerPage, len(entries))
lastPage := currentPage
if page > maxPage || end < len(entries) {
lastPage = currentPage + 1
}
w.Header().Set("Content-Type", "application/json")
require.NoError(t, json.NewEncoder(w).Encode(map[string]any{
"current_page": currentPage,
"last_page": lastPage,
"per_page": testPerPage,
"total": (currentPage + 1) * testPerPage,
"data": entries[start:end],
}))
}))
}
// makeEntries makes n file entries
func makeEntries(n int) []api.Item {
entries := make([]api.Item, n)
for i := range entries {
entries[i] = api.Item{
ID: json.Number(strconv.Itoa(1000 + i)),
Name: fmt.Sprintf("file%05d", i),
Type: "text",
}
}
return entries
}
// testFs makes an Fs which uses server as its API
func testFs(t *testing.T, server *httptest.Server) *Fs {
t.Helper()
f := &Fs{
opt: Options{ListChunk: 1000},
srv: rest.NewClient(server.Client()).SetRoot(server.URL),
}
f.pacer = fs.NewPacer(context.Background(), pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant)))
f.srv.SetErrorHandler(errorHandler)
return f
}
// listNames lists a directory with listAll and returns the names found
func listNames(ctx context.Context, f *Fs) (names []string, err error) {
_, err = f.listAll(ctx, "1", false, false, "", func(item *api.Item) bool {
names = append(names, item.Name)
return false
})
return names, err
}
// TestListAll checks that listAll returns every entry exactly once
func TestListAll(t *testing.T) {
for _, test := range []struct {
name string
entries int
}{
{name: "Empty", entries: 0},
{name: "OnePage", entries: 3},
{name: "ExactPages", entries: 10},
{name: "PartialLastPage", entries: 13},
{name: "AtLimit", entries: testLimit},
} {
t.Run(test.name, func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var requests atomic.Int64
entries := makeEntries(test.entries)
server := listingServer(t, entries, &requests)
defer server.Close()
f := testFs(t, server)
names, err := listNames(ctx, f)
require.NoError(t, err)
var want []string
for _, entry := range entries {
want = append(want, entry.Name)
}
assert.ElementsMatch(t, want, names)
})
}
}
// TestListAllTooManyEntries checks that listAll returns an error for a
// directory with more entries than the server will list, rather than
// looping forever.
func TestListAllTooManyEntries(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var requests atomic.Int64
server := listingServer(t, makeEntries(2*testLimit), &requests)
defer server.Close()
f := testFs(t, server)
_, err := listNames(ctx, f)
require.Error(t, err)
assert.NoError(t, ctx.Err(), "listing didn't finish")
assert.LessOrEqual(t, requests.Load(), int64(testLimit/testPerPage+1))
}
// TestGetItemListError checks that getItem returns listing errors
// rather than fs.ErrorObjectNotFound.
func TestGetItemListError(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, `{"message":"access denied"}`, http.StatusForbidden)
}))
defer server.Close()
f := testFs(t, server)
_, err := f.getItem(context.Background(), "1000", "1", "file00000")
require.Error(t, err)
assert.NotErrorIs(t, err, fs.ErrorObjectNotFound)
assert.Contains(t, err.Error(), "access denied")
}
+9 -22
View File
@@ -568,7 +568,7 @@ recommended usage is using the flag form --drive-size-as-quota when
doing rclone ls/lsl/lsf/lsjson/etc only.
If you do use this flag for syncing (not recommended) then you will
need to use --ignore-size also.`,
need to use --ignore size also.`,
Advanced: true,
Hide: fs.OptionHideConfigurator,
}, {
@@ -2298,9 +2298,15 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
wg.Wait()
// if the input channel overflowed add the collected entries to the channel now
for len(overflow) > 0 {
if !refillListRInput(&mu, &in, &overflow, &wg) {
return
mu.Lock()
// only fill half of the channel to prevent entries being put into overflow again
l := min(len(overflow), listRInputBuffer/2)
wg.Add(l)
for _, d := range overflow[:l] {
in <- d
}
overflow = overflow[l:]
mu.Unlock()
// wait again for the completion of all directories
wg.Wait()
@@ -2348,25 +2354,6 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
return nil
}
// refillListRInput moves queued directory jobs back onto in.
// It returns false when a listing error has already closed in.
// Sending on that nil channel would block forever while holding mu.
func refillListRInput(mu *sync.Mutex, in *chan listREntry, overflow *[]listREntry, wg *sync.WaitGroup) bool {
mu.Lock()
defer mu.Unlock()
if *in == nil {
return false
}
// only fill half of the channel to prevent entries being put into overflow again
l := min(len(*overflow), listRInputBuffer/2)
wg.Add(l)
for _, d := range (*overflow)[:l] {
*in <- d
}
*overflow = (*overflow)[l:]
return true
}
const shortcutSeparator = '\t'
// joinID adds an actual drive ID to the shortcut ID it came from
-152
View File
@@ -8,14 +8,10 @@ import (
"fmt"
"io"
"mime"
"net/http"
"net/http/httptest"
"os"
"path"
"path/filepath"
"strings"
stdsync "sync"
"sync/atomic"
"testing"
"time"
@@ -29,113 +25,13 @@ import (
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/api/drive/v3"
"google.golang.org/api/googleapi"
"google.golang.org/api/option"
)
func newPermissionTestFs(t *testing.T, handler http.Handler) *Fs {
t.Helper()
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
service, err := drive.NewService(
context.Background(),
option.WithHTTPClient(server.Client()),
option.WithEndpoint(server.URL+"/"),
)
require.NoError(t, err)
return &Fs{
svc: service,
pacer: fs.NewPacer(context.Background(), pacer.NewGoogleDrive()),
permissionsMu: new(stdsync.Mutex),
permissions: make(map[string]*drive.Permission),
}
}
func TestInternalGetPermissionConcurrent(t *testing.T) {
const requestCount = 2
arrived := make(chan string, requestCount)
release := make(chan struct{})
var releaseOnce stdsync.Once
releaseRequests := func() {
releaseOnce.Do(func() { close(release) })
}
defer releaseRequests()
f := newPermissionTestFs(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
permissionID := path.Base(r.URL.Path)
arrived <- permissionID
<-release
w.Header().Set("Content-Type", "application/json")
_, err := fmt.Fprintf(w, `{"id":%q,"role":"reader","type":"user"}`, permissionID)
assert.NoError(t, err)
}))
type permissionResult struct {
permission *drive.Permission
inherited bool
err error
}
results := make(chan permissionResult, requestCount)
permissionIDs := []string{"permission-1", "permission-2"}
for _, permissionID := range permissionIDs {
go func() {
permission, inherited, err := f.getPermission(context.Background(), "file-id", permissionID, false)
results <- permissionResult{permission: permission, inherited: inherited, err: err}
}()
}
arrivedIDs := make(map[string]bool, requestCount)
timer := time.NewTimer(time.Second)
defer timer.Stop()
for len(arrivedIDs) < requestCount {
select {
case permissionID := <-arrived:
arrivedIDs[permissionID] = true
case <-timer.C:
releaseRequests()
t.Fatalf("timed out waiting for concurrent permission requests: got %d of %d", len(arrivedIDs), requestCount)
}
}
releaseRequests()
assert.Equal(t, map[string]bool{"permission-1": true, "permission-2": true}, arrivedIDs)
returnedIDs := make(map[string]bool, requestCount)
for range requestCount {
result := <-results
require.NoError(t, result.err)
require.NotNil(t, result.permission)
assert.False(t, result.inherited)
returnedIDs[result.permission.Id] = true
}
assert.Equal(t, arrivedIDs, returnedIDs)
}
func TestInternalGetPermissionCacheHit(t *testing.T) {
var requests atomic.Int32
f := newPermissionTestFs(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
permissionID := path.Base(r.URL.Path)
w.Header().Set("Content-Type", "application/json")
_, err := fmt.Fprintf(w, `{"id":%q,"role":"reader","type":"user"}`, permissionID)
assert.NoError(t, err)
}))
first, inherited, err := f.getPermission(context.Background(), "file-id", "permission-1", true)
require.NoError(t, err)
assert.False(t, inherited)
second, inherited, err := f.getPermission(context.Background(), "file-id", "permission-1", true)
require.NoError(t, err)
assert.False(t, inherited)
assert.Same(t, first, second)
assert.Equal(t, int32(1), requests.Load())
}
func TestDriveScopes(t *testing.T) {
for _, test := range []struct {
in string
@@ -916,51 +812,3 @@ func (f *Fs) InternalTest(t *testing.T) {
}
var _ fstests.InternalTester = (*Fs)(nil)
func TestRefillListRInputStopsWhenInputClosed(t *testing.T) {
mu := stdsync.Mutex{}
wg := stdsync.WaitGroup{}
in := make(chan listREntry, listRInputBuffer)
overflow := []listREntry{{id: "folder", path: "folder"}}
mu.Lock()
close(in)
in = nil
mu.Unlock()
done := make(chan bool, 1)
go func() {
done <- refillListRInput(&mu, &in, &overflow, &wg)
}()
select {
case ok := <-done:
require.False(t, ok, "refill continued after the input channel was closed")
case <-time.After(time.Second):
t.Fatal("refill blocked after the input channel was closed")
}
require.Len(t, overflow, 1)
finished := make(chan struct{})
go func() {
wg.Wait()
close(finished)
}()
select {
case <-finished:
case <-time.After(time.Second):
t.Fatal("refill incremented the wait group after the input channel was closed")
}
}
func TestRefillListRInputSendsQueuedDirectory(t *testing.T) {
mu := stdsync.Mutex{}
wg := stdsync.WaitGroup{}
in := make(chan listREntry, listRInputBuffer)
overflow := []listREntry{{id: "folder", path: "folder"}}
require.True(t, refillListRInput(&mu, &in, &overflow, &wg), "refill stopped while the input channel was open")
require.Empty(t, overflow)
got := <-in
require.Equal(t, listREntry{id: "folder", path: "folder"}, got)
wg.Done()
}
+2 -4
View File
@@ -106,10 +106,10 @@ var permissionsFields = googleapi.Field(strings.Join([]string{
// getPermission returns permissions for the fileID and permissionID passed in
func (f *Fs) getPermission(ctx context.Context, fileID, permissionID string, useCache bool) (perm *drive.Permission, inherited bool, err error) {
f.permissionsMu.Lock()
defer f.permissionsMu.Unlock()
if useCache {
f.permissionsMu.Lock()
perm = f.permissions[permissionID]
f.permissionsMu.Unlock()
if perm != nil {
return perm, false, nil
}
@@ -131,9 +131,7 @@ func (f *Fs) getPermission(ctx context.Context, fileID, permissionID string, use
cleanPermission(perm)
// cache the permission
f.permissionsMu.Lock()
f.permissions[permissionID] = perm
f.permissionsMu.Unlock()
return perm, inherited, err
}
+18 -30
View File
@@ -11,6 +11,7 @@
package drive
import (
"bytes"
"context"
"encoding/json"
"fmt"
@@ -21,7 +22,6 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/readers"
"google.golang.org/api/drive/v3"
"google.golang.org/api/googleapi"
@@ -112,13 +112,8 @@ func (f *Fs) Upload(ctx context.Context, in io.Reader, size int64, contentType,
}
// Make an http.Request for the range passed in
//
// The body is wrapped in readers.NoCloser so the transport can't upgrade it
// to an io.Closer and close it after each attempt — the chunk buffer is
// pool-backed, so an early close would free its pages and a retry would then
// read a dead buffer. The upload loop owns the buffer's lifetime.
func (rx *resumableUpload) makeRequest(ctx context.Context, start int64, body io.ReadSeeker, reqSize int64) *http.Request {
req, _ := http.NewRequestWithContext(ctx, "POST", rx.URI, readers.NoCloser(body))
req, _ := http.NewRequestWithContext(ctx, "POST", rx.URI, body)
req.ContentLength = reqSize
totalSize := "*"
if rx.ContentLength >= 0 {
@@ -173,40 +168,37 @@ func (rx *resumableUpload) Upload(ctx context.Context) (*drive.File, error) {
start := int64(0)
var StatusCode int
var err error
chunkSize := int64(rx.f.opt.ChunkSize)
buf := make([]byte, int(rx.f.opt.ChunkSize))
for finished := false; !finished; {
reqSize := chunkSize
var reqSize int64
var chunk io.ReadSeeker
if rx.ContentLength >= 0 {
// If size known use repeatable reader for smoother bwlimit
if start >= rx.ContentLength {
break
}
reqSize = min(rx.ContentLength-start, chunkSize)
}
// Buffer the chunk in memory from the global pool so reads are
// repeatable for retries
rw := multipart.NewRW()
var n int64
n, err = io.CopyN(rw, rx.Media, reqSize)
if rx.ContentLength < 0 {
reqSize = min(rx.ContentLength-start, int64(rx.f.opt.ChunkSize))
chunk = readers.NewRepeatableLimitReaderBuffer(rx.Media, buf, reqSize)
} else {
// If size unknown read into buffer
var n int
n, err = readers.ReadFill(rx.Media, buf)
if err == io.EOF {
// Send the last chunk with the correct ContentLength
// otherwise Google doesn't know we've finished
rx.ContentLength = start + n
rx.ContentLength = start + int64(n)
finished = true
err = nil
} else if err != nil {
return nil, err
}
reqSize = n
}
if err != nil {
_ = rw.Close()
return nil, err
reqSize = int64(n)
chunk = bytes.NewReader(buf[:reqSize])
}
// Transfer the chunk
err = rx.f.pacer.Call(func() (bool, error) {
fs.Debugf(rx.remote, "Sending chunk %d length %d", start, reqSize)
StatusCode, err = rx.transferChunk(ctx, start, rw, reqSize)
StatusCode, err = rx.transferChunk(ctx, start, chunk, reqSize)
again, err := rx.f.shouldRetry(ctx, err)
if StatusCode == statusResumeIncomplete || StatusCode == http.StatusCreated || StatusCode == http.StatusOK {
again = false
@@ -214,13 +206,9 @@ func (rx *resumableUpload) Upload(ctx context.Context) (*drive.File, error) {
}
return again, err
})
closeErr := rw.Close()
if err != nil {
return nil, err
}
if closeErr != nil {
return nil, closeErr
}
start += reqSize
}
-57
View File
@@ -1,57 +0,0 @@
package drive
import (
"bytes"
"context"
"io"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestResumableUploadRetry checks that a chunk which fails with a 5xx is
// retried successfully. The chunk buffer is pool-backed and implements
// io.Closer, so if it reaches the http transport unwrapped the transport
// closes it after the failed attempt, returning its pages to the pool, and
// the retry then reads a freed buffer.
func TestResumableUploadRetry(t *testing.T) {
content := bytes.Repeat([]byte("resumable"), 512)
var attempts atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
require.NoError(t, err)
if attempts.Add(1) == 1 {
w.WriteHeader(http.StatusInternalServerError)
return
}
assert.Equal(t, content, body, "retried chunk should re-send the full chunk")
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"fake-id","name":"remote"}`))
}))
defer server.Close()
f := &Fs{
pacer: fs.NewPacer(context.Background(), pacer.NewGoogleDrive()),
client: server.Client(),
}
f.opt.ChunkSize = fs.SizeSuffix(len(content))
rx := &resumableUpload{
f: f,
remote: "remote",
URI: server.URL,
Media: bytes.NewReader(content),
MediaType: "application/octet-stream",
ContentLength: int64(len(content)),
}
info, err := rx.Upload(context.Background())
require.NoError(t, err)
require.NotNil(t, info)
assert.Equal(t, "fake-id", info.Id)
assert.Equal(t, int32(2), attempts.Load(), "expected exactly one failed attempt and one retry")
}
+10 -17
View File
@@ -51,7 +51,6 @@ import (
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/lib/batcher"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/readers"
@@ -2083,6 +2082,11 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
// write chunks
in := readers.NewCountingReader(in0)
bufSize := chunkSize
if size >= 0 && size < bufSize {
bufSize = size
}
buf := make([]byte, int(bufSize))
cursor := files.UploadSessionCursor{
SessionId: res.SessionId,
Offset: 0,
@@ -2102,21 +2106,14 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
fs.Debugf(o, "Uploading chunk %d/%d", currentChunk, chunks)
}
// Buffer the chunk in memory from the global pool for retries
rw := multipart.NewRW()
var n int64
n, err = io.CopyN(rw, in, chunkSize)
if err != nil && err != io.EOF {
_ = rw.Close()
return nil, err
}
chunk := readers.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
skip := int64(0)
err = o.fs.pacer.Call(func() (bool, error) {
// seek to the start in case this is a retry
if _, err = rw.Seek(skip, io.SeekStart); err != nil {
if _, err = chunk.Seek(skip, io.SeekStart); err != nil {
return false, err
}
err = o.fs.srv.UploadSessionAppendV2Context(ctx, &appendArg, rw)
err = o.fs.srv.UploadSessionAppendV2Context(ctx, &appendArg, chunk)
// after session is started, we retry everything
if err != nil {
// Check for incorrect offset error and retry with new offset
@@ -2128,10 +2125,10 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
what := fmt.Sprintf("incorrect offset error received: sent %d, need %d, skip %d", cursor.Offset, correctOffset, skip)
if skip < 0 {
return false, fmt.Errorf("can't seek backwards to correct offset: %s", what)
} else if skip == n {
} else if skip == chunkSize {
fs.Debugf(o, "%s: chunk received OK - continuing", what)
return false, nil
} else if skip > n {
} else if skip > chunkSize {
// This error should never happen
return false, fmt.Errorf("can't seek forwards by more than a chunk to correct offset: %s", what)
}
@@ -2148,13 +2145,9 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
}
return err != nil, err
})
closeErr := rw.Close()
if err != nil {
return nil, err
}
if closeErr != nil {
return nil, closeErr
}
if size >= 0 {
// Check for sources which truncate early
expected := min(uint64(currentChunk)*uint64(chunkSize), uint64(size))
+2 -5
View File
@@ -16,16 +16,13 @@ import (
// multipartInit starts a new multipart upload and returns server details.
func (f *Fs) multipartInit(ctx context.Context, folderPath, filename string) (*api.MultipartInitResponse, error) {
encodedFolderPath := f.fromStandardPath(folderPath)
encodedFilename := f.fromStandardPath(filename)
opts := rest.Opts{
Method: "GET",
Path: "/multipart/init",
Parameters: url.Values{
"key": {f.opt.Key},
"filename": {encodedFilename},
"folder_path": {encodedFolderPath},
"filename": {filename},
"folder_path": {folderPath},
},
}
+54 -56
View File
@@ -1,6 +1,7 @@
package filelu
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -10,11 +11,9 @@ import (
"net/http"
"net/url"
"path"
"strconv"
"strings"
"github.com/rclone/rclone/fs"
rclonemultipart "github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/rest"
)
@@ -48,32 +47,40 @@ func (f *Fs) multipartUpload(ctx context.Context, in io.Reader, remote string) e
server := initResp.Result.Server
objectPath := initResp.Result.ObjectPath
chunkSize := int64(f.opt.ChunkSize)
if chunkSize <= 0 {
return fmt.Errorf("multipart upload: chunk_size must be positive: %v", f.opt.ChunkSize)
}
for partNo := 1; ; partNo++ {
// Buffer the part in memory from the global pool so it can be
// re-sent on retry
rw := rclonemultipart.NewRW()
n, err := io.CopyN(rw, in, chunkSize)
if err != nil && err != io.EOF {
_ = rw.Close()
return fmt.Errorf("read failed: %w", err)
}
chunkSize := int(f.opt.ChunkSize)
buf := make([]byte, 0, chunkSize)
tmp := make([]byte, 1024*1024)
partNo := 1
for {
n, errRead := in.Read(tmp)
if n > 0 {
uploadErr := f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, rw, n)
if uploadErr != nil {
_ = rw.Close()
return fmt.Errorf("upload part %d failed: %w", partNo, uploadErr)
buf = append(buf, tmp[:n]...)
// If buffer reached chunkSize, upload a full part
if len(buf) >= chunkSize {
err = f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, bytes.NewReader(buf))
if err != nil {
return fmt.Errorf("upload part %d failed: %w", partNo, err)
}
partNo++
buf = buf[:0]
}
}
if closeErr := rw.Close(); closeErr != nil {
return closeErr
}
if err == io.EOF {
if errRead == io.EOF {
break
}
if errRead != nil {
return fmt.Errorf("read failed: %w", errRead)
}
}
if len(buf) > 0 {
err = f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, bytes.NewReader(buf))
if err != nil {
return fmt.Errorf("upload part %d failed: %w", partNo, err)
}
}
err = f.completeMultipart(ctx, server, uploadID, sessID, objectPath)
@@ -84,40 +91,31 @@ func (f *Fs) multipartUpload(ctx context.Context, in io.Reader, remote string) e
return nil
}
// uploadPart sends a single multipart chunk of size bytes to the upload server.
func (f *Fs) uploadPart(ctx context.Context, server, uploadID, sessID, objectPath string, partNo int, r io.ReadSeeker, size int64) error {
opts := rest.Opts{
Method: "PUT",
RootURL: server,
Parameters: url.Values{
"partNumber": {strconv.Itoa(partNo)},
"uploadId": {uploadID},
},
Body: r,
ContentLength: &size,
ExtraHeaders: map[string]string{
"X-RC-Upload-Id": uploadID,
"X-RC-Part-No": strconv.Itoa(partNo),
"X-Sess-ID": sessID,
"X-Object-Path": objectPath,
},
NoResponse: true,
// uploadPart sends a single multipart chunk to the upload server.
func (f *Fs) uploadPart(ctx context.Context, server, uploadID, sessID, objectPath string, partNo int, r io.Reader) error {
url := fmt.Sprintf("%s?partNumber=%d&uploadId=%s", server, partNo, uploadID)
req, err := http.NewRequestWithContext(ctx, "PUT", url, r)
if err != nil {
return err
}
return f.pacer.Call(func() (bool, error) {
// rewind the pooled buffer so each attempt sends the whole chunk
_, err := r.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
resp, err := f.srv.Call(ctx, &opts)
if err != nil {
if resp != nil {
return shouldRetryHTTP(resp.StatusCode), fmt.Errorf("uploadPart failed: %w", err)
}
return shouldRetry(err), err
}
return false, nil
})
req.Header.Set("X-RC-Upload-Id", uploadID)
req.Header.Set("X-RC-Part-No", fmt.Sprintf("%d", partNo))
req.Header.Set("X-Sess-ID", sessID)
req.Header.Set("X-Object-Path", objectPath)
resp, err := f.client.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != 200 {
return fmt.Errorf("uploadPart failed: %s", resp.Status)
}
return nil
}
// uploadFile uploads a file to FileLu
-29
View File
@@ -1,29 +0,0 @@
package filelu
import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest/fstests"
)
// SetUploadChunkSize changes the configured chunk size, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.ChunkSize = f.opt.ChunkSize, cs
return old, nil
}
// SetUploadCutoff changes the configured upload cutoff, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadCutoff(cutoff fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.UploadCutoff = f.opt.UploadCutoff, cutoff
return old, nil
}
var (
_ fstests.SetUploadChunkSizer = (*Fs)(nil)
_ fstests.SetUploadCutoffer = (*Fs)(nil)
)
-3
View File
@@ -12,8 +12,5 @@ func TestIntegration(t *testing.T) {
RemoteName: "TestFileLu:",
NilObject: nil,
SkipInvalidUTF8: true,
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: 1,
},
})
}
+10 -5
View File
@@ -353,7 +353,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
}
for it.Next() {
item := new(it.File())
item := ptr(it.File())
remote := f.opt.Enc.ToStandardPath(item.DisplayName)
remote = path.Join(dir, remote)
if remote == dir {
@@ -424,7 +424,7 @@ func (f *Fs) mkdir(ctx context.Context, path string) error {
params := files_sdk.FolderCreateParams{
Path: path,
MkdirParents: new(true),
MkdirParents: ptr(true),
}
err := f.pacer.Call(func() (bool, error) {
@@ -466,7 +466,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
params := files_sdk.FileDeleteParams{
Path: path,
Recursive: new(!check),
Recursive: ptr(!check),
}
err := f.pacer.Call(func() (bool, error) {
@@ -537,7 +537,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (dstObj fs.
params := files_sdk.FileCopyParams{
Path: srcPath,
Destination: dstPath,
Overwrite: new(true),
Overwrite: ptr(true),
}
var action files_sdk.FileAction
@@ -680,7 +680,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
Paths: []string{f.absPath(remote)},
}
if expire < fs.DurationOff {
params.ExpiresAt = new(time.Now().Add(time.Duration(expire)))
params.ExpiresAt = ptr(time.Now().Add(time.Duration(expire)))
}
var bundle files_sdk.Bundle
@@ -842,6 +842,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
return
}
// Returns a pointer to t - useful for returning pointers to constants
func ptr[T any](t T) *T {
return &t
}
func isFolderNotEmpty(err error) bool {
var re files_sdk.ResponseError
ok := errors.As(err, &re)
+14 -32
View File
@@ -584,9 +584,8 @@ func (f *Fs) getFtpConnection(ctx context.Context) (c *ftp.ServerConn, err error
//
// It nils the pointed to connection out so it can't be reused
//
// if err is not nil then it discards the connection unless the error
// carries an FTP reply code, in which case it checks the connection
// is alive using a NOOP request
// if err is not nil then it checks the connection is alive using a
// NOOP request
func (f *Fs) putFtpConnection(pc **ftp.ServerConn, err error) {
if f.opt.Concurrency > 0 {
defer f.tokens.Put()
@@ -600,17 +599,14 @@ func (f *Fs) putFtpConnection(pc **ftp.ServerConn, err error) {
}
*pc = nil
if err != nil {
// An error without an FTP reply code means the control
// connection can no longer be trusted, so discard it.
if textprotoError(err) == nil {
fs.Debugf(f, "Discarding connection after non-FTP error: %v", err)
_ = c.Quit()
return
}
if nopErr := c.NoOp(); nopErr != nil {
fs.Debugf(f, "Connection failed, closing: %v", nopErr)
_ = c.Quit()
return
// If not a regular FTP error code then check the connection
if tpErr := textprotoError(err); tpErr != nil {
nopErr := c.NoOp()
if nopErr != nil {
fs.Debugf(f, "Connection failed, closing: %v", nopErr)
_ = c.Quit()
return
}
}
}
f.poolMu.Lock()
@@ -1269,18 +1265,15 @@ func (o *Object) Storable() bool {
// ftpReadCloser implements io.ReadCloser for FTP objects.
type ftpReadCloser struct {
rc io.ReadCloser
c *ftp.ServerConn
f *Fs
err error // errors found during read
closed bool // set by Close so a second call is a no-op
read int64 // bytes returned by Read
rc io.ReadCloser
c *ftp.ServerConn
f *Fs
err error // errors found during read
}
// Read bytes into p
func (f *ftpReadCloser) Read(p []byte) (n int, err error) {
n, err = f.rc.Read(p)
f.read += int64(n)
if err != nil && err != io.EOF {
f.err = err // store any errors for Close to examine
}
@@ -1289,10 +1282,6 @@ func (f *ftpReadCloser) Read(p []byte) (n int, err error) {
// Close the FTP reader and return the connection to the pool
func (f *ftpReadCloser) Close() error {
if f.closed {
return nil
}
f.closed = true
var err error
errchan := make(chan error, 1)
go func() {
@@ -1316,7 +1305,6 @@ func (f *ftpReadCloser) Close() error {
// if errors while reading or closing, dump the connection
if err != nil || f.err != nil {
_ = f.c.Quit()
f.c = nil // never pool a connection after Quit
f.f.putFtpConnection(nil, nil)
} else {
f.f.putFtpConnection(&f.c, nil)
@@ -1328,12 +1316,6 @@ func (f *ftpReadCloser) Close() error {
switch errX.Code {
case ftp.StatusTransfertAborted, ftp.StatusFileUnavailable, ftp.StatusAboutToSend, ftp.StatusRequestedFileActionOK:
err = nil
case ftp.StatusCanNotOpenDataConnection:
// Egnyte answers 425 when a RETR is closed before the first
// read; a transfer cut after data was read stays an error
if f.read == 0 {
err = nil
}
}
}
return err
+1 -164
View File
@@ -1,19 +1,12 @@
package ftp
import (
"bufio"
"context"
"errors"
"fmt"
"io"
"net"
"net/textproto"
"strings"
"sync"
"testing"
"time"
"github.com/jlaffaye/ftp"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/object"
@@ -50,7 +43,7 @@ func TestCommandEncodingNeutralisesCRLF(t *testing.T) {
type settings map[string]any
func deriveFs(ctx context.Context, t *testing.T, f fs.Fs, opts settings) fs.Fs {
fsName, _, _ := strings.Cut(f.Name(), "{") // strip off hash
fsName := strings.Split(f.Name(), "{")[0] // strip off hash
configMap := configmap.Simple{}
for key, val := range opts {
configMap[key] = fmt.Sprintf("%v", val)
@@ -151,159 +144,3 @@ func (f *Fs) InternalTest(t *testing.T) {
}
var _ fstests.InternalTester = (*Fs)(nil)
// fakeFTPServer is a minimal FTP control-connection server which answers
// every command with 200 OK.
type fakeFTPServer struct {
ln net.Listener
mu sync.Mutex
conn net.Conn
}
func startFakeFTPServer(t *testing.T) *fakeFTPServer {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
s := &fakeFTPServer{ln: ln}
go s.serve()
t.Cleanup(func() { s.close() })
return s
}
func (s *fakeFTPServer) serve() {
conn, err := s.ln.Accept()
if err != nil {
return
}
s.mu.Lock()
s.conn = conn
s.mu.Unlock()
defer func() { _ = conn.Close() }()
_, _ = fmt.Fprint(conn, "220 fake FTP server\r\n")
scanner := bufio.NewScanner(conn)
for scanner.Scan() {
_, _ = fmt.Fprint(conn, "200 OK\r\n")
}
}
// killConnection closes the server side of the control connection,
// simulating a server that dropped an idle connection.
func (s *fakeFTPServer) killConnection() {
s.mu.Lock()
defer s.mu.Unlock()
if s.conn != nil {
_ = s.conn.Close()
}
}
func (s *fakeFTPServer) close() {
s.mu.Lock()
defer s.mu.Unlock()
if s.conn != nil {
_ = s.conn.Close()
}
_ = s.ln.Close()
}
// TestPutFtpConnectionDiscardsDeadConnection checks that a connection
// returned with a raw network error is closed and never re-pooled, while
// a connection returned with a plain FTP reply-code error is probed with
// NOOP and pooled while the probe succeeds.
func TestPutFtpConnectionDiscardsDeadConnection(t *testing.T) {
t.Run("network error discards", func(t *testing.T) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
f := &Fs{opt: Options{}}
f.putFtpConnection(&c, errors.New("unexpected EOF"))
assert.Nil(t, c)
assert.Empty(t, f.pool)
})
t.Run("FTP reply error with live connection pools", func(t *testing.T) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
f := &Fs{opt: Options{}}
f.putFtpConnection(&c, &textproto.Error{Code: 550, Msg: "not found"})
assert.Nil(t, c)
require.Len(t, f.pool, 1)
assert.NoError(t, f.pool[0].NoOp())
_ = f.pool[0].Quit()
})
t.Run("FTP reply error with dead connection discards", func(t *testing.T) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
server.killConnection()
// Let the FIN propagate so the NOOP probe observes the dead connection.
time.Sleep(100 * time.Millisecond)
f := &Fs{opt: Options{}}
f.putFtpConnection(&c, &textproto.Error{Code: 550, Msg: "not found"})
assert.Nil(t, c)
assert.Empty(t, f.pool)
})
}
// fakeDataReader stands in for the RETR data stream of ftpReadCloser.
type fakeDataReader struct {
data []byte
closeErr error
}
func (r *fakeDataReader) Read(p []byte) (int, error) {
if len(r.data) == 0 {
return 0, io.EOF
}
n := copy(p, r.data)
r.data = r.data[n:]
return n, nil
}
func (r *fakeDataReader) Close() error { return r.closeErr }
// TestFtpReadCloserClose checks that an early-closed RETR answered with 425
// is masked only when nothing was read, that a connection which got QUIT is
// never pooled, and that a second Close does not touch the pool.
func TestFtpReadCloserClose(t *testing.T) {
err425 := &textproto.Error{Code: ftp.StatusCanNotOpenDataConnection, Msg: "Can't open data connection."}
newReader := func(t *testing.T, data []byte, closeErr error) (*ftpReadCloser, *Fs) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
f := &Fs{opt: Options{}}
return &ftpReadCloser{rc: &fakeDataReader{data: data, closeErr: closeErr}, c: c, f: f}, f
}
t.Run("425 before any read is masked and the connection is not pooled", func(t *testing.T) {
rc, f := newReader(t, nil, err425)
assert.NoError(t, rc.Close())
assert.Empty(t, f.pool)
assert.NoError(t, rc.Close())
assert.Empty(t, f.pool)
})
t.Run("425 after data was read stays an error", func(t *testing.T) {
rc, f := newReader(t, []byte("partial"), err425)
buf := make([]byte, 4)
n, err := rc.Read(buf)
require.NoError(t, err)
require.Equal(t, 4, n)
err = rc.Close()
require.Error(t, err)
assert.Equal(t, ftp.StatusCanNotOpenDataConnection, textprotoError(err).Code)
assert.Empty(t, f.pool)
})
t.Run("clean close pools the connection once", func(t *testing.T) {
rc, f := newReader(t, []byte("data"), nil)
_, err := io.ReadAll(rc)
require.NoError(t, err)
assert.NoError(t, rc.Close())
require.Len(t, f.pool, 1)
assert.NoError(t, rc.Close())
assert.Len(t, f.pool, 1)
_ = f.pool[0].Quit()
})
}
+2 -1
View File
@@ -194,7 +194,8 @@ var retryErrorCodes = []int{
// Return true if the api error has the status given
func isAPIErr(err error, status string) bool {
if apiErr, ok := errors.AsType[api.Error](err); ok {
var apiErr api.Error
if errors.As(err, &apiErr) {
return apiErr.Status == status
}
return false
-44
View File
@@ -122,49 +122,6 @@ func (f *Fs) testUpdateStoresHash(t *testing.T) {
_ = operations.Purge(ctx, f, dirName)
}
func (f *Fs) testSetModTimeKeepsHash(t *testing.T) {
ctx := context.Background()
const dirName = "setmodtime_hash_1"
const fileName = dirName + "/file_setmodtime_1"
const longTime = fs.ModTimeNotSupported
hashType := f.keepHashes.GetOne()
// upload a file to hasher via Put
src := putFile(ctx, t, f, fileName, "setmodtime content")
dst, err := f.NewObject(ctx, fileName)
require.NoError(t, err)
require.NotNil(t, dst)
_ = src
// verify hash was stored after Put
if f.opt.MaxAge <= 0 {
t.Skip("hash cache is disabled")
}
hash1, err := f.getRawHash(ctx, hashType, fileName, anyFingerprint, longTime)
require.NoError(t, err)
require.NotEmpty(t, hash1)
// change only the modification time, as operations does when it
// syncs the modtime of an otherwise unchanged destination file
newTime := fstest.Time("2002-03-04T05:06:07.499999999Z")
require.NoError(t, dst.SetModTime(ctx, newTime))
// the cached hash must survive a pure modtime change
hash2, err := f.getRawHash(ctx, hashType, fileName, anyFingerprint, longTime)
assert.NoError(t, err)
assert.Equal(t, hash1, hash2, "hash should be kept after SetModTime")
// and it must be found via the fingerprint-checked lookup too
dst2, err := f.NewObject(ctx, fileName)
require.NoError(t, err)
hashVal, err := dst2.Hash(ctx, hashType)
assert.NoError(t, err)
assert.Equal(t, hash1, hashVal, "hash should be returned after SetModTime")
_ = operations.Purge(ctx, f, dirName)
}
// InternalTest dispatches all internal tests
func (f *Fs) InternalTest(t *testing.T) {
if !kv.Supported() {
@@ -172,7 +129,6 @@ func (f *Fs) InternalTest(t *testing.T) {
}
t.Run("UploadFromCrypt", f.testUploadFromCrypt)
t.Run("UpdateStoresHash", f.testUpdateStoresHash)
t.Run("SetModTimeKeepsHash", f.testSetModTimeKeepsHash)
}
var _ fstests.InternalTester = (*Fs)(nil)
-28
View File
@@ -206,34 +206,6 @@ func (op *kvPut) Do(ctx context.Context, b kv.Bucket) (err error) {
return err
}
// kvUpdateFingerprint: rewrite the fingerprint of an existing record
type kvUpdateFingerprint struct {
key string
oldFp string
newFp string
age time.Duration
}
func (op *kvUpdateFingerprint) Do(ctx context.Context, b kv.Bucket) error {
data := b.Get([]byte(op.key))
if len(data) == 0 {
return nil
}
var r hashRecord
if err := r.decode(op.key, data); err != nil {
return nil
}
if r.Fp != op.oldFp || time.Since(r.Created) > op.age {
return nil
}
r.Fp = op.newFp
data, err := r.encode(op.key)
if err != nil {
return err
}
return b.Put([]byte(op.key), data)
}
// kvDump: dump the database.
// Note: long dump can cause concurrent operations to fail.
type kvDump struct {
+5 -35
View File
@@ -56,23 +56,6 @@ func (o *Object) putHashes(ctx context.Context, rawHashes hashMap) error {
return o.f.putRawHashes(ctx, key, fp, hashes)
}
// updateFingerprint rewrites the fingerprint of the cached hash
// record for remote from oldFp to newFp, keeping the hashes. It is
// used when only the modification time of an object changes, so the
// cached hashes remain valid under the new fingerprint.
func (f *Fs) updateFingerprint(ctx context.Context, remote, oldFp, newFp string) error {
if oldFp == "" || newFp == "" || oldFp == newFp || f.opt.MaxAge <= 0 {
return nil
}
key := path.Join(f.Fs.Root(), remote)
return f.db.Do(true, &kvUpdateFingerprint{
key: key,
oldFp: oldFp,
newFp: newFp,
age: time.Duration(f.opt.MaxAge),
})
}
// set hashes for a path without any validation
func (f *Fs) putRawHashes(ctx context.Context, key, fp string, hashes operations.HashSums) error {
return f.db.Do(true, &kvPut{
@@ -213,27 +196,14 @@ func (o *Object) Remove(ctx context.Context) error {
}
// SetModTime sets the modification time of the file.
//
// When the fingerprint includes the modtime, the contents have not
// changed, so the cached hashes are carried over to the fingerprint
// for the new modtime. Otherwise the cache entry is pruned when the
// modtime changes so that touching a file will trigger checksum
// recalculation even on backends that don't provide modTime with
// fingerprint.
// Also prunes the cache entry when modtime changes so that
// touching a file will trigger checksum recalculation even
// on backends that don't provide modTime with fingerprint.
func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error {
if mtime == o.Object.ModTime(ctx) {
return o.Object.SetModTime(ctx, mtime)
}
if !o.f.fpTime {
if mtime != o.Object.ModTime(ctx) {
_ = o.f.pruneHash(o.Remote())
return o.Object.SetModTime(ctx, mtime)
}
oldFp := o.fingerprint(ctx)
if err := o.Object.SetModTime(ctx, mtime); err != nil {
return err
}
_ = o.f.updateFingerprint(ctx, o.Remote(), oldFp, o.fingerprint(ctx))
return nil
return o.Object.SetModTime(ctx, mtime)
}
// Open opens the file for read.
+61 -82
View File
@@ -10,6 +10,7 @@ package hidrive
// to be resolved prior to execution with resolvePath(...).
import (
"bytes"
"context"
"errors"
"io"
@@ -23,9 +24,8 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/ranges"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/sync/errgroup"
"golang.org/x/sync/semaphore"
@@ -454,17 +454,17 @@ func (f *Fs) deleteObject(ctx context.Context, path string) error {
}
// createFile creates a file at the given path
// with the content of the buffer.
// with the content of the io.ReadSeeker.
// This guarantees that existing files will not be overwritten.
// The maximum size of the content is limited by MaximumUploadBytes.
// The caller remains responsible for closing the buffer.
// The io.ReadSeeker should be resettable by seeking to its start.
// If modTime is not the zero time instant,
// it will be set as the file's modification time after the operation.
//
// This returns fs.ErrorDirNotFound
// if the parent directory of the file is not found.
// This returns ErrorFileExists if a file already exists at the specified path.
func (f *Fs) createFile(ctx context.Context, path string, content *pool.RW, modTime time.Time, onExist OnExistAction) (*api.HiDriveObject, error) {
func (f *Fs) createFile(ctx context.Context, path string, content io.ReadSeeker, modTime time.Time, onExist OnExistAction) (*api.HiDriveObject, error) {
parameters := api.NewQueryParameters()
parameters.SetFileInDirectory(path)
if onExist == AutoNameOnExist {
@@ -479,14 +479,12 @@ func (f *Fs) createFile(ctx context.Context, path string, content *pool.RW, modT
}
}
contentLength := content.Size()
opts := rest.Opts{
Method: "POST",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
ContentLength: &contentLength,
Parameters: parameters.Values,
Method: "POST",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
Parameters: parameters.Values,
}
var result api.HiDriveObject
@@ -512,16 +510,16 @@ func (f *Fs) createFile(ctx context.Context, path string, content *pool.RW, modT
}
// overwriteFile updates the content of the file at the given path
// with the content of the buffer.
// with the content of the io.ReadSeeker.
// If the file does not exist it will be created.
// The maximum size of the content is limited by MaximumUploadBytes.
// The caller remains responsible for closing the buffer.
// The io.ReadSeeker should be resettable by seeking to its start.
// If modTime is not the zero time instant,
// it will be set as the file's modification time after the operation.
//
// This returns fs.ErrorDirNotFound
// if the parent directory of the file is not found.
func (f *Fs) overwriteFile(ctx context.Context, path string, content *pool.RW, modTime time.Time) (*api.HiDriveObject, error) {
func (f *Fs) overwriteFile(ctx context.Context, path string, content io.ReadSeeker, modTime time.Time) (*api.HiDriveObject, error) {
parameters := api.NewQueryParameters()
parameters.SetFileInDirectory(path)
@@ -533,14 +531,12 @@ func (f *Fs) overwriteFile(ctx context.Context, path string, content *pool.RW, m
}
}
contentLength := content.Size()
opts := rest.Opts{
Method: "PUT",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
ContentLength: &contentLength,
Parameters: parameters.Values,
Method: "PUT",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
Parameters: parameters.Values,
}
var result api.HiDriveObject
@@ -638,18 +634,11 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
}
// Read a chunk of data.
var (
chunkReader *pool.RW
bytesRead int64
)
chunkReader, bytesRead, readErr = readerForChunk(content, int64(chunkSize), true)
if bytesRead < int64(chunkSize) {
chunkReader, bytesRead, readErr := readerForChunk(content, chunkSize)
if bytesRead < chunkSize {
startMoreTransfers = false
}
if readErr != nil || bytesRead <= 0 {
if chunkReader != nil {
_ = chunkReader.Close()
}
break
}
@@ -659,12 +648,11 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
// After this upload is done,
// signal that another transfer can be started.
defer transferSemaphore.Release(1)
defer func() { _ = chunkReader.Close() }()
uploadErr := f.patchFile(gCtx, path, chunkReader, chunkOffset, zeroTime)
uploadErr := f.patchFile(gCtx, path, cachedReader(chunkReader), chunkOffset, zeroTime)
if uploadErr == nil {
// Remember successfully written chunks.
okChunksMu.Lock()
okChunks = append(okChunks, ranges.Range{Pos: int64(chunkOffset), Size: bytesRead})
okChunks = append(okChunks, ranges.Range{Pos: int64(chunkOffset), Size: int64(bytesRead)})
okChunksMu.Unlock()
fs.Debugf(f, "Done uploading chunk of size %v at offset %v.", bytesRead, chunkOffset)
} else {
@@ -708,10 +696,10 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
// starting at the given offset.
//
// Replaces the file contents starting from the given byte offset
// with the content of the buffer.
// with the content of the io.ReadSeeker.
// If the offset is beyond the file end, the file is extended up to the offset.
// The maximum size of the update is limited by MaximumUploadBytes.
// The caller remains responsible for closing the buffer.
// The io.ReadSeeker should be resettable by seeking to its start.
// If modTime is not the zero time instant,
// it will be set as the file's modification time after the operation.
//
@@ -723,7 +711,7 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
// usually expand the holes into regions of 0-bytes.
//
// This returns fs.ErrorObjectNotFound if the object is not found.
func (f *Fs) patchFile(ctx context.Context, path string, content *pool.RW, offset uint64, modTime time.Time) error {
func (f *Fs) patchFile(ctx context.Context, path string, content io.ReadSeeker, offset uint64, modTime time.Time) error {
parameters := api.NewQueryParameters()
parameters.SetPath(path)
parameters.Set("offset", strconv.FormatUint(offset, 10))
@@ -735,15 +723,13 @@ func (f *Fs) patchFile(ctx context.Context, path string, content *pool.RW, offse
}
}
contentLength := content.Size()
opts := rest.Opts{
Method: "PATCH",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
ContentLength: &contentLength,
Parameters: parameters.Values,
NoResponse: true,
Method: "PATCH",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
Parameters: parameters.Values,
NoResponse: true,
}
var resp *http.Response
@@ -853,48 +839,41 @@ func createHiDriveScopes(role string, access string) []string {
return []string{}
}
// unwrapAccounting splits any transfer accounting off reader,
// returning the raw stream and the accounting (nil if there is none),
// so the stream can be buffered and accounted when the buffer is uploaded.
func unwrapAccounting(reader io.Reader) (unwrapped io.Reader, acc *accounting.Account) {
// Any kind of accounter re-wraps into the one type UnWrapAccounting
// knows how to take the *Account out of.
unwrapped, wrap := accounting.UnWrap(reader)
_, acc = accounting.UnWrapAccounting(wrap(unwrapped))
return unwrapped, acc
}
// readerForChunk reads up to length bytes from reader into a buffer
// from the global memory pool and returns it
// along with the number of bytes that have been read from reader.
// Reaching the end of reader early is not an error.
//
// If accountUpload is set, any accounting on reader is applied
// when the buffer is read instead, so the upload is what gets accounted.
//
// This must not be set for a buffer which creates or replaces a file
// in a single request. Accounting can fail part way through the upload,
// for example when --max-transfer is reached, and HiDrive keeps
// what it has received of an aborted request as the file.
//
// The caller must Close the buffer when done with it.
func readerForChunk(reader io.Reader, length int64, accountUpload bool) (chunkReader *pool.RW, bytesRead int64, err error) {
chunkReader = multipart.NewRW()
if accountUpload {
var acc *accounting.Account
reader, acc = unwrapAccounting(reader)
if acc != nil {
chunkReader.SetAccounting(acc.AccountRead)
}
// cachedReader returns a version of the reader that caches its contents and
// can therefore be reset using Seek.
func cachedReader(reader io.Reader) io.ReadSeeker {
bytesReader, ok := reader.(*bytes.Reader)
if ok {
return bytesReader
}
bytesRead, err = io.CopyN(chunkReader, reader, length)
if err == io.EOF {
repeatableReader, ok := reader.(*readers.RepeatableReader)
if ok {
return repeatableReader
}
return readers.NewRepeatableReader(reader)
}
// readerForChunk reads a chunk of bytes from reader (after handling any accounting).
// Returns a new io.Reader (chunkReader) for that chunk
// and the number of bytes that have been read from reader.
func readerForChunk(reader io.Reader, length int) (chunkReader io.Reader, bytesRead int, err error) {
// Unwrap any accounting from the input if present.
reader, wrap := accounting.UnWrap(reader)
// Read a chunk of data.
buffer := make([]byte, length)
bytesRead, err = io.ReadFull(reader, buffer)
if err == io.EOF || err == io.ErrUnexpectedEOF {
err = nil
}
if err != nil {
_ = chunkReader.Close()
return nil, bytesRead, err
}
return chunkReader, bytesRead, nil
// Truncate unused capacity.
buffer = buffer[:bytesRead]
// Use wrap to put any accounting back for chunkReader.
return wrap(bytes.NewReader(buffer)), bytesRead, nil
}
+4 -17
View File
@@ -30,7 +30,6 @@ import (
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/rest"
)
@@ -515,14 +514,8 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
// (i.e. everything up to the cutoff) in the first request,
// avoids files being created on upload failure for small files.
// (As opposed to creating an empty file and then uploading the content.)
//
// Only buffer as much as the source declares it has,
// so small files do not cost the whole cutoff in memory.
prefixSize := int64(f.opt.UploadCutoff)
if size := src.Size(); size >= 0 && size < prefixSize {
prefixSize = size
}
cutoffReader, bytesRead, err := readerForChunk(in, prefixSize, false)
tmpReader, bytesRead, err := readerForChunk(in, int(f.opt.UploadCutoff))
cutoffReader := cachedReader(tmpReader)
if err != nil {
return nil, err
}
@@ -547,7 +540,6 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
}
return false, createErr
})
_ = cutoffReader.Close()
if err != nil {
return nil, err
@@ -564,7 +556,7 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
}
// If there is more left to write, o.Update needs to skip ahead.
// Use a fs.SeekOption with the current offset to do this.
options = append(options, &fs.SeekOption{Offset: bytesRead})
options = append(options, &fs.SeekOption{Offset: int64(bytesRead)})
err = o.Update(ctx, in, src, options...)
if err == nil {
@@ -979,12 +971,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// Metadata should be updated even if the upload fails.
info, metaErr = o.fs.fetchMetadataForPath(ctx, resolvedPath, api.HiDriveObjectWithMetadataFields)
} else {
var content *pool.RW
content, _, err = readerForChunk(in, src.Size(), false)
if err == nil {
info, err = o.fs.overwriteFile(ctx, resolvedPath, content, modTime)
_ = content.Close()
}
info, err = o.fs.overwriteFile(ctx, resolvedPath, cachedReader(in), modTime)
metaErr = err
}
+11 -44
View File
@@ -63,8 +63,7 @@ The headers are only sent to the host in the configured URL. If the
server redirects to another host (including a subdomain or a different
port) the headers are not sent to it, or to any further hop in that
redirect chain. When headers are set, a redirect from https to http is
refused as it would send them in cleartext, even with
--allow-https-downgrade.`,
refused as it would send them in cleartext.`,
Default: fs.CommaSepList{},
Advanced: true,
}, {
@@ -101,30 +100,6 @@ that directory listings are much quicker, but rclone won't have the times or
sizes of any files, and some files that don't exist may be in the listing.`,
Default: false,
Advanced: true,
}, {
Name: "no_head_size",
Help: `Set the reported file size for --http-no-head.
When HEAD requests are disabled, rclone normally reports an unknown file size.
Set this option to a non-negative value to report that size for every file,
including 0. Leave it unset (the default, shown as "off") to keep reporting
an unknown size.
The configured size is authoritative. If it does not match the actual file
size, transfers may fail their size check and retry unless --ignore-size is
used. It can also make multi-thread transfers request incorrect ranges and
make mounted files unreadable past the reported size.`,
Default: fs.SizeSuffix(-1),
Advanced: true,
}, {
Name: "no_head_time",
Help: `Set a default modification time for --http-no-head.
When HEAD requests are disabled, rclone normally reports an unset modification
time. Set this option to report the specified time for every file instead.
Changing the configured time later makes every file appear modified to a sync.`,
Default: fs.Time{},
Advanced: true,
}, {
Name: "no_escape",
Help: "Do not escape URL metacharacters in path names.",
@@ -170,13 +145,11 @@ var systemMetadataInfo = map[string]fs.MetadataHelp{
// Options defines the configuration for this backend
type Options struct {
Endpoint string `config:"url"`
NoSlash bool `config:"no_slash"`
NoHead bool `config:"no_head"`
NoHeadSize fs.SizeSuffix `config:"no_head_size"`
NoHeadTime fs.Time `config:"no_head_time"`
Headers fs.CommaSepList `config:"headers"`
NoEscape bool `config:"no_escape"`
Endpoint string `config:"url"`
NoSlash bool `config:"no_slash"`
NoHead bool `config:"no_head"`
Headers fs.CommaSepList `config:"headers"`
NoEscape bool `config:"no_escape"`
}
// Fs stores the interface to the remote HTTP files
@@ -316,9 +289,8 @@ func (f *Fs) httpConnection(ctx context.Context, opt *Options) (isFile bool, err
}
client := fshttp.NewClient(ctx)
// Without configured headers keep the default policy so
// --allow-https-downgrade can follow public mirrors that
// redirect from https to http.
// Without configured headers keep the default policy which
// public mirrors that redirect from https to http rely on.
if len(opt.Headers) > 0 {
client.CheckRedirect = checkRedirect(opt)
}
@@ -350,9 +322,7 @@ func (f *Fs) httpConnection(ctx context.Context, opt *Options) (isFile bool, err
// the host specified in the config file.
func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
// Parse config into Options struct
opt := &Options{
NoHeadSize: fs.SizeSuffix(-1),
}
opt := new(Options)
err := configstruct.Set(m, opt)
if err != nil {
return nil, err
@@ -603,7 +573,7 @@ func (f *Fs) readDir(ctx context.Context, dir string) (names []string, err error
return nil, fmt.Errorf("failed to readDir: %w", err)
}
contentType, _, _ := strings.Cut(res.Header.Get("Content-Type"), ";")
contentType := strings.SplitN(res.Header.Get("Content-Type"), ";", 2)[0]
switch contentType {
case "text/html":
names, err = parse(u, res.Body)
@@ -747,11 +717,8 @@ func (o *Object) url() string {
// head sends a HEAD request to update info fields in the Object
func (o *Object) head(ctx context.Context) error {
if o.fs.opt.NoHead {
o.size = int64(o.fs.opt.NoHeadSize)
o.size = -1
o.modTime = timeUnset
if o.fs.opt.NoHeadTime.IsSet() {
o.modTime = time.Time(o.fs.opt.NoHeadTime)
}
o.contentType = fs.MimeType(ctx, o)
return nil
}
+4 -65
View File
@@ -314,54 +314,6 @@ func TestOpen(t *testing.T) {
}
}
func TestNoHeadMetadataDefaults(t *testing.T) {
tests := []struct {
name string
size string
modTime string
wantSize int64
wantModTime time.Time
}{
{
name: "defaults",
wantSize: -1,
wantModTime: timeUnset,
}, {
name: "zero size",
size: "0B",
wantSize: 0,
wantModTime: timeUnset,
}, {
name: "configured metadata",
size: "100B",
modTime: "2024-08-05T07:36:07Z",
wantSize: 100,
wantModTime: time.Date(2024, 8, 5, 7, 36, 7, 0, time.UTC),
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
m := prepareServer(t)
m.Set("no_head", "true")
if test.size != "" {
m.Set("no_head_size", test.size)
}
if test.modTime != "" {
m.Set("no_head_time", test.modTime)
}
f, err := NewFs(context.Background(), remoteName, "", m)
require.NoError(t, err)
o, err := f.NewObject(context.Background(), "four/under four.txt")
require.NoError(t, err)
assert.Equal(t, test.wantSize, o.Size())
assert.Equal(t, test.wantModTime, o.ModTime(context.Background()))
})
}
}
func TestMimeType(t *testing.T) {
f := prepare(t)
@@ -688,7 +640,7 @@ func TestRedirectRefusesHTTPSDowngrade(t *testing.T) {
// open makes a remote pointing at ts with or without headers
// and opens file.txt on it
open := func(t *testing.T, withHeaders, allowDowngrade bool) (io.ReadCloser, error) {
open := func(t *testing.T, withHeaders bool) (io.ReadCloser, error) {
configfile.Install()
m := configmap.Simple{
"type": "http",
@@ -700,7 +652,6 @@ func TestRedirectRefusesHTTPSDowngrade(t *testing.T) {
}
ctx, ci := fs.AddConfig(context.Background())
ci.InsecureSkipVerify = true
ci.AllowHTTPSDowngrade = allowDowngrade
f, err := NewFs(ctx, remoteName, "", m)
require.NoError(t, err)
@@ -712,25 +663,13 @@ func TestRedirectRefusesHTTPSDowngrade(t *testing.T) {
t.Run("WithHeaders", func(t *testing.T) {
// Headers could leak so the downgrade must be refused
_, err := open(t, true, false)
require.ErrorIs(t, err, rest.ErrHTTPSDowngrade)
})
t.Run("WithHeadersAllowDowngrade", func(t *testing.T) {
// Headers could leak so the downgrade is refused even when allowed
_, err := open(t, true, true)
_, err := open(t, true)
require.ErrorIs(t, err, rest.ErrHTTPSDowngrade)
})
t.Run("WithoutHeaders", func(t *testing.T) {
// The transport refuses downgrades by default
_, err := open(t, false, false)
require.ErrorIs(t, err, rest.ErrHTTPSDowngrade)
})
t.Run("WithoutHeadersAllowDowngrade", func(t *testing.T) {
// Nothing to leak so the downgrade is followed when allowed
fd, err := open(t, false, true)
// Nothing to leak so the downgrade is followed
fd, err := open(t, false)
require.NoError(t, err)
data, err := io.ReadAll(fd)
require.NoError(t, err)
+70 -65
View File
@@ -37,44 +37,44 @@ type FileList struct {
// File represents a file or folder in Huawei Drive
type File struct {
ID string `json:"id"`
FileName string `json:"fileName"`
OriginalFilename string `json:"originalFilename,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
Category string `json:"category"`
Size int64 `json:"size,omitempty"`
SHA256 string `json:"sha256,omitempty"`
CreatedTime time.Time `json:"createdTime"`
EditedTime time.Time `json:"editedTime"`
EditedByMeTime time.Time `json:"editedByMeTime"`
ParentFolder []string `json:"parentFolder,omitempty"`
Owners []User `json:"owners"`
LastEditor User `json:"lastEditor"`
Permissions []Permission `json:"permissions"`
PermissionIDs []string `json:"permissionIds"`
Capabilities FileCapabilities `json:"capabilities"`
OwnedByMe bool `json:"ownedByMe"`
EditedByMe bool `json:"editedByMe"`
ViewedByMe bool `json:"viewedByMe"`
HasShared bool `json:"hasShared"`
Recycled bool `json:"recycled"`
DirectlyRecycled bool `json:"directlyRecycled"`
Favorite bool `json:"favorite"`
ExistThumbnail bool `json:"existThumbnail"`
ThumbnailVersion int64 `json:"thumbnailVersion"`
IconDownloadLink string `json:"iconDownloadLink,omitempty"`
ContentDownloadLink string `json:"contentDownloadLink,omitempty"`
ContentVersion string `json:"contentVersion,omitempty"`
LastHistoryVersionID string `json:"lastHistoryVersionId,omitempty"`
OccupiedSpace int64 `json:"occupiedSpace,omitempty"`
Version int64 `json:"version"`
WritersHasSharePermission bool `json:"writersHasSharePermission"`
WriterHasCopyPermission bool `json:"writerHasCopyPermission"`
Containers []string `json:"containers"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
ContentExtras *ContentExtras `json:"contentExtras,omitempty"`
ID string `json:"id"`
FileName string `json:"fileName"`
OriginalFilename string `json:"originalFilename,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
Category string `json:"category"`
Size int64 `json:"size,omitempty"`
SHA256 string `json:"sha256,omitempty"`
CreatedTime time.Time `json:"createdTime"`
EditedTime time.Time `json:"editedTime"`
EditedByMeTime time.Time `json:"editedByMeTime,omitempty"`
ParentFolder []string `json:"parentFolder,omitempty"`
Owners []User `json:"owners"`
LastEditor User `json:"lastEditor"`
Permissions []Permission `json:"permissions"`
PermissionIDs []string `json:"permissionIds"`
Capabilities FileCapabilities `json:"capabilities"`
OwnedByMe bool `json:"ownedByMe"`
EditedByMe bool `json:"editedByMe"`
ViewedByMe bool `json:"viewedByMe"`
HasShared bool `json:"hasShared"`
Recycled bool `json:"recycled"`
DirectlyRecycled bool `json:"directlyRecycled"`
Favorite bool `json:"favorite"`
ExistThumbnail bool `json:"existThumbnail"`
ThumbnailVersion int64 `json:"thumbnailVersion"`
IconDownloadLink string `json:"iconDownloadLink,omitempty"`
ContentDownloadLink string `json:"contentDownloadLink,omitempty"`
ContentVersion string `json:"contentVersion,omitempty"`
LastHistoryVersionID string `json:"lastHistoryVersionId,omitempty"`
OccupiedSpace int64 `json:"occupiedSpace,omitempty"`
Version int64 `json:"version"`
WritersHasSharePermission bool `json:"writersHasSharePermission"`
WriterHasCopyPermission bool `json:"writerHasCopyPermission"`
Containers []string `json:"containers"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
ContentExtras *ContentExtras `json:"contentExtras,omitempty"`
}
// IsDir returns true if the file is a directory
@@ -125,41 +125,41 @@ type Thumbnail struct {
// CreateFolderRequest represents request to create a folder
type CreateFolderRequest struct {
FileName string `json:"fileName"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
ParentFolder []string `json:"parentFolder,omitempty"`
Favorite bool `json:"favorite,omitempty"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
AutoRename int `json:"autoRename,omitempty"`
FileName string `json:"fileName"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
ParentFolder []string `json:"parentFolder,omitempty"`
Favorite bool `json:"favorite,omitempty"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
AutoRename int `json:"autoRename,omitempty"`
}
// UpdateFileRequest represents request to update a file
type UpdateFileRequest struct {
FileName string `json:"fileName,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType,omitempty"`
Favorite *bool `json:"favorite,omitempty"`
Recycled *bool `json:"recycled,omitempty"`
OriginalFilename string `json:"originalFilename,omitempty"`
WriterHasCopyPermission *bool `json:"writerHasCopyPermission,omitempty"`
WritersHasSharePermission *bool `json:"writersHasSharePermission,omitempty"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
AddParentFolder []string `json:"addParentFolder,omitempty"`
RemoveParentFolder []string `json:"removeParentFolder,omitempty"`
CreatedTime *time.Time `json:"createdTime,omitempty"`
EditedTime *time.Time `json:"editedTime,omitempty"`
FileName string `json:"fileName,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType,omitempty"`
Favorite *bool `json:"favorite,omitempty"`
Recycled *bool `json:"recycled,omitempty"`
OriginalFilename string `json:"originalFilename,omitempty"`
WriterHasCopyPermission *bool `json:"writerHasCopyPermission,omitempty"`
WritersHasSharePermission *bool `json:"writersHasSharePermission,omitempty"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
AddParentFolder []string `json:"addParentFolder,omitempty"`
RemoveParentFolder []string `json:"removeParentFolder,omitempty"`
CreatedTime *time.Time `json:"createdTime,omitempty"`
EditedTime *time.Time `json:"editedTime,omitempty"`
}
// CopyFileRequest represents request to copy a file
type CopyFileRequest struct {
FileName string `json:"fileName,omitempty"`
MimeType string `json:"mimeType,omitempty"`
ParentFolder []string `json:"parentFolder,omitempty"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
FileName string `json:"fileName,omitempty"`
MimeType string `json:"mimeType,omitempty"`
ParentFolder []string `json:"parentFolder,omitempty"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
}
// ResumeUploadInitResponse represents response to resume upload init
@@ -224,6 +224,11 @@ var GlobalDomains = map[string]bool{
"drive.cloud.hicloud.com": true,
}
// BoolPtr returns a pointer to a bool value (helper for UpdateFileRequest)
func BoolPtr(b bool) *bool {
return &b
}
// StartCursor represents response to Changes.getStartCursor endpoint
type StartCursor struct {
Category string `json:"category"` // "drive#startCursor"
+34 -59
View File
@@ -33,7 +33,6 @@ import (
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/oauth2"
@@ -50,10 +49,6 @@ const (
defaultChunkSize = 8 * fs.Mebi
)
// jsonContentType is sent with JSON request bodies as the API doesn't
// decode non-ASCII file names in them as UTF-8 unless told the charset.
const jsonContentType = "application/json; charset=UTF-8"
// OAuth2 configuration
var oauthConfig = &oauthutil.Config{
Scopes: []string{
@@ -551,7 +546,6 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
var info *api.File
@@ -1020,7 +1014,6 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
copyReq := api.CopyFileRequest{
@@ -1038,7 +1031,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
fs.Debugf(f, "Copy: failed to get metadata options: %v", err)
}
if len(mergedMeta) > 0 {
copyReq.Properties = make(map[string]any)
copyReq.Properties = make(map[string]interface{})
for key, value := range mergedMeta {
switch key {
case "content-type", "sha256", "btime", "mtime", "utime",
@@ -1205,10 +1198,9 @@ func (f *Fs) updateFile(ctx context.Context, fileID string, parameters url.Value
}
parameters.Set("fields", "*")
opts := rest.Opts{
Method: "PATCH",
Path: "/files/" + fileID,
Parameters: parameters,
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
Method: "PATCH",
Path: "/files/" + fileID,
Parameters: parameters,
}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(ctx, &opts, req, &info)
@@ -1678,7 +1670,6 @@ func (o *Object) uploadEmpty(ctx context.Context, leaf, directoryID string, src
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
var info *api.File
@@ -1730,7 +1721,7 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
}
// Create metadata
metadata := map[string]any{
metadata := map[string]interface{}{
"fileName": leaf,
"mimeType": mimeType,
}
@@ -1750,10 +1741,9 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
metadata["autoRename"] = 3
}
// Build the multipart body in pooled memory so it can be re-sent on retry
rw := pool.NewRW(pool.Global())
defer fs.CheckClose(rw, &err)
writer := multipart.NewWriter(rw)
// Create multipart form
var buf bytes.Buffer
writer := multipart.NewWriter(&buf)
// Add metadata part - use exact format from Huawei docs
metadataWriter, err := writer.CreatePart(textproto.MIMEHeader{
@@ -1799,17 +1789,15 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
return fmt.Errorf("failed to close multipart writer: %w", err)
}
bodySize := rw.Size()
bodyBytes := buf.Bytes()
opts := rest.Opts{
Method: "POST",
Parameters: url.Values{
"uploadType": []string{"multipart"},
"fields": []string{"*"},
},
ContentType: fmt.Sprintf("multipart/related; boundary=%s", writer.Boundary()),
ContentLength: &bodySize,
RootURL: o.fs.uploadURL,
Body: rw,
ContentType: fmt.Sprintf("multipart/related; boundary=%s", writer.Boundary()),
RootURL: o.fs.uploadURL,
}
if o.id != "" {
@@ -1823,10 +1811,7 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
var info *api.File
err = o.fs.pacer.Call(func() (bool, error) {
_, err := rw.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
opts.Body = bytes.NewReader(bodyBytes)
resp, err := srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(ctx, resp, err)
})
@@ -1854,7 +1839,6 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{
"Content-Type": jsonContentType,
"X-Upload-Content-Length": strconv.FormatInt(size, 10),
},
RootURL: o.fs.uploadURL,
@@ -1888,7 +1872,7 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
}
// Prepare metadata for resumable upload initialization
metadata := map[string]any{
metadata := map[string]interface{}{
"fileName": leaf,
}
@@ -1921,26 +1905,25 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
}
// Now upload the content in chunks
// Note 256k is the minimum chunk size and 64M the maximum
// single upload size according to the Huawei Drive API
chunkSize := min(max(int64(o.fs.opt.ChunkSize), 256*1024), 64*1024*1024)
chunkSize := int64(o.fs.opt.ChunkSize)
if chunkSize < 256*1024 {
chunkSize = 256 * 1024 // Minimum chunk size according to Huawei Drive API
}
if chunkSize > 64*1024*1024 {
chunkSize = 64 * 1024 * 1024 // Maximum single upload size
}
buf := make([]byte, chunkSize)
var offset int64
for offset < size {
// Buffer the chunk in pooled memory so it can be re-sent on retry
rw := pool.NewRW(pool.Global())
n, err := io.CopyN(rw, in, min(size-offset, chunkSize))
if err != nil {
// io.EOF here means the source ran out before the declared size
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
_ = rw.Close()
n, err := io.ReadFull(in, buf)
if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF {
return fmt.Errorf("failed to read chunk at offset %d: %w", offset, err)
}
end := offset + n - 1
chunk := buf[:n]
end := offset + int64(n) - 1
// Upload this chunk using the original client with OAuth authentication
chunkOpts := rest.Opts{
@@ -1948,20 +1931,17 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
RootURL: location,
Path: "", // Empty path since RootURL contains the full URL
ExtraHeaders: map[string]string{
"Content-Range": fmt.Sprintf("bytes %d-%d/%d", offset, end, size),
"Content-Type": mimeType,
"Content-Range": fmt.Sprintf("bytes %d-%d/%d", offset, end, size),
"Content-Length": strconv.Itoa(n),
"Content-Type": mimeType,
},
ContentLength: &n,
Body: rw,
}
var info *api.File
var chunkResp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
_, err := rw.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
chunkOpts.Body = bytes.NewReader(chunk)
var err error
chunkResp, err = srv.CallJSON(ctx, &chunkOpts, nil, &info)
// 308 means continue uploading (incomplete)
if chunkResp != nil && chunkResp.StatusCode == 308 {
@@ -1975,15 +1955,11 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
}
return shouldRetry(ctx, chunkResp, err)
})
closeErr := rw.Close()
if err != nil {
return fmt.Errorf("failed to upload chunk at offset %d: %w", offset, err)
}
if closeErr != nil {
return closeErr
}
offset += n
offset += int64(n)
// If we got file info back, we're done
if info != nil && info.ID != "" {
@@ -2114,8 +2090,8 @@ func (o *Object) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
// Prepare the update request payload
updateReq := api.UpdateFileRequest{
Properties: make(map[string]any),
AppSettings: make(map[string]any),
Properties: make(map[string]interface{}),
AppSettings: make(map[string]interface{}),
}
// Process metadata and separate into properties and app settings
@@ -2126,7 +2102,7 @@ func (o *Object) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
updateReq.Description = value
case "favorite":
if favorite, err := strconv.ParseBool(value); err == nil {
updateReq.Favorite = new(favorite)
updateReq.Favorite = api.BoolPtr(favorite)
}
case "content-type":
// Allow setting/overriding MIME type
@@ -2154,7 +2130,6 @@ func (o *Object) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
var info *api.File
-34
View File
@@ -19,49 +19,15 @@ import (
"github.com/rclone/rclone/lib/encoder"
)
// Limits on the chunk size accepted by the resumable upload API
const (
minChunkSize = 256 * fs.Kibi
maxChunkSize = 64 * fs.Mebi
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestHuaweiDrive:",
NilObject: (*Object)(nil),
SkipBadWindowsCharacters: true,
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: minChunkSize,
MaxChunkSize: maxChunkSize,
CeilChunkSize: fstests.NextPowerOfTwo,
},
})
}
// SetUploadChunkSize changes the configured chunk size, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.ChunkSize = f.opt.ChunkSize, cs
return old, nil
}
// SetUploadCutoff changes the configured upload cutoff, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadCutoff(cutoff fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.UploadCutoff = f.opt.UploadCutoff, cutoff
return old, nil
}
var (
_ fstests.SetUploadChunkSizer = (*Fs)(nil)
_ fstests.SetUploadCutoffer = (*Fs)(nil)
)
// TestNewFs tests the NewFs constructor
func TestNewFs(t *testing.T) {
ctx := context.Background()
+4 -3
View File
@@ -8,7 +8,6 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"net/http"
"os"
"path"
@@ -1618,7 +1617,9 @@ func (ps *PhotosService) GetLibraryAlbumCounts(ctx context.Context) (map[string]
if err := ps.requestForArea(ctx, area, "internal/records/query/batch", map[string]any{"batch": batch}, &response); err != nil {
return nil, fmt.Errorf("failed to get library album counts: %w", err)
}
maps.Copy(counts, response.toCounts(order))
for k, v := range response.toCounts(order) {
counts[k] = v
}
}
return counts, nil
}
@@ -2031,7 +2032,7 @@ func (album *Album) fetchPhotosParallel(ctx context.Context, totalPhotos int64)
sem := make(chan struct{}, workers)
var wg sync.WaitGroup
for i := range numPartitions {
for i := 0; i < numPartitions; i++ {
wg.Add(1)
go func(idx int) {
defer wg.Done()
+2 -2
View File
@@ -1191,7 +1191,7 @@ func TestFlushCaches_NoPendingDeltaRace(t *testing.T) {
done := make(chan struct{})
go func() {
defer close(done)
for range 100 {
for i := 0; i < 100; i++ {
lib.deltaMu.Lock()
lib.pendingDelta = &deltaPayload{
records: []json.RawMessage{json.RawMessage(`{"recordName":"m1","recordType":"CPLMaster"}`)},
@@ -1201,7 +1201,7 @@ func TestFlushCaches_NoPendingDeltaRace(t *testing.T) {
lib.applyPendingDelta(context.Background())
}
}()
for range 100 {
for i := 0; i < 100; i++ {
ps.FlushCaches()
// Re-add the library since FlushCaches clears it
ps.mu.Lock()
+5 -3
View File
@@ -718,7 +718,7 @@ func (s *Session) GetAuthState(ctx context.Context) (*AuthStateResponse, error)
Path: "",
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
RootURL: authEndpoint,
ContentLength: new(int64(0)),
ContentLength: int64Ptr(0),
}
// Use srv.Call directly to capture the raw response body for debugging
resp, err := s.srv.Call(ctx, &opts)
@@ -828,7 +828,7 @@ func (s *Session) TrustSession(ctx context.Context) error {
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
RootURL: authEndpoint,
NoResponse: true,
ContentLength: new(int64(0)),
ContentLength: int64Ptr(0),
}
_, err := s.Request(ctx, opts, nil, nil)
@@ -846,7 +846,7 @@ func (s *Session) ValidateSession(ctx context.Context) error {
Path: "/validate",
ExtraHeaders: s.GetHeaders(map[string]string{}),
RootURL: setupEndpoint,
ContentLength: new(int64(0)),
ContentLength: int64Ptr(0),
}
_, err := s.Request(ctx, opts, nil, &s.AccountInfo)
if err != nil {
@@ -904,6 +904,8 @@ func GetCommonHeaders(overwrite map[string]string) map[string]string {
return headers
}
func int64Ptr(v int64) *int64 { return &v }
// NewSession creates a new Session instance with default values
func NewSession() *Session {
session := &Session{
-1
View File
@@ -694,7 +694,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
}
f.features = (&fs.Features{
CaseInsensitive: true, // names are matched with EqualFold, see findLeafItem
CanHaveEmptyDirectories: true,
PartialUploads: false,
}).Fill(ctx, f)
+6 -2
View File
@@ -376,7 +376,8 @@ func (f *PhotosFs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut
return "", false, nil
}
if libraryName, ok := strings.CutPrefix(pathID, "lib:"); ok {
if strings.HasPrefix(pathID, "lib:") {
libraryName := strings.TrimPrefix(pathID, "lib:")
libraries, err := photosService.GetLibraries(ctx)
if err != nil {
return "", false, err
@@ -735,7 +736,10 @@ func (f *PhotosFs) ListR(ctx context.Context, dir string, callback fs.ListRCallb
}
close(jobCh)
workers := min(len(jobs), fs.GetConfig(ctx).Checkers)
workers := fs.GetConfig(ctx).Checkers
if len(jobs) < workers {
workers = len(jobs)
}
errs := make(chan error, workers)
for range workers {
-2
View File
@@ -77,8 +77,6 @@ func (ik *ImageKit) Upload(ctx context.Context, file io.Reader, param UploadPara
if err != nil {
return nil, nil, fmt.Errorf("failed to make multipart upload: %w", err)
}
// Stop the form writer reading file if the request fails before sending it all
defer func() { _ = formReader.Close() }()
opts := rest.Opts{
Method: "POST",
@@ -634,27 +634,6 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (_ fs.Objec
"x-archive-filemeta-rclone-update-track": updateTracker,
}
mdata, err := fs.GetMetadataOptions(ctx, f, src, fs.MetadataAsOpenOptions(ctx))
if err != nil {
return nil, err
}
if mdata != nil {
// IA ignores file metadata headers with the COPY directive.
headers["x-amz-metadata-directive"] = "REPLACE"
}
for key, value := range mdata {
key = strings.ToLower(key)
if key == "mtime" {
key = "rclone-mtime"
} else if strings.HasPrefix(key, "rclone-") {
// Keep the new copy tracker and mapped mtime, not the source internals.
continue
} else if _, readOnly := roMetadataKey[key]; readOnly {
continue
}
headers["x-archive-filemeta-"+key] = value
}
// make a PUT request at (IAS3)/:item/:path without body
var resp *http.Response
opts := rest.Opts{
+6 -3
View File
@@ -54,7 +54,8 @@ func (f *Fs) shouldRetry(ctx context.Context, err error) (bool, error) {
if fserrors.ContextError(ctx, &err) {
return false, err
}
if httpErr, ok := errors.AsType[*sdkerrors.HTTPError](err); ok {
var httpErr *sdkerrors.HTTPError
if errors.As(err, &httpErr) {
switch httpErr.StatusCode() {
case 401:
if !f.authFailed {
@@ -1208,7 +1209,8 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
err := o.f.pacer.Call(func() (bool, error) {
err := files.DeleteFile(ctx, o.f.cfg, backupUUID)
if err != nil {
if httpErr, ok := errors.AsType[*sdkerrors.HTTPError](err); ok {
var httpErr *sdkerrors.HTTPError
if errors.As(err, &httpErr) {
// Treat 404 (Not Found) and 204 (No Content) as success
switch httpErr.StatusCode() {
case 404, 204:
@@ -1274,7 +1276,8 @@ func isEmptyFileLimitError(err error) bool {
// fileTooLargeError extracts the SDK's FileTooLargeError from a wrapped error
// chain, returning it (or nil) so callers can branch on the size limit.
func fileTooLargeError(err error) *sdkerrors.FileTooLargeError {
if tooLarge, ok := errors.AsType[*sdkerrors.FileTooLargeError](err); ok {
var tooLarge *sdkerrors.FileTooLargeError
if errors.As(err, &tooLarge) {
return tooLarge
}
return nil
+13 -21
View File
@@ -34,7 +34,6 @@ import (
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/list"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
@@ -86,8 +85,8 @@ func getServices() []service {
{"jottacloud", "Jottacloud", "id.jottacloud.com", "jottacloud", "desktop", []string{"openid", "jotta-default", "offline_access"}}, // Chose client id "desktop" here, will be identified as "Jottacloud for Desktop" in "My logged in devices", but could have used "jottacli" here as well.
{"elgiganten_dk", "Elgiganten Cloud (Denmark)", "cloud.elgiganten.dk", "elgiganten", "desktop", []string{"openid", "jotta-default", "offline_access"}},
{"elgiganten_se", "Elgiganten Cloud (Sweden)", "cloud.elgiganten.se", "elgiganten", "desktop", []string{"openid", "jotta-default", "offline_access"}},
{"elkjop", "Elkjøp Cloud (Norway)", "cloud.elkjop.no", "elkjop", "desktop", []string{"openid", "offline_access"}},
{"elko", "ELKO Cloud (Iceland)", "cloud.elko.is", "elko", "desktop", []string{"openid", "offline_access"}},
{"elkjop", "Elkjøp Cloud (Norway)", "cloud.elkjop.no", "elkjop", "desktop", []string{"openid", "jotta-default", "offline_access"}},
{"elko", "ELKO Cloud (Iceland)", "cloud.elko.is", "elko", "desktop", []string{"openid", "jotta-default", "offline_access"}},
{"gigantti", "Gigantti Cloud (Finland)", "cloud.gigantti.fi", "gigantti", "desktop", []string{"openid", "jotta-default", "offline_access"}},
{"letsgo", "Let's Go Cloud (Germany)", "letsgo.jotta.cloud", "letsgo", "desktop-win", []string{"openid", "offline_access"}},
{"mediamarkt", "MediaMarkt Cloud (Multiregional)", "mediamarkt.jottacloud.com", "mediamarkt", "desktop", []string{"openid", "jotta-default", "offline_access"}},
@@ -258,7 +257,6 @@ service, for the official service on https://www.jottacloud.com/web/secure.`)
}
m.Set("configVersion", fmt.Sprint(configVersion))
m.Set(configClientID, service.clientID)
m.Set(configClientSecret, "")
m.Set(configTokenURL, wellKnown.TokenEndpoint)
return oauthutil.ConfigOut("choose_device", &oauthutil.Options{
OAuth2Config: &oauthutil.Config{
@@ -1501,9 +1499,9 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
info, err := f.copyOrMove(ctx, "cp", srcObj.filePath(), remote)
if err == nil {
createTime := srcObj.createTime
var createTime time.Time
var createTimeMeta bool
modTime := srcObj.modTime
var modTime time.Time
var modTimeMeta bool
if meta != nil {
createTime, createTimeMeta = srcObj.parseFsMetadataTime(meta, "btime")
@@ -1519,11 +1517,6 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
// Workaround necessary when destination was a trashed file, to avoid the copied file also being in trash (bug in api?)
fs.Debugf(src, "Server-side copied to trashed destination, restoring")
info, err = f.createOrUpdate(ctx, remote, createTime, modTime, info.Size, info.MD5)
// The restored file keeps the name it was trashed with, which may differ in case
if err == nil && info != nil && info.Name != f.opt.Enc.FromStandardName(path.Base(remote)) {
fs.Debugf(src, "Renaming restored %q to %q", info.Name, path.Base(remote))
info, err = f.copyOrMove(ctx, "mv", f.filePath(remote), remote)
}
} else if createTimeMeta || modTimeMeta {
info, err = f.createOrUpdate(ctx, remote, createTime, modTime, info.Size, info.MD5)
}
@@ -1931,9 +1924,8 @@ func readMD5(in io.Reader, size, threshold int64) (md5sum string, out io.Reader,
// nothing to clean up by default
cleanup = func() {}
// don't cache small files on disk to reduce wear of the disk, but
// spool unknown sized streams there as they can't be bounded in memory
if size > threshold || size < 0 {
// don't cache small files on disk to reduce wear of the disk
if size > threshold {
var tempFile *os.File
// create the cache file
@@ -1963,15 +1955,15 @@ func readMD5(in io.Reader, size, threshold int64) (md5sum string, out io.Reader,
// replace the already read source with a reader of our cached file
out = tempFile
} else {
// that's a small file, just read it into memory from the global pool
rw := multipart.NewRW()
cleanup = func() {
_ = rw.Close()
}
if _, err = io.Copy(rw, teeReader); err != nil {
// that's a small file, just read it into memory
var inData []byte
inData, err = io.ReadAll(teeReader)
if err != nil {
return
}
out = rw
// set the reader to our read memory block
out = bytes.NewReader(inData)
}
return hex.EncodeToString(md5Hasher.Sum(nil)), out, cleanup, nil
}
+1 -34
View File
@@ -5,14 +5,12 @@ import (
"crypto/md5"
"fmt"
"io"
"os"
"testing"
"time"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/random"
"github.com/rclone/rclone/lib/readers"
"github.com/stretchr/testify/assert"
@@ -30,16 +28,11 @@ func TestReadMD5(t *testing.T) {
wantMD5 := fmt.Sprintf("%x", hasher.Sum(nil))
for _, threshold := range []int64{512, 1024, 10 * 1024, 20 * 1024} {
t.Run(fmt.Sprintf("%d", threshold), func(t *testing.T) {
inUse := pool.Global().InUse()
in := readers.NewPatternReader(size)
gotMD5, out, cleanup, err := readMD5(in, size, threshold)
defer cleanup()
require.NoError(t, err)
assert.Equal(t, wantMD5, gotMD5)
if size > threshold {
assert.IsType(t, (*os.File)(nil), out, "big files should be spooled to disk")
} else {
assert.IsType(t, (*pool.RW)(nil), out, "small files should be buffered in the pool")
}
// check md5hash of out
hasher := md5.New()
@@ -48,36 +41,10 @@ func TestReadMD5(t *testing.T) {
assert.Equal(t, n, size)
outMD5 := fmt.Sprintf("%x", hasher.Sum(nil))
assert.Equal(t, wantMD5, outMD5)
// check cleanup returns the buffer to the pool
cleanup()
assert.Equal(t, inUse, pool.Global().InUse(), "pool buffers leaked")
})
}
})
}
// Unknown sized streams can't be bounded in memory so must be spooled to disk
t.Run("UnknownSize", func(t *testing.T) {
const size = 1024
inUse := pool.Global().InUse()
hasher := md5.New()
_, err := io.Copy(hasher, readers.NewPatternReader(size))
require.NoError(t, err)
wantMD5 := fmt.Sprintf("%x", hasher.Sum(nil))
gotMD5, out, cleanup, err := readMD5(readers.NewPatternReader(size), -1, 10*size)
require.NoError(t, err)
assert.Equal(t, wantMD5, gotMD5)
assert.IsType(t, (*os.File)(nil), out)
hasher.Reset()
n, err := io.Copy(hasher, out)
require.NoError(t, err)
assert.Equal(t, int64(size), n)
assert.Equal(t, wantMD5, fmt.Sprintf("%x", hasher.Sum(nil)))
cleanup()
assert.Equal(t, inUse, pool.Global().InUse())
})
}
func (f *Fs) InternalTestMetadata(t *testing.T) {
-235
View File
@@ -1,235 +0,0 @@
// Package api has type definitions for kDrive
//
// Converted from the API docs with help from https://mholt.github.io/json-to-go/
package api
import (
"fmt"
"strconv"
"time"
)
// Time represents unix timestamp date and time information for the
// kDrive API
type Time time.Time
// UnmarshalJSON turns JSON into a Time
func (t *Time) UnmarshalJSON(data []byte) error {
if string(data) == "null" {
return nil
}
timestamp, err := strconv.ParseInt(string(data), 10, 64)
if err != nil {
return err
}
newT := time.Unix(timestamp, 0)
*t = Time(newT)
return nil
}
// ResultStatus return error details from kdrive when things go wrong
//
// If Status is "success" then everything is OK
type ResultStatus struct {
Status string `json:"result"`
ErrorDetail struct {
Result string `json:"code"`
ErrorString string `json:"description"`
Context struct {
Values []string `json:"values"`
} `json:"context"`
Errors []struct {
Result string `json:"code"`
ErrorString string `json:"description"`
} `json:"errors"`
} `json:"error"`
}
// Error returns a string for the error and satisfies the error interface
func (e *ResultStatus) Error() string {
var details string
for i := range e.ErrorDetail.Errors {
details += "|" + e.ErrorDetail.Errors[i].Result
}
return fmt.Sprintf("kDrive error: %s (%s: %s %s)", e.Status, e.ErrorDetail.Result, e.ErrorDetail.ErrorString, details)
}
// IsError returns true if there is an error
func (e *ResultStatus) IsError() bool {
return e.Status != "success"
}
// Update returns err directly if it was != nil, otherwise it returns
// an Error or nil if no error was detected
func (e *ResultStatus) Update(err error) error {
if err != nil {
return err
}
if e.IsError() {
return e
}
return nil
}
// Check ResultStatus satisfies the error interface
var _ error = (*ResultStatus)(nil)
// Item describes a folder or a file as returned by Get Folder Items and others
type Item struct {
ID int `json:"id"`
Name string `json:"name"`
Type string `json:"type"`
FullPath string `json:"path"`
Status string `json:"status"`
Hash string `json:"hash"`
Size int64 `json:"size"`
Visibility string `json:"visibility"`
DriveID int `json:"drive_id"`
Depth int `json:"depth"`
CreatedBy int `json:"created_by"`
CreatedAt Time `json:"created_at"`
AddedAt int `json:"added_at"`
LastModifiedAt Time `json:"last_modified_at"`
LastModifiedBy int `json:"last_modified_by"`
RevisedAt int `json:"revised_at"`
UpdatedAt int `json:"updated_at"`
MimeType string `json:"mime_type"`
ParentID int `json:"parent_id"`
Color string `json:"color"`
}
// ItemResult is returned when an item is queried
type ItemResult struct {
ResultStatus
Data Item `json:"data"`
}
// SearchResult is returned when a list of items is requested
type SearchResult struct {
ResultStatus
Data []Item `json:"data"`
Cursor string `json:"cursor"`
HasMore bool `json:"has_more"`
ResponseAt int `json:"response_at"`
}
// ModTime returns the modification time of the item
func (i *Item) ModTime() (t time.Time) {
t = time.Time(i.LastModifiedAt)
if t.IsZero() {
t = time.Time(i.CreatedAt)
}
return t
}
// CancelResource is a kdrive resource that can be cancelled after some action
type CancelResource struct {
CancelID string `json:"cancel_id"`
ValidUntil int `json:"valid_until"`
}
// CancellableResponse is returned from kdrive when an action can be cancelled afterwards
type CancellableResponse struct {
ResultStatus
Data CancelResource `json:"data"`
}
// CreateDirResult is returned from kdrive after a call to MkDir
type CreateDirResult struct {
ResultStatus
Data Item `json:"data"`
}
// FileCopyResponse is returned from kdrive after a call to Copy
type FileCopyResponse struct {
ResultStatus
Data Item `json:"data"`
}
// UploadFileResponse is returned from kdrive after a call to Upload
type UploadFileResponse struct {
ResultStatus
Data Item `json:"data"`
}
// ChecksumFileResult is returned from kdrive after a call to Hash
type ChecksumFileResult struct {
ResultStatus
Data struct {
Hash string `json:"hash"`
} `json:"data"`
}
// PubLinkResult is returned from kdrive after a call to public link API
type PubLinkResult struct {
ResultStatus
Data struct {
URL string `json:"url"`
FileID int `json:"file_id"`
Right string `json:"right"`
ValidUntil int `json:"valid_until"`
CreatedBy int `json:"created_by"`
CreatedAt int `json:"created_at"`
UpdatedAt int `json:"updated_at"`
Capabilities struct {
CanEdit bool `json:"can_edit"`
CanSeeStats bool `json:"can_see_stats"`
CanSeeInfo bool `json:"can_see_info"`
CanDownload bool `json:"can_download"`
CanComment bool `json:"can_comment"`
CanRequestAccess bool `json:"can_request_access"`
} `json:"capabilities"`
AccessBlocked bool `json:"access_blocked"`
} `json:"data"`
}
// PubLinkUpdateResult is returned from kdrive after a updating a public link
type PubLinkUpdateResult struct {
ResultStatus
Data bool `json:"data"`
}
// QuotaInfo is return from kdrive after a call get drive info
type QuotaInfo struct {
ResultStatus
Data struct {
Size int64 `json:"size"`
UsedSize int64 `json:"used_size"`
} `json:"data"`
}
// SessionStartResponse is used to start a multichunk upload session
type SessionStartResponse struct {
ResultStatus
Data struct {
Token string `json:"token"`
UploadURL string `json:"upload_url"` // URL pour uploader les chunks
} `json:"data"`
}
// ChunkUploadResponse is returned by kdrive after a chunk upload
type ChunkUploadResponse struct {
ResultStatus
Data struct {
Hash string `json:"hash,omitempty"` // Only present when requested with with:'hash'
} `json:"data"`
}
// SessionFinishResponse is used to end a multichunk upload session
type SessionFinishResponse struct {
ResultStatus
Data struct {
Token string `json:"token"`
File Item `json:"file"`
Result bool `json:"result"`
Message string `json:"message"`
} `json:"data"`
}
// SessionCancelResponse is used to cancel a multichunk upload session
type SessionCancelResponse struct {
ResultStatus
Data bool `json:"data"`
}
File diff suppressed because it is too large. Load diff
-551
View File
@@ -1,551 +0,0 @@
// Unit tests for the kdrive backend
package kdrive
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"github.com/rclone/rclone/backend/kdrive/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestCheckChunkSize checks chunk_size validation against the API limits
func TestCheckChunkSize(t *testing.T) {
for _, test := range []struct {
in fs.SizeSuffix
wantErr string
}{
{in: 20 * fs.Mebi, wantErr: ""}, // default
{in: 1000 * 1000, wantErr: ""}, // API minimum
{in: 1000 * 1000 * 1000, wantErr: ""}, // API maximum
{in: 0, wantErr: "too small"}, // panics without validation
{in: 999 * 1000, wantErr: "too small"}, // below API minimum
{in: 1000*1000*1000 + 1, wantErr: "too large"}, // above API maximum
{in: 2 * 1000 * 1000 * 1000, wantErr: "too large"}, // way above API maximum
} {
t.Run(test.in.String(), func(t *testing.T) {
err := checkChunkSize(test.in)
if test.wantErr == "" {
assert.NoError(t, err)
} else {
require.Error(t, err)
assert.Contains(t, err.Error(), test.wantErr)
}
})
}
}
// TestOpenChunkWriterClampsCallerChunkSize checks that a caller
// supplied chunk size hint below the API minimum is raised to it
func TestOpenChunkWriterClampsCallerChunkSize(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
_, _ = fmt.Fprintln(w, `{"result":"success","data":{"token":"tok","upload_url":"http://unused"}}`)
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
f.dirCache = dircache.New("", "1", f)
src := &mockObjectInfo{remote: "file.txt", size: 4, modTime: time.Unix(1700000000, 0)}
info, _, err := f.OpenChunkWriter(context.Background(), "file.txt", src, &fs.ChunkOption{ChunkSize: 512 * 1024})
require.NoError(t, err)
assert.Equal(t, int64(minChunkSize), info.ChunkSize, "a chunk size hint below the API minimum must be raised to it")
}
// TestObjectSetMetaDataResetsStaleHash checks that setMetaData always
// replaces the hash (a chunked upload finish response carries none)
func TestObjectSetMetaDataResetsStaleHash(t *testing.T) {
for _, test := range []struct {
name string
staleHash string
staleNested bool
newHash string
wantHash string
wantIsNested bool
wantHashKnown bool
}{
{
name: "empty hash clears stale plain hash",
staleHash: "deadbeef",
staleNested: false,
newHash: "",
wantHash: "",
wantIsNested: false,
wantHashKnown: false,
},
{
name: "empty hash clears stale nested hash",
staleHash: "deadbeef",
staleNested: true,
newHash: "",
wantHash: "",
wantIsNested: false,
wantHashKnown: false,
},
{
name: "plain hash is set",
staleHash: "deadbeef",
staleNested: false,
newHash: "xxh3:12345678",
wantHash: "12345678",
wantIsNested: false,
wantHashKnown: true,
},
{
name: "nested hash is set",
staleHash: "deadbeef",
staleNested: false,
newHash: "N:xxh3:87654321",
wantHash: "87654321",
wantIsNested: true,
wantHashKnown: true,
},
} {
t.Run(test.name, func(t *testing.T) {
o := &Object{
remote: "file.txt",
hasMetaData: true,
xxh3: test.staleHash,
nestedHash: test.staleNested,
}
info := &api.Item{
ID: 42,
Name: "file.txt",
Type: "file",
Hash: test.newHash,
}
err := o.setMetaData(info)
require.NoError(t, err)
assert.Equal(t, test.wantHash, o.xxh3, "hash after setMetaData")
assert.Equal(t, test.wantIsNested, o.nestedHash, "nested flag after setMetaData")
assert.True(t, o.hasMetaData)
// Hash() must reflect the new state. Only checked when a
// hash is known: with no hash Hash() would query the API.
// Nested hashes are not comparable with plain XXH3, so
// Hash() returns "" for them.
if test.wantHashKnown {
got, gotErr := o.Hash(context.Background(), hash.XXH3)
require.NoError(t, gotErr)
if test.wantIsNested {
assert.Equal(t, "", got, "nested hash must not be returned as XXH3")
} else {
assert.Equal(t, test.wantHash, got)
}
}
})
}
}
// newTestFs creates an Fs wired to a test server with a plain client
// (no OAuth) and a fast pacer
func newTestFs(t *testing.T, endpoint string) *Fs {
t.Helper()
return &Fs{
name: "TestKdrive",
root: "",
opt: Options{DriveID: "1234", Endpoint: endpoint},
srv: rest.NewClient(&http.Client{}).SetRoot(endpoint),
pacer: fs.NewPacer(context.Background(),
pacer.NewDefault(pacer.MinSleep(time.Millisecond), pacer.MaxSleep(2*time.Millisecond), pacer.DecayConstant(2))),
}
}
// TestWriteChunkDetectsResultError checks that a 200 with result:error
// is reported as an error
func TestWriteChunkDetectsResultError(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"chunk_error","description":"chunk upload failed"}}`)
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
u := &uploadSession{
f: f,
token: "tok",
uploadURL: ts.URL,
fileSize: 4,
chunkSize: 4,
chunkHashes: make(map[int64]string),
}
_, err := u.WriteChunk(context.Background(), 0, bytes.NewReader([]byte("test")))
require.Error(t, err, "result:error with HTTP 200 must not be treated as success")
assert.Contains(t, err.Error(), "chunk upload failed")
}
// TestWriteChunkRetriesAndRewinds checks that transient failures are
// retried with the chunk re-sent from the start of the reader
func TestWriteChunkRetriesAndRewinds(t *testing.T) {
var mu sync.Mutex
var attempts int
var bodies []string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
mu.Lock()
attempts++
bodies = append(bodies, string(body))
current := attempts
mu.Unlock()
w.Header().Set("Content-Type", "application/json")
if current == 1 {
w.WriteHeader(http.StatusInternalServerError)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"internal_error","description":"transient"}}`)
return
}
_, _ = fmt.Fprintln(w, `{"result":"success","data":{"chunk_number":1,"status":"OK"}}`)
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
u := &uploadSession{
f: f,
token: "tok",
uploadURL: ts.URL,
fileSize: 4,
chunkSize: 4,
chunkHashes: make(map[int64]string),
}
written, err := u.WriteChunk(context.Background(), 0, bytes.NewReader([]byte("test")))
require.NoError(t, err)
assert.Equal(t, int64(4), written)
mu.Lock()
defer mu.Unlock()
require.Len(t, bodies, 2, "chunk upload must be retried once after a 5xx")
assert.Equal(t, "test", bodies[0], "first attempt sends the chunk")
assert.Equal(t, "test", bodies[1], "retry must re-send the chunk from the start of the reader")
}
// kdriveMockItem returns a JSON body for a single item result
func kdriveMockItem(id int, name string) string {
return fmt.Sprintf(`{"result":"success","data":{"id":%d,"name":%q,"type":"file","path":%q,"last_modified_at":1700000000}}`, id, name, "/"+name)
}
// mockObjectInfo is a minimal fs.ObjectInfo for testing
type mockObjectInfo struct {
remote string
size int64
modTime time.Time
hash string
}
func (m *mockObjectInfo) Fs() fs.Info { return nil }
func (m *mockObjectInfo) String() string { return m.remote }
func (m *mockObjectInfo) Remote() string { return m.remote }
func (m *mockObjectInfo) Size() int64 { return m.size }
func (m *mockObjectInfo) ModTime(_ context.Context) time.Time { return m.modTime }
func (m *mockObjectInfo) Storable() bool { return true }
func (m *mockObjectInfo) MimeType(_ context.Context) string { return "application/octet-stream" }
func (m *mockObjectInfo) Hash(_ context.Context, t hash.Type) (string, error) {
if m.hash == "" || t != hash.XXH3 {
return "", hash.ErrUnsupported
}
return m.hash, nil
}
// TestUpdateDirectNoRetryOnServerError checks that a failed single
// part upload is not retried here: the input must not be buffered, the
// higher layers retry the whole transfer
func TestUpdateDirectNoRetryOnServerError(t *testing.T) {
var attempts int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&attempts, 1)
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
w.WriteHeader(http.StatusInternalServerError)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"internal_error","description":"transient"}}`)
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
o := &Object{fs: f, remote: "file.txt"}
src := &mockObjectInfo{remote: "file.txt", size: 4, modTime: time.Unix(1700000000, 0), hash: "12345678"}
err := o.updateDirect(context.Background(), bytes.NewReader([]byte("test")), "1", "file.txt", src)
require.Error(t, err)
assert.Equal(t, int32(1), atomic.LoadInt32(&attempts), "single part uploads must not be retried here")
}
// TestUpdateDirect checks the upload parameters and metadata of a
// successful single part upload, and that total_chunk_hash is only
// sent when the source hash is known
func TestUpdateDirect(t *testing.T) {
for _, test := range []struct {
name string
srcHash string
wantTotalHash string
}{
{name: "hash known", srcHash: "12345678", wantTotalHash: "xxh3:12345678"},
{name: "hash unknown", srcHash: "", wantTotalHash: ""},
} {
t.Run(test.name, func(t *testing.T) {
var gotTotalHash, gotBody string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
gotTotalHash = r.URL.Query().Get("total_chunk_hash")
body, _ := io.ReadAll(r.Body)
gotBody = string(body)
_, _ = fmt.Fprintln(w, `{"result":"success","data":{"id":55,"name":"file.txt","type":"file","path":"/file.txt","hash":"xxh3:99998888","last_modified_at":1700000000}}`)
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
o := &Object{fs: f, remote: "file.txt"}
src := &mockObjectInfo{remote: "file.txt", size: 4, modTime: time.Unix(1700000000, 0), hash: test.srcHash}
err := o.updateDirect(context.Background(), bytes.NewReader([]byte("test")), "1", "file.txt", src)
require.NoError(t, err)
assert.Equal(t, "test", gotBody, "the content must be streamed as-is")
assert.Equal(t, test.wantTotalHash, gotTotalHash, "total_chunk_hash")
assert.Equal(t, "55", o.id)
gotHash, hashErr := o.Hash(context.Background(), hash.XXH3)
require.NoError(t, hashErr)
assert.Equal(t, "99998888", gotHash)
})
}
}
// TestNewFsRequiresDriveID checks that NewFs fails fast when drive_id
// is missing
func TestNewFsRequiresDriveID(t *testing.T) {
m := configmap.Simple{"chunk_size": "20Mi"}
_, err := NewFs(context.Background(), "kdrive", "", m)
require.Error(t, err)
assert.Contains(t, err.Error(), "drive_id")
}
// TestUploadSessionStringMasksToken checks that the upload session
// String does not leak the session token in debug logs
func TestUploadSessionStringMasksToken(t *testing.T) {
u := &uploadSession{token: "very-secret-uuid"}
assert.NotContains(t, u.String(), "very-secret-uuid")
}
// TestListNonRecursiveSingleRoundTrip checks that a non recursive
// listing needs a single API request: without depth=unlimited the API
// returns direct children only, so neither the parent item lookup nor
// a child filter is needed
func TestListNonRecursiveSingleRoundTrip(t *testing.T) {
var requests int32
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&requests, 1)
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
switch {
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/files/1/files"):
// Direct children only, as returned without depth=unlimited
_, _ = fmt.Fprintln(w, `{"result":"success","data":[`+
`{"id":10,"name":"sub","type":"dir","path":"/sub"},`+
`{"id":11,"name":"file.txt","type":"file","path":"/file.txt","hash":"xxh3:12345678","last_modified_at":1700000000}],"has_more":false}`)
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"unexpected","description":"unexpected request"}}`)
}
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
f.dirCache = dircache.New("", "1", f)
entries, err := f.List(context.Background(), "")
require.NoError(t, err)
assert.Equal(t, int32(1), atomic.LoadInt32(&requests), "non recursive listing must make a single API request")
names := make([]string, 0, len(entries))
for _, entry := range entries {
names = append(names, entry.Remote())
}
assert.ElementsMatch(t, []string{"sub", "file.txt"}, names)
}
// TestMoveConflictReturnsError checks that moving onto an occupied
// destination reports the API conflict instead of resolving it: rclone
// removes the destination itself before calling Move
func TestMoveConflictReturnsError(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
switch {
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/files/111/move/1"):
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"conflict_error","description":"destination already exists"}}`)
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"unexpected","description":"unexpected request"}}`)
}
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
f.dirCache = dircache.New("", "1", f)
srcObj := &Object{fs: f, remote: "src.txt", id: "111", hasMetaData: true}
_, err := f.Move(context.Background(), srcObj, "file2")
require.Error(t, err, "a move onto an occupied destination must fail")
assert.Contains(t, err.Error(), "conflict", "the API conflict must be reported")
}
// TestMoveDetectsForeignObjectAtDestination checks that Move does not
// return a file re-created by another writer at the destination
func TestMoveDetectsForeignObjectAtDestination(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
switch {
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/files/111/move/1"):
// The move API reports success...
_, _ = fmt.Fprintln(w, `{"result":"success","data":{"cancel_id":"abc","valid_until":1700000000}}`)
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/files/1/name"):
// ...but the file at the destination is not the moved
// file: another writer re-created it
_, _ = fmt.Fprintln(w, kdriveMockItem(999, "file2"))
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"unexpected","description":"unexpected request"}}`)
}
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
f.dirCache = dircache.New("", "1", f)
srcObj := &Object{fs: f, remote: "src.txt", id: "111", hasMetaData: true}
_, err := f.Move(context.Background(), srcObj, "file2")
require.Error(t, err, "Move must not report success when the destination is not the moved file")
}
// TestMoveSuccess checks the nominal move: the object returned is the
// moved file (same ID as the source).
func TestMoveSuccess(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = io.Copy(io.Discard, r.Body)
switch {
case r.Method == http.MethodPost && strings.HasSuffix(r.URL.Path, "/files/111/move/1"):
_, _ = fmt.Fprintln(w, `{"result":"success","data":{"cancel_id":"abc","valid_until":1700000000}}`)
case r.Method == http.MethodGet && strings.HasSuffix(r.URL.Path, "/files/1/name"):
// After the move, the moved file is at the destination
_, _ = fmt.Fprintln(w, kdriveMockItem(111, "file2"))
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
_, _ = fmt.Fprintln(w, `{"result":"error","error":{"code":"unexpected","description":"unexpected request"}}`)
}
}))
defer ts.Close()
f := newTestFs(t, ts.URL)
f.dirCache = dircache.New("", "1", f)
srcObj := &Object{fs: f, remote: "src.txt", id: "111", hasMetaData: true}
dstObj, err := f.Move(context.Background(), srcObj, "file2")
require.NoError(t, err)
moved, ok := dstObj.(*Object)
require.True(t, ok)
assert.Equal(t, "111", moved.id, "the returned object must be the moved file")
}
// TestFormatModTime checks that pre-1970 times format as negative
// unix timestamps instead of wrapping through a uint64 cast
func TestFormatModTime(t *testing.T) {
for _, test := range []struct {
in time.Time
want string
}{
{in: time.Unix(1700000000, 0), want: "1700000000"},
{in: time.Unix(0, 0), want: "0"},
{in: time.Unix(-1577923200, 0), want: "-1577923200"}, // Sat Jan 2 1965
} {
t.Run(test.want, func(t *testing.T) {
assert.Equal(t, test.want, formatModTime(test.in))
})
}
}
// TestObjectSetMetaDataIsDir checks that directories are rejected
func TestObjectSetMetaDataIsDir(t *testing.T) {
o := &Object{remote: "dir"}
info := &api.Item{ID: 42, Name: "dir", Type: "dir"}
err := o.setMetaData(info)
assert.Equal(t, fs.ErrorIsDir, err)
}
// TestShouldRetryLockError checks that transient locks held by server
// side copies are retried while locks held by applications and other
// errors are not
func TestShouldRetryLockError(t *testing.T) {
for _, test := range []struct {
name string
body string
status int
wantRetry bool
}{
{
name: "lock held by internal copy",
body: `{"result":"error","error":{"code":"lock_error","description":"There is a lock on this file, locked by internal_copy","context":{"values":["internal_copy"]}}}`,
status: http.StatusForbidden,
wantRetry: true,
},
{
name: "lock held by an application",
body: `{"result":"error","error":{"code":"lock_error","description":"There is a lock on this file, locked by onlyoffice","context":{"values":["onlyoffice"]}}}`,
status: http.StatusForbidden,
wantRetry: false,
},
{
name: "lock without context",
body: `{"result":"error","error":{"code":"lock_error","description":"There is a lock on this file, locked by internal_copy"}}`,
status: http.StatusForbidden,
wantRetry: false,
},
{
name: "other api error",
body: `{"result":"error","error":{"code":"object_not_found","description":"File not found"}}`,
status: http.StatusNotFound,
wantRetry: false,
},
{
name: "server error",
status: http.StatusServiceUnavailable,
wantRetry: true,
},
} {
t.Run(test.name, func(t *testing.T) {
var apiErr error
if test.body != "" {
e := new(api.ResultStatus)
require.NoError(t, json.Unmarshal([]byte(test.body), e))
apiErr = e
}
resp := &http.Response{StatusCode: test.status, Header: http.Header{}}
got, gotErr := shouldRetry(context.Background(), resp, apiErr)
assert.Equal(t, test.wantRetry, got)
assert.Equal(t, apiErr, gotErr)
})
}
}
-25
View File
@@ -1,25 +0,0 @@
// Test kdrive filesystem interface
package kdrive
import (
"testing"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest/fstests"
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestKdrive:",
NilObject: (*Object)(nil),
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: minChunkSize,
MaxChunkSize: maxChunkSize,
},
})
}
func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
return f.setChunkSize(cs)
}
-67
View File
@@ -1,67 +0,0 @@
// Package khash provides the kDrive specific hash handling.
//
// kDrive uses XXH3-64 for file hashing. Files uploaded in chunks (by
// rclone, or by the kDrive apps which each use their own chunk size)
// get a "nested" hash: the XXH3 of the concatenated lowercase hex
// strings of the individual chunk hashes. Such a nested hash can't be
// compared with a plain XXH3 of the file contents.
//
// Hash formats returned by the kDrive API:
// - "xxh3:HASH" for a plain XXH3 of the file contents
// - "N:xxh3:HASH" for the nested hash of a chunked upload
package khash
import (
"encoding/hex"
"errors"
"fmt"
"strings"
"github.com/zeebo/xxh3"
)
// ParseHash parses a kDrive hash string from the API and returns the hex hash value
// along with whether it is a nested (chunked) hash.
func ParseHash(hashStr string) (hexHash string, isNested bool, err error) {
if hashStr == "" {
return "", false, nil
}
// Check for nested hash format: "N:xxh3:HASH"
if strings.HasPrefix(hashStr, "N:xxh3:") {
hashStr = strings.TrimPrefix(hashStr, "N:xxh3:")
isNested = true
} else {
hashStr = strings.TrimPrefix(hashStr, "xxh3:")
}
// Validate hex string
if _, decodeErr := hex.DecodeString(hashStr); decodeErr != nil {
return "", false, fmt.Errorf("invalid hash format: %w", decodeErr)
}
return hashStr, isNested, nil
}
// NestedChunkHash computes the kDrive nested hash from the hashes of the
// file chunks, in order.
//
// kDrive stores the nested hash of a chunked file as the XXH3 of the
// concatenated lowercase hex strings of the individual chunk hashes.
// Each chunk hash may be passed with or without the "xxh3:" prefix.
func NestedChunkHash(chunkHashes []string) (string, error) {
if len(chunkHashes) == 0 {
return "", errors.New("no chunk hashes to nest")
}
hexHashes := make([]string, len(chunkHashes))
for i, chunkHash := range chunkHashes {
hexHash, _, err := ParseHash(chunkHash)
if err != nil {
return "", fmt.Errorf("invalid chunk hash %q: %w", chunkHash, err)
}
hexHashes[i] = hexHash
}
hasher := xxh3.New()
_, _ = hasher.Write([]byte(strings.Join(hexHashes, "")))
return hex.EncodeToString(hasher.Sum(nil)), nil
}
-150
View File
@@ -1,150 +0,0 @@
package khash
import "testing"
func TestParseHash(t *testing.T) {
tests := []struct {
name string
input string
expectedHash string
isNested bool
expectError bool
}{
{
name: "simple hash with prefix",
input: "xxh3:1db797b96febd334",
expectedHash: "1db797b96febd334",
isNested: false,
},
{
name: "nested hash",
input: "N:xxh3:877abf3579f0a5c0",
expectedHash: "877abf3579f0a5c0",
isNested: true,
},
{
name: "plain hex",
input: "1db797b96febd334",
expectedHash: "1db797b96febd334",
isNested: false,
},
{
name: "empty string",
input: "",
expectedHash: "",
isNested: false,
},
{
name: "invalid hex",
input: "xxh3:notahexvalue",
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
hash, isNested, err := ParseHash(tt.input)
if tt.expectError {
if err == nil {
t.Error("Expected error but got none")
}
return
}
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if hash != tt.expectedHash {
t.Errorf("Hash: got %q, expected %q", hash, tt.expectedHash)
}
if isNested != tt.isNested {
t.Errorf("IsNested: got %t, expected %t", isNested, tt.isNested)
}
})
}
}
func TestNestedChunkHash(t *testing.T) {
// Golden values: the nested hash is the XXH3 of the concatenated
// lowercase hex strings of the chunk hashes, in order
chunk1 := "xxh3:d447b1ea40e6988b" // XXH3 of "hello world"
chunk2 := "2d06800538d394c2" // XXH3 of ""
tests := []struct {
name string
chunkHashes []string
expected string
expectError bool
}{
{
name: "two chunks in order",
chunkHashes: []string{chunk1, chunk2},
expected: "6d51edccc9c05a8b",
},
{
name: "order matters",
chunkHashes: []string{chunk2, chunk1},
expected: "596c127c7bcc8820",
},
{
name: "single chunk",
chunkHashes: []string{chunk1},
expected: "b73081dff8fe443c",
},
{
name: "mixed prefixes",
chunkHashes: []string{chunk1, "xxh3:" + chunk2, chunk1},
expected: "ca0b13abcdcd1661",
},
{
name: "no chunks",
chunkHashes: []string{},
expectError: true,
},
{
name: "invalid chunk hash",
chunkHashes: []string{chunk1, "notahexvalue"},
expectError: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
nested, err := NestedChunkHash(tt.chunkHashes)
if tt.expectError {
if err == nil {
t.Error("Expected error but got none")
}
return
}
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if nested != tt.expected {
t.Errorf("NestedChunkHash: got %q, expected %q", nested, tt.expected)
}
})
}
}
func TestNestedChunkHashAcceptsPrefixedHashes(t *testing.T) {
plain := []string{"d447b1ea40e6988b", "2d06800538d394c2"}
prefixed := []string{"xxh3:d447b1ea40e6988b", "N:xxh3:2d06800538d394c2"}
gotPlain, err := NestedChunkHash(plain)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
gotPrefixed, err := NestedChunkHash(prefixed)
if err != nil {
t.Fatalf("Unexpected error: %v", err)
}
if gotPlain != gotPrefixed {
t.Errorf("Prefixed hashes should give the same nested hash: %q != %q", gotPlain, gotPrefixed)
}
}
-383
View File
@@ -1,383 +0,0 @@
package kdrive
import (
"context"
"encoding/hex"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"path"
"strconv"
"strings"
"sync"
"time"
"github.com/rclone/rclone/backend/kdrive/api"
"github.com/rclone/rclone/backend/kdrive/khash"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/chunksize"
"github.com/rclone/rclone/lib/rest"
"github.com/zeebo/xxh3"
"golang.org/x/text/unicode/norm"
)
const (
// maxUploadParts is the maximum number of chunks the kDrive API accepts per file
maxUploadParts = 10000
// maxChunkSize is the largest chunk size the kDrive API accepts
maxChunkSize = 1 * 1000 * 1000 * 1000
// minChunkSize is the smallest chunk size the kDrive API accepts
minChunkSize = 1 * 1000 * 1000
)
// checkChunkSize checks the user supplied chunk_size against the
// limits the kDrive API accepts
func checkChunkSize(chunkSize fs.SizeSuffix) error {
if chunkSize < minChunkSize {
return fmt.Errorf("chunk_size %s is too small: the kDrive API requires chunks of at least 1 MB", chunkSize)
}
if chunkSize > maxChunkSize {
return fmt.Errorf("chunk_size %s is too large: the kDrive API accepts chunks of at most 1 GB (1000000000 bytes)", chunkSize)
}
return nil
}
// setChunkSize sets the chunk size, returning the old value
func (f *Fs) setChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
err = checkChunkSize(cs)
if err == nil {
old, f.opt.ChunkSize = f.opt.ChunkSize, cs
}
return
}
// formatModTime formats the modification time of an upload as the
// unix timestamp string the kDrive API expects
func formatModTime(modTime time.Time) string {
return fmt.Sprintf("%d", modTime.Unix())
}
// uploadSession implements fs.ChunkWriter for kdrive multipart uploads
type uploadSession struct {
f *Fs
token string
uploadURL string
fileInfo *api.Item
fileSize int64 // total size of the file being uploaded
chunkSize int64 // size of the chunks of this session
mu sync.Mutex
chunkHashes map[int64]string // chunk number -> hash of the chunk contents
}
// OpenChunkWriter returns the chunk size and a ChunkWriter
//
// Pass in the remote and the src object
// You can also use options to hint at the desired chunk size
// @see https://developer.infomaniak.com/docs/api/post/3/drive/%7Bdrive_id%7D/upload/session/start
func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectInfo, options ...fs.OpenOption) (info fs.ChunkWriterInfo, writer fs.ChunkWriter, err error) {
fileSize := src.Size()
if fileSize < 0 {
return info, nil, errors.New("kdrive can't upload files with unknown size")
}
dir, leaf := path.Split(remote)
dir = strings.TrimSuffix(dir, "/")
// Create parent directories if they don't exist
parentID, err := f.dirCache.FindDir(ctx, dir, true)
if err != nil {
return info, nil, fmt.Errorf("failed to find parent directory: %w", err)
}
// Calculate the chunk size, honouring any fs.ChunkOption hint
chunkSize := f.opt.ChunkSize
for _, opt := range options {
if chunkOpt, ok := opt.(*fs.ChunkOption); ok && chunkOpt.ChunkSize > 0 {
chunkSize = fs.SizeSuffix(chunkOpt.ChunkSize)
}
}
chunkSize = chunksize.Calculator(src, fileSize, maxUploadParts, chunkSize)
if chunkSize < minChunkSize {
chunkSize = minChunkSize
}
if chunkSize > maxChunkSize {
chunkSize = maxChunkSize
}
// Calculate the total number of chunks
totalChunks := max((fileSize+int64(chunkSize)-1)/int64(chunkSize), 1)
sessionReq := struct {
Conflict string `json:"conflict"`
DirectoryID string `json:"directory_id"`
FileName string `json:"file_name"`
LastModifiedAt string `json:"last_modified_at"`
TotalChunks int64 `json:"total_chunks"`
TotalSize int64 `json:"total_size"`
}{
Conflict: "version",
DirectoryID: parentID,
FileName: f.opt.Enc.FromStandardName(norm.NFC.String(leaf)),
LastModifiedAt: formatModTime(src.ModTime(ctx)),
TotalChunks: totalChunks,
TotalSize: fileSize,
}
// https://developer.infomaniak.com/docs/api/post/3/drive/%7Bdrive_id%7D/upload/session/start
opts := rest.Opts{
Method: "POST",
Path: fmt.Sprintf("/3/drive/%s/upload/session/start", f.opt.DriveID),
}
var sessionResp api.SessionStartResponse
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
var callErr error
resp, callErr = f.srv.CallJSON(ctx, &opts, &sessionReq, &sessionResp)
callErr = sessionResp.ResultStatus.Update(callErr)
return shouldRetry(ctx, resp, callErr)
})
if err != nil {
return info, nil, fmt.Errorf("failed to start upload session: %w", err)
}
info = fs.ChunkWriterInfo{
ChunkSize: int64(chunkSize),
Concurrency: f.opt.UploadConcurrency,
}
writer = &uploadSession{
f: f,
token: sessionResp.Data.Token,
uploadURL: sessionResp.Data.UploadURL,
fileSize: fileSize,
chunkSize: int64(chunkSize),
chunkHashes: make(map[int64]string),
}
fs.Debugf(f, "open chunk writer: started upload session for %d chunks of %v", totalChunks, chunkSize)
return info, writer, nil
}
// WriteChunk uploads a single chunk
// @see https://developer.infomaniak.com/docs/api/post/3/drive/%7Bdrive_id%7D/upload/session/%7Bsession_token%7D/chunk
func (u *uploadSession) WriteChunk(ctx context.Context, chunkNumber int, reader io.ReadSeeker) (bytesWritten int64, err error) {
if chunkNumber < 0 {
return -1, fmt.Errorf("invalid chunk number provided: %v", chunkNumber)
}
// Calculate the size of this chunk from the session parameters so
// the chunk can be streamed to the API in a single pass
chunkLen := u.chunkSize
if remaining := u.fileSize - int64(chunkNumber)*u.chunkSize; remaining < chunkLen {
chunkLen = remaining
}
if chunkLen <= 0 {
return 0, nil
}
sourceChunkNumber := chunkNumber + 1 // kDrive API uses 1-based numbering
// The chunk_hash parameter is optional, so the chunk is streamed
// in a single pass, hashing the contents on the way with a
// TeeReader. The server returns the hash of the chunk it received,
// which is checked below. Retries seek back to the chunk start
// and hash with a fresh hasher.
var chunkHash string
err = u.f.pacer.Call(func() (bool, error) {
if _, seekErr := reader.Seek(0, io.SeekStart); seekErr != nil {
return false, fmt.Errorf("failed to rewind chunk %d: %w", sourceChunkNumber, seekErr)
}
chunkHasher := xxh3.New()
// https://developer.infomaniak.com/docs/api/post/3/drive/%7Bdrive_id%7D/upload/session/%7Bsession_token%7D/chunk
chunkOpts := rest.Opts{
Method: "POST",
RootURL: u.uploadURL,
Path: fmt.Sprintf("/3/drive/%s/upload/session/%s/chunk", u.f.opt.DriveID, u.token),
Parameters: url.Values{
"chunk_number": {fmt.Sprintf("%d", sourceChunkNumber)},
"chunk_size": {fmt.Sprintf("%d", chunkLen)},
"with": {"hash"},
},
Body: io.TeeReader(reader, chunkHasher),
ContentLength: &chunkLen,
}
var chunkResp api.ChunkUploadResponse
resp, callErr := u.f.srv.CallJSON(ctx, &chunkOpts, nil, &chunkResp)
callErr = chunkResp.ResultStatus.Update(callErr)
if callErr != nil {
return shouldRetry(ctx, resp, callErr)
}
chunkHash = hex.EncodeToString(chunkHasher.Sum(nil))
// The server hashed the chunk it received, so compare with what we
// sent to detect any corruption immediately
if chunkResp.Data.Hash != "" {
if serverHash, _, parseErr := khash.ParseHash(chunkResp.Data.Hash); parseErr == nil {
if !strings.EqualFold(chunkHash, serverHash) {
return false, fmt.Errorf("chunk %d hash mismatch: client=%s, server=%s", sourceChunkNumber, chunkHash, serverHash)
}
} else {
fs.Debugf(u, "Failed to parse server chunk hash %q: %v", chunkResp.Data.Hash, parseErr)
}
}
return false, nil
})
if err != nil {
return -1, err
}
u.mu.Lock()
u.chunkHashes[int64(chunkNumber)] = chunkHash
u.mu.Unlock()
fs.Debugf(u, "uploaded chunk %d (size: %d, hash: %s)", sourceChunkNumber, chunkLen, chunkHash)
return chunkLen, nil
}
// Close finalizes the upload session, verifies the hash of the uploaded
// file and returns the created file info
// @see https://developer.infomaniak.com/docs/api/post/3/drive/%7Bdrive_id%7D/upload/session/%7Bsession_token%7D/finish
func (u *uploadSession) Close(ctx context.Context) error {
// https://developer.infomaniak.com/docs/api/post/3/drive/%7Bdrive_id%7D/upload/session/%7Bsession_token%7D/finish
opts := rest.Opts{
Method: "POST",
Path: fmt.Sprintf("/3/drive/%s/upload/session/%s/finish", u.f.opt.DriveID, u.token),
}
var resp *http.Response
var finishResp api.SessionFinishResponse
err := u.f.pacer.Call(func() (bool, error) {
var callErr error
resp, callErr = u.f.srv.CallJSON(ctx, &opts, nil, &finishResp)
callErr = finishResp.ResultStatus.Update(callErr)
return shouldRetry(ctx, resp, callErr)
})
if err != nil {
return fmt.Errorf("failed to finish upload session: %w", err)
}
u.fileInfo = &finishResp.Data.File
fs.Debugf(u, "multipart upload completed: file id %d", finishResp.Data.File.ID)
return u.verifyHash(ctx)
}
// verifyHash checks the hash of the uploaded file against the hashes of
// the chunks we uploaded, deleting the file if it is corrupted
func (u *uploadSession) verifyHash(ctx context.Context) (err error) {
remoteHash := u.fileInfo.Hash
if remoteHash == "" {
// Hash might be missing in the finish response, try to fetch it
obj := &Object{
fs: u.f,
id: strconv.Itoa(u.fileInfo.ID),
}
var hashErr error
if remoteHash, hashErr = obj.retrieveHash(ctx); hashErr != nil {
// skip verification
fs.Debugf(u, "Failed to retrieve hash for verification: %v", hashErr)
return nil
}
}
parsedRemoteHash, isNested, err := khash.ParseHash(remoteHash)
if err != nil {
fs.Debugf(u, "Failed to parse server hash %q: %v", remoteHash, err)
return nil
}
u.mu.Lock()
defer u.mu.Unlock()
// Rebuild the hash the server should have stored from the chunk
// hashes we uploaded, in order
chunkCount := int64(len(u.chunkHashes))
if chunkCount == 0 {
return nil
}
chunkHashes := make([]string, 0, chunkCount)
for i := int64(0); i < chunkCount; i++ {
chunkHash, ok := u.chunkHashes[i]
if !ok {
fs.Debugf(u, "Chunk %d hash missing, skipping hash verification", i)
return nil
}
chunkHashes = append(chunkHashes, chunkHash)
}
var localHash string
if isNested {
if localHash, err = khash.NestedChunkHash(chunkHashes); err != nil {
fs.Debugf(u, "Failed to compute nested hash: %v", err)
return nil
}
} else if chunkCount == 1 {
localHash, _, _ = khash.ParseHash(chunkHashes[0])
} else {
// The server returned a simple hash for a multi chunk upload,
// nothing to compare it against
fs.Debugf(u, "Server returned a simple hash for a %d chunk upload, skipping hash verification", chunkCount)
return nil
}
if !strings.EqualFold(localHash, parsedRemoteHash) {
err = fmt.Errorf(
"multipart upload hash mismatch: local=%s, remote=%s",
localHash, parsedRemoteHash,
)
// Remove the corrupted file
obj := &Object{
fs: u.f,
id: strconv.Itoa(u.fileInfo.ID),
}
if delErr := obj.Remove(ctx); delErr != nil {
fs.Errorf(nil, "Failed to remove corrupted file after hash mismatch: %v", delErr)
} else {
fs.Debugf(nil, "Removed corrupted file after hash mismatch")
}
return err
}
fs.Debugf(u, "Multipart upload hash verified: %s", localHash)
return nil
}
// Abort the upload session
// @see https://developer.infomaniak.com/docs/api/delete/2/drive/%7Bdrive_id%7D/upload/session/%7Bsession_token%7D
func (u *uploadSession) Abort(ctx context.Context) error {
// https://developer.infomaniak.com/docs/api/delete/2/drive/%7Bdrive_id%7D/upload/session/%7Bsession_token%7D
opts := rest.Opts{
Method: "DELETE",
Path: fmt.Sprintf("/2/drive/%s/upload/session/%s", u.f.opt.DriveID, u.token),
}
var resp *http.Response
var cancelResp api.SessionCancelResponse
err := u.f.pacer.Call(func() (bool, error) {
var callErr error
resp, callErr = u.f.srv.CallJSON(ctx, &opts, nil, &cancelResp)
callErr = cancelResp.ResultStatus.Update(callErr)
return shouldRetry(ctx, resp, callErr)
})
if err != nil {
return fmt.Errorf("failed to cancel upload session: %w", err)
}
fs.Debugf(u, "upload session cancelled")
return nil
}
// String implements fmt.Stringer
func (u *uploadSession) String() string {
return "kdrive upload session"
}
// Verify interface compliance
var _ fs.ChunkWriter = (*uploadSession)(nil)
+9 -28
View File
@@ -11,10 +11,10 @@ package linkbox
*/
import (
"bytes"
"context"
"crypto/md5"
"crypto/tls"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
@@ -35,9 +35,7 @@ import (
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/rest"
)
@@ -48,8 +46,7 @@ const (
pacerBurst = 1
linkboxAPIURL = "https://www.linkbox.to/api/open/"
linkboxWebAPIURL = "https://www.linkbox.to/api/"
rootID = "0" // ID of root directory
prefixSize = 10 * 1024 * 1024 // the API wants the MD5 of the first 10 MiB of a file
rootID = "0" // ID of root directory
)
func init() {
@@ -670,21 +667,6 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
return res.Body, nil
}
// readPrefix reads up to prefixSize bytes of in into a buffer from the
// global pool returning it along with the hex MD5 of what was read.
//
// The caller must Close the returned buffer when done with it.
func readPrefix(in io.Reader) (rw *pool.RW, md5sum string, err error) {
rw = multipart.NewRW()
hasher := md5.New()
_, err = io.CopyN(rw, io.TeeReader(in, hasher), prefixSize)
if err != nil && err != io.EOF {
_ = rw.Close()
return nil, "", err
}
return rw, hex.EncodeToString(hasher.Sum(nil)), nil
}
// Update in to the object with the modTime given of the given size
//
// When called from outside an Fs by rclone, src.Size() will always be >= 0.
@@ -719,13 +701,11 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
}
prefix, prefixMD5, err := readPrefix(in)
first10m := io.LimitReader(in, 10_485_760)
first10mBytes, err := io.ReadAll(first10m)
if err != nil {
return fmt.Errorf("Update err in reading file: %w", err)
}
defer func() {
_ = prefix.Close()
}()
// get upload authorization (step 1)
opts := &rest.Opts{
@@ -735,7 +715,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
Options: options,
Parameters: url.Values{
"token": {o.fs.opt.Token},
"fileMd5ofPre10m": {prefixMD5},
"fileMd5ofPre10m": {fmt.Sprintf("%x", md5.Sum(first10mBytes))},
"fileSize": {itoa64(size)},
},
}
@@ -777,7 +757,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
return fmt.Errorf("head upload URL: %w", err)
}
file := io.MultiReader(prefix, in)
file := io.MultiReader(bytes.NewReader(first10mBytes), in)
opts.Method = "PUT"
opts.Body = file
@@ -817,7 +797,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
Options: options,
Parameters: url.Values{
"token": {o.fs.opt.Token},
"fileMd5ofPre10m": {prefixMD5},
"fileMd5ofPre10m": {fmt.Sprintf("%x", md5.Sum(first10mBytes))},
"fileSize": {itoa64(size)},
"pid": {dirID},
"diyName": {leaf},
@@ -1039,7 +1019,8 @@ func (f *Fs) shouldRetry(ctx context.Context, resp *http.Response, err error) (b
// intermittently returns an HTML challenge page with a 200
// status instead of JSON. This surfaces as a JSON syntax error,
// so retry it to let the pacer back off until the block lifts.
if _, ok := errors.AsType[*json.SyntaxError](err); ok {
var syntaxErr *json.SyntaxError
if errors.As(err, &syntaxErr) {
return true, err
}
return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
Loaded 100 of 769 files, more files were not shown because too many files have changed in this diff. Show more