Before this change setting LogLevel in the _config parameter of an rc
call had no effect as both the log functions and the log handler only
looked at the global log level.
The context aware log functions now use the log level from the config
in the context, and the log handler lets a log level in the context
which differs from the global one override its own level. This means
an rc call can be made more or less verbose than the rest of rclone
with, for example
"_config": {"LogLevel": "DEBUG"}
This applies to the logs made with the context aware log functions, so
not to those made by the backends which still use the global log level.
Note that a config copied into a context no longer follows later
changes to the global log level.
Use the context aware log functions throughout the code which runs
sync, copy, move, check and the other operations, and the directory
listings they do, so that the logs they make, such as the per file
"Copied (new)" logs, are attributed to the rc job and stats group of
the operation.
This adds ErrorfCtx, LogfCtx, InfofCtx, DebugfCtx and LogPrintfCtx
which take a context and pass it on to the log handler. The handler
can then attribute the log to what made it using the new
StatsGroupFromContext and JobIDFromContext function pointers, which
fs/accounting and fs/rc/jobs fill in.
The format string is no longer assigned to in logPrintf so that go
vet recognises these as printf wrappers and checks the arguments of
their callers.
Several links on rclone.org pointed at anchors or pages that don't exist:
- bisync: --color and --dry-run anchors, relative commands/rclone_check/ links
- ftp: on-the-fly link pointed at a local anchor instead of /docs/
- faq: --sftp-http-proxy linked to the ftp page instead of sftp
- index: Arvan Cloud and Liara provider config anchors
- rc, sftp: Hugo drops leading dashes from heading ids (#--rc-no-auth -> #rc-no-auth)
- onedrive: webdav #sharepoint -> #sharepoint-online
- webdav: relative commands/rclone_serve_webdav/ link
- s3: https:/// typo in the China Mobile Ecloud link
Directory listings showed every directory with a size of -1, so
`rclone lsd` and `operations/list` couldn't say how big a directory
was, and finding out meant walking it with `rclone size`.
Microsoft Graph already returns each folder's `size` as the total size
of its contents, and the backend read it but dropped it. Directories
now carry that size, as they already carry the item count. Listing a
directory's parent is enough to get the size of all its subdirectories,
without listing the subdirectories themselves.
OneDrive updates a folder's size in the background, so for a few
minutes after a change inside it the size may be out of date.
Fixes#7502
On shutdown the batcher logged "Committing uploads - please wait..."
but nothing once the commit had completed, so the log looked as if
rclone had stopped part way through. Log "Committing uploads - done"
when the pending batches have been committed.
This affects the backends which use the batcher, dropbox and
googlephotos.
When container soft delete is enabled, listing the containers at the
root of the account returned the soft deleted ones too, because the
listing asked for them with Include Deleted. rclone treated them as
normal containers and then failed to list each of them with
"directory not found". With at least one error, sync refused to delete
anything on the destination ("not deleting files as there were IO
errors") and ended with exit code 3.
The listing now asks for live containers only, as it did before the
port to the new SDK in a131da2c3. Nothing else in the backend used the
deleted flag of the containers.
Nextcloud public shares served from public.php/dav/files reject requests
that lack an X-Requested-With header. Send "XMLHttpRequest" for these
URLs when vendor is nextcloud, unless the user already set the header
with --webdav-headers.
This change adds a step to the bisync tests that checks that after a successful
run, the files named in each listing are exactly the files on that path. This
ensures that nothing is silently missing from the listings.
Before this change, when an existing file had changed on both paths, the
conflict winner was chosen using only the attributes that had changed on each
side. An attribute that hadn't changed was treated as zero, so --conflict-resolve
larger could pick the smaller file (for example, when one side only had its
modtime updated), and --conflict-resolve newer could fail to pick a winner when one
side's modtime hadn't changed.
After this change, the current size, modtime, and checksum (to the extent
enabled with compare settings) are now recorded for every changed file.
Before this change, when --conflict-resolve picked Path2 as the winner, the winning
file was copied to Path1 correctly but then removed from both listings while
updating them for the renamed (or deleted) Path1 copy. The next run saw the file
as new on both paths and had to check it again, and a change to it on one path
in that run was handled as if the file had never been synced. When Path1 won,
the listings happened to be updated in an order that avoided this.
This change fixes the issue by adding a guard to remove the file from the
listing only when dstNewName != dstOldName.
See #9939
- add the release freeze rule, GUI update and security advisory steps
- list the extra software and remotes needed
- explain what make tag, make doc and make retag do
- check integration.rclone.org instead of running make test
- describe how to run the integration tests on the stable branch
- spell out the differences when making a point release
- note that stable branches are abandoned once a new release is out
make startdev now starts the next minor version on master and the next
patch version on a v*-stable branch, so the same target works for
releases and point releases. It refuses to run on other branches or if
the current version hasn't been tagged yet, which catches running it
twice. make startstable is kept as an alias.
golang.org/x/net v0.60.0 deprecates its HTTP/2 server and transport in
favour of net/http.
Make the h2c test use http.Transport with unencrypted HTTP/2. serve
restic --stdio still needs http2.Server to serve HTTP/2 on a single
connection as http.Server has no way of doing that.
Upgrade to v0.60.0 of golang.org/x/net in order to address:
- CVE-2026-78659: net/http: HTTP/2 server memory exhaustion due to Trailer headers
- CVE-2026-78669: net/http: excessive CPU consumption from repeated initial window changes
- CVE-2026-78663: net/http: double flow control refund on HTTP/2 server streams
- CVE-2026-97032: net/http: HTTP/2 server crash due to HPACK encoder race
This also updates golang.org/x/crypto, sync, sys, term and text to the
versions golang.org/x/net v0.60.0 requires.
Before go1.27 net/http compares hosts case sensitively when deciding
whether to keep Authorization on a redirect, so it dropped the header
itself on the localhost -> LOCALHOST redirect in the
SameHostDifferentCase test, before StripHeadersOnCrossHostRedirectFn
was consulted. The other headers still show the function treats the
hosts as the same.
The metadata of the objects of all users was kept in one store, so with
--auth-proxy or several --auth-key credentials one user storing the
metadata of many objects could make the server forget that of every
other user.
Each user (each access key) now has a store of their own.
--multipart-max-uploads and --multipart-streaming-buffer-total limited
the uploads of all users together, so with --auth-proxy or several
--auth-key credentials one user could use up the limits and stop every
other user uploading.
They now limit the uploads of each user (each access key) separately.
A multipart part which arrives before the parts ahead of it is buffered
and acknowledged to the client, then written to the backend when its
turn comes. If that write failed the part was dropped, possibly with
some of it already written, but the upload carried on as if nothing
had happened, so it could be completed with the part missing, or with
a resend of the part appended to the partial data.
Such an upload is now failed, so the client starts it again.
The SigV4 verification in gofakes3 only checked the headers the
client listed as signed and never looked at the other headers which
arrived. A presigned PUT URL signs only the host header, so a holder
of one could add an unsigned x-amz-copy-source header and have the
server copy any object it can reach - in any bucket, and in any
remote when serving a cloud backend - into the object the URL names,
then read it back with the same URL. This affected both --auth-key
and --auth-proxy. AWS refuses such a request with AccessDenied, as
it requires every x-amz-* header to be signed.
This updates gofakes3 to a version which refuses requests carrying
an x-amz-* header not in their signed headers list with AccessDenied,
and adds tests that a presigned PUT URL can't be turned into a copy
in either auth mode. Real S3 clients always sign x-amz-* headers so
they are unaffected.
In v1.75 the multipart reorder buffer admitted the next part the stream
needed however big, adding its client-declared length to the running
total unchecked. Two concurrent requests for the next part declaring
huge lengths overflowed the total negative, after which out-of-order
parts were admitted past --multipart-streaming-buffer-limit.
The next part is now streamed straight to the backend without being
charged to the buffer, and a buffered part is only admitted if it fits
in what is left of the limits, so this can no longer happen. Add a
test of the attack against both the per-upload and server-wide limits.
serve s3 keeps the metadata of objects uploaded through it in memory.
If the object was then changed some other way - by another --auth-proxy
user sharing the backend, another server or directly on the backend -
serve s3 went on returning the old metadata, including the old
modification time, for the new object.
The size and modification time of the object are now stored with its
metadata, which is ignored if they no longer match.
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. A request didn't count as using it, so a
long upload or download could have its backend shut down under it and
fail. A multipart upload used the backend it started on throughout, so
failed if the client paused for more than 5 minutes between parts, or
the auth proxy changed the user's secret.
Each request and multipart upload now holds the backend it uses until
it is finished with it. In-progress multipart uploads are aborted when
the server is shut down so their backends are released.
With --auth-proxy each access key ID may be mapped to a different
backend, but gofakes3 kept a single record of multipart uploads for
everyone. Any user could list the uploads of every other user with
ListMultipartUploads (including their upload IDs) and the parts of
them with ListParts.
With --disable-multipart-streaming the uploads are held by gofakes3,
whose upload IDs are sequential so easy to guess. Any user could
upload parts to, abort, or complete another user's upload - which
wrote it to the backend of the user completing it.
This updates gofakes3 to a version which can keep each user's uploads
private and uses the access key ID as the owner.
With --auth-proxy each access key ID may be mapped to a different
backend, but serve s3 found a streaming multipart upload by its upload
ID alone and stored object metadata by path alone.
A user who learnt another user's upload ID (for example from
ListMultipartUploads on a bucket of the same name) could upload parts
to, complete or abort that upload on the other user's backend. A user
with an object at the same bucket and key as another user was returned
the other user's metadata by HeadObject and GetObject and could
overwrite it.
Keep the uploads and metadata of each access key ID apart.
Refactor the state serve s3 keeps between requests - in-flight
multipart uploads and object metadata - so that it is reached through
a tenant struct for the user making the request. This makes it
possible to keep users' state apart when they are serving different
backends.
There is only one tenant for now, so there is no change in behaviour.
serve s3 keeps the metadata of the objects uploaded to it in memory,
with no limit on the number of objects, so memory use grew with every
object uploaded, and a client could use a lot by uploading many empty
objects with large metadata.
The new --metadata-max-objects (default 100000, 0 for no limit) limits
the number of objects whose metadata is kept, forgetting the metadata of
the least recently used first. As the metadata is only kept in memory
it is already lost when the server restarts; the modification time is
stored on the object itself so is never lost.
When an abandoned multipart upload expired after --multipart-expiry,
gofakes3's record of it was left behind, so it stayed in memory and in
ListMultipartUploads until the server was restarted.
That record is now removed too.
This picks up these fixes to serve s3 from gofakes3:
- XML request bodies (CompleteMultipartUpload, DeleteObjects) bigger
than 10 MiB are rejected rather than read into memory.
- DeleteObjects requests with more than 1000 keys are rejected.
- CompleteMultipartUpload checks the upload exists before reading the
request body.
- With --disable-multipart-streaming, parts are no longer allocated at
their declared size before the data arrives.
- Browser POST uploads hold at most 1 MiB in memory and are limited to
5 GB.
It also brings a SlowDown error, so the SlowDown replies serve s3 sends
when a part waits too long for buffer space, or too many multipart
uploads are in progress, now have status 503 as S3 does rather than
500.
There was no limit on the number of multipart uploads in progress, and
each one holds server resources until it completes or expires - once it
has data, a VFS upload and the buffers of the remote's own upload - so a
client could use unbounded memory by starting many uploads.
The new --multipart-max-uploads (default 1000, 0 for no limit) limits
the streamed multipart uploads in progress. Starting another fails with
SlowDown, which S3 clients retry, before anything is created on the
remote.
With --multipart-expiry 0 nothing ever cleans up abandoned uploads, so
they go on counting towards the limit until no new upload can be
started. A warning is logged at startup when both are set.
Starting a multipart upload opened its VFS upload straight away, which
allocates a 100 KiB buffer (--streaming-upload-cutoff) and starts a
goroutine and a transfer, all held until the upload finished or was
expired after --multipart-expiry (default 24h). A client could make the
server hold about 100 KiB per tiny CreateMultipartUpload request.
The VFS upload is now only opened when the first part is written to it,
so an upload with no parts holds none of these.
--multipart-streaming-buffer-limit only limits the memory buffered for
parts arriving out of order in each upload, so a client could use as
much memory as it liked by starting more uploads.
The new --multipart-streaming-buffer-total (default 1G, 0 for no limit)
limits the memory buffered by all uploads together. A part which would
take either over its limit waits as before. Since the part the stream
needs next is never buffered, every upload can still make progress.
Every multipart part was buffered in memory in full before being written
to the backend, even the part the stream needed next. There is no limit
on part size, and several concurrent requests for the current part were
each admitted, so a client could make the server hold as much memory as
it sent, whatever --multipart-streaming-buffer-limit was set to.
The part the stream needs next is now written straight through to the
backend as it arrives, so only parts which arrive ahead of their turn
are buffered, and those always within the limit - a part bigger than
the limit waits until it is the next part rather than being admitted
when the buffer is empty. Further requests for the part being streamed
wait, holding no memory, and are then checked against it.
Since what reaches the backend can't be taken back, a part whose body
fails part way records how far it got, and the client's retry of the
part, which must have the same content, carries on from there. Such an
upload can't be completed without the rest of the part.
The body is read to its end, so that an error it only reports there -
as the Content-MD5 check does - fails the part rather than being
missed, and a body longer than declared is rejected. A part rejected
once it has been streamed leaves data which can be neither completed
nor replaced, so the whole upload is failed and cleaned up and the
client starts again.
A part waiting for room in the out-of-order buffer waited forever,
ignoring the request being cancelled, and a waiting part stopped its
upload being expired, so a client could hold requests, goroutines and
uploads open indefinitely.
The wait now ends when the request is cancelled, and after a minute the
part fails with SlowDown so the client retries it later.
A part uploaded again after it had been streamed to the backend (a
client retrying after a timeout) was buffered in memory in full just to
compare it with the original, and the buffer limit didn't apply to it.
Such a re-upload is now hashed as it is received and compared with the
original without being buffered.
The out-of-order multipart buffer was charged the declared length of
each part, but a buffered part occupies whole 1 MiB pool pages however
small it is. A client could therefore send up to 10000 one-byte parts
ahead of the part the stream needed and pin about 10 GiB of memory in a
single upload while --multipart-streaming-buffer-limit saw only 10 KB.
Each buffered part is now charged the whole pages it occupies.
The headers set with the `headers` option were sent to any host the
server redirected to, as net/http only strips Authorization and Cookie
on a cross domain redirect and copies every other header onto each
hop. A server which could be made to redirect to another host would
therefore receive any credential carried in a custom header such as
X-Auth-Token.
Strip the configured headers from a redirect hop once the chain has
left the configured host (a different hostname, subdomain or port)
using rest.StripHeadersOnCrossHostRedirectFn. This logs at debug level
when it happens so a 401 from the redirect target can be diagnosed.
The `auth_redirect` option is unchanged as it is an explicit opt-in to
keep credentials across redirects.
The Authorization header set from `user`, `pass` and `bearer_token` is
deliberately left to net/http's policy, which strips it on a change of
domain but keeps it for a subdomain or a different port, as servers
which redirect to a subdomain rely on that today.
The manual release verification steps checked the signature with gpg
--verify and then checked the binaries against the whole SHA256SUMS
file. gpg only verifies the clearsigned part, so an unsigned hash line
added outside it would be used by sha256sum -c or by rclone hashsum -C.
The instructions now extract the signed hashes with gpg --decrypt and
check the binaries against those only.
See: GHSA-w33h-3qgq-8mv8
rclone selfupdate checked the PGP signature on the clearsigned block
of SHA256SUMS but then looked up the archive hash in the whole
downloaded file. Unsigned data can appear in that file without
invalidating the signature. A hash line placed before or after the
signed text was accepted, allowing whoever controlled the download to
install an arbitrary unsigned archive.
The hash is now only looked up in the signed text.
Affects stable (non --beta) selfupdate since v1.55.0. --beta updates
are not signature checked.
The transport strips the headers set with --header once a redirect
chain leaves the originally requested host, but the host comparison
ignores the URL scheme. A server which redirects https://host:8443/
to http://host:8443/ keeps the same host and port, so the headers
were added to the plaintext request and could be read by anyone on
the network path.
The redirect chain walk now also treats a hop which downgrades the
original HTTPS request to HTTP as leaving the host, so the headers are
removed from that hop and every one after it.
The transport also logs at debug level when it removes the headers,
for either reason, so that a resulting 401 from the redirect target
can be diagnosed.
Dir.Create, Dir.Mkdir and Dir.Rename joined whatever name they were
given onto the directory's path. A name of "..", "." or one containing
a "/" isn't a child of the directory, and at the VFS root
path.Join("", "..") preserves the ".." so the new node's path pointed
outside the root. Backends which join the Fs root with the remote then
resolved it to a location above the directory the VFS was created on.
serve nfs was exploitable through this. The go-nfs library refuses a
name of exactly "." or ".." in MKDIR only, doesn't check the name in
CREATE, RENAME or SYMLINK, and never refuses a name containing a "/".
serve webdav was exploitable in the same way and has its own fix. The
other serve front-ends rely on their own or their protocol library's
path cleaning to keep such names out of the VFS.
Reject such names with EINVAL in the VFS itself, so that every
front-end and mount is protected regardless of what the layer above it
checks. This is the fix for serve nfs and defence in depth for the
others. The serve webdav check is kept as the front-end is the trust
boundary and gives the client a clear error. The source name of a
rename is only used to look the node up so it isn't checked.
VFS.Mkdir of the root, which previously went through Dir.Mkdir with an
empty name and added a nameless child node to the root, now makes the
root directory on the backend directly, still honouring --read-only.
serve webdav and serve ftp made directories through Dir.Mkdir rather
than VFS.Mkdir, so they are switched to VFS.Mkdir to keep "MKCOL /" and
"MKD /" working.
A request path with a ".." element, such as "MKCOL /..", was passed
straight through to the VFS. At the VFS root path.Join("", "..")
preserves the "..", so a directory node with the path ".." was added
under the root and every path built beneath it started "../". Backends
which join the Fs root with the remote before encoding it - such as
webdav, ftp, sftp, http and memory - resolved that "../" away, letting
an unauthenticated client create, overwrite, copy, move and delete
objects and directories outside the directory rclone was told to serve.
Moving a directory or file from outside the served tree into it also
made it readable.
Reject any request with a "." or ".." element in the request path, or
in the Destination header of COPY and MOVE, with 400 Bad Request. The
check is made once in ServeHTTP rather than in each webdav.FileSystem
method so that the client gets a consistent error. Rejected requests
are logged at INFO level like other requests.
The webdav handler strips --baseurl from the Destination as a string
prefix, so "/base../x" reaches the FileSystem as "../x". The Destination
is therefore checked after stripping the prefix in the same way.
The resume start value returned by the NetStorage list API is joined
onto the endpoint URL as a URL reference to build the next listing
request. A value beginning with `//` is a network-path reference, so a
server (or an on-path attacker when `protocol = http`) which returns
`//other.example/path` could make rclone send the next request, signed
with the account secret, to a host other than the configured one.
Use rest.URLJoinRoot so the resume token can only change the path.
The netstorage backend joins the path after `remote:` onto the
configured host URL according to the URL standard. A path beginning
with `//` is a network-path reference under that standard, so a path
such as `//other.example/dir/` replaced the configured host (and port
and userinfo) rather than being treated as a path. Every request,
including the initial stat, was then signed with the account secret and
sent to the substituted host. The signature covers the request path and
action but not the host, so the substituted host received signatures
valid for replay against the real NetStorage endpoint for the duration
of their validity.
Use rest.URLJoinRoot, which refuses any root that changes the scheme,
host or userinfo of the configured URL.
The http backend resolves the path after `remote:` against the
configured url according to the URL standard. A path beginning with
`//` is a network-path reference under that standard, so a path such as
`//other.example/dir/` replaced the configured host (and port and
userinfo) rather than being treated as a path. The configured headers
were then sent to the substituted host on the initial HEAD request and
every request after it.
Use rest.URLJoinRoot, which refuses any root carrying a scheme, host
or userinfo of its own, to join the path onto the configured url.
Add an integration test which starts a listener on localhost, makes
the remote with roots naming that listener and checks that nothing
connects to it. The listener replies with junk so a backend which does
connect fails at once rather than retrying a closed connection.
rest.URLJoin resolves its path as a URL reference according to the URL
standard, so a path beginning with `//` is a network-path reference
which replaces the host (and port and userinfo) of the base URL. That
is often the wrong behaviour.
Add rest.URLJoinRoot which joins an unescaped root onto a base URL and
refuses any root which parses with a scheme, host or userinfo of its own.
Before this change the basic, htpasswd and auth proxy authentication
middlewares let every OPTIONS request through without checking
credentials so that browser CORS preflights, which can't carry
credentials, would succeed.
However in serve webdav the OPTIONS request is answered by the WebDAV
handler whose Allow header depends on whether the path is an existing
file, an existing directory or doesn't exist, so an unauthenticated
client could discover the existence and type of any path on the served
remote even with --user/--pass or --htpasswd set.
After this change he authentication middlewares no longer special-case
OPTIONS. Instead the CORS middleware, which runs before authentication
and owns the --allow-origin setting, answers genuine CORS preflights
itself with an empty 200 carrying the CORS headers only when
--allow-origin is set.
All other OPTIONS requests need credentials like any other method, so
clients which probe with OPTIONS before authenticating will get a 401
challenge and must retry with credentials.
When pCloud redirects the browser back to rclone's local OAuth
callback server it adds a `hostname` parameter saying which regional
API host (api.pcloud.com or eapi.pcloud.com) the account lives on.
rclone used this for the token exchange and saved it in the config as
the host for all of the remote's later traffic, without checking it.
This only accepts a well formed hostname within pCloud's own domain,
lower cased, failing the config with an error otherwise, and adds a
unit test for the check. This is defence in depth for GHSA-hvpr-p4pv-4f46
The local web server which receives the OAuth callback during
`rclone config` and `rclone authorize` checks that the `state`
parameter matches the random value rclone generated when starting the
flow. The pcloud backend opted out of this check with the
`StateBlankOK` option in lib/oauthutil, which made the local server
accept a callback with no state at all.
The callback URL (http://127.0.0.1:53682/) is fixed and well known,
so while a pcloud OAuth flow was in progress any web page open in the
user's browser could complete it with a single cross-site request
carrying an attacker's authorization code. This is a CSRF
(CWE-352) on the auth callback.
This commit removes the `StateBlankOK` option so that every callback
must carry the correct state and adds a test that the local auth
server rejects callbacks with a blank, missing or wrong state.
The option was added in May 2020 (#4210) because pCloud briefly
omitted the state parameter from its redirect. The callbacks recorded
five weeks later when EU region support was added show both
api.pcloud.com and eapi.pcloud.com returning `state=` again, so the
exemption has been unnecessary since then.
See: GHSA-hvpr-p4pv-4f46
When reading a directory with --links the VFS skipped entries called "."
and ".." before removing the ".rclonelink" suffix, so an object called
"...rclonelink" (or "..rclonelink" or ".rclonelink") produced a directory
entry called ".." (or "." or "") whose path resolved to the parent
directory. The same happened for names added back from the VFS cache.
The check is now done after the suffix is removed.
See GHSA-55qp-jrwr-x827
In --links mode the local backend checked that a source name stayed inside
the destination root and only then removed the ".rclonelink" suffix from the
resulting OS path. A source entry called "...rclonelink" passed the check and
was then trimmed to "..", so the object addressed the parent of the
destination the user chose.
Writing the content of such an entry was already refused, as it is done
through an os.Root, but metadata was not: with --metadata rclone rewrote the
timestamps, ownership and extended attributes of the directory above the
destination from values supplied by the source.
The suffix is now removed from the remote name before it is joined to the
root, so the containment check applies to the name that is actually used. A
translated link must also leave a real native file name once the suffix is
removed.
A regular file whose name is exactly ".rclonelink" (or "..rclonelink" or
"...rclonelink") is not a valid translated link and is now skipped in
directory listings.
In --links mode the local backend removed the ".rclonelink" suffix from
directory names as well as file names when building the OS path. Only files
are ever translated into symlinks, so a directory called "sub.rclonelink"
addressed the sibling "sub" instead: listing it logged "Failed to read link
size", its metadata was read from the wrong place, and setting metadata on
it with --metadata failed or landed on the sibling.
Directories now use their name as is, whatever suffix it has, in the same
way DirSetModTime already does.
Every file download sends the live account authorization token and,
when configured, the SSE-C customer key headers to the download host.
With --b2-download-url that host is a CDN or proxy the user does not
operate, and the http client followed its redirects with Go's default
policy:
- Go only strips Authorization when the redirect target is not the same
host or a subdomain, and never looks at the scheme, so a same-host
HTTPS to HTTP redirect replayed the full-capability B2 token in
cleartext.
- Go has no idea that the X-Bz-Server-Side-Encryption-Customer-* headers
carry the customer encryption key, so it forwarded them to any
redirect target including a different host.
The B2 client now uses rest.StripHeadersOnCrossHostRedirectFn, so it
refuses to follow a redirect which downgrades HTTPS to HTTP and strips
the Authorization and SSE-C headers from any redirect which leaves the
host and port of the original request, matching the protection already
given to the S3, WebDAV and HTTP backends.
Fixes GHSA-gpw7-77cq-9f72
The s3 and http backends each carried their own copy of the same
CheckRedirect policy. This moves it into lib/rest as
StripHeadersOnCrossHostRedirectFn, so that any backend which sends
credentials in headers to a host it does not control can use the same
policy. It logs at debug level when it strips headers so that a
resulting 401 from the redirect target can be diagnosed. A header
starting with "*" is matched without canonicalising, as that is how
SetHeader and Opts.ExtraHeaders send it.
This changes the s3 policy in two edge cases:
- It compared the raw URL host string, so a redirect to the same host
in different case or with an explicit default port stripped the
credentials. It now uses SameHost like the http backend, which treats
those as the same host.
- The downgrade check compared each hop with the previous one. It now
compares with the original request like the other backends, so a
chain which starts on plaintext HTTP may return to it.
The behaviour tests move to lib/rest. The backend tests which check the
policy is wired into each client are kept.
- CVE-2026-97032: net/http: HTTP/2 server crash due to HPACK encoder race
- CVE-2026-78659: net/http: HTTP/2 server memory exhaustion due to Trailer headers
- CVE-2026-97031: crypto/tls: reject malformed ECH outer extension references
- CVE-2026-94444: cmd/go: checksum bypass for golang.org/fips140
- CVE-2026-94447: cmd/go: checksum database bypass for golang.org/toolchain
- CVE-2026-94448: html/template: reset context tracking on consecutive template expressions
- CVE-2026-97030: html/template: recognize |yield| as regexp preceder keyword
- CVE-2026-94440: net/textproto, mime/multipart: memory limit bypass when parsing MIME headers
- CVE-2026-56866: net/http: HTTP/1 client connection desynchronization after CONNECT rejection
- CVE-2026-94439: net/http: HTTP/1 server connection desynchronization after 2xx CONNECT response
- CVE-2026-78669: net/http: excessive CPU consumption from repeated initial window changes
- CVE-2026-78660: net/http: HTTP/2 transport accepts malformed framing-related headers
- CVE-2026-56857: os: Root.Mkdir(All) can follow junctions out of the root on Windows
- CVE-2026-78667: net/http: lack of limit on size of parsed Range headers
- CVE-2026-78663: net/http: double flow control refund on HTTP/2 server streams
This also updates the go1.26 test job to go1.26.9 which contains the
same fixes.
Nextcloud chunk uploads currently return a retryable error when a
chunk PUT fails. The outer low-level retry then restarts the whole
upload, removing the temporary upload directory and sending previously
accepted chunks again.
Retry each chunk in place using the configured low-level retry budget,
rewinding its repeatable body before each attempt. Once that budget is
exhausted, prevent the outer retry from restarting the entire object.
WriteChunk checksums the chunk with io.Copy before sending it. For a
reader that implements neither WriterTo nor a memory backed Read - a
local file opened by a multi-thread copy - io.Copy falls back to its own
32 KiB buffer, so the source is read in 32 KiB pieces and the disk sees
thousands of requests per second where it should see hundreds.
Hand io.CopyBuffer a 1 MiB buffer instead. It is allocated for the
checksum pass rather than taken from the pool, where it could wait for
ever on a --max-buffer-memory below the page size. Readers that do
implement WriterTo, such as pool.RW, keep their fast path and don't get
a buffer.
FsOpenChunkWriter wrote three chunks of its own fixed sizes whatever
ChunkWriterInfo.ChunkSize the backend returned. That happened to be
the size the s3 backend uses by default, so the test passed there, but
a chunk writer which works out where a chunk goes from its number needs
every chunk but the last to be exactly the size it asked for - as a
multi-thread copy writes them - and the test could neither exercise nor
catch that.
Cut the file into chunks of info.ChunkSize, the last one short, still
written out of order. Where the backend lets the test set its chunk
size, ask for 5 MiB first so the file takes several chunks. Where it
doesn't and the chunks leave the file in fewer than three, start again
with a file of two full chunks and a short one, for chunks up to
128 MiB.
A failed delta poll used to wipe the stored resume token: changeNotifyRunner
returned a named nextDeltaToken which stayed empty on error, and the caller
stored it unconditionally. Later polls then asked for /root/delta?token=
instead of resuming, so notifications stopped until the mount was restarted.
changeNotifyNextChange also called the API without the pacer, so a single
transient 503 was enough to trigger this.
While fixing that, walk @odata.nextLink as well. A change set spanning more
than one page returns nextLink without a deltaLink, and the token parsed from
the missing deltaLink was empty - changes were dropped even when no error
occurred.
Fixes#10009
Before this change the parent process kept the remote control and
metrics servers it started in initConfig listening, so the daemon child
could not bind the same addresses and exited.
After this change the parent shuts those servers down just before
daemonizing, and the child starts them as usual.
A symlink in an archive was listed as an ordinary file holding the link
target by the zip backend, and dropped entirely by the squashfs one.
Symlinks are now skipped unless -l/--links is in use, as they are on the
other backends. With that flag they are listed with the .rclonelink
suffix and read back as the path they point at, so they can be copied to
a destination which supports symlinks.
This changes what the zip backend lists by default: a symlink used to
appear as a small file whose contents were the link target, and is now
skipped with a NOTICE naming the flag.
Fixes#9569
If a backend's ListR returned a directory after objects inside it,
the directory was listed twice when using --max-depth or filters. This
happens with azureblob on storage accounts with a hierarchical
namespace.
When a backend saved config, for example a refreshed OAuth token, for
a remote which wasn't in the config file, a section with just that
value was created in the config file for environment variable remotes
and the config file was rewritten for on the fly remotes.
Config is now only saved if the remote is in the config file.
Listing "dir/" on a storage account with a hierarchical namespace
returns the file "dir" if it exists, so a file was listed as a
directory containing itself.
On storage accounts with a hierarchical namespace, directories can be
read as blobs so NewObject on a directory returned fs.ErrorNotAFile
rather than fs.ErrorObjectNotFound.
Storage accounts with a hierarchical namespace (ADLS Gen2) have real
directories. Rclone could not remove these - with directory_markers it
failed with "InvalidUri" and without it rmdir silently did nothing.
Directories are also left behind when the last file in them is
deleted.
This adds the hns option which is read from the storage account if
not set and saved in the config file. If set, rclone:
- supports empty directories whatever the directory_markers setting
- creates directories with Mkdir
- removes directories with Rmdir
- doesn't create directory markers for parent directories on upload
With --auth-proxy, when several logins with the same credentials
arrived together and none was in the cache yet, as when an FTP client
opens several connections at once, each one ran the auth proxy program
and took its own reference to the backend. Only one of them was
remembered, so the others were never given back and the backend was
never shut down.
Logins which arrive together now share one run of the auth proxy
program and one reference to the backend.
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. With --vfs-cache-mode writes or full, files
are uploaded to the backend after the transfer which wrote them has
finished, which didn't count as a use. So an upload which hadn't
finished 5 minutes after the user's last command (or disconnection for
serve sftp) was stopped, leaving the file in the VFS cache but not on
the backend.
The backend is now kept until it has no files open for write and
nothing in the VFS cache waiting to be uploaded.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
Busy returns true if the VFS has files open for write or files in the
VFS cache which are open or waiting to be uploaded. It is for code
which wants to shut a VFS down only once that wouldn't lose any work.
An entry is expired when it hasn't been used for a while, but the user
of the cache may know it is still doing something in the background.
SetCanExpire sets a function which is asked before expiring an entry
and can keep it in the cache until it is ready to go.
The function is called without the cache locked as it may take a
while, so an entry which is used in the meantime isn't expired.
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so an upload or download lasting longer than that had its backend
shut down under it and failed.
Each request now holds the backend it uses until it has been served.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so a download lasting longer than that had its backend shut down
under it and failed.
Each request now holds the backend it uses until it has been served.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each FTP command counted
as a use, so an upload or download lasting longer than that had its
backend shut down under it and failed with "context canceled".
Each FTP command now holds the backend it uses until it has finished,
which for a download is when the file is closed.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
Add a new backend for dosya.dev, a cloud file storage, sharing, and
synchronization platform with workspace-based multi-tenancy.
Features:
- Server-side copy, move, rename (files and directories)
- Purge (recursive directory delete)
- Recursive listing (ListR) for --fast-list
- Multipart upload for large files (>10MB)
- Public link sharing via rclone link
- About (workspace storage quota)
- MIME type detection
- Empty directory support
- Workspace-based multi-tenancy (one remote per workspace)
- Download via presigned R2 URLs with Range header support
Co-authored-by: firatkaya <firat@netiket.com.tr>
When an upload failed during the transfer, the partial file was
removed but its handle was never closed. The server kept the deleted
file open, so its space stayed allocated until the pooled connection
was closed, which in rcd, mount or serve could be until rclone exited.
This closes the handle before removing the partial file.
Tests in fs/operations, fs/sync, cmd/bisync, cmd/gitannex and vfs are
run by the integration tests against every backend, so they need to
cope with backends with limited functionality, ideally by skipping on
the Fs.Features() flags.
Files.com now treats the replacement characters rclone uses for / and
\ in file names (／ and ＼) as if they were the originals, and the
names ． and ．． as if they were . and .., and rejects them all with
"Invalid path". There is no other way of encoding these names, so
document the restriction and ignore the integration tests for them.
When an upload was stopped part way through by --max-transfer, HiDrive
kept the part of the file it had received, leaving a truncated file at
the destination, or replacing an existing file with a truncated one.
This was introduced in e8f421d28 which accounted the buffered start of
each upload as it was sent rather than as it was read, so the transfer
limit could abort the request which creates the file half way through.
Account the requests which create or replace a file in one go as they
are buffered once more, so the limit is hit before anything is sent.
The chunks of larger uploads are still accounted as they are sent.
ownCloud 10.16 answers a PROPFIND for a path which doesn't exist with
a 207 Multi-Status response, rather than a 404, with a body which
starts a multistatus document then appends a Sabre NotFound error to
it. This isn't valid XML so rclone failed with
read metadata failed: XML syntax error on line 2: expected attribute name in element
whenever it was pointed at a directory which didn't exist yet.
Detect the NotFound error in the response and treat it as a 404.
Koofr refuses to download the HTML listings written by the index tests
with "FileBlocked: File download restricted due to possible dangerous
content" and OpenDrive refuses with 403 Forbidden, so the tests can't
read back what they wrote. The index tests which write other formats
still run.
With meta_format = none, listing a directory which contained the
temporary chunks of an interrupted upload, but no completed chunks for
that file, panicked with "invalid chunked object".
The listing made a placeholder object for the file on seeing a
temporary chunk, which then had no chunks in it when it was validated.
Only make the placeholder on seeing a data chunk, so files which have
nothing but temporary chunks are ignored as they are when metadata is
in use.
- Only check the MIME type of the listings on backends which declare
both ReadMimeType and WriteMimeType, and ignore its parameters, as
many backends report a MIME type they chose themselves or drop the
charset.
- Write the test file before turning --dry-run on. Chunker finishes
an upload with operations.Move, which honours --dry-run, so it left
temporary chunks behind which stopped the test directory being
removed and made the tests following fail.
- Don't check the transfer and delete counts on chunker, as moving
its chunks into place is counted in the same stats.
- Don't expect a new file to change the time of the directories above
it on backends which can't set modification times, as it may get the
same time as the files already there.
The Blob Listing with Apache Arrow feature is now public and shipped
in azblob v1.8.1, so the temporary backend/azureblob/arrowlist package
which implemented it on top of the previous SDK has been removed and
the backend now uses the SDK's own listing pager with ResponseFormat
set to Arrow.
As the SDK client handles every credential type this also makes Arrow
and parallel listing work with connection_string auth, and the
use_arrow_list and list_parallelism options are no longer hidden.
Pointing the archive backend at a file inside a squashfs image, for
example `rclone cat :archive:image.sqfs/dir/file.txt`, listed every
file in the directory containing it instead of just that file. The zip
archiver already behaved correctly.
The squashfs archiver now remembers the file the root points at and
only exposes that one, as zip does. Its Root() includes the file so
that the fs cache does not hand back the Fs for the whole directory in
its place.
With -l/--links a .rclonelink object is buffered in memory to become
the target of a symlink. The read was unbounded, so a hostile or
corrupt source serving a large .rclonelink object made rclone use that
much memory and then log the whole body in the resulting error.
A symlink target can never be longer than a path, so the read now stops
at 128 KiB and anything longer is refused without retrying.
Deleting a selection finished with "Successfully deleted all items!",
which claimed more than had happened - only the selected entries were
removed. It now names the number of items deleted.
The screen was not redrawn until the whole selection had been deleted,
which made the UI look hung, so a progress box is drawn before each
deletion.
Fixes#9516
The docs describe a duration as a sequence of numbers each with a unit
suffix, including d, w, M and y, but a sequence that used one of those
units, such as --max-age 1d12h, failed with a confusing error about
parsing it as a date. Only a single number with one of them, like 1.5d,
or a sequence of the time.ParseDuration units was accepted.
Parse such sequences before falling back to dates, rejecting ones too
long to represent.
Add an opt-in preflight that excludes only guaranteed tracked renames
from the max-delete count. Reuse normal bisync listing metadata,
validate strategy and flag conflicts early, and skip preflight work
when --force bypasses the safety check.
The max_delete_track_renames scenario uses the default hash
track-renames strategy, which requires a hash common to both paths.
Skip it on remote combinations without one, such as crypt, instead of
failing the integration tests.
Fixes#8685
Move strategy parsing, capability checks, and matcher behavior into a
reusable package without changing sync behavior. Add a lightweight
candidate API so callers can count guaranteed matches without
retaining filesystem objects.
Setting the modtime of a closed file refreshed the cache item's
fingerprint before the new modtime had been applied to the remote
object. The next open then saw a fingerprint mismatch, logged "removed
cache file as stale (remote is different)" and downloaded the whole
file again, even though only the modtime had changed.
Refresh the cache fingerprint again once the remote modtime has been
set.
When a file was reopened within the --vfs-handle-caching grace period
(default 5s) after its remote fingerprint changed - for example after a
touch or any other modtime change - _checkObject removed the stale
cache file and open recreated an empty one, but nothing sized it. The
next GetSize stat'd the empty file and set the item size to 0, so
ReadAt's _ensure clamped the request to nothing and skipped the
download, then the size check zero-extended the file and returned the
zeros. Reads through a mount returned the correct length but zeros for
the first read (128 KiB through FUSE).
This is easy to hit with git on a --vfs-cache-mode full mount: git
freshens pack files with utime, and a concurrent reader then sees
"not a GIT packfile".
Size the recreated cache file from the object before opening it, as a
normal open does.
Shutdown was calling expiryTimer.Stop without holding ts.mu, so it could
panic if OnExpiry had not created the timer yet, and raced the write in
OnExpiry. Hold the lock and skip Stop when the timer is still nil.
Fixes#9980
Reading an object in an archive storage class failed with "Object in
GLACIER, restore first" even when the object was in DEEP_ARCHIVE or in
an Intelligent-Tiering archive access tier.
Use the storage class and access tier from the InvalidObjectState
error. The storage class is optional in that error, so fall back to
the storage class from the listing or HEAD, and then to GLACIER as
before.
Until the first time slot of the week, the timetable used its last entry
as the limit carried over from the week before. That is only the latest
slot of the week when the entries are in order, so a timetable written
weekend first, like "Sat-00:00,off Mon-00:00,1M", limited Sunday to 1M
instead of leaving it unlimited.
Carry over the latest time slot of the week instead.
When listing with --s3-versions or --s3-version-at the storage class
of each object was dropped, so it read as STANDARD unless the object's
metadata was fetched separately. This meant backend restore skipped
objects in GLACIER or DEEP_ARCHIVE with "Not GLACIER or DEEP_ARCHIVE
or INTELLIGENT_TIERING storage class", and lsf --format T showed the
wrong tier.
This happened because ObjectVersion.StorageClass has a different type
from Object.StorageClass so the generated setFrom helper does not copy
it. Convert it explicitly after the setFrom call.
The Docker image had no /etc/mime.types, so MIME types came only from
Go's built-in table plus rclone's small extra list, and many
extensions uploaded as application/octet-stream.
This installs Alpine's mailcap package to fix the problem.
To find the latest patch release of a minor version, for example
`rclone selfupdate --version 1.75`, selfupdate searched the listing of
downloads.rclone.org for href="./vX.Y.Z/". The leading ./ is a detail
of how Caddy's file server writes its links. A listing which linked to
the same directories as "vX.Y.Z/", which is an equally valid relative
URL, made selfupdate fail with "could not find the minor release".
This makes the ./ optional in the pattern, so selfupdate no longer
depends on which program wrote the listing, and takes the version from
a capture group rather than from fixed offsets into the match.
The listings written by rclone index now include the ./ for the
benefit of rclone versions without this fix, so this is to remove the
dependency for the future.
downloads.rclone.org is about to be served from static listings
written by rclone index instead of by Caddy's file server. Released
versions of rclone selfupdate find the latest patch release of a minor
version by searching that listing for href="./vX.Y.Z/", as written by
Caddy. rclone's listings linked to "vX.Y.Z/" without the ./ so `rclone
selfupdate --version X.Y` would have failed with "could not find the
minor release" for every rclone already installed.
Caddy prefixes every link with ./ so that a name with a colon in its
first path segment is not read as an absolute URL with a scheme (RFC
3986 section 4.2). rclone was already safe from that as url.URL.String
adds the ./ but only to names which need it, so links to plain names
had no prefix.
This prefixes all the links with ./ as Caddy does. It changes the
output of serve http, serve webdav and the rc server as well as rclone
index, since they share the code. The links resolve identically, and
it keeps the listings consistent with each other.
Now every URL has the prefix, the caddy.json template no longer needs
to add it.
After a sync which changed a known set of files there is no need to
walk the whole remote. --changed PATH, --changed-from FILE and
--changed-combined FILE tell rclone index what changed, and it
re-indexes only the directories containing those paths and their
ancestors, each with one non-recursive listing.
The listing now shows the folder path with clickable breadcrumbs
above a card containing the entries:
- A summary of the directories, files and total size
- The directory and file counts are toggles
- The search box sits beside the Name heading (focussable with /)
- Sizes right aligned, empty columns are gone and the icons are simpler.
- The colours are CSS variables
The template data is unchanged so custom templates still work, and
the zip download links and ?sort= parameters work as before.
This makes directory listings for buckets and other remotes served as
static websites, for example S3 website endpoints or R2 behind
Cloudflare, so they can be browsed without directory listing support
on the host.
Index writes a directory listing (e.g. index.html) into every
directory of a remote so it can be browsed when served as a static
website. It is also available over the rc as operations/index.
It works like sync. It walks the remote once, renders the listings in
memory and compares them with the existing ones by size and hash from
the listing, or by reading them where the backend has no hash.
Listings in directories which contain nothing else are deleted on
backends which can't have empty directories.
Listings can be written in the serve http HTML format, the lsjson
format, Caddy's browse JSON format, or from a user template. A second
rule set (--index-include and friends) controls which directories get
listings, --dir-time controls the time shown for directories and
--no-modtime avoids reading modification times altogether.
- .Static hides the "up" link at the root
- .SetLinkIndex makes directory links point at an index doc
- .Render writes the listing to an io.Writer
- .Path, .IsRoot, .UpLink, .NumDirs, .NumFiles, .TotalSize and .MimeType.
- Sorting is now stable so the rendered output is deterministic
Clicking the Name, Size or Modified heading now sorts the listing in
the browser and clicking again reverses it, rather than reloading the
page with ?sort= and ?order= parameters. Names sort naturally so
v1.9.0 comes before v1.10.0, and directories stay first when sorting
by name.
The ?sort= and ?order= parameters still work for the initial order and
are shown in the heading, so existing links and custom templates are
unaffected.
MinIO no longer publishes images on quay.io (nor Docker Hub or ghcr.io)
so pulling quay.io/minio/minio returns "unauthorized". pgsty/minio is a
maintained community fork with the same entrypoint layout.
ENETUNREACH and ENETDOWN were not in the list of retriable errors on
non-Windows platforms, so a single failed connection aborted the
transfer instead of being retried as a low level retry. The Windows
list already includes the equivalent WSAENETUNREACH and WSAENETDOWN.
This is easy to hit on a host with IPv6 enabled but no IPv6 route: if
the A lookup fails transiently while the AAAA lookup succeeds, the only
address to dial is IPv6 and the connect fails with ENETUNREACH. A retry
resolves again and normally succeeds.
BwPair.String printed a sub-KiB bandwidth as a bare number, and Set
reads a bare number as KiB, so a rate under 1 KiB grew by a factor of
1024 whenever it went through a string. rc core/bwlimit reports the
current rate in that format and accepts it back, so reading the limit
and setting it again raised it.
63c4fef27 fixed the same corruption for config values by suffixing bare
numbers with B inside Option.String. Move that into a SizeSuffix method
and use it from BwPair.String as well.
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
Copying an object onto itself, as clients do to replace its metadata,
reported success when the object didn't exist, and copying a missing
object to a different key failed with an internal error. Both now fail
with NoSuchKey as S3 does.
With --etag-hash auto the hash for the ETags was chosen once from the
remote serve s3 was started with. With --auth-proxy there is no such
remote so serve s3 crashed on startup, and when started by the rc it
was the wrong remote, so users whose backends lacked that hash got no
ETags.
The hash is now chosen from the backend of the user making each
request.
The metadata of every object uploaded was kept in memory for as long as
the server ran, even after the object was deleted, so it grew without
limit and the metadata of a deleted object reappeared on any object
later created at the same key other than through serve s3.
Deleting an object now forgets its metadata.
Listing a prefix with no delimiter walked the entire subtree below it into
memory and only then sliced out the requested page, so every page of a
listing cost a full traversal of the tree.
Walk the tree lazily instead, stopping as soon as the page is full and
skipping the subtrees an earlier page already returned. A page now costs a
number of directory reads proportional to the keys it returns rather than to
the size of the subtree, and ETags are computed only for the objects that
are actually returned.
Entries are emitted in the order their keys have in a flat keyspace, with a
directory sorting as if it carried its trailing slash, so that "a.txt" comes
before "a/b" as it does in a real S3 bucket. Without this a resumed listing
would silently skip keys across a page boundary.
When the last user of a VFS shut it down at the same time as a new
user of the same remote asked for one, the new user could be given the
VFS being shut down, with its cache and background tasks stopped.
Only a VFS which is still in use is now reused, otherwise a new one is
made.
A VFS is shut down when the last of the references to it from New is
given back with Shutdown. Hold takes another reference, for code using
a VFS it didn't create itself, which may otherwise be shut down under
it.
Rclone always asked for the delta listing of the whole drive and threw
away the items which weren't under the directory being listed.
The delta API now works on any folder, so ask for the delta listing of
the directory being listed instead, falling back to listing from the
root of the drive for drives which only support delta there.
When a backend rejected the multipart upload as too small, the subtest
skipped without removing its local source file, so the following
upload subtests failed their local listing checks.
Moving a directory can take longer than the API gateway allows, so the
request fails with a 520 or 502 error even though the move completes.
The retry then failed with "Folder ... was already moved to that
location (status 409)".
Treat that error as success.
The Internxt API serves listings from read replicas which lag behind
writes, so for a short while after files or directories are moved or
deleted they can still be listed in their old location.
This caused removing a directory which had just been emptied to fail
with "directory not empty" (eg when moving a directory without server
side directory moves or purging a directory) and a directory which had
just been moved to be found in its old location.
Remember the directories this process has moved or deleted and ignore
directory and file entries which contradict that when listing, finding
directories and checking a directory is empty before removing it.
Moving a file into a directory which had just been created failed with
"Not Found (status 404)", and moving a file over one which had just been
deleted (as sync does with --backup-dir and --suffix) failed with "A file
with the same name already exists in destination folder (status 409)".
The API checks moves against read replicas which lag behind writes, so
retry these errors until the replicas catch up.
The Internxt API serves lookups and listings from read replicas which
lag behind writes, so for a short while after a file is moved it can
still be returned from its old location.
When sync moved a file into the backup location and then uploaded its
replacement, the upload could find the moved file under its old name
and overwrite it. Overwriting renames the existing file by UUID and
deletes it once the upload succeeds, so the file that had just been
moved into the backup location was deleted.
Remember the files this process has moved or deleted for a minute and
ignore lookups and listing entries which contradict that.
With hard_delete set, deleted files and directories carried on being
listed for about a second afterwards because the Drime server doesn't
invalidate its cache of the parent folder listing when entries are
deleted forever. This made removing a directory straight after
emptying it fail with "directory not empty".
Moving an entry to the trash does invalidate the cache, so with
hard_delete set rclone now moves the entry to the trash first and then
deletes it forever.
When server-side copying to a destination which already existed, the
Drime server gave the copy the name "name (1)" and rclone only renamed
it if the source and destination leaf names differed. The existing file
was then deleted leaving the copy under the wrong name.
The server refuses to rename an entry to a name which is already in
use, so this removes the existing file straight after the copy, then
renames the copy whenever its name differs from the destination name.
The premiumize.me upload server has started truncating the multipart
file name at the first ";", so uploading "a;b.txt" created a file
called "a" and the upload then failed with "object not found".
Directory creation and renames still accept ";", so files whose names
contain ";" are now uploaded under a temporary name and renamed into
place. The encoding is left unchanged so existing files and
directories containing ";" remain accessible.
Shortcuts to shared items (and Personal Vault) are stored as remote
items pointing at another drive and listing them can fail with
"The provided drive id appears to be malformed". This aborts a
--fast-list listing of the whole drive.
rest.ReadBody read the whole response body into memory with no limit.
It is used by the default error handler and by many backends' error
handlers and small API calls, so a server which answered with an error
status and then streamed an endless body could make rclone allocate
memory until it was killed.
ReadBody now reads at most 10 MiB (the same limit drainAndClose
already uses to discard unread bodies) and returns an error if the
body is bigger than that. Every caller reads small API responses -
error bodies, status documents and upload tokens - so no legitimate
response is affected.
Reported by @manus-pi
When S3 returns URL-encoded keys from ListObjectVersions, URL encoding
can change their lexical order. This could make mergeDeleteMarkers
place a delete marker after older versions of the same key, so
--s3-version-at reported deleted objects as live.
Compare decoded keys while merging, while preserving the encoded keys
for the existing listing decode path.
Fixes#9948
Testing against OneDrive personal (free) and OneDrive for Business shows
- personal accounts now create versions on setting the modification
time and can delete them, so --onedrive-no-versions and rclone
cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS
See #9917
With --vfs-metadata-extension set, looking up the metadata file of a
file which was open for write and not yet uploaded caused a nil pointer
panic. Such a file has no object to read the modification time from.
Use the modification time of the VFS node instead, which is valid
whether or not the file has been uploaded.
TestDeleteFatalError set --max-delete before writing its files, so on
chunker, which deletes while uploading, the setup failed.
TestDirMoveMoveError and TestDirMoveContext test the core DirMove logic
with a wrapping Fs. This fails on remotes without Move (eg s3, memory)
and on those whose objects don't belong to the wrapped Fs (eg archive),
so they now only run on local.
findItem read the response status code when the lookup failed, so a
failure with no HTTP response panicked.
Thanks to @manus-pi for finding this problem.
DirMove discarded the error from the destination check and read the
response status code, so a failure with no HTTP response panicked.
Other errors were reported as the destination existing; they are now
returned.
Thanks to @manus-pi for finding this problem.
Rmdir and Purge read the response status code before checking for an
error, so a failed DeleteFolder call with no HTTP response (a network
error, or purging the root which fails validation) panicked.
Thanks to @manus-pi for finding this problem.
The request log printed the escaped URL, so non-ASCII file names showed
up as long runs of %XX escapes. Log the unescaped URL path instead, as
the other serve commands do.
VFS.AddVirtual always added a file entry to the directory cache, even
when called with isDir set, so a virtual directory showed up as a file
and nothing could be added inside it.
See #9310
With -L/--copy-links a symlink pointing to one of its parent
directories was followed until the OS gave up with "too many levels
of symbolic links" 40 levels deep. As every looping symlink multiplies
the listing, a small tree with three such symlinks listed millions of
entries.
When a followed symlink points to a directory, compare it with the
directory being listed and each of its parents using os.SameFile and
if it matches report an error and skip it, the same way as a circular
symlink. A symlink to a sibling directory is still followed. Loops
excluded by the directory filters are skipped quietly as before.
The default for --smb-user was the name of the user running rclone
config, and rclone does not write defaults to the config file, so
entering your own user name left it out. A remote made that way then
logged in as whoever ran it later, e.g. root under systemd.
Make the default blank and look up the current user when the remote is
used, as the ftp backend does. Existing configs work as before.
When the source matched a file in --copy-dest, copyDest server-side
copied it over an existing destination which differed, so sync, copy
and copyto with --copy-dest could modify a file --immutable should have
protected.
Leave such a destination for the caller, which rejects it with
ErrorImmutableModified as for any other modified file.
--immutable was only enforced by sync, so copyto, moveto, copy and move
of a single file, and the operations/copyfile and operations/movefile rc
calls overwrote a destination which differed from the source.
Fail these with the same ErrorImmutableModified as sync, and reject
--no-check-dest with --immutable as sync does, since without checking
the destination it cannot be protected.
The box backend logged an int64 with %q which printed
%!q(int64=123) instead of the sequence ID.
serve s3 passed the message from gofakes3 as the format string, so
any % in it was interpreted as a formatting directive and the message
was mangled.
job/status returns the complete output of the job so it is as
sensitive as the call which started the job, e.g. config/dump started
with _async returns the config file including any secrets. It was
possible to read this without authentication where an authenticated
and an unauthenticated rc server share the same process, e.g. rclone
gui --rc.
An unauthenticated client can't start jobs on a server which needs
authentication, other than with the calls which don't need it, so
this shouldn't affect existing users.
Handle read the list of extra outputs without holding the mutex, so
calling AddOutput while logging caused a data race.
This reads the extra outputs once under the mutex.
The wait to obey a Retry-After error from the server used
time.Sleep() so it carried on sleeping when the context was
cancelled, e.g. by job/stop or --max-duration, and it also slept
after the last try when there was nothing left to retry.
Copying a file also logged the low level retry number counting from
0 rather than 1 as everything else does.
When the backend has no DirMove method, operations.DirMove moves the
objects one by one. If one of the moves failed the movers stopped
without receiving the rest of the objects, so DirMove blocked forever
once the channel filled up. This could be seen renaming a directory on
a mount of a backend without DirMove, such as s3.
The moves were also done with context.Background() instead of the
caller's context, so they didn't have the caller's config, couldn't
be cancelled and weren't counted in the caller's stats.
When a fatal error such as --max-delete being reached stopped the
deletions, the deleters returned without receiving the rest of the
objects, so whatever was sending them blocked forever once the
channel filled up.
For example "rclone delete --max-delete 1" on a directory with more
files than --checkers hung instead of returning the error.
The deleters now keep receiving objects without deleting them after a
fatal error.
Params.GetInt64 parsed string parameters with strconv.ParseInt(x, 10, 0).
A bitSize of 0 means int, which is 32 bits on 386, arm, mips and mipsle,
so any value outside the int32 range was rejected as out of range even
though it fits in the int64 the method returns.
Parameters reach the rc API as strings on the paths that matter here: the
rc server puts every URL query and form value into Params as a string, and
rclone rc turns each key=value argument into a string. So on a 32 bit build
operations/getfile could not be given an offset, count, head or tail beyond
2 GiB, and debug/set-soft-memory-limit could not be given a limit beyond
2 GiB, while the same commands worked on a 64 bit build.
Parse with a bitSize of 64 to match the declared return type. This is the
string half of the range checking that was added to the float64 branch of
the same function in 976d05e1.
Return the first non-EOF input error encountered while Rcat probes whether an
upload is small. Previously that error was ignored and the full probe buffer,
including bytes that were never read, could be passed to Put or PutStream.
- Reject uploads when object lookup fails for reasons other than "not found".
- Serialize uploads to the same object to prevent races between the
existence check and the write.
Gzip metadata is read from the wrapped remote and could contain an
invalid block size or incomplete block index. A range read could then
panic in the seekable gzip reader.
Validate the gzip sidecar invariants before constructing a reader so
malformed remote metadata returns an error instead of crashing rclone.
The start function was run inside an if condition, where bash ignores
errexit, so a failed docker run was not noticed. The script then
printed its connection details anyway and the test spent 100 seconds
trying to connect before failing with a message that hid the real
error.
Run start in a subshell with errexit on and check its status
explicitly so the failure is reported immediately with docker's
error message.
The minio/minio repository has been removed from Docker Hub so the
TestS3Minio test server could not be started and every Linux CI run
failed. quay.io/minio/minio still serves the final release so use
that instead.
OCI rejects a multipart completion request with no parts. Probe unknown-size
streams through a buffered reader and use a regular upload when the stream is
empty. Peek preserves a non-empty stream for the selected upload path.
Internxt stores a file as a (plainName, type) pair and never derives the
split itself, so it is a convention shared between clients. This backend
split at the final dot, storing and looking up ".bashrc" as an empty name
of type "bashrc", where the web, desktop, Linux and macOS clients all keep
the leading dot in plainName. List rebuilt the full name so such files
appeared, but NewObject looked them up by the split and missed them.
- Replaced the preUploadCheck function with findFile for better clarity and efficiency in checking file existence.
- Introduced splitNameExt to encapsulate name and extension parsing logic.
- Updated NewObject and Update methods to utilize findFile for improved file metadata handling.
- Enhanced error handling and reduced redundant code in file checks.
Use OCI BatchDeleteObjects to remove up to 1,000 objects per request when
purging. Include directory markers so purging a bucket leaves it empty for
deletion.
OCI Object Storage rejects NUL, carriage return and line feed in
object names. NUL is already encoded by rclone, while carriage returns
and line feeds need EncodeCrLf so rclone can represent those names
without issuing invalid OCI requests.
https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/managingobjects.htm
When rclone archive create is run without a destination the archive
should be written to stdout, but ArchiveCreate called
CheckValidDestination on the nil dst and panicked with a nil pointer
dereference. Skip the destination check when there is no destination.
Also remove a leftover debug Printf that wrote to stdout before the
archive data.
Signed-off-by: Vladimir Babin <vovababin@gmail.com>
Use OCI RenameObject for moves within a bucket, avoiding a server-side copy
followed by source deletion. Cross-bucket moves continue to fall back to copy
and delete.
The link pointed at a bare relative path (--check-filename) instead of
the in-page anchor for the ### --check-filename heading further down the
page, so it 404s on the rendered docs site. Point it at #check-filename
(Hugo strips leading dashes when generating header anchors).
---------
Co-authored-by: no-hup <19599684+no-hup@users.noreply.github.com>
Before this --onedrive-upload-cutoff was capped at 4 MiB and made
larger single-part uploads impossible.
Microsoft documents the limit for a single-request upload as "250 MB".
Measured against SharePoint Online the figure is binary and exclusive:
a body of 262143999 bytes is accepted and one of 262144000 is not.
Raise the constant to 250 MiB and record where it comes from.
The default is unchanged (upload_cutoff = -1, always chunk), so this
only affects users who raise the cutoff deliberately.
Co-authored-by: Can Arslan <carslan@viyaenv.com>
Listing a bucket without -R shows each prefix as one directory entry, so the
objects under it are not in the output. This is easy to read as a truncated
listing rather than as one level of a hierarchy.
Fixes#9797
A vulnerability exists in gRPC-Go servers configured with
xds.NewGRPCServer() where a crafted request missing both :authority
and Host headers can cause a server panic, resulting in a Denial of
Service (DoS).
This update fixes the problem.
NewUsageValue exists to clip an oversized quota to the maximum value of an
int64, which is what dc95f36bc added it for when Box raised the Enterprise
space_amount to 1e+18 and started returning it as a float.
For the float64 instantiation the guard misses its own boundary.
float64(math.MaxInt64) is not 2**63-1, it rounds up to 2**63, so a quota of
exactly 2**63 fails the comparison and falls through to the int64 conversion,
which the spec leaves implementation dependent for an unrepresentable value.
On linux/amd64 it wraps:
Before: rclone about -> Total=-9223372036854775808
After: rclone about -> Total=9223372036854775807
A negative total is not just a wrong number. vfs.Statfs documents -1 as "not
known", vfs.fillInMissingSizes branches on total < 0, and serve sftp only
computes its usage percentage when total > 0, so the value is read back as a
missing quota.
The int64 and uint64 instantiations are unaffected, since for them
T(int64(math.MaxInt64)) is exact and clipping MaxInt64 to MaxInt64 is a no-op.
Set built the timetable with *x = append(*x, ts), so setting a bandwidth
timetable on a value that already held one kept both schedules. The single-value
branch of the same function has always done *x = BwTimetable{ts}, and the other
multi-token Set methods in this package build into a local and assign at the end.
The visible effect is through the rc API. The "main" options block registered in
fs.RegisterGlobalOptions is the live globalConfig, and options/set reshapes JSON
straight into it, so
rclone rc options/set --json '{"main": {"BwLimit": "Mon-10:00,1Mi"}}'
added to the running daemon's timetable rather than replacing it, and the older
slot kept winning: LimitAt for a Sunday returned the previous 10Mi. The same
applies to a _config override on a single call, since AddConfig shallow-copies
the global.
Building into a local also stops a failed parse from leaving the previous
timetable partly overwritten, which the existing error cases already expect.
Ceph RGW (and Linode Object Storage, which is Ceph-backed) can break
SigV4 when Accept-Encoding is included in the signature, especially
when a reverse proxy rewrites that header. GCS already sets this quirk;
apply the same default for Ceph and Linode as suggested in #8206.
Set the OCI source SSE-C request headers when using a customer key.
Server-side copies need these headers to decrypt the source object, in
addition to the existing headers that encrypt the destination.
The test asks InstallUpdate to install the latest beta into an
unwritable file and expects an error. When the binary under test
reports exactly the latest beta version (as make quicktest does right
after a beta is published from the same commit), InstallUpdate
correctly decides there is nothing to do and returns nil, and the test
then dereferences the nil error and panics.
Pin fs.Version to a fixed old value for the duration of the test so an
update is always attempted, and use require.Error so a missing error
fails the test instead of crashing it.
Decrypting the config with --daemon runs --password-command twice, which
means two authentications when the command needs one, such as a hardware
key touch for `pass show`.
SetConfigPassword saves the obscured key to the temp file named by
_RCLONE_CONFIG_KEY_FILE so the daemon process can pick it up, but the
process that wrote it then read and deleted that file itself before
daemonizing. The daemon started with the variable pointing at a file that
was already gone, found no key, and ran the password command again.
Skip acquiring a password when _RCLONE_CONFIG_KEY_FILE is set, as the
PassConfigKeyForDaemonization documentation already describes, and only
consume the key file in a process that has no key of its own. The parent
then leaves the key for the daemon, and the daemon uses it.
Fixes#7341
applyOptions consumes the "path" option into vol.Path rather than leaving
it in vol.Options, but restoreState rebuilt the options with only fs and
type. The explicit path was therefore dropped when the plugin restarted,
and since fsString is rebuilt from those options the volume was remounted
at the root of the remote instead of at its subpath.
Before this change a volume created with type + path lost its path
completely, and one created with remote + path silently fell back to the
path of the connection string. With a backend whose credentials are
scoped to the subpath the restored mount then failed every operation
rather than serving the wrong directory.
Feed the persisted path back like fs and type, so applyOptions applies
the same precedence on restore that it applies when the volume is
first created.
Fixes#9853
The Dropbox backend advertises CaseInsensitive: true, but the two
shared-mode lookup helpers compared names with an exact, case-sensitive
==, so a shared folder or received file named "Project" could not be
found when requested as "project". Use strings.EqualFold in both
findSharedFolder and findSharedFile to honour the advertised
case-insensitivity.
Fixes#9706
In shared_folders mode NewFs derived the shared folder name with
path.Dir(f.root), which returns the parent path rather than the first
path component. For a root like "SharedFolder/subdir/deeper" this yielded
"SharedFolder/subdir", which findSharedFolder cannot match, so NewFs
failed with ErrorDirNotFound. Use the first path component of the root,
as the shared_folders option documents, so deeply nested roots mount.
Fixes#9705
The flag applies to out of space errors while writing and while creating
files or directories, not only while writing. Describe those operations
without naming ENOSPC, which is a Unix error that Windows never reports, so
the help is accurate on every platform.
KickCleaner sets the out of space flag, kicks the cleaner and then waits for
that flag to clear. Only the cleaner clears it, and the cleaner returns
immediately when the cache poll interval is not positive, so when it is
disabled nothing ever reads the kick or clears the flag and the caller waits
forever.
It now returns straight away in that case, under the same condition the
cleaner itself uses to decide it is disabled. Callers already retry a bounded
number of times and then report the error, which is the right outcome when
nothing is going to free space.
IsErrNoSpace compared against syscall.ENOSPC. Go defines that constant on
Windows as a value in its application reserved range which no Windows API
returns, so the comparison could never be true there. A full disk on Windows
reports ERROR_DISK_FULL or ERROR_HANDLE_DISK_FULL instead.
Preallocation failures were still caught, because those return a separate
sentinel, but a disk that is already full fails at the directory creation or
at the open long before preallocation is reached. That is the case reported.
The errors are now held in a list which platform specific files add to in
their init, which is the shape retriable_errors already uses in this package,
and the comparison itself is unchanged. Windows appends the two codes that
lib/file already recognises when preallocation fails. Every other platform
keeps exactly the behaviour it had.
This also reaches the VFS cache, which uses the same helper and has no
preallocation path of its own, so its out of space handling has been inert
on Windows.
Google now requires an app homepage URL and privacy policy URL to be
set on the OAuth consent screen's "Branding" page before the "PUBLISH
APP" button becomes clickable, even for a personal single-user app.
The existing instructions jumped straight to publishing in step 9
without mentioning this, leaving the button greyed out with no
explanation of why.
Fixes#9854
Add operations/cat endpoint to the Remote Control (RC) API to allow reading
and streaming file contents in-process over librclone / FFI and HTTP RC.
Supports range options (offset, count, head, tail), separator, optional maxSize
buffer limit, and returns both string and base64 encoded results.
The code which opens the --combined, --differ etc report files (or
stdout for "-") and closes them afterwards was duplicated between the
check command and the sync logger flags. This moves it into
operations.OpenReportFiles which both now use. It also closes any
files already opened if a later one fails to open.
The sync report flags (--combined, --missing-on-src, --missing-on-dst,
--match, --differ, --error and --dest-after) were only wired up in the
CLI commands so there was no way to get these reports over the rc or
from librclone.
This adds boolean parameters of the same names as operations/check
(combined, missingOnSrc, missingOnDst, match, differ, error and
destAfter) to sync/sync, sync/copy and sync/move. Each requested
report is returned as an array of strings in the output, just as
operations/check does. All default to off so existing callers see no
change in the output.
To share the code between the CLI and the rc the report writer helper
from operations/check is exported as operations.RcReportWriter, the
lsf defaults for --dest-after are moved into
operations.NewSyncLoggerOpt and the listing setup and --no-traverse
warnings from operationsflags.ConfigureLoggers into LoggerOpt.Init.
Before this change `rclone dedupe --dedupe-mode rename` probed the
backend for each candidate `name-N.ext` in turn and gave up when it
had tried 100 names for a given object. With daily runs against the
same duplicated filename this ceiling was eventually reached and
rclone logged "Could not find an available new name". Each probe was
also a backend lookup, so a run against 99 existing names took
minutes on Google Drive.
The rename now uses the listing dedupe has already made to skip names
known to be taken without asking the backend, and only confirms the
final candidate with NewObject (the listing may be incomplete because
of filters). The suffix counter is shared between the objects being
renamed so no name is checked twice. The safety limit is raised to
10000 which, thanks to the listing, no longer costs a lookup per name.
The deferred cleanup in the constructor checked a local error variable
rather than the error being returned, so failures after the VFS was
created (such as an invalid --passive-port) never shut it down. Name
the error return so the cleanup sees the returned error.
When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.
The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.
Writing to the mount with os.WriteFile made the Go runtime register
the file with its poller so the kernel then polled the file from
epoll_ctl and epoll_wait, sending POLL requests to the FUSE server
running in this same process. A thread waiting inside epoll cannot be
preempted by the runtime, so a garbage collection starting while such
a POLL was outstanding stopped the world for good - the test binary
could not be killed even with SIGKILL and the mount was left behind,
wedging anything that touched it.
Now we write through the mount with a descriptor straight from
open(2), which os.NewFile keeps out of the poller, check that it
really is out of the poller with SetDeadline, and check at the end of
the test that the mountpoint is unmounted.
In this commit we fixed the same problem for mount by running in a
subprocess however changing one write file routine here was much
easier than re-arranging the tests.
4a382c09ec mount: run tests in a subprocess to fix deadlock - #3259
Note that go-fuse (and hence mount2) works around this problem it by
forcing an early POLL it can answer with ENOSYS.
See: https://github.com/golang/go/issues/21014
The check that an entry returned by an archiver is a direct child of
the directory being listed normalised a parent of "/" to the root, so
an entry named "/x" passed as a child of the root while "x/" and
"dir//x" were rejected.
Decide by stripping the directory prefix and checking what is left
with sanitize.Leaf, which rejects an empty name, ".", ".." and any
name containing a "/". This also covers the leading slash case.
A file entry in a zip whose name refers to a directory, such as ".",
"/", "" or "sub/.", was only skipped when it named the root of an
archive which was itself the root of the remote. When the archive was
found by listing its parent directory the entry appeared as a file
with the same name as the archive alongside the directory for it, and
copying the archive tried to write both. When the entry named a
subdirectory it appeared as a file alongside that directory, and with
that subdirectory mounted as the archive root the entry was taken to
be the single file the root points at, hiding every real entry.
Skip any file entry whose last path component is "", "." or "..",
checked on the raw name before it is cleaned or joined on the prefix.
The zip archiver handed out its cached directory tree directly. Any
caller which filters a listing in place (as the core listing code
does) altered the cache, so later listings of the same directory could
be corrupted.
Return a copy of the cached listing instead.
When an upload failed with a 500 error the upload was retried with a
new upload link but the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by returning a RetryError instead so the caller retries
the upload with a fresh stream, which will fetch a new upload link.
When an upload failed with a retryable error the pacer retried the
whole upload with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
It also makes 5xx errors from the upload storage servers retryable.
These come back from the SDK as a different error type to API errors
so were not being retried at all.
When an upload failed with a retryable error the pacer retried the
whole PUT with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
MkdirModTime checked --interactive/--dry-run itself and then called
MkdirMetadata or Mkdir which check again, so --interactive asked twice
about making the same directory and --dry-run skipped before the
operation could be shown in the progress display or counted as a
check.
Now MkdirModTime decides how to make the directory first and delegates
entirely to MkdirMetadata, or Mkdir followed by SetDirModTime, each of
which does its own --interactive/--dry-run check exactly once. This
also means the modtime setting fallback shows in the progress display,
respects --no-update-dir-modtime and has its errors counted.
When syncing to a backend which supports directory modtimes, a
directory which needed its modtime updated and which had files
transferred into it would get its modtime set twice - once when the
directory was checked and once in the pass at the end of the sync.
Now, when the end of sync pass is in use (which it is for all default
syncs), directories which need their modtime (or metadata) updating
are marked for that pass instead of being updated immediately. This
halves the number of directory modtime updates in a typical sync and
makes the "Updated dirs" stat count each directory once.
Syncs to backends which preserve directory modification times (eg
sftp, local) can update the modtime or metadata on many directories.
This count makes that work visible in the stats output, the core/stats
rc and the prometheus metrics (as dirs_updated_total).
Syncs which update lots of directories (eg to sftp) could spend a long
time setting directory modification times, making directories or
removing directories with no feedback in the --progress display or
stats, making rclone appear to have hung.
This shows directory operations (setting modtime, updating metadata,
making and removing directories) in the Checking section of the stats
and counts them as checks, in the same way file deletes are shown.
This creates a checking transfer which is shown in the progress
display while it is running but is not kept in the completed
transfers history, so it never appears in core/transferred and is not
retained in memory after it finishes.
This is for repeated bookkeeping operations (eg directory modtime
updates) which would otherwise crowd file transfers out of the
history.
This was fixed in this commit in an inelegant way
399bc6a6a6 fshttp: don't send --header values to other hosts on redirect
The current commit fixes it properly with AddConfig.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The headers set with --header and --header-download are added to
every request by the rclone transport, including redirect hops which
net/http makes to other hosts, so a credential passed with --header
for one host could be sent to any host that server chose to redirect
to.
The transport now walks the redirect chain net/http records on each
redirected request and, once the chain has visited a host other than
the one originally requested, removes the headers rather than adding
them.
Also restore the global --client-cert and --client-key config after
TestCertificates so its temporary files are not used by later tests.
The headers set with --http-headers are documented for passing
credentials such as Authorization or Cookie. The backend used the
default net/http redirect policy which copies all but a handful of
well known headers to any redirect target, so a redirect from the
configured server to another host would send those credentials to
that host, and a redirect from https to http would send them in
plaintext.
When headers are configured this installs a CheckRedirect function
which:
- removes the configured headers from every hop once the redirect
chain has left the originally requested host
- refuses a redirect from https to http with an error
SameHost compares two URLs by host name (case insensitively) and port
(treating the scheme's default port as no port) so redirect policies
can tell a real change of host from a server spelling its own host
differently, e.g. redirecting "https://example.com/" to
"https://EXAMPLE.com:443/".
The HTTPS downgrade check now compares the redirect target against
the original request rather than the previous hop, so a chain which
started on plaintext http, passed through an https server and came
back to http is no longer refused - nothing is being downgraded
relative to what the user asked for. A chain which started on https
and reaches http via any number of hops is still refused.
Whether an archive entry name can escape the archive's namespace was
left entirely to each archiver. Enforce it in the archive backend too.
List only passes on direct children of the directory listed and
NewObject only returns the object asked for, so a future archiver
which forgets to validate names cannot expose a traversal to fs/sync
and fs/operations.
The path inside the archive was compared against the cleaned entry
names without being cleaned itself, so `archive.zip/sub/./dir` or
`archive.zip/sub//dir` failed to list even though `archive.zip/sub/dir`
worked.
A zip containing a file entry whose name refers to the archive's own
root (".", "/" or "") was presented as a single file called "." and
all its other entries disappeared. A file at the root can only be the
archive member the backend was pointed at, so with no root such an
entry is skipped like any other unsafe name.
Entry names read from a squashfs directory are not sanitized by
go-diskfs. The squashfs backend joined each leaf name onto its
directory to form the object's remote, so a crafted image could escape
its directory.
Use sanitize.Leaf to skip unsafe entries in List. A "\" is an
ordinary character in a file name on the systems squashfs images are
made on and in an rclone remote path, so it is deliberately not
rejected; making it safe for the destination is the destination
backend's job.
Skipped entries are logged at DEBUG with a single NOTICE count per
listing so a crafted image under a mount cannot flood the log.
When a zip archive was mounted at a subdirectory root, readZip used a bare
strings.HasPrefix to decide which entries fell inside the root. This
matched on a raw string prefix rather than a path boundary, so mounting
root "foo" also exposed sibling entries such as "foobar/..." with their
names left uncorrected.
Require a path boundary when filtering by root.
The zip backend mounts a zip file as a browsable Fs. Go's archive/zip
does not sanitize entry names, and readZip applied path.Clean but did
not reject a cleaned name that still pointed outside the archive. A
crafted zip could make rclone copy/sync attempt writes outside the
intended destination.
Sanitize entry names with sanitize.Path - the same check used by
rclone archive extract - skipping any entry with a ".." path
component, whether separated by "/" or "\". A backslash is otherwise
kept as an ordinary character in the name, as archive extract does. It
is up to the destination backend to make names safe for its storage.
Skipped entries are logged as a single count per archive so a crafted
archive with many escaping entries cannot flood the log.
Move the archive entry name validation added for CVE-2026-59732 from
cmd/archive/extract into a new lib/sanitize package as sanitize.Path,
so the same check can be shared with the archive backend which mounts
archives as a filesystem.
sanitize.Path keeps the extract semantics - reject any name with a
".." path component, treating both "/" and "\" as separators - and
additionally cleans the name with path.Clean. This corrects two edge
cases in extract: a repeated "./" prefix ("././file.txt") is now fully
stripped rather than only the first, and a bare "." entry is now
treated as the archive root and skipped.
Add sanitize.Leaf, which rejects a name that is empty, ".", ".." or
contains a "/", for checking a single directory entry name read from
an archive.
The names handled are rclone remote paths, in which "/" is the only
separator and "\" an ordinary character, so Leaf does not reject a
backslash: making a name safe for its storage is the destination
backend's job (the local backend encodes "\" on Windows and refuses
paths which escape its root). Path's rejection of ".." between
backslashes is kept as defence in depth for extract.
With --links/-l, a symlink is served as a .rclonelink object whose
content is the target path. A Range request with a start offset beyond
the target length (e.g. "Range: bytes=99999999999-") reached
openTranslatedLink and sliced the target string at that offset, panicking
with "slice bounds out of range".
Clamp the offset to the target length so an out-of-range start reads
empty, matching how a real file read past EOF behaves.
The birth-time (btime) write in writeMetadataToFile followed symlinks for
any object that was not a translated link, so under -l/--links a symlink
planted by an untrusted source at the destination path could redirect the
btime write to a target outside the backup destination on OSes where
birth time is settable (Windows).
Use the NOFOLLOW birth-time write whenever translating symlinks, not only
for translated links. It is a no-op on a real file or directory and stops
a planted symlink from being followed out of the destination.
With -l/--links the local backend faithfully recreates a source ".rclonelink" as
a real symlink at the destination. Directory metadata (chmod/chown/chtimes),
however, was applied with the raw following syscalls
os.Chmod/os.Chown/os.Chtimes rather than through the os.Root sandbox used for
content writes. A Directory is never a translatedLink, so when the destination
path already existed as a symlink planted by an untrusted source, the metadata
was applied through it to a target outside the backup destination.
Route directory metadata through os.Root when translating symlinks, so a planted
symlink can no longer redirect chmod/chown/chtimes out of the destination, while
legitimate in-tree directories are unaffected.
An empty or "." volume name joined onto the base directory resolves to the
base directory itself. newVolume does not call validate, so such a name
would mount a remote over the base directory and shadow every other
volume's mountpoint.
Require the resolved mountpoint to be a strict descendant of the base
directory so these degenerate names are refused.
When the plugin restarts it reads its persisted state file and used the
stored mountpoint verbatim. A state file written by an older rclone that
allowed escaping volume names, or one that was tampered with, could point
the mountpoint outside the base directory, so upgrading did not remediate
an already-escaped volume.
Re-derive the mountpoint from the base directory and the volume name on
restore, confined to the base directory, rather than trusting the stored
path.
A Docker VolumeDriver.Create request carries a raw volume name that was
joined onto the base directory with filepath.Join and used verbatim as the
mountpoint. filepath.Join collapses ".." components, so a crafted name such
as "../../../etc/foo" resolved to a host path outside the base directory,
where the plugin then created a directory and mounted the remote.
Confine the mountpoint to the base directory and refuse any name that
resolves outside it.
When serving FTP with --auth-proxy, the obscured password was cached in a
driver-global map keyed only by the username. Two sessions that logged in
with the same username but different credentials shared one map entry, so a
later login overwrote it and every subsequent operation on the earlier,
still-authenticated session was re-authorized with the later session's
credential and executed against the later session's backend.
Bind the credential to the FTP session by storing the obscured password in
the per-session goftp Session.Data map instead, so each session always
resolves the backend it authenticated for.
With --auth-proxy set and --auth-key unset, serve s3 registered every client
supplied access key ID with an empty secret and verified the SigV4 signature
against that, so anyone could sign a request for an arbitrary access key ID with
an empty secret and be let in. The proxy program was only ever given the access
key ID (as both user and pass) so it had nothing with which to authenticate the
client either.
An S3 client never sends its secret, only a signature made with it, so the
server has to know the secret to check the request. The auth proxy protocol as
been changed to handle this. For serve s3 the proxy program is given just the
access key ID as the user (no pass or public_key) and must return the matching
secret as _secret_access_key in its output. rclone verifies the request's
signature against that secret, refusing the request if the proxy rejects the
access key ID, doesn't return a secret or returns an empty one, or the signature
doesn't match. The secret is only used for this server's own verification and is
never registered with gofakes3, so other serve s3 instances in the same process
don't honour it.
The proxy's answers are cached. If a signature fails against a cached secret the
proxy is consulted again so a rotated secret takes effect immediately - but only
for a signature mismatch, and at most once every 10 seconds per access key ID
and client IP, so a stream of bad signatures can't make the proxy program run
for every request. A rotation never shuts down the cached backend under requests
still using it. A cached answer is checked with the proxy again once it is 5
minutes old even if in constant use, so revoking an access key ID takes effect
within 5 minutes.
This means --auth-key is no longer needed with --auth-proxy: it is ignored and a
warning is given at startup if both are set. The proxy is the source of truth
for both the credentials and the backend they map to. Presigned URLs (credential
in the query string) are now recognised by the proxy middleware too. The auth
proxy docs are added to serve s3.
Note that the serve s3 auth proxy protocol has changed. The proxy program is now
given the access key ID as "user" (it was previously given an MD5 hash of it,
with the access key ID as "pass") and must return the matching secret as
"_secret_access_key".
This needs gofakes3 v0.0.9 for signature.V4SignVerifyWithSecret.
gofakes3 kept the keys given with --auth-key in a store global to the process,
so when more than one serve s3 was running in one rclone (eg started via the rc)
each accepted the others' credentials and a client with the key for one server
could read and write the backend of another.
This updates gofakes3 to v0.0.9 which keeps auth keys per instance and adds a
test that two servers only accept their own keys.
From v1.70.0, an SFTP server started through the rc serve/start API with
a per-server proxyOpt.AuthProxy decided whether to enable proxy
authentication by checking the process-global proxy.Opt.AuthProxy
instead of the supplied proxyOpt.AuthProxy. In the normal rc case the
global is empty, so the auth proxy was silently ignored: the server
either failed to start with "no authorization found" or authenticated
against the local authorized_keys file instead of routing each login
through the proxy the operator configured.
The serve Provider refactor (f425f8d46) fixed the constructor by building the
provider from the supplied proxyOpt, but the authorized-keys handling in
configure() still consulted the global option. Make it depend on whether
proxy mode is actually active, and add a regression test for the
per-server option.
From v1.70.0 until the serve Provider refactor (f425f8d46), an S3 server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server served the fixed
filesystem supplied to serve/start rather than routing each access key
to the backend chosen by the proxy, bypassing the operator's intended
per-key authorization.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again, and only
logs "allowing anonymous access" when neither an auth key nor an auth
proxy is configured so the log reflects the effective mode.
From v1.70.0 until the serve Provider refactor (f425f8d46), an FTP server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server fell back to its
fixed-backend mode, whose default account accepts user "anonymous" with
any password - a complete authentication bypass.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again.
The multipart reorder-buffer admission trusted the client-declared part length.
A negative length was accepted, and `buffered + size` could overflow int64 for
a huge declared length, wrapping the running total negative and admitting
further parts past --multipart-streaming-buffer-limit.
Reject a negative length and use the overflow-safe comparison `size <=
bufferLimit - buffered` so an untrusted Content-Length can neither poison nor
overflow the budget.
Streamed multipart UploadPart called Reserve(contentLength) before reading any
body bytes, so the pool immediately allocated one 1 MiB page per MiB of the
client-declared Content-Length (or X-Amz-Decoded-Content-Length). An client
could declare a huge part size, send no body, and force an arbitrarily large
allocation without paying the bandwidth cost of the declared body.
Drop the Reserve so the pool-backed buffer grows a page at a time as the body
is actually read: memory now tracks the bytes received, not the unverified
header.
The rclone core does not sanitise ".." in an object's Remote(). Such a name can
arrive from a malicious or buggy backend - an object store permits keys
containing ".." or a leading "/" - and, if acted on, lets a listing or transfer
escape the configured root. A source object named "../../other/x" is copied to
"other/x" outside the destination root, and a crafted listing name surfaces
outside the directory being listed.
Add list.RemoteEscapesRoot, which reports whether a Remote climbs above the
root when joined onto it, and list.RemoveEscaping, which drops and logs such
entries.
Apply RemoveEscaping unconditionally - independent of the include/exclude
filters - at the three per-entry filtering points every listing passes through:
filterDir, walk.listR and walk.walkRDirTree (recursive ListR).
operations.StatJSON calls List and NewObject directly, bypassing those, so it
rejects an escaping remote up front.
This confines every backend at once, so no per-backend change is needed.
CVE-2026-78662: a malicious peer could flood an undecided channel's
incoming requests, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6354)
CVE-2026-56855: a malicious peer could send crafted messages on an
established channel, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6355)
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.
Single part uploads with Object Lock parameters need a Content-MD5
header, which the SDK can't compute from a stream, so the whole body
was read into memory with io.ReadAll to hash it - up to
--s3-upload-cutoff per file. prepareUpload already sets Content-MD5
from the source object's hash when it has one, so skip the buffering
entirely in that case and only buffer when the hash is unavailable.
When buffering is needed, read the body into a multipart.NewRW buffer
from the global pool, hashing in transit, so the memory is reused
across uploads and released after the request. The presigned request
path hands the body straight to http.NewRequest, so wrap it in
readers.NoCloser there to stop the transport closing the pooled buffer.
With speedup enabled, files up to --mailru-speedup-max-memory are read
into memory so their hash can be tried against the server before
uploading. This used io.ReadAll, which allocates a fresh heap slice per
file and grows it by doubling, so with the default 32 MiB limit and
several transfers this churned a lot of garbage outside rclone's memory
accounting.
Buffer the file with multipart.NewRW instead, hashing it in transit,
so the memory comes from the global pool and is reused.
When the hash isn't known to the server the buffered file is uploaded
from the same buffer. Previously a low level retry of that upload
resent an already drained reader, so the retry always failed. Rewind
seekable bodies at the start of each attempt so retries resend the
whole file. Add a test which drops the connection on the first attempt
and checks the retried body is complete.
The body is sent through lib/rest, which wraps it in readers.NoCloser,
so the transport can't close the pooled buffer early; Update closes it
when it returns.
The Linkbox API needs the MD5 of the first 10 MiB of each uploaded
file, so Update reads that prefix into memory before the upload. This
used io.ReadAll, which allocates a fresh heap slice per file and grows
it by doubling, churning well over 10 MiB of garbage per upload.
Read the prefix into a multipart.NewRW buffer instead so the memory
comes from rclone's global pool and is reused across uploads, and hash
it in transit rather than computing the same MD5 twice.
The PUT body goes through lib/rest, which stops the http transport
closing it, so Update owns the buffer and closes it on every exit path.
When the source has no MD5, Update reads the whole file into memory to
hash it before uploading if it is under --jottacloud-md5-memory-limit.
This used io.ReadAll, which grows a fresh heap slice per file (up to
10 MiB by default, roughly doubled by the growth strategy), so syncs
of many files churned allocations and GC.
Buffer the data with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across uploads and is released by the
existing cleanup function.
Unknown sized streams previously took the in-memory branch regardless
of the limit, so an rcat of an arbitrarily large stream could read it
all into memory. Spool those to the temporary file instead, as is
already done for files over the limit.
The buffered body is sent through lib/rest, which wraps request bodies
in readers.NoCloser, so the transport can't close the pooled buffer
early.
The multipart upload allocated a fresh chunk-sized buffer (64 MiB by
default) plus a 1 MiB scratch buffer per large file, copying every byte
twice, and never returned them to rclone's memory pool.
Buffer each part with multipart.NewRW instead so the memory is reused
across uploads and part of rclone's central memory management.
The pooled buffer is seekable, so a part can now be re-sent.
uploadPart previously had no retry at all and any transient error
failed the whole upload. It is now wrapped in the pacer with the
backend's usual shouldRetry rules, seeking to the start before each
attempt. The body is wrapped in readers.NoCloser so the http transport
can't close the pooled buffer between attempts, and Content-Length is
set explicitly since net/http can't infer it from a pool.RW.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each part of a multipart upload allocated a fresh part-sized buffer
(the size is chosen by Box, typically 8-32 MiB) with up to --transfers
parts in flight, so large uploads churned allocations and GC.
Buffer parts with multipart.NewRW instead so the memory comes from
rclone's global pool, is reused across parts and files, and is part of
rclone's central memory management.
The pool.RW is seekable so the retry closure seeks back to the start
before each attempt instead of rebuilding a bytes.Reader, and the
per-part SHA1 digest is computed by reading the buffer and seeking
back. The body goes through lib/rest which already stops the transport
from closing it; the uploading goroutine owns and closes the buffer.
The whole-file SHA1 used for the commit is unchanged.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each upload allocated a fresh chunk-sized buffer (10 MiB by default)
regardless of the file size, so bulk transfers of many files churned
allocations and GC.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management. The
pool.RW is seekable, so the existing rewind on retry carries over.
The body goes through lib/rest which already wraps it so the transport
can't close the pool buffer. The upload loop closes it after every
chunk, on error paths included. A source which ends before the
declared size is now reported as a short read before the chunk is sent
rather than as an incomplete write afterwards.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each chunked upload allocated a fresh chunk-sized buffer (48 MiB by
default), so bulk transfers of many files churned allocations and GC.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management. The
pool.RW is seekable, so the existing IncorrectOffset recovery which
skips already-received bytes on retry carries over unchanged, and the
"chunk received OK" check now compares against the bytes actually
buffered so a short final chunk is recognised too.
The Dropbox SDK wraps the request body in io.NopCloser, so the transport
never closes the pool buffer. The upload loop closes it after every
chunk, on error paths included.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each upload allocated a fresh 48 MiB chunk buffer, so bulk transfers of
many files churned allocations and GC, and small files paid for the full
buffer.
Buffer chunks with multipart.NewRW instead so chunk memory is reused
across uploads and is part of rclone's central memory management.
The pool.RW implements io.Closer, so the PATCH request body is wrapped in
readers.NoCloser to stop the http transport closing it after a failed
attempt and freeing its pages before the retry. The retry closure now
seeks the chunk back to the start explicitly before resending, a short
read of the source is reported as an error rather than sent as an
under-length chunk, and the request carries an explicit ContentLength.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
The compressibility heuristic compressed a 1 MiB sample of each upload
into a bytes.Buffer only to read its length, growing up to ~1 MiB of
garbage per file. Write the sample through a counting io.Discard-style
writer instead so no output buffer is allocated at all.
Every compressed upload allocated a fresh buffer the size of
--compress-ram-cache-limit (20 MiB by default) regardless of how big
the file actually was, so uploading many small files churned large
allocations and the memory sat outside rclone's pool accounting.
Read the head of the stream into a multipart.NewRW instead, which takes
pages from the global pool only for the bytes actually read and returns
them when the upload finishes. The pool.RW is seekable, so a wrapped
backend which needs to retry a small upload can rewind the body, which
the previous bytes.Buffer did not allow. A read error while filling the
cache is returned rather than falling through to the streaming path.
Add a unit test covering the buffered, streamed and spooled paths which
checks the body handed to the wrapped remote, that it can be re-read
for a retry, and that the pool pages are returned.
Each resumable upload allocated a fresh chunk-sized buffer (8 MiB by
default, up to 64 MiB), so bulk transfers of large files churned
allocations and GC.
Buffer each chunk in a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's central
memory management.
The pool.RW is seekable, so the chunk is rewound at the start of each
retry rather than re-wrapped. lib/rest wraps request bodies in
readers.NoCloser so the transport cannot return the pages to the pool
between attempts - the Content-Length is set through rest.Opts because
the wrapped body is not a *bytes.Reader net/http can measure. A source
that runs dry before its declared size is reported as an unexpected EOF
rather than sending the short chunk.
Files below upload_cutoff (up to 20 MiB) were assembled into a bytes.Buffer
which grows by doubling, so each upload allocated roughly twice its size
and threw it away afterwards, churning the GC on bulk transfers. Write the
multipart/related body into a pool.RW from the global page pool instead so
the memory is reused across uploads and bounded by rclone's memory
management.
The pool.RW is seekable, so the same body is rewound at the start of each
retry rather than being re-wrapped. lib/rest already wraps request bodies
in readers.NoCloser so the transport cannot free the pages between
attempts; the Content-Length is passed explicitly because the wrapped body
is no longer a *bytes.Reader net/http can measure.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Updating a file below --hidrive-upload-cutoff copied the whole file into
an append-grown slice so the request could be retried, costing roughly
twice the file size in transient allocations for every such update.
Buffer the file in a multipart.NewRW from rclone's global page pool
instead and return it to the pool once the request has finished. The
pool.RW is seekable so retries re-send the same buffer. Accounting is
applied as the buffer is sent so bandwidth limits and progress still
track the upload. The request now carries an explicit Content-Length
rather than being sent chunked.
The upload is bounded by the size the source declares - a source that
delivers more bytes than its declared size has the excess ignored, where
previously the stream was sent to EOF.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Each chunk of a chunked upload allocated a fresh buffer of
--hidrive-chunk-size bytes (48 MiB by default), with up to
--hidrive-upload-concurrency of them in flight, so large uploads churned
allocations and GC and a short final chunk still cost a whole chunk.
Buffer chunks with multipart.NewRW from rclone's global page pool
instead, sized to the data actually read, and return each buffer to the
pool once its PATCH request has finished. The pool.RW is seekable, so a
chunk which fails with a retryable error is re-sent from the same
buffer. Accounting is applied as a chunk is sent so bandwidth limits
and progress still track the upload. Chunk requests now carry an
explicit Content-Length rather than being sent chunked.
The prefix sent with the creating request in PutUnchecked is buffered
through the same helper.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Every upload allocated a fresh buffer of --hidrive-upload-cutoff bytes
(96 MiB by default) to hold the part of the file sent with the creating
request, however small the file was, so copying many small files churned
large allocations and GC.
Buffer that prefix in a multipart.NewRW from rclone's global page pool
instead, sized to the smaller of the declared file size and the cutoff,
and return it to the pool once the file has been created. Accounting is
applied as the buffer is sent so bandwidth limits and progress still
track the upload. The request now carries an explicit Content-Length
rather than being sent chunked.
The upload is bounded by the size the source declares - a source that
delivers more bytes than its declared size has the excess ignored,
where previously they were read up to the cutoff.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Upload chunks are buffered into a bytes.Reader and then, when transfer
accounting is active (always for a real copy), re-wrapped in accounting
before being handed to cachedReader. cachedReader only recognised a bare
*bytes.Reader, so the accounted chunk fell through to
readers.NewRepeatableReader, which copied the whole chunk again into an
append-grown slice. Every chunk (and the upload-cutoff prefix of every
file) therefore cost roughly twice its size in memory.
Look through the accounting wrapper when deciding whether the reader is
already a seekable buffer and, if so, seek the buffer underneath while
still reading through the accounting, so retries rewind without a copy.
Each chunk of a resumable upload was buffered in a fresh RepeatableReader
which grew by appending, so bulk transfers churned up to a chunk size (10
MiB by default) of heap per chunk and GC pressure.
Buffer chunks instead with multipart.NewRW from the global page pool
instead so memory is reused across uploads and bounded by rclone's
central memory management.
A source which delivers fewer bytes than its declared size now fails
before the chunk is sent with an unexpected EOF error rather than being
rejected by the transport.
Note that accounting now happens as the chunk is read into the buffer
rather than as it is sent, as in the drive backend.
When the source can't be re-opened, the whole input is read once to
compute its gcid before upload and held back for the upload proper. For
inputs at or below --pikpak-hash-memory-limit this used a bytes.Buffer,
a fresh heap allocation of up to the limit (and more while growing) per
file.
Hold the data in a buffer from the global memory pool instead so the
pages are reused and released on cleanup.
Inputs of unknown size were also always held in memory regardless of
their length, as only sizes above the limit chose the temp file.
Spool unknown sizes to the temp file so a large stream can't exhaust
memory.
The multipart uploader kept its own private buffer pool, a copy of the
one in lib/pool with identical settings, so its chunk memory was never
shared with the rest of rclone. Pages cached here were invisible to
other backends and vice versa, costing up to 64 MiB of extra idle cache.
Allocate chunks with multipart.NewRW, the global pool used by the
other backends instead.
WriteChunk copied the whole chunk, which lib/multipart already hands over
in a buffer from the global memory pool, into a bytes.Buffer so that
retries could re-send it. That doubled the per-part memory and made a
fresh chunk-sized heap allocation (64 MiB by default) for every part,
times the upload concurrency.
The chunk reader is seekable, so find its size with Seek and rewind it
inside the pacer closure instead, sending the pooled buffer directly.
Also fix the error for a part that fails to upload, which formatted the
buffer instead of the part number.
Each upload chunk is buffered in a pool.RW from the global memory pool
but was never closed, so its pages were never returned to the pool.
Close the buffer after each chunk is uploaded and on the read error
path.
A chunk that failed with a retryable error was also retried without
rewinding the buffer, so the retry sent an empty body with the original
Content-Length and Content-Range and failed.
Seek the chunk back to the start inside the pacer closure so each
attempt re-sends it in full.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
Before this change the transport could still be reading the request
body when Call returned, in two ways:
- the goroutine writing a multipart upload form was never stopped or
waited for, so after a failed request it kept reading the file body
while the caller retried
- the transport itself is documented to close the request body,
possibly in a different goroutine after the request has finished,
and hiding the body's Close method removed any way to wait for that
Both of these raced with callers which seek to the start of a pooled
buffer and retry the request, corrupting the retried body, and the
abandoned multipart goroutine could read pooled pages after they had
been returned to the pool.
Call now waits for the transport to finish with the request body
before returning, and the multipart form writer is stopped and waited
for once the request has finished.
Add an FsPutRetry integration test which uses the fshttp fault injector
to fail the first request carrying upload data, before it is sent, with
either a synthetic 500 or a connection reset, and then checks that Put
either succeeds with the correct contents or fails cleanly without
leaving a corrupt object behind.
It runs a single part upload and, for backends implementing
SetUploadChunkSizer, a chunked one. This catches backends which retry
an upload from a buffer which has already been consumed or returned to
the memory pool. Backends which don't use HTTP skip the test.
Also check in FsPutRetry and the large upload tests that the number of
buffers in use in the global memory pool is unchanged after an upload,
to catch backends which leak pooled buffers.
Add fshttp.SetFaultInjector, which installs a function consulted by
every Transport before a request is sent. The injector can synthesise
an error status code or a transport error for chosen requests. The
request body is drained and closed as a real round trip would, but
nothing reaches the server.
This lets the integration tests check that backends cope with a
transient failure part way through an upload - in particular that a
retry re-sends the same data rather than an already consumed or
freed buffer - without needing a fake server for each backend.
NoCloserNotify hides the Close method of the reader passed in like
NoCloser, but calls a notify function (once only) when the returned
body is closed. This lets callers of http.NewRequest find out when the
transport has finished with a request body, as it is documented to
possibly close it in a different goroutine after the request has
finished.
When both /me/drives and /me/drive fail during config (for example an
account-level 403 serviceReadOnly "Database Is Read Only"), send the
config state machine to the existing manual drive ID entry state
instead of dead-ending at choose_type with the raw error. The drive
itself remains usable when only the enumeration API is blocked.
Fixes#9794
Account.WriteTo wrote each buffer to the destination in full before
trimming the byte count for --max-transfer, so up to one buffer past
the limit could reach the wire and go unaccounted. This matters now
that NoCloser forwards WriteTo.
Truncate the write to the remaining allowance before writing.
NoCloser hides the Close method of an io.Reader but in doing so it
also hid io.WriterTo if the underlying reader implemented it, forcing
io.Copy to fall back to a buffered Read loop.
Following io.NopCloser, return a variant which forwards WriteTo when
the wrapped reader supports it.
Each resumable upload allocated a fresh chunk-sized buffer (8 MiB by
default), so bulk transfers of many files churned allocations and GC.
Buffer chunks with multipart.NewRW instead — the global page pool used
by the other backends — so chunk memory is reused across uploads and
bounded by rclone's central memory management. The pool.RW is seekable,
which also keeps chunk reads repeatable for retries.
The pool.RW implements io.Closer, so http.NewRequestWithContext upgraded
it to the request body and the transport closed it after each attempt,
returning its pages to the global pool — a chunk retried after a 5xx
then read a freed buffer and panicked in pool.(*RW).readPage. Wrap the
request body in readers.NoCloser so the transport can't take ownership
and the upload loop remains solely responsible for the buffer's
lifetime. Add a regression test that fails a chunk with a 500 and then
accepts the retry; it reproduces the panic without the fix.
Fixes#9684
The sftp backend could be used as the source of a multi-thread copy -
each chunk of a download is read on its own connection - but not as the
destination, because it did not implement OpenWriterAt. Uploading a
single large file was therefore limited to one connection while
downloading the same file was not, capping upload throughput well below
the link speed on high latency connections.
This implements OpenWriterAt for the sftp backend using lib/filepool. A
small pool of open write handles, each backed by its own connection,
lets the core write the chunks of a large file concurrently over several
connections. The file is created and truncated once up front so every
chunk offset is valid before the concurrent writes start.
Multi-thread uploads are off by default and turned on with the new
--sftp-multithread-upload flag, since many sftp servers only accept
sequential writes and would fail large uploads otherwise. Even when
enabled they fall back to a single connection when
--sftp-disable-concurrent-writes is set (a server that can't take
out-of-order packets on one handle won't take several handles either) or
when --sftp-connections caps the pool (the per-file fan-out would
otherwise deadlock waiting on it).
The OpenSSH test server (TestSFTPOpenssh) enables
--sftp-multithread-upload so the feature is exercised in CI against a
real server.
Tested against OpenSSH: a single large upload over a high latency link
went from ~12 MB/s to ~90 MB/s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Splainte <r.wycke@hotmail.fr>
Factor the connection-backed write handle pool out of the smb backend
into a generic lib/filepool.Pool[T] with its own tests, so it can be
reused by other backends that implement fs.OpenWriterAter over a
connection pool.
The smb backend keeps its behaviour, opening and releasing handles
through small closures passed to the pool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Splainte <r.wycke@hotmail.fr>
*pool.RW implements io.Closer, so when one is used directly as an
http.Request body the transport closes it after each attempt. A retry
which then seeks and re-reads the RW would index the freed pages and
panic with "index out of range" in readPage.
Read, Write, ReadFrom, WriteTo and Seek now return ErrClosed (which
wraps io/fs.ErrClosed) after Close, and Close is safe to call more
than once.
Nine tests fail on an ordinary Windows machine, eight in backend/local and
TestEnvironmentVariables in cmdtest, all with
symlink file.txt \?\C:\Users\...\symlink.txt: A required privilege is not
held by the client.
Windows grants SeCreateSymbolicLinkPrivilege only to an elevated process or one
running with Developer Mode enabled, and a default install gives an ordinary
user neither. CI does not see this because the windows-latest runner is
elevated, so the failures only show up on a contributor's own machine, where
AGENTS.md asks for make quicktest to pass before opening a pull request.
cmdtest already recognised the situation and attached a note to the failure
saying the test could safely be ignored. If it is safe to ignore then the test
knows it cannot run, so skip it and say why instead.
backend/local gains a helper that tries a symlink in t.TempDir() and skips if it
cannot make one, called from the six tests that need the privilege. Where a
platform can create symlinks the probe succeeds and nothing is skipped, so other
platforms are unchanged.
TestMetadata is skipped whole because it creates its symlink before anything
else and the object built from it is used throughout.
TestSymlinkEscapeConcurrent is left alone: it goes through putLink and ignores
the error, so it never needed the privilege.
Directory names which look like they have a --b2-versions version
string are now encrypted in full, so directories created by older
rclone (which left the version string in plain text) no longer
decrypt and vanished silently from listings.
DecryptDirName now falls back to the old form for such names so the
directory is listed, and logs the name it needs to be renamed to on
the underlying remote to make it accessible again. Document this in
the crypt docs.
The refresh endpoint returns a rotated token with a fresh expiry on
every successful call, but getUserInfo discarded it, so routine use
never extended the stored token's life. Once the stored token aged
out, accounts with 2FA enabled could not recover non-interactively
and required a manual reconnect.
Carry the rotated token out of getUserInfo and persist it in NewFs
via the same jwtToOAuth2Token + oauthutil.PutToken path that
refreshJWTToken uses, keeping f.cfg.Token in sync (same pattern as
refreshOrReLogin).
Fixes#9584
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The --b2-versions support added in 3fe2aaf96 strips a version string
from the last segment of a path before encrypting it, so that the
plain text version suffixes which the underlying backend appends to
encrypted file leaf names can be handled. EncryptDirName and
DecryptDirName share that code, so the last segment of a *directory*
name was version stripped too. Only file leaf names are ever given a
version string by the backend - a directory gets a
version-string-like name from the user, and such a name is encrypted
verbatim when it appears as the parent of a file name, so the same
directory ended up with two different encryptions.
Before this change, with a directory whose name matches rclone's
version format, eg dir-v2001-02-03-040506-123:
rclone copy file.txt crypt:dir-v2001-02-03-040506-123/
rclone ls crypt:dir-v2001-02-03-040506-123
# => "directory not found" - the file is invisible to listings
rclone mkdir crypt:dir-v2001-02-03-040506-123
# => creates a second directory with the same decrypted name
After this change EncryptDirName and DecryptDirName encrypt directory
names verbatim, so a directory encrypts the same way whether it is
named on its own or as the parent of a file. Version strings are only
added to file names by the underlying backend, so --b2-versions is
unaffected and the existing version tests are untouched.
A directory which was created by the old EncryptDirName will no longer
decrypt and will be reported as undecryptable in listings. Such
directories were already unusable - anything copied into one was
written to a different encrypted directory - so nothing which worked
before is broken by this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With no_head_object set, NewObject does not read any metadata, so the
destination object returned from a server side copy had a size of 0.
The size check in operations.Copy then failed with "corrupted on
transfer: sizes differ N vs 0" and deleted the newly copied object.
This also broke Move and hence renames through rclone mount.
Populate the destination object's size and MD5 from the source object
when no_head_object is set, as a server side copy produces an object
with identical content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dropbox is case insensitive and the path_display it returns in
change notifications may not match the case of the configured root.
Before this change the root was trimmed with a case sensitive prefix
match, so when the cases differed the full path was passed to the
ChangeNotify callback and the notification was ignored.
This trims the root case insensitively while preserving the display
case of the remaining path.
Write() overwrote a successful write's nil error with the stale
lastErr returned by kickWaiters() once the downloader had recorded
too many errors. download() then wrapped that stale error again and
stored it back as the new lastErr, so every subsequent write added
another "vfs reader: failed to write to cache file:" prefix - fixes#4998
Several documentation links pointed at anchors or paths that no longer
resolve, and the S3 directory buckets section named the config option
and flag in the plural, which does not match the backend.
Co-authored-by: shaurya <19599684+no-hup@users.noreply.github.com>
Co-authored-by: no-hup <shauryaj.finance@gmail.com>
A successful UploadPart whose response carries no ETag header made
WriteChunk panic dereferencing uout.ETag in a debug log line. The part
ETag is required by CompleteMultipartUpload, so an ETag-less 200 is
unusable: return a retryable error from inside the pacer callback so
the chunk is retried instead of crashing the transfer or completing
the upload with a broken part list.
Fixes#9822
Co-authored-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
Pikpak never returns MD5 for uploads which causes this test to fail.
Perhaps Pikpak should not declare MD5 but that is a bigger decision
being discussed in #9826
Nextcloud only stores a checksum which is supplied in the OC-Checksum
header of an upload, and discards it again when the modification time
is set with PROPPATCH. Re-sending the checksum in the PROPPATCH (as is
done for ownCloud) is rejected by Nextcloud with 403 Forbidden which
made the whole PROPPATCH fail, so SetModTime returned an error on any
object which had a hash. Uploads from sources without hashes, eg
streamed uploads with `rclone rcat`, were stored with no hash at all.
Use the Nextcloud PATCH extension with the X-Recalculate-Hash header
to have the server calculate and store the SHA1 of an object after a
streamed upload and after setting the modification time. This gives
a server side hash of the stored data which also lets rclone verify
streamed uploads.
If the source supplied fewer bytes than its declared size, the single
part upload path accepted the short body and stored a truncated file
recorded with the declared size, reporting a successful upload. The
multipart path already checks for this.
Count the bytes actually read and fail the upload if they do not match
the declared size, which cancels the partially created file.
This was found by the FsPutShortEOF integration test.
listSharedFolders already decoded shared-folder names with
f.opt.Enc.ToStandardName, but listReceivedFiles stored the raw name
returned by the Dropbox API unchanged. Names that require encoding
(e.g. a trailing space, which Dropbox itself rejects, so rclone
stores it as "name␠" via EncodeRightSpace) were therefore shown under
their raw, encoded form for received files instead of being decoded
back to the standard name, and findSharedFile could not resolve such
a file by its standard name.
Apply the same ToStandardName conversion listSharedFolders uses.
On a ranged download the metadata decoder stored the response's
Content-Length (the length of the range, not the blob) in the object's
size and only corrected it from the Content-Range total afterwards.
Object.Size() is read concurrently by the VFS cache and chunked reader
while a download is in progress, so with --vfs-read-chunk-size a reader
could observe the chunk length (e.g. 67108864 for 64M chunks) as the
object size. The VFS cache then logged
vfs cache: cached file (N) is unexpectedly larger than the remote
object (67108864). The cached file is likely corrupted after an
unclean shutdown; recovering ...
and truncated the read request against the bogus size, breaking
sequential reads of large blobs with --vfs-cache-mode full.
This applies the Content-Range correction before the size is stored so
the range length is never published as the object size.
The multipart upload copied the source into the request buffer without
checking how many bytes it had read, so a source that supplied fewer
bytes than its declared size was accepted by the server and reported as
a success with a truncated file stored.
Count the bytes actually read and fail the upload if they do not match
the declared size.
Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
rclone v1.75.0 started creating files in Proton Drive's new
crypto-refresh encryption format, following guidance from Proton that
new file node keys should use the v6/AEAD profile. It turns out the
official Proton web app cannot decrypt files whose node key is a v6
key (but the Android app can), so every file uploaded with v1.75.0 (or
a beta after 2026-07-13) shows 'Item cannot be decrypted' in the web
app, even though rclone itself reads the files fine. Inspecting a file
created by the web app shows Proton itself still creates v4 node keys,
using the new format only for the file content.
New files are now created with the same fully pre-crypto-refresh
format as v1.74.4 (v4 node key, v3 PKESK content key, v1 SEIPD
blocks), which every Proton client can read. Reading files in the new
format still works, new revisions of files which already use the new
content format keep it, and the auxiliary fields (name, node
passphrase, extended attributes, block signatures) are pinned to the
old format regardless of the recipient key's preferences, as Proton
requires.
Files already uploaded with v1.75.0 cannot be repaired in place -
uploading a new revision does not change the file's node key. To make
such a file readable by the Proton apps again, delete it from the
remote and upload it again with a fixed version of rclone.
This updates Proton-API-Bridge to v1.0.5 and go-proton-api to v1.0.4.
See: https://forum.rclone.org/t/proton-drive-unable-to-decrypt/54087
The single-shot upload path sent the source straight to Box as a multipart
body with no Content-Length, so a source that supplied fewer bytes than its
declared size produced a short request that Box accepted and stored, and the
upload was reported as a success.
Count the bytes actually read and fail the upload if they do not match the
declared size. The multipart path already reads each chunk with io.ReadFull
and so already fails in this case.
The concurrent walker created by walk() only stopped when the callback
returned an error or the whole tree had been listed. Cancelling the
context (for example via the rc job/stop endpoint for an async
operations/size or recursive operations/list call) was therefore
ignored: the checkers kept pulling list jobs from the channel and kept
listing the entire tree, burning CPU and making job cancellation
useless for every backend without a native ListR implementation.
Make every checker select on ctx.Done() so a cancelled walk shuts down
promptly through the existing quit/drain path and reports the context
error. Also check the context between directory read chunks in the
local backend so a single huge directory does not block cancellation.
Update already wrapped the source in a counting reader but never looked at the
count, so a source that supplied fewer bytes than its declared size was
uploaded as a chunked request, accepted by the server and reported as a
success with a truncated file stored.
Compare the bytes actually read against the declared size.
When copying to the local backend with checksums enabled (the default),
rclone hashed the incoming data by wrapping the source reader in an
io.TeeReader. TeeReader has no WriteTo method, so io.Copy could not use
the source's fast path and fell back to its generic 32 KiB buffer loop.
The same wrapping also stopped the destination *os.File using
copy_file_range, since the source was no longer a raw fd.
This meant checksummed copies were written in 32 KiB chunks whereas
--ignore-checksum copies were written in much larger blocks (typically
1 MiB). On filesystems where small writes are expensive, such as FUSE
mounts like LucidLink, this made a big difference: copying a 100 MiB
file took 3203 x 32 KiB writes in 3.6s, and now takes 108 x ~1 MiB
writes in 0.47s.
The dataflow analysis behind SA4023, new in the staticcheck 0.8.0
bundled with golangci-lint v2.13.0, makes linting large packages more
than 10x slower (89s vs 7s for backend/s3 alone) which took the CI
lint job past its 30 minute limit. golangci-lint no longer enforces
its run timeout during analysis so the job ran until cancelled, and
the cancellation meant the lint cache was never saved, making every
subsequent run cold and guaranteeing the timeout repeated.
The check also produces false positives (eg claiming operations.Delete
never returns nil).
This commit removes the workings of the old web ui which hasn't been
maintained for 6 years. If users supply --rc-web-gui then rclone will
exit with an error pointing users at the maintained `rclone gui`
command.
Five of the advisories released with v1.75.0 now have CVEs assigned:
- GHSA-45pq-889g-fcgh serve restic path traversal: CVE-2026-71309
- GHSA-xhf4-832v-7xcr lib/proxy CONNECT header OOM: CVE-2026-71310
- GHSA-8c48-q9wj-3w37 ftp command injection: CVE-2026-71311
- GHSA-2m8m-jhrm-w6j2 sftp PowerShell command injection: CVE-2026-71312
- GHSA-7p4m-qxvv-g567 local file name escape: CVE-2026-71313
GHSA-6jcg-q3wp-x2f4 (squashfs) loses its CVE-PENDING marker as GitHub
declined to issue a CVE from the rclone repository - the vulnerable code
is in go-diskfs so any CVE must come from an advisory there.
GHSA-mfvx-7rcj-9m5g (pprof) keeps its CVE-PENDING marker as the CVE
request is still awaiting allocation.
The test skipped every buffer count above 1 under -race, pointing at
golang/go#27070. That issue was closed in September 2018, so the
workaround outlived its cause: -race covered 137 of the 681 subtests.
Without the guard the race build runs all 681 and passes.
_dispatchWaiters decided whether a waiter was satisfied by clipping its
range against dls.src.Size(), the size of the fs.Object snapshot taken
when the Downloaders was created. _ensureDownloader decided whether to
start a downloader from Item.FindMissing, which clips against
item.info.Size instead.
When item.info.Size dropped below the offset a waiter was parked on while
the source object still reported the full size, the two disagreed.
_ensureDownloader found nothing missing so it started no downloader, and
_dispatchWaiters found the range absent so it never released the waiter.
Nothing was downloaded and no error was produced, so the error count never
reached maxErrorCount and the waiter was never woken. The reader blocked
forever with nothing logged at any level.
Wake a waiter when FindMissing reports nothing left to download for it as
well as when its data has arrived. Since _ensureDownloader starts a
downloader only when FindMissing is non empty, a waiter with nothing
missing has nothing that could ever wake it.
Fixes#9769
Allow --order-by to rank files using comma-separated rclone path globs. Patterns are evaluated in order, unmatched files are placed last, and path ordering makes ties deterministic.
Fixes#3975
When a multi-thread source ignores ranged reads, abort the partial
destination and retry the copy as a single stream. Reset accounting
before the fallback so transfer progress remains accurate.
Validate ranged GET responses before returning their bodies. Close invalid
responses, retry malformed partial responses through the pacer, and return
without retrying when a server deterministically ignores Range.
Fixes#6980
Add response validation for calls made with Range open options. Verify
Content-Range, Content-Length, response status, and the complete
representation size before a backend accepts the response body.
Return a shared sentinel when a server ignores a partial range so callers
can avoid retrying the same unsupported request.
Before this change, when no_data_encryption was set, uploads from
local disk advertised the hash of the encrypted data even though the
data was uploaded unencrypted.
On backends which check upload hashes (eg b2) this made uploads of
small files fail with errors like "Checksum did not match data
received", and made chunked uploads store an incorrect hash so the
files failed their checksum on download with "corrupted on transfer:
SHA1 hashes differ".
See: https://forum.rclone.org/t/sha1-mismatches-on-b2-with-no-data-encryption-true/54121
Before this change the environment variable getters in fs/configmap.go
logged the option value with %q, so a password set via
RCLONE_CONFIG_remote_pass (or RCLONE_remote_pass) was printed in full
to the debug log. Values from the config file were already redacted,
which made the leak easy to miss.
This change routes both getters through fs.RedactOptionValue, which
looks up the option in the backend's option list: options marked
IsPassword or Sensitive log as XXX, unknown options are conservatively
redacted, and --dump auth still shows the value for debugging.
Fixes#5794
The NFS section under Mounting on macOS talked about serve nfs without
pointing at rclone nfsmount, which is the command that actually does the
NFS-based mount on macOS.
Fixes#7869
Signed-off-by: Dean Chen <862469039@qq.com>
rclone checksum and rclone check --checkfile hashed one file at a time
unless --download was in use. The non-download branch of CheckSum called
Object.Hash synchronously from inside the listing callback, so --checkers
had no effect there; only the --download branch used the checker token
pool.
Both branches now do their work in the same goroutine bounded by
--checkers, so --checkers means the same thing with and without
--download. Checking a sum file against 2.6 GiB of local files with
--checkers 8 went from 1.70s to 0.31s here.
As a result the "hashing" checking transfer now covers the hashing work
in both branches, rather than being finished as soon as the download
goroutine had been started.
RcatSize streams the body straight into Put behind an ObjectInfo with no
hashes, so a known-size upload had its size checked but its checksum
never verified, unlike the unknown-size path through Rcat which hashes as
it goes.
The stream can only be read once, so hash it on the way past and compare
with the destination after the upload, as operations.Copy does. A
destination which reports no usable hash is still checked by size, and one
which fails either check has the failed copy removed.
Writing any file into a third-party app container - the Obsidian, Pages or
Shortcuts folders that iCloud Drive shows alongside your own - failed with
HTTP error 412 (412 Precondition Failed) returned body:
"{ ... \"error_code\" : \"VALIDATING_REFERENCE_ERROR\", \"reason\" :
\"Request has out of order children to be chained but the parents were
missing\" }"
Reading from those paths worked, and so did creating directories in them, so
the failure looked like a missing parent when the parent was plainly there.
Items in an app container live in a different zone from ordinary iCloud Drive
folders: a folder under Documents has a drive ID like
FOLDER::com.apple.CloudDocs::<uuid>, while the Obsidian container has
FOLDER::iCloud.md.obsidian::documents#o2v. DownloadFile already accounts for
this - it deconstructs the item's own ID and addresses the zone it finds - but
CreateUpload and UpdateFile hardcoded defaultZone, and UpdateFile built the
resulting Drivewsid with a hardcoded com.apple.CloudDocs as well.
So rclone asked Apple to chain the new document to a parent in
com.apple.CloudDocs while the parent lived in iCloud.md.obsidian. The parent
really was missing from the zone being addressed, which is what the error said.
Take the zone from the parent's drive ID instead, the same way the download
path does, and build the new item's ID with ConstructDriveID. Uploads outside
an app container are unaffected: their parents are in com.apple.CloudDocs, so
the derived zone is the value that was previously hardcoded.
Verified against a real remote: files now upload into an Obsidian vault inside
the container and read back correctly with an unpatched binary afterwards.
Before this change, `NewStats` stored the context it was created from on the
`StatsInfo`. Stats groups are never freed -- they are only evicted once there are
`--max-stats-groups` of them -- so each one kept its context, and everything reachable
from it, alive for the life of the process. As the rc creates a group per call,
that included the call's filters and their compiled regexps.
The context was only ever used to get `ci.StatsFileNameLength` from the config.
`StatsInfo` already stores that same `*fs.ConfigInfo`, read from the same
context in `NewStats`.
This change fixes the issue by passing the stored ci to `transferMap.String` and
dropping the context from `StatsInfo`.
Before this change, Transfer.Done closed the account of a completed transfer but
never released it, because it assigned nil to only a local copy of the pointer.
As a result, every completed transfer continued to reference its account.
An Account holds the transfer context and the source reader, and the stats keep
completed transfers around up to `MaxCompletedTransfers` per group, with groups
discarded only at --max-stats-groups. On a long-running `rclone rcd`, this adds up.
It was noticeable when running bisync repeatedly via the rc, where the transfer
context carries `b.WriteResults` (bisync's `LoggerFn`). A `*bisyncRun` holds
Path1 and Path2 listings, which can be quite large, and are not supposed to be
retained between runs. (Naturally they aren't, when running bisync on the
command line -- which is probably why we didn't notice this issue sooner.)
This change fixes the issue by releasing `tr.acc` in Done, instead of the local
copy. `tr.Snapshot` reads the byte counts off the account, so the progress is
recorded on the transfer first. That read happens before taking `tr.mu`, because
`acc.progress()` locks `acc.values.mu`, `checkReadBefore` holds that lock while
calling `StatsInfo.GetBytes`, and `StatsInfo` locks back into `Transfer` in
`Transferred` and `_removeTransfer`.
A client which started a multipart upload and vanished without either
completing or aborting it used to hold on to its resources forever.
Incomplete multipart uploads which have had no activity for
--multipart-expiry (default 24h) are now aborted and cleaned up
exactly as if the client had called AbortMultipartUpload, with a
NOTICE logged.
An upload with a part still being received is never expired, and each
completed part restarts the clock. Late operations on an expired
upload fail with NoSuchUpload, as they do on real S3 when a lifecycle
rule has aborted the upload.
Set --multipart-expiry 0 to keep incomplete uploads forever.
Multipart uploads used to be streamed directly to the remote with their
own PutStream machinery, bypassing the VFS, a design left over from
before the VFS could abandon a streaming write.
They are now written through the VFS exactly like plain object PUTs in
every cache mode. The parts are written, in part-number order, to a
temporary object which is renamed into place server-side on
completion.
With the default --vfs-cache-mode off the parts stream through the VFS
to the remote as they arrive. With --vfs-cache-mode writes or above
they are buffered in the VFS cache and uploaded by its write-back.
User visible changes:
- Multipart uploads now show in rclone's transfer stats and obey
--bwlimit (previously they bypassed both).
- Remotes without streaming upload support now spool the upload to a
temporary file on local disk instead of buffering it in memory.
- Multipart uploads are never buffered in memory because of missing
remote capabilities - only --disable-multipart-streaming does that.
- Remotes that upload atomically now also write to a temporary object
renamed into place, so an in-progress multipart upload is no longer
briefly visible under its final key.
- On the few remotes with no server-side move or copy the parts are
written straight to the final object in all cache modes.
- With --vfs-cache-mode writes, plain PUTs and multipart uploads to the
same key go through the same cache entry, so an earlier PUT still in
the write-back window can no longer be written back over a newer
multipart upload.
- Failed write-backs are retried by the VFS without the client having
to restart the upload, and completed objects are served from the
cache for read-after-write.
The temporary objects that uploads are written to before being renamed
into place are now named .rclone_temp_put_* and .rclone_temp_multipart_*,
and the whole .rclone_temp_ prefix is reserved: any object whose name
starts with it is hidden from S3 listings. This gives a single pattern
for cleaning up leftovers from killed servers:
rclone delete --min-age 24h --include ".rclone_temp_*" remote:path
The .rclone_multipart_upload_* objects rclone v1.75 used are still
hidden from listings so leftovers from an older server stay invisible
to S3 clients.
The mtime metadata fallback was nested inside the X-Amz-Meta-Mtime
branch, so it only ran when X-Amz-Meta-Mtime was present but invalid -
and then set the modtime from the invalid value's failed parse rather
than parsing mtime. An object PUT with only mtime metadata kept the
upload time as its modtime.
Now the two keys are checked independently, as TouchObject already
does.
streamPart did not check whether the upload had been torn down, but
AbortMultipartUpload sets the reorder buffer map to nil, so an abort
arriving while a part body was still being received panicked with an
assignment to a nil map once the part was buffered.
Now a part whose upload has been aborted or completed under it is
rejected with NoSuchUpload.
A PUT which failed part way through removed the object at the
destination key. As well as differing from real S3 (where a failed PUT
never affects the stored object), this raced with the client's
automatic retry of the same PUT: the retry stored the object and
returned 200 OK, then the failed first attempt's cleanup deleted it,
silently losing an acknowledged upload. The interrupted upload could
also be committed as a truncated object, since closing the write handle
gave the streaming upload a clean end of stream.
Now a failed or interrupted PUT never disturbs the object at the key:
- The object at the key is never removed on error.
- On backends where a partial upload is visible at its final name
(PartialUploads), and when the VFS cache mode is writes or above, the
upload is written to a temporary object which is renamed into place
on success and removed on failure, as streamed multipart uploads
already do. Backends which upload atomically are still streamed
straight to the destination.
- An interrupted or short body fails the upload via
WriteFileHandle.CloseWithError instead of committing truncated data,
and a body which ends cleanly short of its declared size is rejected
with IncompleteBody.
Closing a streaming write handle sends a clean EOF to the backend
upload, so a writer which knows its data source failed part way through
had no way to stop the truncated file being stored as if it were
complete.
CloseWithError closes the handle failing the upload with the given
reason instead. EOF-like reasons are rewritten so the layers reading
the pipe can't mistake them for a clean end of stream and commit the
partial file.
Needed by serve s3 to abort interrupted PUTs - see #9718
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.
Count the bytes actually read from the source and if they do not match
the declared size return an error.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.
Count the bytes actually read from the source and if they do not match
the declared size, remove the partially uploaded file and return an
error.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated file and
reporting a successful upload.
Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.
This was found by the new FsPutShortEOF integration test.
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated object and
reporting a successful upload.
Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.
This affects all backends which use lib/multipart for chunked uploads:
azureblob, b2, drime, internxt, oracleobjectstorage, s3 and shade.
This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.
If the source supplied fewer bytes than its declared size, the
truncated file was stored and the upload reported success with the
object claiming the declared size.
Count the bytes actually read from the source and if they do not match
the declared size, remove the truncated file and return an error.
This was found by the new TestRcatSizeShortEOF integration test.
The file is created at the declared size and the data then written
with ranged writes, so if the source supplied fewer bytes than
declared, the remainder of the file was left as zeroes and the upload
reported success.
Count the bytes actually read from the source and if they do not match
the declared size, delete the partially uploaded file (if newly
created) and return an error.
This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.
For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.
This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.
When using Microsoft Entra ID credentials, Azure Blob server-side copy
uses a user delegation SAS URL for the private copy source.
The SAS start time was set to the current local time. Azure Storage
validates the copy source from the service side, and small clock
differences can make that SAS appear not yet valid. The service then
returns 403 CannotVerifyCopySource with AuthenticationFailed.
Start the copy-source SAS 15 minutes in the past, matching Microsoft SAS
guidance for clock skew.
The precision field in the backend overview YAML files can hold
fs.ModTimeNotSupported (100 years in nanoseconds) which overflows int
on 32 bit platforms, making the YAML for those backends fail to parse
and causing rclone to log 18 internal errors on every invocation.
Use int64 for the precision field and add a test that parses every
embedded backend YAML file so this is caught on 32 bit test runs.
When an upload failed part way through with a retryable error (eg a
502 from the block storage servers) the pacer retried the whole upload
call with the same input stream. The stream had already been partially
consumed, so the retry re-created the upload draft and committed just
the remainder of the stream as a complete file, silently truncating
it. With restic over serve restic this corrupted the repository as the
truncated pack was reported as successfully uploaded.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
Uploads through RcatSize with a known size - used by rcat --size, the
rc operations/uploadfile and the serve backends, eg serve restic - did
not check the size of the uploaded object. If the source stream ended
before the declared size worth of data had been read, the truncated
object was reported as a successful upload. This could corrupt data
for callers which trust the result, eg a restic repository accessed
via serve restic (see #9722).
This adds the same size check operations.Copy performs after a copy,
respecting --ignore-size and backends which do not report sizes.
The SFTP page said `--dump-auth`, which no longer exists: it became a
value of `--dump`, so the docs asked for a flag rclone would reject.
The same line, and a line in the Swift troubleshooting section, also
used `--dump-headers` and `--dump-bodies`. Those still parse, but
SetFlags logs "--dump-headers is obsolete - please use --dump headers
instead", so the docs were steering readers onto a deprecated form.
The generated flag listings in docs/content/flags.md and
docs/content/commands/rclone.md are left alone: those flags do still
exist, so `--help` output should keep showing them.
When the source supplied fewer bytes than its declared size, the
compressed data file was stored under a name containing the declared
size while the metadata recorded the actual number of bytes read.
NewObject looks the data file up by the size in the metadata, so the
resulting object could never be read again, and the upload reported
success.
Check that the number of bytes read matches the declared size after
uploading the data and before writing the metadata, and remove the
data file and return an error if it does not.
This was found by the new FsPutShortEOF integration test.
The append loop retries everything once the upload session has
started, so a cancelled context error was retried through all the low
level retries with exponential backoff before the upload gave up.
A source which returned EOF before supplying as many bytes as it
declared would either commit a truncated file (if the shortfall was
within the final chunk) or loop forever appending empty chunks to the
upload session. Return an error wrapping io.ErrUnexpectedEOF instead.
Note that all dropbox uploads use the chunked upload path with the
default batch_mode of sync, so this affected uploads of every size.
A source reader which returns EOF before supplying as many bytes as it
declared should not cause a backend Put to hang, panic or create an
object claiming the declared size.
The layers above detect a truncated transfer by comparing sizes after
the upload, so a backend may either return an error or create a
shorter object.
See #9704
Object.Update held its connection until the deferred putConnection ran at
function exit, so the SetModTime it does at the end of every upload had to take
a second connection from the pool, dialling a whole new SMB session when the
pool was empty. With N transfers in flight the pool grew to roughly 2N sessions
for no reason.
Return the connection as soon as the file is closed. At that point the upload
has succeeded and remove() can no longer be reached, so nothing else needs it,
and SetModTime picks the same connection straight back out of the pool.
putConnection nils the pointer, so the deferred putConnection becomes a no-op
and the connection is not returned twice.
purgeCheck listed a directory's children with includeAll set, so the Drive API
returned trashed children as well as live ones. The only thing that came of
that was the trashedFiles flag, which is used solely to decide whether to trash
the directory or delete it permanently, and when use_trash is on the directory
gets trashed either way.
That made the common case pay for nothing: after trashing N files in a
directory, removing the now empty directory paged through all N trashed entries,
about one API call and pacer delay per 1000, instead of running a single query
that returns nothing.
Ask the server to filter out trashed children when the directory is going to be
trashed anyway. Hard deletes still enumerate them, since there the distinction
decides whether the directory can be removed permanently (#1040), and
--drive-trashed-only is left alone because it deliberately lists trashed items.
Write the git configuration for each test's fake home directory as a
file instead of running three "git config --global" commands, and drop
the "git annex version" invocation from repository setup. This removes
four subprocess launches from each of the fifteen test cases.
The subtests within each end to end test function already run in
parallel, but the three test functions themselves ran one after
another.
Marking the functions parallel lets all their subtests overlap.
The end to end tests exercise rclone via a separate subprocess spawned
by git-annex, and that subprocess is not built with race
instrumentation.
Running them in the race test therefore adds several minutes to CI
without providing any race coverage. The unit tests in this package
still run under the race detector.
Run the full "git annex testremote" suite for a single layout mode and
use "testremote --fast" for the rest.
The full suite repeats the same protocol operations across a matrix of
key sizes and chunk configurations, which exercises client side
git-annex behaviour rather than rclone.
The migration test ran the full "git annex testremote" suite for every
layout mode after verifying the migration with "git annex fsck".
The fsck calls already prove the migrated data is accessible via the
builtin special remote, and TestEndToEnd covers the special remote
protocol with testremote, so the extra five full testremote runs
duplicated coverage at a cost of tens of seconds each on CI.
The cmd/gitannex end to end tests routinely take 230-530s and on slow
CI runners the package exceeds the go test default timeout of 10m.
Raise the per-package timeout to 20m in quicktest and racequicktest to
give slow runners headroom.
Direct lookups of exported Dropbox Paper files retained the
caller-visible extension before export metadata processing appended it
again. Track when metadata was resolved through an export path so the
object keeps the requested remote name while listing behavior remains
unchanged.
The auth proxy was only given the user and their password or public
key, so a proxy program had no way to restrict logins to particular
networks, or to record where an authentication attempt came from.
The JSON sent to the program now has a client_ip key holding the bare
IP the client connected from, with the port stripped so IPv6 arrives
as 2001:db8::1 rather than [2001:db8::1]:52344. An IPv4-mapped IPv6
address is reported as plain IPv4 so that a client arriving over a
dual-stack listener still matches IPv4 networks. The key is omitted
when the client has no IP address.
The IP is also mixed into the backend cache key. That is needed as the
program is only run on a cache miss, so a client from a
non-allowlisted address presenting valid credentials within the 5
minute cache lifetime would get a cache hit and be let in without the
program being consulted at all.
The macOS NFS client sends SETATTR after SYMLINK, which arrives as
Chmod/Chown on the link path. These opened the target with vfs.Open,
which follows symlinks - a freshly created symlink usually dangles, so
the open failed with ENOENT, surfaced to the client as NFS3ERR_IO even
though the link was created.
Add path-based VFS.Chmod and VFS.Chown mirroring VFS.Chtimes. They do
not follow symlinks (lstat semantics, matching VFS.Stat) and return
ENOSYS when the node exists, since the VFS stores neither permissions
nor ownership; serve nfs calls them and masks ENOSYS as before.
Fixes#9627
Previously cross-compile.go ran NumCPU builds in parallel, each of
which ran an unrestricted go build using -p NumCPU internally, giving
up to NumCPU^2 concurrent compile processes and enormous load averages.
Pass -p to each go build, sized so the total parallelism is about
NumCPU, sharing the CPUs between however many builds are actually
selected. This can be overridden with the new -build-p flag.
The local backend built every OS path by joining the root with the source
name converted through the configured encoding, so the encoding was the only
thing keeping a name from turning into path syntax.
With an encoding which omits Dot (Slash, None, Raw) rclone's standard "．．"
decodes back to a real "..", and with an encoding which omits BackSlash a name
like "..\file" becomes a native path on Windows. filepath.Join then resolved
those out of the destination the user chose, so a source object called
"../marker.txt" - an s3 key of "tenant/../marker.txt" listed with the remote
rooted at "tenant", say - created or overwrote a file outside it.
localPath now joins the name to the root and checks with filepath.Rel that the
result is still inside it. localPath is the only place the root is joined to a
name, so threading the error through newObject and newDirectory covers every
operation.
Default configurations were not affected, as encoder.OS includes Dot on all
platforms and BackSlash on Windows.
Fixes GHSA-7p4m-qxvv-g567
A request path beginning with "../" escaped the path the server was
started on, letting a client list, read, create, overwrite and delete
objects outside it.
The check added for CVE-2026-59733 rejected non-canonical paths by
comparing them with path.Clean, but path.Clean cannot resolve leading
".." elements in a relative path so it leaves them in place and the
comparison comes out equal. Only interior traversal such as "a/../../x"
was rejected. Whether a path then escaped depended on the backend:
those which join the root with the remote before encoding it - webdav,
ftp, sftp, http and memory - resolved the ".." away, while local and s3
encode the dot elements first and were unaffected.
A bare "." was accepted for the same reason, which on bucket backends
addresses the served directory's own key.
Validate with io/fs.ValidPath instead, which rejects ".", ".." and empty
elements wherever they appear. The empty path stays valid as the root of
the API, and "." is excluded explicitly because ValidPath accepts it as
the root of an FS.
The VFS calls backends from goroutines of its own. A panic on any of these
cannot be recovered. So a backend panicking on a single file killed the whole
process, taking down a mount or every user of a serve command, even for servers
such as serve http whose library recovers panics raised on its own request
goroutines.
Recover panics at those goroutines and log them with a stack trace. Where the
surrounding code already handles a failure, recover around the backend call
itself rather than the whole goroutine, so a panicking upload is retried like
any other failed upload and a panicking download is reported to the waiters,
instead of abandoning the work part way through and leaving the bookkeeping
inconsistent.
Addresses GHSA-6jcg-q3wp-x2f4
Three ways a single client could deny service to everyone else connected to
the same serve sftp process:
A session "subsystem" request payload is a length-prefixed string, but it was
sliced at a fixed offset without checking its length, so a client sending a
truncated payload panicked the out-of-band request goroutine and killed the
process. Decode the payload instead, the way the neighbouring "exec" request
already does.
Rejecting a request then left the goroutine handling that channel waiting
forever to learn what kind of channel it was, because nothing was ever sent
on the channel it waits on. A client could open channels in a loop making
unsupported requests and grow the server's goroutines and memory without
bound. Signal the waiter when the requests run out so the channel is torn
down.
Separately, the request handlers - and reads, writes and closes on the file
handles they return - run on pkg/sftp packet worker goroutines which have no
panic recovery. A panic raised by a backend while serving one request took the
process down with it. Recover panics at that boundary, log them with a stack
trace, and return them to the requesting client as an error instead.
Addresses GHSA-6jcg-q3wp-x2f4.
The archive backend passes remote .sqfs bytes straight to the go-diskfs squashfs
parser, which does not validate several attacker-controlled superblock and
metadata fields. A crafted image can make the parser panic.
Recover panics at the go-diskfs boundary and return an "invalid or corrupt
squashfs image" error instead. As well as the parse entry points (Read, ReadDir,
OpenFile) this wraps the reader returned by Open, since the parser reads file
data lazily and can panic long after the image opened successfully.
Addresses GHSA-6jcg-q3wp-x2f4.
The builtin authentication compared the configured username and password
with ==, whose run time depends on how much of the value matches, giving a
timing side-channel that could in principle help guess the password.
serve sftp, serve s3 and the auth proxy already use subtle.ConstantTimeCompare
so bring serve ftp in line with them. An empty configured password
still accepts any password.
Addresses GHSA-mfvx-7rcj-9m5g finding 4.
When --cert and --key were supplied TLS was only applied to the listener if
exactly one --addr was given. With two or more --addr flags every listener
without an explicit tls:// prefix silently served cleartext HTTP, so adding a
second --addr to an HTTPS server quietly disabled TLS on both.
Now when TLS is configured every listener serves TLS. An individual listener
can be prefixed with http:// to serve unencrypted HTTP on that address, and
tls:// still marks a listener as TLS explicitly. Using a tls:// address
without --cert and --key is now an error instead of silently serving
cleartext with an https:// URL.
Addresses GHSA-mfvx-7rcj-9m5g finding 3.
With --rc-serve set the root listing enumerated the names of all configured
remotes without any authentication.
Make the root listing obey the same fail-closed rule as the rest of
the rc endpoints: it now requires authentication to be configured or
an explicit opt out with --rc-no-auth.
Addresses GHSA-mfvx-7rcj-9m5g finding 2.
The pprof debug handlers were accessible without authentication disclosing the
process command line (which can carry backend credentials passed on the command
line) and runtime profiles.
Mount the pprof handlers only when when auth is configured or --rc-no-auth was
passed - so they obey the same rule as the rc endpoints.
Addresses GHSA-mfvx-7rcj-9m5g finding 1.
Before this change rclone read a proxy response with http.ReadResponse
over an unrestricted buffered reader. A malicious or compromised
configured proxy, or an active on-path actor controlling a plaintext
HTTP-proxy hop, can grow memory until the process fails.
This fixes the problem by restrincting the read to 1MB maximum.
Before this change, rclone sent stack traces to the client on panic
capture in the rc. Stack traces can leak information which could be
useful to an attacker.
The FTP control channel is line oriented and the ftp library writes
command arguments (paths) straight onto it without escaping, so a
filename containing CR/LF can inject an independent FTP command.
This fix makes sure CR/LF are therefore always encoded to safe symbols
regardless of the configured encoding, which is what the default
encoding already does.
An audit of the shell escaping alongside the PowerShell fix found the
Windows Command Prompt path only rejected the double quote delimiter. A
remote filename containing % or ! can trigger environment variable expansion
even inside double quotes, and a newline or carriage return ends the
command, so a crafted filename could alter the command run during
server-side hashing. Reject paths containing these characters, matching
the existing treatment of the double quote.
PowerShell treats several Unicode smart-quote characters (U+2018, U+2019,
U+201A, U+201B) as single-quote delimiters in addition to the ASCII
apostrophe. The quoting helper only doubled the ASCII apostrophe, so a
remote filename containing one of these could close the quoted path and
inject statements that ran as the SSH account during server-side hashing.
Double all five delimiters when wrapping a PowerShell path so a filename
is always treated as data.
Fixes GHSA-2m8m-jhrm-w6j2
The AWS SDK signs S3 Express (directory bucket) requests with a session
token in the x-amz-s3session-token header. Go's HTTP client treats it as
an ordinary custom header and copies it when following a redirect to
another host, and it was missing from the list of secret headers the
redirect policy strips. Add it to the list.
The redirect tests derived their inputs from the production header list,
so a header accidentally dropped from that list would silently lose test
coverage rather than fail. The test list is now a deliberately literal
copy, kept in sync with the production list by a new test, so removing a
header from either list is a test failure. There is also a new
regression test verifying the Referer header that net/http generates
automatically - which for a presigned request carries the signed query
string - is not forwarded across hosts.
See GHSA-8mxv-9xhp-86h4
The CheckRedirect policy stripped only the X-Amz-Security-Token header when a
redirect crossed a host, leaving other origin-bound secrets exposed:
- Go removes Authorization on a hostname change but not on a same-host scheme
downgrade, so an IBM IAM bearer token was re-sent over plaintext HTTP.
- The SSE-C and copy-source SSE-C customer-key headers (which Go has no reason
to treat as sensitive) were forwarded across a redirect to another host.
- On a cross-host redirect net/http copies the previous request URL into the
Referer header; for a presigned request that URL carries the signature and
session token in its query string, disclosing them to the new origin.
This now refuses outright to follow any HTTPS->HTTP redirect - an S3 endpoint
has no legitimate reason to downgrade the transport, and this closes the
plaintext-exposure class regardless of which header or query string carries the
secret. On a cross-host redirect also strip the known credential headers and the
Referer.
The IDrive e2 region-lookup call posts the access key ID to a fixed HTTPS
endpoint but used a bare http.Client that would follow a redirect downgrading
to plaintext HTTP. Apply the same CheckRedirect policy as the main S3 client so
the request can't be replayed over an unencrypted connection.
Fixes GHSA-8mxv-9xhp-86h4
A server that redirects an HTTPS request to a plaintext HTTP URL on the
same host would cause Go's http.Client to replay the configured
credentials (Basic Authorization, cookies, secret headers) over the
network in cleartext.
Refuse to follow such downgrade redirects by default in lib/rest and wire
the webdav backend's client to use it. The `auth_redirect` option remains
the opt-in escape hatch for servers that legitimately need auth preserved
across redirects.
Fixes GHSA-h4mf-4v27-hggj
When making a release from master, the range since the last minor
release includes all the commits which were cherry-picked to the
previous stable branch and already published in its point releases,
so their changelog lines had to be deleted by hand.
make_changelog.py now finds the previous stable branch (eg
v1.74-stable) from the version being released and skips any commits
released there, detected via cherry-pick -x trailers and git cherry
patch equivalence. Skipped commits are listed on stderr for review.
Releases made from a stable branch are unaffected.
In this commit we attempted to wait for the success report of an
upload to fix the 500 error:
fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime
However Yandex Disk finalizes an upload asynchronously on its servers.
Waiting for the upload operation to report success is not enough -
under load the server reports the operation as successful slightly
before the file is fully finalized, so setting the modification time
straight after an upload can still fail with 500 Internal Server
Error.
Yandex support recommend waiting 1.5s - 3s after the upload before
modifying the file's metadata, so add an --yandex-upload-wait option
(default off) to insert a delay between the upload completing and the
modification time being set.
Cover the portmap registration OpenBSD's kernel NFS client needs (recipe
from hjicks in #8578), the mount_nfs -T requirement, and why the server
advertises AUTH_UNIX, so an OpenBSD user doesn't need to read through the
issue thread to get a working mount.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mounted files showed up owned by 4294967295 (^uint32(0)) on OpenBSD.
vfsflags_unix.go, which calls unix.Geteuid()/unix.Getegid() to get the
real uid/gid, only builds for linux, darwin and freebsd; OpenBSD fell
through to vfsflags_non_unix.go's zero-value stub instead.
OpenBSD has no linux/darwin/freebsd-specific fields here, just the same
POSIX Geteuid/Getegid/Umask calls golang.org/x/sys/unix already ships
for openbsd on every arch, so this just adds openbsd to both build tags
rather than needing a separate file.
Cross-compiled for GOOS=openbsd (arm64, amd64) and go vet clean; ran the
existing vfs test suite on darwin, no regressions. Not yet verified on a
live OpenBSD mount, only that the right uid/gid syscalls now get called
in this codepath.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mount served fine on Linux and macOS but *BSD kernel NFS clients
refused it with "Authentication error". The MOUNT reply only ever
offered AUTH_NULL, and the OpenBSD/FreeBSD/NetBSD clients won't mount a
server unless AUTH_UNIX is among the offered flavors.
Add AUTH_UNIX to the advertised list. The server still doesn't inspect
the credential (there's no per-user access control here) so the AUTH_UNIX
cred the client then sends is read as an opaque blob and ignored, exactly
as the AUTH_NULL one was. No behaviour change for existing Linux/macOS
clients, and OpenBSD now mounts and reads files over the share.
Verified on a real OpenBSD 7.9 arm64 VM: registered rclone's server in
portmap and ran "mount_nfs -T localhost:/ /mnt"; the mount now succeeds
(MOUNT and GetAttr RPCs go through) and files read back correctly through
the mount, where before it stopped at the auth stage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous OpenBSD path built the right options ("-o port=N -T") but
still handed them to mount(8). On OpenBSD "-T" is a mount_nfs(8) flag,
not a mount(8) one, so mount rejected it with "mount: unknown option --
T" and the mount never ran. Call mount_nfs(8) directly on OpenBSD; the
options are already in its native syntax.
Verified on a real OpenBSD 7.9 arm64 VM: with this change the command
becomes "mount_nfs -o port=N -T localhost:/ <mnt>" and mount_nfs accepts
the flags. The mount then fails later with "Stale NFS file handle"
because the OpenBSD client and rclone's in-process NFS server disagree
on the root filehandle - that is a separate issue in the NFS server,
not in the mount options this PR is about.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nfsmount built its mount options for GNU/Linux syntax unconditionally:
"-o port=N", "-o mountport=N" and "-o tcp". OpenBSD's mount_nfs(8)
rejects "-o mountport" outright ("option not supported", per the
reporter's log) and has no "tcp" suboption either, since it selects
TCP with the separate "-T" flag instead of an -o suboption.
Add a runtime.GOOS == "openbsd" branch that builds the option list
OpenBSD's mount_nfs actually accepts: "-o port=N" plus "-T" for TCP,
with no mountport option since OpenBSD's mountd is located via
portmap rather than a fixed, settable port. This follows the same
GOOS-branching pattern already used in this file's unmount function
(darwin) and in cmd/cmount/mount.go for openbsd/freebsd differences.
FreeBSD's mount_nfs(8) documents "port=", "mountport=" and "tcp" as
-o suboptions identical to Linux, so the existing option set is left
unchanged for freebsd and all other platforms.
Verified by cross-compiling (go build and go vet) for GOOS=openbsd,
freebsd, linux and darwin, all of which succeed. Actually mounting
via mount_nfs on OpenBSD needs a BSD machine to confirm at runtime,
which wasn't available here.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runner image ubuntu24/20260726.254 updated podman from 4.9.3 to 5.8.4
which is installed from the podman-static bundle. This bundle ships a
non-setuid fusermount3 in /usr/local/bin which shadows the setuid
/usr/bin/fusermount3 from the fuse3 package (as /usr/local/bin comes
first in PATH). A non-setuid fusermount3 cannot mount as an
unprivileged user, so the cmd/mount and cmd/mount2 tests failed with:
fusermount3: mount failed: Operation not permitted
Fix by removing the podman bundled fusermount binaries so the distro
provided setuid one is used.
The session close flag was only computed after each append, so a
known-size upload which fits in a single chunk sent all its data and
then issued a zero-payload append purely to close the session - one
wasted round trip per small file on the default batched upload path.
Set the close flag before the first append when the size is known to
fit in one chunk.
Since batch mode became the default all uploads go through
uploadChunked, which allocated a full chunk-size retry buffer (48 MiB
by default) regardless of the file size. With the `--transfers 32`
recommended for small file uploads that is ~1.5 GiB of buffer to
upload tiny files.
Size the buffer to the file size when it is known and smaller than a
chunk.
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download, giving
errors like
corrupted on transfer: sizes differ
This change sets "Accept-Encoding: gzip" on all requests which stops
the go runtime decompressing objects on download, so compressed
objects are downloaded as-is with intact size and hash information.
If --oos-decompress is set then rclone will decompress objects with
"Content-Encoding: gzip" as they are received, at the cost of not
being able to check the length or the hash of the downloaded object.
Fixes#9694
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.
Return the underlying error instead when the check fails for any other
reason.
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.
Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
Object.Open dereferenced the response's ContentLength pointer without checking
it. The OCI SDK leaves ContentLength nil when the server replies without a
Content-Length header or ContentRange which caused a nil pointer panic.
Now the size is only updated when the response actually provides one, leaving
the size from the object metadata in place otherwise.
This also fixes the same potential problem in the newObject code.
Previously, GET requests for a directory URL always returned an HTML
directory listing. There was no way to suppress this, unlike
`serve webdav` which has supported --disable-dir-list since #4191.
This adds the same flag to `serve http`. When set, GET requests for
directory URLs return 404 instead of a listing, while file downloads
continue to work normally.
Based on the approach suggested in #6306.
Fixes#4000
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.
Fixed by not attempting to decode the response body.
This was a regression introduced in
a4972de505 shade: retry server errors instead of failing the transfer
which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
This resolves an issue where mount filters supplied to the rc API
(such as `_filter` in remote control requests) were ignored during
FUSE mounts.
By passing the request context containing the parsed filter config to
`vfs.New`, the VFS layer now correctly respects the active filter
rules.
Fixes#8838
The march janitor goroutine, which discards queued jobs when the context is
cancelled, only ever returned on context cancellation. A march that finished
normally never cancels its context, so on an async rc job (whose context
descends from context.Background and is only cancelled by job/stop) the
janitor parked forever, leaking one goroutine per run and pinning that run's
directory listings in memory. A long-running rcd driving async sync or bisync
jobs accumulated these until it ran out of memory.
Signal the janitor to exit once the march completes so it returns on both
normal completion and cancellation.
Since around mid-2026 Apple's idmsa endpoints `POST
/verify/trusteddevice/securitycode` and `POST /verify/phone/securitycode`
return HTTP 409 (instead of 2xx) even when the submitted code is accepted:
the response body carries `"securityCode": {..., "valid": true}` and the
response headers include a fresh X-Apple-Session-Token, scnt and
X-Apple-Auth-Attributes, which are only issued on successful validation.
rclone treated any 409 as failure and aborted before TrustSession, so
configuring an iclouddrive remote always failed after the 2FA step with:
validate2FACode failed: HTTP error 409 (409 ) returned body:
"{... \"securityCode\": {\"code\": \"...\", \"valid\": true} ...}"
Treat a 409 response that carries X-Apple-Session-Token as success: absorb
the session headers and continue to TrustSession. Applies to both the
trusted-device and SMS validation paths.
Fixes#9488Closes#9534
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Add a new "list_parallelism" advanced option (requires use_arrow_list) which
splits a directory's blob-name keyspace into ranges and lists them
concurrently using the Arrow startFrom/endBefore range parameters. Both
recursive (ListR) and single-directory listings benefit.
The keyspace is sharded with a single-case (digit+lowercase) character
ladder. Single case is deliberate: the service validates startFrom /
endBefore case-insensitively (while listing case-sensitively), so a
mixed-case boundary pair is rejected with a 400; its endBefore filtering is
byte-ordered, so the listing is exact.
The service only honours endBefore on the Arrow listing path of accounts with
Blob Listing with Apache Arrow enabled: elsewhere it either rejects the
request with 400 OperationNotSupportedWithFeatureMissing or, if it fell back
to XML, would ignore the shards' upper bounds. Both cases are detected and
the parallel listing falls back to a sequential listing.
Listing 100,000 objects from a well-connected server:
config time vs XML
xml 23.9s 1.0x
use_arrow_list 6.9s 3.5x
list_parallelism=2 4.4s 5.5x
list_parallelism=4 2.8s 8.4x
list_parallelism=6 2.2s 10.9x
list_parallelism=10 1.5s 15.4x
list_parallelism=20 1.4s 17.6x
list_parallelism=30 1.1s 21.7x
So Apache Arrow listing alone is ~3.5x faster than XML, and parallel listing
takes that to ~20x. The ladder caps at 37 shards, so the benefit plateaus
beyond list_parallelism=30.
The option is hidden from the configurator, command line help and docs
until the feature is officially released.
Add a new "use_arrow_list" advanced option which fetches directory listings
using the ListBlobs Apache Arrow response format instead of XML. This can be
substantially faster for very large containers.
This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow" preview
feature to be enabled on the storage account. It is not supported on accounts
with a hierarchical namespace (ADLS Gen2). If the feature is not enabled the
service returns XML and the listing transparently falls back to it. Connection
string auth is not supported - it also falls back to the normal XML listing.
The option is hidden from the configurator, command line help and docs until the
feature is officially released.
This adds a temporary subpackage implementing the experimental "Blob
Listing with Apache Arrow" feature on top of the released Azure azblob
SDK.
The Azure SDK for Go supports Arrow listing on its unreleased
feature/storage/bifrost branch (commit c6fa341ca22b) but that can't be
used in a release. This package exposes the same options and pager
interface as the experimental SDK, built on the released SDK's public
API plus verbatim copies of its Arrow decoder and of the two internal
auth policies (shared key signing and storage bearer challenge) which
the released SDK does not export. Copied files carry MIT attribution
headers pinned to their upstream source.
When Arrow listing ships in a released azblob the whole package will be
deleted and callers pointed back at the SDK - the interface is
deliberately source compatible to make that a mechanical change.
Also exports auth.Transporter so the new package's pipeline can use
rclone's fshttp transport, and adds github.com/apache/arrow-go/v18 as a
direct dependency for decoding the Arrow IPC stream.
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
This updates `mountRc` in `cmd/mountlib/rc.go` to parse options using the
unified `rc.ParseOptions` helper. It also enforces parameter validation by
calling `rc.CheckParamsUsed`.
- Replace custom options parsing with rc.ParseOptions for vfsOpt and mountOpt.
- Delete consumed params (mountPoint, mountType, fs) and call CheckParamsUsed
before initiating FUSE mount to reject unknown parameters.
- Clean up duplication tests (TestRcFlatOptions, TestRcFlatOptionsNull) in rc_test.go.
- Update TestRc in rc_test.go to pass a clean params map to unmount.Fn.
This refactors all 8 serve protocols to use `rc.ParseOptions` for VFS
and protocol options decoding. It also implements parameter validation in the
main runner using `rc.CheckParamsUsed`.
- Update dlna, ftp, http, nfs, restic, s3, sftp, and webdav to call
rc.ParseOptions, enabling nested option block support.
- Remove unused configstruct imports.
- Update startRc in cmd/serve/rc.go to copy input parameters and
call rc.CheckParamsUsed to reject unknown parameters.
- Add TestRcStartFlatNestedAndUnknownRejection to cmd/serve/rc_test.go.
- Update inline documentation in serve/start command help to include
nested blocks information (vfsOpt, proxyOpt, opt) and a new WebDAV example.
When an HTTP/2 server retires a connection with GOAWAY after it has
already sent successful response headers, Go's http2 transport fails the
read of the response body with
http2: server sent GOAWAY and closed the connection; LastStreamID=..., ErrCode=NO_ERROR, debug=""
This was not recognised as a retriable networking error, so a transient
connection retirement aborted the whole command instead of consuming a
low level retry. It was reported against a large S3 check, where an
interrupted ListObjectsV2 page made rclone report destination objects as
missing and exit unsuccessfully.
The concrete error type is unexported by net/http, so match on the
message as we already do for the other http2 transport errors.
Rmdir checked that the directory existed with GetMetadata and then
checked it was empty with ListFolder, but ListFolder already reports a
missing path and a path that is a file, so the first request was
redundant.
Drop the GetMetadata call and map the ListFolder lookup errors onto the
same sentinel errors as before, so a missing directory still returns
fs.ErrorDirNotFound and a file still returns fs.ErrorIsFile. Limit is
set to 1 as only the presence of an entry matters, and HasMore is
checked as well so a full first page is not read as an empty directory.
Add Scality as an S3 provider covering both Scality RING (S3 Connector)
and ARTESCA, which share the same CloudServer + Vault S3 implementation.
The only quirk required is force_path_style: both products support
path-style addressing, and virtual-hosted style needs wildcard DNS that
on-prem deployments usually lack.
When --onedrive-av-override is set and the object is malware-flagged,
download via Microsoft Graph beta contentStream then /content with
Prefer: forceInfectedDownload, and keep Prefer (and AVOverride) on the
SharePoint redirect without re-encoding tempauth.
Clean files keep using the stable v1.0 /content path so permanently
enabled av_override does not put all traffic on beta APIs.
AI-assisted change; tested against OneDrive for Business with
application permissions (client_credentials).
The unauthenticated follow-up to /content redirects used Client.Do,
which does not check HTTP status. A 403 malwareDetected JSON body was
therefore written as file content and surfaced only as a size mismatch.
Check the status code, surface malware errors clearly, and avoid
suggesting --onedrive-av-override when it is already set.
The unauthenticated DLNA control endpoint decoded arbitrary SOAP bodies
into an in-memory XML field. A LAN client could send a large request
and exhaust the server's memory.
Limit SOAP request bodies to 1 MiB and return 413 when the limit is
exceeded.
ZERO-Z3 is S3-compatible object storage built on Ceph RADOS Gateway,
hosted in the EU on Zero Services' own network (AS215197), with
region-specific endpoints (zero-fra1, zero-fra2, zero-eyl1).
This commit introduced aux.go which unfortunately is illegal on windows.
e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
Rename aux_test.go too to keep the pair together.
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.
host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.
known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
The connection pacer in getSftpConnection used to retry every error,
so permanent failures (host key mismatch, certificate rejection, auth
failure, etc.) were looped 10 times before reporting to the user.
Switch to using fserrors.ShouldRetry which matches the pattern other
backends use so only genuinely retriable errors (timeouts, EOF,
network blips) are retried and permanent errors are surfaced
immediately.
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.
Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.
This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.
Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.
Bug discovered while thinking about #9617
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:
Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.
A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
Multiple security vulnerabilities have been identified and addressed
in grpc-go affecting the xDS RBAC authorization engine
(internal/xds/rbac) and the HTTP/2 transport server implementation
(internal/transport).
This updates to master to add the fix which allows it to compile on
plan9.
Fixes#9640
The auto-derived CORS origin logic (added in #9603 for the wildcard-bind
case) ran unconditionally, so it clobbered an explicitly-passed
--rc-allow-origin too - the flag was silently ignored. That breaks the
documented Docker/remote-browser setup where the bind address and the
address the browser actually reaches the GUI at are different.
Now it only kicks in when the user hasn't set --rc-allow-origin
themselves.
Bump go-diskfs to v1.9.4, which contains diskfs/go-diskfs#413 handling
squashfs images that have no fragment table or no xattr table. On v1.9.3
these images fail to list at all (ReadDir errors, or a nil xattr-table
dereference inside the library).
Adapt the backend to the v1.9.4 io/fs API: ReadDir now returns
io/fs.DirEntry (metadata via DirEntry.Info), and files are opened via
FileSystem.OpenFile instead of the removed FileStat.Open. The path
conversion added in c10eb47 (toIOFS) is reused.
Add a regression test with two committed fixtures built from trivial
placeholder content: an empty directory (no fragment table) and a small
tree whose superblock has the NO_XATTRS flag set while inodes still carry
an xattr index.
Fixes#9004
CloudKit is inconsistent about how it encodes the isDeleted field on
album records, returning a JSON boolean (true/false) for some accounts
and a number (0/1) for others. The numeric form caused listing of a
photo library to fail with:
json: cannot unmarshal number into Go struct field
ckBoolField.records.fields.isDeleted.value of type bool
The encoding also varies over time, not just per account: a full HTTP
dump from the reporting user showed the server sending
"isDeleted" : { "value" : 0, "type" : "INT64" }
but the same account later reverted to the boolean encoding with no
client change. Asset records already deliver isDeleted as a number, so
both encodings are in active use server side and either may appear.
Accept both encodings when parsing CloudKit boolean fields.
See: https://forum.rclone.org/t/error-when-trying-to-list-contents-of-primarysync-directory-in-icloud-photos/54028
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
The session cache was only populated on a fresh username and password
login, so once a session ID was stored in the config every Fs
instance created its own Mega session with its own copy of the
account's node tree. The server side move code relies on all Fs
instances of a user sharing one session, and with separate sessions a
move between two rclone remotes grafted a node from one tree into
another, where the asynchronous event replays of the two sessions
raced and could detach the moved file from the destination directory
so it disappeared from listings.
Cache the session however the login was done. This also stops every
extra Fs instance re-downloading the whole account node tree.
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
The go-diskfs library now requires io/fs.ValidPath style paths (no
leading slash, "." for the root) so convert paths at the library
boundary. This fixes listing failures and the resulting test hangs
after the update to go-diskfs v1.9.3.
This fixes the code to compile with the updated dependencies:
- squashfs: implement the new Path method required by the go-diskfs
backend.Storage interface, returning an empty string as there is no
underlying path.
- dropbox: convert to and from the SDK's new DBXTime type (an alias
for time.Time) for ClientModified, TimeInvited and Expires.
- dropbox: remove a stray debug fmt.Printf from the shared folder
listing.
- pin go-systemd to v22.6.0 to fix netbsd builds
go-systemd v22.7.0 uses unix.ClockGettime and unix.CLOCK_MONOTONIC
which golang.org/x/sys does not define for netbsd, so the build fails.
See: https://github.com/coreos/go-systemd/issues/512
- pin google.golang.org/grpc to v1.80.0 to fix plan9 builds
grpc v1.81.0 and later use syscall.Errno, syscall.ECONNRESET and
syscall.ECONNABORTED which do not exist on plan9, so the build fails
See: https://github.com/grpc/grpc-go/issues/9253
The /file-entries/duplicate endpoint returns a 500 Server Error for
every request (reported to Drime 2026-06) which made all server side
copies fail after 10 retries. Remove the Copy method so rclone falls
back to downloading and re-uploading instead. It can be restored if
Drime fix the endpoint.
This updates go-mega to fix the handling of the event stream when a
file is moved or renamed. The server reports a move as a delete of
the old node followed by an add of the same node, and go-mega was
losing track of the node's identity in the process, which could make
the old name reappear in listings and leave the backend unable to
remove it.
Microsoft has been progressively disabling sharing invitations on
both Business and Personal accounts - the driveItem invite API
returns 400 sharingFailed for any recipient on affected accounts -
which makes the permission writing tests impossible. Probe the API
once and skip the tests which need it when it is refused.
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.
Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
Open decoded range options against an unknown size, producing a
negative offset for suffix ranges (eg the last N bytes) and sending
syntactically invalid Range headers which the server ignored. A
suffix range then returned the whole file instead of the requested
tail.
Decode the options against the known object size so the offset is
always absolute, only send a Range header when one was requested, and
honour the requested count when the server ignores the Range header.
The Linkbox API is fronted by bot protection which under load
intermittently returns an HTML challenge page with a 200 status
instead of JSON. This surfaced as a fatal JSON decode error, and the
web API path misread the empty decoded result as an expired token,
hammering the login endpoint with requests which also failed to
decode and keeping the bot protection triggered.
Treat responses which fail to decode as retryable errors so the pacer
backs off until the block lifts, and only refresh the token when the
response actually parsed as JSON and reported an error.
Yandex Disk stores the modtime in a custom property set just after
the upload completes. The server sometimes silently drops this
property (the request returns success but the property is gone when
read back), leaving the object showing the upload time instead.
Check the modtime read back after upload and set it again if it
didn't stick.
put.io serializes trash operations per account so concurrent deletes
can fail with a 400 TRASH_LOCK_TIMEOUT error saying "There is an
ongoing blocking trash operation". This transient error was treated
as fatal - retry it instead.
Objects looked up by name used the /folder/itembyname.json endpoint
which does not return the file's MD5 hash, so NewObject returned
objects with no hash. Read the metadata with /file/info.json (which
does return the hash) instead, using itembyname only to find the file
id when it isn't known.
Copy and Move returned objects with only the id and size filled in,
leaving the modtime and hash empty, and SetModTime kept nanosecond
precision in memory while the server stores seconds. These made the
returned objects differ from a fresh listing of the same objects.
The Move method compared the source and destination paths relative to
their own Fs roots, so a server side move between two differently
rooted Fs instances on the same remote (as used by --backup-dir and
sync moves between remotes) was skipped as "already there" whenever
the relative paths coincided. With --backup-dir this destroyed the
file which was supposed to be backed up.
Compare the full paths including the Fs roots instead.
The rclone web GUI uses the Prefer header for sync/copy operations,
but the CORS middleware's preflight response did not include it in
Access-Control-Allow-Headers. This caused the browser to block
remote GUI copy requests with a CORS policy error:
Request header field prefer is not allowed by
Access-Control-Allow-Headers in preflight response.
Add 'Prefer' to the allowed headers list so remote GUI
access works correctly for file copy operations.
Fixes#9614
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Zoho config system ended every interactive create, update and
reconnect by calling m.Set("root_folder_id", workspaceID) in the
workspace_end state, with fs.ConfigChoose defaulting to the first
workspace. Any existing root_folder_id was therefore overwritten and
the remote silently repointed to the first workspace root.
This matters because reconnect is the documented fix for the 401
INVALID_OAUTHSCOPE download error - tokens issued before the
ZohoFiles.files.ALL scope was added lack download access - so users are
told to reconnect and then find all subsequent list/copy/sync/delete
operations pointed at a different, often shared, workspace.
Gate the workspace selection the way the drive backend does for team
drives (#5454): if a root_folder_id is already set, ask "Change current
root folder id ...?" defaulting to No and keep it; only run workspace
selection when it is empty or the user opts in. The token type rewrite
still runs on every reconnect so the scope refresh is unaffected. Also
expose root_folder_id as a standard advanced option (Sensitive, so
config redacted masks it) so it can be set and discovered like on
drive/box/onedrive.
Fixes#9575
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.
This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.
These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.
Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.
This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.
See: https://github.com/rclone/rclone/security/code-scanning/183
Go 1.27 changes compress/flate output, which also changes archive/zip
byte output. The HTTP zip download tests currently compare raw zip bytes
against golden files, so they fail even though the generated zip archives
contain the expected files.
Compare zip entries and decompressed contents instead of the exact
compressed byte stream.
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
When processItems filled the last free transfer slot it checked the
next queued item's expiry before the transfer limit, so if that
expiry was still fractionally in the future the timer was reset
instead of stopped. This caused intermittent failures in
TestWriteBackMaxQueue which asserts the timer is stopped once
--transfers uploads are in progress.
Check the transfer limit first so the timer is always stopped when
the transfer limit is reached. The timer is restarted when an upload
finishes so nothing stalls.
Also fix a typo in TestWriteBackMaxQueue which named every queued
item "number1".
The server stores the SMB last write time with 100 ns (FILETIME)
precision and returns it in full in listings, so advertise that as
the precision instead of one second. This makes syncs preserve
sub-second modtimes when comparing and copying files.
Note that this means rclone will consider files with modtimes
differing by less than a second as needing their modtime set where
before it did not.
The server computes the MD5 checksum asynchronously after upload, so
the object returned from Put and Update often had no MD5 while a
fresh listing shortly afterwards would report one. This broke
wrappers which compare exact hashes, such as the hasher backend's
fingerprint and the VFS cache. Retry reading the metadata for a short
time after upload until the MD5 appears.
The object returned from Put and Update had an empty MD5 checksum
while a fresh listing would report the MD5 the server computed for
the upload. This broke wrappers which compare exact hashes, such as
the hasher backend's fingerprint and the VFS cache. Stat the file
after upload to pick up the server computed MD5.
Update (which Put also uses) cleared the object's MD5 checksum after
upload, so until the object was re-read it had no hash while a fresh
listing would report the MD5 the server computed. This broke wrappers
which compare exact hashes, such as the hasher backend's fingerprint
and the VFS cache. Re-read the object's metadata after upload to pick
up the server computed MD5 (and authoritative size).
Update (which Put also uses) kept the caller's full precision modtime
in memory while the upload finalize request sends it rounded to
microseconds, so until the object was re-read the modtime did not
match what a fresh listing would report. This broke wrappers which
compare exact modtimes, such as the hasher backend's fingerprint and
the VFS cache. Round the modtime to microseconds to match what the
server stores.
Update (which Put also uses) kept the caller's full precision modtime
in memory while the server stores second precision, so until the
object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
The objects returned from Put, Move and Update kept the caller's full
nanosecond precision modtime (which write responses echo back) while
the server stores millisecond precision, so until the object was
re-read the modtime did not match what a fresh listing would report.
Update also discarded the upload response so the object had no SHA256
checksum until re-read. This broke wrappers which compare exact
modtimes and hashes, such as the hasher backend's fingerprint and the
VFS cache. Truncate the modtime to millisecond precision to match
what the server stores and populate the object from the upload
response.
Update and SetModTime kept the caller's full nanosecond precision
modtime in memory while the server stores SMB LastWriteTime with
100 ns (FILETIME) precision, so until the object was re-read the
modtime did not match what a fresh listing would report. This broke
wrappers which compare exact modtimes, such as the hasher backend's
fingerprint and the VFS cache. Truncate the modtime to 100 ns
precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the PROPPATCH sets it on the server with second precision, so until
the object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
SetModTime (and the no_check_upload Update path) kept the caller's
full precision modtime in memory while the server stores second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
TestRcDu relied on the default cache directory already existing. In clean
or container environments diskusage.New returned ENOENT, which the test
ignored before type asserting a nil result.
Use a temporary directory and require a successful response before
checking the disk usage values.
Add an AGENTS.md at the repository root describing the project's build,
test and code conventions for AI coding agents (Claude Code, Codex,
Cursor and similar). CLAUDE.md imports it so Claude Code reads the same
guidance.
Add an "AI-assisted contributions" section to CONTRIBUTING.md asking
contributors to understand and test AI-generated code themselves and to
trim the verbose comments these tools tend to produce, and add a
matching checkbox to the pull request template.
Previously if a cached file had grown larger than the remote object
and the cached range metadata was out of sync with the cache file
(e.g. after an unclean shutdown) - reloading the file failed with an
"invalid seek position" error. This aborted the writeback and left the
file inaccessible.
Rclone now recovers the bytes that are still available from the remote
and logs an ERROR that the local file is likely corrupted after an
interrupted upload.
See #9231.
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".
Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
Previously, restoreState in NewDriver would restore volumes AND perform
FUSE mounts synchronously before the Docker plugin socket was created.
This caused two problems:
1. The monChan was created after restoreState, but mount() sends on
monChan, causing a deadlock (send on nil channel blocks forever).
2. Even with the channel fix, slow or hanging mounts during state
restore would block the server socket from being created, causing
Docker to time out after ~13 seconds with "no such file or
directory" for the plugin socket.
Fix by:
- Moving monChan creation and monitor goroutine start before
restoreState
- Splitting state restore into two phases: restoreState (metadata +
filesystem setup only) and RestoreMounts (actual FUSE mounts)
- Calling RestoreMounts asynchronously after the server starts
listening
- Performing mount restoration concurrently across volumes
Fixes#9231
shouldRetry treated every Code=200501 API error as a transient storage
block error and retried it. Proton also returns Code=200501 with an
HTTP 422 for permanent validation failures (e.g. a content key packet
that cannot be verified, or an upload format the account is not
enabled for), so these were retried until the operation timed out
causing a multi-minute hang.
This fixes it by only retrying Code=200501 when it is not a permanent
client (4xx) error.
Update to github.com/ProtonMail/gopenpgp/v3 from v2 by updating to
- github.com/rclone/go-proton-api@v1.0.3
- github.com/rclone/Proton-API-Bridge@v1.0.4
Fixes#9434
SetModTime (which Put and Update also use) kept the caller's full
precision modtime in memory while Chtimes sets it with second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
Update and SetModTime kept the caller's full precision modtime in
memory while the server stores second precision, so until the object
was re-read the modtime did not match what a fresh listing would
report. This broke wrappers which compare exact modtimes, such as the
hasher backend's fingerprint and the VFS cache. Truncate the modtime
to second precision to match what the server stores.
Layers which wrap backends, such as the hasher backend and the VFS
cache, use fs.Fingerprint to detect whether an object has changed.
This only works if the object a backend hands back from a write
operation is identical to the object a fresh listing returns. If a
backend keeps the source's nanosecond precision modtime in memory
while the server stores milliseconds, or omits a hash the server
knows, every fingerprint comparison fails and cached hashes or files
are silently invalidated.
The existing tests compared modtimes within f.Precision() so they
could not detect these sub-precision divergences.
Add a checkFingerprint helper which asserts the fast and slow
fs.Fingerprint of the object returned from a write operation exactly
match those of the object read back from the remote, and wire it into
all the write paths which return or mutate an object: Put (all
variants), PutStream, large and streamed uploads, Update, SetModTime,
server side Copy and Move, and the metadata mutating Copy/Move
subtests.
Rework the ObjectOpenFingerprint test to use the same helper against
a fully read reference object, with subtests reporting which kind of
read is being checked.
Object.Update and SetModTime left the in-memory modtime at the
source's nanosecond precision, while the server stores milliseconds.
Afterwards the in-memory state did not match what a fresh List would
report, which broke wrappers that compare modtimes (e.g. the hasher
backend's fingerprint, which made it lose track of hashes whenever a
file was replaced).
The SDK serializes modtimes as UnixMilli and NewIncompleteFile
already rounds to milliseconds, so Put and the chunked upload path
are unaffected. Round the modtime to milliseconds in Update and
SetModTime too, so every path holds the same value the server
returns, without needing to re-read the metadata from the server
after upload.
Fixes#9308
When rclone sync replaced an existing file, it called Update which
pruned the old hash but never computed or stored the new one. This
left the file with no hash entry in the bolt DB.
This applies the same hashing logic in Put to Update: compute hashes
during the transfer via a hashingReader and store them afterwards. The
common hash-wrapping and hash-storing logic was extracted into a
function.
Fixes#9308
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.
This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.
Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.
Fixes#9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.
Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.
Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.
This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
After PUTting a file to the upload URL, Yandex keeps the file locked
for writing until the upload operation finishes committing on the
server. The PUT returned before this happened, so the following
SetModTime raced the still-in-progress write and got spurious 500
Internal Server Error responses.
Capture the operation_id returned with the upload URL and poll the
operation status until it reports success before returning, so the file
is fully committed before we access it.
The background chunk cleaner slept for the whole ChunkCleanInterval
(default 1 minute) before checking its stop channel, and only ran
CleanUpCache via the select default branch. This meant a cache that
had been stopped by StopBackgroundRunners could keep running
CleanUpCache for up to an interval afterwards.
The cache backend tests all share a single on-disk chunk store (the
TestInternalCache remote), so a lingering cleaner from a finished test
could call CleanChunksBySize and os.RemoveAll chunks that a later,
unrelated test had just written. The later test would then read a
chunk back and get an unexpected EOF - eg
TestInternalMaxChunkSizeRespected failing intermittently on CI.
Wait on a timer and the stop channel together so a stop is honoured
immediately and the cleaner can never run again once stopped.
The TestLogger/TestRepoCompare and TestLogger/TestBeforeVsAfter
testscript scenarios filled src and dst by downloading two old rclone
source archives from GitHub with `rclone copyurl`. Whenever GitHub or
the network hiccuped (eg a 502 Bad Gateway) the downloads failed and
the tests failed with it, making them flaky on CI.
Generate two overlapping trees of files in the test Setup instead.
They cover the same comparison categories the scripts exercise
(matching, differing, src-only and dst-only files) so the tests are
just as meaningful but no longer depend on the network.
The volume plugin parses the remote option as a trusted connection
string, which can run local commands via backend options. Spell out
that access to the unix or TCP socket is equivalent to command
execution as the serving user, document the unix socket permissions,
and warn that the TCP socket is unauthenticated.
A folder shortcut pointing at one of its own ancestor folders (for
example a shortcut to FolderX placed inside FolderX) made rclone recurse
forever when dereferencing shortcuts, duplicating the folder contents
until the disk was full.
Rclone now detects when a folder shortcut targets an ancestor directory
using the directory cache, leaves that shortcut out of the listing and
logs an ERROR, so the rest of the drive can still be copied.
Fixes#7118Fixes#9565Closes#9051
cleanRootPath used filepath.Abs which prepends the current directory,
but the resulting absolute path does not always refer to the same
directory as the original relative path - for example when the current
directory is shadowed by a mount or has been removed. This made
"rclone copy --links . ../dst" fail where "cp -ra . ../dst" succeeds.
rclone now cleans the path lexically on non-Windows platforms instead,
leaving relative roots relative so the OS resolves them against the live
working directory. Windows still makes the path absolute as required for
UNC long-path conversion.
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.
This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
Before this change a streamed multipart upload wrote its parts
straight to the object's final path on the underlying remote. That
meant an in-progress upload overwrote any object already stored under
that name, and aborting or failing the upload destroyed it. The
opposite of the S3 guarantee that an object only changes on a
successful CompleteMultipartUpload.
Remotes that upload atomically already (PartialUploads is false, e.g.
object stores) are safe to stream straight to the destination, so they
still do. Remotes where a partial upload is visible (PartialUploads is
true, e.g. local) now stream the parts to a temporary object instead
and move it, server-side, into its final place only when the upload
completes. A failed or aborted upload then just removes the temporary
object and leaves any pre-existing object untouched.
The temporary-object path needs the remote to support a server-side
move or copy in addition to PutStream uploads fall back to being
buffered in memory as before. The temporary objects are named with a
leading ".rclone_multipart_upload_" and hidden from listings.
Before this change when an aborted upload had overwritten a
pre-existing object of the same name, this left a ghost of that object
in every listing.
This invalidate the VFS cache on all the multipart upload abort paths,
so listings reflect what is actually on the underlying Fs.
When the stored token is expired and refreshOrReLogin fails, the
returned error wrapped the original 401 and discarded authErr, hiding
the actionable reason (e.g. "account requires 2FA - please run: rclone
config reconnect remote:"). Wrap both errors so the user sees why
re-auth failed and what to do about it.
Fixes#9583
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Zoho's WorkDrive listing API returns "401 R008 Unauthorized access"
(not a 404) when a folder id no longer resolves to a listable folder,
because it was deleted or never existed. A freshly refreshed token still
gets it, so it is not a token problem and retrying it is futile - and can
escalate to a 429 F7008 rate-limit penalty.
Handle it as a missing directory instead: shouldRetry no longer retries a
bare R008 401, listAll maps it to fs.ErrorDirNotFound, and
readMetaDataForPath flushes the stale parent from the dircache and reports
the object as not found so a later create re-resolves the parent. This lets
the VFS self-heal a stale cached directory id instead of hard-failing the
operation, and stops the VFS integration tests failing on a stale directory id.
Fixes#9578
The /stream/upload endpoint's overwrite flag is the x-prefixed header
x-override-name-exist, matching its siblings x-filename, x-parent_id and
x-streammode. uploadLargeFile sent the un-prefixed override-name-exist, which
the endpoint ignores, so overwriting a file of 10 MiB or larger created a
renamed duplicate instead of updating the existing file. The small upload API
passes the flag as a query parameter and is unaffected.
The folder list limiter granted the caller-supplied time immediately in
the burst phase, so concurrent callers observing time.Now() out of order
could record grant times that moved backwards. The sliding safety log
indexes grants as an ordered history, so out-of-order grants could also
breach the rolling-window cap. Clamp each grant to be at or after the
previous one so grant times are always monotonic.
Seafile returns a content-addressed id (a git-like hash of the file
contents) rather than a unique per-object identifier. Two different
files with identical contents are therefore given the same id, even on
different servers.
operations.SameObject assumes IDs uniquely identify an object, so when
syncing between Seafile remotes it treated two distinct but
identical-content files as the same object. The guard in NeedTransfer
if Equal(ctx, src, dst) && !SameObject(src, dst)
then evaluated to false for unchanged files (Equal was true because the
sizes matched, and SameObject was true because the ids matched), so
rclone re-uploaded them on every sync.
Seafile has no modtime and no usable content hash, so dropping the IDer
implementation makes SameObject fall back to path comparison and lets
unchanged files be skipped correctly.
Fixes#8300
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closing a stream in the parallel chunked reader cancels the stream's
context, so the in-flight read returns context.Canceled. This was wrapped
and returned as "failed to read stream", which the VFS cache downloader
treats as a real download error - it only recognises
asyncreader.ErrorStreamAbandoned as a benign teardown, as returned by the
sequential reader.
Return asyncreader.ErrorStreamAbandoned for a cancellation so tearing down
the parallel reader (on close, seek or reposition) is recognised as benign,
matching the sequential reader, instead of logging download errors and
retrying when --vfs-read-chunk-streams is used with --vfs-cache-mode full.
_createFile opened the cache file with O_RDWR but not O_CREATE, relying
on the file already existing. When the file had been removed underneath
rclone - either by _checkObject dropping a stale entry during open, or by
external deletion - the open failed and surfaced a hard "IO error" to the
application instead of recreating the file. Add O_CREATE so the cache
self-heals in that case.
Add TestObjectOpenFingerprint which reads an object's fast and slow
fingerprints, then opens it in several ways - a full read, a seek read
and ranged reads - and after a refresh, checking the fingerprint never
changes.
On a range request Azure returns the whole-blob MD5 in the
x-ms-blob-content-md5 header (BlobContentMD5) and leaves Content-MD5
empty. The download metadata decoder only read Content-MD5, so every
ranged read overwrote the object's MD5 with an empty string.
With --vfs-cache-mode full this changed the object's fingerprint between
opens, so the VFS cache judged every reopened file as stale and
re-downloaded it, defeating the cache and inflating egress. Prefer
BlobContentMD5 and never overwrite a known hash with an empty one.
When creating or updating a remote through the rc api (config/create,
config/update), parameters whose name starts with the ephemeral prefix
"config_" (for example config_template_file and config_template used to
customise the OAuth success page) were silently ignored.
updateRemote sets each supplied parameter into the config mapper, but
skips the "config_" prefixed keys so they are never written to the
config file. That guard is correct, because the mapper's setter writes
to the config file and these values are ephemeral. However backends read
these values back from the mapper (oauthutil reads config_template_file
and config_template via m.Get), so dropping them entirely meant the
values could never reach the backend and the default template was always
used.
Collect the ephemeral parameters into a separate map and add it to the
mapper as a getter overlay at PriorityNormal after the loop. The values
are now readable through m.Get without being written to the config file,
which is the same approach rclone authorize already uses to expose a
template supplied on the command line.
Fixes#9572
Co-authored-by: Hakanbaban53 <93117749+Hakanbaban53@users.noreply.github.com>
Co-authored-by: maximilize <3752128+maximilize@users.noreply.github.com>
With -l/--links rclone recreates a .rclonelink object as a symlink. A
malicious or compromised source could serve a symlink whose target points
outside the destination, plus a sibling object whose path traverses it, so
that rclone followed the planted symlink and wrote outside the destination
causing arbitrary file write.
When translating symlinks, rclone now performs all destination writes
(directory creation, file writes and symlink creation) through an os.Root
anchored at the destination. os.Root resolves every path component relative
to the destination's file descriptor and refuses any that escapes the root,
even under concurrent modification, so a planted symlink can never be
traversed out of the destination.
Symlinks are still reproduced verbatim - including ones whose target points
outside the destination - so backups remain faithful. Only writing
*through* such a link is refused. In-tree symlinks are unaffected.
Fixes CVE-2026-54572
Fixes GHSA-cf44-9pgv-m4xc
When applying the "mode" from --metadata the local backend cast the
source value straight to an os.FileMode, so a source that supplied a
mode with Go's setuid, setgid or sticky bits set would have those bits
applied to the freshly written file. As both the file content and its
metadata come from the source remote, a malicious source could plant a
setuid binary, and a victim running "rclone copy -M" as root against
an untrusted remote could end up with a root-owned setuid binary with
attacker-controlled content.
Rclone records "mode" in the unix st_mode layout where the special
bits live in different positions to Go's os.FileMode, so honest
sources never actually round-tripped these bits in the first place.
Apply only the permission bits by default, which closes this off and
is backwards compatible, and add the --local-metadata-restore-special-bits
lag to restore the previous behaviour for trusted sources such as
restoring a system backup made by rclone.
See: GHSA-945v-v9p3-v5xw
The CheckRedirect policy strips the X-Amz-Security-Token header when a
redirect chain crosses a host, but it only compared the host and ignored
the scheme. A redirect that kept the same host:port but downgraded
https:// to http:// was treated as the same host, so the STS session
token was re-sent over a plaintext connection where it could be observed.
Fixes GHSA-cf44-9pgv-m4xc
A user could reach another user's private repository by sending a path
such as /<me>/../<victim>/config. The authorization check compares the
first path segment against the authenticated user, while the backend
object key was built from the raw, un-cleaned URL path.
Reject any non-canonical request path so the authorization segment and
the backend object key can no longer disagree.
Fixes GHSA-fqj9-69pf-6pjg
Archive entry names are attacker controlled. `rclone archive extract` stripped
only a leading `./` and then joined the entry name onto the destination
directory with `path.Join`, which collapses `..` segments. An entry such as
`../escaped.txt` extracted into `:s3:bucket/safe/prefix` therefore resolved to
`bucket/safe/escaped.txt`, outside the selected `prefix` directory - a path
traversal ("Zip Slip") attack that could create or overwrite sibling objects on
any destination remote.
Entry names are now validated before use: a leading `./` is still stripped (tar
archives created with `tar -czf archive.tar.gz .` rely on this), but any entry
with a `..` path component is rejected. Both `/` and `\` are treated as
separators when looking for `..`, as the local backend treats `\` as a path
separator on Windows.
Fixes: GHSA-4vr5-p2gc-h23p
S3 object keys are opaque names that may legally contain `..` segments. `serve
s3` built backend paths with `path.Join(bucket, key)`, which normalised the key
so a request such as `GET /bucket/../root-secret.txt` resolved to a file outside
the selected bucket elsewhere under the serve root. Listing prefixes and
multipart uploads were affected also.
This did not allow reading of files outside the root, but did allow reading of
files in the root which normally aren't visible; only directories are visible as
buckets normally.
Because `serve s3` maps keys to file paths it cannot represent every opaque S3
key, so rather than normalising keys (which would alias distinct keys onto one
file as well as allow traversal) it now rejects any key that is not already in
canonical path form - containing `..`, `.`, `//` or a leading or trailing slash
- with a 400 Bad Request, as MinIO does. Directory listing prefixes are
validated the same way but allow the empty bucket-root prefix and an optional
trailing slash.
Fixes: GHSA-8v25-v8p6-qf7v
When mounting or otherwise opening an S3 prefix without a trailing
slash, rclone probes the path with a HEAD request to see whether it is
actually a file. Since v1.72.0 (#8975) any error other than "not
found" from that probe was fatal, so credentials scoped to a prefix -
which return 403 rather than 404 for the prefix key - could no longer
open the prefix at all.
6440052fbd s3: fix single file copying behavior with low permission
A 403 on the probe is ambiguous: it can mean either "this is the file
you named but you may not HEAD it" or "this is a prefix you may list
but not HEAD". When the HEAD is not permitted we now fall back to a
listing to disambiguate: if the path has children it is treated as a
directory, otherwise it is treated as a file.
Fixes#9582
rclone's shared Google Drive and Google Photos client_id is being
retired and will stop working during 2026. When creating a new remote
that would use it, the config wizard now warns the user and asks the
user to enter their own client_id and secret instead. Service account
and environment auth are unaffected as they don't use the shared
client_id.
See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
The shared Google Drive and Google Photos client_id is being retired and
will stop working during 2026. Warn users who rely on it (ie who have not
configured their own client_id) so they can create their own in advance.
The warning is only shown for auth flows that actually use the shared
client_id, not for service account, environment or anonymous auth.
See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
The --disable-zip flag was registered manually and was missing from
OptionsInfo, so it could not be set over the rc interface. Move it
into OptionsInfo like serve webdav does, which keeps the command line
flag and also makes it settable via rc.
WorkDrive throttles its listing API (GET files/{id}/files) PER
folder, independently of the overall request rate: at most ~19
listings of one folder are allowed in any rolling ~60s window and
the 20th returns F7008 with a ~300s Retry-After (measured live -
every observed trip landed exactly on the 20th listing inside a
window). fstests re-lists the same working directory after almost
every sub-operation, which is why the integration suite could not
pass.
Add a per-folder listing limiter with a true per-window cap: each
window starts with --zoho-list-folder-burst listings passing
back-to-back (the burst re-arms at every window boundary, so a
sync re-listing one directory a few times never waits), the rest
of the budget is spaced evenly across the window, and a sliding
log of recent listings guarantees no rolling window ever exceeds
--zoho-list-folder-limit (default 19) per --zoho-list-folder-window
(default 60s) for any traffic pattern. The registry is
process-wide and keyed by region+folder id so every Fs instance
shares one budget per physical folder; idle entries are evicted
after a window, which is lossless because Zoho's window has also
cleared by then.
Defaults were validated against the live service: bursts of 4-6
under the 19-per-60s cap ran clean while an over-cap probe tripped
F7008 exactly at the 20th listing, and a full test_all -backends
zoho run passes cleanly.
Fixes#9570
A 429 stall was only visible as a DEBUG pacer line, so without -vv
rclone appeared to hang for 2-5 minutes. In one night's batch logs 17
job starts produced only 4 completions because the silent stalls
looked like hangs and the jobs kept getting killed, re-triggering the
throttle.
Log the first 429 of each throttle episode at NOTICE with the server
message and the wait time. An episode ends when a request succeeds
after the penalty window; retries within an episode stay at DEBUG via
the existing pacer logging. State is two atomics behind a pointer on
Fs, so shallow Fs copies share it and concurrent checkers are safe.
See #9570
Zoho throttling is account/plan-dependent; measurements show a
sustainable listing rate of ~6 requests/s on a production account -
going faster drains a token bucket and stalls ~2 minutes per
Retry-After, which is strictly slower overall. Add per-remote pacer
options (default 6/1) using the same token-bucket pacer as the Google
Drive backend. Set --zoho-tpslimit 0 to disable the cap.
See #9570
Zoho WorkDrive now sends a Retry-After header on 429 (it did not when
the backend was written). Waiting the hard-coded 60s retried too early
when the server asked for more (Retry-After: 299 is common) and the
penalty escalated (observed 84s -> 239s). Honour the header plus a 1s
margin - retrying at exactly Retry-After still finds an empty token
bucket and burns ~16 immediate 429s - and keep 60s as the fallback
when the header is absent.
shouldRetry becomes a method on *Fs so the retry decision has access
to the remote's state; later commits build on this.
See #9570
Per RFC 4918 section 10.6, when the Overwrite header is omitted from a
COPY or MOVE request the resource MUST treat the request as if
Overwrite: T had been sent.
The upstream golang.org/x/net/webdav library mishandles this for MOVE
by checking == "T" instead of != "F", so an absent header is treated
as Overwrite: F and the request fails with 412 Precondition Failed.
Normalise the header to T in the rclone WebDAV server before
delegating to the upstream handler when the client did not send one.
This restores RFC-compliant default behaviour and can be removed once
the upstream fix in golang/go#66059 lands and the golang.org/x/net
dependency is bumped.
Fixes#9496
A Range header requesting a suffix longer than the object (e.g.
"bytes=-90407" against a 5 byte object) caused RangeOption.Decode to
compute a negative offset (size - End), which serve.Object then used
directly as a slice/seek offset and panicked with "slice bounds out of
range". FixRangeOption (used by backends like OneDrive/Box that lack
native suffix-range support) had the same root cause: it produced a
RangeOption with a negative Start, which Header() silently dropped,
turning the request into the wrong byte range instead of erroring or
serving the whole object.
Per RFC 7233 section 2.1, when the suffix-length exceeds the
representation size, the entire representation should be served.
Clamp the computed offset/start to 0 in both places.
Fixes#6310
NewStatsGroup started the averageLoop goroutine unconditionally at
group creation. In an rcd daemon driven by many short rc sync/move
calls (a common pattern for scheduled spool flushes), each call gets
a fresh job/N stats group. When such a job transferred zero files
the loop was never stopped, because _stopAverageLoop is only reached
via DoneTransferring once transferring and checking both go from
non-empty back to empty, which never happens if nothing was ever
transferring in the first place. The result was one leaked goroutine
per rc call, growing unbounded until the daemon was OOM-killed
(reported: ~61k goroutines and ~640 MB RSS after ~7 days from a
per-minute timer over 6 mappings).
This is the same class of leak as #8571, which fixed the equivalent
auto-start in NewStats. Fix it the same way: do not start the average
loop at group creation. NewTransfer and NewTransferRemoteSize already
call startAverageLoop when real transfer activity begins, and
DoneTransferring already stops it when the last transfer completes,
so on-demand behaviour is unchanged for groups that actually do work.
Groups that never transfer anything now cost zero goroutines.
Adds a regression test that fails without the fix.
Fixes#9567
Mknod is now implemented by mount, mount2 and cmount, so exercise it from
the shared vfstest suite: create a regular file (S_IFREG) through the
mounted path and check it reads back as a 0-byte regular file. This is the
path the kernel NFS server drives when a client creates a file over an
exported mount.
Before this there was no shared coverage for Mknod; it now runs against
each backend via RunTests.
Suggested in #9548.
Signed-off-by: Sandy Luppino <s.luppino@opendrives.com>
Seekdir handled only a rewind to offset 0 and returned ENOTSUP otherwise.
The stateless kernel NFS server opens a fresh directory handle and seeks to
the last returned cookie on every readdir continuation, so any listing
spanning more than one readdir batch failed over NFS. dirStream is a
snapshot taken at Readdir time and go-fuse assigns each entry a sequential
offset, so seeking to off positions the stream at index off; off == 0 still
resets to the start, preserving the rewind/re-read behaviour.
Before: ls of a directory that spans more than one readdir batch failed
over NFS with "Unknown error 524".
After: it lists correctly.
Adds TestDirStreamSeekdir covering rewind, mid-stream resume and the EOF
clamp. The full NFS path was validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.
Fixes#9547
setAttr left attr.Ino unset (0) and the NewInode sites left StableAttr.Ino
unset, so the kernel saw inode 0 while the node identity differed, which
breaks NFS file-handle validation. Set both to the stable VFS inode. The
bazil cmd/mount backend does not hit this because its framework assigns
stable inodes automatically; go-fuse needs them set explicitly.
Before: chmod/chown/truncate on a just-written file through an
NFS-exported mount2 mount failed with ESTALE.
After: they succeed.
Exercising the NFS handle-validation path needs a kernel NFS server, so it
isn't covered by the local vfstest harness; validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.
#9547
The kernel NFS server creates regular files with MKNOD (it
creates-then-opens, so vfs_create routes through fuse_create -> FUSE_MKNOD
when there is no open intent), but mount2 only implemented Create
(FUSE_CREATE, used by local and SMB clients). Without Mknod every NFS file
creation failed with ENOTSUPP. This mirrors the cmd/mount mknod handler from
#2115.
Before: touch through an NFS-exported mount2 mount failed with ENOTSUPP.
After: files create normally.
Exercising this needs a kernel NFS server, so it isn't covered by the
local vfstest harness; validated against a real Linux nfs-kernel-server
export over NFSv3, NFSv4.0 and NFSv4.2.
#9547
The mega backend keeps the whole account as an in-memory tree. go-mega's
Delete with destroy=true (hard_delete) removed the node from its lookup
table but left it in its parent's children, so a hard deleted file kept
showing up in directory listings until the tree was reloaded. In a
long-running mount or serve the file reappeared once the VFS directory
cache expired and re-read the backend.
Update go-mega to pick up the fix, and let the ghost test exercise
whichever delete mode the remote is configured with.
The mega backend keeps the whole account as an in-memory tree which
go-mega reconciles asynchronously from the server's event stream. After
an upload, delete or move the optimistic local update could be undone
moments later when go-mega replayed the corresponding server event,
re-adding a node to its parent. In a long-running process such as mount
or serve this could briefly show a just-deleted file in a listing, or
make a moved file reappear shortly afterwards.
Wait for the server to confirm uploads, deletes and moves so the
in-memory tree is settled before returning, matching what Rmdir and
Purge already do. The move wait was also previously started after the
server-side move so only the rename was covered.
Add an internal test reproducing the lingering listing via a tight
put/remove/list loop.
When a cached file was closed, --vfs-handle-caching kept its handle and
downloaders alive for a grace period and closed them later from a timer.
The deferred close drops the item lock while tearing down the
downloaders, leaving the file handle open. A reopen landing in that
window saw no grace timer and a live handle, failed to create the cache
file with "internal error: didn't Close file" and removed the cache
file, which hung the application reopening the file.
Reopens now wait for an in-progress grace-period close to finish so they
start from a fully closed item.
See: https://forum.rclone.org/t/opening-a-recently-closed-and-cached-file-hangs-rclone/53986/
This commit allows global config options to be set as flat parameters on all rc
commands. This makes the rc commands much more like command line parameters and
will aid understanding of how the rc is used.
It is backwards compatible with the old method using _config and _filter.
A backend flag set on the command line to a value that happened to
equal its default was silently ignored, letting the config file win
instead.
For example --sftp-user defaults to the current user, so connecting as
that same user with --sftp-user=USER had no effect and caused rclone
to use the value from the config file.
See: https://forum.rclone.org/t/sftp-user-parsing-as-cli-argument-broken/53955
DeleteFile always passes a nil backupDir to DeleteFileWithBackupDir, so
it never honours --backup-dir. The previous comment said it would move
the file into the backup dir when --backup-dir was in effect, which does
not match the code. Update the comment to state that DeleteFile always
deletes and that callers should use DeleteFileWithBackupDir when
--backup-dir support is required.
Also document on DeleteFileWithBackupDir that the backupDir is found with
BackupDir, which is relatively expensive, so it should be looked up once
outside any delete loop rather than per object.
#7566
govulncheck reported four vulnerabilities in golang.org/x/image v0.41.0,
all reachable via the internxt backend's thumbnail/image upload path:
- CVE-2026-46604 (GO-2026-5066): panic decoding a TIFF image with an
out-of-bounds strip offset
- CVE-2026-46602 (GO-2026-5062): unbounded memory use from lack of a
limit on TIFF tile sizes
- CVE-2026-46601 (GO-2026-5061): panic on a WEBP VP8 alpha channel size
mismatch
- CVE-2026-33813 (GO-2026-4961): panic decoding a large WEBP image on
32-bit platforms
Updating to v0.43.0 fixes all four.
After an upload (notably multipart) to a slow backing remote, the file
lives in the VFS and is returned by ListBucket, but node.DirEntry() stays
nil until the --vfs-write-back writeback completes. HeadObject and
GetObject returned gofakes3.KeyNotFound while it was nil, so a HEAD/GET in
that window 404'd even though the object existed.
getFileHashByte already falls back to hashing the VFS cache when the
backing object isn't available yet. Drop the early nil return, pass the
node (not the fs.Object) to getFileHashByte, and take the Content-Type
from fs.MimeTypeFromName when the backing object isn't there yet.
The HeadObject path translates a 404 (Not Found) and a 405 (Method Not
Allowed) response into fs.ErrorObjectNotFound, but the GetObject path
returns the raw AWS SDK error instead. With --s3-no-head-object set,
Object.Open() is the first request to hit the source for a GET
operation, so callers that rely on fs.ErrorObjectNotFound see an
opaque "operation error S3: GetObject, ... StatusCode: 404" string
when --s3-no-head-object is set.
Mirror the error handling of HeadObject. The 405 (Method Not Allowed)
case applies to GetObject for the same reason it applies to
HeadObject: A request that specifies the versionId of a delete marker
returns 405 instead of 404 (commit 8470bdf810, AWS S3 delete-marker
docs).
Co-authored-by: Jan Schlien <rclone-github@jan-o-sch.net>
This adds two new Dropbox backend flags:
--dropbox-skip-shared-folders skips all shared folder mount points
regardless of ownership.
--dropbox-skip-unowned-folders only skips shared folders that are
not owned by the current user.
These help avoid backing up the same shared folder multiple times when
backing up multiple Dropbox accounts.
Fixes#9514
Before this change, with --files-from and --no-traverse, a single file
that could not be read (for example permission denied) stopped all the
other files in the list being copied.
This happened because the error was returned from the listing, which
caused the whole source listing to be discarded.
This change counts and logs such per-file errors and carries on, so the
readable files are still copied and rclone exits with a non-zero error
code.
Fixes#9115
Option help strings are also used to generate the website documentation,
so some contain markdown links with root-relative targets such as
[encoding section in the overview](/overview/#encoding). These render
correctly on rclone.org but are confusing in the interactive config
prompt, where the user sees the raw markdown and the link has no
reachable root.
Rewrite such links to text (https://rclone.org/path) when showing an
option's help in the interactive config. The raw help is left unchanged
so documentation generation is unaffected.
Add a .github/CODEOWNERS file mapping backends and core subsystems to
their maintainers so GitHub auto-requests reviews. Move the per-area
responsibility mapping out of MAINTAINERS.md into CODEOWNERS, leaving
MAINTAINERS.md as the maintainer list with a pointer to CODEOWNERS.
Since v1.74.0, ncdu doubled every keystroke on Windows: pressing Enter
entered a directory and immediately descended a level deeper, "n" sorted
by name and then reversed the order, and so on. Arrow keys were
unaffected.
The cause is an upstream regression in github.com/gdamore/tcell/v2
pulled in by the v2.9.0 -> v2.13.8 bump in commit 23917555. The same
regression affects lazygit and micro.
There is no fixed tcell release, so pin tcell back to v2.9.0, the last
version before that commit, until the input handling is fixed upstream.
See: https://github.com/gdamore/tcell/issues/1124
See: https://github.com/jesseduffield/lazygit/issues/5344Fixes#9539
The Cobra generated PowerShell completion script captures rclone's output
through a pipeline with Invoke-Expression. PowerShell decodes that output
using [Console]::OutputEncoding, which on non-UTF-8 hosts (for example
PowerShell 5.1 on a Windows install with an OEM code page such as CP852)
misinterprets the UTF-8 bytes rclone emits and corrupts remote and path
names containing non-ASCII characters, so tab completion produces a path
that does not exist.
Inject "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8" into the
generated script immediately before the Invoke-Expression call. This is safe
on PowerShell 7+, where UTF-8 is already the default. If the expected line is
not present (for example after a Cobra template change) the script is emitted
unmodified so we never produce a corrupted completion script.
Convert the bug report and feature request templates to GitHub issue forms.
Rewrite the pull request template to require that non-trivial changes are
discussed in an issue first, and to spell out that backend changes need a
clean test_all run and a test account for the integration tester before
they can be merged. Document the latter requirement in CONTRIBUTING.md.
Remove the legacy ISSUE_TEMPLATE.md which is superseded by the forms.
The new "trace" dump flag attaches a net/http/httptrace ClientTrace to
each HTTP transaction and logs the connection level events - DNS
resolution, TCP connect, TLS handshake (including the negotiated TLS
version, cipher, ALPN protocol and server certificate), connection
reuse, request write and time to first response byte. Each line is
tagged with the time elapsed since the start of the transaction and the
request pointer so it can be correlated with the other dumps.
This is complementary to the existing dump flags: it shows how the
connection behaved rather than what was sent, which is useful for
debugging connectivity, DNS, TLS, proxy and keep-alive problems.
The new "errors" dump flag makes the HTTP dump conditional on the
transaction failing with a retryable error (a transport error, HTTP 429
or HTTP 5xx), so first-failure diagnostics can be captured without the
noise of dumping every transaction. The existing dump flags continue to
control what is dumped, for example --dump errors,bodies, and on its own
--dump errors dumps the headers.
TestS3Minio brings up a minio container via the fstest/testserver
framework, which exec's bash init.d scripts that shell out to docker.
This is not available on all platforms - Windows has no POSIX shell to
run the scripts, and macOS CI runners have no docker daemon - which
caused the build to fail there.
Add testy.SkipUnlessDocker to detect whether the framework can run and
skip the test when it cannot.
Previously serve s3 buffered every part of a multipart upload in memory
(in the gofakes3 S3 library) and concatenated them when the upload
completed, so memory use grew with the size of the upload.
serve s3 now streams the parts, in part-number order, into a single
PutStream upload to the underlying remote, which performs its own upload
with bounded memory. The whole file is never held in memory - memory use
is bounded by the parts in flight. This works for any remote that
supports PutStream (nearly all, including crypt) and for any part size,
so clients that don't produce uniform-sized parts (for example
PostgreSQL backup tools such as pgBarman and pgBackRest) work too.
Parts must arrive in ascending, contiguous part-number order; parts
uploaded out of order are buffered until their turn, and there is no
per-part retry (a failure aborts the whole upload). These trade-offs are
documented.
Passing --disable-multipart-streaming, or using a remote without
PutStream, reverts to buffering the parts in memory (the previous
behaviour); a one-off NOTICE is logged the first time this happens.
Fixes#7453
Run still uses a fresh local directory as the backing Fs that the
server wraps. RunWithBackend takes an extra remote name (e.g.
"TestS3Minio:") and uses a random subdirectory of that remote instead,
starting the matching fstest/testserver/init.d script on the way in
and tearing it down on the way out.
AuthProxy is only run for the local backend.
Added information about an alternative, easier way to access folders inside
"Computers" using rclone. Expanded details about folder behavior in "Computers".
Before this change, if the user changed their password or public-key
and the auth proxy script returned updated config parameters for the
backend (eg a rotated api_key) rclone would continue to re-use the old
backend with the old config parameters out of the fscache.
This was because both the VFS cache and the fs/cache key were derived
from the user name only, so a change in the user's password or
public-key did not invalidate the cached backend.
Fix this by deriving the cache key from the user plus a hash of the
password/public-key, so a credential change forces a fresh backend.
The hash uses a per-process random HMAC key so the fragment that
appears in logs cannot be brute-forced offline.
ResetCounters unconditionally restarted the average loop, spawning a
ticker goroutine that pinned the StatsInfo even when no loop had been
running before. statsGroups.delete calls ResetCounters on every removed
group, so deleting N stats groups leaked N goroutines and prevented GC
of the underlying StatsInfo objects.
Only restart the loop if it was active before the reset.
The behavior of the --track-renames and --max-delete flags combination for
bisync have confused me and some other users. So with this PR i added a
paragraph to clarify this.
Tests cover Dir size/count accounting, AttrI behaviour for files and
directories, error propagation, and Remove correctly updating ancestor
totals up the tree.
A connection string can carry global.* options which change rclone's
process-wide configuration (e.g. global.http_proxy). This is
undesirable for the rc interface which was designed to have multiple
users or connections at once. The rc interface has the `_config`
mechanism for setting request scoped global config.
This blocks global.* options on all rc paths by marking the context as
a remote control request at the rc boundaries. fs.NewFs then skips
applying global.* to the process-wide config for a marked context.
The marker is reapplied in fs.CopyConfig, which is the call rclone
uses to detach context but keep config.
global.* options still apply to the individual backend they are set
on, exactly like override.* options; they just no longer leak into the
rest of the process. Remotes created directly on the command line are
unaffected as are remotes defined in the config file.
See: GHSA-qw24-gh76-8rvv
The --rc-serve GET/HEAD file serving path accepted bracketed inline
remotes from the URL and instantiated them, so a single
unauthenticated request could run a command as the rclone user via
backend options such as webdav bearer_token_command or sftp ssh, read
arbitrary local files, or change process-wide config via global.*
options.
This was the GET/HEAD equivalent of the POST hole fixed for
CVE-2026-41179, which only guarded the rc call dispatch path.
Now, unless the rc server has authentication configured or
--rc-no-auth is set, the serve path only allows remotes already
present in the config file: inline remotes, connection string
parameters and bare local paths are rejected. Connection string
global.* options are never honoured on the serve path, even when
authenticated.
See: GHSA-qw24-gh76-8rvv
This PR fixes a bug in the WebDAV backend where directories or files could
randomly "disappear" from listings due to strict and fragile multi-status code
parsing.
Co-authored-by: bright <nako_ruru@sina.com>
Add a redirect policy to the S3 HTTP client so X-Amz-Security-Token is
removed once a redirect chain crosses hosts. Keep stripping it on later
same-host hops in the same chain, since net/http copies headers from the
initial request for each redirect and can otherwise restore the token.
Preserve same-host redirect behavior, retain the standard redirect limit,
and add tests for cross-host, same-host, multi-hop, and redirect-loop cases.
- CVE-2026-42504: mime: quadratic complexity in WordDecoder.DecodeHeader
- CVE-2026-42507: net/textproto: arbitrary input are included in errors without any escaping
- CVE-2026-27145: crypto/x509: split candidate hostname only once
Accounts with shared-album zones (named CMM-*) returned these zones in
the photos changes/database listing alongside the real photo
libraries. rclone treated them as libraries and queried their albums,
but these zones have no CPLAlbumByPositionLive index so the query
failed with BAD_REQUEST / "Index has invalid data", aborting the whole
listing.
Only PrimarySync and SharedSync-* zones are photo libraries, so skip
any other zone during discovery.
See: https://forum.rclone.org/t/sync-to-icloud-gives-index-has-invalid-data-errors/53852
The build matrix relied on setup-go's built-in cache, which keys only on
go.sum with no job differentiation. All matrix jobs computed the same
cache key and raced to save it; since cache keys are immutable, only the
first job to finish saved its cache. That winner was usually a fast job
whose build cache contained none of the cross-compiled architectures, so
the compile_all and ci_beta steps started from a cold cache on every run.
Disable setup-go's cache and add two explicit actions/cache steps to the
build matrix, the android job and the lint job:
- the module cache (~/go/pkg/mod) depends only on go.sum, so it is
shared across all jobs under a single key; it used to be duplicated in
every job's cache. The downloaded module .zip archives are pruned
before saving as they are not needed to build from the extracted
module cache, roughly halving it to ~260 MiB per OS;
- the build cache (compiled artifacts) is specific to OS, arch and Go
version, so it is kept per job, keyed on the job name.
This lets the cross-compile steps reuse per-architecture build artifacts
and keeps the total cache within the repository limit.
Measured on CI, comparing a cold-cache run against the following
warm-cache run:
other_os 23m12s -> 3m35s (compile_all 14m -> 21s)
linux 23m13s -> 12m14s (deploy 11m -> 1m37s,
race test 8m -> 4m45s)
Both jobs now finish well under 15 minutes once the cache is warm.
JSON logs reported "source":"slog/logger.go:256" instead of the real
caller. getCaller skips logging-machinery frames by file path, but
release builds use -trimpath which rewrites the standard library slog
frame's path to "log/slog/logger.go" - matching neither the "/log/" nor
the "log.go" check, so it was reported as the source. Also skip frames
whose function belongs to the log/slog package, which is immune to
-trimpath.
Non-exportable Google documents such as Google My Maps, Sites and
Forms have no export format, so rclone silently leaves them out of all
listings. During a server side move/copy this means they are quietly
left behind.
Emit a one-off notice when such a document is skipped, telling the
user the files are invisible to rclone and pointing them at
--drive-show-all-gdocs to include them in server side copies and
moves.
With cmd/mount2, reading a directory more than once returned the correct
entries on the first read but nothing on subsequent reads. Plain `ls`
triggers this: it does lseek(fd, 0, SEEK_SET) to rewind the directory
before a second getdents.
go-fuse v2.9.0 rewinds a directory stream by calling Seekdir on the
FileSeekdirer interface. dirStream did not implement it, so go-fuse
returned ENOTSUP and produced an empty listing on every read after the
first.
This implements Seekdir on dirStream: a rewind to offset 0 resets the
stream to the start, restoring correct listings on re-read. Non-zero
offsets are uncommon for in-memory listings and still return ENOTSUP,
matching go-fuse's own default. A compile-time interface assertion is
added so signature drift on future go-fuse updates is caught at build
time.
Before: second and subsequent reads of a directory returned no entries.
After: directories list correctly on every read.
See: https://github.com/hanwen/go-fuse/issues/549
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
When moving a file rclone removed the file from its old parent by
looking the parent ID up from the path using the directory cache. When
the source contained two directories with the same name and path, the
cache could resolve to the wrong duplicate, so the removeParents
request didn't match the file's real parent.
This left the file with both its old and new parent, which fails on
shared drives with:
A shared drive item must have exactly one parent., teamDrivesParentLimit
This uses the source object's actual parent ID instead when it is
known, falling back to the path lookup only when the object has zero
or multiple parents.
At least on my macOS Sequoia 15.7.4, the system automatically adds a
com.apple.provenance xattr to files created by processes. This xattr
lacks the "user." prefix so getXattr filters it out, but the metadata
map was already allocated, resulting in an empty non-nil map being
returned instead of nil.
This caused TestMetadata/Symlink/Xattr and TestMetadata/File/Xattr
to fail because they assert the return value is nil when no user
xattrs are present.
The fix checks if the metadata map is empty after filtering and
returns nil if so.
Listing the parts of a multipart upload with a MaxParts smaller than
the number of parts returned wrong PartNumber values and duplicate
parts on pages after the first, and the part number marker regressed
instead of advancing, causing clients to loop forever.
This bumps github.com/rclone/gofakes3 to v0.0.6 pick up the fix.
Drime's folder rename returns success immediately but the children
listing of the renamed folder briefly returns empty before settling,
which made VFS see an empty directory right after a successful rename
and broke subsequent operations on its contents.
After a successful DirMove, poll the renamed folder's listing until it
shows the pre-rename child count, with a 30s timeout and exponential
backoff. The backend integration test already tolerated this via its
own list-consistency retries; this brings VFS into line.
Fixes#9450
Large files (sent as multipart uploads) were placed in the wrong folder
for two reasons:
- the parent folder was sent as "parent_id", but the API ignores that
and expects "parentId", so the parent was never honoured
- relativePath was sent as the full path from the drive root, which made
the server build folders from it and silently drop any "0" path
segment (e.g. ".../data/0/file" lost the "0")
Send the parent as "parentId" and use just the leaf as relativePath,
matching the working single-part upload. This also lets us remove the
now-unneeded absolute-path resolution code.
Fixes#9392
Co-authored-by: Brian King <BrianDKing@gmail.com>
The drime origin returns a malformed response (reported by Cloudflare as
a 520 error) for a literal PUT request to the file-entries update
endpoint, which broke renaming, and so server-side copy and move.
Use a POST with the X-HTTP-Method-Override: PUT header instead - the API
routes this to the same handler and it works reliably.
Also retry Cloudflare 520-524 errors which may occur transiently.
When a SETSTAT request set the access and modification times, the
handler passed the modification time for both, discarding the
requested access time. Pass the requested access time through instead.
The VFS currently ignores the atime, but it might use it one day.
The statvfs@openssh.com extension was advertised but returned an
unsupported status, so clients couldn't query the amount of free and
used space. Implement it using the VFS Statfs method, which reports the
backend's usage where the backend supports About.
The SFTP serve handler ignored the size attribute of SETSTAT/FSETSTAT
requests, only acting on the modification time. This meant a client
asking to truncate a file (eg setting the final size of an upload, or
an explicit truncate) had no effect at all.
This respects the size attribute (if present) by truncating the file
to the requested size.
The SFTP serve write handler always opened files with O_TRUNC,
ignoring the flags requested in the SFTP OPEN packet. Some clients
(notably WinSCP's "Process in Background", which resumes an upload on
a second connection) re-open the partially written file without the
truncate flag and continue writing from the offset they had reached,
relying on the existing data being preserved. Forcing O_TRUNC zeroed
that prefix, so the start of the uploaded file ended up as a block of
zero bytes.
This fix respects the requested open flags instead so a resume open
without truncate keeps the already written data intact.
See: https://forum.rclone.org/t/rclone-serve-sftp-winscp-background-mode-uploading-causes-file-corruption/53841
rclone starts a http listener if the user is adding a remote that requires
oauth. The problem is that sometimes this breaks or the user stops midway with
no way of us knowing that. Once that happens, clients of the api break. In the
best scenario, new remotes cannot be configured without a restart.
This change adds 2 oauth related endpoints, one to get the status of the listener
and one to stop it. This allows clients to go back to normal without having to
restart rclone.
- config/oauthstop - cancels an in-progress OAuth flow and releases the port.
- config/oauthstatus - reports "running" / "stopped".
Make rc respond with a 202 status code (instead of 200) if `Prefer: respond-
async` was passed. Keeps backwards compatibility for current clients while also
allowing the OpenAPI schema & generators to differentiate the responses
properly.
Lets the kernel id-map a mount2 mount into a user namespace
(e.g. Kubernetes pods with hostUsers: false). Off by default;
requires Linux 6.12+ and implies default_permissions.
Previously the Mount RPC ignored the path component of the mount
request, so `server:/sub/dir` and `server:/` both landed at the root
of the served remote. The Mount handler now cleans the requested path
with path.Clean, looks it up in the VFS and serves a billy.Filesystem
rooted at that directory, refusing the mount if the path does not
exist or is not a plain directory.
A pathRewriter cache wraps the inner handle cache so that the same
file always produces the same NFS file handle regardless of which
mount minted it (and stable across server restarts for the disk and
symlink caches). This matches the traditional NFS expectation that a
subpath mount behaves like `cd` into a subtree.
nfsmount gains a --nfs-mount-path flag (default /) so clients can
select a subpath at mount time. This replaces a latent misuse of
--volname as the NFS mount path that was previously masked by the
server ignoring it.
Fixes#9442
shouldRetry was a stub returning false unconditionally, which makes
protondrive the only rclone backend that disables pacer-level retries
entirely. Every other backend at minimum falls back to
fserrors.ShouldRetry(err) so genuine transport-level transients (TCP
resets, brief 5xx) get retried.
- Use errors.As to unwrap proton.APIError instead of string matching
- Retry transient storage block errors (Code=200501)
- Retry server errors (5xx, except 503)
- Skip 429 and 503 (handled by go-proton-api's resty retry layer
via catchTooManyRequests / catchRetryAfter, which honours Retry-After)
- Fall back to fserrors.ShouldRetry for non-API errors
Co-authored-by: tomholford <tomholford@users.noreply.github.com>
The golang.org/x/net/http2/h2c package was deprecated in v0.54.0 in
favour of setting the http.Server Protocols field to enable unencrypted
HTTP/2.
This replaces the h2c.NewHandler wrapping added in e863f751f with
http.Server.Protocols, which is supported by the standard library
since Go 1.24.
Note that the stdlib only supports HTTP/2 prior-knowledge on cleartext
connections, not HTTP/1.1 Upgrade: h2c negotiation. In practice clients
use prior-knowledge or require TLS, so this should not affect users.
Upgrade to v0.55.0 of golang.org/x/net in order to address:
- CVE-2026-42506: html: incorrect handling of namespaced elements in foreign content
- CVE-2026-39821: idna: failure to reject ASCII-only Punycode-encoded labels
- CVE-2026-42502: html: incorrect handling of HTML elements in foreign content
- CVE-2026-25680: html: denial of service when parsing arbitrary HTML
- CVE-2026-25681: html: incorrect handling of character references in DOCTYPE nodes
- CVE-2026-27136: html: duplicate attributes can cause XSS
Upgrade to version v0.52.0 of golang.org/x/crypto to address:
- CVE-2026-46598: ssh/agent: pathological inputs can lead to client panic
- CVE-2026-46597: ssh: byte arithmetic causes underflow and panic
- CVE-2026-39828: ssh: bypass of certificate restrictions
- CVE-2026-39835: ssh: server panic during CheckHostKey/Authenticate
- CVE-2026-39833: ssh/agent: key constraints not enforced
- CVE-2026-39832: ssh/agent: agent constraints dropped when forwarding keys
- CVE-2026-39827: ssh: memory leak when rejecting channels can lead to DoS
- CVE-2026-39830: ssh: client can cause server deadlock on unexpected responses
- CVE-2026-39829: ssh: pathological RSA/DSA parameters may cause DoS
- CVE-2026-39831: ssh: bypass of FIDO/U2F security keys physical interaction
- CVE-2026-39834: ssh: infinite loop on large channel writes
- CVE-2026-42508: ssh/knownhosts: auth bypass via unenforced @revoked status
- CVE-2026-46595: ssh: VerifiedPublicKeyCallback permissions skip enforcement
Upgrade to version v0.41.0 of golang.org/x/image to address:
- CVE-2026-42500: bmp: panic when reading out of bound palette index
- CVE-2026-33809: tiff: excessive resource consumption in PackBits decompression
Upgrade to version v0.45.0 of golang.org/x/sys to address:
- CVE-2026-39824: windows: integer overflow in NewNTUnicodeString
MEGA S4 has launched a new dedicated domain megas4.com with three new
endpoints, including their first Asia-Pacific location in Tokyo, plus
new endpoints in Paris and Barcelona. The existing Amsterdam,
Luxembourg, Montreal and Vancouver locations have been renamed onto
the new domain.
All current s4.mega.io endpoints continue to be supported and are
retained as legacy options. New accounts use the megas4.com endpoints.
The "Import/Export of google documents" section in drive.md and the
"--b2-versions" examples section in b2.md were both at H4 instead of H3,
which excluded them from the ToC even though they are top-level topics
in their respective sections.
Add tenant_url configuration option for accessing OneDrive/SharePoint
without admin privileges. When tenant_url is set, rclone uses the
SharePoint v2.0 API endpoint instead of the standard Microsoft Graph
API.
Co-Authored-By: KyokoMiki <161209740+kyokomiki@users.noreply.github.com>
When mounting with 'remote:.' (dot notation for current directory),
the '.' is passed through toOSPath() which encodes it to full-width
'.' (U+FF0E) via EncodeDot. This caused the VFS cache data/meta root
to be written to a full-width-dot directory while the local cache
backend pointed to the canonicalized path, resulting in writes being
cached but never uploaded.
Fix by calling clean() on the relativeDirPath before toOSPath(),
so path.Clean() strips the trailing '/.' removing the dot before
encoding.
Phonero is a Norwegian mobile provider focusing on the enterprise
market. They provide rebranded Jottacloud service through their «Phonero
Sky» (Norwegian for «Phonero Cloud»).
The Client ID `desktop` used by most of the other whitelabel services
does not work for Phonero Sky. Both `desktop-win` and `desktop-mac`
work, though. Authentication does not work when the `jotta-default`
scope is present, so drop it. This results in a Client ID and scope
setup that is identical with that of Let's Go Cloud (Germany).
When using the drime backend to access a folder shared from another
Drime account (via root_folder_id config option), listing the folder's
children fails with HTTP 500. This makes cross-account read-only
mounts non-functional.
This fixes the problem by changing listAll to use `folderId` instead
of `parentIds[]`
Fixes#9420
Before this change, --conflict-loser pathname assumed --conflict-resolve none,
following the legacy behavior prior to v1.66. This produced unexpected behavior
when used with a different --conflict-resolve option.
This change fixes the issue by ensuring that --conflict-loser pathname looks for
the correct name on the side not being renamed, when only one side should be
renamed.
https://forum.rclone.org/t/bisync-does-not-copy-the-winner-file-to-the-loser-site/53768
The metrics_addr option was registered twice: once explicitly and once
implicitly via AddPrefix(libhttp.ConfigInfo, "metrics", ...). Both
pointed at the same MetricsHTTP.ListenAddr field, so options/info
returned a duplicate entry.
Drop the explicit entry and use SetDefault to keep the empty default
(so the metrics server stays off unless configured), matching the
pattern already used for rc_addr.
Fixes#9419
When using rcat to upload a new version of a file that already existed,
the file upload would succeed. The subsequent deletion of the old file
is attempted after the upload. Drime appears to handle the deletion of
the old file automatically and returns HTTP status code 422, stating
the "The selected entry ids is invalid."
The deletion and the rcat would fail before this change. This is with
file history enabled on my Drime account.
This change detects the error and ignores it since the file has
already been deleted.
- CVE-2026-42501: cmd/go: malicious module proxy can bypass checksum database
- CVE-2026-39825: net/http/httputil: ReverseProxy forwards queries with more than urlmaxqueryparams parameters
- CVE-2026-39836: net: panic in Dial and LookupPort when handling NUL byte on Windows
- CVE-2026-42499: net/mail: quadratic string concatenation in consumePhrase
- CVE-2026-39820: net/mail: quadratic string concatentation in consumeComment
- CVE-2026-39819: cmd/go: "go bug" follows symlinks in predictable temporary filenames
- CVE-2026-39817: cmd/go: "go tool pack" does not sanitize output paths
- CVE-2026-33814: net/http: infinite loop in HTTP/2 transport when given bad SETTINGS_MAX_FRAME_SIZE
- CVE-2026-39826: html/template: escaper bypass leads to XSS
- CVE-2026-33811: net: crash when handling long CNAME response
- CVE-2026-39823: html/template: bypass of meta content URL escaping causes XSS
The bisync normalization test relies on uploading distinct NFC and NFD
versions of the same filename and on the backend supporting in-place
modtime updates. Dropbox normalizes unicode server-side (NFD -> NFC)
and can't set modtime in place, so the test inevitably takes a
different code path on Dropbox and the log diverges from the golden
output without any functional difference.
operations.NeedTransfer's equality check may have deleted pair.Dst as
a precursor to re-uploading it if SetModTime returns
ErrorCantSetModTimeWithoutDelete (e.g. Dropbox). If so skip the eager
delete of the destination if --fix-case will rename it to a different
name. The rename itself replaces the destination, and any subsequent
re-upload happens at the correctly-cased path.
See: #8881
This reverts commit de67f29b3f.
This solved the original Dropbox "from_lookup/not_found" failure, but
broke --fix-case on case-sensitive backends that update modtime via a
server-side copy (such as S3 on Cloudflare R2).
At some point Drime recommended 200M for the upload cutoff for
switching to multipart upload. However uploads have stopped working
using single part upload for 100..200Mish files.
Their docs now recommend 5M as the cutoff for multipart upload so this
changes the default.
The /s3/multipart/create and /s3/entries endpoints interpret relativePath
as an absolute path from the drive root, not relative to parent_id. When
root_folder_id was set to a non-root folder, files larger than
upload_cutoff ended up at the user's drive root instead of the configured
folder.
Resolve the absolute path of the Fs root once via GET /folders/{hash}/path
(cached on first OpenChunkWriter call) and use that to build the correct
relativePath.
Fixes#9392
- Add Data Raven as a silver sponsor
- Add Impossible Cloud as a bronze sponsor
- Shuffle silver sponsors once per page load
- Remove TOC from sponsors page
When enabled, an out-of-space error during a local write returns a
fatal error that aborts the run, instead of being retried.
Without this option, ENOSPC errors are treated as retryable and
rclone may spin through the retry loop many times on a full disk
before giving up. That is fine for transient network errors but
unhelpful when the disk is genuinely full and the operator wants
the run to fail loudly. Default is off so existing behaviour is
unchanged.
Implementation follows the pattern suggested in the issue: a defer
at the top of Update wraps the error with fserrors.FatalError when
the option is on and the error is disk-full. Detection covers both
file.ErrDiskFull from the preallocate path and syscall.ENOSPC from
io.Copy or Close, via a small helper that uses fserrors.IsErrNoSpace.
Add three new regions and their endpoints for Fastly Object Storage:
- eu-west-1 (Paris)
- us-east-1 (Virginia)
- us-west-1 (Oregon)
These are distinct from the existing us-east, us-west and eu-central
endpoints, which are kept in place.
shouldRetry treated every non-nil error as retryable, so permanent
failures (auth, 4xx, not-found) burned through the LowLevelRetries
budget instead of returning fast.
This also fixes the pacer sleeps: pacer.MinSleep(1000) and
MaxSleep(10000) are time.Duration values, so they were 1µs and 10µs -
almost certainly intended as 10ms and 2s.
The TransformFile tests in fs/sync call operations.TransformFile
immediately after MoveDir. On eventually-consistent backends the
internal NewObject lookup can momentarily fail with "object not
found", making the tests flaky.
This wraps the two operations.TransformFile calls in TestTransformFile
and TestManualTransformFile with fstest.Retry
When --fix-case was used (e.g. by bisync) on backends that can't set
modification times in place - such as Dropbox - files whose content
matched but whose modtimes differed would fail to rename with a
"from_lookup/not_found" error and abort the operation.
This happened because operations.NeedTransfer was called before the
fix-case rename. NeedTransfer's equality check would delete the
destination as a precursor to re-uploading it (the standard way to
update a modtime on these backends), so by the time the rename ran the
file no longer existed on the remote.
Fix by running the fix-case rename first, so that any subsequent
delete/re-upload happens at the correctly-cased destination path.
See: #8881
The test set the short idle timeout before creating the test Fs, which
made fs.NewFs fail to read the FTP welcome banner within 1s on slow CI
hosts. Restore the long timeout while NewFs dials the control
connection, then apply the short idle timeout before the upload so the
data connection still exercises the close race that shut_timeout fixes.
The previous wording "Already have a token - refresh?" was misleading
because answering yes triggers a full re-authorization flow, not an
OAuth2 refresh token grant. Updated to "Token already configured -
replace it?" to accurately describe what happens.
Also updated the SugarSync backend which has its own copy of the prompt,
and the docs for box, drive, and onedrive that reference it.
Enable on-the-fly response compression for WebDAV when the client sends
Accept-Encoding and the response content type is suitable for
compression.
This adds compression for the WebDAV responses that benefit most in
practice, notably PROPFIND XML responses and text file downloads.
I tested this with Cyberduck, which sends
`Accept-Encoding: gzip,deflate` and accepted the compressed responses.
Range requests are explicitly left uncompressed.
Fixes#5777
Before this change, the GUI server sent all static files uncompressed,
meaning the browser had to download the full size of every JS, CSS,
and HTML asset.
After this change, the GUI server uses chi's Compress middleware at
level 5, which negotiates gzip or deflate encoding based on the
client's Accept-Encoding header.
This reduces transfer sizes significantly for the web UI assets, for
example assets/index-CvfdU_RR.js is 874 KB uncompressed, and
265 KB compressed.
This is consistent with how rclone serve http, webdav, and restic
already compress their responses.
The stscreds.AssumeRoleProvider from AWS SDK Go v2 does not cache
credentials by itself. The SDK only auto-wraps providers with
aws.CredentialsCache when they are loaded via
config.LoadDefaultConfig; when assigned directly to
aws.Config.Credentials it must be wrapped manually, as documented on
stscreds.NewAssumeRoleProvider.
Without the cache, configurations using role_arn would call AssumeRole
once per S3 request, flooding STS and CloudTrail.
See: https://forum.rclone.org/t/aws-iam-roles-credentials-arent-cached/53732
When a Proton Drive file has no active revision attributes,
readMetaDataForLink returns a nil FileSystemAttrs and Object.originalSize
is left as nil. Object.Open then dereferenced this nil pointer when
calling fs.FixRangeOption, causing a SIGSEGV during copy.
Use Object.Size() instead, which already implements the correct fallback
to the link size when originalSize is unavailable.
This updates the github.com/rclone/Proton-API-Bridge package to fix a
segfault when reading files with no metadata.
Fixes#9377Fixes#9117
Previously all log output produced by Proton-API-Bridge (stdlib log)
and go-proton-api (logrus + resty's logger) bypassed rclone's
logging: it ignored -v / -vv levels and didn't reach --log-file.
Add a small adapter implementing the resty.Logger / bridge Logger
shape that calls fs.Errorf / fs.Logf / fs.Debugf, and pass it via
the new Config.Logger hook. The bridge in turn forwards the same
value to go-proton-api's WithLogger option, so HTTP-layer warnings
and the formerly-hardcoded logrus warnings inside go-proton-api
also surface through rclone's log levels.
The Proton Drive backend constructed the upstream Proton-API-Bridge
without ever passing rclone's HTTP transport. As a result none of
rclone's HTTP flags reached Proton: --dump headers, --dump bodies,
--no-check-certificate, --user-agent, --bind, --ca-cert, --header,
--tpslimit etc. all silently did nothing for this remote, and HTTP
traffic was invisible to -vv.
Pass fshttp.NewTransport(ctx) through the new Config.Transport hook on
the bridge, which forwards it to the updated go-proton-api's
WithTransport option and so to the underlying resty client.
Avoid side effects by using own logger instance
- Importing fs/log only sets rclone's private logger via fs.SetLogger,
so internal rclone logging works from the moment the package is
imported but the process-wide slog default is left untouched.
- slog.SetDefault and slog.SetLogLoggerLevel move into InitLogging,
which is called explicitly from the CLI (cmd/cmd.go), the librclone
wrapper and the integration test framework. So rclone-as-a-program
keeps capturing log.Print/log.Fatal and slog.Default() output as
before.
Library consumers that import fs/log without calling InitLogging now
keep their own slog default and can safely route rclone output back
into it via log.Handler.SetOutput without recursing.
Fixes#8907
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
The S3 ListObjects response from `rclone serve s3` was sorting object
contents by modification time instead of object key. This made the
listing order incompatible with S3 clients which expect lexicographic
key ordering.
In particular, `aws s3 sync` assumes both source and destination
iterators are ordered by key. With the old modtime ordering it could
misidentify files as missing or outdated and re-download objects that
were already up to date.
Change the pager to sort returned objects by key and add a regression
test which uses keys and modtimes arranged so the old behaviour would
fail.
Fixes#9002
Previously `make fetch-gui` extracted the GUI release into cmd/gui/dist/
and the unpacked tree was embedded uncompressed via `//go:embed dist`.
This commits and embeds the GUI bundle (dist.zip) and its release tag
(dist.tag) to the repo so:
- the rclone binary is smaller
- `go build` works on a fresh clone without first running fetch-gui
- a given commit pins an exact GUI version
The "Fetch GUI" step was removed from .github/workflows/build.yml.
next_page is not currently being returned on listings which is causing
the rclone listing code to go wrong. This was returned so is likely a
regression in Drime.
This changes the page counter to calculate using current_page and
last_page. last_page on the first page request is just current_page+1.
drime appears to be capping per_page to 200. as more pages are
requested, last_page increments by 1 until current_page = last_page
Bump go-proton-api and Proton-API-Bridge to versions that send the new
NameSignatureEmail field and omit NodePassphraseSignature/SignatureEmail
for ordinary nodes, matching the schema accepted by the Proton Drive
API. Without this rclone moveto, --backup-dir, server-side rename and
DirMove all failed with "value cannot be empty" / "outdated app" 422.
Fixes#8512
Add read-only iCloud Photos support to the existing iclouddrive
backend via `service = photos` config option.
Also includes auth improvements on top of #9209's SRP authentication.
**Photos features:**
- 3-level hierarchy: libraries (Personal + Shared Photo Library) →
albums → photos/videos
- server-side smart albums (All Photos, Videos, Favorites,
Screenshots, Live, Bursts, Panoramas, Slo-mo, Time-lapse, Portrait,
Long Exposure, Animated, Hidden, Recently Deleted)
- User-created albums and nested album folders
- Live Photo `.MOV` companions as first-class entries
- Edited photo versions (`-edited` suffix) and RAW alternatives
- Duplicate filename dedup for camera counter wrap collisions
- Parallel cold listing for large albums
- Delta sync via CloudKit `changes/zone` - warm listings near-instant from disk cache
- Disk cache (libraries, albums, photos) with atomic writes for crash safety
- `ChangeNotify` support for FUSE mounts via `changes/zone` polling
- `ListR` support for `--fast-list` and recursive operations
- `--metadata` support - width, height, added-time, favorite, hidden
- Fresh download URLs per file - no stale URL failures on long copies
- FUSE mount documentation with recommended flags
**Auth improvements over #9209:**
- SMS 2FA fallback for users without trusted Apple devices
- Explicit push notification request - fixes iOS/macOS 26.4+ where 409
no longer auto-pushes
- Thread safety for concurrent FUSE callers (mutexes on session and client state)
- Session endpoint caching - skips ~5s `/validate` round-trip on warm start
- `Disconnect` support - clears auth state + disk cache
- PCS cookie support for Advanced Data Protection accounts, including
trusted-device approval for PCS cookies
Built on @coughlanio's Photos PoC (Closes#8734) and @mikegillan's SRP auth (#9209).
Fixes#7982
Co-authored-by: Chris Coughlan <chris@coughlan.io>
On Windows, passing "*" as mountPoint to the mount/mount RC command
auto-assigns a drive letter (e.g. "Z:"), but the resolved letter was
never propagated back to mountlib. This caused liveMounts to be keyed
on the literal "*", breaking tracking of multiple mounts and making
unmount unreliable.
Change MountFn to return the actual mount point as an additional
return value. Update MountPoint.Mount() to store the resolved value,
and mountRc() to use it as the liveMounts key. The mount/mount RC
response now returns the actual mountPoint so callers can discover
which drive letter was assigned.
The background kicker goroutine had a bare select outside a for loop,
so the 5s ticker fired at most once before the goroutine exited. The
intent was to run every 5s for the lifetime of the Downloaders.
This wraps the select in a for loop so the ticker fires repeatedly
until ctx is cancelled.
In practice this was benign because every downloader exit and every
successful Write already calls kickWaiters, so the background kicker
is only load-bearing when a waiter is queued, no downloader is
running, and _ensureDownloader failed transiently. In that state,
before this fix, the waiter would hang until another Download() call
or Close() arrived; now it gets retried every 5s and will either
recover or accumulate enough errors to trip maxErrorCount and error
out cleanly.
Implement multipart upload support with configurable chunk size and concurrency options
Enable OpenChunkWriter with per-chunk encryption
Enhance multipart upload handling with new upload cutoff and error management for small files
Samsung TVs have a bug where they duplicate file extensions when both
the title contains an extension and the MIME type indicates the same
file type. For example, "photo.jpg" becomes "photo.jpg.jpg".
Remove extensions from <dc:title> while keeping them in the resource URL
and MIME type. This provides a cleaner display and prevents Samsung TVs
from incorrectly "fixing" what they perceive as missing extensions.
Samsung TVs have strict XML parsers that fail to interpret "
(numeric quote entity) correctly within DIDL-Lite metadata, causing
files to appear as empty folders. By replacing " with "
(named quote entity) in all marshaled XML, Samsung TVs can now
properly parse the metadata and display files.
This handles the "Big 5" XML entities that might cause parsing issues:
- " -> " (double quotes)
- ' -> ' (apostrophes)
- & -> & (ampersands)
- < -> < (less than)
- > -> > (greater than)
While Go's xml.Marshal already uses named entities for &, <, >
characters, this ensures complete protection against any edge cases
where numeric entities might be generated. Samsung TVs are known
to have strict XML parsers that can't handle numeric entities.
Fixes#9346
Samsung TVs sometimes send Browse requests with empty ObjectID
parameters (<ObjectID></ObjectID>) which causes DLNA servers to
return errors. Default empty ObjectID to "0" (root container) to
maintain compatibility.
This fix is based on ReadyMedia/MiniDLNA Bug 311 which documented
the same issue and solution for Samsung TVs.
See #9346
Add xmlns:sec="http://www.sec.co.kr/" namespace to DIDL-Lite responses
as required by Samsung TV DLNA implementations. This namespace is used
by working DLNA servers like MediaBrowser/Emby for Samsung compatibility.
Based on research of open source DLNA servers that successfully work
with Samsung TVs.
See #9346
Containers (directories) never had their Date field set, producing
<dc:date>0001-01-01</dc:date> (Go's zero time) in DIDL-Lite metadata.
This invalid date can confuse strict DLNA clients.
Set the dc:date to the directory's modification time, and as a safety
net, omit the dc:date element entirely when the timestamp is zero.
See #9346
The childCount attribute on DLNA containers was hardcoded to 1
regardless of how many items the directory actually contained. Some
DLNA clients (notably Samsung TVs) use childCount to decide whether
to browse into a container. Report the actual number of directory
entries instead.
See #9346
Samsung TVs are strict DLNA clients that expect SOAP response arguments
in the order defined by the service SCPD (Service Control Protocol
Description). The Browse response was using a Go map which produces
random iteration order, causing arguments like Result, NumberReturned,
TotalMatches, and UpdateID to appear in unpredictable order. Samsung TVs
fail to parse such responses and never proceed to browse directory
children, showing "no content" to the user.
Replace the map[string]string return type with an ordered []soapArg
slice throughout the UPnPService.Handle() interface, ensuring response
arguments always appear in SCPD-defined order.
See #9346
Fix CVE-2026-32952: A malicious NTLM challenge message can causes an slice out
of bounds panic, which can crash any Go process using ntlmssp.Negotiator as an
HTTP transport.
This is in use in rclone in the webdav backend to access sharepoint.
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The fix for #8241 set FileRequestIntent=Backup on
service.ClientOptions so the azfile SDK emits the
x-ms-file-request-intent header was inadvertently dropped in this
commit (released in v1.73.0)
846f193806 azureblob,azurefiles: factor the common auth into a library
This broke azurefiles with OAuth (service principal secret,
certificate, MSI, etc.) with:
400 MissingRequiredHeader: x-ms-file-request-intent
This restores it in the azurefiles SetClientOptions callback. The SDK
only emits the header for TokenCredential auth, so shared-key and SAS
paths are unaffected.
Before this change, uploading to an existing bucket on Ceph (radosgw)
could fail with:
BucketAlreadyExists: 409 Conflict
when rclone attempted to create the destination bucket (which it does
by default unless --s3-no-check-bucket is set).
The Ceph rgw S3 implementation never returns BucketAlreadyOwnedByYou;
it returns BucketAlreadyExists for every CreateBucket on an existing
bucket, even one the caller owns. With the use_already_exists quirk
set to true, rclone wraps BucketAlreadyExists as a non-retriable error
and aborts the transfer.
The Ceph provider used to set useAlreadyExists = true explicitly. When
the s3 providers were converted to YAML files in f28c83c, Ceph did not
set use_already_exists so it picked up the default of true (via
set(&opt.UseAlreadyExists, true, provider.Quirks.UseAlreadyExists)),
which matched the previous behaviour but is the wrong setting for
Ceph.
This sets use_already_exists: false for the Ceph provider so rclone
ignores BucketAlreadyExists on CreateBucket and continues with the
upload.
Side effect: this partially reverts #7871 for the Ceph provider. If a
user tries to create a bucket on Ceph that is actually owned by
someone else, rclone will no longer fail fast at CreateBucket time;
the subsequent object PUT will fail instead. This is unavoidable on
Ceph since the server does not distinguish "already owned by you" from
"owned by someone else".
Parsing a WEBP image with an invalid, large size panics on 32-bit platforms.
This only affects users on 32 bit platforms using the Internxt backend.
See: https://pkg.go.dev/vuln/GO-2026-4961
When Mega.LoginWithKeys() fails to make the API request, it leaves the
object in a state where Mega.FS.root is nil because it could never query
any information about the filesystem tree. An easy way for this to
happen is if the device is not connected to the internet.
Previously, these failures would be ignored, but Fs.findRoot() on the
rclone side is written in a way that assumes the go-meta filesystem will
have a non-nil root. This leads to an immediate nil pointer dereference
when NewFs() calls Fs.findRoot().
This commit fixes the problem by making LoginWithKeys() failures hard
failures, similar to the MultiFactorLogin() path.
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
Replace AuthRequired bool with NoAuth bool on the rc.Call struct and
flip the auth check logic. Previously endpoints were unauthenticated
by default and had to opt in with AuthRequired: true, which led to
security vulnerabilities when developers forgot to set the flag.
Now all endpoints require authentication by default. Only explicitly
safe read-only endpoints are marked with NoAuth: true:
- rc/noop
- rc/error
- rc/list
- core/version
- core/stats
- core/group-list
- core/transferred
- core/du
- cache/stats
- vfs/list
- vfs/stats
- vfs/queue
- job/status
- job/list
See GHSA-25qr-6mpr-f7qx, GHSA-jfwf-28xr-xw6q
The operations/fsinfo RC endpoint was registered without AuthRequired,
allowing unauthenticated callers to instantiate arbitrary backends via
inline backend definitions.
See GHSA-jfwf-28xr-xw6q
Snapshot the NoAuth setting when the RC server is created rather than
reading it from the mutable options struct on each request. This
prevents any runtime mutation of rc.NoAuth (e.g. via options/set)
from disabling the auth gate for protected RC methods.
See GHSA-25qr-6mpr-f7qx
The options/set RC endpoint was registered without AuthRequired,
allowing unauthenticated callers to mutate global runtime options
including rc.NoAuth, which disables the auth gate for all protected
RC methods. Require authentication for options/set.
See GHSA-25qr-6mpr-f7qx
The files.com integration tests for rcat/copyurl were failing because
fs/account.Account was declaring a ReadAt method when the underlying
handle did not support it. The files.com SDK decided to use the ReadAt
method to speed transfers up which failed.
ReadAt and Seek methods were added in this commit to support the
archive command:
409dc75328 accounting: add io.Seeker/io.ReaderAt support to accounting.Account
This fixes the problem by adding new methods to the Account object
WithSeeker/WithReaderAt/WithReadAtSeeker which produce an object with
the desired methods or errors if it isn't possible.
This stops Account advertising things it can't do which is bad Go
practice.
Until now test_all relied entirely on per-goroutine defer finish()
calls in fstest/runs to stop test servers. A Ctrl-C, kill, or panic
aborted those defers and left docker containers running, breaking the
next run.
Register testserver.CleanupAll with lib/atexit so SIGINT/SIGTERM
delivery runs the sweep automatically. Also defer atexit.Run for the
normal exit and unrecovered-panic paths, and call it explicitly
before os.Exit(1) since os.Exit does not fire defers. The fs.Fatalf
call sites above only fire before any server starts so they need no
explicit sweep.
Cleanup today is entirely per-goroutine via the stop closure that Start
returns. If the driver process is killed or panics, those deferred
stops never run and the underlying container keeps running.
Track every remote Start has brought up in a process-local map, and
expose CleanupAll which force-stops each tracked remote via the new
run.bash "force-stop" verb. The returned stop closure is now
sync.Once-wrapped so it and CleanupAll can both fire harmlessly. No
callers yet; wired up in fstest/test_all in a follow-up commit.
run.bash holds a persistent refcount file in the shared state directory
so multiple concurrent tests can share a single container. If a prior
test_all run is killed (e.g. Ctrl-C), the count never reaches zero on
the next run and the container is never stopped - forcing manual
cleanup.
Three fixes, all in fstest/testserver/init.d/run.bash:
- On start, if the refcount is non-zero but no container is running,
treat it as zero. Stops leaking through future runs.
- reset now rm -rfs RUN_ROOT (the per-server state) instead of
RUN_BASE (the shared parent) which was clobbering sibling services.
- New force-stop verb unconditionally stops the container and zeroes
the refcount. This is the primitive that the Go-side cleanup sweep
will call at end-of-run.
Some S3-compatible servers (e.g. Archiware P5) reject requests with an
empty `?delimiter=` query parameter. For recursive listings, pass `nil`
instead of a pointer to an empty string so the parameter is omitted
entirely from the request.
Fixes#9342
The temp directory name used random.String(2) giving only 676 possible
values. When multiple concurrent tests started in the same second, they
shared the same timestamp prefix, causing name collisions and shared
temp directories. This led to lock file conflicts, listing file races,
and file deletion errors.
Increase to random.String(8) to make collisions effectively impossible.
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download.
If --azureblob-decompress is set, this change erases the length and hash on
compressed objects so they can be downloaded successfully, at the cost
of not being able to check the length or the hash of the downloaded
object.
If --azureblob-decompress is not set the compressed files will be downloaded
as-is providing compressed objects with intact size and hash
information.
Fixes#9337
Add a Hugo page at /backends/index.json that exports all the
backend YAML data from docs/data/backends/ as a single JSON file
for external consumption.
Set the User-Agent to include the APN prefix for Azure backends
(azureblob, azurefiles, onelake) to identify rclone as a Microsoft
Partner. The User-Agent is now:
APN/1.0 rclone/1.0 rclone/<version>
Add a ctx parameter to vfs.New() so callers can pass in context
carrying ConfigInfo and FilterInfo. The context is stripped of
cancellation but config and filter values are preserved into a fresh
background context.
Add a ctx field to the VFS struct, initialized in New() from the
existing cancellable context. Propagate this through the cache
subsystem hierarchy.
This ensures proper context cancellation when a VFS shuts down, rather
than using disconnected context.TODO() or context.Background() calls
throughout and paves the way for VFS to have its own config.
The flag.Lookup("test.v") check existed to skip opening a browser
during tests, but the tests don't exercise RunE, so this was never
used. The --no-open-browser flag is sufficient on its own.
Bind the RC server to localhost:0 and read the bound URL back via a
new rcserver.Server.URLs() accessor instead of pre-allocating a port
in cmd/gui. This removes the small TOCTOU race window between
freePort() closing its listener and rcserver claiming the same port.
- Fail gracefully if `make fetch-gui` hasn't been run
- Return errors instead of panic or fatal errrors
- Don't run `make fetch-gui` on every make since we have it in the workflow
The fetch-gui-dist.sh script calls the GitHub releases API
unauthenticated, which is limited to 60 requests/hour per source IP.
GitHub Actions runners share outbound IPs, so this quota is regularly
exhausted.
Pass GITHUB_TOKEN (or GH_TOKEN) as an Authorization header when
present, raising the limit to 1000/hour, and wire secrets.GITHUB_TOKEN
into the workflow step. Local unauthenticated runs still work.
`json:"entry_permissions"` is known to be either empty [] or of
structure {string: boolean}. This may have been a breaking API change on
Drime's side. Because EntryPermissions is not used, the type was changed
to `any` to capture both cases, otherwise we could implement custom
unmarshalling for that type.
These Debugf calls in NewFilter() ran during GlobalOptionsInit(), before
InitLogging() configured the JSON log format. This caused plain-text
debug lines to leak to stderr when --use-json-log was set, breaking
tooling that expected only JSON output.
The resolved time values are already available via --dump filters so
this commit removes the debug messages.
This adds a new gui command which runs an embedded copy of the GUI at
https://github.com/rclone/rclone-web/
The GUI release is fetched as part of the CI build.
The Global Acceleration Endpoint (cos.accelerate.myqcloud.com) of
Tencent COS does not seem to support "CreateBucket" (maybe also other
bucket management operations). Since the acceleration functionality must
be enabled per-bucket in the Tencent Cloud console, the bucket will
always exist before this endpoint is used, so this check can be safely
skipped.
Now, "no_check_bucket = true" will be auto set when using this endpoint.
Why "NewFs()": on-the-fly remotes (connection string remotes), for
example, ":s3,provider=TencentCOS,...:..." will also be fixed.
Why no unit test: I can't find a good way to test "NewFs()" without
leveraging live endpoints. I think we can extract all existing mutations
for different providers (e.g., AWS, Fastly, and Rabata) from "NewFs()"
to a new function in the future.
Some Tencent docs about this CDN endpoint:
- English: Global Acceleration Endpoint | https://www.tencentcloud.com/pt/document/product/436/40700
- Chinese: 对象存储 全球加速概述_腾讯云 | https://cloud.tencent.com/document/product/436/38866
Assisted-By: OpenCode
When a user has --s3-versions set but lacks the s3:GetBucketVersioning
permission, GetBucketVersioning returns an error and isVersioned() caches
the result as false. This caused CleanUpHidden (backend cleanup-hidden) to
silently exit with "bucket is not versioned so not removing old versions",
ignoring the user's explicit --s3-versions flag.
Fix this by trusting the explicit --s3-versions flag in purge(), bypassing
the GetBucketVersioning check when the user has explicitly declared the
bucket is versioned.
Previously ssh.InsecureIgnoreHostKey() was set unconditionally as the
default HostKeyCallback with no indication to the user.
This logs a warning pointing users to the documentation on how to
enable host key validation.
See: https://github.com/rclone/rclone/security/code-scanning/167
Before this change when a cache item was in its grace period (with
HandleCaching) and the file is reopened, _checkObject runs before the
grace timer recovery check. If the remote object's fingerprint changed
_checkObject removes the cache file from disk. However the grace
recovery path still reused the now-stale fd pointing to a deleted
inode, skipping _createFile entirely. This left no cache file on disk,
causing cache.Exists() to return false and breaking
rename-while-writing logic.
Fix this by checking the cache file still exists before reusing the fd
in grace recovery. If the file was removed, close the stale fd and
downloaders and fall through to _createFile.
Also update the fingerprint in item.rename after setting the new object,
preventing unnecessary cache invalidation when a file is reopened after
a rename.
This was discovered in the integration tests on backends that update
modtime on rename (like mailru).
The Linkbox open API (/api/open/file_search) no longer returns download
URLs, breaking all downloads. This switches to using the web API
(/api/file/my_file_list/web) which requires email+password authentication
but returns working download URLs.
This will unfortunately require changing your existing rclone config.
- Add email, password, and web_token config options
- Add web API login via /api/user/login_email with token caching and retry
- Create separate CDN HTTP client with HTTP/2 disabled and browser
User-Agent to avoid CDN fingerprint blocking
- Remove searchOK regex and name-filtering (web API doesn't support it)
In this commit
0db3e7a2a0 vfs: fix slow nfs serve by adding --vfs-handle-caching
We added --vfs-handle-caching but unfortunately forgot to disable it
for the TestRWCacheUpdate test.
Add a configurable grace period (default 5s) that delays closing file
handles and downloaders when the last handle closes. If a new handle
opens within the grace period, it reuses the existing resources.
This fixes 40x performance degradation with serve nfs vs serve sftp
caused by go-nfs opening/reading/closing on every NFS READ RPC, which
destroyed read-ahead prefetch before it could accumulate.
The grace period only applies to non-dirty files so that writeback
proceeds immediately on close.
Fixes#9251
Apple IDs are case-insensitive, but the SRP proof computation (M1)
hashes the username client-side. The old plaintext signin let the
server normalize the case, but with SRP the client must match.
Lowercase the Apple ID before use so mixed-case IDs authenticate
correctly.
Reported-by: ArturKlauser
This fixes China mainland iCloud authentication by deriving the Origin
and Referer headers from authEndpoint instead of hardcoding idmsa.apple.com.
Fixes compatibility with PR #8818 (China region support) and PR #9209
(SRP authentication).
Signed-off-by: Xiangzhe <xiangzhedev@gmail.com>
Apple has deprecated the legacy /appleauth/auth/signin endpoint and
now blocks it, causing "Invalid Session Token" errors for all users
when their trust token expires. The browser login flow now requires
SRP (Secure Remote Password), a cryptographic handshake that never
transmits the password.
Replace Session.SignIn() with a multi-step SRP-6a flow:
1. authStart - initialize session at /authorize/signin
2. authFederate - submit account name to /federate
3. authSRPInit - exchange client public value for salt/B at /signin/init
4. authSRPComplete - send M1/M2 proofs to /signin/complete
The SRP implementation uses the RFC 5054 2048-bit group with SHA-256
and Apple's NoUserNameInX variant. Password derivation supports both
s2k and s2k_fo protocols via SHA-256 + PBKDF2.
The 2FA and trust token flow is unchanged. Auth headers for all
idmsa.apple.com requests now include X-Apple-Auth-Attributes,
X-Apple-Frame-Id, and use Origin/Referer of https://idmsa.apple.com.
Fixes#8587
- replace Bootstrap/jQuery with purpose-built CSS and JS
- remove backend icons from navbar and content pages
- replace remaining FontAwesome icons with inline SVGs, remove FontAwesome
- modernize CSS styling for menus, typography, cards, tables, and code blocks
- add copy-to-clipboard buttons on code blocks using SVG icon
- move TOC to left sidebar with responsive overlay drawer
- add sticky header, top scrollbar and first column for wide tables
- add left/right arrow buttons to scrollable tables
- hide homepage logo on mobile
- make wide menus with filter for Commands and Storage Systems
- add dark mode support based on browser preference
- fix CSS/JS cache busting to use build time
Lockfiles with invalid JSON content caused bisync to fail permanently
because lockFileIsExpired() logged the decode error but still fell
through to the "valid lock file" path with zero-value TimeExpires.
Now when a JSON decode error is detected:
- If --max-lock is set (< basicallyforever): treat garbled lockfile as
expired, mark listings failed, and proceed (safe assumption: the
previous bisync run crashed and left garbage).
- If --max-lock is not set (default): log a clear error telling the
user the lockfile needs manual inspection, and return false.
Make ctest build and run on Windows in addition to Linux/macOS:
- Add OS detection in Makefile using ifeq ($(OS),Windows_NT)
- Use .lib extension and .exe suffix on Windows
- Link Windows system libraries (winmm, ws2_32, ole32)
- Remove unused dlfcn.h include that prevented compilation on Windows
Fix memory management to use RcloneFreeString instead of free for
strings returned by RcloneRPC, as documented in the librclone README.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Commit a3e1312d accidentally replaced io.NopCloser(in) with a bare
io.Reader when assigning req.Body in uploadSinglepartPutObject.
rclone wraps upload readers in an accounting.Account for progress
tracking. When the AWS SDK calls Seek on the body, Account.Seek does
a type assert on the inner reader. With a bare io.Reader the type
is unexpected and causes:
operation error S3: PutObject, serialization failed: internal error:
Seek not implemented for io.nopCloser
With io.NopCloser(in) the type assert works correctly for both seekable
and non-seekable readers.
Restore io.NopCloser(in) to wrap the reader correctly in all cases.
Verified by running both before (regression confirmed) and after (fix
confirmed):
go test ./backend/s3/... ./fs/operations/... -remote TestS3:
Remove the POSIX_FADV_DONTNEED and POSIX_FADV_SEQUENTIAL calls
from the local backend. The DONTNEED calls cause severe spinlock
contention on parallel file systems (and any system with many
concurrent transfers), because each call triggers per-page cache
teardown under a global lock.
Observed on a 256-core system running rclone with 64 parallel
transfers over Lustre: 69% of all CPU cycles were spent in
kernel spinlock contention from the fadvise path, with effective
throughput well below hardware capability.
The kernel's own page reclaim (kswapd) handles eviction more
efficiently from a single context. Since rclone does not always
read files sequentially (e.g. multipart uploads rewind and
re-read blocks), FADV_SEQUENTIAL was also not reliably correct.
This is consistent with the non-Linux behavior (which never
called fadvise) and with restic's decision to remove identical
code (restic/restic#670).
Fixes#7886
This PR optimizes the PROPFIND requests in the webdav backend to only ask for
the specific properties rclone actually needs.
Currently, the generic webdav backend sends an empty XML body during directory
listing (listAll), which causes the server to fall back to allprops by default.
This forces the server to return properties we never use, such as
getcontenttype.
Fetching getcontenttype can be a very expensive operation on the server side.
For instance, in the official golang.org/x/net/webdav library, determining the
content type requires the server to open the file and read the first 500 bytes.
For a directory with 1,300 files in my environment, rclone ls time dropped from
~30s to ~4s (as fast as native ls).
This only applies to the other vendor for backwards compatibility which could be
expanded.
AWS S3 requires Content-MD5 for PutObject with Object Lock parameters.
Since rclone passes a non-seekable io.Reader, the SDK cannot compute
checksums automatically. Buffer the body and compute MD5 manually for
singlepart PutObject and presigned request uploads when Object Lock
parameters are set. Multipart uploads are unaffected as Object Lock
headers go on CreateMultipartUpload which has no body.
Add object_lock_supported provider quirk (default true) to allow
skipping Object Lock integration tests on providers with incomplete
S3 API support. Set to false for GCS which uses non-standard
x-goog-bypass-governance-retention header and doesn't implement
PutObjectLegalHold/GetObjectLegalHold.
Add Multipart and Presigned subtests to Object Lock integration tests
to cover all three upload paths.
Fixes#9199
URLPathEscapeAll was only passing [A-Za-z0-9/] through unencoded, causing
it to percent-encode RFC 3986 unreserved characters (-, ., _, ~). Per RFC
3986 §2.3, unreserved characters MUST NOT be percent-encoded, and a URI
that unnecessarily encodes them is not equivalent to one that does not.
Servers that perform strict path matching without normalising
percent-encoded characters will reject the over-encoded form with a 404.
Before: /files/my-report.pdf → /files/my%2Dreport%2Epdf
After: /files/my-report.pdf → /files/my-report.pdf
Reserved characters (spaces, semicolons, colons, etc.) continue to be
encoded as before.
When extsort.Strings() cannot create temporary files (e.g. due to
apparmor restrictions or permission denied), it returns a nil sorter
with the error on errChan. The code then called Sort() on the nil
sorter, causing a panic.
Check for nil sorter and return the error instead of panicking.
Fixes#9244
The format field was read in Handle() without synchronization while
setFormat() could write it concurrently from InitLogging(). This
caused a data race detected by the race detector, failing
TestListBucketsAuthProxy in cmd/serve/s3.
Fix by protecting all access to format with the existing mutex.
This adds 11 previously-missing rc params for newer bisync features.
It also makes optional parameters truly optional. (Previously, callers were
required to supply every single one, even if using the default value.)
This reduces the number of go routines which can get out of hand when
using large --transfers and --multi-thread-streams from potentially
--multi-thread-streams * --transfers Go routines to --max-memory /
--multi-thread-chunk-size
It serializes the memory allocator in each transfer which should be
good for performance and reduce lock contention.
Before this change server side copies would show at 0% until they were
done then show at 100%.
With support from the backend, server side copies can now be accounted
in real time. This will only work for backends which have been
modified and themselves get feedback about how copies are going.
If the transfer fails, the bytes accounted will be reversed.
Add gzip compression for directory listings and text assets served over HTTP.
This reduces the rclone repository file listing from 40 kB to 8 kB and reduces
the rclone MANUAL.txt from 2.7 MB to 700 kB.
This makes listings and assets served across the network load faster.
The compression level of 5 should be a good balance between size and speed.
The WebDAV implementation already permits redirects on PROPFIND for
listing paths in the `listAll` method but does not permit this for
metadata in `readMetaDataForPath`. This results in a strange experience
for endpoints that heavily use redirects -
```
rclone lsl endpoint:
```
functions and lists `hello_world.txt` in its output but
```
rclone lsl endpoint:hello_world.txt
```
Fails with a HTTP 307.
The git history for this setting indicates this was done to avoid
an issue where redirects cause a verb change to GET in the Go HTTP
client; it does not appear to be problematic with HTTP 307.
To fix, a new `CheckRedirect` function is added in the `rest` library
to force the client to use the same verb across redirects, forcing this
for the PROPFIND case.
Fixes#5063 by documenting that S3 object keys containing
consecutive forward slashes (//) are not supported by rclone.
The issue occurs because rclone normalizes paths like "a//b" to "a/b",
causing "object not found" errors when trying to access the original
object. This documentation addition explicitly warns users about this
limitation and provides workarounds.
Changes:
- Added new subsection "Important note about double slashes (//)"
under "Restricted filename characters" in S3 documentation
- Explains the normalization behavior and its consequences
- Provides clear examples and workarounds
AI Model/Tool Attribution:
- Implemented using opencode AI assistant
- Issue analysis and documentation update performed by AI tools
Resolves: #5063
When specifying --drime-workspace-id, a file greater than the limit at
which file uploads get chunked would ignore the specified ID and get put
into the default workspace instead.
Completes the fix described in commit 2360e65 by properly closing the
chunkwriter by providing the workspace ID to the Drime API call.
Add AU East 1, EU South 1, JP Central 1, UK East 1, and US Central 1
regions and endpoints for Fastly Object Storage.
Also sort the entries alphabetically.
Browsers make a request to /favicon.ico when visiting pages generated
by the HTTP server.
Previously, if remotes did not have a /favicon.ico then the server
responded with a 404, causing browsers to show a default icon.
This adds a tiny fallback embedded PNG rclone favicon to help users
identify the rclone browser tab.
Add support for S3 Object Lock with the following new options:
- --s3-object-lock-mode: set retention mode (GOVERNANCE/COMPLIANCE/copy)
- --s3-object-lock-retain-until-date: set retention date (RFC3339/duration/copy)
- --s3-object-lock-legal-hold-status: set legal hold (ON/OFF/copy)
- --s3-bypass-governance-retention: bypass GOVERNANCE lock on delete
- --s3-bucket-object-lock-enabled: enable Object Lock on bucket creation
- --s3-object-lock-set-after-upload: apply lock via separate API calls
The special value "copy" preserves the source object's setting when used
with --metadata flag, enabling scenarios like cloning objects from
COMPLIANCE to GOVERNANCE mode while preserving the original retention date.
Includes integration tests that create a temporary Object Lock bucket covering:
- Retention Mode and Date
- Legal Hold
- Apply settings after upload
- Override protections using bypass-governance flag
The tests are gracefully skipped on providers that do not support Object Lock.
Fixes#4683Closes#7894#7893#8866
Use URLPathEscapeAll instead of URLPathEscape for path encoding.
URLPathEscape relies on Go's url.URL.String() which only minimally
escapes paths - reserved sub-delimiter characters like semicolons and
equals signs pass through unescaped. Per RFC 3986 section 3.3, these
characters must be percent-encoded when used as literal values in
path segments.
Some WebDAV servers (notably dCache/Jetty) interpret unescaped
semicolons as path parameter delimiters, which truncates filenames
at the semicolon position. URLPathEscapeAll encodes everything
except [A-Za-z0-9/], which is safe for all servers.
Fixes#9082
Tar files created from the current directory (e.g. tar -czf archive.tar.gz .)
produce entries prefixed with "./". When extracting, rclone's character
encoding replaces the "." with a full-width dot (U+FF0E), creating a
spurious directory instead of merging into the destination root.
Strip the leading "./" from NameInArchive before processing. Only "./"
is stripped specifically to avoid enabling path traversal attacks via
"../".
Fixes#9168
Before this change when doing a sync with `--no-traverse` and
`--files-from` we could call `NewObject` a total of `--checkers` *
`--checkers` times simultaneously.
With `--checkers 128` this can exceed the 10,000 thread limit and
fails when run on a local to local transfer because `NewObject` calls
`lstat` which is a syscall which needs an OS thread of its own.
This patch uses a weighted semaphore to limit the number of
simultaneous calls to `NewObject` to `--checkers` instead which won't
blow the 10,000 thread limit and is far more sensible use of OS
resources.
Fixes#9073
These stats weren't being updated in the global stats read by rc
core/stats:
- transferQueue
- deletesSize
- serverSideCopies
- serverSideCopyBytes
- serverSideMoves
- serverSideMoveBytes
Before this change we read sleepTime before acquiring the pacer token
and uses that possibly stale value to schedule the token return. When
many goroutines enter while sleepTime is high (e.g., 10s), each
goroutine caches this 10s value. Even if successful calls rapidly
decay the pacer state to 0, the queued goroutines still schedule 10s
token returns, so the queue drains at 1 req/10s for the entire herd.
This can create multi‑minute delays even after the pacer has dropped
to 0.
After this change we refresh the sleep time after getting the token.
This problem was introduced by the desire to skip reading the pacer
token entirely when sleepTime is 0 in high performance backends (eg
s3, azure blob).
It was possible in the presence of --max-connections and recursive
calls to the pacer to deadlock it leaving all connections waiting on
either a max connection token or a pacer token.
This fixes the problem by making sure we return the pacer token on
schedule if we take it.
This also short circuits the pacer token if sleepTime is 0.
StackPath's object storage service no longer exists and all S3
endpoints are no longer operational.
Before this change, users could select StackPath as an S3 provider
during configuration, but connections would fail as the endpoints no
longer respond and the service has been discontinued.
After this change, StackPath is removed from the list of supported
S3 providers, preventing users from attempting to configure a
non-functional service.
Fixes#9148
- Add new Fastly provider with US East, US West, and EU Central regions
- Add `etag_is_not_md5` quirk for providers with mandatory encryption
- Disable server-side copy for Fastly (not supported)
Before this change, we uses an unguarded type assertion which failed
when NewObject returned a google document instead of an Object.
This change uses the ID() method to read the id which works for all
types of file instead and returns a sensible error if one isn't found.
Fixes#8990
Before this change it was giving back "401 Unauthorized" however
removing the "Authorization: XXXX" from the request fixes the problem
as the auth is in the URL already.
This updates rclone to use forks of the upstream proton drive modules
in preparation for making changes.
The go-proton-api modules has had changes from master merged so rclone
and Proton-API-Bridge are using the same version.
The Google Photos custom client_id documentation was incomplete - it
only mentioned changing the scopes but did not mention that you need
to enable the "Photos Library API" instead of the "Google Drive API"
in step 3 of the referenced Google Drive instructions.
This fixes the documentation to clearly state both differences from
the Google Drive setup:
- Enable "Photos Library API" instead of "Google Drive API"
- Use the Google Photos specific scopes
Co-authored-by: hyusap <paulayush@gmail.com>
This was being done in NewFs instead of NewFsWithConnection like it
should have been which meant calls to NewFsWithConnection were not
initialising the proxy correctly.
Previously if auth keys were provided without a comma then rclone
would only log an INFO message which could mean it went on to serve
without any auth.
The parsing for environment variables was changed in v1.70.0 to make
them work properly with multiple inputs. This means the input is
treated like a mini CSV file which works well except in this case when
the input has commas. This meant `user,auth` without quotes is treated
as two key pairs `user` and `quote`. The correct syntax is
`"user,auth"`. This updates the documentation accordingly.
Before the log re-organisation in:
8d353039a6 log: add log rotation to --log-file
rclone would write any backtraces to the --log-file which was very
convenient for users.
This got accidentally disabled due to a typo which meant backtraces
started going to stderr even if --log-file was supplied.
This fixes the problem.
Fixes previous pull request #8978
An oversight meant that unrestricted API keys
never called b2_list_buckets,
meaning the root remote could not be listed.
The call is now made in the event there are no allowed buckets,
indicating an unrestricted API key
Fixes#9007
When specifying a custom endpoint with a subpath, there is a limitation
in the Google cloud storage integration that the subpath is ignored
during upload operations. For example with the custom endpoint
"example.org/custom/endpoint" on upload the /custom/endpoint is not
reflected.
As this is most likely an issue with the underlying API client, there is
no way to fix this in rclone. By extending the documentation at least
rclone users are made aware of this limitation.
Related forum thread: https://forum.rclone.org/t/googlecloudstorage-custom-endpoint-subpath-removed-for-upload/53059
This change adds first-class metadata support to the Azure Blob backend,
including headers, user metadata, tags, and modtime overrides, and wires
it through uploads and server-side copies.
There is a behavior change in that rclone will now set the "mtime"
custom metadata when doing server side copies to azure and the
`--metadata` argument is given.
- Map standard headers: cache-control, content-disposition, content-encoding,
content-language, content-type to corresponding x-ms-blob-* HTTP headers.
- Map user metadata: any non-reserved keys (excluding x-ms-*) are sent as
blob user metadata. Keys are normalized to lowercase for consistency.
- Support tags: parse `x-ms-tags` as a comma-separated list of key=value
pairs and apply them on uploads and copies.
- Support mtime override: accept `mtime` in metadata (RFC3339/RFC3339Nano)
to override the stored modtime persisted in user metadata.
Backblaze has updated its b2_authorize_account API endpoint, newly created
application keys are now "multi-bucket" keys, capable of being limited to
multiple buckets. These keys can only be used with the v4 endpoint, not v1 which
returns an HTTP 400.
This commit switches authorization to the v4 endpoint, and allowing such keys to
work with any of the allowed buckets.
With multi-bucket keys, missing restricted buckets can be non-fatal.
Supports listing root with multi-bucket API keys
#8947 implemented support for the If-Match and If-None-Match headers for S3 PUT
Object requests; however, this support did not extend to multi-part copy and
upload requests. These headers are implemented via inclusion in the
CompleteMultipartUpload request.
This updates the auto generated code also which was needed for multipart copy.
We accidentally added a non `camelCase` parameter to the rc
(`config_password`)- this fixes it (to `configPassword`) but accepts
the old name too as it has been in a release.
Especially when using rclone via rc it is helpful to configure the box
backend using the contents of the config file instead of heaving to
upload the file to the server that is running rclone.
Before Go 1.23, x509.ParseCertificate accepted certificates with
negative serial numbers. Rejecting these certificates caused a small
number of users to see this error.
From Go 1.23 debug flags can be added to go.mod so this change adds a
debug flag to ensure negative serial numbers are still allowed since
this is a spec violation, not a security issue.
See: https://forum.rclone.org/t/ssl-validation-broken-between-v1-69-1-latest-version/
The If-Match and If-None-Match headers were being dropped rather
than implemented in the Put Object request to S3. These headers
make requests conditional which allow AWS S3 Bucket Policies to
prevent Object overwriting.
The bisync tests have been failing as Dropbox is failing to move just
created objects. This seems to be caused by an eventual consistency
problem so this attempts to fix it by retrying the specific error.
Before this change, if any code called fs.Fatal(f) then it would stop
rclone as designed. However this is not appropriate when using the RC
API - we want the error returned to the user.
This change turns the fs.Fatal(f) call into a panic which is caught by
the RC API handler and returned to the user as a 500 error.
The uloz.to backend was failing to download files, instead returning
an HTML page with a "Slow download" message. This was caused by
recent changes in the uloz.to API.
This commit fixes the issue by making the following changes to the
download process:
1. The `hash` received from the download link API is now appended as a
query parameter to the download URL.
2. The download is now performed using the authenticated `rest` client
to ensure premium access is recognized.
3. The `DeviceID` is now generated dynamically for each download request
to avoid potential rate-limiting of a static ID.
Replace custom rem hugo shortcode template with HTML comment. HTML comments are now
allowed in Hugo without enabling unsafe HTML parsing.
Improve the text in the comment: Remove unnecessary quoting, and avoid impression that
make backenddocs has to be run and results committed, since we have a lint check which
will then report error because we want to prevent manual changes in autogenerated sections.
Disable the markdownlint rule line-length on the autogenerated marker line.
Make the autogenerated marker detection a bit more robust.
See #8942 for more details.
Added support for reading and writing zstd-compressed archives in seekable format
using "github.com/klauspost/compress/zstd" and
"github.com/SaveTheRbtz/zstd-seekable-format-go/pkg".
Bumped Go version from 1.24.0 to 1.24.4 due to requirements of
"github.com/SaveTheRbtz/zstd-seekable-format-go/pkg".
Before this fix using --sftp-ssh with the sftp backend could leave
zombie processes.
This patch fixes the problem that sshClientExternal.session was never
assigned, so Wait() always returned nil without waiting for the SSH
process to exit. This caused zombie processes because the process was
never reaped.
It also ensures that Wait() is only called once on each process.
I gave this issue to Copilot to fix as an experiment. It went off in
the wrong direction to start with and fixed something which wasn't the
problem but still needed fixing. With a bit of a nudge it fixed the
correct problem too.
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
Before this fix there were various issues with the test server
framework, most noticeably servers stopping when they shouldn't
causing timeouts. This was caused by the reference counting in the Go
code not being engineered to work in multiple processes so it was not
working at all properly.
This fix moves the reference counting logic to the start scripts and
in turn removes that logic from the Go code. This means that the
reference counting is now global and works correctly over multiple
processes.
In this commit
d240d044c3 check: improved reporting of differences in sizes and contents
We adjusted the sense of operations.CheckIdenticalDownload to return
true if files are identical as is implied by the name, but we forgot
to invert the logic in the bisync DownloadCheckFn which caused lots of
tests to fail.
94deb6bd6f b2: Add Server-Side encryption support
From the commit above, without setting SSE, rclone would send invalid
SSE requests with empty strings. This is as omitempty only works with
struct pointers not structs.
This commit adds SSE-C (Server-Side Encryption - Customer) support to
the B2 native backend. The server uses a customer provided AES-256 key
to encrypt the files when you upload them to the bucket, and then it
discards your key from the servers RAM after you're done uploading.
The option names and descriptions are based off the S3 backend
implementation as the way S3 and B2 does SSE-C is pretty similar.
Fixes#6585
Give users a way to explicitly acknowledge that pipes, sockets and block
devices are to be ignored without warnings.
This follows the precedent set in commit 6152bab28 (local: add
--skip-links to suppress symlink warnings, 2017-07-21) for ignoring
warnings about symlinks.
* Adds "aix/ppc64" to the cross-compile target list.
* Including AIX in the build tag of "metadata_other.go".
* Excluding AIX from the main ncdu build tags.
* Marking AIX as an unsupported platform for ncdu.
* Excluding AIX from the fallback redirect implementation.
* Excluding AIX from unix build tags to avoid undefined unix.WNOHANG.
We were seeing a (non-fatal) error in our logs:
```
Failed to seek log file to end: seek /proc/1/fd/1: illegal seek
```
Because we open the log file with O_APPEND,
we don't need to manually seek to the end.
As https://pkg.go.dev/os#File.Seek also confirms
that the behavior of `Seek` is not specified
if the file has been opened with O_APPEND,
remove the `Seek` call.
Before this change, you had to modify a fragile data-structure
containing all providers. This often led to things being out of order,
duplicates and conflicts whilst merging. As well as the changes for
one provider being in different places across the file.
After this change, new providers are defined in an easy to edit YAML file,
one per provider.
The config output has been tested before and after for all providers
and any changes are cosmetic only.
Before this change --no-traverse was calling NewObject on directories
(where it would always fail) as well as files. This was very
noticeable when doing syncs with --max-age which were only
transferring a small number of objects. This should have been very
quick, but the NewObject calls for each directory slowed the sync down
a lot.
This changes replaces the check to see if the source entry is an
Object that got missed out from this commit:
88e30eecbf march: fix deadlock when using --no-traverse - fixes#8656
This renames whitelabel authentication to traditional authentication and adds support for
the main Jottacloud service also here, as it can be used as an alternative to the
authentication based on personal login token for those who prefer it. Documentation
also adjusted correspondingly, and restructured the authentication section a bit more
since some of the sections that was under standard authentication in reality also
applies to the traditional authentication.
The oauthutil.Renew was initialized early in NewFs, before the first request to the
service where a token is needed. When token is already expired at the time NewFs is
called, the Renew operation would be triggered immediately, only to abort before actually
performing a token refresh, for reason described in debug message:
Token expired but no uploads in progress - doing nothing
Then later in NewFs, a request to the customer endpoint was made, and since it requires
a valid token it would perform a token refresh after all.
This was not a big problem, but a bit unnecessary, and the debug log messages made it
confusing to understand what rclone was actually doing regarding token refreshing.
If, from debugger, we were forcing the Renew operation to perform actual token refresh,
even if no uploads in process, then it would fail because it actually needs the username
which is retrieved from the customer endpoint
jottacloud root '': Token refresh failed: read metadata failed: error 400: org.springframework.security.core.userdetails.UsernameNotFoundException: Username not found in url! (Bad Request)
Don't think this can happen in any real situations, but better to make sure it never can.
The bouncestorage image hasn't been updated for 4 years and has this
message at the top of the docs:
This repository is outdated; please use dockerswiftaio/docker-swift instead.
However, dockerswiftaio/docker-swift hasn't been updated for 2 years.
Switch to openstackswift/saio instead, which is getting regular updates.
This requires some minor changes to one test, and how we start the
container.
On my system there needs to be a slight pause between stopping and
checking to see if SwiftAIO has stopped. Without the pause the tests fail for
a non-obvious reason.
Instead of using a magic sleep, re-use the retry logic that is used for
starting the test server.
We understand you are having a problem with rclone; we want to help you with that!
**STOP and READ**
**YOUR POST WILL BE REMOVED IF IT IS LOW QUALITY**:
Please show the effort you've put into solving the problem and please be specific.
People are volunteering their time to help! Low effort posts are not likely to get good answers!
If you think you might have found a bug, try to replicate it with the latest beta (or stable).
The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
If you can still replicate it or just got a question then please use the rclone forum:
https://forum.rclone.org/
for a quick response instead of filing an issue on this repo.
If nothing else helps, then please fill in the info below which helps us help you.
**DO NOT REDACT** any information except passwords/keys/personal info.
You should use 3 backticks to begin and end your paste to make it readable.
Make sure to include a log obtained with '-vv'.
You can also use '-vv --log-file bug.log' and a service such as https://pastebin.com or https://gist.github.com/
Thank you
The Rclone Developers
-->
#### The associated forum post URL from `https://forum.rclone.org`
#### What is the problem you are having with rclone?
#### What is your rclone version (output from `rclone version`)
#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)
#### Which cloud storage system are you using? (e.g. Google Drive)
#### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)
#### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)
<!--- Please keep the note below for others who read your bug report. -->
#### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
about:Suggest a new feature or enhancement for rclone
---
<!--
Welcome :-)
So you've got an idea to improve rclone? We love that!
You'll be glad to hear we've incorporated hundreds of ideas from contributors already.
Probably the latest beta (or stable) release has your feature, so try to update your rclone.
The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
If it still isn't there, here is a checklist of things to do:
1. Search the old issues for your idea and +1 or comment on an existing issue if possible.
2. Discuss on the forum: https://forum.rclone.org/
3. Make a feature request issue (this is the right place!).
4. Be prepared to get involved making the feature :-)
Looking forward to your great idea!
The Rclone Developers
-->
#### The associated forum post URL from `https://forum.rclone.org`
#### What is your current rclone version (output from `rclone version`)?
#### What problem are you are trying to solve?
#### How do you think rclone should be changed to solve that?
<!--- Please keep the note below for others who read your feature request. -->
#### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
Thank you very much for contributing code or documentation to rclone! Please
fill out the following questions to make it easier for us to review your
changes.
Thank you for contributing to rclone!
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
Please discuss anything more than a trivial fix in an issue FIRST - see the
"Linked issue" section below. Then fill in the questions to help us review.
-->
#### What is the purpose of this change?
#### What does this change do?
<!-- Describe the change here. -->
#### Linked issue
<!--
Describe the changes here
Put `Fixes #1234` here if this closes an issue, or `#1234` to link without closing.
IMPORTANT: Anything beyond a trivial fix (typo, doc tweak, small obvious bug fix)
should be discussed and agreed in an issue BEFORE you open a pull request.
Pull requests for larger changes that have not been discussed in an issue first
may be closed. This saves everyone's time if the change needs a different approach
or isn't a good fit.
-->
#### Was the change discussed in an issue or in the forum before?
Fixes #
#### For new or changed backends
<!--
Link issues and relevant forum posts here.
Backend pull requests are often sent without the integration tests having been run.
We cannot merge a backend change until it passes the integration tests.
To be merged, a backend change REQUIRES:
1. A clean run of `go run ./fstest/test_all -backends <remote>` (summarise the
result below or paste a screenshot of the test_all output in the web browser).
2. A test account so the maintainers can run the integration tests daily against
the backend on the integration test server.
If the tests aren't quite passing yet and you need help getting them
to pass, then submit the PR and we can help getting you over the line.
See https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-new-backend
and https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#integration-tests
-->
#### Checklist
- [ ] This change is trivial **OR** it has been discussed and agreed in the linked issue.
- [ ] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [ ]**(If I used AI tools to help write this code)** I have read and understood the [AI-assisted contributions guidance](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#ai-assisted-contributions), and I have tested and take ownership of this change myself.
- [ ] I have added tests for all changes in this PR if appropriate.
- [ ] I have added documentation for the changes if appropriate.
- [ ] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [ ]I'm done, this Pull Request is ready for review :-)
- [ ]**(Backend changes only)** `test_all` passes for this backend and if submitting a new backend can provide a test account for the integration tester - see [CONTRIBUTING.md](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#integration-tests).
This file provides guidance to AI coding agents (e.g. Claude Code, Codex, Cursor, Gemini CLI, and similar tools) when working with code in this repository.
Rclone welcomes AI-assisted contributions, but the expectation is that you, the human submitter, understand every line you propose and have compiled and tested it against real rclone code - not just generated it. See the "AI-assisted contributions" section of [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request.
## Project Overview
Rclone is a command-line program to sync files and directories to and from cloud storage providers. It's written in Go and supports 70+ backends (cloud storage systems). Think "rsync for cloud storage".
## General Notes
**We take backwards compatibility very seriously.** PRs should not change the observable behaviour of existing commands, flags or rc API without very good reason. Rclone does not try to preserve a stable Go API but try not to change it gratuitously.
Rclone operates with a lot of different backends, so **compatibility is key**. It is the backend integration tests which guarantee that compatibility. Changes should consider both known and unknown backends and should take care not to break functionality of existing installations.
The core parts of rclone under `fs` and `vfs` need to work with all backends and **backend specific hacks won't be merged**. Fixes likely need to go in the relevant backend or if new behaviour is really needed, a new Feature flag needs to be added.
**Changes should be kept to the minimum.** Work hard to make the most elegant, smallest change you can. Do not refactor or re-order code unless necessary as this makes review more challenging. Re-use existing test scaffolding, existing or library routines (e.g. `lib`) where possible.
Make sure added tests **actually test the code you have written** and test the intention behind the change. If you are fixing a problem, write the tests first to reproduce the problem before starting on the fix.
Tests in `fs/operations`, `fs/sync`, `cmd/bisync`, `cmd/gitannex` and `vfs` (the definitive list is in `fstest/test_all/config.yaml`) are run on the integration test machine against **all** the backends, so they **must be prepared to run on backends with limited functionality**. Ideally tests, or parts of tests, should be skipped using the `Fs.Features()` flags on backends which can't support them.
## Build and Test Commands
```bash
# Build rclone (simple)
go build
# Build with version info (preferred)
make
# Run all unit tests (no cloud credentials needed)
make quicktest
# or equivalently:
RCLONE_CONFIG="/notfound" go test ./...
# Run tests for a specific package
cd backend/memory && go test -v
# or from root:
go test -v ./backend/memory/
# Run a single test
go test -v -run TestIntegration/FsCheckWrap ./backend/memory/
# Run tests with race detector
make racequicktest
# Lint (requires golangci-lint)
golangci-lint run ./...
# Run backend integration tests (requires configured TestRemote remote)
cd backend/drive && go test -v
# Run sync/operations integration tests against a remote
cd fs/sync && go test -v -remote TestDrive:
cd fs/operations && go test -v -remote TestDrive:
# Run integration tests via test framework
go run ./fstest/test_all -backends drive
```
## Architecture
### Entry Point and Plugin Registration
`rclone.go` is the main entry point. It imports `backend/all` and `cmd/all` which use Go's `init()` pattern to register all backends and commands. Each backend calls `fs.Register()` with a `fs.RegInfo` struct during init.
### Core Interfaces (`fs/`)
The `fs` package defines the core abstractions:
- **`fs.Fs`** (`fs/types.go`): The filesystem interface every backend must implement (List, NewObject, Put, Mkdir, Rmdir).
- **`fs.Object`** (`fs/types.go`): Interface for a file/object (Open, Update, Remove, SetModTime).
- **`fs.Features`** (`fs/features.go`): Optional capabilities a backend can declare (Purge, Copy, Move, DirMove, etc.). Backends set function pointers for operations they support; nil means not supported.
- **`fs.RegInfo`** (`fs/registry.go`): Registration metadata for a backend including its name, config options, and NewFs constructor.
### Backend Structure (`backend/`)
Each backend is a single Go package (e.g., `backend/s3/`, `backend/drive/`). Key conventions:
- Main implementation in a single file (e.g., `s3.go`) - **do not** split into `fs.go`/`object.go`.
- API types go in a separate `api/types.go` file.
- Test file (e.g., `s3_test.go`) uses `fstests.Run()` from `fstest/fstests` for standardized integration tests.
- Register in `backend/all/all.go` via blank import.
- HTTP-based backends should use `lib/rest` for HTTP calls and `fs/fshttp` for the HTTP client.
- Use `lib/dircache` for directory-ID-based remotes, `lib/oauthutil` for OAuth, `lib/pacer` for rate limiting.
### Command Structure (`cmd/`)
Each command is a package under `cmd/` registered in `cmd/all/all.go` via blank import. Commands use cobra via `cmd.Main()`.
- **`fs/march/`**: Parallel directory tree walker used by sync
- **`fs/filter/`**: Include/exclude filtering
- **`fs/accounting/`**: Transfer statistics and bandwidth limiting
- **`fs/config/`**: Configuration file management
- **`vfs/`**: Virtual filesystem layer (used by mount, serve)
- **`librclone/`**: C-compatible library interface for embedding rclone
- **`fstest/`**: Integration test framework; `fstest/fstests/` has the generic backend test suite
## Commit Message Convention
Prefix with the directory of the change, then a colon: `drive: add team drive support - fixes #885`. For cross-cutting changes use a broader prefix like `fs` or `operations`.
Make the first line of your commit message a summary of the change that a user (not a developer) of rclone would like to read. So write `drive: fix server side copy of big files` instead of `drive: no longer set the MimeType in Move or Copy`. This is important because these lines go into the change log which is read by users.
## Code Commenting Style
Comments describe the code as it is now, for a future reader who has no knowledge of the change that introduced it.
Every exported type, function, field, and constant has a **godoc comment** that starts with its name and is phrased as a present-tense statement of what it is or does (`// Mkdir makes the directory (container, bucket)`)
**Document the contract** callers need - preconditions, what's returned, which sentinel errors are returned and when, and any "shouldn't return an error if it already exists" style caveats - rather than the implementation.
**Keep comments terse**: a single line for most things, with extra paragraphs (separated by blank `//` lines) reserved for genuine subtlety.
Inline comments inside function bodies should **explain why** - a non-obvious API quirk, a workaround, a gotcha, or an ordering constraint - and may cite an external reference (forum thread, vendor docs, RFC) when that's what makes the behaviour non-obvious; skip comments that merely restate what the code plainly does.
Use `FIXME` and `TODO` for known shortcomings.
**Do not write comments that narrate the change itself** or compare against the previous behaviour (no "now we also handle...", "changed to...", "previously this returned...", or references to bug/PR numbers in the code) - that context belongs in the commit message, not in source that will outlive the change.
Code comments should only refer to **current** behaviour - and shouldn't describe behaviour that is trivially deducible from reading the code.
## Linting Configuration
Uses golangci-lint v2 with config in `.golangci.yml`. Enabled linters: errcheck, govet, ineffassign, staticcheck, unused, gocritic, misspell, revive, unconvert. The `goimports` formatter is also enabled.
## Documentation
- Backend option docs come from `Help:` fields in the Go source Options structs, not from markdown files.
- Command docs are in the command source code (e.g., `cmd/ls/ls.go`).
- Don't commit autogenerated doc changes from `make backenddocs` or `make commanddocs`.
- Website docs are in `docs/content/` as markdown, built with Hugo (`make serve` to preview).
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
It may be more convenient to put the credentials directly into the
rclone config file under the `+"`client_id`, `tenant` and `client_secret`"+`
keys instead of setting `+"`service_principal_file`"+`.
`,
Advanced:true,
},{
Name:"disable_instance_discovery",
Help:`Skip requesting Microsoft Entra instance metadata
This should be set true only by applications authenticating in
disconnected clouds, or private clouds such as Azure Stack.
It determines whether rclone requests Microsoft Entra instance
metadata from `+"`https://login.microsoft.com/`"+` before
authenticating.
Setting this to true will skip this request, making you responsible
for ensuring the configured authority is valid and trustworthy.
`,
Default:false,
Advanced:true,
},{
Name:"use_msi",
Help:`Use a managed service identity to authenticate (only works in Azure).
When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
to authenticate to Azure Storage instead of a SAS token or account key.
If the VM(SS) on which this program is running has a system-assigned identity, it will
be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
the user-assigned identity will be used by default. If the resource has multiple user-assigned
identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
msi_client_id, or msi_mi_res_id parameters.`,
Default:false,
Advanced:true,
},{
Name:"msi_object_id",
Help:"Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_mi_res_id specified.",
Advanced:true,
Sensitive:true,
},{
Name:"msi_client_id",
Help:"Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_object_id or msi_mi_res_id specified.",
Advanced:true,
Sensitive:true,
},{
Name:"msi_mi_res_id",
Help:"Azure resource ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_object_id specified.",
Advanced:true,
Sensitive:true,
},{
Name:"use_emulator",
Help:"Uses local storage emulator if provided as 'true'.\n\nLeave blank if using real azure storage endpoint.",
Default:false,
Advanced:true,
},{
Name:"use_az",
Help:`Use Azure CLI tool az for authentication
Set to use the [Azure CLI tool az](https://learn.microsoft.com/en-us/cli/azure/)
as the sole means of authentication.
Setting this can be useful if you wish to use the az CLI on a host with
a System Managed Identity that you do not want to use.
Don't set env_auth at the same time.
`,
Default:false,
Advanced:true,
},{
Name:"endpoint",
Help:"Endpoint for the service.\n\nLeave blank normally.",
Advanced:true,
}}
// Options defines the common auth configuration for azure backends
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to files data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
**NB** this section needs updating for Azure Files - pull requests appreciated!
It may be more convenient to put the credentials directly into the
rclone config file under the `+"`client_id`, `tenant` and `client_secret`"+`
keys instead of setting `+"`service_principal_file`"+`.
`,
Advanced:true,
},{
Name:"use_msi",
Help:`Use a managed service identity to authenticate (only works in Azure).
When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
to authenticate to Azure Storage instead of a SAS token or account key.
If the VM(SS) on which this program is running has a system-assigned identity, it will
be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
the user-assigned identity will be used by default. If the resource has multiple user-assigned
identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
msi_client_id, or msi_mi_res_id parameters.`,
Default:false,
Advanced:true,
},{
Name:"msi_object_id",
Help:"Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_mi_res_id specified.",
Advanced:true,
Sensitive:true,
},{
Name:"msi_client_id",
Help:"Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_object_id or msi_mi_res_id specified.",
Advanced:true,
Sensitive:true,
},{
Name:"msi_mi_res_id",
Help:"Azure resource ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_object_id specified.",
Advanced:true,
Sensitive:true,
},{
Name:"disable_instance_discovery",
Help:`Skip requesting Microsoft Entra instance metadata
This should be set true only by applications authenticating in
disconnected clouds, or private clouds such as Azure Stack.
It determines whether rclone requests Microsoft Entra instance
metadata from `+"`https://login.microsoft.com/`"+` before
authenticating.
Setting this to true will skip this request, making you responsible
for ensuring the configured authority is valid and trustworthy.
`,
Default:false,
Advanced:true,
},{
Name:"use_az",
Help:`Use Azure CLI tool az for authentication
Set to use the [Azure CLI tool az](https://learn.microsoft.com/en-us/cli/azure/)
as the sole means of authentication.
Setting this can be useful if you wish to use the az CLI on a host with
a System Managed Identity that you do not want to use.
Don't set env_auth at the same time.
`,
Default:false,
Advanced:true,
},{
Name:"endpoint",
Help:"Endpoint for the service.\n\nLeave blank normally.",
Advanced:true,
},{
Name:"chunk_size",
Help:`Upload chunk size.
@@ -322,38 +131,18 @@ You will need this much free space in the share as the file will be this size te
encoder.EncodeInvalidUtf8|
encoder.EncodeCtl|encoder.EncodeDel|
encoder.EncodeDot|encoder.EncodeRightPeriod),
}},
}}),
})
}
// Options defines the configuration for this backend
// ServerSideEncryption is a configuration object for B2 Server-Side Encryption
typeServerSideEncryptionstruct{
Modestring`json:"mode"`
Algorithmstring`json:"algorithm"`// Encryption algorithm to use
CustomerKeystring`json:"customerKey"`// User provided Base64 encoded key that is used by the server to encrypt files
CustomerKeyMd5string`json:"customerKeyMd5"`// An MD5 hash of the decoded key
}
// Timestamp is a UTC time when this file was uploaded. It is a base
// 10 number of milliseconds since midnight, January 1, 1970 UTC. This
// fits in a 64 bit integer such as the type "long" in the programming
@@ -125,23 +133,32 @@ type File struct {
Infomap[string]string`json:"fileInfo"`// The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
}
// AuthorizeAccountResponse is as returned from the b2_authorize_account call
typeAuthorizeAccountResponsestruct{
// StorageAPI is as returned from the b2_authorize_account call
typeStorageAPIstruct{
AbsoluteMinimumPartSizeint`json:"absoluteMinimumPartSize"`// The smallest possible size of a part of a large file.
AccountIDstring`json:"accountId"`// The identifier for the account.
Allowedstruct{// An object (see below) containing the capabilities of this auth token, and any restrictions on using it.
BucketIDstring`json:"bucketId"`// When present, access is restricted to one bucket.
BucketNamestring`json:"bucketName"`// When present, name of bucket - may be empty
Capabilities[]string`json:"capabilities"`// A list of strings, each one naming a capability the key has.
Buckets[]struct{// When present, access is restricted to one or more buckets.
IDstring`json:"id"`// ID of bucket
Namestring`json:"name"`// When present, name of bucket - may be empty
}`json:"buckets"`
Capabilities[]string`json:"capabilities"`// A list of strings, each one naming a capability the key has for every bucket.
NamePrefixany`json:"namePrefix"`// When present, access is restricted to files whose names start with the prefix
}`json:"allowed"`
APIURLstring`json:"apiUrl"`// The base URL to use for all API calls except for uploading and downloading files.
AuthorizationTokenstring`json:"authorizationToken"`// An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
DownloadURLstring`json:"downloadUrl"`// The base URL to use for downloading files.
MinimumPartSizeint`json:"minimumPartSize"`// DEPRECATED: This field will always have the same value as recommendedPartSize. Use recommendedPartSize instead.
RecommendedPartSizeint`json:"recommendedPartSize"`// The recommended size for each part of a large file. We recommend using this part size for optimal upload performance.
}
// AuthorizeAccountResponse is as returned from the b2_authorize_account call
typeAuthorizeAccountResponsestruct{
AccountIDstring`json:"accountId"`// The identifier for the account.
AuthorizationTokenstring`json:"authorizationToken"`// An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
APIsstruct{// Supported APIs for this account / key. These are API-dependent JSON objects.
StorageStorageAPI`json:"storageApi"`
}`json:"apiInfo"`
}
// ListBucketsRequest is parameters for b2_list_buckets call
typeListBucketsRequeststruct{
AccountIDstring`json:"accountId"`// The identifier for the account.
@@ -261,10 +278,11 @@ type GetFileInfoRequest struct {
BucketIDstring`json:"bucketId"`//The ID of the bucket that the file will go in.
Namestring`json:"fileName"`// The name of the file. See Files for requirements on file names.
ContentTypestring`json:"contentType"`// The MIME type of the content of the file, which will be returned in the Content-Type header when downloading the file. Use the Content-Type b2/x-auto to automatically set the stored Content-Type post upload. In the case where a file extension is absent or the lookup fails, the Content-Type is set to application/octet-stream.
Infomap[string]string`json:"fileInfo"`// A JSON object holding the name/value pairs for the custom file info.
BucketIDstring`json:"bucketId"`//The ID of the bucket that the file will go in.
Namestring`json:"fileName"`// The name of the file. See Files for requirements on file names.
ContentTypestring`json:"contentType"`// The MIME type of the content of the file, which will be returned in the Content-Type header when downloading the file. Use the Content-Type b2/x-auto to automatically set the stored Content-Type post upload. In the case where a file extension is absent or the lookup fails, the Content-Type is set to application/octet-stream.
Infomap[string]string`json:"fileInfo"`// A JSON object holding the name/value pairs for the custom file info.
ServerSideEncryption*ServerSideEncryption`json:"serverSideEncryption,omitempty"`// A JSON object holding values related to Server-Side Encryption
}
// StartLargeFileResponse is the response to StartLargeFileRequest
@@ -325,21 +343,25 @@ type CancelLargeFileResponse struct {
// CopyFileRequest is as passed to b2_copy_file
typeCopyFileRequeststruct{
SourceIDstring`json:"sourceFileId"`// The ID of the source file being copied.
Namestring`json:"fileName"`// The name of the new file being created.
Rangestring`json:"range,omitempty"`// The range of bytes to copy. If not provided, the whole source file will be copied.
MetadataDirectivestring`json:"metadataDirective,omitempty"`// The strategy for how to populate metadata for the new file: COPY or REPLACE
ContentTypestring`json:"contentType,omitempty"`// The MIME type of the content of the file (REPLACE only)
Infomap[string]string`json:"fileInfo,omitempty"`// This field stores the metadata that will be stored with the file. (REPLACE only)
DestBucketIDstring`json:"destinationBucketId,omitempty"`// The destination ID of the bucket if set, if not the source bucket will be used
SourceIDstring`json:"sourceFileId"`// The ID of the source file being copied.
Namestring`json:"fileName"`// The name of the new file being created.
Rangestring`json:"range,omitempty"`// The range of bytes to copy. If not provided, the whole source file will be copied.
MetadataDirectivestring`json:"metadataDirective,omitempty"`// The strategy for how to populate metadata for the new file: COPY or REPLACE
ContentTypestring`json:"contentType,omitempty"`// The MIME type of the content of the file (REPLACE only)
Infomap[string]string`json:"fileInfo,omitempty"`// This field stores the metadata that will be stored with the file. (REPLACE only)
DestBucketIDstring`json:"destinationBucketId,omitempty"`// The destination ID of the bucket if set, if not the source bucket will be used
SourceServerSideEncryption*ServerSideEncryption`json:"sourceServerSideEncryption,omitempty"`// A JSON object holding values related to Server-Side Encryption for the source file
DestinationServerSideEncryption*ServerSideEncryption`json:"destinationServerSideEncryption,omitempty"`// A JSON object holding values related to Server-Side Encryption for the destination file
}
// CopyPartRequest is the request for b2_copy_part - the response is UploadPartResponse
typeCopyPartRequeststruct{
SourceIDstring`json:"sourceFileId"`// The ID of the source file being copied.
LargeFileIDstring`json:"largeFileId"`// The ID of the large file the part will belong to, as returned by b2_start_large_file.
PartNumberint64`json:"partNumber"`// Which part this is (starting from 1)
Rangestring`json:"range,omitempty"`// The range of bytes to copy. If not provided, the whole source file will be copied.
SourceIDstring`json:"sourceFileId"`// The ID of the source file being copied.
LargeFileIDstring`json:"largeFileId"`// The ID of the large file the part will belong to, as returned by b2_start_large_file.
PartNumberint64`json:"partNumber"`// Which part this is (starting from 1)
Rangestring`json:"range,omitempty"`// The range of bytes to copy. If not provided, the whole source file will be copied.
SourceServerSideEncryption*ServerSideEncryption`json:"sourceServerSideEncryption,omitempty"`// A JSON object holding values related to Server-Side Encryption for the source file
DestinationServerSideEncryption*ServerSideEncryption`json:"destinationServerSideEncryption,omitempty"`// A JSON object holding values related to Server-Side Encryption for the destination file
}
// UpdateBucketRequest describes a request to modify a B2 bucket
// Read the chunk into memory from the global pool, hashing
// it for the whole file hash (must be done sequentially)
rw:=multipart.NewRW().Reserve(reqSize)
_,err=io.CopyN(rw,io.TeeReader(in,hash),reqSize)
iferr!=nil{
_=rw.Close()
err=fmt.Errorf("multipart upload failed to read source: %w",err)
breakouter
}
// Make the global hash (must be done sequentially)
_,_=hash.Write(buf)
// Transfer the chunk
wg.Add(1)
o.fs.uploadToken.Get()
gofunc(partint,positionint64){
deferwg.Done()
defero.fs.uploadToken.Put()
deferfunc(){_=rw.Close()}()
fs.Debugf(o,"Uploading part %d/%d offset %v/%v part size %v",part+1,session.TotalParts,fs.SizeSuffix(position),fs.SizeSuffix(size),fs.SizeSuffix(chunkSize))
fs.Logf(nil,"crypt: directory %q has a legacy encrypted name - rename %q on the underlying remote to %q to make it accessible",out,in,c.encryptFileName(out,false))
returnout,nil
}
}
returnout,err
}
// NameEncryptionMode returns the encryption mode in use for names
constconcurrentLimitBody=`{"ok":false,"error":"You have 5 uploads in progress. Wait for them to finish before starting more.","error_code":"concurrent_upload_limit","max_concurrent_uploads":5}`
opt.Help="Google Application Client Id\nSetting your own is recommended.\nSee https://rclone.org/drive/#making-your-own-client-id for how to create your own.\nIf you leave this blank, it will use an internal key which is low performance."
opt.Help="Google Application Client Id\nLeave blank to use rclone's shared client_id, or if you are using a service account.\nThe shared client_id is being retired and will stop working during 2026, so creating your own is now strongly recommended.\nSee https://rclone.org/drive/#making-your-own-client-id for how to create your own."
}
ifopt.Name==config.ConfigClientSecret{
opt.Help="Google Application Client Secret\nLeave blank to use rclone's shared client_id, or if you are using a service account.\nIf you created your own client_id then enter its client secret here."
returnfs.ConfigConfirm("teamdrive_ok",false,"config_change_team_drive","Configure this as a Shared Drive (Team Drive)?\n")
}
returnfs.ConfigConfirm("teamdrive_change",false,"config_change_team_drive",fmt.Sprintf("Change current Shared Drive (Team Drive) ID %q?\n",opt.TeamDriveID))
case"teamdrive_ok":
ifconfig.Result=="false"{
ifconfigIn.Result=="false"{
m.Set("team_drive","")
returnnil,nil
}
returnfs.ConfigGoto("teamdrive_config")
case"teamdrive_change":
ifconfig.Result=="false"{
ifconfigIn.Result=="false"{
returnnil,nil
}
returnfs.ConfigGoto("teamdrive_config")
@@ -278,14 +301,14 @@ func init() {
returnteamDrive.Id,teamDrive.Name
})
case"teamdrive_final":
driveID:=config.Result
driveID:=configIn.Result
m.Set("team_drive",driveID)
m.Set("root_folder_id","")
opt.TeamDriveID=driveID
opt.RootFolderID=""
returnnil,nil
}
returnnil,fmt.Errorf("unknown state %q",config.State)
returnnil,fmt.Errorf("unknown state %q",configIn.State)
assert.Equal(t,int32(2),attempts.Load(),"expected exactly one failed attempt and one retry")
}
Loaded 100 of 1210 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Nick Craig-Wood