The rc server decoded the whole POST body before looking up the method
and checking whether it needed authorisation, then echoed the decoded
input back in the 403 or 404 error. A client that was not allowed to
call a method could still make rclone read, decode and re-encode its
request.
The method lookup and authorisation check now happen before the body
is read, and these errors no longer echo the request input.
When the rc server was running without HTTP authentication and without
--rc-no-auth, anyone able to connect to it could POST an arbitrarily
large body to any rc path. A JSON body was decoded in full and then
echoed back in the response, so a single large request could make
rclone use several times the body size in memory and be killed by the
OOM killer.
In this mode only the small set of NoAuth methods can be called, so
POST bodies are now limited to 64 KiB and oversized requests are
rejected with 413 Request Entity Too Large without echoing the input.
Servers with authentication or --rc-no-auth are unaffected.
A server which redirects an https:// request to an http:// URL makes
rclone replay the request in cleartext. Any credentials the backend
sets as headers are then readable by anyone on the network path, and
the response can be altered in transit.
The transport now refuses to follow such a redirect, so this holds for
every client built on it, including those of libraries which install
their own redirect policy. Use --allow-https-downgrade to follow the
redirect anyway, for example when downloading from a mirror which
redirects to plain HTTP.
Backends which check redirects themselves still refuse a downgrade
which would send their credentials in cleartext, regardless of the
flag.
The 5 chunker remotes with 50 byte chunks split each test file into
many objects which made them expensive to run against AWS S3. Run
them against the TestS3Minio docker server instead and keep
TestChunkerS3 with the default chunk size on AWS.
When a negative --offset reached back past the start of a file and
--count was set, rclone cat printed bytes from the end of the file
instead of from its start. For example on a 10 byte file
rclone cat --offset -12 --count 5 file
printed the last 2 bytes rather than the first 5. With several files
of different sizes, each file shorter than -offset was affected.
The start of the range stayed negative after the file size was added,
and a negative Start with an End set is a suffix range (the last End
bytes). Clamp the start to 0 instead, which matches what
`tail -c 12 file | head -c 5` prints. --tail and --offset without
--count were not affected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
strconv.ParseFloat accepts "NaN", and NaN compares false with every
bound, so it got past the overflow check added for 300y and was then
converted to a time.Duration, which gives the minimum int64 on amd64.
--max-age NaN or --min-age NaNM silently became about -292 years,
which moves the filter's ModTimeFrom into the future and excludes every
file, instead of being rejected.
When a server-side copy or move finished without reporting progress
along the way, its bytes were added to the stats but not to the
transfer, so until the transfer ended core/stats reported totalBytes as
the bytes done plus the whole size of the transfer still in progress.
A 1 GiB server-side copy on the local backend showed 1 GiB of 2 GiB
while rclone checked the hashes.
The bytes now go on the transfer as well, as they already did for
backends which report progress through NewServerSideCopyAccounter.
validateHour only checked the length of the hh:mm field and then
parsed the two halves with strconv.Atoi. It never checked the third
character, so "10h20" or "Mon-10.20" were accepted as 10:20, and
since Atoi accepts a sign, "+1:00" and "Mon-10:+5" were accepted
as 01:00 and 10:05.
Require the colon and four digits so these are reported as invalid
time specifications like other malformed times.
SizeSuffix.Set and CountSuffix.Set accepted "nan" before a suffix, so
--max-transfer nank became -1 ("off") without an error. CountSuffix.Set
also had no overflow check, unlike SizeSuffix, so 10E or infk wrapped
to the same value.
A glob like file[^a].txt was translated to (^|/)file[^a]\.txt$ in path
mode, so it matched file/.txt across a directory boundary, unlike ?
and the [!...] form.
Treat a leading ^ in a character class the same as a leading !: in
path mode it now also excludes /, and [^] is rejected as an empty
character class. Document both negation forms in the filtering
grammar.
The filtering docs give the character class syntax as [ [ ! ]
{ character-range } ], but globToRegexp copied the ! into the regexp
unchanged, so file[!a].txt matched filea.txt and file!.txt and
excluded fileb.txt - the opposite of what was asked for.
Translate a ! at the start of a character class to ^. In path mode
the class also excludes /, in the same way ? does. [!] is rejected
as an empty character class.
The docs say Time options take the same relative forms as Duration
options, with "1d12h" as an example, but ParseTime only tried
time.ParseDuration and single-suffix values, so --max-age style
Time options such as --s3-version-at 1d12h were rejected.
The test looked up the object it had just uploaded without retrying, so
it failed with "object not found" on backends such as Cloudinary and
Internxt where new objects take a moment to appear.
Add the hash type crc32c. It is the checksum that AWS S3 stores for
objects uploaded with x-amz-checksum-crc32c (FULL_OBJECT checksum),
and the checksum that iSCSI, ext4 and Btrfs use. Go calculates it with
the SSE 4.2 and ARMv8 CRC32 instructions where they are available.
With it, rclone can calculate the value that S3 returns in
ChecksumCRC32C for a local file:
rclone hashsum crc32c file
prints it as hex; `--base64` prints it in the URL-safe base64 alphabet
that rclone uses for every hash (`-` and `_` where S3 prints `+` and
`/`).
The new type is registered after the existing core types, so their
Type values do not change. The types that backends register (dropbox,
hidrive, mailru, quickxor) move up one bit, as they did when blake3,
xxh3 and xxh128 were added; no Type value is persisted (the hasher
cache and bisync listings store hash names).
Before this change, `TestMoveUnicodeEquivalentWithoutServerSideMove`
failed on backends that can't store a file under the NFD name it was given:
`mailru` rejects the name ("invalid characters in object name"), and
`cloudinary`, `filelu` and `seafile` don't find the file again under it.
Tests in `fs/operations` run against every backend, so this one needs to
skip where the precondition doesn't hold.
After this change, the test skips when the NFD name can't be uploaded or
the file isn't found again under it.
Before this change, `operations.Move` and `operations.MoveTransfer`
deleted a file when asked to move it onto its own path without a `dst`
object. Without server-side `Move`, `operations.move` copied the file onto
itself and then deleted the source, which was the same file, and the
`memory` backend's server-side `Move` did the same. `storj` stores every
name in NFC, so there a move from a file's NFD name to its NFC name is a
move onto its own path, and `TestMoveUnicodeEquivalentWithoutServerSideMove`
lost the file on `TestStorj`.
After this change, a move whose destination is the file's own path on the
same remote does nothing, including when the destination `Fs` has a
different root.
ParseDuration checked for overflow in multi-unit durations such as
300y1d but not in single-suffix ones, so --max-age 300y silently
became a large negative duration, which moved the filter's
ModTimeFrom into the future and excluded every file.
Before this change, on backends without server-side `Move`, renaming a
file to an equivalent name deleted it: a name differing only in case on
a case-insensitive backend, or only in Unicode normalization on a
backend such as Cloudflare R2, which treats NFC and NFD names as
equivalent. `operations.move` fell back to copying the file to the new
name and then deleting the source, but both resolved to the same object,
so the delete removed the file it just copied. This affected
`--fix-case` renames in `sync` and `bisync`, `--track-renames` with
`--no-unicode-normalization`, and renames through the VFS
(`rclone mount`, `rclone serve`). `MoveFile` and backends with
server-side `Move` already handled these renames by moving through a
temporary name.
After this change, the copy-and-delete fallback also moves through a
temporary name when the two names are equivalent. The issue was discovered on
the bisync integration tests.
Sorter.Send selects on both the output and error channels of the ext
sort. If the ext sort failed it closes both, so the select could pick
the closed output channel and return without the error. This was seen
as a flaky TestSorterExtTempFileError under the race detector.
extsort now only creates temp files when the data doesn't fit in
one chunk, so the test needs to add more than a chunk of entries to
provoke the temp file error.
deleteEmptyDirectories gated on the cumulative accounting stats
(accounting.Stats(ctx).Errored()) instead of the current operation's error.
On a long-running rcd that reuses one stats group (rc "_group") across many
sync/move calls, a single earlier error left Errored() true forever, which
permanently disabled empty-source-directory deletion for every later,
error-free operation sharing the group - each run just logged "not deleting
directories as there were IO errors" and never cleaned up.
Gate on s.currentError() instead, so empty-directory cleanup depends only on
the current operation's outcome. This matches the per-operation guards already
used elsewhere in run(), and is a no-op for normal single runs where
Errored() (s.errors != 0) tracks the same errors that reach s.currentError().
Before this change setting LogLevel in the _config parameter of an rc
call had no effect as both the log functions and the log handler only
looked at the global log level.
The context aware log functions now use the log level from the config
in the context, and the log handler lets a log level in the context
which differs from the global one override its own level. This means
an rc call can be made more or less verbose than the rest of rclone
with, for example
"_config": {"LogLevel": "DEBUG"}
This applies to the logs made with the context aware log functions, so
not to those made by the backends which still use the global log level.
Note that a config copied into a context no longer follows later
changes to the global log level.
Use the context aware log functions throughout the code which runs
sync, copy, move, check and the other operations, and the directory
listings they do, so that the logs they make, such as the per file
"Copied (new)" logs, are attributed to the rc job and stats group of
the operation.
This adds ErrorfCtx, LogfCtx, InfofCtx, DebugfCtx and LogPrintfCtx
which take a context and pass it on to the log handler. The handler
can then attribute the log to what made it using the new
StatsGroupFromContext and JobIDFromContext function pointers, which
fs/accounting and fs/rc/jobs fill in.
The format string is no longer assigned to in logPrintf so that go
vet recognises these as printf wrappers and checks the arguments of
their callers.
The transport strips the headers set with --header once a redirect
chain leaves the originally requested host, but the host comparison
ignores the URL scheme. A server which redirects https://host:8443/
to http://host:8443/ keeps the same host and port, so the headers
were added to the plaintext request and could be read by anyone on
the network path.
The redirect chain walk now also treats a hop which downgrades the
original HTTPS request to HTTP as leaving the host, so the headers are
removed from that hop and every one after it.
The transport also logs at debug level when it removes the headers,
for either reason, so that a resulting 401 from the redirect target
can be diagnosed.
If a backend's ListR returned a directory after objects inside it,
the directory was listed twice when using --max-depth or filters. This
happens with azureblob on storage accounts with a hierarchical
namespace.
When a backend saved config, for example a refreshed OAuth token, for
a remote which wasn't in the config file, a section with just that
value was created in the config file for environment variable remotes
and the config file was rewritten for on the fly remotes.
Config is now only saved if the remote is in the config file.
- Only check the MIME type of the listings on backends which declare
both ReadMimeType and WriteMimeType, and ignore its parameters, as
many backends report a MIME type they chose themselves or drop the
charset.
- Write the test file before turning --dry-run on. Chunker finishes
an upload with operations.Move, which honours --dry-run, so it left
temporary chunks behind which stopped the test directory being
removed and made the tests following fail.
- Don't check the transfer and delete counts on chunker, as moving
its chunks into place is counted in the same stats.
- Don't expect a new file to change the time of the directories above
it on backends which can't set modification times, as it may get the
same time as the files already there.
The docs describe a duration as a sequence of numbers each with a unit
suffix, including d, w, M and y, but a sequence that used one of those
units, such as --max-age 1d12h, failed with a confusing error about
parsing it as a date. Only a single number with one of them, like 1.5d,
or a sequence of the time.ParseDuration units was accepted.
Parse such sequences before falling back to dates, rejecting ones too
long to represent.
Move strategy parsing, capability checks, and matcher behavior into a
reusable package without changing sync behavior. Add a lightweight
candidate API so callers can count guaranteed matches without
retaining filesystem objects.
Until the first time slot of the week, the timetable used its last entry
as the limit carried over from the week before. That is only the latest
slot of the week when the entries are in order, so a timetable written
weekend first, like "Sat-00:00,off Mon-00:00,1M", limited Sunday to 1M
instead of leaving it unlimited.
Carry over the latest time slot of the week instead.
downloads.rclone.org is about to be served from static listings
written by rclone index instead of by Caddy's file server. Released
versions of rclone selfupdate find the latest patch release of a minor
version by searching that listing for href="./vX.Y.Z/", as written by
Caddy. rclone's listings linked to "vX.Y.Z/" without the ./ so `rclone
selfupdate --version X.Y` would have failed with "could not find the
minor release" for every rclone already installed.
Caddy prefixes every link with ./ so that a name with a colon in its
first path segment is not read as an absolute URL with a scheme (RFC
3986 section 4.2). rclone was already safe from that as url.URL.String
adds the ./ but only to names which need it, so links to plain names
had no prefix.
This prefixes all the links with ./ as Caddy does. It changes the
output of serve http, serve webdav and the rc server as well as rclone
index, since they share the code. The links resolve identically, and
it keeps the listings consistent with each other.
Now every URL has the prefix, the caddy.json template no longer needs
to add it.
After a sync which changed a known set of files there is no need to
walk the whole remote. --changed PATH, --changed-from FILE and
--changed-combined FILE tell rclone index what changed, and it
re-indexes only the directories containing those paths and their
ancestors, each with one non-recursive listing.
Index writes a directory listing (e.g. index.html) into every
directory of a remote so it can be browsed when served as a static
website. It is also available over the rc as operations/index.
It works like sync. It walks the remote once, renders the listings in
memory and compares them with the existing ones by size and hash from
the listing, or by reading them where the backend has no hash.
Listings in directories which contain nothing else are deleted on
backends which can't have empty directories.
Listings can be written in the serve http HTML format, the lsjson
format, Caddy's browse JSON format, or from a user template. A second
rule set (--index-include and friends) controls which directories get
listings, --dir-time controls the time shown for directories and
--no-modtime avoids reading modification times altogether.
ENETUNREACH and ENETDOWN were not in the list of retriable errors on
non-Windows platforms, so a single failed connection aborted the
transfer instead of being retried as a low level retry. The Windows
list already includes the equivalent WSAENETUNREACH and WSAENETDOWN.
This is easy to hit on a host with IPv6 enabled but no IPv6 route: if
the A lookup fails transiently while the AAAA lookup succeeds, the only
address to dial is IPv6 and the connect fails with ENETUNREACH. A retry
resolves again and normally succeeds.
BwPair.String printed a sub-KiB bandwidth as a bare number, and Set
reads a bare number as KiB, so a rate under 1 KiB grew by a factor of
1024 whenever it went through a string. rc core/bwlimit reports the
current rate in that format and accepts it back, so reading the limit
and setting it again raised it.
63c4fef27 fixed the same corruption for config values by suffixing bare
numbers with B inside Option.String. Move that into a SizeSuffix method
and use it from BwPair.String as well.
When a backend rejected the multipart upload as too small, the subtest
skipped without removing its local source file, so the following
upload subtests failed their local listing checks.
TestDeleteFatalError set --max-delete before writing its files, so on
chunker, which deletes while uploading, the setup failed.
TestDirMoveMoveError and TestDirMoveContext test the core DirMove logic
with a wrapping Fs. This fails on remotes without Move (eg s3, memory)
and on those whose objects don't belong to the wrapped Fs (eg archive),
so they now only run on local.
When the source matched a file in --copy-dest, copyDest server-side
copied it over an existing destination which differed, so sync, copy
and copyto with --copy-dest could modify a file --immutable should have
protected.
Leave such a destination for the caller, which rejects it with
ErrorImmutableModified as for any other modified file.
--immutable was only enforced by sync, so copyto, moveto, copy and move
of a single file, and the operations/copyfile and operations/movefile rc
calls overwrote a destination which differed from the source.
Fail these with the same ErrorImmutableModified as sync, and reject
--no-check-dest with --immutable as sync does, since without checking
the destination it cannot be protected.
job/status returns the complete output of the job so it is as
sensitive as the call which started the job, e.g. config/dump started
with _async returns the config file including any secrets. It was
possible to read this without authentication where an authenticated
and an unauthenticated rc server share the same process, e.g. rclone
gui --rc.
An unauthenticated client can't start jobs on a server which needs
authentication, other than with the calls which don't need it, so
this shouldn't affect existing users.
Nick Craig-Wood