Compare commits

...
1171 Commits
Author SHA1 Message Date
Nick Craig-Wood 9ee9d0a0ca Version v1.75.0 2026-07-31 16:56:33 +01:00
Nick Craig-Wood 3f8df416c8 build: stop make compile_all overloading the machine
Previously cross-compile.go ran NumCPU builds in parallel, each of
which ran an unrestricted go build using -p NumCPU internally, giving
up to NumCPU^2 concurrent compile processes and enormous load averages.

Pass -p to each go build, sized so the total parallelism is about
NumCPU, sharing the CPUs between however many builds are actually
selected. This can be overridden with the new -build-p flag.
2026-07-31 15:31:24 +01:00
Nick Craig-Wood 6a69713864 local: stop source file names escaping the destination directory GHSA-7p4m-qxvv-g567 CVE-PENDING
The local backend built every OS path by joining the root with the source
name converted through the configured encoding, so the encoding was the only
thing keeping a name from turning into path syntax.

With an encoding which omits Dot (Slash, None, Raw) rclone's standard ".."
decodes back to a real "..", and with an encoding which omits BackSlash a name
like "..\file" becomes a native path on Windows. filepath.Join then resolved
those out of the destination the user chose, so a source object called
"../marker.txt" - an s3 key of "tenant/../marker.txt" listed with the remote
rooted at "tenant", say - created or overwrote a file outside it.

localPath now joins the name to the root and checks with filepath.Rel that the
result is still inside it. localPath is the only place the root is joined to a
name, so threading the error through newObject and newDirectory covers every
operation.

Default configurations were not affected, as encoder.OS includes Dot on all
platforms and BackSlash on Windows.

Fixes GHSA-7p4m-qxvv-g567
2026-07-31 13:21:59 +01:00
Nick Craig-Wood cc5a189f00 serve restic: fix path traversal above the served directory GHSA-45pq-889g-fcgh CVE-PENDING
A request path beginning with "../" escaped the path the server was
started on, letting a client list, read, create, overwrite and delete
objects outside it.

The check added for CVE-2026-59733 rejected non-canonical paths by
comparing them with path.Clean, but path.Clean cannot resolve leading
".." elements in a relative path so it leaves them in place and the
comparison comes out equal. Only interior traversal such as "a/../../x"
was rejected. Whether a path then escaped depended on the backend:
those which join the root with the remote before encoding it - webdav,
ftp, sftp, http and memory - resolved the ".." away, while local and s3
encode the dot elements first and were unaffected.

A bare "." was accepted for the same reason, which on bucket backends
addresses the served directory's own key.

Validate with io/fs.ValidPath instead, which rejects ".", ".." and empty
elements wherever they appear. The empty path stays valid as the root of
the API, and "." is excluded explicitly because ValidPath accepts it as
the root of an FS.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 208d7df877 vfs: don't crash the process if a backend panics on a background goroutine GHSA-6jcg-q3wp-x2f4
The VFS calls backends from goroutines of its own. A panic on any of these
cannot be recovered. So a backend panicking on a single file killed the whole
process, taking down a mount or every user of a serve command, even for servers
such as serve http whose library recovers panics raised on its own request
goroutines.

Recover panics at those goroutines and log them with a stack trace. Where the
surrounding code already handles a failure, recover around the backend call
itself rather than the whole goroutine, so a panicking upload is retried like
any other failed upload and a panicking download is reported to the waiters,
instead of abandoning the work part way through and leaving the bookkeeping
inconsistent.

Addresses GHSA-6jcg-q3wp-x2f4
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 50b4d4c745 serve sftp: don't crash the whole server on a bad request GHSA-6jcg-q3wp-x2f4
Three ways a single client could deny service to everyone else connected to
the same serve sftp process:

A session "subsystem" request payload is a length-prefixed string, but it was
sliced at a fixed offset without checking its length, so a client sending a
truncated payload panicked the out-of-band request goroutine and killed the
process. Decode the payload instead, the way the neighbouring "exec" request
already does.

Rejecting a request then left the goroutine handling that channel waiting
forever to learn what kind of channel it was, because nothing was ever sent
on the channel it waits on. A client could open channels in a loop making
unsupported requests and grow the server's goroutines and memory without
bound. Signal the waiter when the requests run out so the channel is torn
down.

Separately, the request handlers - and reads, writes and closes on the file
handles they return - run on pkg/sftp packet worker goroutines which have no
panic recovery. A panic raised by a backend while serving one request took the
process down with it. Recover panics at that boundary, log them with a stack
trace, and return them to the requesting client as an error instead.

Addresses GHSA-6jcg-q3wp-x2f4.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 31f3856281 archive/squashfs: don't crash on malformed squashfs images GHSA-6jcg-q3wp-x2f4 CVE-PENDING
The archive backend passes remote .sqfs bytes straight to the go-diskfs squashfs
parser, which does not validate several attacker-controlled superblock and
metadata fields. A crafted image can make the parser panic.

Recover panics at the go-diskfs boundary and return an "invalid or corrupt
squashfs image" error instead. As well as the parse entry points (Read, ReadDir,
OpenFile) this wraps the reader returned by Open, since the parser reads file
data lazily and can panic long after the image opened successfully.

Addresses GHSA-6jcg-q3wp-x2f4.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood b7a1184019 serve ftp: use constant time comparison for password check GHSA-mfvx-7rcj-9m5g
The builtin authentication compared the configured username and password
with ==, whose run time depends on how much of the value matches, giving a
timing side-channel that could in principle help guess the password.

serve sftp, serve s3 and the auth proxy already use subtle.ConstantTimeCompare
so bring serve ftp in line with them. An empty configured password
still accepts any password.

Addresses GHSA-mfvx-7rcj-9m5g finding 4.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 043e58b83c lib/http: use TLS on all --addr listeners when --cert and --key are set GHSA-mfvx-7rcj-9m5g
When --cert and --key were supplied TLS was only applied to the listener if
exactly one --addr was given. With two or more --addr flags every listener
without an explicit tls:// prefix silently served cleartext HTTP, so adding a
second --addr to an HTTPS server quietly disabled TLS on both.

Now when TLS is configured every listener serves TLS. An individual listener
can be prefixed with http:// to serve unencrypted HTTP on that address, and
tls:// still marks a listener as TLS explicitly. Using a tls:// address
without --cert and --key is now an error instead of silently serving
cleartext with an https:// URL.

Addresses GHSA-mfvx-7rcj-9m5g finding 3.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 4bb6a1edf6 rc: require authentication to list the remotes with --rc-serve GHSA-mfvx-7rcj-9m5g
With --rc-serve set the root listing enumerated the names of all configured
remotes without any authentication.

Make the root listing obey the same fail-closed rule as the rest of
the rc endpoints: it now requires authentication to be configured or
an explicit opt out with --rc-no-auth.

Addresses GHSA-mfvx-7rcj-9m5g finding 2.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood faaf716e9b rc: don't expose pprof debug handlers on an unauthenticated server GHSA-mfvx-7rcj-9m5g CVE-PENDING
The pprof debug handlers were accessible without authentication disclosing the
process command line (which can carry backend credentials passed on the command
line) and runtime profiles.

Mount the pprof handlers only when when auth is configured or --rc-no-auth was
passed - so they obey the same rule as the rc endpoints.

Addresses GHSA-mfvx-7rcj-9m5g finding 1.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 21d8cd3b92 lib/proxy: fix unbounded HTTP CONNECT headers causing OOM GHSA-xhf4-832v-7xcr CVE-PENDING
Before this change rclone read a proxy response with http.ReadResponse
over an unrestricted buffered reader. A malicious or compromised
configured proxy, or an active on-path actor controlling a plaintext
HTTP-proxy hop, can grow memory until the process fails.

This fixes the problem by restrincting the read to 1MB maximum.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood ff43a1e3ae rc: fix leaking stack traces on panics GHSA-gwfq-86j8-7qhv
Before this change, rclone sent stack traces to the client on panic
capture in the rc. Stack traces can leak information which could be
useful to an attacker.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 1df2b70753 ftp: fix ftp command injection when encoding doesn't include CRLF GHSA-8c48-q9wj-3w37 CVE-PENDING
The FTP control channel is line oriented and the ftp library writes
command arguments (paths) straight onto it without escaping, so a
filename containing CR/LF can inject an independent FTP command.

This fix makes sure CR/LF are therefore always encoded to safe symbols
regardless of the configured encoding, which is what the default
encoding already does.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 5871d98c36 webdav: tus: fix potential nil pointer crash GHSA-3x6r-wxxg-53vv 2026-07-31 13:21:59 +01:00
Nick Craig-Wood 52cf74dc59 sftp: fix cmd shell execution of paths containing variable-expansion or newline characters
An audit of the shell escaping alongside the PowerShell fix found the
Windows Command Prompt path only rejected the double quote delimiter. A
remote filename containing % or ! can trigger environment variable expansion
even inside double quotes, and a newline or carriage return ends the
command, so a crafted filename could alter the command run during
server-side hashing. Reject paths containing these characters, matching
the existing treatment of the double quote.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood e122fba1a5 sftp: fix command injection via crafted filenames on PowerShell remotes GHSA-2m8m-jhrm-w6j2 CVE-PENDING
PowerShell treats several Unicode smart-quote characters (U+2018, U+2019,
U+201A, U+201B) as single-quote delimiters in addition to the ASCII
apostrophe. The quoting helper only doubled the ASCII apostrophe, so a
remote filename containing one of these could close the quoted path and
inject statements that ran as the SSH account during server-side hashing.

Double all five delimiters when wrapping a PowerShell path so a filename
is always treated as data.

Fixes GHSA-2m8m-jhrm-w6j2
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 7543a7a878 s3: strip S3 Express session token on cross-host redirects GHSA-8mxv-9xhp-86h4
The AWS SDK signs S3 Express (directory bucket) requests with a session
token in the x-amz-s3session-token header. Go's HTTP client treats it as
an ordinary custom header and copies it when following a redirect to
another host, and it was missing from the list of secret headers the
redirect policy strips. Add it to the list.

The redirect tests derived their inputs from the production header list,
so a header accidentally dropped from that list would silently lose test
coverage rather than fail. The test list is now a deliberately literal
copy, kept in sync with the production list by a new test, so removing a
header from either list is a test failure. There is also a new
regression test verifying the Referer header that net/http generates
automatically - which for a presigned request carries the signed query
string - is not forwarded across hosts.

See GHSA-8mxv-9xhp-86h4
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 9328763d1b s3: fix redirect credential leaks, reject HTTPS->HTTP and strip secrets GHSA-8mxv-9xhp-86h4
The CheckRedirect policy stripped only the X-Amz-Security-Token header when a
redirect crossed a host, leaving other origin-bound secrets exposed:

- Go removes Authorization on a hostname change but not on a same-host scheme
  downgrade, so an IBM IAM bearer token was re-sent over plaintext HTTP.
- The SSE-C and copy-source SSE-C customer-key headers (which Go has no reason
  to treat as sensitive) were forwarded across a redirect to another host.
- On a cross-host redirect net/http copies the previous request URL into the
  Referer header; for a presigned request that URL carries the signature and
  session token in its query string, disclosing them to the new origin.

This now refuses outright to follow any HTTPS->HTTP redirect - an S3 endpoint
has no legitimate reason to downgrade the transport, and this closes the
plaintext-exposure class regardless of which header or query string carries the
secret. On a cross-host redirect also strip the known credential headers and the
Referer.

The IDrive e2 region-lookup call posts the access key ID to a fixed HTTPS
endpoint but used a bare http.Client that would follow a redirect downgrading
to plaintext HTTP. Apply the same CheckRedirect policy as the main S3 client so
the request can't be replayed over an unencrypted connection.

Fixes GHSA-8mxv-9xhp-86h4
2026-07-31 13:21:59 +01:00
Nick Craig-Wood 59b513b0e7 webdav: fix HTTPS to HTTP redirects leaking credentials GHSA-h4mf-4v27-hggj
A server that redirects an HTTPS request to a plaintext HTTP URL on the
same host would cause Go's http.Client to replay the configured
credentials (Basic Authorization, cookies, secret headers) over the
network in cleartext.

Refuse to follow such downgrade redirects by default in lib/rest and wire
the webdav backend's client to use it. The `auth_redirect` option remains
the opt-in escape hatch for servers that legitimately need auth preserved
across redirects.

Fixes GHSA-h4mf-4v27-hggj
2026-07-31 13:21:59 +01:00
Nick Craig-Wood ed983c952d bin: remove commits already released on the stable branch from the changelog
When making a release from master, the range since the last minor
release includes all the commits which were cherry-picked to the
previous stable branch and already published in its point releases,
so their changelog lines had to be deleted by hand.

make_changelog.py now finds the previous stable branch (eg
v1.74-stable) from the version being released and skips any commits
released there, detected via cherry-pick -x trailers and git cherry
patch equivalence. Skipped commits are listed on stderr for review.

Releases made from a stable branch are unaffected.
2026-07-31 13:21:59 +01:00
Nick Craig-Wood c3f556b9c0 gui: update embedded release to 1.1.11 2026-07-31 12:01:05 +01:00
Nick Craig-Wood a4d288f8d2 docs: update sponsors 2026-07-31 11:59:18 +01:00
Nick Craig-Wood 7eef70c8b8 docs: fix hugo build after adding .go files 2026-07-31 11:44:15 +01:00
Nick Craig-Wood 92fbc85f10 yandex: add --yandex-upload-wait to fix 500 errors when uploading
In this commit we attempted to wait for the success report of an
upload to fix the 500 error:

fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime

However Yandex Disk finalizes an upload asynchronously on its servers.
Waiting for the upload operation to report success is not enough -
under load the server reports the operation as successful slightly
before the file is fully finalized, so setting the modification time
straight after an upload can still fail with 500 Internal Server
Error.

Yandex support recommend waiting 1.5s - 3s after the upload before
modifying the file's metadata, so add an --yandex-upload-wait option
(default off) to insert a delay between the upload completing and the
modification time being set.
2026-07-30 20:01:18 +01:00
SocialprankerandClaude Opus 4.8 afa25e9cff docs: add OpenBSD mounting section for nfsmount
Cover the portmap registration OpenBSD's kernel NFS client needs (recipe
from hjicks in #8578), the mount_nfs -T requirement, and why the server
advertises AUTH_UNIX, so an OpenBSD user doesn't need to read through the
issue thread to get a working mount.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-30 19:54:16 +01:00
SocialprankerandClaude Opus 4.8 36ea20b0cc vfs: build the real uid/gid lookup on OpenBSD too
Mounted files showed up owned by 4294967295 (^uint32(0)) on OpenBSD.
vfsflags_unix.go, which calls unix.Geteuid()/unix.Getegid() to get the
real uid/gid, only builds for linux, darwin and freebsd; OpenBSD fell
through to vfsflags_non_unix.go's zero-value stub instead.

OpenBSD has no linux/darwin/freebsd-specific fields here, just the same
POSIX Geteuid/Getegid/Umask calls golang.org/x/sys/unix already ships
for openbsd on every arch, so this just adds openbsd to both build tags
rather than needing a separate file.

Cross-compiled for GOOS=openbsd (arm64, amd64) and go vet clean; ran the
existing vfs test suite on darwin, no regressions. Not yet verified on a
live OpenBSD mount, only that the right uid/gid syscalls now get called
in this codepath.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-30 19:54:16 +01:00
SocialprankerandClaude Opus 4.8 9d41369d77 serve nfs: advertise AUTH_UNIX so the *BSD NFS clients can mount
The mount served fine on Linux and macOS but *BSD kernel NFS clients
refused it with "Authentication error". The MOUNT reply only ever
offered AUTH_NULL, and the OpenBSD/FreeBSD/NetBSD clients won't mount a
server unless AUTH_UNIX is among the offered flavors.

Add AUTH_UNIX to the advertised list. The server still doesn't inspect
the credential (there's no per-user access control here) so the AUTH_UNIX
cred the client then sends is read as an opaque blob and ignored, exactly
as the AUTH_NULL one was. No behaviour change for existing Linux/macOS
clients, and OpenBSD now mounts and reads files over the share.

Verified on a real OpenBSD 7.9 arm64 VM: registered rclone's server in
portmap and ran "mount_nfs -T localhost:/ /mnt"; the mount now succeeds
(MOUNT and GetAttr RPCs go through) and files read back correctly through
the mount, where before it stopped at the auth stage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-30 19:54:16 +01:00
SocialprankerandClaude Opus 4.8 b78a43d05f nfsmount: call mount_nfs directly on OpenBSD so -T is accepted
The previous OpenBSD path built the right options ("-o port=N -T") but
still handed them to mount(8). On OpenBSD "-T" is a mount_nfs(8) flag,
not a mount(8) one, so mount rejected it with "mount: unknown option --
T" and the mount never ran. Call mount_nfs(8) directly on OpenBSD; the
options are already in its native syntax.

Verified on a real OpenBSD 7.9 arm64 VM: with this change the command
becomes "mount_nfs -o port=N -T localhost:/ <mnt>" and mount_nfs accepts
the flags. The mount then fails later with "Stale NFS file handle"
because the OpenBSD client and rclone's in-process NFS server disagree
on the root filehandle - that is a separate issue in the NFS server,
not in the mount options this PR is about.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-30 19:54:16 +01:00
SocialprankerandClaude Opus 4.8 47fb3c7156 nfsmount: fix mount_nfs options incompatible with OpenBSD - fixes #8578
nfsmount built its mount options for GNU/Linux syntax unconditionally:
"-o port=N", "-o mountport=N" and "-o tcp". OpenBSD's mount_nfs(8)
rejects "-o mountport" outright ("option not supported", per the
reporter's log) and has no "tcp" suboption either, since it selects
TCP with the separate "-T" flag instead of an -o suboption.

Add a runtime.GOOS == "openbsd" branch that builds the option list
OpenBSD's mount_nfs actually accepts: "-o port=N" plus "-T" for TCP,
with no mountport option since OpenBSD's mountd is located via
portmap rather than a fixed, settable port. This follows the same
GOOS-branching pattern already used in this file's unmount function
(darwin) and in cmd/cmount/mount.go for openbsd/freebsd differences.

FreeBSD's mount_nfs(8) documents "port=", "mountport=" and "tcp" as
-o suboptions identical to Linux, so the existing option set is left
unchanged for freebsd and all other platforms.

Verified by cross-compiling (go build and go vet) for GOOS=openbsd,
freebsd, linux and darwin, all of which succeed. Actually mounting
via mount_nfs on OpenBSD needs a BSD machine to confirm at runtime,
which wasn't available here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-30 19:54:16 +01:00
Nick Craig-Wood be6c689a68 build: fix FUSE mount test failures on GitHub actions runners
Runner image ubuntu24/20260726.254 updated podman from 4.9.3 to 5.8.4
which is installed from the podman-static bundle. This bundle ships a
non-setuid fusermount3 in /usr/local/bin which shadows the setuid
/usr/bin/fusermount3 from the fuse3 package (as /usr/local/bin comes
first in PATH). A non-setuid fusermount3 cannot mount as an
unprivileged user, so the cmd/mount and cmd/mount2 tests failed with:

    fusermount3: mount failed: Operation not permitted

Fix by removing the podman bundled fusermount binaries so the distro
provided setuid one is used.
2026-07-30 19:35:40 +01:00
Nick Craig-Wood 1032813f91 gitannex: skip e2e tests on macOS CI to avoid timeout
The TestEndToEnd tests consistently time out after 10 minutes on
macOS CI runners. Skip them on macOS when running in CI.
2026-07-30 19:33:42 +01:00
Nick Craig-Wood e6a2347bb3 dropbox: remove an unnecessary API call when uploading small files - fixes #9686
The session close flag was only computed after each append, so a
known-size upload which fits in a single chunk sent all its data and
then issued a zero-payload append purely to close the session - one
wasted round trip per small file on the default batched upload path.

Set the close flag before the first append when the size is known to
fit in one chunk.
2026-07-30 19:30:38 +01:00
Nick Craig-Wood 71a173472b dropbox: use much less memory when uploading small files - fixes #9685
Since batch mode became the default all uploads go through
uploadChunked, which allocated a full chunk-size retry buffer (48 MiB
by default) regardless of the file size. With the `--transfers 32`
recommended for small file uploads that is ~1.5 GiB of buffer to
upload tiny files.

Size the buffer to the file size when it is known and smaller than a
chunk.
2026-07-30 19:30:38 +01:00
Nick Craig-Wood b3a41206da oracleobjectstorage: add --oos-decompress flag to download gzip-encoded files
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download, giving
errors like

    corrupted on transfer: sizes differ

This change sets "Accept-Encoding: gzip" on all requests which stops
the go runtime decompressing objects on download, so compressed
objects are downloaded as-is with intact size and hash information.

If --oos-decompress is set then rclone will decompress objects with
"Content-Encoding: gzip" as they are received, at the cost of not
being able to check the length or the hash of the downloaded object.

Fixes #9694
2026-07-30 19:30:08 +01:00
Nick Craig-Wood 8772c94011 smb: fix spurious "Directory already exists" errors when moving directories - fixes #9680
DirMove returned fs.ErrorDirExists for any error from the destination
existence check not just when the destination actually exists. That
made sync silently fall back to file-by-file moves and masked the real
failure.

Return the underlying error instead when the check fails for any other
reason.
2026-07-30 17:50:45 +01:00
Nick Craig-Wood 088f68f3c3 smb: fix server-side move of directories with special characters in the name - fixes #9677
DirMove checked whether the destination exists using the raw path but
performed the rename with the encoded path, so for directory names
needing encoding (trailing spaces or periods, characters like
\* ? : < > | " or a literal backslash) the existence check looked at
a different server path than the rename used.
2026-07-30 17:50:08 +01:00
Nick Craig-Wood 74f9f182aa smb: fix TCP connection leak when connection setup fails - fixes #9678
If revealing the password, creating the Kerberos client or the SMB
handshake failed after the TCP connection was established, the
connection was never closed.
2026-07-30 17:48:54 +01:00
Nick Craig-Wood 76d1adb7a6 smb: fix Kerberos credentials being reloaded for every connection - fixes #9674
A new KerberosFactory was constructed for every dial, so the client,
error and ccache modification time caches it holds were discarded
after a single use. Every new SMB connection re-read the Kerberos
config, re-parsed the ccache and did a fresh KDC exchange.

Share a single factory so clients are cached across connections as
intended, and refreshed when the ccache file changes.
2026-07-30 17:47:21 +01:00
Nick Craig-Wood 862ed2b7ac oracleobjectstorage: fix crash when downloading objects with unknown length - fixes #9694
Object.Open dereferenced the response's ContentLength pointer without checking
it. The OCI SDK leaves ContentLength nil when the server replies without a
Content-Length header or ContentRange which caused a nil pointer panic.

Now the size is only updated when the response actually provides one, leaving
the size from the object metadata in place otherwise.

This also fixes the same potential problem in the newObject code.
2026-07-30 15:22:03 +01:00
Leon Brocard c1ff08a627 serve/http: add --disable-dir-list flag
Previously, GET requests for a directory URL always returned an HTML
directory listing. There was no way to suppress this, unlike
`serve webdav` which has supported --disable-dir-list since #4191.

This adds the same flag to `serve http`. When set, GET requests for
directory URLs return 404 instead of a listing, while file downloads
continue to work normally.

Based on the approach suggested in #6306.

Fixes #4000
2026-07-30 14:58:04 +01:00
Nick Craig-Wood aba5c11eab shade: fix uploads failing with EOF when completing multipart uploads
The multipart upload complete endpoint returns 200 with an empty body,
but rclone tried to decode that body as JSON, failed with EOF and
retried until the retries ran out, so every upload failed even though
the server had actually completed it.

Fixed by not attempting to decode the response body.

This was a regression introduced in

a4972de505 shade: retry server errors instead of failing the transfer

which started treating the JSON decode error as fatal where previously
it was accidentally ignored.
2026-07-30 14:41:09 +01:00
Nick Craig-Wood 4638d4a83c Add Punya Jain to contributors 2026-07-30 14:41:09 +01:00
Hakan İSMAİL f47ea6eb4a rc: fix _filter and _config parameters being ignored by mount/* commands
This resolves an issue where mount filters supplied to the rc API
(such as `_filter` in remote control requests) were ignored during
FUSE mounts.

By passing the request context containing the parsed filter config to
`vfs.New`, the VFS layer now correctly respects the active filter
rules.

Fixes #8838
2026-07-30 11:02:25 +01:00
Yash Anil bd4c6571ec march: fix goroutine leak on completed async rc jobs - fixes #9620
The march janitor goroutine, which discards queued jobs when the context is
cancelled, only ever returned on context cancellation. A march that finished
normally never cancels its context, so on an async rc job (whose context
descends from context.Background and is only cancelled by job/stop) the
janitor parked forever, leaking one goroutine per run and pinning that run's
directory listings in memory. A long-running rcd driving async sync or bisync
jobs accumulated these until it ran out of memory.

Signal the janitor to exit once the march completes so it returns on both
normal completion and cancellation.
2026-07-29 20:29:13 +01:00
Punya JainandClaude Opus 5 731f2a6c29 iclouddrive: fix 2FA failing with 409 even when the code is valid
Since around mid-2026 Apple's idmsa endpoints `POST
/verify/trusteddevice/securitycode` and `POST /verify/phone/securitycode`
return HTTP 409 (instead of 2xx) even when the submitted code is accepted:
the response body carries `"securityCode": {..., "valid": true}` and the
response headers include a fresh X-Apple-Session-Token, scnt and
X-Apple-Auth-Attributes, which are only issued on successful validation.

rclone treated any 409 as failure and aborted before TrustSession, so
configuring an iclouddrive remote always failed after the 2FA step with:

    validate2FACode failed: HTTP error 409 (409 ) returned body:
    "{... \"securityCode\": {\"code\": \"...\", \"valid\": true} ...}"

Treat a 409 response that carries X-Apple-Session-Token as success: absorb
the session headers and continue to TrustSession. Applies to both the
trusted-device and SMS validation paths.

Fixes #9488
Closes #9534

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-07-29 20:25:57 +01:00
Nick Craig-Wood d183f4a43d azureblob: list very large containers in parallel with list_parallelism
Add a new "list_parallelism" advanced option (requires use_arrow_list) which
splits a directory's blob-name keyspace into ranges and lists them
concurrently using the Arrow startFrom/endBefore range parameters. Both
recursive (ListR) and single-directory listings benefit.

The keyspace is sharded with a single-case (digit+lowercase) character
ladder. Single case is deliberate: the service validates startFrom /
endBefore case-insensitively (while listing case-sensitively), so a
mixed-case boundary pair is rejected with a 400; its endBefore filtering is
byte-ordered, so the listing is exact.

The service only honours endBefore on the Arrow listing path of accounts with
Blob Listing with Apache Arrow enabled: elsewhere it either rejects the
request with 400 OperationNotSupportedWithFeatureMissing or, if it fell back
to XML, would ignore the shards' upper bounds. Both cases are detected and
the parallel listing falls back to a sequential listing.

Listing 100,000 objects from a well-connected server:

    config                 time    vs XML
    xml                   23.9s     1.0x
    use_arrow_list         6.9s     3.5x
    list_parallelism=2     4.4s     5.5x
    list_parallelism=4     2.8s     8.4x
    list_parallelism=6     2.2s    10.9x
    list_parallelism=10    1.5s    15.4x
    list_parallelism=20    1.4s    17.6x
    list_parallelism=30    1.1s    21.7x

So Apache Arrow listing alone is ~3.5x faster than XML, and parallel listing
takes that to ~20x. The ladder caps at 37 shards, so the benefit plateaus
beyond list_parallelism=30.

The option is hidden from the configurator, command line help and docs
until the feature is officially released.
2026-07-29 20:16:22 +01:00
Nick Craig-Wood 19478503af azureblob: add use_arrow_list flag for experimental Apache Arrow listing
Add a new "use_arrow_list" advanced option which fetches directory listings
using the ListBlobs Apache Arrow response format instead of XML. This can be
substantially faster for very large containers.

This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow" preview
feature to be enabled on the storage account. It is not supported on accounts
with a hierarchical namespace (ADLS Gen2). If the feature is not enabled the
service returns XML and the listing transparently falls back to it. Connection
string auth is not supported - it also falls back to the normal XML listing.

The option is hidden from the configurator, command line help and docs until the
feature is officially released.
2026-07-29 20:16:22 +01:00
Nick Craig-Wood 267f1560d2 azureblob: add arrowlist package for Apache Arrow blob listing
This adds a temporary subpackage implementing the experimental "Blob
Listing with Apache Arrow" feature on top of the released Azure azblob
SDK.

The Azure SDK for Go supports Arrow listing on its unreleased
feature/storage/bifrost branch (commit c6fa341ca22b) but that can't be
used in a release. This package exposes the same options and pager
interface as the experimental SDK, built on the released SDK's public
API plus verbatim copies of its Arrow decoder and of the two internal
auth policies (shared key signing and storage bearer challenge) which
the released SDK does not export. Copied files carry MIT attribution
headers pinned to their upstream source.

When Arrow listing ships in a released azblob the whole package will be
deleted and callers pointed back at the SDK - the interface is
deliberately source compatible to make that a mechanical change.

Also exports auth.Transporter so the new package's pipeline can use
rclone's fshttp transport, and adds github.com/apache/arrow-go/v18 as a
direct dependency for decoding the Arrow IPC stream.
2026-07-29 20:16:22 +01:00
Nick Craig-Wood 3bb1cadc23 docs: don't show fully hidden backend options in the backend docs
Backend options with Hide: fs.OptionHideBoth are hidden from the config
wizard and from the command line flag listing, but "rclone help backend"
(and therefore the autogenerated backend documentation) still showed
them. Skip them there too so fully hidden options no longer appear in
the docs.
2026-07-29 20:16:22 +01:00
Nick Craig-Wood e25344fb11 Add phatlc to contributors 2026-07-29 20:16:17 +01:00
Nick Craig-Wood 4e66c96507 Add Socialpranker to contributors 2026-07-29 20:16:17 +01:00
Nick Craig-Wood 04a56a5a39 Add Anupam Mediratta to contributors 2026-07-29 20:16:14 +01:00
Nick Craig-Wood 2150dfa56e Add Dzmitry Nianakhau to contributors 2026-07-29 20:16:14 +01:00
Nick Craig-Wood c37ab1dc7f Add ifloppy to contributors 2026-07-29 20:16:14 +01:00
Nick Craig-Wood 70222fa408 Add Acts1631 to contributors 2026-07-29 20:16:14 +01:00
Nick Craig-Wood 63b3a934ec Add Noah Zalev to contributors 2026-07-29 20:16:14 +01:00
Nick Craig-Wood 2ab104ba06 Add Zero Services GmbH to contributors 2026-07-29 20:16:14 +01:00
Hakan İSMAİL e8ff1b123b mount: refactor mount remote control to use ParseOptions
This updates `mountRc` in `cmd/mountlib/rc.go` to parse options using the
unified `rc.ParseOptions` helper. It also enforces parameter validation by
calling `rc.CheckParamsUsed`.

- Replace custom options parsing with rc.ParseOptions for vfsOpt and mountOpt.
- Delete consumed params (mountPoint, mountType, fs) and call CheckParamsUsed
  before initiating FUSE mount to reject unknown parameters.
- Clean up duplication tests (TestRcFlatOptions, TestRcFlatOptionsNull) in rc_test.go.
- Update TestRc in rc_test.go to pass a clean params map to unmount.Fn.
2026-07-29 19:42:45 +01:00
Hakan İSMAİL 01495c8ded serve: update serve remote control to use ParseOptions
This refactors all 8 serve protocols to use `rc.ParseOptions` for VFS
and protocol options decoding. It also implements parameter validation in the
main runner using `rc.CheckParamsUsed`.

- Update dlna, ftp, http, nfs, restic, s3, sftp, and webdav to call
  rc.ParseOptions, enabling nested option block support.
- Remove unused configstruct imports.
- Update startRc in cmd/serve/rc.go to copy input parameters and
  call rc.CheckParamsUsed to reject unknown parameters.
- Add TestRcStartFlatNestedAndUnknownRejection to cmd/serve/rc_test.go.
- Update inline documentation in serve/start command help to include
  nested blocks information (vfsOpt, proxyOpt, opt) and a new WebDAV example.
2026-07-29 19:42:45 +01:00
Hakan İSMAİL a50d1137a3 fs/rc: add ParseOptions and CheckParamsUsed unified options helpers 2026-07-29 19:42:45 +01:00
Hakan İSMAİL 71aef129cf mountlib: support flat VFS and Mount options in mount RC command 2026-07-29 19:42:45 +01:00
phatlc ab93058560 fserrors: make http2 "server sent GOAWAY" a retriable error - fixes #9664
When an HTTP/2 server retires a connection with GOAWAY after it has
already sent successful response headers, Go's http2 transport fails the
read of the response body with

    http2: server sent GOAWAY and closed the connection; LastStreamID=..., ErrCode=NO_ERROR, debug=""

This was not recognised as a retriable networking error, so a transient
connection retirement aborted the whole command instead of consuming a
low level retry. It was reported against a large S3 check, where an
interrupted ListObjectsV2 page made rclone report destination objects as
missing and exit unsuccessfully.

The concrete error type is unexported by net/http, so match on the
message as we already do for the other http2 transport errors.
2026-07-29 17:35:11 +01:00
dependabot[bot] 0f49ceab75 build(deps): bump actions/setup-go from 6 to 7
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6 to 7.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-29 17:24:15 +01:00
dependabot[bot] d74b458386 build(deps): bump actions/cache from 5 to 6
Bumps [actions/cache](https://github.com/actions/cache) from 5 to 6.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-29 17:23:50 +01:00
dependabot[bot] dcedb721f0 build(deps): bump actions/checkout from 6 to 7
Bumps [actions/checkout](https://github.com/actions/checkout) from 6 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-29 17:23:22 +01:00
Socialpranker 3444500e74 dropbox: make Rmdir use one less API call - fixes #9663
Rmdir checked that the directory existed with GetMetadata and then
checked it was empty with ListFolder, but ListFolder already reports a
missing path and a path that is a file, so the first request was
redundant.

Drop the GetMetadata call and map the ListFolder lookup errors onto the
same sentinel errors as before, so a missing directory still returns
fs.ErrorDirNotFound and a file still returns fs.ErrorIsFile. Limit is
set to 1 as only the presence of an entry matters, and HasMore is
checked as well so a full first page is not read as an empty directory.
2026-07-29 17:22:30 +01:00
Anupam Mediratta 17629d67b2 build: add 7 day cool down period for dependabot security fixes
Automated security fix generated by OrbisAI Security
2026-07-28 18:03:58 +01:00
Dzmitry Nianakhau 0257ae9b50 s3: add Scality (RING / ARTESCA) provider
Add Scality as an S3 provider covering both Scality RING (S3 Connector)
and ARTESCA, which share the same CloudServer + Vault S3 implementation.

The only quirk required is force_path_style: both products support
path-style addressing, and virtual-hosted style needs wildcard DNS that
on-prem deployments usually lack.
2026-07-28 17:35:11 +01:00
ifloppy 512ed643f7 onedrive: download malware-flagged files via Graph Prefer header
When --onedrive-av-override is set and the object is malware-flagged,
download via Microsoft Graph beta contentStream then /content with
Prefer: forceInfectedDownload, and keep Prefer (and AVOverride) on the
SharePoint redirect without re-encoding tempauth.

Clean files keep using the stable v1.0 /content path so permanently
enabled av_override does not put all traffic on beta APIs.

AI-assisted change; tested against OneDrive for Business with
application permissions (client_credentials).
2026-07-28 17:33:55 +01:00
ifloppy 9a0d7e57dd onedrive: treat non-2xx preauth download as error
The unauthenticated follow-up to /content redirects used Client.Do,
which does not check HTTP status. A 403 malwareDetected JSON body was
therefore written as file content and surfaced only as a size mismatch.

Check the status code, surface malware errors clearly, and avoid
suggesting --onedrive-av-override when it is already set.
2026-07-28 17:33:55 +01:00
Acts1631 439e518bda serve dlna: bound SOAP request bodies
The unauthenticated DLNA control endpoint decoded arbitrary SOAP bodies
into an in-memory XML field. A LAN client could send a large request
and exhaust the server's memory.

Limit SOAP request bodies to 1 MiB and return 413 when the limit is
exceeded.
2026-07-28 17:32:00 +01:00
Noah Zalev 19f8b69518 sftp: allow silencing no hostkey validation warning 2026-07-28 15:47:26 +01:00
Zero Services GmbH 631bd09ce4 s3: add Zero Services (ZERO-Z3) provider
ZERO-Z3 is S3-compatible object storage built on Ceph RADOS Gateway,
hosted in the EU on Zero Services' own network (AS215197), with
region-specific endpoints (zero-fra1, zero-fra2, zero-eyl1).
2026-07-28 12:07:54 +01:00
Nick Craig-Wood b6ffdfa8e6 vfs: rename aux.go to nodeaux.go as aux is a reserved file name on Windows
This commit introduced aux.go which unfortunately is illegal on windows.

e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS

Rename aux_test.go too to keep the pair together.
2026-07-27 16:35:08 +01:00
Nick Craig-Wood b2aa82061f sftp: add --sftp-pin-host-key - Trust On First Use host key pinning
Add two new options, pin_host_key and host_keys, that
together provide a TOFU host-key validation mode for users who don't
maintain a known_hosts file. When --sftp-pin-host-key is used, rclone
records the server's host key into host_keys on the first successful
connection and verifies it on every subsequent connection.

host_keys is always validated when non-empty, so it can also be used
by hand to pin a known fingerprint without enabling TOFU writing.

known_hosts_file takes precedence if both are set. SSH host
certificates are rejected with a clear message pointing at
known_hosts_file. On-the-fly remotes log a warning since the captured
key cannot be persisted.
2026-07-27 14:57:21 +01:00
Nick Craig-Wood 83a366beae sftp: don't retry permanent connection errors
The connection pacer in getSftpConnection used to retry every error,
so permanent failures (host key mismatch, certificate rejection, auth
failure, etc.) were looped 10 times before reporting to the user.

Switch to using fserrors.ShouldRetry which matches the pattern other
backends use so only genuinely retriable errors (timeouts, EOF,
network blips) are retried and permanent errors are surfaced
immediately.
2026-07-27 14:57:21 +01:00
Nick Craig-Wood e006d7c13f vfs: fix crash when multiple mounts or servers share the same VFS
The VFS is shared between users with the same remote and options, for
example two mounts created over the rc, or a mount and an NFS server.

Each node has a single Sys() slot which mount, mount2 and serve nfs
all used to attach their per-node data. With a shared VFS the users
overwrote each other's data: at best churning the cached FUSE nodes,
and since the slot was an atomic.Value, panicking with "store of
inconsistently typed value" as soon as two users stored different
types on the same node.

This change gives each node auxiliary values keyed by owner, set with
SetAux and read with Aux, so each user of the VFS has an independent
slot. The mounts now cache their FUSE nodes under their own key,
leaving Sys - which is read through the os.FileInfo interface -
reserved for users like serve nfs which need to control what that
returns.

Nodes with nothing attached use less memory than before (one pointer
instead of an atomic.Value) and reads remain lock free.

Bug discovered while thinking about #9617
2026-07-27 14:55:20 +01:00
Nick Craig-Wood b2d0642825 serve s3: fix OOM and InvalidPart errors with concurrent multipart uploads - fixes #9616
Streamed multipart uploads had two problems when the client uploaded parts
concurrently:

Parts arriving ahead of the next part needed by the backend stream were buffered
in memory without limit, acknowledging each part as soon as it was received. A
client uploading faster than the backend could drain would therefore balloon the
server's memory to the size of the upload. Buffering is now bounded a new
--multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part
that would take the buffer over the limit is not read until the stream drains,
applying backpressure to the client instead of using unbounded memory.

A part uploaded again with the same number - typically a client retrying after
its request timed out - left a stale copy in the reorder buffer which made
CompleteMultipartUpload fail with InvalidPart, aborting the whole upload.
Re-uploaded parts are now handled properly: a copy still in the buffer is
replaced, an identical copy of an already streamed part is accepted as a no-op,
and only replacing an already streamed part with different content (which the
in-order stream cannot honour) is rejected.
2026-07-27 14:51:06 +01:00
Nick Craig-Wood d97e33fc88 Revert "drime: disable server side copy as it always fails"
This reverts commit 961266888f.

This has been fixed on the server side.
2026-07-27 12:27:02 +01:00
Nick Craig-Wood c81bc3aca2 test_all: mask internet archive tests which can't work 2026-07-27 12:25:22 +01:00
Nick Craig-Wood 2217d38273 test_all: run the Internet Archive tests with -short to speed them up
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
2026-07-25 18:48:31 +01:00
Nick Craig-Wood 662272e374 Add p1 to contributors 2026-07-25 18:48:31 +01:00
Nick Craig-Wood 479d67bef2 Add Giridhar to contributors 2026-07-25 18:48:31 +01:00
p1 c99b2d11ed drive: document shortcut IDs in lsf 2026-07-24 15:00:09 +01:00
Giridhar caf3d1cc3c lsf: document Google Drive shortcut IDs - fixes #7863
Signed-off-by: Giridhar <80974392+giri256@users.noreply.github.com>
2026-07-24 15:00:09 +01:00
Nick Craig-Wood a110fa5d47 build: update google.golang.org/grpc to fix multiple security problems
Multiple security vulnerabilities have been identified and addressed
in grpc-go affecting the xDS RBAC authorization engine
(internal/xds/rbac) and the HTTP/2 transport server implementation
(internal/transport).

This updates to master to add the fix which allows it to compile on
plan9.
2026-07-23 17:03:42 +01:00
Nick Craig-Wood 2cd9516037 Add Kyue to contributors 2026-07-23 17:03:42 +01:00
Nick Craig-Wood 339f7a2f45 Add Søren Lindberg to contributors 2026-07-23 17:03:42 +01:00
Kyue 7de6d611ba gui: respect explicit --rc-allow-origin instead of always deriving it from the bind address
Fixes #9640

The auto-derived CORS origin logic (added in #9603 for the wildcard-bind
case) ran unconditionally, so it clobbered an explicitly-passed
--rc-allow-origin too - the flag was silently ignored. That breaks the
documented Docker/remote-browser setup where the bind address and the
address the browser actually reaches the GUI at are different.

Now it only kicks in when the user hasn't set --rc-allow-origin
themselves.
2026-07-23 16:35:22 +01:00
maximilize 2f3895fa3c archive/squashfs: fix reading images with no fragment or xattr table
Bump go-diskfs to v1.9.4, which contains diskfs/go-diskfs#413 handling
squashfs images that have no fragment table or no xattr table. On v1.9.3
these images fail to list at all (ReadDir errors, or a nil xattr-table
dereference inside the library).

Adapt the backend to the v1.9.4 io/fs API: ReadDir now returns
io/fs.DirEntry (metadata via DirEntry.Info), and files are opened via
FileSystem.OpenFile instead of the removed FileStat.Open. The path
conversion added in c10eb47 (toIOFS) is reused.

Add a regression test with two committed fixtures built from trivial
placeholder content: an empty directory (no fragment table) and a small
tree whose superblock has the NO_XATTRS flag set while inodes still carry
an xattr index.

Fixes #9004
2026-07-22 11:30:21 +01:00
Søren Lindberg a1d906fd3d operations: fix Move godoc to note Copy fallback is accounted as a transfer - fixes #8799 2026-07-21 16:33:30 +01:00
FTCHD d67eff43ba gui: fix cross-origin API requests when bound to a wildcard address 2026-07-21 16:08:54 +01:00
Nick Craig-Wood 5e9b809a82 iclouddrive: fix "cannot unmarshal number" error when listing photo albums
CloudKit is inconsistent about how it encodes the isDeleted field on
album records, returning a JSON boolean (true/false) for some accounts
and a number (0/1) for others. The numeric form caused listing of a
photo library to fail with:

    json: cannot unmarshal number into Go struct field
    ckBoolField.records.fields.isDeleted.value of type bool

The encoding also varies over time, not just per account: a full HTTP
dump from the reporting user showed the server sending

    "isDeleted" : { "value" : 0, "type" : "INT64" }

but the same account later reverted to the boolean encoding with no
client change. Asset records already deliver isDeleted as a number, so
both encodings are in active use server side and either may appear.

Accept both encodings when parsing CloudKit boolean fields.

See: https://forum.rclone.org/t/error-when-trying-to-list-contents-of-primarysync-directory-in-icloud-photos/54028
2026-07-21 15:18:23 +01:00
Nick Craig-Wood b9009b1c13 test_all: use a fresh Internet Archive item for the integration tests
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
2026-07-21 10:28:47 +01:00
Nick Craig-Wood 74be61735c mega: fix moved files disappearing from listings between remotes
The session cache was only populated on a fresh username and password
login, so once a session ID was stored in the config every Fs
instance created its own Mega session with its own copy of the
account's node tree. The server side move code relies on all Fs
instances of a user sharing one session, and with separate sessions a
move between two rclone remotes grafted a node from one tree into
another, where the asynchronous event replays of the two sessions
raced and could detach the moved file from the destination directory
so it disappeared from listings.

Cache the session however the login was done. This also stops every
extra Fs instance re-downloading the whole account node tree.
2026-07-20 17:37:16 +01:00
Nick Craig-Wood 4d6cc0ba1f test_all: ignore TestListDirSortedFn for linkbox
Linkbox can't upload files starting with . - the same reason
TestListDirSorted is already ignored.
2026-07-20 17:22:12 +01:00
Nick Craig-Wood 4db5b91610 sync: fix one transform test error failing all the following tests
Sync refuses to delete files when the global error stats are non-zero
so a single backend error in one transform test made every following
transform test in the same test binary fail with "not deleting files
as there were IO errors". Reset the stats at the start of each test.
2026-07-20 17:21:33 +01:00
Nick Craig-Wood a4972de505 shade: retry server errors instead of failing the transfer
The pacer callbacks only retried 429 responses, so transient server
errors like 502 Bad Gateway failed transfers immediately. Retry the
standard retryable status codes and transport errors everywhere, make
chunk upload retries resend the whole chunk, and fix a potential nil
pointer dereference when logging a bad token response.
2026-07-20 17:21:33 +01:00
Nick Craig-Wood c10eb47bb6 archive: fix squashfs listings failing with invalid argument after update
The go-diskfs library now requires io/fs.ValidPath style paths (no
leading slash, "." for the root) so convert paths at the library
boundary. This fixes listing failures and the resulting test hangs
after the update to go-diskfs v1.9.3.
2026-07-19 19:03:09 +01:00
Nick Craig-Wood cbbf588c47 build: update all dependencies
This fixes the code to compile with the updated dependencies:

- squashfs: implement the new Path method required by the go-diskfs
  backend.Storage interface, returning an empty string as there is no
  underlying path.
- dropbox: convert to and from the SDK's new DBXTime type (an alias
  for time.Time) for ClientModified, TimeInvited and Expires.
- dropbox: remove a stray debug fmt.Printf from the shared folder
  listing.
- pin go-systemd to v22.6.0 to fix netbsd builds
  go-systemd v22.7.0 uses unix.ClockGettime and unix.CLOCK_MONOTONIC
  which golang.org/x/sys does not define for netbsd, so the build fails.
  See: https://github.com/coreos/go-systemd/issues/512
- pin google.golang.org/grpc to v1.80.0 to fix plan9 builds
  grpc v1.81.0 and later use syscall.Errno, syscall.ECONNRESET and
  syscall.ECONNABORTED which do not exist on plan9, so the build fails
  See: https://github.com/grpc/grpc-go/issues/9253
2026-07-19 19:01:12 +01:00
Nick Craig-Wood 961266888f drime: disable server side copy as it always fails
The /file-entries/duplicate endpoint returns a 500 Server Error for
every request (reported to Drime 2026-06) which made all server side
copies fail after 10 retries. Remove the Copy method so rclone falls
back to downloading and re-uploading instead. It can be restored if
Drime fix the endpoint.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood f0d77d07a2 mega: fix files reappearing in listings after being renamed
This updates go-mega to fix the handling of the event stream when a
file is moved or renamed. The server reports a move as a delete of
the old node followed by an add of the same node, and go-mega was
losing track of the node's identity in the process, which could make
the old name reappear in listings and leave the backend unable to
remove it.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 2cb127943d onedrive: skip permission tests when the server refuses sharing invitations
Microsoft has been progressively disabling sharing invitations on
both Business and Personal accounts - the driveItem invite API
returns 400 sharingFailed for any recipient on affected accounts -
which makes the permission writing tests impossible. Probe the API
once and skip the tests which need it when it is refused.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood ed60580730 sync: fix tests failing on backends that drop hashes on server side copy
ownCloud does not carry the checksum over to the destination of a
server side copy and refuses attempts to set it afterwards, so the
destination legitimately has no hash. The logger vs lsf check
compared the predicted hash against the empty hash and failed.

Treat an empty hash in the listing as unknown rather than wrong,
matching how sync itself compares hashes.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 8ac8d1821c test_all: ignore TestRWFileHandleWriteNoWrite for imagekit
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 0775b15ccf test_all: ignore TestSyncCompareDest for cloudinary
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 55b95489f3 imagekit: add mtime to the available metadata
The backend declares ReadMetadata but did not return the standard
mtime key, so the modification time was missing from the metadata.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 846f571eb7 imagekit: fix Open with a RangeOption returning the wrong data
Open decoded range options against an unknown size, producing a
negative offset for suffix ranges (eg the last N bytes) and sending
syntactically invalid Range headers which the server ignored. A
suffix range then returned the whole file instead of the requested
tail.

Decode the options against the known object size so the offset is
always absolute, only send a Range header when one was requested, and
honour the requested count when the server ignores the Range header.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood d6190dc4f2 linkbox: retry bot protection HTML challenge responses instead of failing
The Linkbox API is fronted by bot protection which under load
intermittently returns an HTML challenge page with a 200 status
instead of JSON. This surfaced as a fatal JSON decode error, and the
web API path misread the empty decoded result as an expired token,
hammering the login endpoint with requests which also failed to
decode and keeping the bot protection triggered.

Treat responses which fail to decode as retryable errors so the pacer
backs off until the block lifts, and only refresh the token when the
response actually parsed as JSON and reported an error.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood ba6ee46d39 yandex: fix modtime randomly reverting to the upload time after upload
Yandex Disk stores the modtime in a custom property set just after
the upload completes. The server sometimes silently drops this
property (the request returns success but the property is gone when
read back), leaving the object showing the upload time instead.

Check the modtime read back after upload and set it again if it
didn't stick.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 74d376a7fe putio: fix sync deletions failing with 400 TRASH_LOCK_TIMEOUT errors
put.io serializes trash operations per account so concurrent deletes
can fail with a 400 TRASH_LOCK_TIMEOUT error saying "There is an
ongoing blocking trash operation". This transient error was treated
as fatal - retry it instead.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 13354ac438 opendrive: fix uploaded objects returning the wrong hash and modtime
Objects looked up by name used the /folder/itembyname.json endpoint
which does not return the file's MD5 hash, so NewObject returned
objects with no hash. Read the metadata with /file/info.json (which
does return the hash) instead, using itembyname only to find the file
id when it isn't known.

Copy and Move returned objects with only the id and size filled in,
leaving the modtime and hash empty, and SetModTime kept nanosecond
precision in memory while the server stores seconds. These made the
returned objects differ from a fresh listing of the same objects.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 727c11e081 ulozto: fix server side moves between differently rooted remotes losing files
The Move method compared the source and destination paths relative to
their own Fs roots, so a server side move between two differently
rooted Fs instances on the same remote (as used by --backup-dir and
sync moves between remotes) was skipped as "already there" whenever
the relative paths coincided. With --backup-dir this destroyed the
file which was supposed to be backed up.

Compare the full paths including the Fs roots instead.
2026-07-17 18:29:39 +01:00
Nick Craig-Wood 99bef2d269 Add sijie-Z to contributors 2026-07-17 18:29:39 +01:00
sijie-ZandClaude Opus 4.8 31604f623f http: add Prefer to CORS Access-Control-Allow-Headers header
The rclone web GUI uses the Prefer header for sync/copy operations,
but the CORS middleware's preflight response did not include it in
Access-Control-Allow-Headers. This caused the browser to block
remote GUI copy requests with a CORS policy error:

  Request header field prefer is not allowed by
  Access-Control-Allow-Headers in preflight response.

Add 'Prefer' to the allowed headers list so remote GUI
access works correctly for file copy operations.

Fixes #9614

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 11:55:33 +01:00
Erol Ozcan fb25801f35 zoho: preserve root_folder_id on reconnect and allow setting it
The Zoho config system ended every interactive create, update and
reconnect by calling m.Set("root_folder_id", workspaceID) in the
workspace_end state, with fs.ConfigChoose defaulting to the first
workspace. Any existing root_folder_id was therefore overwritten and
the remote silently repointed to the first workspace root.

This matters because reconnect is the documented fix for the 401
INVALID_OAUTHSCOPE download error - tokens issued before the
ZohoFiles.files.ALL scope was added lack download access - so users are
told to reconnect and then find all subsequent list/copy/sync/delete
operations pointed at a different, often shared, workspace.

Gate the workspace selection the way the drive backend does for team
drives (#5454): if a root_folder_id is already set, ask "Change current
root folder id ...?" defaulting to No and keep it; only run workspace
selection when it is empty or the user opts in. The token type rewrite
still runs on every reconnect so the scope refresh is unaffected. Also
expose root_folder_id as a standard advanced option (Sensitive, so
config redacted masks it) so it can be set and discovered like on
drive/box/onedrive.

Fixes #9575
2026-07-16 18:12:00 +01:00
Nick Craig-Wood 8b812fff28 fs: fix passwords and tokens appearing in the debug log during rclone config
Previously running rclone config (or driving it via the rc API or web
GUI) with -vv would write secrets to the debug log.

This was dangerous as users debugging a failing config flow often
paste their -vv logs into the forum or GitHub issues.

These values are now redacted from the log as "XXX". Values whose
option is known are only redacted if the option is marked IsPassword
or Sensitive, so normal answers remain visible.

Use --dump auth to see the unredacted values when debugging a config
flow - rclone prints a warning that secrets will appear in the log
when this is in effect.

This was discovered by CodeQL: https://github.com/rclone/rclone/security/code-scanning/182
2026-07-16 16:11:22 +01:00
Nick Craig-Wood 454430a057 combine: don't return an error message as the remote name for a bad object
This fixes 30 spurious CodeQL path-injection alerts which used the
error message as a taint path from HTTP responses into filesystem
paths.
2026-07-16 14:47:03 +01:00
Nick Craig-Wood 2eb6f6d961 fs: don't log the contents of objects without a String method
The logging functions take an object which is rendered into the log
line. Rendering it with %+v dumps all its fields, which for an object
holding backend config would include decrypted credentials. Every
object currently logged is a string or has a String method, so render
anything else as just its type to keep credentials out of the logs.

See: https://github.com/rclone/rclone/security/code-scanning/183
2026-07-15 16:43:37 +01:00
Nick Craig-Wood d8b4966fa6 sftp: docs: clarify the security boundaries of --sftp-skip-links 2026-07-15 11:46:47 +01:00
Mikel Olasagasti Uranga 169f5b714c serve/http: compare zip test output semantically
Go 1.27 changes compress/flate output, which also changes archive/zip
byte output. The HTTP zip download tests currently compare raw zip bytes
against golden files, so they fail even though the generated zip archives
contain the expected files.

Compare zip entries and decompressed contents instead of the exact
compressed byte stream.

Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
2026-07-15 11:00:18 +01:00
Nick Craig-Wood c851d4dec5 vfs: fix vfs cache writeback timer not being stopped when --transfers reached
When processItems filled the last free transfer slot it checked the
next queued item's expiry before the transfer limit, so if that
expiry was still fractionally in the future the timer was reset
instead of stopped. This caused intermittent failures in
TestWriteBackMaxQueue which asserts the timer is stopped once
--transfers uploads are in progress.

Check the transfer limit first so the timer is always stopped when
the transfer limit is reached. The timer is restarted when an upload
finishes so nothing stalls.

Also fix a typo in TestWriteBackMaxQueue which named every queued
item "number1".
2026-07-14 14:29:21 +01:00
dougal c4d87bd6d4 fs/config: add tier to config wizard
This makes the overview from the docs accessible in the code.
2026-07-14 14:16:19 +01:00
dougal 384dff5e52 docs/data: export backend data 2026-07-14 14:16:19 +01:00
Nick Craig-Wood 4089e5af48 azurefiles: improve modtime precision from 1s to 100ns
The server stores the SMB last write time with 100 ns (FILETIME)
precision and returns it in full in listings, so advertise that as
the precision instead of one second. This makes syncs preserve
sub-second modtimes when comparing and copying files.

Note that this means rclone will consider files with modtimes
differing by less than a second as needing their modtime set where
before it did not.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 3e19032656 filescom: fix missing MD5 hash after uploading a file
The server computes the MD5 checksum asynchronously after upload, so
the object returned from Put and Update often had no MD5 while a
fresh listing shortly afterwards would report one. This broke
wrappers which compare exact hashes, such as the hasher backend's
fingerprint and the VFS cache. Retry reading the metadata for a short
time after upload until the MD5 appears.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 57b1f24566 netstorage: fix missing MD5 hash after uploading a file
The object returned from Put and Update had an empty MD5 checksum
while a fresh listing would report the MD5 the server computed for
the upload. This broke wrappers which compare exact hashes, such as
the hasher backend's fingerprint and the VFS cache. Stat the file
after upload to pick up the server computed MD5.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 0efafc5210 yandex: fix missing MD5 hash after uploading a file
Update (which Put also uses) cleared the object's MD5 checksum after
upload, so until the object was re-read it had no hash while a fresh
listing would report the MD5 the server computed. This broke wrappers
which compare exact hashes, such as the hasher backend's fingerprint
and the VFS cache. Re-read the object's metadata after upload to pick
up the server computed MD5 (and authoritative size).
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 449397b8df quatrix: fix incorrect modtime after uploading a file
Update (which Put also uses) kept the caller's full precision modtime
in memory while the upload finalize request sends it rounded to
microseconds, so until the object was re-read the modtime did not
match what a fresh listing would report. This broke wrappers which
compare exact modtimes, such as the hasher backend's fingerprint and
the VFS cache. Round the modtime to microseconds to match what the
server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 1404a9f9d1 protondrive: fix incorrect modtime after uploading a file
Update (which Put also uses) kept the caller's full precision modtime
in memory while the server stores second precision, so until the
object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 72907a7a90 pixeldrain: fix incorrect modtime and missing hash after uploading a file
The objects returned from Put, Move and Update kept the caller's full
nanosecond precision modtime (which write responses echo back) while
the server stores millisecond precision, so until the object was
re-read the modtime did not match what a fresh listing would report.
Update also discarded the upload response so the object had no SHA256
checksum until re-read. This broke wrappers which compare exact
modtimes and hashes, such as the hasher backend's fingerprint and the
VFS cache. Truncate the modtime to millisecond precision to match
what the server stores and populate the object from the upload
response.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 397c0f872a azurefiles: fix incorrect modtime after uploading a file or setting its modtime
Update and SetModTime kept the caller's full nanosecond precision
modtime in memory while the server stores SMB LastWriteTime with
100 ns (FILETIME) precision, so until the object was re-read the
modtime did not match what a fresh listing would report. This broke
wrappers which compare exact modtimes, such as the hasher backend's
fingerprint and the VFS cache. Truncate the modtime to 100 ns
precision to match what the server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood d1fcbadc1f webdav: fix incorrect modtime after setting a file's modtime
SetModTime kept the caller's full precision modtime in memory while
the PROPPATCH sets it on the server with second precision, so until
the object was re-read the modtime did not match what a fresh listing
would report. This broke wrappers which compare exact modtimes, such
as the hasher backend's fingerprint and the VFS cache. Truncate the
modtime to second precision to match what the server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood dee6e23d34 putio: fix incorrect modtime after setting a file's modtime
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood fc87735f80 jottacloud: fix incorrect modtime after setting a file's modtime
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 5d4c2c32a0 hidrive: fix incorrect modtime after setting a file's modtime
SetModTime kept the caller's full precision modtime in memory while
the server stores second precision, so until the object was re-read
the modtime did not match what a fresh listing would report. This
broke wrappers which compare exact modtimes, such as the hasher
backend's fingerprint and the VFS cache. Truncate the modtime to
second precision to match what the server stores.
2026-07-14 14:13:49 +01:00
Nick Craig-Wood 2a91ea8cf2 ftp: fix incorrect modtime after uploading a file or setting its modtime
SetModTime (and the no_check_upload Update path) kept the caller's
full precision modtime in memory while the server stores second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
2026-07-14 14:13:49 +01:00
Cao Yuhang 76196a2897 operations: fix core/du test with missing cache dir
TestRcDu relied on the default cache directory already existing. In clean
or container environments diskusage.New returned ENOENT, which the test
ignored before type asserting a nil result.

Use a temporary directory and require a successful response before
checking the disk usage values.
2026-07-14 11:23:58 +01:00
Nick Craig-Wood a0c09f1381 docs: add guidance for AI-assisted contributions
Add an AGENTS.md at the repository root describing the project's build,
test and code conventions for AI coding agents (Claude Code, Codex,
Cursor and similar). CLAUDE.md imports it so Claude Code reads the same
guidance.

Add an "AI-assisted contributions" section to CONTRIBUTING.md asking
contributors to understand and test AI-generated code themselves and to
trim the verbose comments these tools tend to produce, and add a
matching checkbox to the pull request template.
2026-07-13 17:48:10 +01:00
Nick Craig-Wood 16e199067e vfs/vfscache: fix "invalid seek position" error when cache files larger than the remote
Previously if a cached file had grown larger than the remote object
and the cached range metadata was out of sync with the cache file
(e.g. after an unclean shutdown) - reloading the file failed with an
"invalid seek position" error. This aborted the writeback and left the
file inaccessible.

Rclone now recovers the bytes that are still available from the remote
and logs an ERROR that the local file is likely corrupted after an
interrupted upload.

See #9231.
2026-07-13 15:30:03 +01:00
Nick Craig-Wood 7b9c5fd4d5 docker serve: add docs on what to expect when the plugin is restarted 2026-07-13 15:30:03 +01:00
Nick Craig-Wood 7a5175f223 docker serve: make Create idempotent to avoid "volume already exists" after restart
Docker may re-send Create requests for volumes that already exist,
especially after a plugin restart. Previously this returned
ErrVolumeExists which Docker surfaced as "volume name must be unique".

Now if a volume with the same name already exists, Create returns
success (no-op), matching the Docker volume plugin protocol's
expectation of idempotent operations.
2026-07-13 15:30:03 +01:00
Nick Craig-Wood 8bfe94770a docker serve: restore volumes concurrently so one slow remote doesn't block others
When restoring multiple volumes from saved state, each volume's
filesystem setup (including connecting to the remote) was done
sequentially. Now volumes are restored concurrently, so a single
slow or unreachable remote doesn't delay the restoration of other
volumes.
2026-07-13 15:30:03 +01:00
Nick Craig-Wood 64331a4eae docker serve: add timeout to volume restore so slow remotes don't block startup
When restoring volumes from saved state during plugin startup, a slow
or unreachable remote could block indefinitely in fs.NewFs. Add a
30-second per-volume timeout context so that individual volume failures
are logged and skipped rather than blocking the entire plugin.
2026-07-13 15:30:03 +01:00
Nick Craig-Wood 560d3928d0 docker serve: fix plugin timeout on restart when volumes have active mounts
Previously, restoreState in NewDriver would restore volumes AND perform
FUSE mounts synchronously before the Docker plugin socket was created.
This caused two problems:

1. The monChan was created after restoreState, but mount() sends on
   monChan, causing a deadlock (send on nil channel blocks forever).

2. Even with the channel fix, slow or hanging mounts during state
   restore would block the server socket from being created, causing
   Docker to time out after ~13 seconds with "no such file or
   directory" for the plugin socket.

Fix by:
- Moving monChan creation and monitor goroutine start before
  restoreState
- Splitting state restore into two phases: restoreState (metadata +
  filesystem setup only) and RestoreMounts (actual FUSE mounts)
- Calling RestoreMounts asynchronously after the server starts
  listening
- Performing mount restoration concurrently across volumes

Fixes #9231
2026-07-13 15:30:03 +01:00
Nick Craig-Wood 4235aa4fe5 protondrive: fix long hangs on permanent validation failures
shouldRetry treated every Code=200501 API error as a transient storage
block error and retried it. Proton also returns Code=200501 with an
HTTP 422 for permanent validation failures (e.g. a content key packet
that cannot be verified, or an upload format the account is not
enabled for), so these were retried until the operation timed out
causing a multi-minute hang.

This fixes it by only retrying Code=200501 when it is not a permanent
client (4xx) error.
2026-07-13 15:22:47 +01:00
Nick Craig-Wood bb17d07eb7 protondrive: fix gopenpgp: invalid data: user ID signature with wrong type on custom-domain account
Update to github.com/ProtonMail/gopenpgp/v3 from v2 by updating to

- github.com/rclone/go-proton-api@v1.0.3
- github.com/rclone/Proton-API-Bridge@v1.0.4

Fixes #9434
2026-07-13 15:22:47 +01:00
Nick Craig-Wood 0a39aa4d32 hdfs: fix incorrect modtime after uploading a file or setting its modtime
SetModTime (which Put and Update also use) kept the caller's full
precision modtime in memory while Chtimes sets it with second
precision, so until the object was re-read the modtime did not match
what a fresh listing would report. This broke wrappers which compare
exact modtimes, such as the hasher backend's fingerprint and the VFS
cache. Truncate the modtime to second precision to match what the
server stores.
2026-07-13 13:53:29 +01:00
Nick Craig-Wood 047e0daae3 mailru: fix incorrect modtime after updating a file or setting its modtime
Update and SetModTime kept the caller's full precision modtime in
memory while the server stores second precision, so until the object
was re-read the modtime did not match what a fresh listing would
report. This broke wrappers which compare exact modtimes, such as the
hasher backend's fingerprint and the VFS cache. Truncate the modtime
to second precision to match what the server stores.
2026-07-13 13:49:11 +01:00
Nick Craig-Wood 7dd42c93cf fstests: check backends return accurate object fingerprints after writes
Layers which wrap backends, such as the hasher backend and the VFS
cache, use fs.Fingerprint to detect whether an object has changed.
This only works if the object a backend hands back from a write
operation is identical to the object a fresh listing returns. If a
backend keeps the source's nanosecond precision modtime in memory
while the server stores milliseconds, or omits a hash the server
knows, every fingerprint comparison fails and cached hashes or files
are silently invalidated.

The existing tests compared modtimes within f.Precision() so they
could not detect these sub-precision divergences.

Add a checkFingerprint helper which asserts the fast and slow
fs.Fingerprint of the object returned from a write operation exactly
match those of the object read back from the remote, and wire it into
all the write paths which return or mutate an object: Put (all
variants), PutStream, large and streamed uploads, Update, SetModTime,
server side Copy and Move, and the metadata mutating Copy/Move
subtests.

Rework the ObjectOpenFingerprint test to use the same helper against
a fully read reference object, with subtests reporting which kind of
read is being checked.
2026-07-13 13:40:53 +01:00
Nick Craig-Wood 2eaaf91b83 filen: fix incorrect modtime after updating a file or setting its modtime
Object.Update and SetModTime left the in-memory modtime at the
source's nanosecond precision, while the server stores milliseconds.
Afterwards the in-memory state did not match what a fresh List would
report, which broke wrappers that compare modtimes (e.g. the hasher
backend's fingerprint, which made it lose track of hashes whenever a
file was replaced).

The SDK serializes modtimes as UnixMilli and NewIncompleteFile
already rounds to milliseconds, so Put and the chunked upload path
are unaffected. Round the modtime to milliseconds in Update and
SetModTime too, so every path holds the same value the server
returns, without needing to re-read the metadata from the server
after upload.

Fixes #9308
2026-07-13 11:22:04 +01:00
Nick Craig-Wood 7c803db8f3 hasher: fix Update not storing hashes in bolt DB after file replacement
When rclone sync replaced an existing file, it called Update which
pruned the old hash but never computed or stored the new one. This
left the file with no hash entry in the bolt DB.

This applies the same hashing logic in Put to Update: compute hashes
during the transfer via a hashingReader and store them afterwards. The
common hash-wrapping and hash-storing logic was extracted into a
function.

Fixes #9308
2026-07-13 10:44:03 +01:00
Nick Craig-Wood 9e0a5b66a4 march: fix unnecessarily listing dst directory when src listing finished
When doing a copy (no delete mode) without a logger, the destination
listing can be cancelled as soon as the source listing finishes, since
dst-only entries won't be processed.

This is particularly beneficial with --fast-list where the dst listing
may fetch the entire directory tree upfront via ListR. Cancelling it
early avoids waiting for a potentially large listing that won't be used.

Adds NoProcessDstOnly flag to March which, when set, cancels the dst
listing context once the source channel is exhausted in matchListings.

Fixes #9226
2026-07-12 17:10:10 +01:00
Nick Craig-Wood 2228e7c866 march: add context parameter to listDirFn for cancellable listings #9226
Add a context parameter to listDirFn so that each call site can pass
its own context. The closures in makeListDir previously captured
m.Ctx at creation time; they now use the context passed at call time
instead. This is needed so that processJob can pass a cancellable
context for the destination listing independently of the source.

Note: callers must pass m.Ctx (or a child of it) to preserve the
existing cancellation behaviour where listings stop when the march
context is cancelled.
2026-07-12 17:10:10 +01:00
Nick Craig-Wood c6cdb89935 config: fix normalization when obscuring passwords - fixes #9507
Interactively-entered passwords were run through NFKC Unicode
normalization before being obscured, which silently rewrote characters
such as ª (U+00AA) to a. The obscured password then revealed to
something different from what the user typed confusing everyone.

Normalization is only needed for the config encryption master
password, so apply it there (in SetConfigPassword) rather than in the
shared checkPassword used for backend password options.
2026-07-12 13:27:46 +01:00
Nick Craig-Wood 0a44cbff37 operations: fix operations/stat for directories wth large parent dirs
When `operations/stat` / StatJSON is called on a directory path it
lists the parent directory to find the target entry. If the parent has
millions of entries this is very expensive.

This fixes the problem for bucket-based backends with ListP by listing
the target directory itself first. It will stop the listing
immediately if any files are found meaning it is safe to run on
directories with millions of files.
2026-07-12 13:27:19 +01:00
Nick Craig-Wood fe78b559d1 yandex: fix 500 errors by waiting for uploads to complete before setting modtime
After PUTting a file to the upload URL, Yandex keeps the file locked
for writing until the upload operation finishes committing on the
server. The PUT returned before this happened, so the following
SetModTime raced the still-in-progress write and got spurious 500
Internal Server Error responses.

Capture the operation_id returned with the upload URL and poll the
operation status until it reports success before returning, so the file
is fully committed before we access it.
2026-07-12 13:26:39 +01:00
Nick Craig-Wood 63439b4444 cache: fix test flakiness by stopping the chunk cleaner promptly
The background chunk cleaner slept for the whole ChunkCleanInterval
(default 1 minute) before checking its stop channel, and only ran
CleanUpCache via the select default branch. This meant a cache that
had been stopped by StopBackgroundRunners could keep running
CleanUpCache for up to an interval afterwards.

The cache backend tests all share a single on-disk chunk store (the
TestInternalCache remote), so a lingering cleaner from a finished test
could call CleanChunksBySize and os.RemoveAll chunks that a later,
unrelated test had just written. The later test would then read a
chunk back and get an unexpected EOF - eg
TestInternalMaxChunkSizeRespected failing intermittently on CI.

Wait on a timer and the stop channel together so a stop is honoured
immediately and the cleaner can never run again once stopped.
2026-07-12 13:26:17 +01:00
Nick Craig-Wood 9a49790797 fs/logger: fix flaky tests by generating test data locally
The TestLogger/TestRepoCompare and TestLogger/TestBeforeVsAfter
testscript scenarios filled src and dst by downloading two old rclone
source archives from GitHub with `rclone copyurl`. Whenever GitHub or
the network hiccuped (eg a 502 Bad Gateway) the downloads failed and
the tests failed with it, making them flaky on CI.

Generate two overlapping trees of files in the test Setup instead.
They cover the same comparison categories the scripts exercise
(matching, differing, src-only and dst-only files) so the tests are
just as meaningful but no longer depend on the network.
2026-07-12 13:26:17 +01:00
Nick Craig-Wood 931126dd6e serve docker: document that socket access allows command execution
The volume plugin parses the remote option as a trusted connection
string, which can run local commands via backend options. Spell out
that access to the unix or TCP socket is equivalent to command
execution as the serving user, document the unix socket permissions,
and warn that the TCP socket is unauthenticated.
2026-07-11 16:49:01 +01:00
Nick Craig-Wood 0baac15c49 drive: detect shortcut loops to avoid infinite recursion
A folder shortcut pointing at one of its own ancestor folders (for
example a shortcut to FolderX placed inside FolderX) made rclone recurse
forever when dereferencing shortcuts, duplicating the folder contents
until the disk was full.

Rclone now detects when a folder shortcut targets an ancestor directory
using the directory cache, leaves that shortcut out of the listing and
logs an ERROR, so the rest of the drive can still be copied.

Fixes #7118
Fixes #9565
Closes #9051
2026-07-10 18:45:41 +01:00
Nick Craig-Wood 887c2b6b58 local: don't resolve relative roots to absolute paths - fixes #9510
cleanRootPath used filepath.Abs which prepends the current directory,
but the resulting absolute path does not always refer to the same
directory as the original relative path - for example when the current
directory is shadowed by a mount or has been removed. This made
"rclone copy --links . ../dst" fail where "cp -ra . ../dst" succeeds.

rclone now cleans the path lexically on non-Windows platforms instead,
leaving relative roots relative so the OS resolves them against the live
working directory. Windows still makes the path absolute as required for
UNC long-path conversion.
2026-07-10 18:45:41 +01:00
Nick Craig-Wood d40423765b config: add config unset command to remove options from a remote - fixes #9541
Previously the only way to remove an option from a remote was to set it
to an empty string, which is not the same as deleting it - a present but
empty value overrides the option's default whereas a deleted key
restores it. Editing the file by hand isn't an option for an encrypted
config either.

This adds a "config unset" command and a "config/unset" rc endpoint to
remove one or more keys from an existing remote.
2026-07-10 18:45:41 +01:00
Nick Craig-Wood 060b10c8cc serve s3: fix streamed multipart uploads not being atomic
Before this change a streamed multipart upload wrote its parts
straight to the object's final path on the underlying remote. That
meant an in-progress upload overwrote any object already stored under
that name, and aborting or failing the upload destroyed it. The
opposite of the S3 guarantee that an object only changes on a
successful CompleteMultipartUpload.

Remotes that upload atomically already (PartialUploads is false, e.g.
object stores) are safe to stream straight to the destination, so they
still do. Remotes where a partial upload is visible (PartialUploads is
true, e.g. local) now stream the parts to a temporary object instead
and move it, server-side, into its final place only when the upload
completes. A failed or aborted upload then just removes the temporary
object and leaves any pre-existing object untouched.

The temporary-object path needs the remote to support a server-side
move or copy in addition to PutStream uploads fall back to being
buffered in memory as before. The temporary objects are named with a
leading ".rclone_multipart_upload_" and hidden from listings.
2026-07-10 18:45:41 +01:00
Nick Craig-Wood ac7d1bbdfd serve s3: fix aborted multipart uploads appearing as ghosts
Before this change when an aborted upload had overwritten a
pre-existing object of the same name, this left a ghost of that object
in every listing.

This invalidate the VFS cache on all the multipart upload abort paths,
so listings reflect what is actually on the underlying Fs.
2026-07-10 18:45:41 +01:00
Nick Craig-Wood 0aba1fd2eb Add 0rangeSeaW0lf to contributors 2026-07-10 18:45:41 +01:00
0rangeSeaW0lfandClaude Fable 5 9f4845a40c internxt: surface re-login error when re-auth fails in NewFs
When the stored token is expired and refreshOrReLogin fails, the
returned error wrapped the original 401 and discarded authErr, hiding
the actionable reason (e.g. "account requires 2FA - please run: rclone
config reconnect remote:"). Wrap both errors so the user sees why
re-auth failed and what to do about it.

Fixes #9583

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-10 11:01:57 +01:00
Erol Ozcan 171e86369a zoho: treat R008 unauthorized as directory not found
Zoho's WorkDrive listing API returns "401 R008 Unauthorized access"
(not a 404) when a folder id no longer resolves to a listable folder,
because it was deleted or never existed. A freshly refreshed token still
gets it, so it is not a token problem and retrying it is futile - and can
escalate to a 429 F7008 rate-limit penalty.

Handle it as a missing directory instead: shouldRetry no longer retries a
bare R008 401, listAll maps it to fs.ErrorDirNotFound, and
readMetaDataForPath flushes the stale parent from the dircache and reports
the object as not found so a later create re-resolves the parent. This lets
the VFS self-heal a stale cached directory id instead of hard-failing the
operation, and stops the VFS integration tests failing on a stale directory id.

Fixes #9578
2026-07-09 18:13:55 +01:00
Erol Ozcan 037340b1b5 zoho: fix large file overwrite creating a duplicate instead of replacing - fixes #9585
The /stream/upload endpoint's overwrite flag is the x-prefixed header
x-override-name-exist, matching its siblings x-filename, x-parent_id and
x-streammode. uploadLargeFile sent the un-prefixed override-name-exist, which
the endpoint ignores, so overwriting a file of 10 MiB or larger created a
renamed duplicate instead of updating the existing file. The small upload API
passes the flag as a query parameter and is unaffected.
2026-07-09 18:11:29 +01:00
Nick Craig-Wood 2b099f2667 zoho: fix flaky folder list limiter test under concurrent listings
The folder list limiter granted the caller-supplied time immediately in
the burst phase, so concurrent callers observing time.Now() out of order
could record grant times that moved backwards. The sliding safety log
indexes grants as an ordered history, so out-of-order grants could also
breach the rolling-window cap. Clamp each grant to be at or after the
previous one so grant times are always monotonic.
2026-07-09 18:09:13 +01:00
Nick Craig-Wood 5b589e8f7c Add TowyTowy to contributors 2026-07-09 18:09:13 +01:00
TowyTowyandClaude Fable 5 f2e43b7c14 seafile: don't implement IDer as the id is a content hash not a unique ID
Seafile returns a content-addressed id (a git-like hash of the file
contents) rather than a unique per-object identifier. Two different
files with identical contents are therefore given the same id, even on
different servers.

operations.SameObject assumes IDs uniquely identify an object, so when
syncing between Seafile remotes it treated two distinct but
identical-content files as the same object. The guard in NeedTransfer

    if Equal(ctx, src, dst) && !SameObject(src, dst)

then evaluated to false for unchanged files (Equal was true because the
sizes matched, and SameObject was true because the ids matched), so
rclone re-uploaded them on every sync.

Seafile has no modtime and no usable content hash, so dropping the IDer
implementation makes SameObject fall back to path comparison and lets
unchanged files be skipped correctly.

Fixes #8300

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 17:50:26 +01:00
Nick Craig-Wood ed9eec0f20 fs/chunkedreader: fix spurious errors when a parallel stream is closed early
Closing a stream in the parallel chunked reader cancels the stream's
context, so the in-flight read returns context.Canceled. This was wrapped
and returned as "failed to read stream", which the VFS cache downloader
treats as a real download error - it only recognises
asyncreader.ErrorStreamAbandoned as a benign teardown, as returned by the
sequential reader.

Return asyncreader.ErrorStreamAbandoned for a cancellation so tearing down
the parallel reader (on close, seek or reposition) is recognised as benign,
matching the sequential reader, instead of logging download errors and
retrying when --vfs-read-chunk-streams is used with --vfs-cache-mode full.
2026-07-09 17:46:31 +01:00
Nick Craig-Wood 6e0cde076a vfs/vfscache: fix IO error by recreating the cache file if it has been removed
_createFile opened the cache file with O_RDWR but not O_CREATE, relying
on the file already existing. When the file had been removed underneath
rclone - either by _checkObject dropping a stale entry during open, or by
external deletion - the open failed and surfaced a hard "IO error" to the
application instead of recreating the file. Add O_CREATE so the cache
self-heals in that case.
2026-07-09 17:46:31 +01:00
Nick Craig-Wood d66235d556 fstests: check opening an object doesn't change its fingerprint
Add TestObjectOpenFingerprint which reads an object's fast and slow
fingerprints, then opens it in several ways - a full read, a seek read
and ranged reads - and after a refresh, checking the fingerprint never
changes.
2026-07-09 17:46:31 +01:00
Nick Craig-Wood 70c815d1b7 azureblob: fix MD5 being dropped on range reads causing vfs cache re-downloads
On a range request Azure returns the whole-blob MD5 in the
x-ms-blob-content-md5 header (BlobContentMD5) and leaves Content-MD5
empty. The download metadata decoder only read Content-MD5, so every
ranged read overwrote the object's MD5 with an empty string.

With --vfs-cache-mode full this changed the object's fingerprint between
opens, so the VFS cache judged every reopened file as stale and
re-downloaded it, defeating the cache and inflating egress. Prefer
BlobContentMD5 and never overwrite a known hash with an empty one.
2026-07-09 17:46:31 +01:00
859439c1e0 config: fix config_template_file and config_template being ignored via config/create - fixes #9572
When creating or updating a remote through the rc api (config/create,
config/update), parameters whose name starts with the ephemeral prefix
"config_" (for example config_template_file and config_template used to
customise the OAuth success page) were silently ignored.

updateRemote sets each supplied parameter into the config mapper, but
skips the "config_" prefixed keys so they are never written to the
config file. That guard is correct, because the mapper's setter writes
to the config file and these values are ephemeral. However backends read
these values back from the mapper (oauthutil reads config_template_file
and config_template via m.Get), so dropping them entirely meant the
values could never reach the backend and the default template was always
used.

Collect the ephemeral parameters into a separate map and add it to the
mapper as a getter overlay at PriorityNormal after the loop. The values
are now readable through m.Get without being written to the config file,
which is the same approach rclone authorize already uses to expose a
template supplied on the command line.

Fixes #9572

Co-authored-by: Hakanbaban53 <93117749+Hakanbaban53@users.noreply.github.com>
Co-authored-by: maximilize <3752128+maximilize@users.noreply.github.com>
2026-07-09 15:44:49 +01:00
jzunigax2 75429ca647 internxt: handle file size limit errors during uploads 2026-07-09 12:42:21 +01:00
jzunigax2 0ce75add5a internxt: add Move and DirMove methods for server-side file and directory operations 2026-07-09 12:42:21 +01:00
mathieulongtin 86f58d7972 docs: azureblob,azurefiles: clarify documentation on use_az 2026-07-09 12:37:07 +01:00
Nick Craig-Wood b0c47d19b1 Changelog updates from Version v1.74.4 2026-07-08 20:10:17 +01:00
Nick Craig-Wood 1154afebee local: stop --links symlinks escaping the destination directory CVE-2026-54572
With -l/--links rclone recreates a .rclonelink object as a symlink. A
malicious or compromised source could serve a symlink whose target points
outside the destination, plus a sibling object whose path traverses it, so
that rclone followed the planted symlink and wrote outside the destination
causing arbitrary file write.

When translating symlinks, rclone now performs all destination writes
(directory creation, file writes and symlink creation) through an os.Root
anchored at the destination. os.Root resolves every path component relative
to the destination's file descriptor and refuses any that escapes the root,
even under concurrent modification, so a planted symlink can never be
traversed out of the destination.

Symlinks are still reproduced verbatim - including ones whose target points
outside the destination - so backups remain faithful. Only writing
*through* such a link is refused. In-tree symlinks are unaffected.

Fixes CVE-2026-54572
Fixes GHSA-cf44-9pgv-m4xc
2026-07-08 16:12:24 +01:00
Nick Craig-Wood 637a830002 local: don't restore setuid/setgid/sticky bits from metadata by default GHSA-945v-v9p3-v5xw
When applying the "mode" from --metadata the local backend cast the
source value straight to an os.FileMode, so a source that supplied a
mode with Go's setuid, setgid or sticky bits set would have those bits
applied to the freshly written file. As both the file content and its
metadata come from the source remote, a malicious source could plant a
setuid binary, and a victim running "rclone copy -M" as root against
an untrusted remote could end up with a root-owned setuid binary with
attacker-controlled content.

Rclone records "mode" in the unix st_mode layout where the special
bits live in different positions to Go's os.FileMode, so honest
sources never actually round-tripped these bits in the first place.
Apply only the permission bits by default, which closes this off and
is backwards compatible, and add the --local-metadata-restore-special-bits
lag to restore the previous behaviour for trusted sources such as
restoring a system backup made by rclone.

See: GHSA-945v-v9p3-v5xw
2026-07-08 16:09:47 +01:00
Nick Craig-Wood 1a28451ea6 s3: strip STS security token on same-host HTTPS->HTTP redirect GHSA-cf44-9pgv-m4xc
The CheckRedirect policy strips the X-Amz-Security-Token header when a
redirect chain crosses a host, but it only compared the host and ignored
the scheme. A redirect that kept the same host:port but downgraded
https:// to http:// was treated as the same host, so the STS session
token was re-sent over a plaintext connection where it could be observed.

Fixes GHSA-cf44-9pgv-m4xc
2026-07-08 16:08:32 +01:00
Nick Craig-Wood dade21c161 serve restic: fix --private-repos isolation bypass CVE-2026-59733
A user could reach another user's private repository by sending a path
such as /<me>/../<victim>/config. The authorization check compares the
first path segment against the authenticated user, while the backend
object key was built from the raw, un-cleaned URL path.

Reject any non-canonical request path so the authorization segment and
the backend object key can no longer disagree.

Fixes GHSA-fqj9-69pf-6pjg
2026-07-08 16:07:11 +01:00
Nick Craig-Wood d11efe0d58 archive extract: fix path traversal letting archives escape the destination CVE-2026-59732
Archive entry names are attacker controlled. `rclone archive extract` stripped
only a leading `./` and then joined the entry name onto the destination
directory with `path.Join`, which collapses `..` segments. An entry such as
`../escaped.txt` extracted into `:s3:bucket/safe/prefix` therefore resolved to
`bucket/safe/escaped.txt`, outside the selected `prefix` directory - a path
traversal ("Zip Slip") attack that could create or overwrite sibling objects on
any destination remote.

Entry names are now validated before use: a leading `./` is still stripped (tar
archives created with `tar -czf archive.tar.gz .` rely on this), but any entry
with a `..` path component is rejected. Both `/` and `\` are treated as
separators when looking for `..`, as the local backend treats `\` as a path
separator on Windows.

Fixes: GHSA-4vr5-p2gc-h23p
2026-07-08 16:05:38 +01:00
Nick Craig-Wood 83d1e62aa9 serve s3: fix path traversal letting clients see files in the root GHSA-8v25-v8p6-qf7v
S3 object keys are opaque names that may legally contain `..` segments. `serve
s3` built backend paths with `path.Join(bucket, key)`, which normalised the key
so a request such as `GET /bucket/../root-secret.txt` resolved to a file outside
the selected bucket elsewhere under the serve root. Listing prefixes and
multipart uploads were affected also.

This did not allow reading of files outside the root, but did allow reading of
files in the root which normally aren't visible; only directories are visible as
buckets normally.

Because `serve s3` maps keys to file paths it cannot represent every opaque S3
key, so rather than normalising keys (which would alias distinct keys onto one
file as well as allow traversal) it now rejects any key that is not already in
canonical path form - containing `..`, `.`, `//` or a leading or trailing slash
- with a 400 Bad Request, as MinIO does. Directory listing prefixes are
validated the same way but allow the empty bucket-root prefix and an optional
trailing slash.

Fixes: GHSA-8v25-v8p6-qf7v
2026-07-08 16:02:29 +01:00
Nick Craig-Wood e753736df6 build: fix multiple CVEs by upgrading to go1.26.5
- CVE-2026-39822: os: Root escape via symlink plus trailing slash
- CVE-2026-42505: crypto/tls: Encrypted Client Hello privacy leak
2026-07-08 15:00:33 +01:00
Nick Craig-Wood 2d6d0da37b s3: fix mounting a prefix failing with 403 when HEAD is not permitted
When mounting or otherwise opening an S3 prefix without a trailing
slash, rclone probes the path with a HEAD request to see whether it is
actually a file. Since v1.72.0 (#8975) any error other than "not
found" from that probe was fatal, so credentials scoped to a prefix -
which return 403 rather than 404 for the prefix key - could no longer
open the prefix at all.

6440052fbd s3: fix single file copying behavior with low permission

A 403 on the probe is ambiguous: it can mean either "this is the file
you named but you may not HEAD it" or "this is a prefix you may list
but not HEAD". When the HEAD is not permitted we now fall back to a
listing to disambiguate: if the path has children it is treated as a
directory, otherwise it is treated as a file.

Fixes #9582
2026-07-08 12:02:43 +01:00
Nick Craig-Wood b5a81dab76 drive, googlephotos: warn in config wizard before using the shared client_id #9580
rclone's shared Google Drive and Google Photos client_id is being
retired and will stop working during 2026. When creating a new remote
that would use it, the config wizard now warns the user and asks the
user to enter their own client_id and secret instead. Service account
and environment auth are unaffected as they don't use the shared
client_id.

See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
2026-07-07 12:35:37 +01:00
Nick Craig-Wood d03eb58586 drive, googlephotos: warn when using rclone's shared client_id #9580
The shared Google Drive and Google Photos client_id is being retired and
will stop working during 2026. Warn users who rely on it (ie who have not
configured their own client_id) so they can create their own in advance.

The warning is only shown for auth flows that actually use the shared
client_id, not for service account, environment or anonymous auth.

See: https://forum.rclone.org/t/google-drive-and-google-photos-users-action-required/54005
2026-07-07 12:35:37 +01:00
Nick Craig-Wood 42f7eda4f1 drive: fix stray %!(EXTRA) in unexportable google document log message 2026-07-07 12:35:37 +01:00
Nick Craig-Wood 60cb844f9a serve/http: fix --disable-zip so it works over rc
The --disable-zip flag was registered manually and was missing from
OptionsInfo, so it could not be set over the rc interface. Move it
into OptionsInfo like serve webdav does, which keeps the command line
flag and also makes it settable via rc.
2026-07-07 12:35:37 +01:00
Nick Craig-Wood f4fea9c777 Add Erol Ozcan to contributors 2026-07-07 12:35:37 +01:00
Nick Craig-Wood 43ee4dab32 Add Amit Mishra to contributors 2026-07-07 12:35:37 +01:00
Nick Craig-Wood f843300e32 Add blackflytech to contributors 2026-07-07 12:35:36 +01:00
Nick Craig-Wood f62db1f37c Add Sanjays2402 to contributors 2026-07-07 12:35:36 +01:00
Nick Craig-Wood f7953c5785 Add Sandy Luppino to contributors 2026-07-07 12:35:36 +01:00
Nick Craig-Wood 56aa296cd1 Add dithwick to contributors 2026-07-07 12:35:36 +01:00
Nick Craig-Wood 0db0fe1e43 Add maximilize to contributors 2026-07-07 12:35:36 +01:00
Erol Ozcan 2e2a50cc44 zoho: rate limit repeated listings of the same folder
WorkDrive throttles its listing API (GET files/{id}/files) PER
folder, independently of the overall request rate: at most ~19
listings of one folder are allowed in any rolling ~60s window and
the 20th returns F7008 with a ~300s Retry-After (measured live -
every observed trip landed exactly on the 20th listing inside a
window). fstests re-lists the same working directory after almost
every sub-operation, which is why the integration suite could not
pass.

Add a per-folder listing limiter with a true per-window cap: each
window starts with --zoho-list-folder-burst listings passing
back-to-back (the burst re-arms at every window boundary, so a
sync re-listing one directory a few times never waits), the rest
of the budget is spaced evenly across the window, and a sliding
log of recent listings guarantees no rolling window ever exceeds
--zoho-list-folder-limit (default 19) per --zoho-list-folder-window
(default 60s) for any traffic pattern. The registry is
process-wide and keyed by region+folder id so every Fs instance
shares one budget per physical folder; idle entries are evicted
after a window, which is lossless because Zoho's window has also
cleared by then.

Defaults were validated against the live service: bursts of 4-6
under the 19-per-60s cap ran clean while an over-cap probe tripped
F7008 exactly at the 20th listing, and a full test_all -backends
zoho run passes cleanly.

Fixes #9570
2026-07-05 12:28:52 +01:00
Erol Ozcan 1daa03f108 zoho: log throttling once per episode at NOTICE
A 429 stall was only visible as a DEBUG pacer line, so without -vv
rclone appeared to hang for 2-5 minutes. In one night's batch logs 17
job starts produced only 4 completions because the silent stalls
looked like hangs and the jobs kept getting killed, re-triggering the
throttle.

Log the first 429 of each throttle episode at NOTICE with the server
message and the wait time. An episode ends when a request succeeds
after the penalty window; retries within an episode stay at DEBUG via
the existing pacer logging. State is two atomics behind a pointer on
Fs, so shallow Fs copies share it and concurrent checkers are safe.

See #9570
2026-07-05 12:28:52 +01:00
Erol Ozcan ef94788d5a zoho: add --zoho-tpslimit and --zoho-tpslimit-burst
Zoho throttling is account/plan-dependent; measurements show a
sustainable listing rate of ~6 requests/s on a production account -
going faster drains a token bucket and stalls ~2 minutes per
Retry-After, which is strictly slower overall. Add per-remote pacer
options (default 6/1) using the same token-bucket pacer as the Google
Drive backend. Set --zoho-tpslimit 0 to disable the cap.

See #9570
2026-07-05 12:28:52 +01:00
Erol Ozcan 4ab4b952be zoho: honour Retry-After header on 429
Zoho WorkDrive now sends a Retry-After header on 429 (it did not when
the backend was written). Waiting the hard-coded 60s retried too early
when the server asked for more (Retry-After: 299 is common) and the
penalty escalated (observed 84s -> 239s). Honour the header plus a 1s
margin - retrying at exactly Retry-After still finds an empty token
bucket and burns ~16 immediate 429s - and keep 60s as the fallback
when the header is absent.

shouldRetry becomes a method on *Fs so the retry decision has access
to the remote's state; later commits build on this.

See #9570
2026-07-05 12:28:52 +01:00
Sanjay Santhanam cfb9a10a3d serve webdav: fix MOVE overwrite failing without Overwrite header
Per RFC 4918 section 10.6, when the Overwrite header is omitted from a
COPY or MOVE request the resource MUST treat the request as if
Overwrite: T had been sent.

The upstream golang.org/x/net/webdav library mishandles this for MOVE
by checking == "T" instead of != "F", so an absent header is treated
as Overwrite: F and the request fails with 412 Precondition Failed.

Normalise the header to T in the rclone WebDAV server before
delegating to the upstream handler when the client did not send one.
This restores RFC-compliant default behaviour and can be removed once
the upstream fix in golang/go#66059 lands and the golang.org/x/net
dependency is bumped.

Fixes #9496
2026-07-04 09:57:27 +01:00
Amit Mishra cb41e42d04 fs: fix negative offset when a suffix Range request exceeds object size
A Range header requesting a suffix longer than the object (e.g.
"bytes=-90407" against a 5 byte object) caused RangeOption.Decode to
compute a negative offset (size - End), which serve.Object then used
directly as a slice/seek offset and panicked with "slice bounds out of
range". FixRangeOption (used by backends like OneDrive/Box that lack
native suffix-range support) had the same root cause: it produced a
RangeOption with a negative Start, which Header() silently dropped,
turning the request into the wrong byte range instead of erroring or
serving the whole object.

Per RFC 7233 section 2.1, when the suffix-length exceeds the
representation size, the entire representation should be served.
Clamp the computed offset/start to 0 in both places.

Fixes #6310
2026-07-04 09:11:52 +01:00
blackflytech b12251f07f chore: fix some function names in comments
Signed-off-by: blackflytech <blackflytech@outlook.com>
2026-07-02 11:30:15 +01:00
Gaurav 761af75a25 filter: add --files-from0 to support NUL-delimited input - fixes #9537 2026-07-02 11:21:21 +01:00
Sanjays2402 c91c4cbbff accounting: fix goroutine leak in NewStatsGroup for zero-transfer rc jobs
NewStatsGroup started the averageLoop goroutine unconditionally at
group creation. In an rcd daemon driven by many short rc sync/move
calls (a common pattern for scheduled spool flushes), each call gets
a fresh job/N stats group. When such a job transferred zero files
the loop was never stopped, because _stopAverageLoop is only reached
via DoneTransferring once transferring and checking both go from
non-empty back to empty, which never happens if nothing was ever
transferring in the first place. The result was one leaked goroutine
per rc call, growing unbounded until the daemon was OOM-killed
(reported: ~61k goroutines and ~640 MB RSS after ~7 days from a
per-minute timer over 6 mappings).

This is the same class of leak as #8571, which fixed the equivalent
auto-start in NewStats. Fix it the same way: do not start the average
loop at group creation. NewTransfer and NewTransferRemoteSize already
call startAverageLoop when real transfer activity begins, and
DoneTransferring already stops it when the last transfer completes,
so on-demand behaviour is unchanged for groups that actually do work.
Groups that never transfer anything now cost zero goroutines.

Adds a regression test that fails without the fix.

Fixes #9567
2026-07-02 11:15:18 +01:00
Sandy Luppino c1b5756eec vfs/vfstest: add Mknod test exercising all three mount backends
Mknod is now implemented by mount, mount2 and cmount, so exercise it from
the shared vfstest suite: create a regular file (S_IFREG) through the
mounted path and check it reads back as a 0-byte regular file. This is the
path the kernel NFS server drives when a client creates a file over an
exported mount.

Before this there was no shared coverage for Mknod; it now runs against
each backend via RunTests.

Suggested in #9548.

Signed-off-by: Sandy Luppino <s.luppino@opendrives.com>
2026-07-02 11:14:13 +01:00
Sandy Luppino 22aba81074 cmd/mount2: fix NFS directory listings by supporting non-zero Seekdir offsets
Seekdir handled only a rewind to offset 0 and returned ENOTSUP otherwise.
The stateless kernel NFS server opens a fresh directory handle and seeks to
the last returned cookie on every readdir continuation, so any listing
spanning more than one readdir batch failed over NFS. dirStream is a
snapshot taken at Readdir time and go-fuse assigns each entry a sequential
offset, so seeking to off positions the stream at index off; off == 0 still
resets to the start, preserving the rewind/re-read behaviour.

Before: ls of a directory that spans more than one readdir batch failed
over NFS with "Unknown error 524".
After: it lists correctly.

Adds TestDirStreamSeekdir covering rewind, mid-stream resume and the EOF
clamp. The full NFS path was validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.

Fixes #9547
2026-07-01 16:37:54 +01:00
Sandy Luppino 92b78aefa4 cmd/mount2: fix ESTALE over NFS by reporting stable inode numbers
setAttr left attr.Ino unset (0) and the NewInode sites left StableAttr.Ino
unset, so the kernel saw inode 0 while the node identity differed, which
breaks NFS file-handle validation. Set both to the stable VFS inode. The
bazil cmd/mount backend does not hit this because its framework assigns
stable inodes automatically; go-fuse needs them set explicitly.

Before: chmod/chown/truncate on a just-written file through an
NFS-exported mount2 mount failed with ESTALE.
After: they succeed.

Exercising the NFS handle-validation path needs a kernel NFS server, so it
isn't covered by the local vfstest harness; validated against a real Linux
nfs-kernel-server export over NFSv3, NFSv4.0 and NFSv4.2.

#9547
2026-07-01 16:37:54 +01:00
Sandy Luppino 6eecd45bed cmd/mount2: fix NFS file creation by implementing Mknod
The kernel NFS server creates regular files with MKNOD (it
creates-then-opens, so vfs_create routes through fuse_create -> FUSE_MKNOD
when there is no open intent), but mount2 only implemented Create
(FUSE_CREATE, used by local and SMB clients). Without Mknod every NFS file
creation failed with ENOTSUPP. This mirrors the cmd/mount mknod handler from
#2115.

Before: touch through an NFS-exported mount2 mount failed with ENOTSUPP.
After: files create normally.

Exercising this needs a kernel NFS server, so it isn't covered by the
local vfstest harness; validated against a real Linux nfs-kernel-server
export over NFSv3, NFSv4.0 and NFSv4.2.

#9547
2026-07-01 16:37:54 +01:00
dithwick bf2945bf40 smb: fix for IBM iSeries and signature verification
This commit updates go-smb2 to a version that fixes the following issues.

- Issue connecting to SMB share from IBM iSeries
  (see https://forum.rclone.org/t/smb-shares-on-ibm-iseries-update/53550)
- Fix for signature verification for async STATUS_PENDING response which
  resolves a transfer issue with Lenovo DSS-G.
  (see https://github.com/CloudSoda/go-smb2/pull/48)

It also provides additional security hardening, fixing panics on malicious
server input
(see https://github.com/CloudSoda/go-smb2/pull/51)
2026-07-01 12:00:26 +01:00
maximilize 294e985e2e filter: support nested {} alternates in glob filters - fixes #7220 2026-07-01 10:26:24 +01:00
Nick Craig-Wood b59a17e5e8 mega: fix hard deleted files reappearing in listings - fixes #9554
The mega backend keeps the whole account as an in-memory tree. go-mega's
Delete with destroy=true (hard_delete) removed the node from its lookup
table but left it in its parent's children, so a hard deleted file kept
showing up in directory listings until the tree was reloaded. In a
long-running mount or serve the file reappeared once the VFS directory
cache expired and re-read the backend.

Update go-mega to pick up the fix, and let the ghost test exercise
whichever delete mode the remote is configured with.
2026-07-01 10:06:53 +01:00
Nick Craig-Wood d489c59a59 mega: wait for server events after upload, delete and move
The mega backend keeps the whole account as an in-memory tree which
go-mega reconciles asynchronously from the server's event stream. After
an upload, delete or move the optimistic local update could be undone
moments later when go-mega replayed the corresponding server event,
re-adding a node to its parent. In a long-running process such as mount
or serve this could briefly show a just-deleted file in a listing, or
make a moved file reappear shortly afterwards.

Wait for the server to confirm uploads, deletes and moves so the
in-memory tree is settled before returning, matching what Rmdir and
Purge already do. The move wait was also previously started after the
server-side move so only the rename was covered.

Add an internal test reproducing the lingering listing via a tight
put/remove/list loop.
2026-07-01 10:06:53 +01:00
Nick Craig-Wood 9728fa0ede vfs: fix hang reopening a file during the handle-caching grace period
When a cached file was closed, --vfs-handle-caching kept its handle and
downloaders alive for a grace period and closed them later from a timer.
The deferred close drops the item lock while tearing down the
downloaders, leaving the file handle open. A reopen landing in that
window saw no grace timer and a live handle, failed to create the cache
file with "internal error: didn't Close file" and removed the cache
file, which hung the application reopening the file.

Reopens now wait for an in-progress grace-period close to finish so they
start from a fully closed item.

See: https://forum.rclone.org/t/opening-a-recently-closed-and-cached-file-hangs-rclone/53986/
2026-06-30 20:30:49 +01:00
max 48f8d63d9e s3: correct documented copy_cutoff minimum to 1 byte - Fixes #7391 2026-06-30 20:28:50 +01:00
Hakan İSMAİL b36b679202 rc: allow setting rc config and filter options as flat parameters
This commit allows global config options to be set as flat parameters on all rc
commands. This makes the rc commands much more like command line parameters and
will aid understanding of how the rc is used.

It is backwards compatible with the old method using _config and _filter.
2026-06-30 17:04:33 +01:00
Nick Craig-Wood 105feca22b fs: fix command line flag being ignored when set to its default value
A backend flag set on the command line to a value that happened to
equal its default was silently ignored, letting the config file win
instead.

For example --sftp-user defaults to the current user, so connecting as
that same user with --sftp-user=USER had no effect and caused rclone
to use the value from the config file.

See: https://forum.rclone.org/t/sftp-user-parsing-as-cli-argument-broken/53955
2026-06-30 14:00:54 +01:00
Nick Craig-Wood 68478f1538 Add user77 to contributors 2026-06-30 14:00:54 +01:00
Gaurav 0d1937f872 docs: fix copy to clipboard functionality for code blocks 2026-06-30 13:44:58 +01:00
user77 0d3c9e929b fs/operations: correct DeleteFile --backup-dir documentation
DeleteFile always passes a nil backupDir to DeleteFileWithBackupDir, so
it never honours --backup-dir. The previous comment said it would move
the file into the backup dir when --backup-dir was in effect, which does
not match the code. Update the comment to state that DeleteFile always
deletes and that callers should use DeleteFileWithBackupDir when
--backup-dir support is required.

Also document on DeleteFileWithBackupDir that the backupDir is found with
BackupDir, which is relatively expensive, so it should be looked up once
outside any delete loop rather than per object.

#7566
2026-06-30 12:29:19 +01:00
Nick Craig-Wood fee55edd15 build: update golang.org/x/image to v0.43.0 to fix image decoding vulnerabilities
govulncheck reported four vulnerabilities in golang.org/x/image v0.41.0,
all reachable via the internxt backend's thumbnail/image upload path:

- CVE-2026-46604 (GO-2026-5066): panic decoding a TIFF image with an
  out-of-bounds strip offset
- CVE-2026-46602 (GO-2026-5062): unbounded memory use from lack of a
  limit on TIFF tile sizes
- CVE-2026-46601 (GO-2026-5061): panic on a WEBP VP8 alpha channel size
  mismatch
- CVE-2026-33813 (GO-2026-4961): panic decoding a large WEBP image on
  32-bit platforms

Updating to v0.43.0 fixes all four.
2026-06-30 10:56:49 +01:00
Nick Craig-Wood fb482756b5 Add Jan Schlien to contributors 2026-06-30 10:56:49 +01:00
Nick Craig-Wood a7fb2367c4 Add happysnaker to contributors 2026-06-30 10:56:49 +01:00
Nick Craig-Wood 4274d8ec64 Add Bryan Stenson to contributors 2026-06-30 10:56:49 +01:00
Nick Craig-Wood e72d4a1609 Add max to contributors 2026-06-30 10:56:49 +01:00
Nick Craig-Wood 4d07ba57e3 Add lewoberst to contributors 2026-06-30 10:56:49 +01:00
happysnaker 8ac978c269 docs: clarify copyto command description - Fixes #9527 2026-06-29 18:19:49 +01:00
Bryan Stenson 6bbc28cf02 docs: fix typo in remote setup docs 2026-06-28 23:53:05 +02:00
max 445528a3fb serve s3: fix spurious 404 on HEAD/GET during VFS writeback - fixes #8188
After an upload (notably multipart) to a slow backing remote, the file
lives in the VFS and is returned by ListBucket, but node.DirEntry() stays
nil until the --vfs-write-back writeback completes. HeadObject and
GetObject returned gofakes3.KeyNotFound while it was nil, so a HEAD/GET in
that window 404'd even though the object existed.

getFileHashByte already falls back to hashing the VFS cache when the
backing object isn't available yet. Drop the early nil return, pass the
node (not the fs.Object) to getFileHashByte, and take the Content-Type
from fs.MimeTypeFromName when the backing object isn't there yet.
2026-06-26 18:38:04 +01:00
lewoberstandJan Schlien c1df24f142 * s3: fix error mapping in GetObject to match HeadObject
The HeadObject path translates a 404 (Not Found) and a 405 (Method Not
Allowed) response into fs.ErrorObjectNotFound, but the GetObject path
returns the raw AWS SDK error instead. With --s3-no-head-object set,
Object.Open() is the first request to hit the source for a GET
operation, so callers that rely on fs.ErrorObjectNotFound see an
opaque "operation error S3: GetObject, ... StatusCode: 404" string
when --s3-no-head-object is set.

Mirror the error handling of HeadObject. The 405 (Method Not Allowed)
case applies to GetObject for the same reason it applies to
HeadObject: A request that specifies the versionId of a delete marker
returns 405 instead of 404 (commit 8470bdf810, AWS S3 delete-marker
docs).

Co-authored-by: Jan Schlien <rclone-github@jan-o-sch.net>
2026-06-26 18:37:02 +01:00
Gaurav 19e7562fbe dropbox: add--dropbox-skip-shared-folders and --dropbox-skip-unowned-folders
This adds two new Dropbox backend flags:

--dropbox-skip-shared-folders skips all shared folder mount points
regardless of ownership.

--dropbox-skip-unowned-folders only skips shared folders that are
not owned by the current user.

These help avoid backing up the same shared folder multiple times when
backing up multiple Dropbox accounts.

Fixes #9514
2026-06-25 11:54:14 +01:00
Nick Craig-Wood 9ab8e4724a filter: fix --files-from copy stopping at the first unreadable file
Before this change, with --files-from and --no-traverse, a single file
that could not be read (for example permission denied) stopped all the
other files in the list being copied.

This happened because the error was returned from the listing, which
caused the whole source listing to be discarded.

This change counts and logs such per-file errors and carries on, so the
readable files are still copied and rclone exits with a non-zero error
code.

Fixes #9115
2026-06-25 10:21:07 +01:00
Nick Craig-Wood 6338aba55a Add yashanil98 to contributors 2026-06-25 10:21:02 +01:00
yashanil98 c7c6646ea3 config: fix root-relative markdown links in interactive config help - fixes #8239
Option help strings are also used to generate the website documentation,
so some contain markdown links with root-relative targets such as
[encoding section in the overview](/overview/#encoding). These render
correctly on rclone.org but are confusing in the interactive config
prompt, where the user sees the raw markdown and the link has no
reachable root.

Rewrite such links to text (https://rclone.org/path) when showing an
option's help in the interactive config. The raw help is left unchanged
so documentation generation is unaffected.
2026-06-25 10:20:14 +01:00
Nick Craig-Wood d1e85a7d9c github: add CODEOWNERS for per-area review routing
Add a .github/CODEOWNERS file mapping backends and core subsystems to
their maintainers so GitHub auto-requests reviews. Move the per-area
responsibility mapping out of MAINTAINERS.md into CODEOWNERS, leaving
MAINTAINERS.md as the maintainer list with a pointer to CODEOWNERS.
2026-06-22 15:18:26 +01:00
Nick Craig-Wood 89c745c0aa docs: Fix RELEASE.md 2026-06-22 15:18:26 +01:00
Nick Craig-Wood d204b29cce ncdu: fix duplicated keystrokes on Windows by pinning tcell to v2.9.0
Since v1.74.0, ncdu doubled every keystroke on Windows: pressing Enter
entered a directory and immediately descended a level deeper, "n" sorted
by name and then reversed the order, and so on. Arrow keys were
unaffected.

The cause is an upstream regression in github.com/gdamore/tcell/v2
pulled in by the v2.9.0 -> v2.13.8 bump in commit 23917555. The same
regression affects lazygit and micro.

There is no fixed tcell release, so pin tcell back to v2.9.0, the last
version before that commit, until the input handling is fixed upstream.

See: https://github.com/gdamore/tcell/issues/1124
See: https://github.com/jesseduffield/lazygit/issues/5344
Fixes #9539
2026-06-22 12:16:27 +01:00
Nick Craig-Wood 99da9d36b9 gui: update embedded release to 1.1.10 2026-06-22 12:04:29 +01:00
Nick Craig-Wood 86d5d8b462 Add Yash Anil to contributors 2026-06-22 12:04:12 +01:00
Yash Anil 59c86b01bb completion: fix powershell completion corrupting non-ASCII names - fixes #9412
The Cobra generated PowerShell completion script captures rclone's output
through a pipeline with Invoke-Expression. PowerShell decodes that output
using [Console]::OutputEncoding, which on non-UTF-8 hosts (for example
PowerShell 5.1 on a Windows install with an OEM code page such as CP852)
misinterprets the UTF-8 bytes rclone emits and corrupts remote and path
names containing non-ASCII characters, so tab completion produces a path
that does not exist.

Inject "[Console]::OutputEncoding = [System.Text.Encoding]::UTF8" into the
generated script immediately before the Invoke-Expression call. This is safe
on PowerShell 7+, where UTF-8 is already the default. If the expected line is
not present (for example after a Cobra template change) the script is emitted
unmodified so we never produce a corrupted completion script.
2026-06-18 13:28:23 +01:00
Nick Craig-Wood 3732e010e2 github: modernise issue and pull request templates
Convert the bug report and feature request templates to GitHub issue forms.

Rewrite the pull request template to require that non-trivial changes are
discussed in an issue first, and to spell out that backend changes need a
clean test_all run and a test account for the integration tester before
they can be merged. Document the latter requirement in CONTRIBUTING.md.

Remove the legacy ISSUE_TEMPLATE.md which is superseded by the forms.
2026-06-18 13:22:19 +01:00
Nick Craig-Wood 34176fb294 Add Yuhang Cao to contributors 2026-06-18 13:22:19 +01:00
Castronaut 056f20800a docs/crypt: fix encrypted size example - Fixes #9202 2026-06-15 12:49:36 +01:00
Yuhang Cao 710514afb3 fs/hash: fix xxh128 hasher size 2026-06-15 08:14:03 +02:00
Nick Craig-Wood 16091ce365 fshttp: add --dump trace to log connection level events via httptrace
The new "trace" dump flag attaches a net/http/httptrace ClientTrace to
each HTTP transaction and logs the connection level events - DNS
resolution, TCP connect, TLS handshake (including the negotiated TLS
version, cipher, ALPN protocol and server certificate), connection
reuse, request write and time to first response byte. Each line is
tagged with the time elapsed since the start of the transaction and the
request pointer so it can be correlated with the other dumps.

This is complementary to the existing dump flags: it shows how the
connection behaved rather than what was sent, which is useful for
debugging connectivity, DNS, TLS, proxy and keep-alive problems.
2026-06-11 21:29:45 +01:00
Nick Craig-Wood 875a666f9c fshttp: add --dump errors to dump only failed HTTP transactions - fixes #9471
The new "errors" dump flag makes the HTTP dump conditional on the
transaction failing with a retryable error (a transport error, HTTP 429
or HTTP 5xx), so first-failure diagnostics can be captured without the
noise of dumping every transaction. The existing dump flags continue to
control what is dumped, for example --dump errors,bodies, and on its own
--dump errors dumps the headers.
2026-06-11 21:29:20 +01:00
Nick Craig-Wood 9c9fbebf7f serve s3: skip TestS3Minio when the docker test framework is unavailable
TestS3Minio brings up a minio container via the fstest/testserver
framework, which exec's bash init.d scripts that shell out to docker.
This is not available on all platforms - Windows has no POSIX shell to
run the scripts, and macOS CI runners have no docker daemon - which
caused the build to fail there.

Add testy.SkipUnlessDocker to detect whether the framework can run and
skip the test when it cannot.
2026-06-11 17:13:11 +01:00
Nick Craig-Wood 3d246a2aea serve s3: stream multipart uploads to the backend instead of buffering in memory
Previously serve s3 buffered every part of a multipart upload in memory
(in the gofakes3 S3 library) and concatenated them when the upload
completed, so memory use grew with the size of the upload.

serve s3 now streams the parts, in part-number order, into a single
PutStream upload to the underlying remote, which performs its own upload
with bounded memory. The whole file is never held in memory - memory use
is bounded by the parts in flight. This works for any remote that
supports PutStream (nearly all, including crypt) and for any part size,
so clients that don't produce uniform-sized parts (for example
PostgreSQL backup tools such as pgBarman and pgBackRest) work too.

Parts must arrive in ascending, contiguous part-number order; parts
uploaded out of order are buffered until their turn, and there is no
per-part retry (a failure aborts the whole upload). These trade-offs are
documented.

Passing --disable-multipart-streaming, or using a remote without
PutStream, reverts to buffering the parts in memory (the previous
behaviour); a one-off NOTICE is logged the first time this happens.

Fixes #7453
2026-06-11 12:30:19 +01:00
Nick Craig-Wood 6267d29b86 servetest: add RunWithBackend so serve tests can pick a non-local backing
Run still uses a fresh local directory as the backing Fs that the
server wraps. RunWithBackend takes an extra remote name (e.g.
"TestS3Minio:") and uses a random subdirectory of that remote instead,
starting the matching fstest/testserver/init.d script on the way in
and tearing it down on the way out.

AuthProxy is only run for the local backend.
2026-06-11 12:30:19 +01:00
Nick Craig-Wood a8c45fd262 docs: drive: note Google verification exemption for personal use apps 2026-06-11 12:30:19 +01:00
Nick Craig-Wood 3286540921 Add Gaurav to contributors 2026-06-11 12:30:19 +01:00
Gaurav e15ce77c8d dropbox: add support for impersonate_admin - fixes #9497 2026-06-10 11:01:31 +01:00
Nick Craig-Wood a37d54b11a rc: document that rc API access is equivalent to shell access 2026-06-09 15:29:40 +01:00
Nick Craig-Wood 64e124f4c5 Add Filippo to contributors 2026-06-09 15:29:40 +01:00
Filippo fa87df9c4d docs: drive: update documentation about "Computers" folder
Added information about an alternative, easier way to access folders inside
"Computers" using rclone. Expanded details about folder behavior in "Computers".
2026-06-09 15:21:47 +01:00
Nick Craig-Wood df9935d71e serve: fix auth proxy using stale config parameters when making a backend
Before this change, if the user changed their password or public-key
and the auth proxy script returned updated config parameters for the
backend (eg a rotated api_key) rclone would continue to re-use the old
backend with the old config parameters out of the fscache.

This was because both the VFS cache and the fs/cache key were derived
from the user name only, so a change in the user's password or
public-key did not invalidate the cached backend.

Fix this by deriving the cache key from the user plus a hash of the
password/public-key, so a credential change forces a fresh backend.
The hash uses a per-process random HMAC key so the fragment that
appears in logs cannot be brute-forced offline.
2026-06-08 16:10:20 +01:00
Nick Craig-Wood a8f102ce8f accounting: fix goroutine leak in ResetCounters
ResetCounters unconditionally restarted the average loop, spawning a
ticker goroutine that pinned the StatsInfo even when no loop had been
running before. statsGroups.delete calls ResetCounters on every removed
group, so deleting N stats groups leaked N goroutines and prevented GC
of the underlying StatsInfo objects.

Only restart the loop if it was active before the reset.
2026-06-08 16:10:20 +01:00
Nick Craig-Wood ae4a054818 docs: fix --windows-event-log-level help 2026-06-08 16:10:20 +01:00
Nick Craig-Wood 4bc5bdab97 Add bright to contributors 2026-06-08 16:10:20 +01:00
Nick Craig-Wood 67f3783215 Add nako-ruru to contributors 2026-06-08 16:10:20 +01:00
Nick Craig-Wood f14f2ca4cf Add IceLocke to contributors 2026-06-08 16:10:20 +01:00
kingston125 e64480f634 backend/filelu: fix recursive listing path handling and file filtering 2026-06-08 11:54:43 +01:00
Jan-Philipp Reßler ec97bb4d7f docs: bisync - clarify flag interaction and minor changes
The behavior of the --track-renames and --max-delete flags combination for
bisync have confused me and some other users. So with this PR i added a
paragraph to clarify this.
2026-06-06 12:17:06 +01:00
Leon Brocard d3530cb317 ncdu/scan: add unit tests for the scan package
Tests cover Dir size/count accounting, AttrI behaviour for files and
directories, error propagation, and Remove correctly updating ancestor
totals up the tree.
2026-06-05 18:13:16 +01:00
Nick Craig-Wood 97aefa49a8 Changelog updates from Version v1.74.3 2026-06-05 17:43:07 +01:00
Nick Craig-Wood 53f972830c rc: stop global.* connection string options changing config CVE-2026-49980
A connection string can carry global.* options which change rclone's
process-wide configuration (e.g. global.http_proxy). This is
undesirable for the rc interface which was designed to have multiple
users or connections at once. The rc interface has the `_config`
mechanism for setting request scoped global config.

This blocks global.* options on all rc paths by marking the context as
a remote control request at the rc boundaries. fs.NewFs then skips
applying global.* to the process-wide config for a marked context.

The marker is reapplied in fs.CopyConfig, which is the call rclone
uses to detach context but keep config.

global.* options still apply to the individual backend they are set
on, exactly like override.* options; they just no longer leak into the
rest of the process. Remotes created directly on the command line are
unaffected as are remotes defined in the config file.

See: GHSA-qw24-gh76-8rvv
2026-06-05 15:21:01 +01:00
Nick Craig-Wood 2326ea79f7 rc: fix unauthenticated command execution via --rc-serve inline remotes CVE-2026-49980
The --rc-serve GET/HEAD file serving path accepted bracketed inline
remotes from the URL and instantiated them, so a single
unauthenticated request could run a command as the rclone user via
backend options such as webdav bearer_token_command or sftp ssh, read
arbitrary local files, or change process-wide config via global.*
options.

This was the GET/HEAD equivalent of the POST hole fixed for
CVE-2026-41179, which only guarded the rc call dispatch path.

Now, unless the rc server has authentication configured or
--rc-no-auth is set, the serve path only allows remotes already
present in the config file: inline remotes, connection string
parameters and bare local paths are rejected. Connection string
global.* options are never honoured on the serve path, even when
authenticated.

See: GHSA-qw24-gh76-8rvv
2026-06-05 15:21:01 +01:00
Nick Craig-Wood 25939a1d00 docs: Update RELEASE.md to use cherry-pick -x 2026-06-05 15:21:01 +01:00
nako-ruruandbright 9693b3df09 webdav: fix mixed property statuses in multi-status responses
This PR fixes a bug in the WebDAV backend where directories or files could
randomly "disappear" from listings due to strict and fragile multi-status code
parsing.

Co-authored-by: bright <nako_ruru@sina.com>
2026-06-03 18:04:15 +01:00
IceLocke e7b1eb774c s3: remove session token on cross-host redirects
Add a redirect policy to the S3 HTTP client so X-Amz-Security-Token is
removed once a redirect chain crosses hosts. Keep stripping it on later
same-host hops in the same chain, since net/http copies headers from the
initial request for each redirect and can otherwise restore the token.

Preserve same-host redirect behavior, retain the standard redirect limit,
and add tests for cross-host, same-host, multi-hop, and redirect-loop cases.
2026-06-03 17:58:26 +01:00
Nick Craig-Wood c96385c280 build: fix multiple CVEs by upgrading to go1.26.4
- CVE-2026-42504: mime: quadratic complexity in WordDecoder.DecodeHeader
- CVE-2026-42507: net/textproto: arbitrary input are included in errors without any escaping
- CVE-2026-27145: crypto/x509: split candidate hostname only once
2026-06-03 12:40:28 +01:00
Nick Craig-Wood ffd36fc2d7 iclouddrive: fix "Index has invalid data" error listing iCloud Photos
Accounts with shared-album zones (named CMM-*) returned these zones in
the photos changes/database listing alongside the real photo
libraries. rclone treated them as libraries and queried their albums,
but these zones have no CPLAlbumByPositionLive index so the query
failed with BAD_REQUEST / "Index has invalid data", aborting the whole
listing.

Only PrimarySync and SharedSync-* zones are photo libraries, so skip
any other zone during discovery.

See: https://forum.rclone.org/t/sync-to-icloud-gives-index-has-invalid-data-errors/53852
2026-06-01 17:11:23 +01:00
Nick Craig-Wood 8b981d8a69 build: cache Go build and module dirs to speed up CI
The build matrix relied on setup-go's built-in cache, which keys only on
go.sum with no job differentiation. All matrix jobs computed the same
cache key and raced to save it; since cache keys are immutable, only the
first job to finish saved its cache. That winner was usually a fast job
whose build cache contained none of the cross-compiled architectures, so
the compile_all and ci_beta steps started from a cold cache on every run.

Disable setup-go's cache and add two explicit actions/cache steps to the
build matrix, the android job and the lint job:

  - the module cache (~/go/pkg/mod) depends only on go.sum, so it is
    shared across all jobs under a single key; it used to be duplicated in
    every job's cache. The downloaded module .zip archives are pruned
    before saving as they are not needed to build from the extracted
    module cache, roughly halving it to ~260 MiB per OS;
  - the build cache (compiled artifacts) is specific to OS, arch and Go
    version, so it is kept per job, keyed on the job name.

This lets the cross-compile steps reuse per-architecture build artifacts
and keeps the total cache within the repository limit.

Measured on CI, comparing a cold-cache run against the following
warm-cache run:

    other_os    23m12s -> 3m35s    (compile_all 14m -> 21s)
    linux       23m13s -> 12m14s   (deploy 11m -> 1m37s,
                                     race test 8m -> 4m45s)

Both jobs now finish well under 15 minutes once the cache is warm.
2026-06-01 17:08:21 +01:00
Nick Craig-Wood c34ed0a9ab log: fix wrong source file:line in JSON logs from release builds
JSON logs reported "source":"slog/logger.go:256" instead of the real
caller. getCaller skips logging-machinery frames by file path, but
release builds use -trimpath which rewrites the standard library slog
frame's path to "log/slog/logger.go" - matching neither the "/log/" nor
the "log.go" check, so it was reported as the source. Also skip frames
whose function belongs to the log/slog package, which is immune to
-trimpath.
2026-06-01 15:49:20 +01:00
Nick Craig-Wood 786e91d6a7 drive: warn when non-exportable Google documents are skipped - #9475
Non-exportable Google documents such as Google My Maps, Sites and
Forms have no export format, so rclone silently leaves them out of all
listings. During a server side move/copy this means they are quietly
left behind.

Emit a one-off notice when such a document is skipped, telling the
user the files are invisible to rclone and pointing them at
--drive-show-all-gdocs to include them in server side copies and
moves.
2026-06-01 15:41:48 +01:00
Nick Craig-Wood e85c2e11db Add Janne Beate Bakeng to contributors 2026-06-01 15:41:48 +01:00
Nick Craig-Wood b6b995525a Add Yakov Till to contributors 2026-06-01 15:41:48 +01:00
Janne Beate BakengandNick Craig-Wood 00bd00d83d mount2: fix empty directory listings on re-read
With cmd/mount2, reading a directory more than once returned the correct
entries on the first read but nothing on subsequent reads. Plain `ls`
triggers this: it does lseek(fd, 0, SEEK_SET) to rewind the directory
before a second getdents.

go-fuse v2.9.0 rewinds a directory stream by calling Seekdir on the
FileSeekdirer interface. dirStream did not implement it, so go-fuse
returned ENOTSUP and produced an empty listing on every read after the
first.

This implements Seekdir on dirStream: a rewind to offset 0 resets the
stream to the start, restoring correct listings on re-read. Non-zero
offsets are uncommon for in-memory listings and still return ENOTSUP,
matching go-fuse's own default. A compile-time interface assertion is
added so signature drift on future go-fuse updates is caught at build
time.

Before: second and subsequent reads of a directory returned no entries.
After: directories list correctly on every read.

See: https://github.com/hanwen/go-fuse/issues/549
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2026-06-01 12:06:03 +01:00
Yakov Till 2dbad62a11 iclouddrive: fix ADP/PCS cookie acquisition for iCloud Drive 2026-05-31 10:14:07 +01:00
Nick Craig-Wood 74436281ed drive: fix server-side move failing on shared drives with duplicate dirs - fixes #9472
When moving a file rclone removed the file from its old parent by
looking the parent ID up from the path using the directory cache. When
the source contained two directories with the same name and path, the
cache could resolve to the wrong duplicate, so the removeParents
request didn't match the file's real parent.

This left the file with both its old and new parent, which fails on
shared drives with:

    A shared drive item must have exactly one parent., teamDrivesParentLimit

This uses the source object's actual parent ID instead when it is
known, falling back to the path lookup only when the object has zero
or multiple parents.
2026-05-30 17:58:49 +01:00
Nick Craig-Wood aec530d1e8 Add Puneet Dixit to contributors 2026-05-30 17:58:49 +01:00
Puneet Dixit 1c92cecaa3 sftp: add --sftp-encoding support 2026-05-30 11:01:20 +01:00
Leon Brocard d2b5ff8384 local: fix getXattr returning empty map instead of nil
At least on my macOS Sequoia 15.7.4, the system automatically adds a
com.apple.provenance xattr to files created by processes. This xattr
lacks the "user." prefix so getXattr filters it out, but the metadata
map was already allocated, resulting in an empty non-nil map being
returned instead of nil.

This caused TestMetadata/Symlink/Xattr and TestMetadata/File/Xattr
to fail because they assert the return value is nil when no user
xattrs are present.

The fix checks if the metadata map is empty after filtering and
returns nil if so.
2026-05-30 10:57:55 +01:00
Nick Craig-Wood 79f42d37ff serve s3: fix multipart ListParts pagination returning wrong part numbers - fixes #9460
Listing the parts of a multipart upload with a MaxParts smaller than
the number of parts returned wrong PartNumber values and duplicate
parts on pages after the first, and the part number marker regressed
instead of advancing, causing clients to loop forever.

This bumps github.com/rclone/gofakes3 to v0.0.6 pick up the fix.
2026-05-29 17:04:01 +01:00
Nick Craig-Wood 387be8d8c0 drime: fix directory rename leaving the renamed folder empty in VFS
Drime's folder rename returns success immediately but the children
listing of the renamed folder briefly returns empty before settling,
which made VFS see an empty directory right after a successful rename
and broke subsequent operations on its contents.

After a successful DirMove, poll the renamed folder's listing until it
shows the pre-rename child count, with a 30s timeout and exponential
backoff. The backend integration test already tolerated this via its
own list-consistency retries; this brings VFS into line.

Fixes #9450
2026-05-29 12:51:26 +01:00
Nick Craig-Wood d742334be5 Add Brian King to contributors 2026-05-29 12:51:26 +01:00
Nick Craig-Wood c262d86ab2 Add Hakan İSMAİL to contributors 2026-05-29 12:51:26 +01:00
Hakan İSMAİL f2832217aa rc: include OAuth authorization URL in rc config/oauthstatus response 2026-05-28 17:51:58 +01:00
Nick Craig-Wood 8a13055688 drime: remove stale mux_status field from Item
The API no longer returns mux_status (it now uses transcode_status and
stream_status). The field was unused, so just remove it.
2026-05-26 18:09:39 +01:00
Nick Craig-Wood b6cfcb97c2 drime: remove duplicate upload_cutoff config option
The upload_cutoff option was registered twice with identical content.
Remove the second copy.
2026-05-26 18:09:28 +01:00
Nick Craig-Wood afb29064e4 drime: fix files being uploaded to the wrong directory
Large files (sent as multipart uploads) were placed in the wrong folder
for two reasons:

- the parent folder was sent as "parent_id", but the API ignores that
  and expects "parentId", so the parent was never honoured
- relativePath was sent as the full path from the drive root, which made
  the server build folders from it and silently drop any "0" path
  segment (e.g. ".../data/0/file" lost the "0")

Send the parent as "parentId" and use just the leaf as relativePath,
matching the working single-part upload. This also lets us remove the
now-unneeded absolute-path resolution code.

Fixes #9392
Co-authored-by: Brian King <BrianDKing@gmail.com>
2026-05-26 18:09:09 +01:00
Nick Craig-Wood 8cb4bb19b0 drime: fix server-side copy and move failing with Cloudflare 520 error
The drime origin returns a malformed response (reported by Cloudflare as
a 520 error) for a literal PUT request to the file-entries update
endpoint, which broke renaming, and so server-side copy and move.

Use a POST with the X-HTTP-Method-Override: PUT header instead - the API
routes this to the same handler and it works reliably.

Also retry Cloudflare 520-524 errors which may occur transiently.
2026-05-26 18:08:42 +01:00
Nick Craig-Wood f71bebab44 serve sftp: use the requested atime when setting file times
When a SETSTAT request set the access and modification times, the
handler passed the modification time for both, discarding the
requested access time. Pass the requested access time through instead.

The VFS currently ignores the atime, but it might use it one day.
2026-05-25 20:43:23 +01:00
Nick Craig-Wood 54cd7d6750 serve sftp: implement statvfs@openssh.com to report disk usage
The statvfs@openssh.com extension was advertised but returned an
unsupported status, so clients couldn't query the amount of free and
used space. Implement it using the VFS Statfs method, which reports the
backend's usage where the backend supports About.
2026-05-25 20:43:23 +01:00
Nick Craig-Wood 90308de5d1 serve sftp: fix truncate request being silently ignored
The SFTP serve handler ignored the size attribute of SETSTAT/FSETSTAT
requests, only acting on the modification time. This meant a client
asking to truncate a file (eg setting the final size of an upload, or
an explicit truncate) had no effect at all.

This respects the size attribute (if present) by truncating the file
to the requested size.
2026-05-25 20:43:22 +01:00
Nick Craig-Wood 4dead760dd serve sftp: fix file corruption when a client resumes an upload
The SFTP serve write handler always opened files with O_TRUNC,
ignoring the flags requested in the SFTP OPEN packet. Some clients
(notably WinSCP's "Process in Background", which resumes an upload on
a second connection) re-open the partially written file without the
truncate flag and continue writing from the offset they had reached,
relying on the existing data being preserved. Forcing O_TRUNC zeroed
that prefix, so the start of the uploaded file ended up as a block of
zero bytes.

This fix respects the requested open flags instead so a resume open
without truncate keeps the already written data intact.

See: https://forum.rclone.org/t/rclone-serve-sftp-winscp-background-mode-uploading-causes-file-corruption/53841
2026-05-25 20:43:22 +01:00
Nick Craig-Wood 6efe605b97 Add Valerij Fredriksen to contributors 2026-05-25 20:43:17 +01:00
Nick Craig-Wood 5794a22983 Add caltechustc to contributors 2026-05-25 20:43:17 +01:00
FTCHD 35b95aa2a3 rc: add config/oauthstop and config/oauthstatus to control oauth listener
rclone starts a http listener if the user is adding a remote that requires
oauth. The problem is that sometimes this breaks or the user stops midway with
no way of us knowing that. Once that happens, clients of the api break. In the
best scenario, new remotes cannot be configured without a restart.

This change adds 2 oauth related endpoints, one to get the status of the listener
and one to stop it. This allows clients to go back to normal without having to
restart rclone.

- config/oauthstop - cancels an in-progress OAuth flow and releases the port.
- config/oauthstatus - reports "running" / "stopped".
2026-05-25 19:59:09 +01:00
FTCHD 605eb30674 rc: respond with 202 if prefer-async header is passed
Make rc respond with a 202 status code (instead of 200) if `Prefer: respond-
async` was passed. Keeps backwards compatibility for current clients while also
allowing the OpenAPI schema & generators to differentiate the responses
properly.
2026-05-25 19:50:41 +01:00
Valerij Fredriksen 675806067a mount2: add --allow-idmap to advertise FUSE_ALLOW_IDMAP
Lets the kernel id-map a mount2 mount into a user namespace
(e.g. Kubernetes pods with hostUsers: false). Off by default;
requires Linux 6.12+ and implies default_permissions.
2026-05-25 17:52:36 +01:00
caltechustc be3bfe2d19 refactor: use slices.Contains to simplify code
Signed-off-by: caltechustc <caltechustc@outlook.com>
2026-05-24 19:57:31 +01:00
Nick Craig-Wood 04d1e2563a serve nfs: allow NFS clients to mount subpaths of the served remote
Previously the Mount RPC ignored the path component of the mount
request, so `server:/sub/dir` and `server:/` both landed at the root
of the served remote. The Mount handler now cleans the requested path
with path.Clean, looks it up in the VFS and serves a billy.Filesystem
rooted at that directory, refusing the mount if the path does not
exist or is not a plain directory.

A pathRewriter cache wraps the inner handle cache so that the same
file always produces the same NFS file handle regardless of which
mount minted it (and stable across server restarts for the disk and
symlink caches). This matches the traditional NFS expectation that a
subpath mount behaves like `cd` into a subtree.

nfsmount gains a --nfs-mount-path flag (default /) so clients can
select a subpath at mount time. This replaces a latent misuse of
--volname as the NFS mount path that was previously masked by the
server ignoring it.

Fixes #9442
2026-05-24 18:09:03 +01:00
Nick Craig-Wood 761157714b Add tomholford to contributors 2026-05-24 18:08:57 +01:00
tomholfordandtomholford f22a1b05a8 protondrive: implement shouldRetry instead of always returning false
shouldRetry was a stub returning false unconditionally, which makes
protondrive the only rclone backend that disables pacer-level retries
entirely. Every other backend at minimum falls back to
fserrors.ShouldRetry(err) so genuine transport-level transients (TCP
resets, brief 5xx) get retried.

- Use errors.As to unwrap proton.APIError instead of string matching
- Retry transient storage block errors (Code=200501)
- Retry server errors (5xx, except 503)
- Skip 429 and 503 (handled by go-proton-api's resty retry layer
  via catchTooManyRequests / catchRetryAfter, which honours Retry-After)
- Fall back to fserrors.ShouldRetry for non-API errors

Co-authored-by: tomholford <tomholford@users.noreply.github.com>
2026-05-24 17:50:01 +01:00
Nick Craig-Wood 7dcbc8b4f3 Changelog updates from Version v1.74.2 2026-05-23 11:34:17 +01:00
Nick Craig-Wood 05e5c05cdb Add Iizuki to contributors 2026-05-23 11:34:17 +01:00
Nick Craig-Wood ad8a108453 lib/http: replace deprecated h2c.NewHandler with http.Server.Protocols
The golang.org/x/net/http2/h2c package was deprecated in v0.54.0 in
favour of setting the http.Server Protocols field to enable unencrypted
HTTP/2.

This replaces the h2c.NewHandler wrapping added in e863f751f with
http.Server.Protocols, which is supported by the standard library
since Go 1.24.

Note that the stdlib only supports HTTP/2 prior-knowledge on cleartext
connections, not HTTP/1.1 Upgrade: h2c negotiation. In practice clients
use prior-knowledge or require TLS, so this should not affect users.
2026-05-22 12:20:24 +01:00
Nick Craig-Wood a03070c960 build: update golang.org/x dependencies to address multiple CVEs
Upgrade to v0.55.0 of golang.org/x/net in order to address:

- CVE-2026-42506: html: incorrect handling of namespaced elements in foreign content
- CVE-2026-39821: idna: failure to reject ASCII-only Punycode-encoded labels
- CVE-2026-42502: html: incorrect handling of HTML elements in foreign content
- CVE-2026-25680: html: denial of service when parsing arbitrary HTML
- CVE-2026-25681: html: incorrect handling of character references in DOCTYPE nodes
- CVE-2026-27136: html: duplicate attributes can cause XSS

Upgrade to version v0.52.0 of golang.org/x/crypto to address:

- CVE-2026-46598: ssh/agent: pathological inputs can lead to client panic
- CVE-2026-46597: ssh: byte arithmetic causes underflow and panic
- CVE-2026-39828: ssh: bypass of certificate restrictions
- CVE-2026-39835: ssh: server panic during CheckHostKey/Authenticate
- CVE-2026-39833: ssh/agent: key constraints not enforced
- CVE-2026-39832: ssh/agent: agent constraints dropped when forwarding keys
- CVE-2026-39827: ssh: memory leak when rejecting channels can lead to DoS
- CVE-2026-39830: ssh: client can cause server deadlock on unexpected responses
- CVE-2026-39829: ssh: pathological RSA/DSA parameters may cause DoS
- CVE-2026-39831: ssh: bypass of FIDO/U2F security keys physical interaction
- CVE-2026-39834: ssh: infinite loop on large channel writes
- CVE-2026-42508: ssh/knownhosts: auth bypass via unenforced @revoked status
- CVE-2026-46595: ssh: VerifiedPublicKeyCallback permissions skip enforcement

Upgrade to version v0.41.0 of golang.org/x/image to address:

- CVE-2026-42500: bmp: panic when reading out of bound palette index
- CVE-2026-33809: tiff: excessive resource consumption in PackBits decompression

Upgrade to version v0.45.0 of golang.org/x/sys to address:

- CVE-2026-39824: windows: integer overflow in NewNTUnicodeString
2026-05-22 11:54:29 +01:00
Iizuki 28d6b0b7b9 docs: s3: update doc link for force_path_style 2026-05-21 10:09:25 +01:00
Nick Craig-Wood cbc64a6e6c s3: add new MEGA S4 endpoints on megas4.com including Asia-Pacific region
MEGA S4 has launched a new dedicated domain megas4.com with three new
endpoints, including their first Asia-Pacific location in Tokyo, plus
new endpoints in Paris and Barcelona. The existing Amsterdam,
Luxembourg, Montreal and Vancouver locations have been renamed onto
the new domain.

All current s4.mega.io endpoints continue to be supported and are
retained as legacy options. New accounts use the megas4.com endpoints.
2026-05-20 12:09:31 +01:00
Nick Craig-Wood d97056cb22 docs: fix heading levels so sections appear in the table of contents
The "Import/Export of google documents" section in drive.md and the
"--b2-versions" examples section in b2.md were both at H4 instead of H3,
which excluded them from the ToC even though they are top-level topics
in their respective sections.
2026-05-20 12:09:31 +01:00
Nick Craig-Wood 3a914d9088 Add KyokoMiki to contributors 2026-05-20 12:09:31 +01:00
Nick Craig-Wood 7d4829bf7d Add TaterLi to contributors 2026-05-20 12:09:31 +01:00
Nick Craig-Wood 08bafc38ba Add William Tange to contributors 2026-05-20 12:09:31 +01:00
Nick Craig-Wood ec011c0658 Add Lucky945H to contributors 2026-05-20 12:09:31 +01:00
Nick Craig-Wood 5c6d133389 Add Sai Asish Y to contributors 2026-05-20 12:09:31 +01:00
Nick Craig-Wood c917ad9d36 Add Tore Anderson to contributors 2026-05-20 12:09:31 +01:00
TaterLiandKyokoMiki 7bd05530eb onedrive: add support for no admin mode
Add tenant_url configuration option for accessing OneDrive/SharePoint
without admin privileges. When tenant_url is set, rclone uses the
SharePoint v2.0 API endpoint instead of the standard Microsoft Graph
API.

Co-Authored-By: KyokoMiki <161209740+kyokomiki@users.noreply.github.com>
2026-05-19 11:21:17 +01:00
William Tange 930a733594 protondrive: fix corrupted on transfer: sha1 hashes differ
Some SHA1 hashes of protondrive files are uppercase, so always
converting to lowercase before comparing seems to be the solution.

Fixes #7345
2026-05-19 11:18:58 +01:00
Lucky945H f7d371fc37 vfs/vfscache: fix silent write failure when mounting with remote:.
When mounting with 'remote:.' (dot notation for current directory),
the '.' is passed through toOSPath() which encodes it to full-width
'.' (U+FF0E) via EncodeDot. This caused the VFS cache data/meta root
to be written to a full-width-dot directory while the local cache
backend pointed to the canonicalized path, resulting in writes being
cached but never uploaded.

Fix by calling clean() on the relativeDirPath before toOSPath(),
so path.Clean() strips the trailing '/.' removing the dot before
encoding.
2026-05-18 18:34:55 +01:00
Leon Brocard 793f2d2f06 docs: fix misspellings
Fix several misspellings in documentation and config comments detected by `misspell`.
2026-05-15 20:12:47 +02:00
Sai Asish Y 75773e4d72 webdav: honour auth_redirect on listAll PROPFIND - fixes #9159 2026-05-14 22:27:57 +01:00
Tore Anderson 7e08899a2c jottacloud: support whitelabel service Phonero Sky
Phonero is a Norwegian mobile provider focusing on the enterprise
market. They provide rebranded Jottacloud service through their «Phonero
Sky» (Norwegian for «Phonero Cloud»).

The Client ID `desktop` used by most of the other whitelabel services
does not work for Phonero Sky. Both `desktop-win` and `desktop-mac`
work, though. Authentication does not work when the `jotta-default`
scope is present, so drop it. This results in a Client ID and scope
setup that is identical with that of Let's Go Cloud (Germany).
2026-05-14 10:53:04 +01:00
FTCHD 14cc32a1e4 docs: add note about using config/update to reconnect an oauth remote 2026-05-13 22:29:37 +01:00
dependabot[bot] 72fcef5183 build: bump github.com/go-git/go-billy/v5 to 5.9.0 to fix CVE-2026-44740 (again)
The previous commit had no content due to copilot deleting it as I was
merging it!

Bumps [github.com/go-git/go-billy/v5](https://github.com/go-git/go-billy) from 5.8.0 to 5.9.0.
- [Release notes](https://github.com/go-git/go-billy/releases)
- [Commits](https://github.com/go-git/go-billy/compare/v5.8.0...v5.9.0)

---
updated-dependencies:
- dependency-name: github.com/go-git/go-billy/v5
  dependency-version: 5.9.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-13 20:01:04 +01:00
Nick Craig-Wood 57939d4157 docs: upgrade protondrive to tier 4 tests pass and community maintains 2026-05-13 20:01:04 +01:00
Nick Craig-Wood 70a0665dc7 Add Alvinwylim to contributors 2026-05-13 20:01:04 +01:00
dependabot[bot] 845c1d5e7b build: bump github.com/go-git/go-billy/v5 to 5.9.0 to fix CVE-2026-44740
Bumps [github.com/go-git/go-billy/v5](https://github.com/go-git/go-billy) from 5.8.0 to 5.9.0.
- [Release notes](https://github.com/go-git/go-billy/releases)
- [Commits](https://github.com/go-git/go-billy/compare/v5.8.0...v5.9.0)

---
updated-dependencies:
- dependency-name: github.com/go-git/go-billy/v5
  dependency-version: 5.9.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-13 19:56:17 +01:00
Alvinwylim e95b64be08 drime: fix 500 errors when listing shared folders
When using the drime backend to access a folder shared from another
Drime account (via root_folder_id config option), listing the folder's
children fails with HTTP 500. This makes cross-account read-only
mounts non-functional.

This fixes the problem by changing listAll to use `folderId` instead
of `parentIds[]`

Fixes #9420
2026-05-13 09:55:06 +01:00
nielash 35752d0079 bisync: fix --conflict-loser pathname with --conflict-resolve newer
Before this change, --conflict-loser pathname assumed --conflict-resolve none,
following the legacy behavior prior to v1.66. This produced unexpected behavior
when used with a different --conflict-resolve option.

This change fixes the issue by ensuring that --conflict-loser pathname looks for
the correct name on the side not being renamed, when only one side should be
renamed.

https://forum.rclone.org/t/bisync-does-not-copy-the-winner-file-to-the-loser-site/53768
2026-05-11 18:34:02 -04:00
Nick Craig-Wood acda43a74f rc: remove duplicate metrics_addr option registration
The metrics_addr option was registered twice: once explicitly and once
implicitly via AddPrefix(libhttp.ConfigInfo, "metrics", ...). Both
pointed at the same MetricsHTTP.ListenAddr field, so options/info
returned a duplicate entry.

Drop the explicit entry and use SetDefault to keep the empty default
(so the metrics server stays off unless configured), matching the
pattern already used for rc_addr.

Fixes #9419
2026-05-11 16:34:45 +01:00
John Volk 4343b80949 drime: fix file doesn't exists error when trying to delete
When using rcat to upload a new version of a file that already existed,
the file upload would succeed. The subsequent deletion of the old file
is attempted after the upload. Drime appears to handle the deletion of
the old file automatically and returns HTTP status code 422, stating
the "The selected entry ids is invalid."

The deletion and the rcat would fail before this change. This is with
file history enabled on my Drime account.

This change detects the error and ignores it since the file has
already been deleted.
2026-05-11 13:04:49 +01:00
Nick Craig-Wood c00756810a http: don't list parent directory when pointing at a single file
When an HTTP URL points to a single file, listing the parent
directory is unnecessary and may fail entirely on servers that
disable directory listings but still serve HEAD/GET on the file.

Remember the file name in the Fs and short-circuit List to return
just that one object.

See: https://forum.rclone.org/t/how-to-combine-on-the-fly-http-archive-remote-to-list-crc32s-in-a-http-hosted-zip/53761
2026-05-11 11:05:22 +01:00
Nick Craig-Wood 0157a1fb1e Changelog updates from Version v1.74.1 2026-05-08 17:15:06 +01:00
Nick Craig-Wood fc50f3e731 build: update golang.org/x/net to v0.53.0 to fix CVE-2026-33814 2026-05-08 10:59:20 +01:00
Nick Craig-Wood 869f777594 build: fix multiple CVEs by upgrading to go1.26.3
- CVE-2026-42501: cmd/go: malicious module proxy can bypass checksum database
- CVE-2026-39825: net/http/httputil: ReverseProxy forwards queries with more than urlmaxqueryparams parameters
- CVE-2026-39836: net: panic in Dial and LookupPort when handling NUL byte on Windows
- CVE-2026-42499: net/mail: quadratic string concatenation in consumePhrase
- CVE-2026-39820: net/mail: quadratic string concatentation in consumeComment
- CVE-2026-39819: cmd/go: "go bug" follows symlinks in predictable temporary filenames
- CVE-2026-39817: cmd/go: "go tool pack" does not sanitize output paths
- CVE-2026-33814: net/http: infinite loop in HTTP/2 transport when given bad SETTINGS_MAX_FRAME_SIZE
- CVE-2026-39826: html/template: escaper bypass leads to XSS
- CVE-2026-33811: net: crash when handling long CNAME response
- CVE-2026-39823: html/template: bypass of meta content URL escaping causes XSS
2026-05-08 10:47:04 +01:00
Nick Craig-Wood acb6155654 test_all: skip bisync normalization tests on Dropbox
The bisync normalization test relies on uploading distinct NFC and NFD
versions of the same filename and on the backend supporting in-place
modtime updates. Dropbox normalizes unicode server-side (NFD -> NFC)
and can't set modtime in place, so the test inevitably takes a
different code path on Dropbox and the log diverges from the golden
output without any functional difference.
2026-05-07 18:08:37 +01:00
Nick Craig-Wood f60213545b sync: fix --fix-case rename on backends that need upload before overwrite
operations.NeedTransfer's equality check may have deleted pair.Dst as
a precursor to re-uploading it if SetModTime returns
ErrorCantSetModTimeWithoutDelete (e.g. Dropbox). If so skip the eager
delete of the destination if --fix-case will rename it to a different
name. The rename itself replaces the destination, and any subsequent
re-upload happens at the correctly-cased path.

See: #8881
2026-05-07 18:08:29 +01:00
Nick Craig-Wood 92058f15c4 Revert "sync: fix --fix-case rename failing on backends that can't update modtime"
This reverts commit de67f29b3f.

This solved the original Dropbox "from_lookup/not_found" failure, but
broke --fix-case on case-sensitive backends that update modtime via a
server-side copy (such as S3 on Cloudflare R2).
2026-05-07 17:48:10 +01:00
Nick Craig-Wood c55634bdf8 drime: fix uploads of 100..200M files
At some point Drime recommended 200M for the upload cutoff for
switching to multipart upload. However uploads have stopped working
using single part upload for 100..200Mish files.

Their docs now recommend 5M as the cutoff for multipart upload so this
changes the default.
2026-05-07 17:38:27 +01:00
Nick Craig-Wood 667903dca0 drime: fix large file uploads landing in drive root instead of configured folder
The /s3/multipart/create and /s3/entries endpoints interpret relativePath
as an absolute path from the drive root, not relative to parent_id. When
root_folder_id was set to a non-root folder, files larger than
upload_cutoff ended up at the user's drive root instead of the configured
folder.

Resolve the absolute path of the Fs root once via GET /folders/{hash}/path
(cached on first OpenChunkWriter call) and use that to build the correct
relativePath.

Fixes #9392
2026-05-07 17:38:27 +01:00
Nick Craig-Wood 15419f90c2 docs: sponsor updates
- Add Data Raven as a silver sponsor
- Add Impossible Cloud as a bronze sponsor
- Shuffle silver sponsors once per page load
- Remove TOC from sponsors page
2026-05-07 17:38:27 +01:00
Nick Craig-Wood 84698c3d28 Add ferrumclaudepilgrim to contributors 2026-05-07 17:38:27 +01:00
ferrumclaudepilgrim 1bbe758bc5 local: add --local-fatal-if-no-space flag - fixes #8011
When enabled, an out-of-space error during a local write returns a
fatal error that aborts the run, instead of being retried.

Without this option, ENOSPC errors are treated as retryable and
rclone may spin through the retry loop many times on a full disk
before giving up. That is fine for transient network errors but
unhelpful when the disk is genuinely full and the operator wants
the run to fail loudly. Default is off so existing behaviour is
unchanged.

Implementation follows the pattern suggested in the issue: a defer
at the top of Update wraps the error with fserrors.FatalError when
the option is on and the error is disk-full. Detection covers both
file.ErrDiskFull from the preallocate path and syscall.ENOSPC from
io.Copy or Close, via a small helper that uses fserrors.IsErrNoSpace.
2026-05-07 10:38:47 +01:00
Leon Brocard 4c8bfb7500 s3: add new Fastly Object Storage regions
Add three new regions and their endpoints for Fastly Object Storage:

- eu-west-1 (Paris)
- us-east-1 (Virginia)
- us-west-1 (Oregon)

These are distinct from the existing us-east, us-west and eu-central
endpoints, which are kept in place.
2026-05-07 10:36:26 +01:00
Nick Craig-Wood 0c8d098b7f cloudinary: fix retrying every error and fix pacer sleep units
shouldRetry treated every non-nil error as retryable, so permanent
failures (auth, 4xx, not-found) burned through the LowLevelRetries
budget instead of returning fast.

This also fixes the pacer sleeps: pacer.MinSleep(1000) and
MaxSleep(10000) are time.Duration values, so they were 1µs and 10µs -
almost certainly intended as 10ms and 2s.
2026-05-06 17:47:53 +01:00
Nick Craig-Wood 895e56e401 test_all: give Cloudinary more list retries to fix flaky tests 2026-05-06 17:47:53 +01:00
Nick Craig-Wood daacfb6035 sync: fix flaky transform tests with retries
The TransformFile tests in fs/sync call operations.TransformFile
immediately after MoveDir. On eventually-consistent backends the
internal NewObject lookup can momentarily fail with "object not
found", making the tests flaky.

This wraps the two operations.TransformFile calls in TestTransformFile
and TestManualTransformFile with fstest.Retry
2026-05-06 17:47:53 +01:00
Nick Craig-Wood 384e905373 fstest: make Retry helper public
Add a public fstest.Retry helper that retries any function with
exponential backoff up to *ListRetries attempts.
2026-05-06 17:47:53 +01:00
Nick Craig-Wood de67f29b3f sync: fix --fix-case rename failing on backends that can't update modtime
When --fix-case was used (e.g. by bisync) on backends that can't set
modification times in place - such as Dropbox - files whose content
matched but whose modtimes differed would fail to rename with a
"from_lookup/not_found" error and abort the operation.

This happened because operations.NeedTransfer was called before the
fix-case rename. NeedTransfer's equality check would delete the
destination as a precursor to re-uploading it (the standard way to
update a modtime on these backends), so by the time the rename ran the
file no longer existed on the remote.

Fix by running the fix-case rename first, so that any subsequent
delete/re-upload happens at the correctly-cased destination path.

See: #8881
2026-05-06 17:47:53 +01:00
Nick Craig-Wood 40642fee01 Add Tim Schumacher to contributors 2026-05-06 17:47:53 +01:00
Nick Craig-Wood fdda89ae63 Add kkocdko to contributors 2026-05-06 17:47:53 +01:00
kkocdkoandTim Schumacher d86b72c405 serve: support custom http response headers
Co-authored-by: Tim Schumacher <tim@tschumacher.net>
2026-05-06 12:41:15 +01:00
Nick Craig-Wood 03b06ac459 ftp: fix flaky UploadTimeout test on slow integration servers
The test set the short idle timeout before creating the test Fs, which
made fs.NewFs fail to read the FTP welcome banner within 1s on slow CI
hosts. Restore the long timeout while NewFs dials the control
connection, then apply the short idle timeout before the upload so the
data connection still exercises the close race that shut_timeout fixes.
2026-05-06 11:40:07 +01:00
Nick Craig-Wood 38926f43be test_all: give Drime more list retries to fix flaky tests 2026-05-06 11:40:07 +01:00
Nick Craig-Wood 3e78426fc9 test_all: remove Webdav Infinite Scale integration tests
These are broken and the submitter has not shown any interest in
fixing them.
2026-05-06 11:40:07 +01:00
Nick Craig-Wood dac3fa851e test_all: remove Seafile V6 tests as they are broken and V6 is 10 years old 2026-05-06 11:40:07 +01:00
Nick Craig-Wood 7b54c7a6e6 Add KTibow to contributors 2026-05-06 11:40:07 +01:00
KTibow 7200e377dd oauthutil: clarify token replacement prompt wording
The previous wording "Already have a token - refresh?" was misleading
because answering yes triggers a full re-authorization flow, not an
OAuth2 refresh token grant. Updated to "Token already configured -
replace it?" to accurately describe what happens.

Also updated the SugarSync backend which has its own copy of the prompt,
and the docs for box, drive, and onedrive that reference it.
2026-05-06 10:51:16 +01:00
Leon Brocard 6f1678419f serve webdav: add gzip compression for compressible responses
Enable on-the-fly response compression for WebDAV when the client sends
Accept-Encoding and the response content type is suitable for
compression.

This adds compression for the WebDAV responses that benefit most in
practice, notably PROPFIND XML responses and text file downloads.
I tested this with Cyberduck, which sends
`Accept-Encoding: gzip,deflate` and accepted the compressed responses.

Range requests are explicitly left uncompressed.

Fixes #5777
2026-05-06 10:43:55 +01:00
Leon Brocard 6e99f8b301 gui: serve static files with gzip/deflate compression
Before this change, the GUI server sent all static files uncompressed,
meaning the browser had to download the full size of every JS, CSS,
and HTML asset.

After this change, the GUI server uses chi's Compress middleware at
level 5, which negotiates gzip or deflate encoding based on the
client's Accept-Encoding header.

This reduces transfer sizes significantly for the web UI assets, for
example assets/index-CvfdU_RR.js is 874 KB uncompressed, and
265 KB compressed.

This is consistent with how rclone serve http, webdav, and restic
already compress their responses.
2026-05-06 10:40:34 +01:00
Nick Craig-Wood 9d4c912e0e s3: fix STS call per request by caching AssumeRole credentials
The stscreds.AssumeRoleProvider from AWS SDK Go v2 does not cache
credentials by itself. The SDK only auto-wraps providers with
aws.CredentialsCache when they are loaded via
config.LoadDefaultConfig; when assigned directly to
aws.Config.Credentials it must be wrapped manually, as documented on
stscreds.NewAssumeRoleProvider.

Without the cache, configurations using role_arn would call AssumeRole
once per S3 request, flooding STS and CloudTrail.

See: https://forum.rclone.org/t/aws-iam-roles-credentials-arent-cached/53732
2026-05-05 15:47:18 +01:00
Nick Craig-Wood 0737599cd4 protondrive: fix segfault when copying files missing revision metadata
When a Proton Drive file has no active revision attributes,
readMetaDataForLink returns a nil FileSystemAttrs and Object.originalSize
is left as nil. Object.Open then dereferenced this nil pointer when
calling fs.FixRangeOption, causing a SIGSEGV during copy.

Use Object.Size() instead, which already implements the correct fallback
to the link size when originalSize is unavailable.

This updates the github.com/rclone/Proton-API-Bridge package to fix a
segfault when reading files with no metadata.

Fixes #9377
Fixes #9117
2026-05-05 15:02:34 +01:00
Nick Craig-Wood 3b2011c7a0 protondrive: route library logging through rclone's logger
Previously all log output produced by Proton-API-Bridge (stdlib log)
and go-proton-api (logrus + resty's logger) bypassed rclone's
logging: it ignored -v / -vv levels and didn't reach --log-file.

Add a small adapter implementing the resty.Logger / bridge Logger
shape that calls fs.Errorf / fs.Logf / fs.Debugf, and pass it via
the new Config.Logger hook. The bridge in turn forwards the same
value to go-proton-api's WithLogger option, so HTTP-layer warnings
and the formerly-hardcoded logrus warnings inside go-proton-api
also surface through rclone's log levels.
2026-05-05 09:43:39 +01:00
Nick Craig-Wood ef26e6d26d protondrive: route HTTP through rclone's transport
The Proton Drive backend constructed the upstream Proton-API-Bridge
without ever passing rclone's HTTP transport. As a result none of
rclone's HTTP flags reached Proton: --dump headers, --dump bodies,
--no-check-certificate, --user-agent, --bind, --ca-cert, --header,
--tpslimit etc. all silently did nothing for this remote, and HTTP
traffic was invisible to -vv.

Pass fshttp.NewTransport(ctx) through the new Config.Transport hook on
the bridge, which forwards it to the updated go-proton-api's
WithTransport option and so to the underlying resty client.
2026-05-05 09:43:39 +01:00
Nick Craig-Wood c0a8b2597d Add Copilot to contributors 2026-05-05 09:43:39 +01:00
Nick Craig-Wood 74d65bf670 Add Sven Rebhan to contributors 2026-05-05 09:43:39 +01:00
Nick Craig-Wood a4e6476642 Add Gustavo V. F. to contributors 2026-05-05 09:43:39 +01:00
Nick Craig-Wood 6c66b4985d Add 王一赫 to contributors 2026-05-05 09:43:39 +01:00
Sven RebhanandNick Craig-Wood b8b3346499 log: fix side effects when importing rclone as a library
Avoid side effects by using own logger instance

- Importing fs/log only sets rclone's private logger via fs.SetLogger,
  so internal rclone logging works from the moment the package is
  imported but the process-wide slog default is left untouched.

- slog.SetDefault and slog.SetLogLoggerLevel move into InitLogging,
  which is called explicitly from the CLI (cmd/cmd.go), the librclone
  wrapper and the integration test framework. So rclone-as-a-program
  keeps capturing log.Print/log.Fatal and slog.Default() output as
  before.

Library consumers that import fs/log without calling InitLogging now
keep their own slog default and can safely route rclone output back
into it via log.Handler.SetOutput without recursing.

Fixes #8907

Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2026-05-04 11:06:30 +01:00
Gustavo V. F. 9f89102a57 bisync: fix retryable without --resync error message when --resync has a critical failure 2026-05-02 16:47:07 +01:00
Leon Brocard 075552367e cmd/serve/s3: return object listings in key order
The S3 ListObjects response from `rclone serve s3` was sorting object
contents by modification time instead of object key. This made the
listing order incompatible with S3 clients which expect lexicographic
key ordering.

In particular, `aws s3 sync` assumes both source and destination
iterators are ordered by key. With the old modtime ordering it could
misidentify files as missing or outdated and re-download objects that
were already up to date.

Change the pager to sort returned objects by key and add a regression
test which uses keys and modtimes arranged so the old behaviour would
fail.

Fixes #9002
2026-05-02 12:28:30 +01:00
Nick Craig-Wood ada5559fe1 Start v1.75.0-DEV development 2026-05-01 17:15:20 +01:00
Nick Craig-Wood aa031c51cc Version v1.74.0 2026-05-01 15:56:56 +01:00
Nick Craig-Wood 45eb93c3af docs: add missing Huawei Drive docs 2026-05-01 14:06:11 +01:00
18899a58f3 Add Huawei Drive support
Add Huawei Drive backend implementation and tests

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2026-05-01 13:41:07 +01:00
Nick Craig-Wood 7c3909589c s3: add Impossible Cloud as a new S3 provider 2026-05-01 12:47:07 +01:00
Nick Craig-Wood d6377a283d build: add make fetch-gui-and-commit to fetch and commit the embedded GUI 2026-05-01 12:46:46 +01:00
Nick Craig-Wood 56b7d7500e gui: embed compressed dist.zip in the binary for smaller, reproducible builds
Previously `make fetch-gui` extracted the GUI release into cmd/gui/dist/
and the unpacked tree was embedded uncompressed via `//go:embed dist`.

This commits and embeds the GUI bundle (dist.zip) and its release tag
(dist.tag) to the repo so:

- the rclone binary is smaller
- `go build` works on a fresh clone without first running fetch-gui
- a given commit pins an exact GUI version

The "Fetch GUI" step was removed from .github/workflows/build.yml.
2026-05-01 12:46:46 +01:00
Nick Craig-Wood 7400a811fd docs: update the GUI docs to reflect the new rclone gui 2026-05-01 12:46:46 +01:00
Nick Craig-Wood bd62357998 Add John Volk to contributors 2026-05-01 12:46:46 +01:00
John Volk 306fb0a304 drime: fix listings of large directories
next_page is not currently being returned on listings which is causing
the rclone listing code to go wrong. This was returned so is likely a
regression in Drime.

This changes the page counter to calculate using current_page and
last_page. last_page on the first page request is just current_page+1.
drime appears to be capping per_page to 200. as more pages are
requested, last_page increments by 1 until current_page = last_page
2026-05-01 12:37:38 +01:00
Nick Craig-Wood 9c67942986 docs: fix iCloud docs after website update (missed in the merge) 2026-04-30 16:45:49 +01:00
Nick Craig-Wood 0812cee9e9 protondrive: fix server-side moveto and DirMove against current API
Bump go-proton-api and Proton-API-Bridge to versions that send the new
NameSignatureEmail field and omit NodePassphraseSignature/SignatureEmail
for ordinary nodes, matching the schema accepted by the Proton Drive
API. Without this rclone moveto, --backup-dir, server-side rename and
DirMove all failed with "value cannot be empty" / "outdated app" 422.

Fixes #8512
2026-04-30 12:40:52 +01:00
Nick Craig-Wood 7447f23d66 build: Update all packages with pseudo versions which aren't v0.0.0
These don't get updated by `go get` and were missed until recently. It
turns out `github.com/jlaffaye/ftp` was the only one that needed
updating.
2026-04-28 09:24:51 +01:00
Nick Craig-Wood 77954e5cd2 Add Chris Coughlan to contributors 2026-04-28 09:23:46 +01:00
Nick Craig-Wood b30cf8fbe7 Add Yakov Till to contributors 2026-04-28 09:23:46 +01:00
Yakov Till d0c469c3c0 iclouddrive: add read only iCloud Photos support and SRP authentication
Add read-only iCloud Photos support to the existing iclouddrive
backend via `service = photos` config option.

Also includes auth improvements on top of #9209's SRP authentication.

**Photos features:**
- 3-level hierarchy: libraries (Personal + Shared Photo Library) →
  albums → photos/videos
- server-side smart albums (All Photos, Videos, Favorites,
  Screenshots, Live, Bursts, Panoramas, Slo-mo, Time-lapse, Portrait,
  Long Exposure, Animated, Hidden, Recently Deleted)
- User-created albums and nested album folders
- Live Photo `.MOV` companions as first-class entries
- Edited photo versions (`-edited` suffix) and RAW alternatives
- Duplicate filename dedup for camera counter wrap collisions
- Parallel cold listing for large albums
- Delta sync via CloudKit `changes/zone` - warm listings near-instant from disk cache
- Disk cache (libraries, albums, photos) with atomic writes for crash safety
- `ChangeNotify` support for FUSE mounts via `changes/zone` polling
- `ListR` support for `--fast-list` and recursive operations
- `--metadata` support - width, height, added-time, favorite, hidden
- Fresh download URLs per file - no stale URL failures on long copies
- FUSE mount documentation with recommended flags

**Auth improvements over #9209:**
- SMS 2FA fallback for users without trusted Apple devices
- Explicit push notification request - fixes iOS/macOS 26.4+ where 409
  no longer auto-pushes
- Thread safety for concurrent FUSE callers (mutexes on session and client state)
- Session endpoint caching - skips ~5s `/validate` round-trip on warm start
- `Disconnect` support - clears auth state + disk cache
- PCS cookie support for Advanced Data Protection accounts, including
  trusted-device approval for PCS cookies

Built on @coughlanio's Photos PoC (Closes #8734) and @mikegillan's SRP auth (#9209).

Fixes #7982
Co-authored-by: Chris Coughlan <chris@coughlan.io>
2026-04-27 16:55:31 +01:00
Nick Craig-Wood 6b67be9d48 mountlib: rc: fix mounts created with mountPoint "*" overwriting each other
On Windows, passing "*" as mountPoint to the mount/mount RC command
auto-assigns a drive letter (e.g. "Z:"), but the resolved letter was
never propagated back to mountlib. This caused liveMounts to be keyed
on the literal "*", breaking tracking of multiple mounts and making
unmount unreliable.

Change MountFn to return the actual mount point as an additional
return value. Update MountPoint.Mount() to store the resolved value,
and mountRc() to use it as the liveMounts key. The mount/mount RC
response now returns the actual mountPoint so callers can discover
which drive letter was assigned.
2026-04-27 15:09:14 +01:00
Nick Craig-Wood d2b8b73ea3 vfs/vfscache/downloaders: kick waiters periodically, not just once
The background kicker goroutine had a bare select outside a for loop,
so the 5s ticker fired at most once before the goroutine exited. The
intent was to run every 5s for the lifetime of the Downloaders.

This wraps the select in a for loop so the ticker fires repeatedly
until ctx is cancelled.

In practice this was benign because every downloader exit and every
successful Write already calls kickWaiters, so the background kicker
is only load-bearing when a waiter is queued, no downloader is
running, and _ensureDownloader failed transiently. In that state,
before this fix, the waiter would hang until another Download() call
or Close() arrived; now it gets retried every 5s and will either
recover or accumulate enough errors to trip maxErrorCount and error
out cleanly.
2026-04-27 15:08:33 +01:00
Nick Craig-Wood 7c56eff1a7 rc: add user directories to core/disks and filter mounts better 2026-04-27 15:07:33 +01:00
Nick Craig-Wood 55da1abb23 docs: notes on how to update pseudo versions 2026-04-27 15:07:13 +01:00
Nick Craig-Wood b8bfcbc0c0 Add dlaumen to contributors 2026-04-27 15:07:13 +01:00
Nick Craig-Wood 96f0979b66 Add Luke Cyca to contributors 2026-04-27 15:07:13 +01:00
Nick Craig-Wood a55ad46201 Add mathieulongtin to contributors 2026-04-27 15:07:13 +01:00
dlaumen a71c6f0459 protondrive: update to latest go-proton-api to use new host 2026-04-27 14:33:47 +01:00
Luke Cyca 96760f1c14 docs: amend Google Drive client_id instructions to include running web-based auth flow 2026-04-24 18:51:28 +01:00
mathieulongtin 8e60792d96 azureblob,azurefile: fix documentation about federated identity 2026-04-24 18:21:22 +01:00
José Zúniga c385d8586a internxt: implement multi-part uploads
Implement multipart upload support with configurable chunk size and concurrency options

Enable OpenChunkWriter with per-chunk encryption

Enhance multipart upload handling with new upload cutoff and error management for small files
2026-04-24 17:20:18 +01:00
Nick Craig-Wood 328ac017c1 serve dlna: remove file extensions from titles to prevent Samsung TV duplication
Samsung TVs have a bug where they duplicate file extensions when both
the title contains an extension and the MIME type indicates the same
file type. For example, "photo.jpg" becomes "photo.jpg.jpg".

Remove extensions from <dc:title> while keeping them in the resource URL
and MIME type. This provides a cleaner display and prevents Samsung TVs
from incorrectly "fixing" what they perceive as missing extensions.
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 8502532c22 serve dlna: fix XML quote escaping for Samsung TV compatibility
Samsung TVs have strict XML parsers that fail to interpret &#34;
(numeric quote entity) correctly within DIDL-Lite metadata, causing
files to appear as empty folders. By replacing &#34; with &quot;
(named quote entity) in all marshaled XML, Samsung TVs can now
properly parse the metadata and display files.

This handles the "Big 5" XML entities that might cause parsing issues:

- &#34; -> &quot; (double quotes)
- &#39; -> &apos; (apostrophes)
- &#38; -> &amp;  (ampersands)
- &#60; -> &lt;   (less than)
- &#62; -> &gt;   (greater than)

While Go's xml.Marshal already uses named entities for &, <, >
characters, this ensures complete protection against any edge cases
where numeric entities might be generated. Samsung TVs are known
to have strict XML parsers that can't handle numeric entities.

Fixes #9346
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 3e9e29ba8f serve dlna: handle empty ObjectID from Samsung TVs
Samsung TVs sometimes send Browse requests with empty ObjectID
parameters (<ObjectID></ObjectID>) which causes DLNA servers to
return errors. Default empty ObjectID to "0" (root container) to
maintain compatibility.

This fix is based on ReadyMedia/MiniDLNA Bug 311 which documented
the same issue and solution for Samsung TVs.

See #9346
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 9cb329809d serve dlna: add Samsung-specific XML namespace
Add xmlns:sec="http://www.sec.co.kr/" namespace to DIDL-Lite responses
as required by Samsung TV DLNA implementations. This namespace is used
by working DLNA servers like MediaBrowser/Emby for Samsung compatibility.

Based on research of open source DLNA servers that successfully work
with Samsung TVs.

See #9346
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 6d0bca0fc8 serve dlna: fix invalid dc:date for containers
Containers (directories) never had their Date field set, producing
<dc:date>0001-01-01</dc:date> (Go's zero time) in DIDL-Lite metadata.
This invalid date can confuse strict DLNA clients.

Set the dc:date to the directory's modification time, and as a safety
net, omit the dc:date element entirely when the timestamp is zero.

See #9346
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 49650db8af serve dlna: fix container childCount to reflect actual contents
The childCount attribute on DLNA containers was hardcoded to 1
regardless of how many items the directory actually contained. Some
DLNA clients (notably Samsung TVs) use childCount to decide whether
to browse into a container. Report the actual number of directory
entries instead.

See #9346
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 9b7f960a24 serve dlna: fix SOAP response argument ordering for Samsung TV compatibility
Samsung TVs are strict DLNA clients that expect SOAP response arguments
in the order defined by the service SCPD (Service Control Protocol
Description). The Browse response was using a Go map which produces
random iteration order, causing arguments like Result, NumberReturned,
TotalMatches, and UpdateID to appear in unpredictable order. Samsung TVs
fail to parse such responses and never proceed to browse directory
children, showing "no content" to the user.

Replace the map[string]string return type with an ordered []soapArg
slice throughout the UPnPService.Handle() interface, ensuring response
arguments always appear in SCPD-defined order.

See #9346
2026-04-24 16:27:09 +01:00
Nick Craig-Wood 18aa4b2f29 Add Anton Bordwine to contributors 2026-04-24 16:27:09 +01:00
Anton Bordwine 8e9ea05a67 listremotes: add --exact flag for filtering - fixes #9076 2026-04-24 16:23:12 +01:00
dependabot[bot] 1f44b2ee93 build: bump github.com/Azure/go-ntlmssp to 0.1.1 to fix CVE-2026-32952
Fix CVE-2026-32952: A malicious NTLM challenge message can causes an slice out
of bounds panic, which can crash any Go process using ntlmssp.Negotiator as an
HTTP transport.

This is in use in rclone in the webdav backend to access sharepoint.

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-24 16:18:05 +01:00
Nick Craig-Wood 90028ab3da azurefiles: fix missing x-ms-file-request-intent header with OAuth - fixes #9367
The fix for #8241 set FileRequestIntent=Backup on
service.ClientOptions so the azfile SDK emits the
x-ms-file-request-intent header was inadvertently dropped in this
commit (released in v1.73.0)

846f193806 azureblob,azurefiles: factor the common auth into a library

This broke azurefiles with OAuth (service principal secret,
certificate, MSI, etc.) with:

    400 MissingRequiredHeader: x-ms-file-request-intent

This restores it in the azurefiles SetClientOptions callback. The SDK
only emits the header for TokenCredential auth, so shared-key and SAS
paths are unaffected.
2026-04-24 15:55:53 +01:00
Nick Craig-Wood d45f66f840 Add tdawe to contributors 2026-04-24 15:55:53 +01:00
Nick Craig-Wood 4942541f02 Add Jan Heylen to contributors 2026-04-24 15:55:53 +01:00
tdawe ae5d388ea3 protondrive: align backend with newer Proton SDK stack
send SDK-era app headers for move and upload compatibility
2026-04-24 14:40:29 +01:00
Jan Heylen a1ad9b3f46 s3: fix bucket creation failing on Ceph/radosgw
Before this change, uploading to an existing bucket on Ceph (radosgw)
could fail with:

    BucketAlreadyExists: 409 Conflict

when rclone attempted to create the destination bucket (which it does
by default unless --s3-no-check-bucket is set).

The Ceph rgw S3 implementation never returns BucketAlreadyOwnedByYou;
it returns BucketAlreadyExists for every CreateBucket on an existing
bucket, even one the caller owns. With the use_already_exists quirk
set to true, rclone wraps BucketAlreadyExists as a non-retriable error
and aborts the transfer.

The Ceph provider used to set useAlreadyExists = true explicitly. When
the s3 providers were converted to YAML files in f28c83c, Ceph did not
set use_already_exists so it picked up the default of true (via
set(&opt.UseAlreadyExists, true, provider.Quirks.UseAlreadyExists)),
which matched the previous behaviour but is the wrong setting for
Ceph.

This sets use_already_exists: false for the Ceph provider so rclone
ignores BucketAlreadyExists on CreateBucket and continues with the
upload.

Side effect: this partially reverts #7871 for the Ceph provider. If a
user tries to create a bucket on Ceph that is actually owned by
someone else, rclone will no longer fail fast at CreateBucket time;
the subsequent object PUT will fail instead. This is unavoidable on
Ceph since the server does not distinguish "already owned by you" from
"owned by someone else".
2026-04-23 19:13:29 +01:00
Nick Craig-Wood 79379faeac rc: add core/disks to enumerate attached disks 2026-04-23 18:47:46 +01:00
Nick Craig-Wood 9dedb12b9d build: update golang.org/x/image/webp to v0.39.0 to fix CVE-2026-33813
Parsing a WEBP image with an invalid, large size panics on 32-bit platforms.

This only affects users on 32 bit platforms using the Internxt backend.

See: https://pkg.go.dev/vuln/GO-2026-4961
2026-04-23 18:46:18 +01:00
Nick Craig-Wood 53a571abf1 Add SyoBoN to contributors 2026-04-23 18:46:18 +01:00
SyoBoN 8688785a7a docs: fix typo 2026-04-21 21:06:49 +02:00
albertony f1a5e44d50 docs: fix code comment regarding cmount tag 2026-04-20 20:09:52 +02:00
Chris 65ef7d8e6c s3: add HCP provider and list_versions_oldest_first quirk
Hitachi Content Platform (HCP) returns object versions in ascending
chronological order (oldest first), unlike the S3 standard which
returns them newest first. This causes --s3-version-at to return the
wrong version when used with HCP.

Add a new list_versions_oldest_first quirk which reverses the Versions
and DeleteMarkers lists before merging, so the existing versionAt
filter works correctly regardless of backend sort order.

Add HCP as a new provider with this quirk enabled by default.

See: https://docs.hitachivantara.com/r/en-us/content-platform/9.6.x/mk-95hcph002/using-the-hitachi-api-for-amazon-s3/working-with-buckets/listing-bucket-contents-version-2
2026-04-20 13:45:18 +01:00
Andrew Gunnerson c744949d91 mega: fix crash when logging in with previous auth keys fails
When Mega.LoginWithKeys() fails to make the API request, it leaves the
object in a state where Mega.FS.root is nil because it could never query
any information about the filesystem tree. An easy way for this to
happen is if the device is not connected to the internet.

Previously, these failures would be ignored, but Fs.findRoot() on the
rclone side is written in a way that assumes the go-meta filesystem will
have a non-nil root. This leads to an immediate nil pointer dereference
when NewFs() calls Fs.findRoot().

This commit fixes the problem by making LoginWithKeys() failures hard
failures, similar to the MultiFactorLogin() path.

Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
2026-04-20 13:37:45 +01:00
Nick Craig-Wood 94e7adaeba pcloud: fix recursive listing from the root - fixes #9315
pCloud now disallow recursive listing from the root so this change
lists the root normally then uses recursive listings for
subdirectories.
2026-04-20 12:16:10 +01:00
Nick Craig-Wood f191448b0d rc: flip auth default so all endpoints require auth unless opted out
Replace AuthRequired bool with NoAuth bool on the rc.Call struct and
flip the auth check logic. Previously endpoints were unauthenticated
by default and had to opt in with AuthRequired: true, which led to
security vulnerabilities when developers forgot to set the flag.

Now all endpoints require authentication by default. Only explicitly
safe read-only endpoints are marked with NoAuth: true:

- rc/noop
- rc/error
- rc/list
- core/version
- core/stats
- core/group-list
- core/transferred
- core/du
- cache/stats
- vfs/list
- vfs/stats
- vfs/queue
- job/status
- job/list

See GHSA-25qr-6mpr-f7qx, GHSA-jfwf-28xr-xw6q
2026-04-19 13:31:27 +01:00
Nick Craig-Wood c5df1ddf4b Changelog updates from Version v1.73.5 2026-04-19 13:30:46 +01:00
Nick Craig-Wood 9e3e68d00c operations: add AuthRequired to operations/fsinfo to prevent backend creation CVE-2026-41179
The operations/fsinfo RC endpoint was registered without AuthRequired,
allowing unauthenticated callers to instantiate arbitrary backends via
inline backend definitions.

See GHSA-jfwf-28xr-xw6q
2026-04-19 13:30:46 +01:00
Nick Craig-Wood 08490972a5 rc: snapshot NoAuth at startup to prevent runtime auth bypass CVE-2026-41176
Snapshot the NoAuth setting when the RC server is created rather than
reading it from the mutable options struct on each request. This
prevents any runtime mutation of rc.NoAuth (e.g. via options/set)
from disabling the auth gate for protected RC methods.

See GHSA-25qr-6mpr-f7qx
2026-04-19 13:30:46 +01:00
Nick Craig-Wood 06aa958ad6 rc: add AuthRequired to options/set to prevent auth bypass CVE-2026-41176
The options/set RC endpoint was registered without AuthRequired,
allowing unauthenticated callers to mutate global runtime options
including rc.NoAuth, which disables the auth gate for all protected
RC methods. Require authentication for options/set.

See GHSA-25qr-6mpr-f7qx
2026-04-19 13:30:46 +01:00
Nick Craig-Wood 67e5f435c6 accounting: fix rcat/copyurl for files.com
The files.com integration tests for rcat/copyurl were failing because
fs/account.Account was declaring a ReadAt method when the underlying
handle did not support it. The files.com SDK decided to use the ReadAt
method to speed transfers up which failed.

ReadAt and Seek methods were added in this commit to support the
archive command:

409dc75328 accounting: add io.Seeker/io.ReaderAt support to accounting.Account

This fixes the problem by adding new methods to the Account object
WithSeeker/WithReaderAt/WithReadAtSeeker which produce an object with
the desired methods or errors if it isn't possible.

This stops Account advertising things it can't do which is bad Go
practice.
2026-04-18 17:48:03 +01:00
Nick Craig-Wood e76a30471a bisync: fix integration tests after sftp log changes
We added a new log message here which we need to ignore in the bisync tests

3658470022 sftp: warn the user if no host key validation is configured
2026-04-18 16:21:43 +01:00
dependabot[bot] 2b570c788c build: bump actions/github-script from 8 to 9
Bumps [actions/github-script](https://github.com/actions/github-script) from 8 to 9.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v8...v9)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '9'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-04-17 18:10:03 +02:00
Nick Craig-Wood 9b889684b0 fstest/test_all: stop test servers on signal, panic, or exit
Until now test_all relied entirely on per-goroutine defer finish()
calls in fstest/runs to stop test servers. A Ctrl-C, kill, or panic
aborted those defers and left docker containers running, breaking the
next run.

Register testserver.CleanupAll with lib/atexit so SIGINT/SIGTERM
delivery runs the sweep automatically. Also defer atexit.Run for the
normal exit and unrecovered-panic paths, and call it explicitly
before os.Exit(1) since os.Exit does not fire defers. The fs.Fatalf
call sites above only fire before any server starts so they need no
explicit sweep.
2026-04-16 17:37:16 +01:00
Nick Craig-Wood 48a6584baf fstest/testserver: add CleanupAll for end-of-run server sweep
Cleanup today is entirely per-goroutine via the stop closure that Start
returns. If the driver process is killed or panics, those deferred
stops never run and the underlying container keeps running.

Track every remote Start has brought up in a process-local map, and
expose CleanupAll which force-stops each tracked remote via the new
run.bash "force-stop" verb. The returned stop closure is now
sync.Once-wrapped so it and CleanupAll can both fire harmlessly. No
callers yet; wired up in fstest/test_all in a follow-up commit.
2026-04-16 17:30:10 +01:00
Nick Craig-Wood 58a51c2ce9 fstest/testserver: add force-stop and reconcile stale refcounts
run.bash holds a persistent refcount file in the shared state directory
so multiple concurrent tests can share a single container. If a prior
test_all run is killed (e.g. Ctrl-C), the count never reaches zero on
the next run and the container is never stopped - forcing manual
cleanup.

Three fixes, all in fstest/testserver/init.d/run.bash:

- On start, if the refcount is non-zero but no container is running,
  treat it as zero. Stops leaking through future runs.
- reset now rm -rfs RUN_ROOT (the per-server state) instead of
  RUN_BASE (the shared parent) which was clobbering sibling services.
- New force-stop verb unconditionally stops the container and zeroes
  the refcount. This is the primitive that the Go-side cleanup sweep
  will call at end-of-run.
2026-04-16 17:29:06 +01:00
Nick Craig-Wood b715ad2a97 fshttp: add --dump curl for dumping HTTP requests as curl commands 2026-04-15 18:22:05 +01:00
Nick Craig-Wood e12c250705 s3: fix empty delimiter parameter rejected by Archiware P5 server
Some S3-compatible servers (e.g. Archiware P5) reject requests with an
empty `?delimiter=` query parameter. For recursive listings, pass `nil`
instead of a pointer to an empty string so the parameter is omitted
entirely from the request.

Fixes #9342
2026-04-15 18:01:14 +01:00
Nick Craig-Wood 513d2fc17e serve nfs: fix EOF flag in READ response not being set when read reaches end of file
This was fixed in the upstream library

See: https://github.com/willscott/go-nfs/pull/149
2026-04-15 17:53:00 +01:00
ZRHan 3be3347e86 webdav: optimize performance by using Depth=0 for metadata requests 2026-04-15 17:40:27 +01:00
Nick Craig-Wood bf55d5e6d3 bisync: fix flaky TestBisyncConcurrent by increasing random name entropy
The temp directory name used random.String(2) giving only 676 possible
values. When multiple concurrent tests started in the same second, they
shared the same timestamp prefix, causing name collisions and shared
temp directories. This led to lock file conflicts, listing file races,
and file deletion errors.

Increase to random.String(8) to make collisions effectively impossible.
2026-04-13 18:21:22 +01:00
Nick Craig-Wood dd5250ca55 azureblob: add --azureblob-decompress flag to download gzip-encoded files
Before this change, if an object compressed with "Content-Encoding:
gzip" was downloaded, a length and hash mismatch would occur since the
go runtime automatically decompressed the object on download.

If --azureblob-decompress is set, this change erases the length and hash on
compressed objects so they can be downloaded successfully, at the cost
of not being able to check the length or the hash of the downloaded
object.

If --azureblob-decompress is not set the compressed files will be downloaded
as-is providing compressed objects with intact size and hash
information.

Fixes #9337
2026-04-13 18:05:52 +01:00
Nick Craig-Wood cf11c8bbd9 docs: serve backend metadata as JSON on the website
Add a Hugo page at /backends/index.json that exports all the
backend YAML data from docs/data/backends/ as a single JSON file
for external consumption.
2026-04-13 16:45:00 +01:00
Nick Craig-Wood bbd7297b33 azureblob/auth: add Microsoft Partner Network User-Agent prefix
Set the User-Agent to include the APN prefix for Azure backends
(azureblob, azurefiles, onelake) to identify rclone as a Microsoft
Partner. The User-Agent is now:

    APN/1.0 rclone/1.0 rclone/<version>
2026-04-13 15:27:20 +01:00
Nick Craig-Wood 7b8994ab32 vfs: add context parameter to New() for config propagation
Add a ctx parameter to vfs.New() so callers can pass in context
carrying ConfigInfo and FilterInfo. The context is stripped of
cancellation but config and filter values are preserved into a fresh
background context.
2026-04-13 12:48:38 +01:00
Nick Craig-Wood 9f2edc3077 vfs: replace context.TODO/Background with stored VFS context
Add a ctx field to the VFS struct, initialized in New() from the
existing cancellable context. Propagate this through the cache
subsystem hierarchy.

This ensures proper context cancellation when a VFS shuts down, rather
than using disconnected context.TODO() or context.Background() calls
throughout and paves the way for VFS to have its own config.
2026-04-13 12:48:38 +01:00
Nick Craig-Wood 771267d713 build: fix make fetch-gui in CI workflow - it was in the wrong place 2026-04-13 12:19:00 +01:00
Nick Craig-Wood 5f791079fc gui: join Wait goroutines on shutdown 2026-04-11 15:27:05 +01:00
Nick Craig-Wood 7b9ac79ab4 gui: remove flag.Lookup test guard around browser open
The flag.Lookup("test.v") check existed to skip opening a browser
during tests, but the tests don't exercise RunE, so this was never
used. The --no-open-browser flag is sufficient on its own.
2026-04-11 15:27:05 +01:00
Nick Craig-Wood 8ddccc1285 gui: drop freePort helper, use libhttp port binding for the RC server
Bind the RC server to localhost:0 and read the bound URL back via a
new rcserver.Server.URLs() accessor instead of pre-allocating a port
in cmd/gui. This removes the small TOCTOU race window between
freePort() closing its listener and rcserver claiming the same port.
2026-04-11 15:27:05 +01:00
Nick Craig-Wood 55afa13921 gui: allow serving from a local zip file or an unpacked directory
This helps with local development and allows users to try older and
newer releases of rclone-web.
2026-04-11 15:27:05 +01:00
Nick Craig-Wood a08b48adaa gui: don't run fetch-gui on make
- Fail gracefully if `make fetch-gui` hasn't been run
- Return errors instead of panic or fatal errrors
- Don't run `make fetch-gui` on every make since we have it in the workflow
2026-04-11 15:27:05 +01:00
Nick Craig-Wood 5337b14739 build: fix GitHub API rate limit errors when fetching GUI dist in CI
The fetch-gui-dist.sh script calls the GitHub releases API
unauthenticated, which is limited to 60 requests/hour per source IP.
GitHub Actions runners share outbound IPs, so this quota is regularly
exhausted.

Pass GITHUB_TOKEN (or GH_TOKEN) as an Authorization header when
present, raising the limit to 1000/hour, and wire secrets.GITHUB_TOKEN
into the workflow step. Local unauthenticated runs still work.
2026-04-11 15:27:05 +01:00
a1pcm 3ad0178b5b drime: fix User.EntryPermissions JSON unmarshalling
`json:"entry_permissions"` is known to be either empty [] or of
structure {string: boolean}. This may have been a breaking API change on
Drime's side. Because EntryPermissions is not used, the type was changed
to `any` to capture both cases, otherwise we could implement custom
unmarshalling for that type.
2026-04-10 20:48:21 +01:00
Enduriel 1a924aa746 filen: make multi-threaded upload chunks individually retryable 2026-04-10 20:46:55 +01:00
Enduriel 33996381df chore: add Enduriel as filen backend maintainer 2026-04-10 18:45:29 +02:00
Nick Craig-Wood 299e7274f8 filter: fix debug logs that fire before logger is configured - fixes #9291
These Debugf calls in NewFilter() ran during GlobalOptionsInit(), before
InitLogging() configured the JSON log format. This caused plain-text
debug lines to leak to stderr when --use-json-log was set, breaking
tooling that expected only JSON output.

The resolved time values are already available via --dump filters so
this commit removes the debug messages.
2026-04-10 13:02:24 +01:00
Nick Craig-Wood c5b4745cb9 Add Mozi to contributors 2026-04-10 13:02:24 +01:00
Nick Craig-Wood 560882de2c Add Brais Couce to contributors 2026-04-10 13:02:24 +01:00
FTCHD acf887b464 gui: new command to launch the https://github.com/rclone/rclone-web/ GUI
This adds a new gui command which runs an embedded copy of the GUI at

https://github.com/rclone/rclone-web/

The GUI release is fetched as part of the CI build.
2026-04-10 11:39:50 +01:00
Mozi 082031cc85 s3: fix TencentCOS CDN endpoint failing on bucket check
The Global Acceleration Endpoint (cos.accelerate.myqcloud.com) of
Tencent COS does not seem to support "CreateBucket" (maybe also other
bucket management operations). Since the acceleration functionality must
be enabled per-bucket in the Tencent Cloud console, the bucket will
always exist before this endpoint is used, so this check can be safely
skipped.

Now, "no_check_bucket = true" will be auto set when using this endpoint.

Why "NewFs()": on-the-fly remotes (connection string remotes), for
example, ":s3,provider=TencentCOS,...:..." will also be fixed.

Why no unit test: I can't find a good way to test "NewFs()" without
leveraging live endpoints. I think we can extract all existing mutations
for different providers (e.g., AWS, Fastly, and Rabata) from "NewFs()"
to a new function in the future.

Some Tencent docs about this CDN endpoint:
- English: Global Acceleration Endpoint | https://www.tencentcloud.com/pt/document/product/436/40700
- Chinese: 对象存储 全球加速概述_腾讯云 | https://cloud.tencent.com/document/product/436/38866

Assisted-By: OpenCode
2026-04-09 17:36:29 +01:00
Chris 40b064993e s3: fix --s3-versions flag ignored by cleanup-hidden when GetBucketVersioning fails
When a user has --s3-versions set but lacks the s3:GetBucketVersioning
permission, GetBucketVersioning returns an error and isVersioned() caches
the result as false. This caused CleanUpHidden (backend cleanup-hidden) to
silently exit with "bucket is not versioned so not removing old versions",
ignoring the user's explicit --s3-versions flag.

Fix this by trusting the explicit --s3-versions flag in purge(), bypassing
the GetBucketVersioning check when the user has explicitly declared the
bucket is versioned.
2026-04-09 17:08:03 +01:00
Brais Couce d15f1142ef iclouddrive: fix 'directory not found' error when the directory contains accent marks 2026-04-09 17:04:20 +01:00
Nick Craig-Wood 0b4c21fbda downloaders: fix flaky TestDownloaders/EnsureDownloader test
Replace racy assert.False/Sleep/assert.True pattern with
assert.Eventually which polls until the async download completes.
2026-04-09 17:00:45 +01:00
Nick Craig-Wood 3658470022 sftp: warn the user if no host key validation is configured
Previously ssh.InsecureIgnoreHostKey() was set unconditionally as the
default HostKeyCallback with no indication to the user.

This logs a warning pointing users to the documentation on how to
enable host key validation.

See: https://github.com/rclone/rclone/security/code-scanning/167
2026-04-09 17:00:45 +01:00
Nick Craig-Wood 9623850979 Add TheBabu to contributors 2026-04-09 17:00:45 +01:00
TheBabu e863f751f0 lib/http: Add HTTP/2 cleartext support in server configuration
Enable HTTP/2 cleartext support for non-TLS listeners.
2026-04-09 17:00:18 +01:00
Nick Craig-Wood a2ce8b04f3 build: add explicit permissions to GitHub Actions workflows
This restricts the GITHUB_TOKEN to least-privilege in all workflows,
fixing CodeQL code-scanning alerts for actions/missing-workflow-permissions.
2026-04-09 14:37:30 +01:00
Nick Craig-Wood bfd650b428 vfscache: fix grace timer reusing stale fd after _checkObject removes cache file
Before this change when a cache item was in its grace period (with
HandleCaching) and the file is reopened, _checkObject runs before the
grace timer recovery check. If the remote object's fingerprint changed
_checkObject removes the cache file from disk. However the grace
recovery path still reused the now-stale fd pointing to a deleted
inode, skipping _createFile entirely. This left no cache file on disk,
causing cache.Exists() to return false and breaking
rename-while-writing logic.

Fix this by checking the cache file still exists before reusing the fd
in grace recovery. If the file was removed, close the stale fd and
downloaders and fall through to _createFile.

Also update the fingerprint in item.rename after setting the new object,
preventing unnecessary cache invalidation when a file is reopened after
a rename.

This was discovered in the integration tests on backends that update
modtime on rename (like mailru).
2026-04-09 12:49:21 +01:00
Nick Craig-Wood 16591fdc21 webdav: Add a section on symlink/junction points in the help
This notes in particular not to use `--links` but to use
`--local-links`.

Fixes #9317
2026-04-09 11:52:54 +01:00
Nick Craig-Wood c1f5df2090 Changelog updates from Version v1.73.4 2026-04-08 16:43:41 +01:00
Nick Craig-Wood 0fa16e8638 build: update all dependencies
This also streamlines the update process in RELEASE.md and the Makefile
2026-04-08 12:08:28 +01:00
Nick Craig-Wood d58ce7b1b7 docs: fix XSS vulnerability in dropdown mobile header
Use textContent instead of innerHTML when inserting the dropdown
label to prevent DOM text being reinterpreted as HTML.

Fixes https://github.com/rclone/rclone/security/code-scanning/176
2026-04-08 09:19:10 +01:00
dependabot[bot] f757e6c0a7 build: fix Denial of Service due to Panic in AWS SDK for Go v2 SDK EventStream Decoder
Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.96.0 to 1.97.3.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/s3/v1.96.0...service/s3/v1.97.3)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/s3
  dependency-version: 1.97.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-04-08 09:03:49 +01:00
Nick Craig-Wood 20eaad4b6d linkbox: fix downloading files by using web API - fixes #8665
The Linkbox open API (/api/open/file_search) no longer returns download
URLs, breaking all downloads. This switches to using the web API
(/api/file/my_file_list/web) which requires email+password authentication
but returns working download URLs.

This will unfortunately require changing your existing rclone config.

- Add email, password, and web_token config options
- Add web API login via /api/user/login_email with token caching and retry
- Create separate CDN HTTP client with HTTP/2 disabled and browser
  User-Agent to avoid CDN fingerprint blocking
- Remove searchOK regex and name-filtering (web API doesn't support it)
2026-04-08 08:49:42 +01:00
Nick Craig-Wood 1ac0c3eecb vfs: fix tests after --vfs-handle-caching
In this commit

0db3e7a2a0 vfs: fix slow nfs serve by adding --vfs-handle-caching

We added --vfs-handle-caching but unfortunately forgot to disable it
for the TestRWCacheUpdate test.
2026-04-07 23:27:25 +01:00
Nick Craig-Wood 561f20f83f Add Suyun to contributors 2026-04-07 23:27:25 +01:00
Suyun e7deebdf40 build: fix loong64 and s390x build 2026-04-06 10:54:17 +01:00
albertony cb9bdf629c jottacloud: add encoding of percent character to default backend encoding
Fixes #9153
2026-04-06 08:28:28 +01:00
albertony 1f3770a57f docs: fix markdown issues in mount docs 2026-04-03 17:30:56 +01:00
Clément Notin 829b06d39b docs: fix header level for metadata option 2026-04-03 14:32:24 +02:00
Nick Craig-Wood 0db3e7a2a0 vfs: fix slow nfs serve by adding --vfs-handle-caching
Add a configurable grace period (default 5s) that delays closing file
handles and downloaders when the last handle closes. If a new handle
opens within the grace period, it reuses the existing resources.

This fixes 40x performance degradation with serve nfs vs serve sftp
caused by go-nfs opening/reading/closing on every NFS READ RPC, which
destroyed read-ahead prefetch before it could accumulate.

The grace period only applies to non-dirty files so that writeback
proceeds immediately on close.

Fixes #9251
2026-04-03 11:10:50 +01:00
Nick Craig-Wood 654df91375 Add Xiangzhe to contributors 2026-04-03 11:10:50 +01:00
Nick Craig-Wood f079361a01 Add Mike GIllan to contributors 2026-04-03 11:10:50 +01:00
Ross Smith II 3677b7ff77 fix(docs): Fix link to not be language specific 2026-04-02 17:57:02 +01:00
Mike GIllan 4a00a4dc4b iclouddrive: lowercase Apple ID for SRP authentication
Apple IDs are case-insensitive, but the SRP proof computation (M1)
hashes the username client-side. The old plaintext signin let the
server normalize the case, but with SRP the client must match.
Lowercase the Apple ID before use so mixed-case IDs authenticate
correctly.

Reported-by: ArturKlauser
2026-04-02 17:52:56 +01:00
Xiangzhe 2610beb18d iclouddrive: use dynamic origin for SRP auth headers
This fixes China mainland iCloud authentication by deriving the Origin
and Referer headers from authEndpoint instead of hardcoding idmsa.apple.com.

Fixes compatibility with PR #8818 (China region support) and PR #9209
(SRP authentication).

Signed-off-by: Xiangzhe <xiangzhedev@gmail.com>
2026-04-02 17:52:56 +01:00
Mike GIllan 35e4f60548 iclouddrive: replace plaintext signin with SRP authentication
Apple has deprecated the legacy /appleauth/auth/signin endpoint and
now blocks it, causing "Invalid Session Token" errors for all users
when their trust token expires. The browser login flow now requires
SRP (Secure Remote Password), a cryptographic handshake that never
transmits the password.

Replace Session.SignIn() with a multi-step SRP-6a flow:
1. authStart - initialize session at /authorize/signin
2. authFederate - submit account name to /federate
3. authSRPInit - exchange client public value for salt/B at /signin/init
4. authSRPComplete - send M1/M2 proofs to /signin/complete

The SRP implementation uses the RFC 5054 2048-bit group with SHA-256
and Apple's NoUserNameInX variant. Password derivation supports both
s2k and s2k_fo protocols via SHA-256 + PBKDF2.

The 2FA and trust token flow is unchanged. Auth headers for all
idmsa.apple.com requests now include X-Apple-Auth-Attributes,
X-Apple-Frame-Id, and use Origin/Referer of https://idmsa.apple.com.

Fixes #8587
2026-04-02 17:52:56 +01:00
Nick Craig-Wood 839a3b92e0 docs: modernize rclone.org site design
- replace Bootstrap/jQuery with purpose-built CSS and JS
- remove backend icons from navbar and content pages
- replace remaining FontAwesome icons with inline SVGs, remove FontAwesome
- modernize CSS styling for menus, typography, cards, tables, and code blocks
- add copy-to-clipboard buttons on code blocks using SVG icon
- move TOC to left sidebar with responsive overlay drawer
- add sticky header, top scrollbar and first column for wide tables
- add left/right arrow buttons to scrollable tables
- hide homepage logo on mobile
- make wide menus with filter for Commands and Storage Systems
- add dark mode support based on browser preference
- fix CSS/JS cache busting to use build time
2026-04-02 16:32:54 +01:00
Nick Craig-Wood 492e0bd616 Add Andriy Senyshyn to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood d4894db765 Add Claude Opus 4.6 to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 8d09f16681 Add jinyu.han to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 550889c096 Add jinkeyuu to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 896961404b Add lif to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 5eabfc74f4 Add BizaNator to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 787d033376 Add Patrick Farrell to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood db258d404a Add Jason to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 3cdf680a15 Add ZRHan to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood bc57110920 Add Andrew Furman to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 133eae4d5b Add Andriy Senyshyn to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 24c40463eb Add Bhagyashreek8 to contributors 2026-04-02 16:32:54 +01:00
jinkeyuuandjinyu.han e9fddaabeb s3: add UCloud Object Storage provider (#9230)
Co-authored-by: jinyu.han <jinyu.han@ucloud.cn>
2026-03-31 11:45:40 +01:00
lif c49015552c bisync: fix handling of unreadable lockfiles - fixes #9290
Lockfiles with invalid JSON content caused bisync to fail permanently
because lockFileIsExpired() logged the decode error but still fell
through to the "valid lock file" path with zero-value TimeExpires.

Now when a JSON decode error is detected:
- If --max-lock is set (< basicallyforever): treat garbled lockfile as
  expired, mark listings failed, and proceed (safe assumption: the
  previous bisync run crashed and left garbage).
- If --max-lock is not set (default): log a clear error telling the
  user the lockfile needs manual inspection, and return false.
2026-03-31 10:56:28 +01:00
BizaNatorandClaude Opus 4.6 ffdd043b95 librclone/ctest: add Windows support and fix memory management
Make ctest build and run on Windows in addition to Linux/macOS:

- Add OS detection in Makefile using ifeq ($(OS),Windows_NT)
- Use .lib extension and .exe suffix on Windows
- Link Windows system libraries (winmm, ws2_32, ole32)
- Remove unused dlfcn.h include that prevented compilation on Windows

Fix memory management to use RcloneFreeString instead of free for
strings returned by RcloneRPC, as documented in the librclone README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 10:51:15 +01:00
Chris 434edba275 s3: fix regression where PutObject fails with non-seekable readers
Commit a3e1312d accidentally replaced io.NopCloser(in) with a bare
io.Reader when assigning req.Body in uploadSinglepartPutObject.

rclone wraps upload readers in an accounting.Account for progress
tracking. When the AWS SDK calls Seek on the body, Account.Seek does
a type assert on the inner reader. With a bare io.Reader the type
is unexpected and causes:

  operation error S3: PutObject, serialization failed: internal error:
  Seek not implemented for io.nopCloser

With io.NopCloser(in) the type assert works correctly for both seekable
and non-seekable readers.

Restore io.NopCloser(in) to wrap the reader correctly in all cases.

Verified by running both before (regression confirmed) and after (fix
confirmed):
  go test ./backend/s3/... ./fs/operations/... -remote TestS3:
2026-03-31 10:22:27 +01:00
Enduriel da7a3d8d64 filen: update SDK version
- increase timeout for http requests
- only use a single URL per request type
2026-03-30 22:56:32 +01:00
dependabot[bot] 2362b4ca50 build(deps): bump golang.org/x/image from 0.36.0 to 0.38.0
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.36.0 to 0.38.0.
- [Commits](https://github.com/golang/image/compare/v0.36.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/image
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-30 22:55:35 +01:00
kapitainsky 5e3d783b3e docs: note macOS 10.15 (Catalina) support with version v1.70.3
due to min golang requirements macOS Catalina (10.15) can not run newer rclone versions
2026-03-28 12:29:53 +00:00
Bjoern Franke 7a63990df2 Add OVHcloud storage classes
Added OVHcloud storage classes according to https://help.ovhcloud.com/csm/en-ie-public-cloud-storage-s3-choosing-right-storage-class?id=kb_article_view&sysparm_article=KB0047293 and https://help.ovhcloud.com/csm/en-ie-public-cloud-storage-s3-choosing-right-storage-class?id=kb_article_view&sysparm_article=KB0047293
2026-03-25 21:48:56 +00:00
Patrick Farrell 7ca667d35d local: remove fadvise calls that cause spinlock contention
Remove the POSIX_FADV_DONTNEED and POSIX_FADV_SEQUENTIAL calls
from the local backend. The DONTNEED calls cause severe spinlock
contention on parallel file systems (and any system with many
concurrent transfers), because each call triggers per-page cache
teardown under a global lock.

Observed on a 256-core system running rclone with 64 parallel
transfers over Lustre: 69% of all CPU cycles were spent in
kernel spinlock contention from the fadvise path, with effective
throughput well below hardware capability.

The kernel's own page reclaim (kswapd) handles eviction more
efficiently from a single context. Since rclone does not always
read files sequentially (e.g. multipart uploads rewind and
re-read blocks), FADV_SEQUENTIAL was also not reliably correct.

This is consistent with the non-Linux behavior (which never
called fadvise) and with restic's decision to remove identical
code (restic/restic#670).

Fixes #7886
2026-03-24 10:17:36 +00:00
Nick Craig-Wood 1e22af8358 Changelog updates from Version v1.73.3 2026-03-23 23:03:59 +00:00
dependabot[bot] 589d45a0bf build(deps): bump github.com/buger/jsonparser from 1.1.1 to 1.1.2
Bumps [github.com/buger/jsonparser](https://github.com/buger/jsonparser) from 1.1.1 to 1.1.2.
- [Release notes](https://github.com/buger/jsonparser/releases)
- [Commits](https://github.com/buger/jsonparser/compare/v1.1.1...v1.1.2)

---
updated-dependencies:
- dependency-name: github.com/buger/jsonparser
  dependency-version: 1.1.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-23 17:05:35 +00:00
albertony d5deec3534 docs/jottacloud: fix broken link 2026-03-20 23:33:27 +01:00
Jason 53c19e04ba docs: clarify Filen password change requires updating both password and API key in rclone config 2026-03-20 15:26:20 +00:00
Jason e7fadc7e3d docs: note that Filen API key changes on password change 2026-03-20 11:08:59 +00:00
dependabot[bot] 7dfc5b3076 build(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.3
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.79.1 to 1.79.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.79.1...v1.79.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.79.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-19 11:02:37 +00:00
ZRHan 523a29a4e9 webdav: request only required properties in listAll to improve performance
This PR optimizes the PROPFIND requests in the webdav backend to only ask for
the specific properties rclone actually needs.

Currently, the generic webdav backend sends an empty XML body during directory
listing (listAll), which causes the server to fall back to allprops by default.
This forces the server to return properties we never use, such as
getcontenttype.

Fetching getcontenttype can be a very expensive operation on the server side.
For instance, in the official golang.org/x/net/webdav library, determining the
content type requires the server to open the file and read the first 500 bytes.

For a directory with 1,300 files in my environment, rclone ls time dropped from
~30s to ~4s (as fast as native ls).

This only applies to the other vendor for backwards compatibility which could be
expanded.
2026-03-16 17:29:45 +00:00
Chris a3e1312d9d s3: fix Content-MD5 for Object Lock uploads and add GCS quirk
AWS S3 requires Content-MD5 for PutObject with Object Lock parameters.
Since rclone passes a non-seekable io.Reader, the SDK cannot compute
checksums automatically. Buffer the body and compute MD5 manually for
singlepart PutObject and presigned request uploads when Object Lock
parameters are set. Multipart uploads are unaffected as Object Lock
headers go on CreateMultipartUpload which has no body.

Add object_lock_supported provider quirk (default true) to allow
skipping Object Lock integration tests on providers with incomplete
S3 API support. Set to false for GCS which uses non-standard
x-goog-bypass-governance-retention header and doesn't implement
PutObjectLegalHold/GetObjectLegalHold.

Add Multipart and Presigned subtests to Object Lock integration tests
to cover all three upload paths.

Fixes #9199
2026-03-14 22:18:43 +00:00
Marco Ferretti e987d4f351 s3: add multi tenant support for Cubbit 2026-03-14 22:15:47 +00:00
Andrew Furman 639bd88959 lib/rest: fix URLPathEscapeAll breaking WebDAV servers (eg nzbdav) with strict path matching
URLPathEscapeAll was only passing [A-Za-z0-9/] through unencoded, causing
it to percent-encode RFC 3986 unreserved characters (-, ., _, ~). Per RFC
3986 §2.3, unreserved characters MUST NOT be percent-encoded, and a URI
that unnecessarily encodes them is not equivalent to one that does not.

Servers that perform strict path matching without normalising
percent-encoded characters will reject the over-encoded form with a 404.

Before: /files/my-report.pdf → /files/my%2Dreport%2Epdf
After:  /files/my-report.pdf → /files/my-report.pdf

Reserved characters (spaces, semicolons, colons, etc.) continue to be
encoded as before.
2026-03-12 21:58:22 +00:00
Andriy SenyshynandAndriy Senyshyn bfa2fbb78e copyurl: fix ignored --upload-headers and --download-headers
Co-authored-by: Andriy Senyshyn <andriy.senyshyn@kiteworks.com>
2026-03-12 17:30:13 +00:00
Bhagyashreek8 69ccbacf30 s3: IBM COS: provide ibm_iam_endpoint as a configurable param for IBM IAM-based auth 2026-03-12 10:04:16 +00:00
Nick Craig-Wood 5d6690eb20 list: fix nil pointer panic in Sorter when temp file creation fails
When extsort.Strings() cannot create temporary files (e.g. due to
apparmor restrictions or permission denied), it returns a nil sorter
with the error on errChan. The code then called Sort() on the nil
sorter, causing a panic.

Check for nil sorter and return the error instead of panicking.

Fixes #9244
2026-03-12 10:00:56 +00:00
Nick Craig-Wood c7b8654b17 docs: update RELEASE procedure to avoid mistakes 2026-03-11 11:26:16 +00:00
Nick Craig-Wood 906b27d8b9 Add Billy Hughes to contributors 2026-03-11 11:26:16 +00:00
Billy Hughes e233f8b0a7 accounting: Add deletedDirs stat to core/stats help output 2026-03-09 18:43:19 +00:00
Jan-Philipp ReßlerandJan-Philipp Reßler 17a547a980 docs: added text to the label showing version-introduced info
Co-authored-by: Jan-Philipp Reßler <xodarap@xodarap.de>
2026-03-09 18:35:20 +00:00
Nick Craig-Wood cdb3ecb0fb Changelog updates from Version v1.73.2 2026-03-06 20:43:59 +00:00
Nick Craig-Wood 2d7f84576a fs/log: fix data race on OutputHandler.format field
The format field was read in Handle() without synchronization while
setFormat() could write it concurrently from InitLogging(). This
caused a data race detected by the race detector, failing
TestListBucketsAuthProxy in cmd/serve/s3.

Fix by protecting all access to format with the existing mutex.
2026-03-06 14:41:23 +00:00
dependabot[bot] ab234dcfd7 build(deps): bump docker/build-push-action from 6 to 7
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-06 09:49:05 +00:00
dependabot[bot] c5bcfbac8f build(deps): bump docker/setup-buildx-action from 3 to 4
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-06 09:48:25 +00:00
dependabot[bot] 88ed8e2b43 build(deps): bump docker/metadata-action from 5 to 6
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](https://github.com/docker/metadata-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-06 09:47:33 +00:00
dependabot[bot] b097454f83 build(deps): bump docker/setup-qemu-action from 3 to 4
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-05 12:36:44 +00:00
dependabot[bot] 2f890114b6 build(deps): bump docker/login-action from 3 to 4
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-05 12:36:13 +00:00
nielash 78a7d9bb8e bisync: update changelog 2026-03-03 16:13:00 -05:00
nielash 72c561d209 bisync: auto-generate rc help docs
This adds a go generate ./cmd/bisync command to autogenerate the bisync rc docs,
including the list of params.
2026-03-03 16:13:00 -05:00
nielash b864c4f9c9 bisync: add more structured info to rc output
This adds a few handy bits of info, like the session name, workdir location, and
listing file paths, to the rc output.
2026-03-03 16:13:00 -05:00
nielash bb78eb8ab2 bisync: add missing rc params - fixes #7799
This adds 11 previously-missing rc params for newer bisync features.

It also makes optional parameters truly optional. (Previously, callers were
required to supply every single one, even if using the default value.)
2026-03-03 16:13:00 -05:00
Nick Craig-Wood e11a066e00 operations: multithread copy: grab memory before making go routines
This reduces the number of go routines which can get out of hand when
using large --transfers and --multi-thread-streams from potentially
--multi-thread-streams * --transfers Go routines to --max-memory /
--multi-thread-chunk-size

It serializes the memory allocator in each transfer which should be
good for performance and reduce lock contention.
2026-03-03 14:01:11 +00:00
Nick Craig-Wood f0dfe9280c b2: add server side copy real time accounting 2026-03-03 14:01:11 +00:00
Nick Craig-Wood b8d2ce8962 s3: add server side copy real time accounting 2026-03-03 14:01:11 +00:00
Nick Craig-Wood f60b09e7ab azureblob: add server side copy real time accounting 2026-03-03 14:01:11 +00:00
Nick Craig-Wood 307f1edaf4 operations: add method to real time account server side copy
Before this change server side copies would show at 0% until they were
done then show at 100%.

With support from the backend, server side copies can now be accounted
in real time. This will only work for backends which have been
modified and themselves get feedback about how copies are going.

If the transfer fails, the bytes accounted will be reversed.
2026-03-03 14:01:11 +00:00
Nick Craig-Wood a7a3bee488 Add Duncan F to contributors 2026-03-03 14:01:11 +00:00
Duncan F 8e24a21ef2 azureblob: add --azureblob-copy-total-concurrency to limit total multipart copy concurrency 2026-03-03 14:01:11 +00:00
Nick Craig-Wood 239dae97db Add razorloves to contributors 2026-03-03 14:01:11 +00:00
razorloves 7056989972 docs: fix new drive flag typo in changelog 2026-03-03 09:53:42 +01:00
Nick Craig-Wood 056adc97d6 build: update to golang.org/x/net v0.51.0 to fix CVE-2026-27141 #9220
See: https://pkg.go.dev/vuln/GO-2026-4559
2026-02-27 11:24:16 +00:00
Nick Craig-Wood c2fd08192b Add Bjoern Franke to contributors 2026-02-27 11:22:12 +00:00
Nick Craig-Wood 9e0054983c Add Brian Bockelman to contributors 2026-02-27 11:22:12 +00:00
Nick Craig-Wood 7c336fdb6f Add Romāns Potašovs to contributors 2026-02-27 11:22:12 +00:00
Nick Craig-Wood 5d07f521dc Add Adam Kasztenny to contributors 2026-02-27 11:22:11 +00:00
Nick Craig-Wood 0d759d1280 Add hxnd to contributors 2026-02-27 11:22:11 +00:00
Nick Craig-Wood 93b1c50c14 Add Bjoern Franke to contributors 2026-02-27 11:22:11 +00:00
Nick Craig-Wood d6819d9d46 Add FTCHD to contributors 2026-02-27 11:22:11 +00:00
dependabot[bot] 65f2f59734 build(deps): bump actions/upload-artifact from 6 to 7
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-27 08:25:05 +00:00
dependabot[bot] ac177f70cb build(deps): bump actions/download-artifact from 7 to 8
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 7 to 8.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-02-27 08:23:32 +00:00
Leon Brocard 9be4fc8c2b serve http: add gzip compression
Add gzip compression for directory listings and text assets served over HTTP.

This reduces the rclone repository file listing from 40 kB to 8 kB and reduces
the rclone MANUAL.txt from 2.7 MB to 700 kB.

This makes listings and assets served across the network load faster.

The compression level of 5 should be a good balance between size and speed.
2026-02-26 17:18:52 +00:00
Brian Bockelman cf240b6c0f webdav: permit redirects on PROPFIND for metadata
The WebDAV implementation already permits redirects on PROPFIND for
listing paths in the `listAll` method but does not permit this for
metadata in `readMetaDataForPath`.  This results in a strange experience
for endpoints that heavily use redirects -

```
rclone lsl endpoint:
```

functions and lists `hello_world.txt` in its output but

```
rclone lsl endpoint:hello_world.txt
```

Fails with a HTTP 307.

The git history for this setting indicates this was done to avoid
an issue where redirects cause a verb change to GET in the Go HTTP
client; it does not appear to be problematic with HTTP 307.

To fix, a new `CheckRedirect` function is added in the `rest` library
to force the client to use the same verb across redirects, forcing this
for the PROPFIND case.
2026-02-26 16:23:52 +00:00
Romāns Potašovs 6159ea9cf5 webdav: add missing headers for CORS
fix #7492
2026-02-26 16:23:07 +00:00
Adam Kasztenny be1fb99aee docs: Document unsupported S3 object keys with double slashes
Fixes #5063 by documenting that S3 object keys containing
consecutive forward slashes (//) are not supported by rclone.

The issue occurs because rclone normalizes paths like "a//b" to "a/b",
causing "object not found" errors when trying to access the original
object. This documentation addition explicitly warns users about this
limitation and provides workarounds.

Changes:
- Added new subsection "Important note about double slashes (//)"
  under "Restricted filename characters" in S3 documentation
- Explains the normalization behavior and its consequences
- Provides clear examples and workarounds

AI Model/Tool Attribution:
- Implemented using opencode AI assistant
- Issue analysis and documentation update performed by AI tools

Resolves: #5063
2026-02-26 16:21:56 +00:00
Prakhar Chhalotre 01095a539b touch: add metadata when using --metadata-set 2026-02-26 16:20:24 +00:00
hxnd 6578f0ad08 s3: ionos: updated regions & endpoints 2026-02-26 12:41:24 +00:00
Bjoern FrankeandBjoern Franke f4ffa54952 s3: scaleway: ONEZONE_IA is available in all zones, GLACIER only in FR-PAR
Co-authored-by: Bjoern Franke <bjo@schafweide.org>
2026-02-26 12:38:22 +00:00
Prakhar Chhalotre abed695720 drive: add integration test for handling folder names with single quotes 2026-02-26 12:37:10 +00:00
FTCHD 77892a8289 http: dark mode for browser 2026-02-26 12:29:40 +00:00
Nick Craig-Wood 9f9569ad6f docs: note that --use-server-modtime only works on some backends
Closes #6977
2026-02-26 12:00:56 +00:00
Nick Craig-Wood 80cdc4dd06 Add a1pcm to contributors 2026-02-26 12:00:56 +00:00
Nick Craig-Wood 216c9de2c4 Add Leon Brocard to contributors 2026-02-26 12:00:56 +00:00
Nick Craig-Wood 7952910e40 Add Dark Dragon to contributors 2026-02-26 12:00:56 +00:00
jzunigax2 7eed0b6825 internxt: fix Entry doesn't belong in directory errors on windows 2026-02-26 11:13:16 +00:00
a1pcm c7da3ab77f drime: fix chunk-uploaded files ignoring workspace ID
When specifying --drime-workspace-id, a file greater than the limit at
which file uploads get chunked would ignore the specified ID and get put
into the default workspace instead.

Completes the fix described in commit 2360e65 by properly closing the
chunkwriter by providing the workspace ID to the Drime API call.
2026-02-26 10:58:13 +00:00
Leon Brocard 879913da41 s3: add new Fastly Object Storage regions
Add AU East 1, EU South 1, JP Central 1, UK East 1, and US Central 1
regions and endpoints for Fastly Object Storage.

Also sort the entries alphabetically.
2026-02-26 09:02:27 +00:00
Dark Dragon 576e6145fc docs: Fix headers hierarchy for mount.md
Similar to "Mounting on macOS", "Mounting on Linux" needs to move one layer down. This also fixes the missing anchor overlay on hover.
2026-02-25 14:47:30 +01:00
Leon Brocard be73a72f93 serve http: add fallback embedded favicon
Browsers make a request to /favicon.ico when visiting pages generated
by the HTTP server.

Previously, if remotes did not have a /favicon.ico then the server
responded with a 404, causing browsers to show a default icon.

This adds a tiny fallback embedded PNG rclone favicon to help users
identify the rclone browser tab.
2026-02-25 12:48:17 +00:00
Leon Brocard 36913cda94 graphics: optimise images losslessly with ImageOptim
All PNG and JPEG images in the graphics/ and docs/static/img/
directories have been run through ImageOptim with lossless compression.
This reduces the checkout size and the size of logo files served,
without any change in visual quality.

```bash
git diff --cached --stat
docs/static/img/ncw-bitcoin-address.png                                  | Bin 1100 -> 404 bytes
docs/static/img/rclone-1200x630.png                                      | Bin 36503 -> 24220 bytes
docs/static/img/rclone-32x32.png                                         | Bin 1106 -> 704 bytes
graphics/cover.jpg                                                       | Bin 9986330 -> 8998666 bytes
graphics/logo/favicon/favicon_16px.png                                   | Bin 559 -> 389 bytes
graphics/logo/favicon/favicon_32px.png                                   | Bin 1106 -> 704 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_color_128px.png      | Bin 10651 -> 6655 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_color_256px.png      | Bin 22388 -> 14378 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_color_32px.png       | Bin 2656 -> 1650 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_color_64px.png       | Bin 5284 -> 3245 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_mono_128px.png       | Bin 9875 -> 5790 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_mono_256px.png       | Bin 21247 -> 10809 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_mono_32px.png        | Bin 2234 -> 1406 bytes
graphics/logo/logo_on_dark/logo_on_dark__horizontal_mono_64px.png        | Bin 4622 -> 2751 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_color_128px.png        | Bin 3946 -> 2444 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_color_256px.png        | Bin 8234 -> 5134 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_color_512px.png        | Bin 17223 -> 10797 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_color_64px.png         | Bin 1979 -> 1271 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_mono_128px.png         | Bin 3490 -> 2104 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_mono_256px.png         | Bin 7372 -> 4630 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_mono_512px.png         | Bin 16451 -> 8358 bytes
graphics/logo/logo_on_dark/logo_on_dark__vertical_mono_64px.png          | Bin 1684 -> 1049 bytes
graphics/logo/logo_on_dark/logo_symbol_on_dark_mono_128px.png            | Bin 3889 -> 2239 bytes
graphics/logo/logo_on_dark/logo_symbol_on_dark_mono_256px.png            | Bin 8843 -> 4917 bytes
graphics/logo/logo_on_dark/logo_symbol_on_dark_mono_512px.png            | Bin 19071 -> 9074 bytes
graphics/logo/logo_on_dark/logo_symbol_on_dark_mono_64px.png             | Bin 1731 -> 1130 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_color_128px.png    | Bin 10621 -> 6860 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_color_256px.png    | Bin 22317 -> 14747 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_color_32px.png     | Bin 2594 -> 1655 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_color_64px.png     | Bin 5184 -> 3316 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_mono_128px.png     | Bin 8839 -> 5907 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_mono_256px.png     | Bin 19305 -> 11010 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_mono_32px.png      | Bin 2092 -> 1422 bytes
graphics/logo/logo_on_light/logo_on_light__horizontal_mono_64px.png      | Bin 4198 -> 2835 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_color_128px.png      | Bin 3856 -> 2446 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_color_256px.png      | Bin 8083 -> 5179 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_color_512px.png      | Bin 16804 -> 11148 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_color_64px.png       | Bin 1932 -> 1258 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_color_800px_2to1.png | Bin 23388 -> 12019 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_mono_128px.png       | Bin 3139 -> 2106 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_mono_256px.png       | Bin 6347 -> 4630 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_mono_512px.png       | Bin 14138 -> 8358 bytes
graphics/logo/logo_on_light/logo_on_light__vertical_mono_64px.png        | Bin 1522 -> 1038 bytes
graphics/logo/logo_on_light/logo_symbol_on_light_mono_128px.png          | Bin 3193 -> 2241 bytes
graphics/logo/logo_on_light/logo_symbol_on_light_mono_256px.png          | Bin 6992 -> 4917 bytes
graphics/logo/logo_on_light/logo_symbol_on_light_mono_512px.png          | Bin 16638 -> 9074 bytes
graphics/logo/logo_on_light/logo_symbol_on_light_mono_64px.png           | Bin 1568 -> 1127 bytes
graphics/logo/logo_symbol/logo_symbol_color_128px.png                    | Bin 4468 -> 2783 bytes
graphics/logo/logo_symbol/logo_symbol_color_16px.png                     | Bin 1787 -> 391 bytes
graphics/logo/logo_symbol/logo_symbol_color_256px.png                    | Bin 9362 -> 5835 bytes
graphics/logo/logo_symbol/logo_symbol_color_512px.png                    | Bin 19898 -> 12946 bytes
graphics/logo/logo_symbol/logo_symbol_color_600px_spaced.png             | Bin 18775 -> 13537 bytes
graphics/logo/logo_symbol/logo_symbol_color_64px.png                     | Bin 2235 -> 1380 bytes
```
2026-02-25 12:45:49 +00:00
Nick Craig-Wood 699e4cfb23 docs: update sponsors 2026-02-23 13:05:33 +00:00
Nick Craig-Wood bc5cad8792 Add Jan-Philipp Reßler to contributors 2026-02-23 13:05:25 +00:00
Nick Craig-Wood 9024962fc4 Add Chris to contributors 2026-02-23 13:05:25 +00:00
Nick Craig-Wood aa1f4ace64 Add Shlomi Avihou to contributors 2026-02-23 13:05:25 +00:00
Nick Craig-Wood dd1d750c55 Add Jan-Philipp Reßler to contributors 2026-02-23 13:05:25 +00:00
Nick Craig-Wood 5a2564c6e2 Add Varun Chawla to contributors 2026-02-23 13:05:24 +00:00
Nick Craig-Wood 82cc80cc6f Add Prakhar Chhalotre to contributors 2026-02-23 13:05:24 +00:00
Chris 7d0a8bf850 s3: add Object Lock support
Add support for S3 Object Lock with the following new options:

- --s3-object-lock-mode: set retention mode (GOVERNANCE/COMPLIANCE/copy)
- --s3-object-lock-retain-until-date: set retention date (RFC3339/duration/copy)
- --s3-object-lock-legal-hold-status: set legal hold (ON/OFF/copy)
- --s3-bypass-governance-retention: bypass GOVERNANCE lock on delete
- --s3-bucket-object-lock-enabled: enable Object Lock on bucket creation
- --s3-object-lock-set-after-upload: apply lock via separate API calls

The special value "copy" preserves the source object's setting when used
with --metadata flag, enabling scenarios like cloning objects from
COMPLIANCE to GOVERNANCE mode while preserving the original retention date.

Includes integration tests that create a temporary Object Lock bucket covering:
- Retention Mode and Date
- Legal Hold
- Apply settings after upload
- Override protections using bypass-governance flag
The tests are gracefully skipped on providers that do not support Object Lock.

Fixes #4683
Closes #7894 #7893 #8866
2026-02-20 16:40:24 +00:00
Varun Chawla fd8b28d36d webdav: escape reserved characters in URL path segments
Use URLPathEscapeAll instead of URLPathEscape for path encoding.

URLPathEscape relies on Go's url.URL.String() which only minimally
escapes paths - reserved sub-delimiter characters like semicolons and
equals signs pass through unescaped. Per RFC 3986 section 3.3, these
characters must be percent-encoded when used as literal values in
path segments.

Some WebDAV servers (notably dCache/Jetty) interpret unescaped
semicolons as path parameter delimiters, which truncates filenames
at the semicolon position. URLPathEscapeAll encodes everything
except [A-Za-z0-9/], which is safe for all servers.

Fixes #9082
2026-02-20 16:30:15 +00:00
Shlomi Avihou c63ecace41 s3: add Zadara Object Storage provider 2026-02-20 16:27:17 +00:00
Jan-Philipp ReßlerandJan-Philipp Reßler 0c8c3d8fb9 bisync: add group Sync to the bisync command
Co-authored-by: Jan-Philipp Reßler <xodarap@xodarap.de>
2026-02-20 16:21:48 +00:00
Varun Chawla 5042f360f0 archive: extract: strip "./" prefix from tar entry paths
Tar files created from the current directory (e.g. tar -czf archive.tar.gz .)
produce entries prefixed with "./". When extracting, rclone's character
encoding replaces the "." with a full-width dot (U+FF0E), creating a
spurious directory instead of merging into the destination root.

Strip the leading "./" from NameInArchive before processing. Only "./"
is stripped specifically to avoid enabling path traversal attacks via
"../".

Fixes #9168
2026-02-20 11:46:53 +00:00
Prakhar Chhalotre 9601dbce87 accounting: update String method output format for clarity in transfer rate representation - fixes #9129 2026-02-18 15:19:47 +00:00
Nick Craig-Wood e06f0b0595 docs: add instructions on how to update Go version 2026-02-18 12:38:50 +00:00
Nick Craig-Wood b2866f0291 build: modernize Go code with go fix for go1.25 2026-02-18 12:11:52 +00:00
Nick Craig-Wood cf97f250df build: update all dependencies
Could not update github.com/coreos/go-systemd as the new version doesn't build under freebsd

See: https://github.com/coreos/go-systemd/issues/509
2026-02-18 11:33:48 +00:00
Nick Craig-Wood 627b763d4b lib/rest: remove go1.24 workaround now go1.25 is the minimum 2026-02-18 11:33:48 +00:00
Nick Craig-Wood f14945f9c1 build: update to go1.26 and make go1.25 the minimum required version 2026-02-18 11:33:48 +00:00
Nick Craig-Wood 391661fdb4 Add Jack Kelly to contributors 2026-02-18 11:33:48 +00:00
Nick Craig-Wood faffd0a6f1 Changelog updates from Version v1.73.1 2026-02-17 18:22:21 +00:00
Nick Craig-Wood 6cc3356f8e build: fix build using go 1.26.0 instead of go 1.25.7
In the actions config use Go ~1.25.7 to pin the go version to 1.25.x,
x >= 7.

Before this it was choosing Go 1.26.0 which isn't what we want.
2026-02-17 17:05:45 +00:00
Nick Craig-Wood 07e76419c9 fs/march: fix runtime: program exceeds 10000-thread limit
Before this change when doing a sync with `--no-traverse` and
`--files-from` we could call `NewObject` a total of `--checkers` *
`--checkers` times simultaneously.

With `--checkers 128` this can exceed the 10,000 thread limit and
fails when run on a local to local transfer because `NewObject` calls
`lstat` which is a syscall which needs an OS thread of its own.

This patch uses a weighted semaphore to limit the number of
simultaneous calls to `NewObject` to `--checkers` instead which won't
blow the 10,000 thread limit and is far more sensible use of OS
resources.

Fixes #9073
2026-02-17 12:27:17 +00:00
Nick Craig-Wood 60c4f35b56 accounting: fix missing server side stats from core/stats rc
These stats weren't being updated in the global stats read by rc
core/stats:

- transferQueue
- deletesSize
- serverSideCopies
- serverSideCopyBytes
- serverSideMoves
- serverSideMoveBytes
2026-02-17 12:27:17 +00:00
Nick Craig-Wood 15a9c0fd36 pacer: re-read the sleep time as it may be stale
Before this change we read sleepTime before acquiring the pacer token
and uses that possibly stale value to schedule the token return. When
many goroutines enter while sleepTime is high (e.g., 10s), each
goroutine caches this 10s value. Even if successful calls rapidly
decay the pacer state to 0, the queued goroutines still schedule 10s
token returns, so the queue drains at 1 req/10s for the entire herd.
This can create multi‑minute delays even after the pacer has dropped
to 0.

After this change we refresh the sleep time after getting the token.

This problem was introduced by the desire to skip reading the pacer
token entirely when sleepTime is 0 in high performance backends (eg
s3, azure blob).
2026-02-17 12:27:17 +00:00
Nick Craig-Wood 8b85ffbf03 pacer: fix deadlock between pacer token and --max-connections
It was possible in the presence of --max-connections and recursive
calls to the pacer to deadlock it leaving all connections waiting on
either a max connection token or a pacer token.

This fixes the problem by making sure we return the pacer token on
schedule if we take it.

This also short circuits the pacer token if sleepTime is 0.
2026-02-17 12:27:17 +00:00
Nick Craig-Wood 26fb659fe4 test_all: increase retries for Internxt eventual consistency 2026-02-17 12:27:17 +00:00
Nick Craig-Wood 7aa3d8a32f build: fix CVE-2025-68121 by updating go to 1.25.7 or later - fixes #9167 2026-02-17 12:27:17 +00:00
Nick Craig-Wood b7ebec865b drime: fix files and directories being created in the default workspace
Before this change directories and files were created in the default
workspace, not the workspace specified by --drime-workspace-id.
2026-02-17 12:27:17 +00:00
Nick Craig-Wood a60d09c43d docs: update sponsors 2026-02-17 12:27:04 +00:00
Nick Craig-Wood 14a47937c0 Add kingston125 to contributors 2026-02-17 12:22:11 +00:00
Jack Kelly 64d6916161 copyurl: Extend copyurl docs with an example of CSV FILENAMEs starting with a path. 2026-02-17 11:46:37 +00:00
kingston125 ae778f1413 filelu: migrate API calls to lib/rest 2026-02-16 17:45:42 +00:00
José Zúniga 33859568d6 internxt: implement re-login under refresh logic, improve retry logic - fixes #9174 2026-02-13 19:18:51 +00:00
Nick Craig-Wood 4b3aa5aea0 docs: add ExchangeRate-API as a sponsor 2026-02-12 14:08:20 +00:00
Nick Craig-Wood 349487bb7f Add Cohinem to contributors 2026-02-12 14:08:20 +00:00
Nick Craig-Wood b70b2fff16 Add Leon Brocard to contributors 2026-02-12 14:08:20 +00:00
Leon Brocard 32307e9226 s3: remove StackPath Object Storage provider
StackPath's object storage service no longer exists and all S3
endpoints are no longer operational.

Before this change, users could select StackPath as an S3 provider
during configuration, but connections would fail as the endpoints no
longer respond and the service has been discontinued.

After this change, StackPath is removed from the list of supported
S3 providers, preventing users from attempting to configure a
non-functional service.

Fixes #9148
2026-02-11 23:12:35 +01:00
Cohinem 2bd6630c2e drime: implement About 2026-02-11 14:06:31 +00:00
albertony 54c2078f25 build: bump github.com/go-chi/chi/v5 from 5.2.3 to 5.2.5 to fix GO-2026-4316 2026-02-11 11:34:29 +01:00
kingston125 4f284614a4 Set list_version to 2 for FileLu S3 configuration 2026-02-11 11:32:11 +01:00
kingston125 eef0b39a2c filelu: add multipart upload support with configurable cutoff 2026-02-05 13:09:16 +00:00
kingston125 37f6336636 filelu: add multipart init response type 2026-02-05 13:09:16 +00:00
kingston125 1049f88a1d filelu: add comment for response body wrapping 2026-02-05 13:09:16 +00:00
kingston125 327ca25a4d filelu: avoid buffering entire file in memory
Avoid buffering the entire file in memory during download, especially
for large files.
2026-02-05 13:09:16 +00:00
Nick Craig-Wood 673e24a60f docs: update sponsor logos 2026-02-05 12:27:51 +00:00
Leon Brocard 43db4c5dc7 s3: add Fastly Object Storage provider
- Add new Fastly provider with US East, US West, and EU Central regions
- Add `etag_is_not_md5` quirk for providers with mandatory encryption
- Disable server-side copy for Fastly (not supported)
2026-02-05 12:10:53 +00:00
Enduriel 88b484722a filen: fix potential panic in case of error during upload 2026-02-05 12:08:20 +00:00
Enduriel ed5bd327c0 filen: fix 32 bit targets not being able to list directories Fixes #9142
or do pretty much anything,
this was caused by timestamps not being read to 64 bit integers
2026-02-05 12:06:20 +00:00
wiserain 341ce61a2a pikpak: support custom filenames for addurl backend command - fixes #9111
Extended the addurl backend command to accept an optional filename parameter, 
enabling customized naming for downloaded files.
2026-02-03 08:13:03 +09:00
Nick Craig-Wood 9abf9d38c0 Start v1.74.0-DEV development 2026-01-30 22:19:04 +00:00
Nick Craig-Wood f42aa5a586 Version v1.73.0 2026-01-30 20:53:56 +00:00
Nick Craig-Wood 40f2e847a3 drive: fix crash when trying to creating shortcut to a Google doc
Before this change, we uses an unguarded type assertion which failed
when NewObject returned a google document instead of an Object.

This change uses the ID() method to read the id which works for all
types of file instead and returns a sensible error if one isn't found.

Fixes #8990
2026-01-30 18:17:56 +00:00
Nick Craig-Wood 846f193806 azureblob,azurefiles: factor the common auth into a library 2026-01-30 17:50:18 +00:00
Nick Craig-Wood 116d0f3708 test: allow backends to return fs.ErrorCantListRoot to skip Root tests 2026-01-30 17:50:18 +00:00
Nick Craig-Wood 140ec62683 build: add privatebeta Makefile target 2026-01-30 17:50:18 +00:00
Nick Craig-Wood 8c746f36dd docs: add Internxt as a sponsor 2026-01-30 17:38:13 +00:00
Nick Craig-Wood fac7ba4bd0 internxt: remove use of CVE laden github.com/disintegration/imaging 2026-01-30 17:28:52 +00:00
Nick Craig-Wood e3be333c29 docs: fix Internxt docs after merge 2026-01-30 17:08:44 +00:00
Nick Craig-Wood 67013ccf35 docs: update making a new backend docs 2026-01-30 17:08:44 +00:00
Nick Craig-Wood a1522468db docs: build overview page from the backend data 2026-01-30 17:08:44 +00:00
Nick Craig-Wood 58f30ab538 docs: add tiering to the documentation - fixes #8873 2026-01-30 17:08:44 +00:00
Nick Craig-Wood 7539b2c5e1 docs: add data about each backend in YAML format 2026-01-30 17:08:44 +00:00
Nick Craig-Wood 6d39c63983 docs: add bin/manage_backends.py for managing the backend data files 2026-01-30 16:54:28 +00:00
Nick Craig-Wood 1bf73d862a internxt: use rclone's http.Client to enable more features
e.g. --bwlimit, HTTP proxies, connection timeouts, --dump headers/bodies etc
2026-01-30 16:33:03 +00:00
Nick Craig-Wood e2a079d73f internxt: fix lint problems 2026-01-30 16:06:32 +00:00
Nick Craig-Wood 7d7f82854f Add StarHack to contributors 2026-01-30 16:06:32 +00:00
Nick Craig-Wood b6113a505f Add lullius to contributors 2026-01-30 16:06:32 +00:00
Nick Craig-Wood 2a754ef2fa Add jzunigax2 to contributors 2026-01-30 16:06:32 +00:00
85d13c4e34 internxt: add Internxt backend - fixes #7610
Co-Authored-By: lullius <lullius@users.noreply.github.com>
Co-Authored-By: StarHack <StarHack@users.noreply.github.com>
2026-01-30 16:02:50 +00:00
Nick Craig-Wood 49b4ca3412 drive: add --drive-metadata-force-expansive-access flag - Fixes #8980
This allows the permissions changes happening in Feb 2026 to be tested
in advance.
2026-01-30 12:38:59 +00:00
Nick Craig-Wood 315a148820 test_all: allow drime more time to complete 2026-01-30 11:51:56 +00:00
Nick Craig-Wood b873f838db onedrive: fix permissions on onedrive Personal 2026-01-29 21:31:51 +00:00
Nick Craig-Wood 1d6f86900b onedrive: fix require sign in for Onedrive Personal
require_sign_in and regain_inherited_permissions in Onedrive Personal
have been brought into line with Onedrive Business.
2026-01-29 21:31:51 +00:00
Nick Craig-Wood 8a31921fb0 onedrive: Onedrive Personal no longer supports description
Uploading files with description set is no longer allowed with
Onedrive personal. This brings it into line with Onedrive business.
2026-01-29 21:31:51 +00:00
Nick Craig-Wood c64161822d onedrive: fix setting modification time on directories for onedrive Personal
Onedrive Personal started ignoring the time parameters given to create
directory in the same way as onedrive Business.
2026-01-29 21:31:51 +00:00
Nick Craig-Wood d660960775 onedrive: fix cancelling multipart upload
Before this change it was giving back "401 Unauthorized" however
removing the "Authorization: XXXX" from the request fixes the problem
as the auth is in the URL already.
2026-01-29 21:31:51 +00:00
Duncan Smart 7615636093 docs: fix WinFsp link in mount documentation 2026-01-29 10:27:24 +00:00
Nick Craig-Wood cc9abcad06 cmount: make work under OpenBSD - fixes #1727 2026-01-29 10:24:33 +00:00
Nick Craig-Wood 5c4af237e4 vfs: make mount tests run on OpenBSD 2026-01-29 10:24:33 +00:00
albertony 90a57da9d5 docs: improve alignment of icons 2026-01-29 10:22:25 +00:00
Nick Craig-Wood 824257583c protondrive: update to use forks of upstream modules
This updates rclone to use forks of the upstream proton drive modules
in preparation for making changes.

The go-proton-api modules has had changes from master merged so rclone
and Proton-API-Bridge are using the same version.
2026-01-28 11:38:38 +00:00
Nick Craig-Wood cd857754c7 Add hyusap to contributors 2026-01-28 11:38:31 +00:00
Nick Craig-Wood 017d930dfc Add Nick Owens to contributors 2026-01-28 11:38:31 +00:00
Nick Craig-Wood 6ed68883fe Add Mikel Olasagasti Uranga to contributors 2026-01-28 11:38:31 +00:00
hyusap 2a5e0b28c9 docs: fix googlephotos custom client_id instructions
The Google Photos custom client_id documentation was incomplete - it
only mentioned changing the scopes but did not mention that you need
to enable the "Photos Library API" instead of the "Google Drive API"
in step 3 of the referenced Google Drive instructions.

This fixes the documentation to clearly state both differences from
the Google Drive setup:
- Enable "Photos Library API" instead of "Google Drive API"
- Use the Google Photos specific scopes

Co-authored-by: hyusap <paulayush@gmail.com>
2026-01-27 13:48:12 +00:00
Nick Owens 264e75d892 cmount: fix OpenBSD mount support.
this pulls in https://github.com/winfsp/cgofuse/pull/100 to fix OpenBSD
mount support.

part of #1727
2026-01-27 13:47:00 +00:00
Mikel Olasagasti Uranga 4553c3de7b fs: fix bwlimit: correctly report minutes
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
2026-01-27 11:44:47 +00:00
Mikel Olasagasti Uranga 2e9e4a47a2 fs: fix bwlimit: use %d instead of %q for ints
Signed-off-by: Mikel Olasagasti Uranga <mikel@olasagasti.info>
2026-01-27 11:44:47 +00:00
necaran e44829a448 mega: reverts TLS workaround
The workaround to include the obsolete cipher suite is no longer needed because storage endpoints now support TLS1.3.
2026-01-24 12:35:02 +00:00
Nick Craig-Wood 6b9f77459a docs: fix formatting 2026-01-23 16:55:52 +00:00
Nick Craig-Wood 4b2c39c585 docs: add faq entry about re-enabling old TLS ciphers 2026-01-23 15:40:06 +00:00
Nick Craig-Wood dbf2499d85 Add Marc-Philip to contributors 2026-01-23 15:40:06 +00:00
Nick Craig-Wood 760a134c95 Add yy to contributors 2026-01-23 15:40:06 +00:00
Enduriel 63cfe260a2 filen: swap to blake3 hashes 2026-01-22 00:11:46 +00:00
Marc-Philip 3550275cd3 docs: fix echo command syntax for password input 2026-01-18 20:56:26 +01:00
yy b728929f44 docs: fix typos in comments and messages 2026-01-18 20:48:33 +01:00
Nick Craig-Wood 9ec918f137 docs: fix use of removed rem macro 2026-01-14 18:37:19 +00:00
Nick Craig-Wood 3a9c7ceeb1 uptobox: remove backend as service is no longer available
Uptobox was shutdown in September 2023 and does not appear to be
returning.
2026-01-14 15:31:46 +00:00
Nick Craig-Wood 5502c0f8ae rc: add operations/hashsumfile to sum a single file only 2026-01-14 12:29:48 +00:00
Nick Craig-Wood d707ae7cf4 docs: update sponsor link 2026-01-14 12:29:48 +00:00
Enduriel 9bef7f0dbf filen: add Filen backend - Fixes #6728 2026-01-13 12:50:27 +00:00
Nick Craig-Wood 933bbf3ac8 sftp: fix proxy initialisation
This was being done in NewFs instead of NewFsWithConnection like it
should have been which meant calls to NewFsWithConnection were not
initialising the proxy correctly.
2026-01-13 12:40:17 +00:00
Nick Craig-Wood ecc5972d6f fstest: skip Copy mutation test with --sftp-copy-is-hardlink 2026-01-13 12:40:17 +00:00
Nick Craig-Wood 07805796ab fstest: Make Copy mutation test work properly
Before this change it could miss a mutation if the Modtime was cached
2026-01-13 12:40:17 +00:00
Nick Craig-Wood 189e6dbf6a Add Qingwei Li to contributors 2026-01-13 12:40:12 +00:00
Nick Craig-Wood d47e289165 Add Nicolas Dessart to contributors 2026-01-13 12:40:12 +00:00
dougal e51a0599a0 log: fix systemd adding extra newline - fixes #9086
This was broken in v1.71.0 as a typo.
2026-01-09 16:30:01 +00:00
Qingwei Li 530a901de3 oracleobjectstorage, sftp: eliminate unnecessary heap allocation
Move the declaration location of variables to eliminate heap
allocation which may make rclone faster and reduce memory usage slightly.

Fixes #9078
2026-01-09 16:10:02 +00:00
Nicolas Dessart a64a8aad0e sftp,ftp: add http proxy authentication support
This change supports the `http://user:pass@host:port` syntax for the
http_proxy setting.
2026-01-08 16:31:11 +00:00
dougalandNick Craig-Wood 6529d2cd8f Add Drime backend
Co-Authored-By: Nick Craig-Wood <nick@craig-wood.com>
2026-01-08 12:05:37 +00:00
Nick Craig-Wood d9895fef9d lib/rest: add opts.MultipartContentType to explicitly set Content-Type of attachements
Before this the standard library set it to application/octet-stream for some reason
2026-01-08 12:05:37 +00:00
dougal 8c7b7ac891 dircache: allow empty string as root parent id
This was causing an internal error with the drime backend which has the
root parent id as an empty string. This shouldn't affect anything else.
2026-01-08 12:05:37 +00:00
Nick Craig-Wood f814498561 docs: update sponsors 2026-01-08 12:05:30 +00:00
vupn0712andsys6101 5f4e4b1a20 s3: add provider Bizfly Cloud Simple Storage
Co-authored-by: sys6101 <csvmen@gmail.com>
2026-01-06 14:56:49 +00:00
Nick Craig-Wood 28c187b9b4 docs: update sponsor logos 2025-12-31 17:04:11 +00:00
Nick Craig-Wood e07afc4645 Add sys6101 to contributors 2025-12-31 17:04:11 +00:00
Nick Craig-Wood 08932ab92a Add darkdragon-001 to contributors 2025-12-31 17:04:11 +00:00
Nick Craig-Wood 356ee57edb Add vupn0712 to contributors 2025-12-31 17:04:11 +00:00
yuval-cloudinary 7c1660214d docs: add cloudinary to readme 2025-12-22 22:39:53 +01:00
darkdragon-001 51b197c86f docs: fix headers hierarchy in mount docs 2025-12-21 12:23:59 +01:00
vupn0712andsys6101 029ffd2761 s3: fix Copy ignoring storage class
Co-authored-by: sys6101 <csvmen@gmail.com>
2025-12-20 14:42:00 +00:00
Nick Craig-Wood f81cd7d279 serve s3: make errors in --s3-auth-key fatal - fixes #9044
Previously if auth keys were provided without a comma then rclone
would only log an INFO message which could mean it went on to serve
without any auth.

The parsing for environment variables was changed in v1.70.0 to make
them work properly with multiple inputs. This means the input is
treated like a mini CSV file which works well except in this case when
the input has commas. This meant `user,auth` without quotes is treated
as two key pairs `user` and `quote`. The correct syntax is
`"user,auth"`. This updates the documentation accordingly.
2025-12-18 10:17:41 +00:00
Nick Craig-Wood 1a0a4628d7 Add masrlinu to contributors 2025-12-18 10:17:41 +00:00
masrlinuandmasrlinu c10a4d465c pcloud: add support for real-time updates in mount
Co-authored-by: masrlinu <5259918+masrlinu@users.noreply.github.com>
2025-12-17 15:13:25 +00:00
Nick Craig-Wood 3a6e07a613 memory: add --memory-discard flag for speed testing - fixes #9037 2025-12-17 10:21:12 +00:00
Nick Craig-Wood c36f99d343 Add vyv03354 to contributors 2025-12-17 10:21:12 +00:00
jhasse-shade 3e21a7261b shade: Fix VFS test issues 2025-12-16 17:21:22 +00:00
vyv03354 fd439fab62 docs: mention use of ListR feature in ls docs 2025-12-15 09:11:00 +01:00
dependabot[bot] 976aa6b416 build: bump actions/download-artifact from 6 to 7
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 6 to 7.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-13 11:01:27 +01:00
dependabot[bot] b3a0383ca3 build: bump actions/upload-artifact from 5 to 6
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 5 to 6.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-13 11:00:59 +01:00
dependabot[bot] c13f129339 build: bump actions/cache from 4 to 5
Bumps [actions/cache](https://github.com/actions/cache) from 4 to 5.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-12-12 14:52:57 +01:00
vyv03354 748d8c8957 docs: reflects the fact that pCloud supports ListR 2025-12-11 20:32:53 +01:00
jbagwell-akamai 4d379efcbb S3: Linode: updated endpoints to use ISO 3166-1 alpha-2 standard
ISO 3166-1 alpha-2 standard for countries and region short name in parentheses instead of separated by another comma
2025-12-11 17:20:34 +00:00
dougal e5e6a4b5ae sync: fix error propagation in tests (#9025)
This commit fixes the sync transform test IO errors by resetting the
error flag which stops subsequent tests failing.
2025-12-10 15:43:22 +00:00
Nick Craig-Wood df18e8c55b Changelog updates from Version v1.72.1 2025-12-10 15:31:48 +00:00
Nick Craig-Wood f4e17d8b0b s3: add more regions for Selectel 2025-12-10 15:31:48 +00:00
Nick Craig-Wood e5c69511bc Add jhasse-shade to contributors 2025-12-10 15:31:48 +00:00
jhasse-shade 175d4bc553 Add Shade backend 2025-12-09 17:08:57 +00:00
Nick Craig-Wood 4851f1796c log: fix backtrace not going to the --log-file #9014
Before the log re-organisation in:

8d353039a6 log: add log rotation to --log-file

rclone would write any backtraces to the --log-file which was very
convenient for users.

This got accidentally disabled due to a typo which meant backtraces
started going to stderr even if --log-file was supplied.

This fixes the problem.
2025-12-09 16:35:07 +00:00
Nick Craig-Wood 4ff8899b2c build: fix lint warning after linter upgrade 2025-12-09 16:15:17 +00:00
Nick Craig-Wood 8f29a0b0a1 Add Jonas Tingeborn to contributors 2025-12-09 16:15:17 +00:00
Nick Craig-Wood 8b0e76e53b Add Tingsong Xu to contributors 2025-12-09 16:15:17 +00:00
Jonas Tingeborn 233fef5c4d configfile: add piped config support - fixes #9012 2025-12-08 18:42:17 +00:00
Tingsong Xu b9586c3e03 fs/log: fix PID not included in JSON log output
When using `--log-format pid,json`, the PID was not being added to the JSON log output. This fix adds PID support to JSON logging.
2025-12-08 18:41:58 +00:00
Nick Craig-Wood 0dc0ab1330 build: adjust lint rules to exclude new errors from linter update 2025-12-08 14:45:06 +00:00
Nick Craig-Wood a6bbdb35a0 proxy: fix error handling in tests spotted by the linter 2025-12-08 14:45:06 +00:00
Nick Craig-Wood b33cb77b6c Add Johannes Rothe to contributors 2025-12-08 14:45:06 +00:00
Nick Craig-Wood d51322bb5f Add Leo to contributors 2025-12-08 14:45:06 +00:00
Nick Craig-Wood e718ab6091 Add Vladislav Tropnikov to contributors 2025-12-08 14:45:06 +00:00
Nick Craig-Wood 0a9e6e130f Add Cliff Frey to contributors 2025-12-08 14:45:06 +00:00
Nick Craig-Wood 3358b9049c Add vicerace to contributors 2025-12-08 14:45:06 +00:00
DianaNites 847734d421 b2: Fix listing root buckets with unrestricted API key
Fixes previous pull request #8978

An oversight meant that unrestricted API keys
never called b2_list_buckets,
meaning the root remote could not be listed.

The call is now made in the event there are no allowed buckets,
indicating an unrestricted API key

Fixes #9007
2025-12-04 15:55:17 +00:00
Johannes Rothe f7b255d4ec googlecloudstorage: improve endpoint parameter docs
When specifying a custom endpoint with a subpath, there is a limitation
in the Google cloud storage integration that the subpath is ignored
during upload operations. For example with the custom endpoint
"example.org/custom/endpoint" on upload the /custom/endpoint is not
reflected.

As this is most likely an issue with the underlying API client, there is
no way to fix this in rclone. By extending the documentation at least
rclone users are made aware of this limitation.

Related forum thread: https://forum.rclone.org/t/googlecloudstorage-custom-endpoint-subpath-removed-for-upload/53059
2025-12-01 19:04:02 +00:00
Leo 24c752ed9e serve webdav: implement download-directory-as-zip
Signed-off-by: Leo <i@hardrain980.com>
2025-12-01 15:42:16 +00:00
Vladislav Tropnikov a99d155fd4 s3: The ability to specify an IAM role for cross-account interaction 2025-11-29 13:53:00 +00:00
Cliff Frey f72b32b470 azureblob: add metadata and tags support across upload and copy paths
This change adds first-class metadata support to the Azure Blob backend,
including headers, user metadata, tags, and modtime overrides, and wires
it through uploads and server-side copies.

There is a behavior change in that rclone will now set the "mtime"
custom metadata when doing server side copies to azure and the
`--metadata` argument is given.

- Map standard headers: cache-control, content-disposition, content-encoding,
  content-language, content-type to corresponding x-ms-blob-* HTTP headers.
- Map user metadata: any non-reserved keys (excluding x-ms-*) are sent as
  blob user metadata. Keys are normalized to lowercase for consistency.
- Support tags: parse `x-ms-tags` as a comma-separated list of key=value
  pairs and apply them on uploads and copies.
- Support mtime override: accept `mtime` in metadata (RFC3339/RFC3339Nano)
  to override the stored modtime persisted in user metadata.
2025-11-27 16:58:07 +00:00
vicerace 9be7f99bf8 refactor: use strings.Cut to simplify code
Signed-off-by: vicerace <vicerace@sohu.com>
2025-11-27 14:42:11 +00:00
Nick Craig-Wood 6858bf242e docs: note where a provider has an S3 compatible alternative 2025-11-26 12:22:48 +00:00
Nick Craig-Wood e8c6867e4c Add Shade as sponsor 2025-11-26 12:22:48 +00:00
Nick Craig-Wood 50fbd6b049 Add Duncan Smart to contributors 2025-11-26 12:22:48 +00:00
Nick Craig-Wood 0783cab952 Add Diana to contributors 2025-11-26 12:22:48 +00:00
Duncan Smart 886ac7af1d docs: Clarify OAuth scopes for readonly Google Drive access 2025-11-24 15:58:53 +00:00
Diana 3c40238f02 b2: support authentication with new bucket restricted application keys
Backblaze has updated its b2_authorize_account API endpoint, newly created
application keys are now "multi-bucket" keys, capable of being limited to
multiple buckets. These keys can only be used with the v4 endpoint, not v1 which
returns an HTTP 400.

This commit switches authorization to the v4 endpoint, and allowing such keys to
work with any of the allowed buckets.

With multi-bucket keys, missing restricted buckets can be non-fatal.

Supports listing root with multi-bucket API keys
2025-11-24 15:46:41 +00:00
Nick Craig-Wood 46ca0dd7fe docs: update sponsor logos 2025-11-24 14:58:33 +00:00
Nick Craig-Wood 2e968e7ce0 docs: fix lint error in changelog 2025-11-21 18:23:16 +00:00
Nick Craig-Wood 1886c552db Start v1.73.0-DEV development 2025-11-21 18:23:07 +00:00
Nick Craig-Wood 38ab3dd5b1 Version v1.72.0 2025-11-21 17:10:17 +00:00
Nick Craig-Wood 1d02e1219a rc: fix formatting in job/batch 2025-11-21 17:06:18 +00:00
Nick Craig-Wood 035d3f344c test speed: fix formatting of help 2025-11-21 17:02:45 +00:00
Nick Craig-Wood 7d45aee70f docs: update sponsor logos 2025-11-21 12:48:29 +00:00
dependabot[bot] f30789180d build: bump actions/checkout from 5 to 6
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 6.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-20 23:16:24 +00:00
Sean Turner 7cb05a84e9 s3: add multi-part-upload support for If-Match and If-None-Match
#8947 implemented support for the If-Match and If-None-Match headers for S3 PUT
Object requests; however, this support did not extend to multi-part copy and
upload requests. These headers are implemented via inclusion in the
CompleteMultipartUpload request.

This updates the auto generated code also which was needed for multipart copy.
2025-11-20 17:31:15 +00:00
Nick Craig-Wood 6d4c625bfb rc: config/unlock: rename parameter to configPassword accept old as well
We accidentally added a non `camelCase` parameter to the rc
(`config_password`)- this fixes it (to `configPassword`) but accepts
the old name too as it has been in a release.
2025-11-20 16:46:01 +00:00
Nick Craig-Wood 4eccc40168 rc: correct names of parameters in job/list output
These were accidentally committed as snake_case whereas we use
camelCase elsewhere.

This corrects the issue before the first release in v1.72.0
2025-11-20 16:46:01 +00:00
Nick Craig-Wood e451f9c999 Add Nikolay Kiryanov to contributors 2025-11-20 16:46:01 +00:00
Nikolay Kiryanov 321488441e rc: add executeId to job statuses - fixes #8972 2025-11-20 13:15:22 +00:00
dependabot[bot] bd99e05ff0 build: bump golang.org/x/crypto from 0.43.0 to 0.45.0 to fix CVE-2025-58181
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-20 13:09:29 +00:00
hunshcn 6440052fbd s3: fix single file copying behavior with low permission - Fixes #8975 2025-11-18 17:01:07 +00:00
Nick Craig-Wood 4afb59bc93 docs: onedrive: note how to backup up any user's data 2025-11-18 16:21:06 +00:00
Nick Craig-Wood 0343670375 Add Dominik Sander to contributors 2025-11-18 16:21:06 +00:00
Nick Craig-Wood 5b2b372ba9 Add jijamik to contributors 2025-11-18 16:21:06 +00:00
Dominik Sander 08c35ae741 box: allow to configure with config file contents
Especially when using rclone via rc it is helpful to configure the box
backend using the contents of the config file instead of heaving to
upload the file to the server that is running rclone.
2025-11-18 16:09:06 +00:00
Oleg Kunitsynanddougal ecea0cd6f9 http: add basic metadata and provide it via serve
Co-authored-by: dougal <147946567+roucc@users.noreply.github.com>
2025-11-17 16:52:30 +00:00
jijamik 80e6389a50 ftp: fix transfers from servers that return 250 ok messages 2025-11-14 21:01:25 +00:00
dougal a3ccf4d8a0 b2: allow individual old versions to be deleted with --b2-versions - fixes #1626 2025-11-14 17:04:45 +00:00
Nick Craig-Wood 31df39d356 build: fix tls: failed to verify certificate: x509: negative serial number
Before Go 1.23, x509.ParseCertificate accepted certificates with
negative serial numbers. Rejecting these certificates caused a small
number of users to see this error.

From Go 1.23 debug flags can be added to go.mod so this change adds a
debug flag to ensure negative serial numbers are still allowed since
this is a spec violation, not a security issue.

See: https://forum.rclone.org/t/ssl-validation-broken-between-v1-69-1-latest-version/
2025-11-14 12:51:17 +00:00
Nick Craig-Wood 03d3811f7f Add Sean Turner to contributors 2025-11-14 12:51:17 +00:00
Sean Turner 83b83f7768 s3: add support for --upload-header If-Match and If-None-Match
The If-Match and If-None-Match headers were being dropped rather
than implemented in the Put Object request to S3. These headers
make requests conditional which allow AWS S3 Bucket Policies to
prevent Object overwriting.
2025-11-13 13:50:47 +00:00
n4n5 71138082ea fix: comment typos 2025-11-13 13:47:40 +00:00
Nick Craig-Wood cf94824426 dropbox: fix error moving just created objects - fixes #8881
The bisync tests have been failing as Dropbox is failing to move just
created objects. This seems to be caused by an eventual consistency
problem so this attempts to fix it by retrying the specific error.
2025-11-12 15:54:01 +00:00
hunshcn 16971ab6b9 s3: add --s3-use-data-integrity-protections to fix BadDigest error in Alibaba, Tencent
Since aws/aws-sdk-go-v2#2960, aws-go-sdk-v2 changes its default integrity
behavior. This breaks some s3 providers (eg Tencent, Alibaba)

https://github.com/aws/aws-sdk-go-v2/discussions/2960

This introduces `use_data_integrity_protections` option to disable it.

Defaults to false with it set to true for AWS.

Fixes #8432
Fixes #8483
2025-11-12 15:15:13 +00:00
Nick Craig-Wood 9f75af38e3 rc: make sure fatal errors don't crash rclone - fixes #8955
Before this change, if any code called fs.Fatal(f) then it would stop
rclone as designed. However this is not appropriate when using the RC
API - we want the error returned to the user.

This change turns the fs.Fatal(f) call into a panic which is caught by
the RC API handler and returned to the user as a 500 error.
2025-11-12 12:22:04 +00:00
Nick Craig-Wood b5e4d39b05 pacer: factor call stack searching into its own package 2025-11-12 12:22:04 +00:00
Nick Craig-Wood 4d19afdbbf rc: add osVersion, osKernel and osArch to core/version
This makes it return the same info as `rclone version`
2025-11-12 11:16:48 +00:00
Nick Craig-Wood 2ebfedce85 build: update all dependencies 2025-11-12 10:36:30 +00:00
dependabot[bot] 1a4b85b6e7 build(deps): bump golangci/golangci-lint-action from 8 to 9
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 8 to 9.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v8...v9)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-version: '9'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-11 17:10:10 +01:00
Nick Craig-Wood 5052b80298 webdav: fix out of memory with sharepoint-ntlm when uploading large file
Fixes #7469
Fixes #8959
See: https://forum.rclone.org/t/huge-memory-usage-10gb-when-upload-a-single-large-file-16gb-in-webdav/43312/
2025-11-10 16:57:18 +00:00
Nick Craig-Wood fada870ff0 testserver: fix owncloud test server startup 2025-11-10 16:57:18 +00:00
Nick Craig-Wood 38f456c527 Add aliaj1 to contributors 2025-11-10 16:57:18 +00:00
aliaj1 e6d82ac6ee ulozto: Fix downloads returning HTML error page
The uloz.to backend was failing to download files, instead returning
an HTML page with a "Slow download" message. This was caused by
recent changes in the uloz.to API.

This commit fixes the issue by making the following changes to the
download process:

1.  The `hash` received from the download link API is now appended as a
    query parameter to the download URL.
2.  The download is now performed using the authenticated `rest` client
    to ensure premium access is recognized.
3.  The `DeviceID` is now generated dynamically for each download request
    to avoid potential rate-limiting of a static ID.
2025-11-10 15:56:06 +00:00
Nick Craig-Wood 4c74ded85a docs: adjust spectra logic example endpoint name 2025-11-10 13:47:33 +00:00
kapitainsky 43848f5c42 docs: update version introduced to v1.70 in doi docs
Fixes #8948
2025-11-08 21:33:38 +00:00
Nick Craig-Wood fb895f69a1 testserver: fix HDFS server after run.bash adjustments 2025-11-05 17:56:28 +00:00
Nick Craig-Wood b204090325 testserver: remind developers about allocating a port 2025-11-05 17:56:28 +00:00
Nick Craig-Wood 1821d86911 testserver: make run.bash variables less likely to collide with scripts 2025-11-05 17:56:28 +00:00
Nick Craig-Wood 7ce67347fb testserver: fix seafile servers messing up _connect string 2025-11-05 17:56:28 +00:00
Nick Craig-Wood 0228bbff39 testserver: make sure TestWebdavInfiniteScale uses an assigned port 2025-11-05 17:56:28 +00:00
Nick Craig-Wood 6890bd7738 testserver: make sure we don't overwrite the NAME variable set
This fixes some oddities stopping and starting servers
2025-11-05 17:56:28 +00:00
Nick Craig-Wood bc5d1dfaf3 Add n4n5 to contributors 2025-11-05 17:56:28 +00:00
Nick Craig-Wood c33aeb705f Add Alex to contributors 2025-11-05 17:56:28 +00:00
Nick Craig-Wood 12cf8e71df Add Copilot to contributors 2025-11-05 17:56:28 +00:00
albertony ec5ddb68a8 docs: update contributing docs regarding backend documentation 2025-11-05 14:06:09 +01:00
n4n5 8335596207 rc: add jobs stats 2025-11-05 12:36:39 +00:00
albertony 4f56ab2341 docs: fix alignment of some of the icons in the storage system dropdown 2025-11-04 23:00:46 +01:00
albertony 8b5b7ecfd9 docs: run markdownlint on _index.md 2025-11-04 23:00:46 +01:00
albertony 2aa2cfc70e docs: fix markdownlint issues and other styling improvements in backend command docs 2025-11-04 23:00:46 +01:00
albertony 7265b2331f docs: fix markdownlint issue md046/code-block-style in backend command docs 2025-11-04 23:00:46 +01:00
albertony 0dd56ff2a3 docs: fix missing punctuation in backend commands short description 2025-11-04 23:00:46 +01:00
albertony 2443cb284e docs: fix markdownlint issues in backend command generated output 2025-11-04 23:00:46 +01:00
albertony 0f3aa17fb6 build: improve backend docs autogenerated marker line
Replace custom rem hugo shortcode template with HTML comment. HTML comments are now
allowed in Hugo without enabling unsafe HTML parsing.

Improve the text in the comment: Remove unnecessary quoting, and avoid impression that
make backenddocs has to be run and results committed, since we have a lint check which
will then report error because we want to prevent manual changes in autogenerated sections.

Disable the markdownlint rule line-length on the autogenerated marker line.

Make the autogenerated marker detection a bit more robust.

See #8942 for more details.
2025-11-04 21:56:01 +01:00
Alex 8f74e7d331 backend/compress: add zstd compression
Added support for reading and writing zstd-compressed archives in seekable format
using "github.com/klauspost/compress/zstd" and
"github.com/SaveTheRbtz/zstd-seekable-format-go/pkg".

Bumped Go version from 1.24.0 to 1.24.4 due to requirements of
"github.com/SaveTheRbtz/zstd-seekable-format-go/pkg".
2025-11-04 14:50:56 +00:00
CopilotandNick Craig-Wood ee92673e1b sftp: fix zombie SSH processes with --sftp-ssh - Fixes #8929
Before this fix using --sftp-ssh with the sftp backend could leave
zombie processes.

This patch fixes the problem that sshClientExternal.session was never
assigned, so Wait() always returned nil without waiting for the SSH
process to exit. This caused zombie processes because the process was
never reaped.

It also ensures that Wait() is only called once on each process.

I gave this issue to Copilot to fix as an experiment. It went off in
the wrong direction to start with and fixed something which wasn't the
problem but still needed fixing. With a bit of a nudge it fixed the
correct problem too.

Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2025-11-04 12:09:47 +00:00
Nick Craig-Wood 55655efabf testserver: fix tests failing due to stopped servers
Before this fix there were various issues with the test server
framework, most noticeably servers stopping when they shouldn't
causing timeouts. This was caused by the reference counting in the Go
code not being engineered to work in multiple processes so it was not
working at all properly.

This fix moves the reference counting logic to the start scripts and
in turn removes that logic from the Go code. This means that the
reference counting is now global and works correctly over multiple
processes.
2025-11-04 11:45:15 +00:00
dougal 700e6e11fd docs: add new integration tester site link 2025-11-03 17:15:53 +00:00
Nick Craig-Wood edb47076b5 docs: update the method for running integration tests 2025-11-03 16:52:33 +00:00
Nick Craig-Wood e5fd97b8d2 bisync: fix failing tests
In this commit

d240d044c3 check: improved reporting of differences in sizes and contents

We adjusted the sense of operations.CheckIdenticalDownload to return
true if files are identical as is implied by the name, but we forgot
to invert the logic in the bisync DownloadCheckFn which caused lots of
tests to fail.
2025-11-03 16:52:33 +00:00
Nick Craig-Wood bc57a31859 Add SublimePeace to contributors 2025-11-03 16:52:33 +00:00
dougal 4adb48fbbc b2: fix "expected a FileSseMode but found: ''"
94deb6bd6f b2: Add Server-Side encryption support

From the commit above, without setting SSE, rclone would send invalid
SSE requests with empty strings. This is as omitempty only works with
struct pointers not structs.
2025-11-03 16:42:40 +00:00
SublimePeace c41d0f7d3a docs: s3: clarify multipart uploads memory usage
Clarified phrasing to avoid confusion. Fixed a typo.

Fixes #8525
2025-11-03 16:35:33 +00:00
Nick Craig-Wood d34ba258b0 test_all: fix detection of running servers
Before this change stopping servers was unreliable, expecially the non
docker based ones. This caused timeouts and connection errors in the
tests.
2025-11-03 14:44:39 +00:00
Nick Craig-Wood 05d54a95b8 accounting: add AccountReadN for use in cluster 2025-11-03 14:44:39 +00:00
Nick Craig-Wood f16b39165b fs: add NonDefaultRC for discovering options in use
This enables us to send rc messages with the config in use.
2025-11-03 14:44:39 +00:00
Nick Craig-Wood 86edb26fd5 fs: move tests into correct files 2025-11-03 14:44:39 +00:00
Nick Craig-Wood 203e1bdbf9 rc: add NewJobFromBytes for reading jobs from non HTTP transactions 2025-11-03 14:44:39 +00:00
Nick Craig-Wood a522c056fe rc: add job/batch for sending batches of rc commands to run concurrently 2025-11-03 14:44:39 +00:00
Nick Craig-Wood 31adc7d89f Add Ted Robertson to contributors 2025-11-03 14:44:39 +00:00
Nick Craig-Wood c559ab7c58 Add Joseph Brownlee to contributors 2025-11-03 14:44:39 +00:00
Nick Craig-Wood 80610ef774 Add fries1234 to contributors 2025-11-03 14:44:39 +00:00
Nick Craig-Wood a6c943a1ad Add Fawzib Rojas to contributors 2025-11-03 14:43:56 +00:00
Nick Craig-Wood 53e0dbb5cb Add Riaz Arbi to contributors 2025-11-03 14:43:56 +00:00
Nick Craig-Wood 3a0000526b Add Lukas Krejci to contributors 2025-11-03 14:43:56 +00:00
Nick Craig-Wood 1fa6941e26 Add Adam Dinwoodie to contributors 2025-11-03 14:43:56 +00:00
Nick Craig-Wood 9bb7ad31e6 Add dulanting to contributors 2025-11-03 14:43:56 +00:00
Ted Robertson da8c6847ad docs: add AppArmor restrictions to rclone mount 2025-11-01 19:28:14 +00:00
albertony d240d044c3 check: improved reporting of differences in sizes and contents
fixes rclone check --download not showing differing files
2025-11-01 19:23:01 +00:00
iTrooz 1056ace80f mega: implement 2FA login 2025-11-01 19:03:49 +00:00
albertony a06c1c0cb7 docs: change to light code block style to better match overall theme 2025-11-01 18:55:11 +01:00
albertony 7672c3d586 docs: fix various markdownlint issues 2025-11-01 18:54:19 +01:00
albertony f361cdf1cb build: restrict the markdown languages to use for code blocks 2025-11-01 15:52:41 +01:00
albertony 26d3c71bab docs: fix various markdownlint issues 2025-11-01 15:33:38 +01:00
albertony c76396f03c docs: fix markdownlint issue md013/line-length 2025-11-01 15:33:38 +01:00
albertony 059ad47336 docs: change syntax hightlighting for command examples from sh to console 2025-11-01 15:33:38 +01:00
becc068d36 docs: Clarify remote naming convention
Co-authored-by: dougal <147946567+roucc@users.noreply.github.com>
Co-authored-by: dougal <dougal.craigwood@gmail.com>
2025-10-31 15:42:38 +00:00
fries1234 94deb6bd6f b2: Add Server-Side encryption support
This commit adds SSE-C (Server-Side Encryption - Customer) support to
the B2 native backend. The server uses a customer provided AES-256 key
to encrypt the files when you upload them to the bucket, and then it
discards your key from the servers RAM after you're done uploading.

The option names and descriptions are based off the S3 backend
implementation as the way S3 and B2 does SSE-C is pretty similar.

Fixes #6585
2025-10-31 15:33:31 +00:00
Fawzib RojasandNick Craig-Wood cc09978b79 Added rclone archive command to create and read archive files
Co-Authored-By: Nick Craig-Wood <nick@craig-wood.com>
2025-10-30 16:20:48 +00:00
Fawzib Rojas 409dc75328 accounting: add io.Seeker/io.ReaderAt support to accounting.Account
This is a pass through implementation which will fail if the
underlying reader does not have the interface.
2025-10-30 16:20:48 +00:00
Nick Craig-Wood fb30c5f8dd operations: add ReadAt method to ReOpen 2025-10-30 16:20:48 +00:00
Nick Craig-Wood 203df6cc58 fstest: add ResetRun to allow the remote to be reset in tests 2025-10-30 16:20:48 +00:00
Riaz Arbi 459e10d599 gcs: fix --gcs-storage-class to work with server side copy for objects 2025-10-30 15:20:16 +00:00
Lukas Krejci 1ba4fd1d83 ulozto: implement the about functionality 2025-10-30 15:06:37 +00:00
Adam Dinwoodie 77553b8dd5 local: add --skip-specials to ignore special files
Give users a way to explicitly acknowledge that pipes, sockets and block
devices are to be ignored without warnings.

This follows the precedent set in commit 6152bab28 (local: add
--skip-links to suppress symlink warnings, 2017-07-21) for ignoring
warnings about symlinks.
2025-10-29 17:00:25 +00:00
Andrew Ruthven 5420dbbe38 swift: Report disk usage in segment containers
Large objects are split and stored in a _segments container in Swift.
These should be included when reporting on the space used.

Fixes #8857
2025-10-29 16:55:53 +00:00
dulanting 87b71dd6b9 refactor: use strings.Builder to improve performance 2025-10-29 16:48:34 +00:00
Nick Craig-Wood a0bcdc2638 Archive backend to read archives on cloud storage.
Initial support with Zip and Squashfs archives.

Fixes #8633
See #2815
2025-10-28 11:05:41 +00:00
Nick Craig-Wood e42fa9f92d vfs: remove unecessary import in tests to fix import cycles 2025-10-28 11:05:41 +00:00
Nick Craig-Wood 4586104dc7 Add Lakshmi-Surekha to contributors 2025-10-28 11:05:35 +00:00
Nick Craig-Wood c4c360a285 Add Andrew Gunnerson to contributors 2025-10-28 11:05:35 +00:00
Nick Craig-Wood ce4860b9b6 Add divinity76 to contributors 2025-10-28 11:05:35 +00:00
Lakshmi-Surekha ed87f82d21 build: enable support for aix/ppc64
* Adds "aix/ppc64" to the cross-compile target list.
* Including AIX in the build tag of "metadata_other.go".
* Excluding AIX from the main ncdu build tags.
* Marking AIX as an unsupported platform for ncdu.
* Excluding AIX from the fallback redirect implementation.
* Excluding AIX from unix build tags to avoid undefined unix.WNOHANG.
2025-10-27 13:34:58 +00:00
Andrew Gunnerson 0a82929b94 rc: fix name of "queue" JSON key in docs for vfs/cache
Signed-off-by: Andrew Gunnerson <accounts+github@chiller3.com>
2025-10-27 13:28:24 +00:00
divinity76 1e8ee3b813 cmount: windows: improve error message on missing winfsp 2025-10-27 13:22:04 +00:00
Nick Craig-Wood eaab3f5271 docs: add the Provider to the options examples in the backend docs 2025-10-26 10:25:12 +00:00
Nick Craig-Wood 25b05f1210 Add Aneesh Agrawal to contributors 2025-10-26 10:25:12 +00:00
Nick Craig-Wood 2dc1b07863 Add viocha to contributors 2025-10-26 10:25:12 +00:00
Nick Craig-Wood 49acacec2e Add reddaisyy to contributors 2025-10-26 10:25:12 +00:00
Aneesh Agrawal 70d2fe6568 fs: remove unnecessary Seek call on log file
We were seeing a (non-fatal) error in our logs:
```
Failed to seek log file to end: seek /proc/1/fd/1: illegal seek
```

Because we open the log file with O_APPEND,
we don't need to manually seek to the end.
As https://pkg.go.dev/os#File.Seek also confirms
that the behavior of `Seek` is not specified
if the file has been opened with O_APPEND,
remove the `Seek` call.
2025-10-25 19:38:57 +01:00
dougal f28c83c6de s3: make it easier to add new S3 providers
Before this change, you had to modify a fragile data-structure
containing all providers. This often led to things being out of order,
duplicates and conflicts whilst merging. As well as the changes for
one provider being in different places across the file.

After this change, new providers are defined in an easy to edit YAML file,
one per provider.

The config output has been tested before and after for all providers
and any changes are cosmetic only.
2025-10-25 19:37:29 +01:00
dependabot[bot] 2cf44e584c build(deps): bump actions/upload-artifact from 4 to 5
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4 to 5.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-25 12:09:16 +02:00
dependabot[bot] bba9027817 build(deps): bump actions/download-artifact from 5 to 6
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 5 to 6.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-25 12:09:06 +02:00
dougal 51859af8d9 ftp: fix SOCK proxy support - fixes #8892 (#8918) 2025-10-24 14:50:13 +01:00
viocha 4f60f8915d webdav: Add Access-Control-Max-Age header for CORS preflight caching - fixes #5078 2025-10-24 10:19:22 +01:00
hunshcn 6663eb346f webdav: use SpaceSepList to parse bearer token command 2025-10-23 19:56:37 +01:00
reddaisyy 1d0e1ea0b5 refactor: use strings.Builder to improve performance 2025-10-23 16:40:30 +01:00
Nick Craig-Wood 71631621c4 docs: re-arrange sponsors page 2025-10-23 14:50:51 +01:00
Nick Craig-Wood 31e904d84c docs: add Spectra Logic as a sponsor 2025-10-23 14:50:51 +01:00
Nick Craig-Wood 30c9843e3d Add Oleksandr Redko to contributors 2025-10-23 14:50:51 +01:00
Oleksandr Redko c8a834f0e8 build: enable all govet checks (except fieldalignment and shadow) and fix issues. 2025-10-22 18:37:58 +01:00
Nick Craig-Wood b272c50c4c march: fix --no-traverse being very slow - fixes #8860
Before this change --no-traverse was calling NewObject on directories
(where it would always fail) as well as files. This was very
noticeable when doing syncs with --max-age which were only
transferring a small number of objects. This should have been very
quick, but the NewObject calls for each directory slowed the sync down
a lot.

This changes replaces the check to see if the source entry is an
Object that got missed out from this commit:

88e30eecbf march: fix deadlock when using --no-traverse - fixes #8656
2025-10-22 14:14:52 +01:00
Nick Craig-Wood b8700e8042 Add vastonus to contributors 2025-10-22 14:14:52 +01:00
kingston125 73193b0565 s3: add new FileLu S5 endpoints
Add US, EU, AP, and ME endpoints
2025-10-22 12:25:05 +01:00
vastonus c4eef3065f build: remove obsolete build tag 2025-10-21 18:56:06 +01:00
Nick Craig-Wood ba2a642961 azurefiles: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood 979c6a573d dropbox: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood bbb866018e webdav: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood 7706f02294 pcloud: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood 6df7913181 box: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood c079495d1f onedrive: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood 3bf1ac5b07 drive: add ListP interface - #4788 2025-10-21 18:40:23 +01:00
Nick Craig-Wood 091caa34c6 Add hunshcn to contributors 2025-10-21 18:40:23 +01:00
hunshcn d507e9be39 webdav: optimize bearer token fetching with singleflight 2025-10-21 11:14:37 +01:00
Nick Craig-Wood 40b3251e41 Changelog updates from Version v1.71.2 2025-10-20 16:56:47 +01:00
albertony 484d955ea8 lib/http: cleanup indentation and other whitespace in http serve template 2025-10-20 11:53:55 +01:00
albertony 8fa9f255a0 docs: improve formatting of http serve template parameters 2025-10-20 11:53:55 +01:00
Nick Craig-Wood e7f11af1ca build: stop markdown linter leaving behind docker containers 2025-10-20 11:51:23 +01:00
Nick Craig-Wood 0b5c4cc442 Add Marco Ferretti to contributors 2025-10-20 11:51:23 +01:00
Marco Ferretti 178ddafdc7 s3: add cubbit as provider 2025-10-20 11:01:34 +01:00
dougal ad316ec6e3 s3: add servercore as a provider 2025-10-17 16:35:06 +01:00
Nick Craig-Wood 61b022dfc3 docs: update sponsors 2025-10-17 12:04:51 +01:00
Nick Craig-Wood 1903b4c1a2 docs: update sponsor images 2025-10-15 16:33:10 +01:00
Nick Craig-Wood f7cbcf556f docs: update privacy policy with a section on user data 2025-10-14 16:24:07 +01:00
Nick Craig-Wood 3581e628c0 Add Dulani Woods to contributors 2025-10-14 16:24:07 +01:00
Nick Craig-Wood 62c41bf449 Add spiffytech to contributors 2025-10-14 16:24:07 +01:00
Dulani Woods c5864e113b gcs: add region us-east5 - fixes #8863 2025-10-14 14:13:56 +01:00
albertony 39259a5bd1 jottacloud: refactor service list from map to slice to get predefined order 2025-10-11 20:57:19 +02:00
albertony 2e376eb3b9 jottacloud: added support for traditional oauth authentication also for the main service
This renames whitelabel authentication to traditional authentication and adds support for
the main Jottacloud service also here, as it can be used as an alternative to the
authentication based on personal login token for those who prefer it. Documentation
also adjusted correspondingly, and restructured the authentication section a bit more
since some of the sections that was under standard authentication in reality also
applies to the traditional authentication.
2025-10-11 20:57:19 +02:00
albertony de8e9d4693 oauthutil: improved debug logs from token refresh 2025-10-10 20:10:21 +02:00
spiffytech 710cf49bc6 backend: add S3 provider for Hetzner object storage #8183 2025-10-10 18:20:43 +01:00
albertony 8dacac60ea jottacloud: improved token refresh handling
The oauthutil.Renew was initialized early in NewFs, before the first request to the
service where a token is needed. When token is already expired at the time NewFs is
called, the Renew operation would be triggered immediately, only to abort before actually
performing a token refresh, for reason described in debug message:

    Token expired but no uploads in progress - doing nothing

Then later in NewFs, a request to the customer endpoint was made, and since it requires
a valid token it would perform a token refresh after all.

This was not a big problem, but a bit unnecessary, and the debug log messages made it
confusing to understand what rclone was actually doing regarding token refreshing.

If, from debugger, we were forcing the Renew operation to perform actual token refresh,
even if no uploads in process, then it would fail because it actually needs the username
which is retrieved from the customer endpoint

    jottacloud root '': Token refresh failed: read metadata failed: error 400: org.springframework.security.core.userdetails.UsernameNotFoundException: Username not found in url! (Bad Request)

Don't think this can happen in any real situations, but better to make sure it never can.
2025-10-10 18:59:19 +02:00
dougal 3a80d4d4b4 s3: provider reordering
+ fixing some typos
2025-10-10 16:30:03 +01:00
dougal a531f987a8 index: add missing providers 2025-10-10 16:30:03 +01:00
dougal e906b8d0c4 docs: add missing ` 2025-10-10 16:30:03 +01:00
dougal a5932ef91a s3: add rabata as a provider 2025-10-10 16:30:03 +01:00
Nick Craig-Wood 3afa563eaf mega: fix 402 payment required errors - fixes #8758
The underlying library now supports hashcash which should fix this
problem.
2025-10-09 11:58:49 +01:00
Nick Craig-Wood 9d9654b31f Add Andrew Ruthven to contributors 2025-10-09 11:58:49 +01:00
Nick Craig-Wood cfe257f13d Add Microscotch to contributors 2025-10-09 11:58:49 +01:00
Nick Craig-Wood 0375efbd35 Add iTrooz to contributors 2025-10-09 11:58:49 +01:00
Andrew Ruthven cad1954213 build: Bump SwiftAIO container to a newer one
The bouncestorage image hasn't been updated for 4 years and has this
message at the top of the docs:

  This repository is outdated; please use dockerswiftaio/docker-swift instead.

However, dockerswiftaio/docker-swift hasn't been updated for 2 years.
Switch to openstackswift/saio instead, which is getting regular updates.

This requires some minor changes to one test, and how we start the
container.
2025-10-06 16:55:48 +01:00
Andrew Ruthven 604e37caa5 build: Retry stopping the test server
On my system there needs to be a slight pause between stopping and
checking to see if SwiftAIO has stopped. Without the pause the tests fail for
a non-obvious reason.

Instead of using a magic sleep, re-use the retry logic that is used for
starting the test server.
2025-10-06 16:55:48 +01:00
Andrew Ruthven b249d384b9 build: Increase attempts to connect to test server
On the system I'm testing Swift on it can take ~90 retries for SwiftAIO to
be ready. Extend the retry attempts.
2025-10-06 16:55:48 +01:00
Andrew Ruthven 04e91838db swift: If storage_policy isn't set, use the root containers policy
Ensure that if we need to create a segments container it uses the same
storage policy as the root container.

Fixes #8858
2025-10-06 16:55:48 +01:00
Microscotch 94829aaec5 proton: automated 2FA login with OTP secret key
add OTP secret key to config to generate 2FA code
2025-10-06 16:18:38 +01:00
iTrooz f574e3395c serve s3: fix log output to remove the EXTRA messages
As shown in

https://github.com/rclone/gofakes3/blob/81e56a30c86f942861f105439339812439b53415/log.go#L74

it seems like the wanted behaviour for merging arguments is the one of PrintLn,
which is "put a space between each arg"
2025-10-06 15:17:21 +01:00
albertony 2bc155a96a docs/jottacloud: update description of invalid_grant error according to changes 2025-10-05 11:22:27 +02:00
albertony adc8ea3427 jottacloud: add support for MediaMarkt Cloud as a whitelabel service
This was requested in issue #8852, after authentication was already fixed for existing
whitelabels.
2025-10-05 00:48:01 +02:00
kingston125 068eea025c s3: add FileLu S5 provider 2025-10-04 15:48:01 +01:00
iTrooz 4510aa679a docs: fix variants of --user-from-header 2025-10-04 08:10:49 +02:00
dougal 79281354c7 vfs: fix chunker integration test 2025-10-03 17:10:24 +01:00
Nick Craig-Wood f57a178719 test_all: give TestZoho: extra time as it has been timing out 2025-10-03 16:03:29 +01:00
Nick Craig-Wood 44f2e2ed39 test_all: give TestCompressDrive: extra time as it has been timing out 2025-10-03 16:02:07 +01:00
Nick Craig-Wood 13e1752d94 rclone config string: reduce quoting with Human rendering for strings #8859 2025-10-03 15:54:15 +01:00
Nick Craig-Wood bb82c0e43b Add juejinyuxitu to contributors 2025-10-03 15:54:15 +01:00
albertony 1af7151e73 docs/jottacloud: update documentation with new whitelabel services and changed configuration flow 2025-10-02 19:16:03 +02:00
albertony fd63478ed6 jottacloud: abort attempts to run unsupported rclone authorize command 2025-10-02 19:16:03 +02:00
albertony 5133b05c74 jottacloud: minor adjustment of texts in config ui 2025-10-02 19:16:03 +02:00
albertony 6ba96ede4b jottacloud: add support for Let's Go Cloud (from MediaMarkt) as a whitelabel service 2025-10-02 19:16:03 +02:00
albertony 2896973964 jottacloud: fix authentication for whitelabel services from Elkjøp subsidiaries
This adds support for them in the whitelabel autentication type, relying on OpenID
Connect, same as Telia, Tele2 etc already uses.

Until recently the Elkjøp subsidiaries still supported the legacy authentication type
only, but that seem to have changed. They no longer support legacy authentication, which
made existing rclone version incompatible with them.

With this the legacy authentication has no known uses, however the implementation of
it is still kept for now.

Fixes #8852
2025-10-02 19:16:03 +02:00
albertony be123d85ff jottacloud: refactor config handling of whitelabel services to use openid provider configuration 2025-10-02 19:16:03 +02:00
albertony b1b9562ab7 jottacloud: remove nil error object from error message 2025-10-02 19:16:03 +02:00
albertony 5146b66569 jottacloud: fix legacy authentication
This fixes the issue where configuration would fail after supplying passoword:

    Reveal failed: input too short when revealing password - is it obscured?
2025-10-02 19:16:03 +02:00
albertony 8898372d5a docs: add remote setup page to main docs dropdown 2025-10-02 18:46:16 +02:00
albertony 091fe9e453 docs: update remote setup page 2025-10-02 18:46:16 +02:00
albertony 8fdb68e41a docs: add link from authorize command docs to remote setup docs 2025-10-02 18:46:16 +02:00
albertony c124aa2ed3 docs: lowercase internet and web browser instead of Internet browser 2025-10-02 18:46:16 +02:00
albertony 54e8bb89f7 docs: use the term backend name instead of fs name for authorize command 2025-10-02 18:46:16 +02:00
Nick Craig-Wood 50c1b594ab add rclone config string for making connection strings #8859 2025-10-02 17:30:08 +01:00
Nick Craig-Wood 72437a9ca2 config: add more human readable configmap.Simple output
Before this, String() quoted every part of the config map even if it
wasn't necessary.

The new Human() method removes the quoting and adds the special case
for "true" values.
2025-10-02 17:30:08 +01:00
dougal 8ed55c61e1 serve http: download folders as zip
Now folders can be downloaded as a zip. You can also use --disable-zip
to not show this.
2025-09-26 15:18:02 +01:00
dougal bd598c1ceb s3: reorder providers to be in alphabetical order 2025-09-26 15:14:45 +01:00
juejinyuxitu 7e30665102 refactor: use strings.FieldsFuncSeq to reduce memory allocations
Signed-off-by: juejinyuxitu <juejinyuxitu@outlook.com>
2025-09-26 15:12:53 +01:00
Nick Craig-Wood d44957a09c accounting: add SetMaxCompletedTransfers method to fix bisync race #8815
Before this change bisync adjusted the global MaxCompletedTransfers
variable which caused races.

This adds a SetMaxCompletedTransfers method and uses it in bisync.

The MaxCompletedTransfers global becomes the default. This can be
changed externally if rclone is in use as a library, and the commit
history indicates that MaxCompletedTransfers was added for exactly
this purpose so we try not to break it here.
2025-09-26 14:54:47 +01:00
Nick Craig-Wood 37524e2dea accounting: add RemoveDoneTransfers method to fix bisync race #8815
Before this change bisync was adjusting MaxCompletedTransfers in order
to clear the done transfers from the stats.

This wasn't working (because it was only clearing one transfer) and
was part of a race adjusting MaxCompletedTransfers.

This fixes the problem by introducing a new method RemoveDoneTransfers
to clear the done transfers explicitly and calling it in bisync.
2025-09-26 14:54:47 +01:00
Nick Craig-Wood 2f6a6c8233 bisync: fix race when CaptureOutput is used concurrently #8815
Before this change CaptureOutput could trip the race detector when
used concurrently. In particular if go routines using the logging are
outlasting the return from `fun()`.

This fixes the problem with a mutex.
2025-09-26 14:54:47 +01:00
Nick Craig-Wood 4ad40b6554 build: update all dependencies 2025-09-26 14:53:36 +01:00
Nick Craig-Wood 4f33d64f25 Makefile: remove deprecated go mod usage 2025-09-26 14:53:36 +01:00
Vikas Bhansali 519623d9f1 azurefiles: Fix server side copy not waiting for completion - fixes #8848 2025-09-26 12:41:42 +01:00
Nick Craig-Wood 913278327b Changelog updates from Version v1.71.1 2025-09-24 17:34:26 +01:00
Nick Craig-Wood a9b05e4c7a test_all: fix branch name in test report 2025-09-24 15:35:09 +01:00
Nick Craig-Wood 5d6d79e7d4 pacer: fix deadlock with --max-connections
If the pacer was used recursively and --max-connections was in use
then it could deadlock if all the connections were in use at the time
of recursive call (likely).

This affected the azureblob backend because when it receives an
InvalidBlockOrBlob error it attempts to clear the condition before
retrying. This in turn involves recursively calling the pacer.

This fixes the problem by skipping the --max-connections check if the
pacer is called recursively.

The recursive detection is done by stack inspection which isn't ideal,
but the alternative would be to add ctx to all >1,000 pacer calls. The
benchmark reveals stack inspection takes about 55nS per stack level so
it is relatively cheap.
2025-09-22 17:39:27 +01:00
Nick Craig-Wood 11de074cbf Revert "azureblob: fix deadlock with --max-connections with InvalidBlockOrBlob errors"
This reverts commit 0c1902cc6037d81eaf95e931172879517a25d529.

This turns out not to be sufficient so we need a better approach
2025-09-22 17:39:27 +01:00
Nick Craig-Wood e9ab177a32 Add Youfu Zhang to contributors 2025-09-22 17:39:27 +01:00
Nick Craig-Wood f3f4fba98d Add Matt LaPaglia to contributors 2025-09-22 17:39:27 +01:00
Sudipto Baral 03fccdd67b smb: optimize smb mount performance by avoiding stat checks during initialization
add IsPathDir function and tests for trailing slash optimization
2025-09-22 15:33:44 +01:00
Youfu Zhang 231083647e pikpak: fix unnecessary retries by using URL expire parameter - fixes #8601
Before this change, rclone would unnecessarily retry downloads when
the `Link.Expire` field was unreliable but the download URL contained
a valid expire query parameter. This primarily affects cases where
media links are unavailable or when `no_media_link` is enabled.

The `Link.Valid()` method now primarily checks the URL's expire query
parameter (as Unix timestamp) and falls back to the Expire field
only when URL parsing fails. This eliminates the `error no link`
retry loops while maintaining backward compatibility.

Signed-off-by: Youfu Zhang <zhangyoufu@gmail.com>
2025-09-19 12:46:26 +09:00
dougal 0e203a7546 serve http: fix: logging url on start 2025-09-18 14:49:58 +01:00
Matt LaPaglia a7dd787569 docs: fix typo 2025-09-16 14:27:10 +02:00
dougal 689555033e b2: fix 1TB+ uploads
Before this change the minimum chunk size would default to 96M which
would allow a maximum size of just below 1TB file to be uploaded, due to
the 10000 part rule for b2.

Now the calculated chunk size is used so the chunk size can be 5GB
making a max file size of 50TB.

Fixes #8460
2025-09-15 13:05:20 +01:00
Nick Craig-Wood 4fc4898287 march: fix deadlock when using --fast-list on syncs - fixes #8811
Before this change, it was possible to have a deadlock when using
--fast-list for a sync if both the source and destination supported
ListR.

This fixes the problem by shortening the locking window.
2025-09-15 12:55:29 +01:00
Nick Craig-Wood b003169088 build: slices.Contains, added in go1.21 2025-09-15 12:45:57 +01:00
Nick Craig-Wood babd112665 build: use strings.CutPrefix introduced in go1.20 2025-09-15 12:45:57 +01:00
Nick Craig-Wood 71b9b4ad7a build: use sequence Split introduced in go1.24 2025-09-15 12:45:57 +01:00
Nick Craig-Wood 4368863fcb build: use "for i := range n", added in go1.22 2025-09-15 12:45:57 +01:00
Nick Craig-Wood 04d49bf0ea build: modernize benchmark usage 2025-09-15 12:45:57 +01:00
Nick Craig-Wood d7aa37d263 build: in tests use t.Context, added in go1.24 2025-09-15 12:45:57 +01:00
Nick Craig-Wood 379dffa61c build: replace interface{} by the 'any' type added in go1.18 2025-09-15 12:45:57 +01:00
Nick Craig-Wood 5fd4ece31f build: use the built-in min or max functions added in go1.21 2025-09-15 12:45:57 +01:00
Nick Craig-Wood fc3f95190b Add russcoss to contributors 2025-09-15 12:45:57 +01:00
russcoss d6f5652b65 build: remove x := x made unnecessary by the new semantics of loops in go1.22
Signed-off-by: russcoss <russcoss@outlook.com>
2025-09-14 15:58:20 +01:00
Nick Craig-Wood b5cbb7520d lib/pool: fix unreliable TestPoolMaxBufferMemory test
This turned out to be a problem in the tests. The tests used to do

1. allocate
2. increment
3. free
4. decrement

But if one goroutine had just completed 2 and another had just
completed 3 then this can cause the test to register too many
allocations.

This was fixed by doing the test in this order instead:

1. allocate
2. increment
3. decrement
4. free

The 4 operations are atomic.

Fixes #8813
2025-09-12 10:39:32 +01:00
Nick Craig-Wood a170dfa55b Update S-Pegg1 email 2025-09-12 10:39:32 +01:00
Nick Craig-Wood 1449c5b5ba Add Jean-Christophe Cura to contributors 2025-09-12 10:39:32 +01:00
dougal 35fe609722 pool: fix flaky unreliability test 2025-09-11 18:09:50 +01:00
dougal cce399515f copyurl: reworked code, added concurrency and tests
- Added Tests
- Fixed file name handling
- Added concurrent downloads
- Limited downloads to --transfers
- Fixes #8127
2025-09-11 13:56:14 +01:00
S-Pegg1 8c5af2f51c copyurl: Added --url to read urls from csv file - #8127 2025-09-11 13:56:14 +01:00
dougal c639d3656e docs: HDFS: erasure coding limitation #8808 2025-09-10 19:26:55 +01:00
nielash d9fbbba5c3 fstest: fix slice bounds out of range error when using -remotes local
Before this change, TestIntegration/FsName could fail with "slice bounds out of
range [:-1]" when run with -remotes local.

It also caused issues with
'^TestGitAnnexFstestBackendCases$/^(TransferStorePathWithInteriorWhitespace|TransferStoreRelative)$'.

This change fixes the issue by accepting either "" or "local" to indicate the
local remote.
2025-09-09 12:09:42 -04:00
nielash fd87560388 local: fix time zones on tests
Before this change, TestMetadata could fail due to a difference between the
user's local time zone and UTC causing the string representation of the date to
be off by one day. This change fixes the issue by comparing both in the Local
time zone.
2025-09-09 12:09:42 -04:00
dougal d87720a787 s3: added SpectraLogic as a provider 2025-09-09 16:40:10 +01:00
nielash d541caa52b local: fix rmdir "Access is denied" on windows - fixes #8363
Before this change, Rmdir (and other commands that rely on Rmdir) would fail
with "Access is denied" on Windows, if the directory had
FILE_ATTRIBUTE_READONLY. This could happen if, for example, an empty folder had
a custom icon added via Windows Explorer's interface (Properties => Customize =>
Change Icon...).

However, Microsoft docs indicate that "This attribute is not honored on
directories."
https://learn.microsoft.com/en-us/windows/win32/fileio/file-attribute-constants#file_attribute_readonly
Accordingly, this created an odd situation where such directories were removable
(by their owner) via File Explorer and the rd command, but not via rclone.

An upstream issue has been open since 2018, but has not yet resulted in a fix.
https://github.com/golang/go/issues/26295

This change gets around the issue by doing os.Chmod on the dir and then retrying
os.Remove. If the dir is not empty, this will still fail with "The directory is
not empty."

A bisync user confirmed that it fixed their issue in
https://forum.rclone.org/t/bisync-leaving-empty-directories-on-unc-path-1-or-local-filesystem-path-2-on-directory-renames/52456/4?u=nielash

It is likely also a fix for #8019, although @ncw is correct that Purge would be
a more efficient solution in that particular scenario.
2025-09-09 11:25:09 -04:00
nielash fd1665ae93 bisync: fix error handling for renamed conflicts
Before this change, rclone could crash during modifyListing if a rename's
srcNewName is known but not found in the srcList
(srcNewName != "" && new == nil).
This scenario should not happen, but if it does, we should print an error
instead of crashing.

On #8458 there is a report of this possibly happening on v1.68.2. It is unknown
what the underlying issue was, and whether it still exists in the latest
version, but if it does, the user will now see an error and debug info instead
of a crash.
2025-09-06 12:43:23 -04:00
Jean-Christophe Cura 457d80e8a9 docs: pcloud: update root_folder_id instructions 2025-09-05 20:50:00 +01:00
Nick Craig-Wood c5a3e86df8 operations: fix partial name collisions for non --inplace copies
In this commit:

c63f1865f3 operations: copy: generate stable partial suffix

We made the partial suffix for non inplace copies stable. This was a
hash based off the file fingerprint.

However, given a directory of files which have the same fingerprint
the partial suffix collides. On some backends (eg the local backend)
the fingerprint is just the size and modification time so files with
different contents can collide.

The effect of collisions was hash failures on copy when using
--transfers > 1. These copies invariably retried successfully which
probably explains why this bug hasn't been reported.

This fixes the problem by adding the file name to the hash.

It also makes sure the hash is always represented as 8 hex bytes for
consistency.
2025-09-05 16:09:46 +01:00
Ed Craig-Wood 4026e8db20 drive: docs: update making your own client ID instructions
update instructions with the most recent changes to google cloud console
2025-09-05 15:30:52 +01:00
dougal c9ce686231 swift: add ListP interface - #4788 2025-09-05 15:29:37 +01:00
dougal b085598cbc memory: add ListP interface - #4788 2025-09-05 15:29:37 +01:00
dougal bb47dccdeb oraceobjectstorage: add ListP interface - #4788 2025-09-05 15:29:37 +01:00
dougal 7a279d2789 B2: add ListP interface - #4788 2025-09-05 15:29:37 +01:00
dougal 9bd5df658a azureblob: add ListP interface - #4788 2025-09-05 15:29:37 +01:00
dougal d512e4d566 googlecloudstorage: add ListP interface - Fixes #8763 2025-09-05 15:29:37 +01:00
dependabot[bot] 3dd68c824a build: bump actions/github-script from 7 to 8
Bumps [actions/github-script](https://github.com/actions/github-script) from 7 to 8.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](https://github.com/actions/github-script/compare/v7...v8)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-05 08:14:32 +02:00
dependabot[bot] fbe73c993b build: bump actions/setup-go from 5 to 6
Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5 to 6.
- [Release notes](https://github.com/actions/setup-go/releases)
- [Commits](https://github.com/actions/setup-go/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-go
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-05 08:12:38 +02:00
nielash d915f75edf bisync: fix chunker integration tests
Before this change, TestChunkerS3: tests were failing because our use of
obj.Remove (for "modtime_write_test") created an unexpected extra transfer.

This is because chunker calls operations.Move for removes, which (per its
function comment) is supposed to be only accounted as a check. But because S3
can Copy but not Move, the move falls back to copy and ends up getting counted
as a transfer anyway.
https://github.com/rclone/rclone/blob/99e8a63df2528e4fd470821095cbeb32e8a827a3/fs/operations/operations.go#L506
https://github.com/rclone/rclone/blob/99e8a63df2528e4fd470821095cbeb32e8a827a3/fs/operations/copy.go#L381

This is probably a bug that should get a more proper fix in operations. But in
the meantime, we can get around it by doing our "modtime_write_test" with its
own unique stats group.
2025-09-04 14:38:10 -04:00
nielash 26b629f42f bisync: fix koofr integration tests
Before this change, koofr failed certain bisync tests because it can't set mod
time without deleting and re-uploading. This caused the "nothing to transfer" log
to not get printed where expected (as it is only printed when there are 0
transfers, but koofr requires extra transfers to set modtime.)

This change fixes the issue by ignoring the absence of the "nothing to transfer"
log line on backends that return `fs.ErrorCantSetModTimeWithoutDelete` for
`obj.SetModTime`.
2025-09-04 14:38:10 -04:00
Nick Craig-Wood ceaac2194c internetarchive: fix server side copy files with spaces
In this commit we broke server side copy for files with spaces

4c5764204d internetarchive: fix server side copy files with &

This fixes the problem by using rest.URLPathEscapeAll which escapes
everything possible.

Fixes #8754
2025-09-04 10:37:27 +01:00
Nick Craig-Wood 1f14b6aa35 lib/rest: add URLPathEscapeAll to URL escape as many chars as possible 2025-09-04 10:37:27 +01:00
Nick Craig-Wood dd75af6a18 Add alternate email for dougal to contributors 2025-09-04 10:37:27 +01:00
dougal 99e8a63df2 test speed: add command to test a specified remotes speed
Run speed test to try and work in a given time budget, uploading
randomly created files to the remote then downloading them again.

Fixes #3198
2025-09-03 12:37:52 +01:00
Nick Craig-Wood 0019e18ac3 docs: add link to MEGA S4 from MEGA page 2025-09-02 17:22:32 +01:00
Nick Craig-Wood 218c3bf6e9 Add Robin Rolf to contributors 2025-09-02 17:22:32 +01:00
Nick Craig-Wood 8f9702583d Add anon-pradip to contributors 2025-09-02 17:22:32 +01:00
Robin Rolf e6578fb5a1 s3: Add Intercolo provider 2025-09-02 16:34:43 +01:00
albertony fa1d7da272 gendocs: refactor and add logging of skipped command docs 2025-09-02 14:06:31 +02:00
albertony 813708c24d gendocs: ignore missing rclone_mount.md, rclone_nfsmount.md, rclone_serve_nfs.md on windows 2025-09-02 14:06:31 +02:00
nielash fee4716343 bin: add bisync.md generator
This change adds make_bisync_docs.go step to dynamically update the list of
ignored and failed tests in bisync.md
2025-09-01 14:43:40 -04:00
nielash 6e9a675b3f fstest: refactor to decouple package from implementation 2025-09-01 14:43:40 -04:00
nielash 7f5a444350 gendocs: ignore missing rclone_mount.md on macOS 2025-09-01 14:43:40 -04:00
nielash d2916ac5c7 bisync: ignore expected "nothing to transfer" differences on tests
The "There was nothing to transfer" log is only printed when the number of
transfers is exactly 0. However, there are a variety of reasons why the transfer
count would be expected to differ between backends. For example, if either side
lacks hashes, the sync may in fact need to transfer, where it would otherwise
skip based on hash or just update modtime. Transfer stats will also differ in
the "src and dst identical but can't set mod time without deleting and re-
uploading" scenario (because the re-upload is a transfer), and where --download-hash
is needed (because calculating the hash requires downloading the file, which is
a transfer).

Before this change, these expected differences would result in erroneous test
failures. This change fixes the issue by ignoring the absence of the "nothing to
transfer" log where it is expected.

Note that this issue did not occur before
https://github.com/rclone/rclone/commit/9e200531b1490000656031c42fff4d95477e1b46
because the number of transfers was not getting reset between test steps,
sometimes resulting in an artificially inflated transfers count.
2025-09-01 14:05:00 -04:00
nielash 3369a15285 bisync: fix TestBisyncConcurrent ignoring -case
Before this change, TestBisyncConcurrent would still run the "basic" test case
if a non-blank -case arg was used to specify a case other than "basic". This
change fixes it by skipping in this scenario.
2025-09-01 14:05:00 -04:00
nielash 58aee30de7 bisync: make number of parallel tests configurable
Example usage:
go test ./cmd/bisync -remote local -race -pcount 10
2025-09-01 14:05:00 -04:00
anon-pradip ef919241a6 docs: clarify subcommand description in rclone usage 2025-09-01 17:09:51 +01:00
albertony d5386bb9a7 docs: fix description of regex syntax of name transform 2025-09-01 16:40:14 +01:00
albertony bf46ea5611 docs: add some more details about supported regex syntax 2025-09-01 16:40:14 +01:00
nielash b8a379c9c9 makefile: fix lib/transform docs not getting updated
As of
https://github.com/rclone/rclone/commit/4280ec75ccfd27fbc4f646d5323b8ed060062be5
the lib/transform docs are generated with //go:generate and embedded with
//go:embed.

Before this change, however, they were not getting automatically updated with
subsequent changes (like
https://github.com/rclone/rclone/commit/fe62a2bb4ea8f62a1567c7883c0bba81827656eb)
because `go generate ./lib/transform` was not being run as part of the release
making process.

This change fixes that by running it in `make commanddocs`.
2025-09-01 16:39:20 +01:00
Nick Craig-Wood 8c37a9c2ef lib/pool: fix flaky test which was causing timeouts
This puts a limit on the number of allocation failures in a row which
stops the test timing out as the exponential backoffs get very large.
2025-09-01 16:25:31 +01:00
Nick Craig-Wood 963a72ce01 Add dougal to contributors 2025-09-01 16:25:31 +01:00
dougal a4962e21d1 vfs: fix SIGHUP killing serve instead of flushing directory caches
Before, rclone serve would crash when sent a SIGHUP which contradicts
the documentation - saying it should flush the directory caches.

Moved signal handling from the mount into the vfs layer, which now
handles SIGHUP on all uses of the VFS including mount and serve.

Fixes #8607
2025-09-01 13:15:11 +01:00
nielash 9e200531b1 bisync: use unique stats groups on tests 2025-08-30 17:46:33 +01:00
Nick Craig-Wood 04683f2032 fstest: stop errors in test cleanup changing the global stats
This was causing the concurrent bisync tests to fail every now and again.
2025-08-30 17:46:33 +01:00
Nick Craig-Wood b41f7994da Add Motte to contributors 2025-08-30 17:46:33 +01:00
Nick Craig-Wood 13a5ffe391 Add Claudius Ellsel to contributors 2025-08-30 17:46:33 +01:00
Nick Craig-Wood 85deea82e4 build: add local markdown linting to make check 2025-08-28 16:56:40 +01:00
Motte 89a8ea7a91 lsf: add support for unix and unixnano time formats 2025-08-28 16:28:49 +01:00
albertony c8912eb6a0 docs: remove broken links from rc to commands 2025-08-28 11:52:18 +02:00
albertony 01674949a1 hashsum: changed output format when listing algorithms 2025-08-27 23:36:28 +02:00
Claudius Ellsel 98e1d3ee73 docs: add example of how to add date as suffix 2025-08-27 22:01:28 +02:00
Nick Craig-Wood 50d7a80331 box: fix about after change in API return - fixes #8776 2025-08-26 18:03:09 +01:00
Nick Craig-Wood bc3e8e1abd Add skbeh to contributors 2025-08-26 18:03:09 +01:00
Nick Craig-Wood 30e80d0716 Add Tilman Vogel to contributors 2025-08-26 18:03:09 +01:00
albertony f288920696 docs: fix incorrectly escaped windows path separators 2025-08-26 14:29:33 +02:00
albertony fa2bbd705c build: restore error handling in gendocs 2025-08-26 14:28:05 +02:00
skbeh 43a794860f combine: propagate SlowHash feature 2025-08-26 12:39:32 +01:00
albertony adfe6b3bad docs/oracleobjectstorage: add introduction before external links and remove broken link 2025-08-26 12:04:00 +02:00
albertony 091ccb649c docs: fix markdown lint issues in backend docs 2025-08-26 12:04:00 +02:00
albertony 2e02d49578 docs: fix markdown lint issues in command docs 2025-08-26 12:04:00 +02:00
albertony 514535ad46 docs: update markdown code block json indent size 2 2025-08-26 12:04:00 +02:00
Tilman Vogel b010591c96 mount: do not log successful unmount as an error - fixes #8766 2025-08-23 16:30:33 +01:00
Nick Craig-Wood 1aaee9edce Start v1.72.0-DEV development 2025-08-22 17:42:25 +01:00
Nick Craig-Wood 3f0e9f5fca Version v1.71.0 2025-08-22 16:03:16 +01:00
Nick Craig-Wood cfd0d28742 fs: tls: add --client-pass support for encrypted --client-key files
This also widens the supported types

- Unencrypted PKCS#1 ("BEGIN RSA PRIVATE KEY")
- Unencrypted PKCS#8 ("BEGIN PRIVATE KEY")
- Encrypted PKCS#8 ("BEGIN ENCRYPTED PRIVATE KEY")
- Legacy PEM encryption (e.g., DEK-Info headers), which are automatically detected.
2025-08-22 12:19:29 +01:00
Nick Craig-Wood e7a2b322ec ftp: make TLS config default to global TLS config - Fixes #6671
This allows --ca-cert, --client-cert, --no-check-certificate etc to be
used.

This also allows `override.ca_cert = XXX` to be used in the config
file.
2025-08-22 12:19:29 +01:00
Nick Craig-Wood d3a0805a2b fshttp: return *Transport rather than http.RoundTripper from NewTransport
This allows further customization, reading the existing config and is
the Go recommended way "accept interfaces, return structs".
2025-08-22 12:19:29 +01:00
nielash d4edf8ac18 bisync: release from beta
As of v1.71, bisync is officially out of beta.

Some history:

- bisync was born in 2018 as https://github.com/cjnaz/rclonesync-V2
by @cjnaz, written in python.
- In 2021, @ivandeex ported it to go with @cjnaz's support.
https://github.com/rclone/rclone/pull/5164
- It was introduced as an "experimental" feature in v1.58.
https://github.com/rclone/rclone/commit/6210e22ab585eccdf27c12642d342056459e2e6e
- In 2023, bisync needed a new maintainer, and @nielash volunteered.
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636
- Later in 2023, bisync received a major overhaul and was relabeled "beta"
(from "experimental"). https://github.com/rclone/rclone/pull/7410
- In 2024, integration tests were introduced for bisync (which previously had
only unit tests). https://github.com/rclone/rclone/pull/7693
- As of August 2025, bisync is stable and integration tests are passing on all
of the "flagship" backends.

Development doesn't stop here, of course. But bisync has come a long way since
its "experimental" days, and the "beta" tag is no longer needed.
2025-08-22 12:13:59 +01:00
nielash 87d14b000a bisync: fix markdown formatting issues flagged by linter in docs 2025-08-22 12:13:59 +01:00
nielash 12bded980b bisync: fix --no-slow-hash settings on path2
Before this change, if path2 had slow hashes, and --no-slow-hash or --slow-hash-sync-only
was in use, bisync was erroneously setting path1's hashtype to 'none' instead of
path2's. This change fixes the issue.

See https://forum.rclone.org/t/hashtype-mismatch-with-slow-hash-sync-only-in-onedrive-local-bisync/52138/2?u=nielash
2025-08-22 12:13:59 +01:00
Nick Craig-Wood 6e0e76af9d Add cui to contributors 2025-08-22 12:13:59 +01:00
Nick Craig-Wood 6f9b2f7b9b docs: add code of conduct 2025-08-22 11:42:51 +01:00
cui f61d79396d lib/mmap: convert to using unsafe.Slice to avoid deprecated reflect.SliceHeader 2025-08-22 00:35:50 +01:00
dependabot[bot] 9b22e38450 build: bump golangci/golangci-lint-action from 6 to 8
Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 6 to 8.
- [Release notes](https://github.com/golangci/golangci-lint-action/releases)
- [Commits](https://github.com/golangci/golangci-lint-action/compare/v6...v8)

---
updated-dependencies:
- dependency-name: golangci/golangci-lint-action
  dependency-version: '8'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-08-22 00:14:01 +01:00
albertony 9e4fe18830 build: update golangci-lint configuration 2025-08-22 00:14:01 +01:00
albertony ae5cc1ab37 build: ignore revive lint issue var-naming: avoid meaningless package names 2025-08-22 00:14:01 +01:00
albertony d4be38ec02 build: fix lint issue: should omit type error from declaration 2025-08-22 00:14:01 +01:00
albertony 115cff3007 Revert "build: downgrade linter to use go1.24 until it is fixed for go1.25"
This reverts commit 8f84f91666.
2025-08-22 00:14:01 +01:00
albertony 70b862f026 build: migrate golangci-lint configuration to v2 format 2025-08-22 00:14:01 +01:00
Nick Craig-Wood 321cf23e9c s3: add --s3-use-arn-region flag - fixes #8686 2025-08-22 00:02:41 +01:00
Nick Craig-Wood 7e8d4bd915 Add Binbin Qian to contributors 2025-08-22 00:02:41 +01:00
Nick Craig-Wood 06f45e0ac0 Add Lucas Bremgartner to contributors 2025-08-22 00:02:41 +01:00
Binbin Qian 4af2f01abc docs: add tips about outdated certificates 2025-08-21 08:21:02 +02:00
Lucas Bremgartner dd3fff6eae FAQ: specify the availability of SSL_CERT_* env vars
SSL_CERT_FILE and SSL_CERT_DIR env vars are only available on Unix systems other than macOS.

Addressing comment https://github.com/rclone/rclone/pull/1977#issuecomment-3201961570
2025-08-20 12:34:04 +01:00
wiserain ca6631746a pikpak: add file name integrity check during upload
This commit introduces a new validation step to ensure data integrity 
during file uploads.

- The API's returned file name (new.File.Name) is now verified 
  against the requested file name (leaf) immediately after 
  the initial upload ticket is created.
- If a mismatch is detected, the upload process is aborted with an error, 
  and the defer cleanup logic is triggered to delete any partially created file.
- This addresses an unexpected API behavior where numbered suffixes 
  might be appended to filenames even without conflicts.
- This change prevents corrupted or misnamed files from being uploaded 
  without client-side awareness.
2025-08-19 22:00:23 +09:00
nielash e5fe0b1476 bisync: skip TestBisyncConcurrent on non-local
See discussion on
https://github.com/rclone/rclone/pull/8708#discussion_r2280308808
2025-08-18 17:57:14 -04:00
Nick Craig-Wood 4c5764204d internetarchive: fix server side copy files with &
Before this change, server side copy of files with & gave the error:

    Invalid Argument</Message><Resource>x-(amz|archive)-copy-source
    header has bad character

This fix switches to using url.QueryEscape which escapes everything
from url.PathEscape which doesn't escape &.

Fixes #8754
2025-08-18 19:37:30 +01:00
Nick Craig-Wood d70f40229e Revert "s3: set useAlreadyExists to false for Alibaba OSS"
This reverts commit 64ed9b175f.

This fails the integration tests with

s3_internal_test.go:434: Creating a bucket we already have created returned code: No Error
s3_internal_test.go:439:
    	Error Trace:	backend/s3/s3_internal_test.go:439
    	Error:      	Should be true
    	Test:       	TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/Mkdir
    	Messages:   	Need to set UseAlreadyExists quirk
2025-08-18 19:37:30 +01:00
Nick Craig-Wood 05b13b47b5 Add huangnauh to contributors 2025-08-18 19:37:30 +01:00
Sudipto Baral ecd52aa809 smb: improve multithreaded upload performance using multiple connections
In the current design, OpenWriterAt provides the interface for random-access
writes, and openChunkWriterFromOpenWriterAt wraps this interface to enable
parallel chunk uploads using multiple goroutines. A global connection pool is
already in place to manage SMB connections across files.

However, currently only one connection is used per file, which makes multiple
goroutines compete for the connection during multithreaded writes.

This changes create separate connections for each goroutine, which allows true
parallelism by giving each goroutine its own SMB connection

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>
2025-08-18 16:29:18 +01:00
nielash 269abb1aee bisync: fix data races on tests 2025-08-17 20:16:46 -04:00
nielash d91cbb2626 bisync: remove unused parameters 2025-08-17 20:16:46 -04:00
nielash 9073d17313 bisync: deglobalize to fix concurrent runs via rc - fixes #8675
Before this change, bisync used some global variables, which could cause errors
if running multiple concurrent bisync runs through the rc. (Running normally
from the command line was not affected.)

This change deglobalizes those variables so that multiple bisync runs can be
safely run at once, from the same rclone instance.
2025-08-17 20:16:46 -04:00
huangnauh cc20d93f47 mount: fix identification of symlinks in directory listings 2025-08-17 12:57:35 +01:00
Nick Craig-Wood cb1507fa96 s3: fix Content-Type: aws-chunked causing upload errors with --metadata
`Content-Type: aws-chunked` is used on S3 PUT requests to signal SigV4
streaming uploads: the body is sent in AWS-formatted chunks, each
chunk framed and HMAC-signed.

When copying from a non S3 compatible object store (like Digital
Ocean) the objects can have `Content-Type: aws-chunked` (which you
won't see on AWS S3). Attempting to copy these objects to S3 with
`--metadata` this produces this error.

    aws-chunked encoding is not supported when x-amz-content-sha256 UNSIGNED-PAYLOAD is supplied

This patch makes sure `aws-chunked` is removed from the `Content-Type`
metadata both on the way in and the way out.

Fixes #8724
2025-08-16 17:11:54 +01:00
Nick Craig-Wood b0b3b04b3b config: fix problem reading pasted tokens over 4095 bytes
Before this change we were reading input from stdin using the terminal
in the default line mode which has a limit of 4095 characters.

The typical culprit was onedrive tokens (which are very long) giving the error

    Couldn't decode response: invalid character 'e' looking for beginning of value

This change swaps over to use the github.com/peterh/liner read line
library which does not have that limitation and also enables more
sensible cursor editing.

Fixes #8688 #8323 #5835
2025-08-16 16:44:35 +01:00
Nick Craig-Wood 8d878d0a5f config: fix test failure on local machine with a config file
This uses a temporary config file instead.
2025-08-16 16:44:00 +01:00
Nick Craig-Wood 8d353039a6 log: add log rotation to --log-file - fixes #2259 2025-08-16 16:38:23 +01:00
Nick Craig-Wood 4b777db20b accounting: Fix stats (speed=0 and eta=nil) when starting jobs via rc
Before this change we used the current context to start the average
loop. This means that if the context came from the rc the average loop
would be cancelled at the end of the rc request leading the speed not
being measured.

This uses the background context for the accounting loop so it doesn't
get cancelled when its parent gets cancelled.
2025-08-16 16:33:38 +01:00
Nick Craig-Wood 16ad0c2aef docs: update overview table for oracle object storage 2025-08-16 16:00:14 +01:00
Nick Craig-Wood e46dec2a94 Add praveen-solanki-oracle to contributors 2025-08-16 16:00:14 +01:00
praveen-solanki-oracle 2b54b63cb3 oracleobjectstorage: add read only metadata support - Fixes #8705 2025-08-16 15:55:53 +01:00
Nick Craig-Wood f2eb5f35f6 doc: sync doesn't symlinks in dest without --link - Fixes #8749 2025-08-16 09:22:31 +01:00
Nick Craig-Wood d9a36ef45c s3: sort providers in docs 2025-08-15 17:38:31 +01:00
Nick Craig-Wood eade7710e7 s3: add docs for Exaba Object Storage 2025-08-15 17:38:31 +01:00
Nick Craig-Wood e6470d998c azureblob: fix double accounting for multipart uploads - fixes #8718
Before this change multipart uploads using OpenChunkWriter would
account for twice the space used.

This fixes the problem by adjusting the accounting delay.
2025-08-14 16:59:34 +01:00
Nick Craig-Wood 0c0fb93111 pool: fix deadlock with --max-buffer-memory
Before this change we used an overcomplicated method of memory
reservations in the pool.RW which caused deadlocks.

This changes it to use a much simpler reservation system where we
actually reserve the memory and store it in the pool.RW. This allows
us to use the semaphore.Weighted to count the actually memory in use
(rather than the memory in use and in the cache). This in turn allows
accurate use of the semaphore by users wanting memory.
2025-08-14 16:14:59 +01:00
Nick Craig-Wood 3f60764bd4 azureblob: fix deadlock with --max-connections with InvalidBlockOrBlob errors
Before this change the azureblob backend could deadlock when using
--max-connections. This is because when it receives InvalidBlockOrBlob
error it attempts to clear the condition before retrying. This in turn
involved recursively calling the pacer. At this point the pacer can
easily have no connections left which causes a deadlock as all the
other pacer connections are waiting for the InvalidBlockOrBlob to be
resolved.

This fixes the problem by using a temporary pacer when resolving the
InvalidBlockOrBlob errors.
2025-08-14 16:14:59 +01:00
Nick Craig-Wood 8f84f91666 build: downgrade linter to use go1.24 until it is fixed for go1.25 2025-08-13 17:54:45 +01:00
Nick Craig-Wood 2c91772bf1 build: update all dependencies 2025-08-13 17:54:45 +01:00
Nick Craig-Wood c3f721755d build: update to go1.25 and make go1.24 the minimum required version 2025-08-13 17:54:45 +01:00
Nick Craig-Wood 8a952583a5 Add Timothy Jacobs to contributors 2025-08-13 17:54:40 +01:00
nielash fc5bd21e28 bisync: fix time.Local data race on tests - fixes #8272
Before this change, the bisync tests were directly setting the time.Local
variable to UTC.

The reason for overriding the time zone on the tests is to make them
deterministic regardless of where in the world the user happens to be. There are
some goldenized strings which have the time zone hard-coded and would result in a
miscompare failure outside of that time zone.

However, mutating the time.Local variable is not the right way to do this, as OP
correctly pointed out on #8272.

Setting the TZ environment variable from within the code was also not an ideal
solution because, while it worked on unix, it did not work on Windows. See
https://github.com/golang/go/blob/fbac94a79998d4730a58592f0634fa8a39d8b9fb/src/time/zoneinfo.go#L79-L80

This change fixes the issue by defining a new bisync.LogTZ setting for use when
printing timestamps in /cmd/bisync/resolve.go. We override this on the tests
instead of time.Local.
2025-08-13 11:58:35 -04:00
nielash be73a10a97 googlecloudstorage: fix rateLimitExceeded error on bisync tests
Additional to googlecloudstorage's general rate limiting, it apparently has a
separate limit for updating the same object more than once per second:

googleapi: Error 429: The object rclone-test-
demilaf1fexu/015108so/check_access/path2/modtime_write_test exceeded the rate
limit for object mutation operations (create, update, and delete). Please reduce
your request rate. See https://cloud.google.com/storage/docs/gcs429.,
rateLimitExceeded

We were encountering this in the part of the bisync tests where we create an
object, verify that we can edit its modtime, then remove it. We were not
encountering it elsewhere because it only concerns manipulations of the same
object -- not the rate of API calls in general. For the same reason, the standard
pacer is not an effective solution for enforcing this (unless, of course, we
want to slow the entire test down by setting a 1s MinSleep across the board.)

While ideally this would be handled in the backend, this gets around it by
sleeping for 1s in the relevant part of the bisync tests.
2025-08-13 11:58:35 -04:00
Timothy Jacobs 7edf8eb233 accounting: populate transfer snapshot with "what" value 2025-08-13 16:25:38 +01:00
dependabot[bot] 99144dcbba build(deps): bump actions/checkout from 4 to 5
Bumps [actions/checkout](https://github.com/actions/checkout) from 4 to 5.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-08-12 19:39:49 +02:00
dependabot[bot] 8f90f830bd build(deps): bump actions/download-artifact from 4 to 5
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 4 to 5.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v4...v5)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: '5'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-08-12 17:49:55 +02:00
nielash 456108f29e googlecloudstorage: enable bisync integration tests
These were habitually failing at some point and ignored for that reason, but
seem to be passing now. It is possible that in the interim, the underlying issue
was resolved by another commit. If there is still an issue lurking, the nightly
tests will surely reveal it (and give us a log to look at.)
2025-08-09 18:12:17 -04:00
nielash f7968aad1c fstest: fix parsing of commas in -remotes
Connection string remotes like "TestGoogleCloudStorage,directory_markers:" use
commas. Before this change, these could not be passed with the -remotes flag,
which expected commas to be used only as separators.

After this change, CSV parsing is used so that commas will be properly
recognized inside a terminal-escaped and quoted value, like:

-remotes local,\"TestGoogleCloudStorage,directory_markers:\"
2025-08-09 18:12:17 -04:00
nielash 2a587d21c4 azurefiles: fix hash getting erased when modtime is set
Before this change, setting an object's modtime with o.SetModTime() (without
updating the file's content) would inadvertently erase its md5 hash.

The documentation notes: "If this property isn't specified on the request, the
property is cleared for the file. Subsequent calls to Get File Properties won't
return this property, unless it's explicitly set on the file again."
https://learn.microsoft.com/en-us/rest/api/storageservices/set-file-properties#common-request-headers

This change fixes the issue by setting ContentMD5 (and ContentType), to the
extent we have it, during SetModTime.

Discovered on bisync integration tests such as TestBisyncRemoteRemote/resolve
2025-08-09 18:12:17 -04:00
nielash 4b0df05907 bisync: disable --sftp-copy-is-hardlink on sftp tests
Before this change, TestSFTPOpenssh integration tests would fail due to setting
copy_is_hardlink=true in /fstest/testserver/init.d/TestSFTPOpenssh.

For example, if a file was server-side copied from path1 to path2 and then the
bisync tests set the path2 modtime, the path1 modtime would also unexpectedly
mutate.

Hardlinks are not the same as copies. The bisync tests assume that they can
modify a file on one side without affecting a file on the other. This change
essentially sets --sftp-copy-is-hardlink to the default of false for the bisync
tests.
2025-08-09 18:12:17 -04:00
Anagh Kumar Baranwal a92af34825 local: fix --copy-links on Windows when listing Junction points 2025-08-10 00:33:34 +05:30
Nick Craig-Wood 8ffde402f6 operations: fix too many connections open when using --max-memory
Before this change we opened the connection before allocating memory.
This meant a long wait sometimes for memory and too many connections
open.

Now we allocate the memory first before opening the connection.
2025-08-07 12:45:44 +01:00
Nick Craig-Wood 117d8d9fdb pool: fix deadlock with --max-memory and multipart transfers
Because multipart transfers can need more than one buffer to complete,
if transfers was set very high, it was possible for lots of multipart
transfers to start, grab fewer buffers than chunk size, then deadlock
because no more memory was available.

This fixes the problem by introducing a reservation system which the
multipart transfer uses to ensure it can reserve all the memory for
one chunk before starting.
2025-08-07 12:45:44 +01:00
Nick Craig-Wood 5050f42b8b pool: unify memory between multipart and asyncreader to use one pool
Before this the multipart code and asyncreader used separate pools
which is inefficient on memory use.
2025-08-07 12:45:44 +01:00
Nick Craig-Wood fcbcdea067 docs: update links to rcloneui 2025-08-05 16:25:58 +01:00
Nick Craig-Wood d4e68bf66b docs: add MEGA S4 as a gold sponsor
This also tidies the menu cards.
2025-08-01 12:40:29 +01:00
Nick Craig-Wood 743d160fdd about: fix potential overflow of about in various backends
Before this fix it was possible for an about call in various backends
to exceed an int64 and wrap.

This patch causes it to clip to the max int64 value instead.
2025-07-31 11:38:51 +01:00
Nick Craig-Wood dc95f36bc1 box: fix about: cannot unmarshal number 1.0e+18 into Go struct field
Before this change rclone about was failing with

    cannot unmarshal number 1.0e+18 into Go struct field User.space_amount of type int64

As Box increased Enterprise accounts user.space_amount from 30PB to
1e+18 or 888.178PB returning it as a floating point number, not an integer.

This fix reads it as a float64 and clips it to the maximum value of an
int64 if necessary.
2025-07-31 11:38:51 +01:00
Nick Craig-Wood d3e3af377a oauthutil: fix nil pointer crash when started with expired token 2025-07-31 11:38:51 +01:00
n4n5 db4812fbfa rc: listremotes should send an empty array instead of nil 2025-07-25 15:37:25 +01:00
n4n5 ff9cbab5fa config: add error if RCLONE_CONFIG_PASS was supplied but didn't decrypt config 2025-07-25 11:24:18 +01:00
n4n5 30d8ab5f2f rc: add config/unlock to unlock the config file 2025-07-25 11:19:07 +01:00
Anagh Kumar Baranwal d71a4195d6 ftp: allow insecure TLS ciphers - fixes #8701
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2025-07-25 10:30:18 +01:00
zjx20 64ed9b175f s3: set useAlreadyExists to false for Alibaba OSS 2025-07-24 23:22:16 +01:00
Nick Craig-Wood 2b10340e4e docs: update sponsors page 2025-07-24 15:19:15 +01:00
Nick Craig-Wood 3c596f8d11 fs: allow global variables to be overriden or set on backend creation
This allows backend config to contain

- `override.var` - set var during remote creation only
- `global.var` - set var in the global config permanently

Fixes #8563
2025-07-23 15:09:51 +01:00
Nick Craig-Wood 6a9c221841 fs: allow setting of --http_proxy from command line
This in turn allows `override.http_proxy` to be set in backend configs
to set an http proxy for a single backend.
2025-07-23 15:09:51 +01:00
Nick Craig-Wood c49b24ff90 tests: cloudinary: remove test ignore after merging fix from #8707 2025-07-23 13:12:55 +01:00
Nick Craig-Wood edbbfd1e86 Add Antonin Goude to contributors 2025-07-23 13:12:55 +01:00
Nick Craig-Wood 0e0af7499c Add Yu Xin to contributors 2025-07-23 13:12:55 +01:00
Nick Craig-Wood eb4fe3ef4c Add houance to contributors 2025-07-23 13:12:55 +01:00
Nick Craig-Wood 70eb0f21d9 Add Florent Vennetier to contributors 2025-07-23 13:12:55 +01:00
Nick Craig-Wood 12378bae27 Add n4n5 to contributors 2025-07-23 13:12:55 +01:00
Nick Craig-Wood 3c08c4df3a Add Albin Parou to contributors 2025-07-23 13:12:55 +01:00
Nick Craig-Wood 897509ae10 Add liubingrun to contributors 2025-07-23 13:12:55 +01:00
nielash 0eb7ee2e16 sync: fix testLoggerVsLsf when backend only reads modtime
There are some backends (like PikPak) that advertise a precision of
fs.ModTimeNotSupported but do actually return a modtime when asked. In the case
of PikPak, it is because the modtime can be read but not written, and is not
considered reliable enough to use for syncing.

Before this change, testLoggerVsLsf got confused in this scenario (expected a
blank modtime but got non-blank). Adding to the confusion, it only reaches this
code if the backend happens to support md5 hashes, and the fsrc and fdst have
the same precision.

This change fixes the issue by setting the modtime string on both sides to
"none" in this scenario. Note that we can't use "" (blank) because
(operations.ListFormat).AddModTime would replace that with "2006-01-02 15:04:05".
2025-07-23 12:49:52 +01:00
nielash c1ebfb7e04 sync: fix testLoggerVsLsf checking wrong fs
Before this change, two tests (TestServerSideCopyOverSelf and
TestServerSideMoveOverSelf) were checking the wrong Fs in the call to
testLoggerVsLsf. This fixes it by making sure we are testing the same two Fs's
we synced.
2025-07-23 12:49:52 +01:00
Nick Craig-Wood 3d62058693 docs: fix make opengraph tags absolute as not all sites understand relative 2025-07-22 18:00:33 +01:00
albertony 122890799f docs: update contributing guide regarding markdown documentation 2025-07-21 20:23:16 +02:00
albertony 65078d5846 build: add markdown linting to workflow 2025-07-21 20:23:16 +02:00
albertony 92f304902d build: add markdownlint configuration 2025-07-21 20:23:16 +02:00
albertony 45477a6c7d docs: minor format cleanup install.md 2025-07-21 20:23:16 +02:00
albertony 79b549b5a4 docs: fix markdownlint issue md049/emphasis-style 2025-07-21 20:23:16 +02:00
albertony 318880b4ad docs: fix markdownlint issue md036/no-emphasis-as-heading 2025-07-21 20:23:16 +02:00
albertony 75521dcf6e docs: fix markdownlint issue md033/no-inline-html 2025-07-21 20:23:16 +02:00
albertony 8bf20dd545 docs: fix markdownlint issue md025/single-title 2025-07-21 20:23:16 +02:00
albertony 744bce1246 docs: fix markdownlint issue md041/first-line-heading 2025-07-21 20:23:16 +02:00
albertony c817fc5c57 docs: fix markdownlint issue md001/heading-increment 2025-07-21 20:23:16 +02:00
albertony 0bb4d0a985 docs: fix markdownlint issue md003/heading-style 2025-07-21 20:23:16 +02:00
albertony a8605abd34 docs: fix markdownlint issue md034/no-bare-urls 2025-07-21 20:23:16 +02:00
albertony 953fb4490b docs: fix markdownlint issue md010/no-hard-tabs 2025-07-21 20:23:16 +02:00
albertony b17c3d18af docs: fix markdownlint issue md013/line-length 2025-07-21 20:23:16 +02:00
albertony b45580fa19 docs: fix markdownlint issue md038/no-space-in-code 2025-07-21 20:23:16 +02:00
albertony 1c26f40078 docs: fix markdownlint issue md040/fenced-code-language 2025-07-21 20:23:16 +02:00
albertony 667ad093eb docs: fix markdownlint issue md046/code-block-style 2025-07-21 20:23:16 +02:00
albertony 2c369aedf5 docs: fix markdownlint issue md037/no-space-in-emphasis 2025-07-21 20:23:16 +02:00
albertony 7a0d5ab0b4 docs: fix markdownlint issue md059/descriptive-link-text 2025-07-21 20:23:16 +02:00
albertony 75582b804b docs: fix markdownlint issues md007/ul-indent md004/ul-style 2025-07-21 20:23:16 +02:00
albertony 73452551c6 docs: fix markdownlint issue md012/no-multiple-blanks 2025-07-21 20:23:16 +02:00
albertony cb3cf5068b docs: fix markdownlint issue md058/blanks-around-tables 2025-07-21 20:23:16 +02:00
albertony 428f518771 docs: fix markdownlint issue md022/blanks-around-headings 2025-07-21 20:23:16 +02:00
albertony 0411a41e11 docs: fix markdownlint issue md031/blanks-around-fences 2025-07-21 20:23:16 +02:00
albertony 07b37bcd12 docs: fix markdownlint issue md032/blanks-around-lists 2025-07-21 20:23:16 +02:00
albertony 0506826ff5 docs: fix markdownlint issue md009/no-trailing-spaces 2025-07-21 20:23:16 +02:00
albertony 4fcd36a5ab docs: fix markdownlint issue md014/commands-show-output 2025-07-21 20:23:16 +02:00
albertony b2f43f39ba docs: fix markdownlint issues md007/ul-indent md004/ul-style (bin/update-authors.py) 2025-07-21 20:23:16 +02:00
albertony 074d73d12b docs: fix markdownlint issues md007/ul-indent md004/ul-style (authors.md) 2025-07-21 20:23:16 +02:00
Nick Craig-Wood 6457bcf51e docs: add opengraph tags for website social media previews 2025-07-21 17:48:23 +01:00
Nick Craig-Wood 8d12519f3d mount: note that bucket based remotes can use directory markers 2025-07-21 17:48:23 +01:00
wiserain 8a7c401366 pikpak: add docs for methods to clarify name collision handling and restrictions 2025-07-21 17:43:15 +01:00
wiserain 0aae8f346f pikpak: enhance Copy method to handle name collisions and improve error management 2025-07-21 17:43:15 +01:00
wiserain e991328967 pikpak: enhance Move for better handling of error and name collision 2025-07-21 17:43:15 +01:00
Yu Xin 614d02a673 accounting: fix incorrect stats with --transfers=1 - fixes #8670 2025-07-21 16:54:19 +01:00
houance 018ebdded5 rc: fix operations/check ignoring oneWay parameter
Change param from parsing "oneway" to "oneWay" as bool value, as the docs
say "oneWay -  check one way only, source files must exist on remote"
2025-07-21 16:41:08 +01:00
Florent VennetierandAntonin Goude fc08983d71 s3: add OVHcloud Object Storage provider
Co-Authored-By: Antonin Goude <antonin.goude@ovhcloud.com>
2025-07-21 16:34:53 +01:00
n4n5 7b61084891 docs: rc: fix description of how to read local config 2025-07-21 15:42:37 +01:00
albertony d1ac6c2fe1 build: limit check for edits of autogenerated files to only commits in a pull request 2025-07-17 16:20:38 +02:00
albertony da9c99272c build: extend check for edits of autogenerated files to all commits in a pull request 2025-07-17 16:20:38 +02:00
Sudipto Baral 9c7594d78f smb: refresh Kerberos credentials when ccache file changes
This change enhances the SMB backend in Rclone to automatically refresh
Kerberos credentials when the associated ccache file is updated.

Previously, credentials were only loaded once per path and cached
indefinitely, which caused issues when service tickets expired or the
cache was renewed on the server.
2025-07-17 14:34:44 +01:00
Albin Parou 70226cc653 s3: fix multipart upload and server side copy when using bucket policy SSE-C
When uploading or moving data within an s3-compatible bucket, the
`SSECustomer*` headers should always be forwarded: on
`CreateMultipartUpload`, `UploadPart`, `UploadCopyPart` and
`CompleteMultipartUpload`. But currently rclone doesn't forward those
headers to `CompleteMultipartUpload`.

This is a requirement if you want to enforce `SSE-C` at the bucket level
via a bucket policy. Cf: `This parameter is required only when the
object was created using a checksum algorithm or if your bucket policy
requires the use of SSE-C.` in
https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html
2025-07-17 14:29:31 +01:00
liubingrun c20e4bd99c backend/s3: Fix memory leak by cloning strings #8683
This commit addresses a potential memory leak in the S3 backend where
strings extracted from large API responses were keeping the entire
response in memory. The issue occurs because Go strings share underlying
memory with their source, preventing garbage collection of large XML
responses even when only small substrings are needed.

Signed-off-by: liubingrun <liubr1@chinatelecom.cn>
2025-07-17 12:31:52 +01:00
Nick Craig-Wood ccfe153e9b purge: exit with a fatal error if filters are set on rclone purge
Fixes #8491
2025-07-17 11:17:08 +01:00
Nick Craig-Wood c9730bcaaf docs: Add Backblaze as a Platinum sponsor 2025-07-17 11:17:08 +01:00
Nick Craig-Wood 03dd7486c1 Add Sam Pegg to contributors 2025-07-17 11:17:08 +01:00
raider13209 6249009fdf googlephotos: added warning for Google Photos compatability-fixes #8672 2025-07-17 10:48:12 +01:00
Nick Craig-Wood 8e2d76459f test: remove flakey TestChunkerChunk50bYandex: test 2025-07-16 16:39:57 +01:00
albertony 5e539c6a72 docs: Consolidate entries for Josh Soref in contributors 2025-07-13 14:05:45 +02:00
albertony 8866112400 docs: remove dead link to example of writing a plugin 2025-07-13 13:51:38 +02:00
1116 changed files with 220085 additions and 80998 deletions

No files matched your search

+78
View File
@@ -0,0 +1,78 @@
# CODEOWNERS for rclone
#
# Maps paths to the maintainers auto-requested for review when a pull request
# touches matching files. Patterns are evaluated top to bottom and the LAST
# matching pattern wins, so the catch-all below is overridden by the more
# specific subsystem/backend lines that follow.
#
# Every owner listed here has write access to the repo (GitHub ignores owners
# without it). Assignments are derived from `git log` history per directory:
# each area lists its dominant contributor(s) alongside @ncw, the project lead
# and global fallback.
# Global fallback - everything not matched below
* @ncw
# -----------------------------------------------------------------------------
# Core subsystems
# -----------------------------------------------------------------------------
/fs/ @ncw
/vfs/ @ncw
/lib/ @ncw
/cmd/ @ncw
/librclone/ @ncw
/fstest/ @ncw
/lib/encoder/ @ncw @klauspost
/fs/sync/ @ncw @nielash
/fs/operations/ @ncw @nielash
/cmd/bisync/ @nielash @ivandeex @ncw
/cmd/serve/ @albertony @ncw
/cmd/selfupdate/ @ivandeex @ncw
/cmd/mount2/ @ncw @darthShadow
/cmd/gitannex/ @dmcardle @ncw
# -----------------------------------------------------------------------------
# Backends - keyed to the dominant / specialist maintainer per `git log`
# -----------------------------------------------------------------------------
# albertony
/backend/jottacloud/ @albertony @buengese @ncw
/backend/local/ @ncw @albertony @nielash
/backend/netstorage/ @albertony @ncw
/backend/hdfs/ @albertony @ncw
# buengese
/backend/yandex/ @ncw @buengese
/backend/zoho/ @buengese @ncw
/backend/fichier/ @ncw @buengese
/backend/compress/ @ncw @buengese
# ivandeex
/backend/chunker/ @ncw @ivandeex
/backend/mailru/ @ncw @ivandeex
/backend/ftp/ @ncw @ivandeex
/backend/sia/ @ncw @ivandeex
# Other maintainer-owned backends (see MAINTAINERS.md)
/backend/cache/ @ncw @remusb
/backend/union/ @ncw @Max-Sum @albertony
/backend/filescom/ @ncw @childish-sambino
# Microsoft Azure
/backend/azureblob/ @ncw @sandeepkru @darthShadow
# Single-maintainer specialist backends
/backend/onedrive/ @ncw @Cnly @nielash
/backend/iclouddrive/ @lostb1t @ncw
/backend/pikpak/ @wiserain @ncw
/backend/protondrive/ @henrybear327 @ncw
/backend/seafile/ @ncw @creativeprojects
/backend/filen/ @Enduriel @ncw
/backend/storj/ @ncw @calebcase
/backend/combine/ @ncw @nielash
# -----------------------------------------------------------------------------
# Repo plumbing
# -----------------------------------------------------------------------------
/.github/ @ncw
-31
View File
@@ -1,31 +0,0 @@
<!--
Welcome :-) We understand you are having a problem with rclone; we want to help you with that!
If you've just got a question or aren't sure if you've found a bug then please use the rclone forum:
https://forum.rclone.org/
instead of filing an issue for a quick response.
If you are reporting a bug or asking for a new feature then please use one of the templates here:
https://github.com/rclone/rclone/issues/new
otherwise fill in the form below.
Thank you
The Rclone Developers
-->
#### Output of `rclone version`
#### Describe the issue
-74
View File
@@ -1,74 +0,0 @@
---
name: Bug report
about: Report a problem with rclone
---
<!--
We understand you are having a problem with rclone; we want to help you with that!
**STOP and READ**
**YOUR POST WILL BE REMOVED IF IT IS LOW QUALITY**:
Please show the effort you've put into solving the problem and please be specific.
People are volunteering their time to help! Low effort posts are not likely to get good answers!
If you think you might have found a bug, try to replicate it with the latest beta (or stable).
The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
If you can still replicate it or just got a question then please use the rclone forum:
https://forum.rclone.org/
for a quick response instead of filing an issue on this repo.
If nothing else helps, then please fill in the info below which helps us help you.
**DO NOT REDACT** any information except passwords/keys/personal info.
You should use 3 backticks to begin and end your paste to make it readable.
Make sure to include a log obtained with '-vv'.
You can also use '-vv --log-file bug.log' and a service such as https://pastebin.com or https://gist.github.com/
Thank you
The Rclone Developers
-->
#### The associated forum post URL from `https://forum.rclone.org`
#### What is the problem you are having with rclone?
#### What is your rclone version (output from `rclone version`)
#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)
#### Which cloud storage system are you using? (e.g. Google Drive)
#### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)
#### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)
<!--- Please keep the note below for others who read your bug report. -->
#### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
-51
View File
@@ -1,51 +0,0 @@
---
name: Feature request
about: Suggest a new feature or enhancement for rclone
---
<!--
Welcome :-)
So you've got an idea to improve rclone? We love that!
You'll be glad to hear we've incorporated hundreds of ideas from contributors already.
Probably the latest beta (or stable) release has your feature, so try to update your rclone.
The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
If it still isn't there, here is a checklist of things to do:
1. Search the old issues for your idea and +1 or comment on an existing issue if possible.
2. Discuss on the forum: https://forum.rclone.org/
3. Make a feature request issue (this is the right place!).
4. Be prepared to get involved making the feature :-)
Looking forward to your great idea!
The Rclone Developers
-->
#### The associated forum post URL from `https://forum.rclone.org`
#### What is your current rclone version (output from `rclone version`)?
#### What problem are you are trying to solve?
#### How do you think rclone should be changed to solve that?
<!--- Please keep the note below for others who read your feature request. -->
#### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
+124
View File
@@ -0,0 +1,124 @@
name: Bug report
description: Report a reproducible problem with rclone
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
We want to help you with rclone - thank you for taking the time to fill this in!
**This tracker is for reproducible bugs only.** If you have a question, are
not sure whether something is a bug, or want help with a configuration, please
use the [rclone forum](https://forum.rclone.org/) instead - you will get a much
quicker answer there.
Before filing, please **test with the [latest beta or stable](https://rclone.org/commands/rclone_selfupdate/)**
(downloads at <https://beta.rclone.org/>) as your problem may already be fixed.
**Do not redact** any information except passwords, keys and personal info.
- type: checkboxes
id: acknowledgements
attributes:
label: Before you start
description: Please confirm the following. Issues that skip these steps may be closed.
options:
- label: I have searched the [forum](https://forum.rclone.org/) and the existing issues for this problem.
required: true
- label: I have tested with the latest beta or stable release and can still reproduce the problem.
required: true
- label: This is a reproducible bug, not a usage question (questions belong on the forum).
required: true
- type: input
id: forum-url
attributes:
label: Associated forum post URL
description: If you discussed this on the forum first (encouraged), paste the link here.
placeholder: https://forum.rclone.org/t/...
validations:
required: false
- type: textarea
id: problem
attributes:
label: What is the problem you are having with rclone?
description: Describe the problem and the exact steps to reproduce it.
validations:
required: true
- type: textarea
id: version
attributes:
label: rclone version
description: The full output of `rclone version`.
placeholder: |
rclone v1.75.0
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-111-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.4
- go/linking: dynamic
- go/tags: none
validations:
required: true
- type: dropdown
id: os
attributes:
label: Operating system
description: Choose the operating system you are using with rclone.
options:
- Windows
- macOS
- Linux
- FreeBSD
- NetBSD
- OpenBSD
- Plan 9
- Solaris
- Android
- iOS
- Other
validations:
required: true
- type: input
id: backend
attributes:
label: Which cloud storage system are you using?
description: e.g. Google Drive, S3, Dropbox, local disk. List all involved.
validations:
required: true
- type: textarea
id: config
attributes:
label: Output of `rclone config redacted`
description: |
Run `rclone config redacted` (or `rclone config redacted <remote>` for a
single remote). This command automatically replaces passwords and tokens
with `XXX`, so it should be safe to paste but please double-check before posting.
render: ini
validations:
required: true
- type: textarea
id: command
attributes:
label: The command you were trying to run
description: e.g. `rclone copy /tmp remote:tmp`
render: shell
validations:
required: true
- type: textarea
id: log
attributes:
label: A log from the command with the `-vv` flag
description: |
e.g. output from `rclone -vv copy /tmp remote:tmp`. For long logs (> 100 lines) use
`-vv --log-file bug.log` and attach it here or send to a service such as
<https://gist.github.com/> or <https://pastebin.com/> and link it here.
render: shell
validations:
required: true
- type: textarea
id: extra
attributes:
label: Anything else?
description: Anything else that might help us, e.g. a minimal way to replicate the problem.
validations:
required: false
+5 -2
View File
@@ -1,5 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Rclone Forum Community Support
- name: Rclone Forum - questions & support
url: https://forum.rclone.org/
about: Please ask and answer questions here.
about: Have a question, need help, or not sure if it's a bug? Please ask here for a quick answer.
- name: Rclone Documentation
url: https://rclone.org/docs/
about: The docs answer many "how do I..." questions, including per-backend guides.
+68
View File
@@ -0,0 +1,68 @@
name: Feature request
description: Suggest a new feature or enhancement for rclone
labels: ["enhancement"]
body:
- type: markdown
attributes:
value: |
So you've got an idea to improve rclone? We love that - we've already
incorporated hundreds of ideas from contributors!
Please **search first**: your idea may already exist in the latest
[beta or stable](https://rclone.org/commands/rclone_selfupdate/), or already
be requested. It is often worth [discussing on the forum](https://forum.rclone.org/)
to refine the idea or checking to see if rclone can do it already before filing it here.
- type: checkboxes
id: acknowledgements
attributes:
label: Before you start
options:
- label: I have searched the [forum](https://forum.rclone.org/) and the existing issues and this hasn't already been requested.
required: true
- label: I have checked the latest beta and this feature doesn't already exist.
required: true
- type: input
id: forum-url
attributes:
label: Associated forum post URL
description: If you discussed this on the forum first (encouraged), paste the link here.
placeholder: https://forum.rclone.org/t/...
validations:
required: false
- type: textarea
id: version
attributes:
label: rclone version
description: The full output of `rclone version`, not just the version number.
placeholder: |
rclone v1.75.0
- os/version: ubuntu 24.04 (64 bit)
- os/kernel: 6.8.0-111-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.26.4
- go/linking: dynamic
- go/tags: none
validations:
required: true
- type: textarea
id: problem
attributes:
label: What problem are you trying to solve?
description: Describe the use case and what is difficult or impossible today.
validations:
required: true
- type: textarea
id: solution
attributes:
label: How do you think rclone should be changed to solve that?
validations:
required: true
- type: checkboxes
id: involvement
attributes:
label: Getting involved
description: Features are much more likely to happen if someone helps make them.
options:
- label: I'm willing to help implement, test or fund this feature.
required: false
+36 -11
View File
@@ -1,29 +1,54 @@
<!--
Thank you very much for contributing code or documentation to rclone! Please
fill out the following questions to make it easier for us to review your
changes.
Thank you for contributing to rclone!
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
Please discuss anything more than a trivial fix in an issue FIRST - see the
"Linked issue" section below. Then fill in the questions to help us review.
-->
#### What is the purpose of this change?
#### What does this change do?
<!-- Describe the change here. -->
#### Linked issue
<!--
Describe the changes here
Put `Fixes #1234` here if this closes an issue, or `#1234` to link without closing.
IMPORTANT: Anything beyond a trivial fix (typo, doc tweak, small obvious bug fix)
should be discussed and agreed in an issue BEFORE you open a pull request.
Pull requests for larger changes that have not been discussed in an issue first
may be closed. This saves everyone's time if the change needs a different approach
or isn't a good fit.
-->
#### Was the change discussed in an issue or in the forum before?
Fixes #
#### For new or changed backends
<!--
Link issues and relevant forum posts here.
Backend pull requests are often sent without the integration tests having been run.
We cannot merge a backend change until it passes the integration tests.
To be merged, a backend change REQUIRES:
1. A clean run of `go run ./fstest/test_all -backends <remote>` (summarise the
result below or paste a screenshot of the test_all output in the web browser).
2. A test account so the maintainers can run the integration tests daily against
the backend on the integration test server.
If the tests aren't quite passing yet and you need help getting them
to pass, then submit the PR and we can help getting you over the line.
See https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-new-backend
and https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#integration-tests
-->
#### Checklist
- [ ] This change is trivial **OR** it has been discussed and agreed in the linked issue.
- [ ] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [ ] **(If I used AI tools to help write this code)** I have read and understood the [AI-assisted contributions guidance](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#ai-assisted-contributions), and I have tested and take ownership of this change myself.
- [ ] I have added tests for all changes in this PR if appropriate.
- [ ] I have added documentation for the changes if appropriate.
- [ ] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [ ] I'm done, this Pull Request is ready for review :-)
- [ ] **(Backend changes only)** `test_all` passes for this backend and if submitting a new backend can provide a test account for the integration tester - see [CONTRIBUTING.md](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#integration-tests).
- [ ] This Pull Request is ready for review.
+2
View File
@@ -4,3 +4,5 @@ updates:
directory: "/"
schedule:
interval: "daily"
cooldown:
default-days: 7
+157 -25
View File
@@ -19,6 +19,9 @@ on:
type: boolean
default: true
permissions:
contents: read
jobs:
build:
if: inputs.manual || (github.repository == 'rclone/rclone' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name))
@@ -29,12 +32,12 @@ jobs:
strategy:
fail-fast: false
matrix:
job_name: ['linux', 'linux_386', 'mac_amd64', 'mac_arm64', 'windows', 'other_os', 'go1.23']
job_name: ['linux', 'linux_386', 'mac_amd64', 'mac_arm64', 'windows', 'other_os', 'go1.25']
include:
- job_name: linux
os: ubuntu-latest
go: '>=1.24.0-rc.1'
go: '~1.26.5'
gotags: cmount
build_flags: '-include "^linux/"'
check: true
@@ -45,14 +48,14 @@ jobs:
- job_name: linux_386
os: ubuntu-latest
go: '>=1.24.0-rc.1'
go: '~1.26.5'
goarch: 386
gotags: cmount
quicktest: true
- job_name: mac_amd64
os: macos-latest
go: '>=1.24.0-rc.1'
go: '~1.26.5'
gotags: 'cmount'
build_flags: '-include "^darwin/amd64" -cgo'
quicktest: true
@@ -61,14 +64,14 @@ jobs:
- job_name: mac_arm64
os: macos-latest
go: '>=1.24.0-rc.1'
go: '~1.26.5'
gotags: 'cmount'
build_flags: '-include "^darwin/arm64" -cgo -macos-arch arm64 -cgo-cflags=-I/usr/local/include -cgo-ldflags=-L/usr/local/lib'
deploy: true
- job_name: windows
os: windows-latest
go: '>=1.24.0-rc.1'
go: '~1.26.5'
gotags: cmount
cgo: '0'
build_flags: '-include "^windows/"'
@@ -78,14 +81,14 @@ jobs:
- job_name: other_os
os: ubuntu-latest
go: '>=1.24.0-rc.1'
go: '~1.26.5'
build_flags: '-exclude "^(windows/|darwin/|linux/)"'
compile_all: true
deploy: true
- job_name: go1.23
- job_name: go1.25
os: ubuntu-latest
go: '1.23'
go: '~1.25.12'
quicktest: true
racequicktest: true
@@ -94,16 +97,58 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- name: Get runner parameters
id: get-runner-parameters
run: |
echo "year-week=$(date -u "+%Y%V")" >> $GITHUB_OUTPUT
echo "runner-os-version=$ImageOS" >> $GITHUB_OUTPUT
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Install Go
uses: actions/setup-go@v5
id: setup-go
uses: actions/setup-go@v7
with:
go-version: ${{ matrix.go }}
check-latest: true
# Caching is handled explicitly below. setup-go's built-in cache
# keys only on go.sum, so all matrix jobs collide on one key and
# only the first to finish saves - which loses the cross-compile
# build cache.
cache: false
- name: Get Go paths
id: go-paths
run: |
echo "build-cache=$(go env GOCACHE)" >> $GITHUB_OUTPUT
echo "mod-cache=$(go env GOMODCACHE)" >> $GITHUB_OUTPUT
# The module cache holds downloaded source and is identical across
# jobs (it depends only on go.sum, not on OS, arch or Go version), so
# it is shared via a single key to avoid storing the same ~600 MiB in
# every job's cache and overflowing the repo cache limit.
- name: Module cache
uses: actions/cache@v6
with:
path: ${{ steps.go-paths.outputs.mod-cache }}
key: go-mod-${{ steps.get-runner-parameters.outputs.year-week }}-${{ hashFiles('go.sum') }}
restore-keys: |
go-mod-${{ steps.get-runner-parameters.outputs.year-week }}-
go-mod-
# The build cache holds compiled artifacts which are specific to the
# OS, arch and Go version, so it is kept per job.
- name: Build cache
uses: actions/cache@v6
with:
path: ${{ steps.go-paths.outputs.build-cache }}
key: go-build-${{ matrix.job_name }}-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-${{ steps.get-runner-parameters.outputs.year-week }}-${{ hashFiles('go.sum') }}
restore-keys: |
go-build-${{ matrix.job_name }}-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-${{ steps.get-runner-parameters.outputs.year-week }}-
go-build-${{ matrix.job_name }}-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-
- name: Set environment variables
run: |
@@ -120,6 +165,11 @@ jobs:
sudo chown root:$USER /etc/fuse.conf
sudo apt-get update
sudo apt-get install -y fuse3 libfuse-dev rpm pkg-config git-annex git-annex-remote-rclone nfs-common
# The podman static bundle in runner image ubuntu24/20260726.254 and
# later ships a non-setuid fusermount3 in /usr/local/bin which
# shadows the setuid one from the fuse3 package and breaks FUSE
# mounts with "Operation not permitted", so remove it.
sudo rm -f /usr/local/bin/fusermount /usr/local/bin/fusermount3
if: matrix.os == 'ubuntu-latest'
- name: Install Libraries on macOS
@@ -202,6 +252,15 @@ jobs:
# Deploy binaries if enabled in config && not a PR && not a fork
if: env.RCLONE_CONFIG_PASS != '' && matrix.deploy && github.head_ref == '' && github.repository == 'rclone/rclone'
- name: Trim module cache before saving
if: always()
run: |
# The downloaded module .zip archives are not needed to build from
# the already-extracted module cache, so drop them before the cache
# is saved - this roughly halves the module cache size.
modcache=$(go env GOMODCACHE | tr '\\' '/')
find "$modcache/cache/download" -name '*.zip' -type f -delete 2>/dev/null || true
lint:
if: inputs.manual || (github.repository == 'rclone/rclone' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name))
timeout-minutes: 30
@@ -216,36 +275,44 @@ jobs:
echo "runner-os-version=$ImageOS" >> $GITHUB_OUTPUT
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Install Go
id: setup-go
uses: actions/setup-go@v5
uses: actions/setup-go@v7
with:
go-version: '>=1.23.0-rc.1'
go-version: '~1.26.5'
check-latest: true
cache: false
- name: Module cache
uses: actions/cache@v6
with:
path: ~/go/pkg/mod
key: go-mod-${{ steps.get-runner-parameters.outputs.year-week }}-${{ hashFiles('go.sum') }}
restore-keys: |
go-mod-${{ steps.get-runner-parameters.outputs.year-week }}-
go-mod-
- name: Cache
uses: actions/cache@v4
uses: actions/cache@v6
with:
path: |
~/go/pkg/mod
~/.cache/go-build
~/.cache/golangci-lint
key: golangci-lint-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-${{ steps.get-runner-parameters.outputs.year-week }}-${{ hashFiles('go.sum') }}
restore-keys: golangci-lint-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-${{ steps.get-runner-parameters.outputs.year-week }}-
- name: Code quality test (Linux)
uses: golangci/golangci-lint-action@v6
uses: golangci/golangci-lint-action@v9
with:
version: latest
skip-cache: true
- name: Code quality test (Windows)
uses: golangci/golangci-lint-action@v6
uses: golangci/golangci-lint-action@v9
env:
GOOS: "windows"
with:
@@ -253,7 +320,7 @@ jobs:
skip-cache: true
- name: Code quality test (macOS)
uses: golangci/golangci-lint-action@v6
uses: golangci/golangci-lint-action@v9
env:
GOOS: "darwin"
with:
@@ -261,7 +328,7 @@ jobs:
skip-cache: true
- name: Code quality test (FreeBSD)
uses: golangci/golangci-lint-action@v6
uses: golangci/golangci-lint-action@v9
env:
GOOS: "freebsd"
with:
@@ -269,7 +336,7 @@ jobs:
skip-cache: true
- name: Code quality test (OpenBSD)
uses: golangci/golangci-lint-action@v6
uses: golangci/golangci-lint-action@v9
env:
GOOS: "openbsd"
with:
@@ -282,10 +349,32 @@ jobs:
- name: Scan for vulnerabilities
run: govulncheck ./...
- name: Check Markdown format
uses: DavidAnson/markdownlint-cli2-action@v20
with:
globs: |
CONTRIBUTING.md
MAINTAINERS.md
README.md
RELEASE.md
CODE_OF_CONDUCT.md
librclone\README.md
backend\s3\README.md
docs/content/{_index,authors,bugs,changelog,docs,downloads,faq,filtering,gui,install,licence,overview,privacy}.md
- name: Scan edits of autogenerated files
run: bin/check_autogenerated_edits.py
run: bin/check_autogenerated_edits.py 'origin/${{ github.base_ref }}'
if: github.event_name == 'pull_request'
- name: Trim module cache before saving
if: always()
run: |
# The downloaded module .zip archives are not needed to build from
# the already-extracted module cache, so drop them before the cache
# is saved - this roughly halves the module cache size.
modcache=$(go env GOMODCACHE | tr '\\' '/')
find "$modcache/cache/download" -name '*.zip' -type f -delete 2>/dev/null || true
android:
if: inputs.manual || (github.repository == 'rclone/rclone' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name))
timeout-minutes: 30
@@ -293,16 +382,50 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Get runner parameters
id: get-runner-parameters
run: |
echo "year-week=$(date -u "+%Y%V")" >> $GITHUB_OUTPUT
echo "runner-os-version=$ImageOS" >> $GITHUB_OUTPUT
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
fetch-depth: 0
# Upgrade together with NDK version
- name: Set up Go
uses: actions/setup-go@v5
id: setup-go
uses: actions/setup-go@v7
with:
go-version: '>=1.24.0-rc.1'
go-version: '~1.26.5'
# Caching is handled explicitly below to share the module cache
# with the other jobs - see the build job for the rationale.
cache: false
- name: Get Go paths
id: go-paths
run: |
echo "build-cache=$(go env GOCACHE)" >> $GITHUB_OUTPUT
echo "mod-cache=$(go env GOMODCACHE)" >> $GITHUB_OUTPUT
- name: Module cache
uses: actions/cache@v6
with:
path: ${{ steps.go-paths.outputs.mod-cache }}
key: go-mod-${{ steps.get-runner-parameters.outputs.year-week }}-${{ hashFiles('go.sum') }}
restore-keys: |
go-mod-${{ steps.get-runner-parameters.outputs.year-week }}-
go-mod-
- name: Build cache
uses: actions/cache@v6
with:
path: ${{ steps.go-paths.outputs.build-cache }}
key: go-build-android-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-${{ steps.get-runner-parameters.outputs.year-week }}-${{ hashFiles('go.sum') }}
restore-keys: |
go-build-android-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-${{ steps.get-runner-parameters.outputs.year-week }}-
go-build-android-${{ steps.get-runner-parameters.outputs.runner-os-version }}-go${{ steps.setup-go.outputs.go-version }}-
- name: Set global environment variables
run: |
@@ -378,3 +501,12 @@ jobs:
RCLONE_CONFIG_PASS: ${{ secrets.RCLONE_CONFIG_PASS }}
# Upload artifacts if not a PR && not a fork
if: env.RCLONE_CONFIG_PASS != '' && github.head_ref == '' && github.repository == 'rclone/rclone'
- name: Trim module cache before saving
if: always()
run: |
# The downloaded module .zip archives are not needed to build from
# the already-extracted module cache, so drop them before the cache
# is saved - this roughly halves the module cache size.
modcache=$(go env GOMODCACHE | tr '\\' '/')
find "$modcache/cache/download" -name '*.zip' -type f -delete 2>/dev/null || true
@@ -18,6 +18,10 @@ on:
type: boolean
default: true
permissions:
contents: read
packages: write
jobs:
build-image:
if: inputs.manual || (github.repository == 'rclone/rclone' && github.event_name != 'pull_request')
@@ -52,7 +56,7 @@ jobs:
df -h .
- name: Checkout Repository
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
fetch-depth: 0
@@ -92,7 +96,7 @@ jobs:
# There's no way around this, because "ImageOS" is only available to
# processes, but the setup-go action uses it in its key.
id: imageos
uses: actions/github-script@v7
uses: actions/github-script@v9
with:
result-encoding: string
script: |
@@ -100,7 +104,7 @@ jobs:
- name: Extract Metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
env:
DOCKER_METADATA_ANNOTATIONS_LEVELS: manifest,manifest-descriptor # Important for digest annotation (used by Github packages)
with:
@@ -122,14 +126,14 @@ jobs:
type=raw,value=beta,enable={{is_default_branch}}
- name: Setup QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Load Go Build Cache for Docker
id: go-cache
uses: actions/cache@v4
uses: actions/cache@v6
with:
key: ${{ runner.os }}-${{ steps.imageos.outputs.result }}-go-${{ env.CACHE_NAME }}-${{ env.PLATFORM }}-${{ hashFiles('**/go.mod') }}-${{ hashFiles('**/go.sum') }}
restore-keys: |
@@ -148,7 +152,7 @@ jobs:
skip-extraction: ${{ steps.go-cache.outputs.cache-hit }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
# This is the user that triggered the Workflow. In this case, it will
@@ -159,7 +163,7 @@ jobs:
- name: Build and Publish Image Digest
id: build
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
file: Dockerfile
context: .
@@ -183,7 +187,7 @@ jobs:
touch "/tmp/digests/${digest#sha256:}"
- name: Upload Image Digest
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: digests-${{ env.PLATFORM }}
path: /tmp/digests/*
@@ -198,7 +202,7 @@ jobs:
steps:
- name: Download Image Digests
uses: actions/download-artifact@v4
uses: actions/download-artifact@v8
with:
path: /tmp/digests
pattern: digests-*
@@ -210,7 +214,7 @@ jobs:
- name: Extract Metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
env:
DOCKER_METADATA_ANNOTATIONS_LEVELS: index
with:
@@ -261,16 +265,16 @@ jobs:
env.write(f"ANNOTATIONS={annotations_string}\n")
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Login to Docker Hub
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
# This is the user that triggered the Workflow. In this case, it will
@@ -14,6 +14,9 @@ on:
type: boolean
default: true
permissions:
contents: read
jobs:
build_docker_volume_plugin:
if: inputs.manual || github.repository == 'rclone/rclone'
@@ -30,7 +33,7 @@ jobs:
sudo rm -rf /usr/share/dotnet || true
df -h .
- name: Checkout master
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
fetch-depth: 0
- name: Build and publish docker plugin
+3
View File
@@ -4,6 +4,9 @@ on:
issues:
types: [labeled]
permissions:
contents: read
jobs:
notify:
runs-on: ubuntu-latest
+3
View File
@@ -3,6 +3,9 @@ on:
release:
types: [released]
permissions:
contents: read
jobs:
publish:
runs-on: ubuntu-latest
+143 -127
View File
@@ -1,144 +1,160 @@
# golangci-lint configuration options
version: "2"
linters:
# Configure the linter set. To avoid unexpected results the implicit default
# set is ignored and all the ones to use are explicitly enabled.
default: none
enable:
# Default
- errcheck
- goimports
- revive
- ineffassign
- govet
- unconvert
- ineffassign
- staticcheck
- gosimple
- stylecheck
- unused
- misspell
# Additional
- gocritic
#- prealloc
#- maligned
disable-all: true
- misspell
#- prealloc # TODO
- revive
- unconvert
exclusions:
rules:
- linters:
- revive
text: 'var-naming: avoid meaningless package names'
- linters:
- revive
text: 'var-naming: avoid package names that conflict with Go standard library package names'
# Configure checks. Mostly using defaults but with some commented exceptions.
settings:
govet:
enable-all: true
disable:
- fieldalignment
- shadow
staticcheck:
# With staticcheck there is only one setting, so to extend the implicit
# default value it must be explicitly included.
checks:
# Default
- all
- -ST1000
- -ST1003
- -ST1016
- -ST1020
- -ST1021
- -ST1022
# Disable quickfix checks
- -QF*
gocritic:
# With gocritic there are different settings, but since enabled-checks
# and disabled-checks cannot both be set, for full customization the
# alternative is to disable all defaults and explicitly enable the ones
# to use.
disable-all: true
enabled-checks:
#- appendAssign # Skip default
- argOrder
- assignOp
- badCall
- badCond
#- captLocal # Skip default
- caseOrder
- codegenComment
#- commentFormatting # Skip default
- defaultCaseOrder
- deprecatedComment
- dupArg
- dupBranchBody
- dupCase
- dupSubExpr
- elseif
#- exitAfterDefer # Skip default
- flagDeref
- flagName
#- ifElseChain # Skip default
- mapKey
- newDeref
- offBy1
- regexpMust
- ruleguard # Enable additional check that are not enabled by default
#- singleCaseSwitch # Skip default
- sloppyLen
- sloppyTypeAssert
- switchTrue
- typeSwitchVar
- underef
- unlambda
- unslice
- valSwap
- wrapperFunc
settings:
ruleguard:
rules: ${base-path}/bin/rules.go
revive:
# With revive there is in reality only one setting, and when at least one
# rule are specified then only these rules will be considered, defaults
# and all others are then implicitly disabled, so must explicitly enable
# all rules to be used.
rules:
- name: blank-imports
disabled: false
- name: context-as-argument
disabled: false
- name: context-keys-type
disabled: false
- name: dot-imports
disabled: false
#- name: empty-block # Skip default
# disabled: true
- name: error-naming
disabled: false
- name: error-return
disabled: false
- name: error-strings
disabled: false
- name: errorf
disabled: false
- name: exported
disabled: false
#- name: increment-decrement # Skip default
# disabled: true
- name: indent-error-flow
disabled: false
- name: package-comments
disabled: false
- name: range
disabled: false
- name: receiver-naming
disabled: false
#- name: redefines-builtin-id # Skip default
# disabled: true
#- name: superfluous-else # Skip default
# disabled: true
- name: time-naming
disabled: false
- name: unexported-return
disabled: false
#- name: unreachable-code # Skip default
# disabled: true
#- name: unused-parameter # Skip default
# disabled: true
- name: var-declaration
disabled: false
- name: var-naming
disabled: false
formatters:
enable:
- goimports
issues:
# Enable some lints excluded by default
exclude-use-default: false
# Maximum issues count per one linter. Set to 0 to disable. Default is 50.
max-issues-per-linter: 0
# Maximum count of issues with the same text. Set to 0 to disable. Default is 3.
max-same-issues: 0
exclude-rules:
- linters:
- staticcheck
text: 'SA1019: "github.com/rclone/rclone/cmd/serve/httplib" is deprecated'
# don't disable the revive messages about comments on exported functions
include:
- EXC0012
- EXC0013
- EXC0014
- EXC0015
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
# Timeout for total work, e.g. 30s, 5m, 5m30s. Default is 0 (disabled).
timeout: 10m
linters-settings:
revive:
# setting rules seems to disable all the rules, so re-enable them here
rules:
- name: blank-imports
disabled: false
- name: context-as-argument
disabled: false
- name: context-keys-type
disabled: false
- name: dot-imports
disabled: false
- name: empty-block
disabled: true
- name: error-naming
disabled: false
- name: error-return
disabled: false
- name: error-strings
disabled: false
- name: errorf
disabled: false
- name: exported
disabled: false
- name: increment-decrement
disabled: true
- name: indent-error-flow
disabled: false
- name: package-comments
disabled: false
- name: range
disabled: false
- name: receiver-naming
disabled: false
- name: redefines-builtin-id
disabled: true
- name: superfluous-else
disabled: true
- name: time-naming
disabled: false
- name: unexported-return
disabled: false
- name: unreachable-code
disabled: true
- name: unused-parameter
disabled: true
- name: var-declaration
disabled: false
- name: var-naming
disabled: false
stylecheck:
# Only enable the checks performed by the staticcheck stand-alone tool,
# as documented here: https://staticcheck.io/docs/configuration/options/#checks
checks: ["all", "-ST1000", "-ST1003", "-ST1016", "-ST1020", "-ST1021", "-ST1022", "-ST1023"]
gocritic:
# Enable all default checks with some exceptions and some additions (commented).
# Cannot use both enabled-checks and disabled-checks, so must specify all to be used.
disable-all: true
enabled-checks:
#- appendAssign # Enabled by default
- argOrder
- assignOp
- badCall
- badCond
#- captLocal # Enabled by default
- caseOrder
- codegenComment
#- commentFormatting # Enabled by default
- defaultCaseOrder
- deprecatedComment
- dupArg
- dupBranchBody
- dupCase
- dupSubExpr
- elseif
#- exitAfterDefer # Enabled by default
- flagDeref
- flagName
#- ifElseChain # Enabled by default
- mapKey
- newDeref
- offBy1
- regexpMust
- ruleguard # Not enabled by default
#- singleCaseSwitch # Enabled by default
- sloppyLen
- sloppyTypeAssert
- switchTrue
- typeSwitchVar
- underef
- unlambda
- unslice
- valSwap
- wrapperFunc
settings:
ruleguard:
rules: "${configDir}/bin/rules.go"
+72
View File
@@ -0,0 +1,72 @@
default: true
# Use specific styles, to be consistent across all documents.
# Default is to accept any as long as it is consistent within the same document.
heading-style: # MD003
style: atx
ul-style: # MD004
style: dash
hr-style: # MD035
style: ---
code-block-style: # MD046
style: fenced
code-fence-style: # MD048
style: backtick
emphasis-style: # MD049
style: asterisk
strong-style: # MD050
style: asterisk
# Allow multiple headers with same text as long as they are not siblings.
no-duplicate-heading: # MD024
siblings_only: true
# Allow long lines in code blocks and tables.
line-length: # MD013
code_blocks: false
tables: false
# The Markdown files used to generated docs with Hugo contain a top level
# header, even though the YAML front matter has a title property (which is
# used for the HTML document title only). Suppress Markdownlint warning:
# Multiple top-level headings in the same document.
single-title: # MD025
level: 1
front_matter_title:
# The HTML docs generated by Hugo from Markdown files may have slightly
# different header anchors than GitHub rendered Markdown, e.g. Hugo trims
# leading dashes so "--config string" becomes "#config-string" while it is
# "#--config-string" in GitHub preview. When writing links to headers in the
# Markdown files we must use whatever works in the final HTML generated docs.
# Suppress Markdownlint warning: Link fragments should be valid.
link-fragments: false # MD051
# Restrict the languages and language identifiers to use for code blocks.
# We only want those supported by both Hugo and GitHub. These are documented
# here:
# https://gohugo.io/content-management/syntax-highlighting/#languages
# https://docs.github.com//get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#syntax-highlighting
# In addition, we only want to allow identifiers (aliases) that correspond to
# the same language in Hugo and GitHub, and preferably also VSCode and other
# commonly used tools, to avoid confusion. An example of this is that "shell"
# by some are considered an identifier for shell scripts, i.e. an alias for
# "sh", while others consider it an identifier for shell sessions, i.e. an
# alias for "console". Although Hugo and GitHub in this case are consistent and
# have chosen the former, using "sh" instead, and not allowing use of "shell",
# avoids the confusion entirely.
fenced-code-language: # MD040
allowed_languages:
- text
- console
- sh
- bat
- ini
- json
- yaml
- go
- python
- c++
- c#
- java
- powershell
+134
View File
@@ -0,0 +1,134 @@
# AGENTS.md
This file provides guidance to AI coding agents (e.g. Claude Code, Codex, Cursor, Gemini CLI, and similar tools) when working with code in this repository.
Rclone welcomes AI-assisted contributions, but the expectation is that you, the human submitter, understand every line you propose and have compiled and tested it against real rclone code - not just generated it. See the "AI-assisted contributions" section of [CONTRIBUTING.md](CONTRIBUTING.md) before opening a pull request.
## Project Overview
Rclone is a command-line program to sync files and directories to and from cloud storage providers. It's written in Go and supports 70+ backends (cloud storage systems). Think "rsync for cloud storage".
## General Notes
**We take backwards compatibility very seriously.** PRs should not change the observable behaviour of existing commands, flags or rc API without very good reason. Rclone does not try to preserve a stable Go API but try not to change it gratuitously.
Rclone operates with a lot of different backends, so **compatibility is key**. It is the backend integration tests which guarantee that compatibility. Changes should consider both known and unknown backends and should take care not to break functionality of existing installations.
The core parts of rclone under `fs` and `vfs` need to work with all backends and **backend specific hacks won't be merged**. Fixes likely need to go in the relevant backend or if new behaviour is really needed, a new Feature flag needs to be added.
**Changes should be kept to the minimum.** Work hard to make the most elegant, smallest change you can. Do not refactor or re-order code unless necessary as this makes review more challenging. Re-use existing test scaffolding, existing or library routines (e.g. `lib`) where possible.
Make sure added tests **actually test the code you have written** and test the intention behind the change. If you are fixing a problem, write the tests first to reproduce the problem before starting on the fix.
## Build and Test Commands
```bash
# Build rclone (simple)
go build
# Build with version info (preferred)
make
# Run all unit tests (no cloud credentials needed)
make quicktest
# or equivalently:
RCLONE_CONFIG="/notfound" go test ./...
# Run tests for a specific package
cd backend/memory && go test -v
# or from root:
go test -v ./backend/memory/
# Run a single test
go test -v -run TestIntegration/FsCheckWrap ./backend/memory/
# Run tests with race detector
make racequicktest
# Lint (requires golangci-lint)
golangci-lint run ./...
# Run backend integration tests (requires configured TestRemote remote)
cd backend/drive && go test -v
# Run sync/operations integration tests against a remote
cd fs/sync && go test -v -remote TestDrive:
cd fs/operations && go test -v -remote TestDrive:
# Run integration tests via test framework
go run ./fstest/test_all -backends drive
```
## Architecture
### Entry Point and Plugin Registration
`rclone.go` is the main entry point. It imports `backend/all` and `cmd/all` which use Go's `init()` pattern to register all backends and commands. Each backend calls `fs.Register()` with a `fs.RegInfo` struct during init.
### Core Interfaces (`fs/`)
The `fs` package defines the core abstractions:
- **`fs.Fs`** (`fs/types.go`): The filesystem interface every backend must implement (List, NewObject, Put, Mkdir, Rmdir).
- **`fs.Object`** (`fs/types.go`): Interface for a file/object (Open, Update, Remove, SetModTime).
- **`fs.Features`** (`fs/features.go`): Optional capabilities a backend can declare (Purge, Copy, Move, DirMove, etc.). Backends set function pointers for operations they support; nil means not supported.
- **`fs.RegInfo`** (`fs/registry.go`): Registration metadata for a backend including its name, config options, and NewFs constructor.
### Backend Structure (`backend/`)
Each backend is a single Go package (e.g., `backend/s3/`, `backend/drive/`). Key conventions:
- Main implementation in a single file (e.g., `s3.go`) - **do not** split into `fs.go`/`object.go`.
- API types go in a separate `api/types.go` file.
- Test file (e.g., `s3_test.go`) uses `fstests.Run()` from `fstest/fstests` for standardized integration tests.
- Register in `backend/all/all.go` via blank import.
- HTTP-based backends should use `lib/rest` for HTTP calls and `fs/fshttp` for the HTTP client.
- Use `lib/dircache` for directory-ID-based remotes, `lib/oauthutil` for OAuth, `lib/pacer` for rate limiting.
### Command Structure (`cmd/`)
Each command is a package under `cmd/` registered in `cmd/all/all.go` via blank import. Commands use cobra via `cmd.Main()`.
### Key Subsystems
- **`fs/operations/`**: Core file operations (Copy, Move, Delete, etc.)
- **`fs/sync/`**: Directory sync logic
- **`fs/march/`**: Parallel directory tree walker used by sync
- **`fs/filter/`**: Include/exclude filtering
- **`fs/accounting/`**: Transfer statistics and bandwidth limiting
- **`fs/config/`**: Configuration file management
- **`vfs/`**: Virtual filesystem layer (used by mount, serve)
- **`librclone/`**: C-compatible library interface for embedding rclone
- **`fstest/`**: Integration test framework; `fstest/fstests/` has the generic backend test suite
## Commit Message Convention
Prefix with the directory of the change, then a colon: `drive: add team drive support - fixes #885`. For cross-cutting changes use a broader prefix like `fs` or `operations`.
Make the first line of your commit message a summary of the change that a user (not a developer) of rclone would like to read. So write `drive: fix server side copy of big files` instead of `drive: no longer set the MimeType in Move or Copy`. This is important because these lines go into the change log which is read by users.
## Code Commenting Style
Comments describe the code as it is now, for a future reader who has no knowledge of the change that introduced it.
Every exported type, function, field, and constant has a **godoc comment** that starts with its name and is phrased as a present-tense statement of what it is or does (`// Mkdir makes the directory (container, bucket)`)
**Document the contract** callers need - preconditions, what's returned, which sentinel errors are returned and when, and any "shouldn't return an error if it already exists" style caveats - rather than the implementation.
**Keep comments terse**: a single line for most things, with extra paragraphs (separated by blank `//` lines) reserved for genuine subtlety.
Inline comments inside function bodies should **explain why** - a non-obvious API quirk, a workaround, a gotcha, or an ordering constraint - and may cite an external reference (forum thread, vendor docs, RFC) when that's what makes the behaviour non-obvious; skip comments that merely restate what the code plainly does.
Use `FIXME` and `TODO` for known shortcomings.
**Do not write comments that narrate the change itself** or compare against the previous behaviour (no "now we also handle...", "changed to...", "previously this returned...", or references to bug/PR numbers in the code) - that context belongs in the commit message, not in source that will outlive the change.
Code comments should only refer to **current** behaviour - and shouldn't describe behaviour that is trivially deducible from reading the code.
## Linting Configuration
Uses golangci-lint v2 with config in `.golangci.yml`. Enabled linters: errcheck, govet, ineffassign, staticcheck, unused, gocritic, misspell, revive, unconvert. The `goimports` formatter is also enabled.
## Documentation
- Backend option docs come from `Help:` fields in the Go source Options structs, not from markdown files.
- Command docs are in the command source code (e.g., `cmd/ls/ls.go`).
- Don't commit autogenerated doc changes from `make backenddocs` or `make commanddocs`.
- Website docs are in `docs/content/` as markdown, built with Hugo (`make serve` to preview).
+6
View File
@@ -0,0 +1,6 @@
# CLAUDE.md
Guidance for this repository lives in AGENTS.md, shared with all AI coding
agents. Claude Code reads it via the import below.
@AGENTS.md
+80
View File
@@ -0,0 +1,80 @@
# Rclone Code of Conduct
Like the technical community as a whole, the Rclone team and community
is made up of a mixture of professionals and volunteers from all over
the world, working on every aspect of the mission - including
mentorship, teaching, and connecting people.
Diversity is one of our huge strengths, but it can also lead to
communication issues and unhappiness. To that end, we have a few
ground rules that we ask people to adhere to. This code applies
equally to founders, mentors and those seeking help and guidance.
This isn't an exhaustive list of things that you can't do. Rather,
take it in the spirit in which it's intended - a guide to make it
easier to enrich all of us and the technical communities in which we
participate.
This code of conduct applies to all spaces managed by the Rclone
project or Rclone Services Ltd. This includes the issue tracker, the
forum, the GitHub site, the wiki, any other online services or
in-person events. In addition, violations of this code outside these
spaces may affect a person's ability to participate within them.
- **Be friendly and patient.**
- **Be welcoming.** We strive to be a community that welcomes and
supports people of all backgrounds and identities. This includes,
but is not limited to members of any race, ethnicity, culture,
national origin, colour, immigration status, social and economic
class, educational level, sex, sexual orientation, gender identity
and expression, age, size, family status, political belief,
religion, and mental and physical ability.
- **Be considerate.** Your work will be used by other people, and you
in turn will depend on the work of others. Any decision you take
will affect users and colleagues, and you should take those
consequences into account when making decisions. Remember that we're
a world-wide community, so you might not be communicating in someone
else's primary language.
- **Be respectful.** Not all of us will agree all the time, but
disagreement is no excuse for poor behavior and poor manners. We
might all experience some frustration now and then, but we cannot
allow that frustration to turn into a personal attack. It's
important to remember that a community where people feel
uncomfortable or threatened is not a productive one. Members of the
Rclone community should be respectful when dealing with other
members as well as with people outside the Rclone community.
- **Be careful in the words that you choose.** We are a community of
professionals, and we conduct ourselves professionally. Be kind to
others. Do not insult or put down other participants. Harassment and
other exclusionary behavior aren't acceptable. This includes, but is
not limited to:
- Violent threats or language directed against another person.
- Discriminatory jokes and language.
- Posting sexually explicit or violent material.
- Posting (or threatening to post) other people's personally
identifying information ("doxing").
- Personal insults, especially those using racist or sexist terms.
- Unwelcome sexual attention.
- Advocating for, or encouraging, any of the above behavior.
- Repeated harassment of others. In general, if someone asks you to
stop, then stop.
- **When we disagree, try to understand why.** Disagreements, both
social and technical, happen all the time and Rclone is no
exception. It is important that we resolve disagreements and
differing views constructively. Remember that we're different. The
strength of Rclone comes from its varied community, people from a
wide range of backgrounds. Different people have different
perspectives on issues. Being unable to understand why someone holds
a viewpoint doesn't mean that they're wrong. Don't forget that it is
human to err and blaming each other doesn't get us anywhere.
Instead, focus on helping to resolve issues and learning from
mistakes.
If you believe someone is violating the code of conduct, we ask that
you report it by emailing [info@rclone.com](mailto:info@rclone.com).
Original text courtesy of the [Speak Up! project](http://web.archive.org/web/20141109123859/http://speakup.io/coc.html).
## Questions?
If you have questions, please feel free to [contact us](mailto:info@rclone.com).
+382 -222
View File
@@ -15,61 +15,81 @@ with the [latest beta of rclone](https://beta.rclone.org/):
- Rclone version (e.g. output from `rclone version`)
- Which OS you are using and how many bits (e.g. Windows 10, 64 bit)
- The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)
- A log of the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)
- if the log contains secrets then edit the file with a text editor first to obscure them
- A log of the command with the `-vv` flag (e.g. output from
`rclone -vv copy /tmp remote:tmp`)
- if the log contains secrets then edit the file with a text editor first to
obscure them
## Submitting a new feature or bug fix
If you find a bug that you'd like to fix, or a new feature that you'd
like to implement then please submit a pull request via GitHub.
If it is a big feature, then [make an issue](https://github.com/rclone/rclone/issues) first so it can be discussed.
If it is a big feature, then [make an issue](https://github.com/rclone/rclone/issues)
first so it can be discussed.
To prepare your pull request first press the fork button on [rclone's GitHub
page](https://github.com/rclone/rclone).
Then [install Git](https://git-scm.com/downloads) and set your public contribution [name](https://docs.github.com/en/github/getting-started-with-github/setting-your-username-in-git) and [email](https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/setting-your-commit-email-address#setting-your-commit-email-address-in-git).
Then [install Git](https://git-scm.com/downloads) and set your public contribution
[name](https://docs.github.com/en/github/getting-started-with-github/setting-your-username-in-git)
and [email](https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/setting-your-commit-email-address#setting-your-commit-email-address-in-git).
Next open your terminal, change directory to your preferred folder and initialise your local rclone project:
Next open your terminal, change directory to your preferred folder and initialise
your local rclone project:
git clone https://github.com/rclone/rclone.git
cd rclone
git remote rename origin upstream
# if you have SSH keys setup in your GitHub account:
git remote add origin git@github.com:YOURUSER/rclone.git
# otherwise:
git remote add origin https://github.com/YOURUSER/rclone.git
```console
git clone https://github.com/rclone/rclone.git
cd rclone
git remote rename origin upstream
# if you have SSH keys setup in your GitHub account:
git remote add origin git@github.com:YOURUSER/rclone.git
# otherwise:
git remote add origin https://github.com/YOURUSER/rclone.git
```
Note that most of the terminal commands in the rest of this guide must be executed from the rclone folder created above.
Note that most of the terminal commands in the rest of this guide must be
executed from the rclone folder created above.
Now [install Go](https://golang.org/doc/install) and verify your installation:
go version
```console
go version
```
Great, you can now compile and execute your own version of rclone:
go build
./rclone version
```console
go build
./rclone version
```
(Note that you can also replace `go build` with `make`, which will include a
more accurate version number in the executable as well as enable you to specify
more build options.) Finally make a branch to add your new feature
git checkout -b my-new-feature
```console
git checkout -b my-new-feature
```
And get hacking.
You may like one of the [popular editors/IDE's for Go](https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins) and a quick view on the rclone [code organisation](#code-organisation).
You may like one of the [popular editors/IDE's for Go](https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins)
and a quick view on the rclone [code organisation](#code-organisation).
When ready - test the affected functionality and run the unit tests for the code you changed
When ready - test the affected functionality and run the unit tests for the
code you changed
cd folder/with/changed/files
go test -v
```console
cd folder/with/changed/files
go test -v
```
Note that you may need to make a test remote, e.g. `TestSwift` for some
of the unit tests.
This is typically enough if you made a simple bug fix, otherwise please read the rclone [testing](#testing) section too.
This is typically enough if you made a simple bug fix, otherwise please read
the rclone [testing](#testing) section too.
Make sure you
@@ -79,14 +99,49 @@ Make sure you
When you are done with that push your changes to GitHub:
git push -u origin my-new-feature
```console
git push -u origin my-new-feature
```
and open the GitHub website to [create your pull
request](https://help.github.com/articles/creating-a-pull-request/).
Your changes will then get reviewed and you might get asked to fix some stuff. If so, then make the changes in the same branch, commit and push your updates to GitHub.
Your changes will then get reviewed and you might get asked to fix some stuff.
If so, then make the changes in the same branch, commit and push your updates to
GitHub.
You may sometimes be asked to [base your changes on the latest master](#basing-your-changes-on-the-latest-master) or [squash your commits](#squashing-your-commits).
You may sometimes be asked to [base your changes on the latest master](#basing-your-changes-on-the-latest-master)
or [squash your commits](#squashing-your-commits).
## AI-assisted contributions
You are welcome to use AI coding assistants (Claude Code, Codex, Cursor, Gemini
CLI, and similar) to help write your contribution. Rclone has an
[AGENTS.md](AGENTS.md) file at the top of the repository describing the project's
conventions; point your tool at it so the code it produces matches rclone's
style.
However, the same standard applies to every pull request whether or not a tool
was involved: **you are responsible for the code you submit.** Before you open a
pull request please make sure that:
- You understand every line of the change and can explain why it is correct. If
a reviewer asks a question about it, you should be able to answer without
going back to the tool.
- You have actually built and run it. At a minimum `go build` and
`make quicktest` must pass, and for a backend change you should run the
[backend tests](#backend-testing) against a real remote where you can.
- The change is a genuine fix or feature that you have verified solves the
problem, not a plausible-looking guess. Unverified, AI-generated pull requests
that do not compile, do not pass the tests, invent APIs that do not exist, or
do not actually do what the description claims waste maintainer time and are
likely to be closed.
- You have trimmed the comments. AI tools tend to add verbose comments that
restate what the code plainly does or narrate the change being made; please
cut these down to match the [code commenting style](AGENTS.md#code-commenting-style).
In short: an AI assistant is a tool to help *you* contribute, not a substitute
for understanding and testing your own work.
## Using Git and GitHub
@@ -94,87 +149,118 @@ You may sometimes be asked to [base your changes on the latest master](#basing-y
Follow the guideline for [commit messages](#commit-messages) and then:
git checkout my-new-feature # To switch to your branch
git status # To see the new and changed files
git add FILENAME # To select FILENAME for the commit
git status # To verify the changes to be committed
git commit # To do the commit
git log # To verify the commit. Use q to quit the log
```console
git checkout my-new-feature # To switch to your branch
git status # To see the new and changed files
git add FILENAME # To select FILENAME for the commit
git status # To verify the changes to be committed
git commit # To do the commit
git log # To verify the commit. Use q to quit the log
```
You can modify the message or changes in the latest commit using:
git commit --amend
```console
git commit --amend
```
If you amend to commits that have been pushed to GitHub, then you will have to [replace your previously pushed commits](#replacing-your-previously-pushed-commits).
If you amend to commits that have been pushed to GitHub, then you will have to
[replace your previously pushed commits](#replacing-your-previously-pushed-commits).
### Replacing your previously pushed commits
Note that you are about to rewrite the GitHub history of your branch. It is good practice to involve your collaborators before modifying commits that have been pushed to GitHub.
Note that you are about to rewrite the GitHub history of your branch. It is good
practice to involve your collaborators before modifying commits that have been
pushed to GitHub.
Your previously pushed commits are replaced by:
git push --force origin my-new-feature
```console
git push --force origin my-new-feature
```
### Basing your changes on the latest master
To base your changes on the latest version of the [rclone master](https://github.com/rclone/rclone/tree/master) (upstream):
To base your changes on the latest version of the
[rclone master](https://github.com/rclone/rclone/tree/master) (upstream):
git checkout master
git fetch upstream
git merge --ff-only
git push origin --follow-tags # optional update of your fork in GitHub
git checkout my-new-feature
git rebase master
```console
git checkout master
git fetch upstream
git merge --ff-only
git push origin --follow-tags # optional update of your fork in GitHub
git checkout my-new-feature
git rebase master
```
If you rebase commits that have been pushed to GitHub, then you will have to [replace your previously pushed commits](#replacing-your-previously-pushed-commits).
If you rebase commits that have been pushed to GitHub, then you will have to
[replace your previously pushed commits](#replacing-your-previously-pushed-commits).
### Squashing your commits ###
### Squashing your commits
To combine your commits into one commit:
git log # To count the commits to squash, e.g. the last 2
git reset --soft HEAD~2 # To undo the 2 latest commits
git status # To check everything is as expected
```console
git log # To count the commits to squash, e.g. the last 2
git reset --soft HEAD~2 # To undo the 2 latest commits
git status # To check everything is as expected
```
If everything is fine, then make the new combined commit:
git commit # To commit the undone commits as one
```console
git commit # To commit the undone commits as one
```
otherwise, you may roll back using:
git reflog # To check that HEAD{1} is your previous state
git reset --soft 'HEAD@{1}' # To roll back to your previous state
```console
git reflog # To check that HEAD{1} is your previous state
git reset --soft 'HEAD@{1}' # To roll back to your previous state
```
If you squash commits that have been pushed to GitHub, then you will have to [replace your previously pushed commits](#replacing-your-previously-pushed-commits).
If you squash commits that have been pushed to GitHub, then you will have to
[replace your previously pushed commits](#replacing-your-previously-pushed-commits).
Tip: You may like to use `git rebase -i master` if you are experienced or have a more complex situation.
Tip: You may like to use `git rebase -i master` if you are experienced or have a
more complex situation.
### GitHub Continuous Integration
rclone currently uses [GitHub Actions](https://github.com/rclone/rclone/actions) to build and test the project, which should be automatically available for your fork too from the `Actions` tab in your repository.
rclone currently uses [GitHub Actions](https://github.com/rclone/rclone/actions)
to build and test the project, which should be automatically available for your
fork too from the `Actions` tab in your repository.
## Testing
### Code quality tests
If you install [golangci-lint](https://github.com/golangci/golangci-lint) then you can run the same tests as get run in the CI which can be very helpful.
If you install [golangci-lint](https://github.com/golangci/golangci-lint) then
you can run the same tests as get run in the CI which can be very helpful.
You can run them with `make check` or with `golangci-lint run ./...`.
Using these tests ensures that the rclone codebase all uses the same coding standards. These tests also check for easy mistakes to make (like forgetting to check an error return).
Using these tests ensures that the rclone codebase all uses the same coding
standards. These tests also check for easy mistakes to make (like forgetting
to check an error return).
### Quick testing
rclone's tests are run from the go testing framework, so at the top
level you can run this to run all the tests.
go test -v ./...
```console
go test -v ./...
```
You can also use `make`, if supported by your platform
make quicktest
```console
make quicktest
```
The quicktest is [automatically run by GitHub](#github-continuous-integration) when you push your branch to GitHub.
The quicktest is [automatically run by GitHub](#github-continuous-integration)
when you push your branch to GitHub.
### Backend testing
@@ -190,41 +276,50 @@ need to make a remote called `TestDrive`.
You can then run the unit tests in the drive directory. These tests
are skipped if `TestDrive:` isn't defined.
cd backend/drive
go test -v
```console
cd backend/drive
go test -v
```
You can then run the integration tests which test all of rclone's
operations. Normally these get run against the local file system,
but they can be run against any of the remotes.
cd fs/sync
go test -v -remote TestDrive:
go test -v -remote TestDrive: -fast-list
```console
cd fs/sync
go test -v -remote TestDrive:
go test -v -remote TestDrive: -fast-list
cd fs/operations
go test -v -remote TestDrive:
cd fs/operations
go test -v -remote TestDrive:
```
If you want to use the integration test framework to run these tests
altogether with an HTML report and test retries then from the
project root:
go install github.com/rclone/rclone/fstest/test_all
test_all -backends drive
```console
go run ./fstest/test_all -backends drive
```
### Full integration testing
If you want to run all the integration tests against all the remotes,
then change into the project root and run
make check
make test
```console
make check
make test
```
The commands may require some extra go packages which you can install with
make build_dep
```console
make build_dep
```
The full integration tests are run daily on the integration test server. You can
find the results at https://pub.rclone.org/integration-tests/
find the results at <https://integration.rclone.org>
## Code Organisation
@@ -232,46 +327,48 @@ Rclone code is organised into a small number of top level directories
with modules beneath.
- backend - the rclone backends for interfacing to cloud providers -
- all - import this to load all the cloud providers
- ...providers
- all - import this to load all the cloud providers
- ...providers
- bin - scripts for use while building or maintaining rclone
- cmd - the rclone commands
- all - import this to load all the commands
- ...commands
- all - import this to load all the commands
- ...commands
- cmdtest - end-to-end tests of commands, flags, environment variables,...
- docs - the documentation and website
- content - adjust these docs only - everything else is autogenerated
- command - these are auto-generated - edit the corresponding .go file
- content - adjust these docs only, except those marked autogenerated
or portions marked autogenerated where the corresponding .go file must be
edited instead, and everything else is autogenerated
- commands - these are auto-generated, edit the corresponding .go file
- fs - main rclone definitions - minimal amount of code
- accounting - bandwidth limiting and statistics
- asyncreader - an io.Reader which reads ahead
- config - manage the config file and flags
- driveletter - detect if a name is a drive letter
- filter - implements include/exclude filtering
- fserrors - rclone specific error handling
- fshttp - http handling for rclone
- fspath - path handling for rclone
- hash - defines rclone's hash types and functions
- list - list a remote
- log - logging facilities
- march - iterates directories in lock step
- object - in memory Fs objects
- operations - primitives for sync, e.g. Copy, Move
- sync - sync directories
- walk - walk a directory
- accounting - bandwidth limiting and statistics
- asyncreader - an io.Reader which reads ahead
- config - manage the config file and flags
- driveletter - detect if a name is a drive letter
- filter - implements include/exclude filtering
- fserrors - rclone specific error handling
- fshttp - http handling for rclone
- fspath - path handling for rclone
- hash - defines rclone's hash types and functions
- list - list a remote
- log - logging facilities
- march - iterates directories in lock step
- object - in memory Fs objects
- operations - primitives for sync, e.g. Copy, Move
- sync - sync directories
- walk - walk a directory
- fstest - provides integration test framework
- fstests - integration tests for the backends
- mockdir - mocks an fs.Directory
- mockobject - mocks an fs.Object
- test_all - Runs integration tests for everything
- fstests - integration tests for the backends
- mockdir - mocks an fs.Directory
- mockobject - mocks an fs.Object
- test_all - Runs integration tests for everything
- graphics - the images used in the website, etc.
- lib - libraries used by the backend
- atexit - register functions to run when rclone exits
- dircache - directory ID to name caching
- oauthutil - helpers for using oauth
- pacer - retries with backoff and paces operations
- readers - a selection of useful io.Readers
- rest - a thin abstraction over net/http for REST
- atexit - register functions to run when rclone exits
- dircache - directory ID to name caching
- oauthutil - helpers for using oauth
- pacer - retries with backoff and paces operations
- readers - a selection of useful io.Readers
- rest - a thin abstraction over net/http for REST
- librclone - in memory interface to rclone's API for embedding rclone
- vfs - Virtual FileSystem layer for implementing rclone mount and similar
@@ -279,47 +376,109 @@ with modules beneath.
If you are adding a new feature then please update the documentation.
The documentation sources are generally in Markdown format, in conformance
with the CommonMark specification and compatible with GitHub Flavored
Markdown (GFM). The markdown format and style is checked as part of the lint
operation that runs automatically on pull requests, to enforce standards and
consistency. This is based on the [markdownlint](https://github.com/DavidAnson/markdownlint)
tool by David Anson, which can also be integrated into editors so you can
perform the same checks while writing. It generally follows Ciro Santilli's
[Markdown Style Guide](https://cirosantilli.com/markdown-style-guide), which
is good source if you want to know more.
HTML pages, served as website <rclone.org>, are generated from the Markdown,
using [Hugo](https://gohugo.io). Note that when generating the HTML pages,
there is currently used a different algorithm for generating header anchors
than what GitHub uses for its Markdown rendering. For example, in the HTML docs
generated by Hugo any leading `-` characters are ignored, which means when
linking to a header with text `--config string` we therefore need to use the
link `#config-string` in our Markdown source, which will not work in GitHub's
preview where `#--config-string` would be the correct link.
Most of the documentation are written directly in text files with extension
`.md`, mainly within folder `docs/content`. Note that several of such files
are autogenerated (e.g. the command documentation, and `docs/content/flags.md`),
or contain autogenerated portions (e.g. the backend documentation under
`docs/content/commands`). These are marked with an `autogenerated` comment.
The sources of the autogenerated text are usually Markdown formatted text
embedded as string values in the Go source code, so you need to locate these
and edit the `.go` file instead. The `MANUAL.*`, `rclone.1` and other text
files in the root of the repository are also autogenerated. The autogeneration
of files, and the website, will be done during the release process. See the
`make doc` and `make website` targets in the Makefile if you are interested in
how. You don't need to run these when adding a feature.
If you add a new general flag (not for a backend), then document it in
`docs/content/docs.md` - the flags there are supposed to be in
alphabetical order.
If you add a new backend option/flag, then it should be documented in
the source file in the `Help:` field.
the source file in the `Help:` field:
- Start with the most important information about the option,
as a single sentence on a single line.
- This text will be used for the command-line flag help.
- It will be combined with other information, such as any default value,
and the result will look odd if not written as a single sentence.
- It should end with a period/full stop character, which will be shown
in docs but automatically removed when producing the flag help.
- Try to keep it below 80 characters, to reduce text wrapping in the terminal.
as a single sentence on a single line.
- This text will be used for the command-line flag help.
- It will be combined with other information, such as any default value,
and the result will look odd if not written as a single sentence.
- It should end with a period/full stop character, which will be shown
in docs but automatically removed when producing the flag help.
- Try to keep it below 80 characters, to reduce text wrapping in the terminal.
- More details can be added in a new paragraph, after an empty line (`"\n\n"`).
- Like with docs generated from Markdown, a single line break is ignored
and two line breaks creates a new paragraph.
- This text will be shown to the user in `rclone config`
and in the docs (where it will be added by `make backenddocs`,
normally run some time before next release).
- Like with docs generated from Markdown, a single line break is ignored
and two line breaks creates a new paragraph.
- This text will be shown to the user in `rclone config`
and in the docs (where it will be added by `make backenddocs`,
normally run some time before next release).
- To create options of enumeration type use the `Examples:` field.
- Each example value have their own `Help:` field, but they are treated
a bit different than the main option help text. They will be shown
as an unordered list, therefore a single line break is enough to
create a new list item. Also, for enumeration texts like name of
countries, it looks better without an ending period/full stop character.
- Each example value have their own `Help:` field, but they are treated
a bit different than the main option help text. They will be shown
as an unordered list, therefore a single line break is enough to
create a new list item. Also, for enumeration texts like name of
countries, it looks better without an ending period/full stop character.
- You can run `make backenddocs` to verify the resulting Markdown.
- This will update the autogenerated sections of the backend docs Markdown
files under `docs/content`.
- It requires you to have [Python](https://www.python.org) installed.
- The `backenddocs` make target runs the Python script `bin/make_backend_docs.py`,
and you can also run this directly, optionally with the name of a backend
as argument to only update the docs for a specific backend.
- **Do not** commit the updated Markdown files. This operation is run as part of
the release process. Since any manual changes in the autogenerated sections
of the Markdown files will then be lost, we have a pull request check that
reports error for any changes within the autogenerated sections. Should you
have done manual changes outside of the autogenerated sections they must be
committed, of course.
- You can run `make serve` to verify the resulting website.
- This will build the website and serve it locally, so you can open it in
your web browser and verify that the end result looks OK. Check specifically
any added links, also in light of the note above regarding different algorithms
for generated header anchors.
- It requires you to have the [Hugo](https://gohugo.io) tool available.
- The `serve` make target depends on the `website` target, which runs the
`hugo` command from the `docs` directory to build the website, and then
it serves the website locally with an embedded web server using a command
`hugo server --logLevel info -w --disableFastRender --ignoreCache`, so you
can run similar Hugo commands directly as well.
The only documentation you need to edit are the `docs/content/*.md`
files. The `MANUAL.*`, `rclone.1`, website, etc. are all auto-generated
from those during the release process. See the `make doc` and `make
website` targets in the Makefile if you are interested in how. You
don't need to run these when adding a feature.
When writing documentation for an entirely new backend,
see [backend documentation](#backend-documentation).
Documentation for rclone sub commands is with their code, e.g.
`cmd/ls/ls.go`. Write flag help strings as a single sentence on a single
line, without a period/full stop character at the end, as it will be
combined unmodified with other information (such as any default value).
If you are updating documentation for a command, you must do that in the
command source code, e.g. `cmd/ls/ls.go`. Write flag help strings as a single
sentence on a single line, without a period/full stop character at the end,
as it will be combined unmodified with other information (such as any default
value).
Note that you can use [GitHub's online editor](https://help.github.com/en/github/managing-files-in-a-repository/editing-files-in-another-users-repository)
for small changes in the docs which makes it very easy.
Note that you can use
[GitHub's online editor](https://help.github.com/en/github/managing-files-in-a-repository/editing-files-in-another-users-repository)
for small changes in the docs which makes it very easy. Just remember the
caveat when linking to header anchors, noted above, which means that GitHub's
Markdown preview may not be an entirely reliable verification of the results.
After your changes have been merged, you can verify them on
[tip.rclone.org](https://tip.rclone.org). This site is updated daily with the
current state of the master branch at 07:00 UTC. The changes will be on the main
[rclone.org](https://rclone.org) site once they have been included in a release.
## Making a release
@@ -350,13 +509,13 @@ change will get linked into the issue.
Here is an example of a short commit message:
```
```text
drive: add team drive support - fixes #885
```
And here is an example of a longer one:
```
```text
mount: fix hang on errored upload
In certain circumstances, if an upload failed then the mount could hang
@@ -379,7 +538,9 @@ To add a dependency `github.com/ncw/new_dependency` see the
instructions below. These will fetch the dependency and add it to
`go.mod` and `go.sum`.
go get github.com/ncw/new_dependency
```console
go get github.com/ncw/new_dependency
```
You can add constraints on that package when doing `go get` (see the
go docs linked above), but don't unless you really need to.
@@ -391,7 +552,9 @@ and `go.sum` in the same commit as your other changes.
If you need to update a dependency then run
go get golang.org/x/crypto
```console
go get golang.org/x/crypto
```
Check in a single commit as above.
@@ -434,25 +597,38 @@ remote or an fs.
### Getting going
- Create `backend/remote/remote.go` (copy this from a similar remote)
- box is a good one to start from if you have a directory-based remote (and shows how to use the directory cache)
- b2 is a good one to start from if you have a bucket-based remote
- box is a good one to start from if you have a directory-based remote (and
shows how to use the directory cache)
- b2 is a good one to start from if you have a bucket-based remote
- Add your remote to the imports in `backend/all/all.go`
- HTTP based remotes are easiest to maintain if they use rclone's [lib/rest](https://pkg.go.dev/github.com/rclone/rclone/lib/rest) module, but if there is a really good Go SDK from the provider then use that instead.
- Try to implement as many optional methods as possible as it makes the remote more usable.
- Use [lib/encoder](https://pkg.go.dev/github.com/rclone/rclone/lib/encoder) to make sure we can encode any path name and `rclone info` to help determine the encodings needed
- `rclone purge -v TestRemote:rclone-info`
- `rclone test info --all --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info`
- `go run cmd/test/info/internal/build_csv/main.go -o remote.csv remote.json`
- open `remote.csv` in a spreadsheet and examine
- HTTP based remotes are easiest to maintain if they use rclone's
[lib/rest](https://pkg.go.dev/github.com/rclone/rclone/lib/rest) module, but
if there is a really good Go SDK from the provider then use that instead.
- Try to implement as many optional methods as possible as it makes the remote
more usable.
- Use [lib/encoder](https://pkg.go.dev/github.com/rclone/rclone/lib/encoder) to
make sure we can encode any path name and `rclone info` to help determine the
encodings needed
- `rclone purge -v TestRemote:rclone-info`
- `rclone test info --all --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info`
- `go run cmd/test/info/internal/build_csv/main.go -o remote.csv remote.json`
- open `remote.csv` in a spreadsheet and examine
### Guidelines for a speedy merge
- **Do** use [lib/rest](https://pkg.go.dev/github.com/rclone/rclone/lib/rest) if you are implementing a REST like backend and parsing XML/JSON in the backend.
- **Do** use rclone's Client or Transport from [fs/fshttp](https://pkg.go.dev/github.com/rclone/rclone/fs/fshttp) if your backend is HTTP based - this adds features like `--dump bodies`, `--tpslimit`, `--user-agent` without you having to code anything!
- **Do** follow your example backend exactly - use the same code order, function names, layout, structure. **Don't** move stuff around and **Don't** delete the comments.
- **Do not** split your backend up into `fs.go` and `object.go` (there are a few backends like that - don't follow them!)
- **Do** use [lib/rest](https://pkg.go.dev/github.com/rclone/rclone/lib/rest)
if you are implementing a REST like backend and parsing XML/JSON in the backend.
- **Do** use rclone's Client or Transport from [fs/fshttp](https://pkg.go.dev/github.com/rclone/rclone/fs/fshttp)
if your backend is HTTP based - this adds features like `--dump bodies`,
`--tpslimit`, `--user-agent` without you having to code anything!
- **Do** follow your example backend exactly - use the same code order, function
names, layout, structure. **Don't** move stuff around and **Don't** delete the
comments.
- **Do not** split your backend up into `fs.go` and `object.go` (there are a few
backends like that - don't follow them!)
- **Do** put your API type definitions in a separate file - by preference `api/types.go`
- **Remember** we have >50 backends to maintain so keeping them as similar as possible to each other is a high priority!
- **Remember** we have >50 backends to maintain so keeping them as similar as
possible to each other is a high priority!
### Unit tests
@@ -463,34 +639,53 @@ remote or an fs.
### Integration tests
- Add your backend to `fstest/test_all/config.yaml`
- Once you've done that then you can use the integration test framework from the project root:
- go install ./...
- test_all -backends remote
- Once you've done that then you can use the integration test framework from
the project root:
- `go run ./fstest/test_all -backends remote`
Or if you want to run the integration tests manually:
- Make sure integration tests pass with
- `cd fs/operations`
- `go test -v -remote TestRemote:`
- `cd fs/sync`
- `go test -v -remote TestRemote:`
- `cd fs/operations`
- `go test -v -remote TestRemote:`
- `cd fs/sync`
- `go test -v -remote TestRemote:`
- If your remote defines `ListR` check with this also
- `go test -v -remote TestRemote: -fast-list`
- `go test -v -remote TestRemote: -fast-list`
Before a new or changed backend can be merged we require:
- A clean run of `go run ./fstest/test_all -backends remote` (please include the
result in your pull request).
- A test account for the backend so the maintainers can add it to the
[integration test server](https://integration.rclone.org) and keep the backend
working as rclone evolves. A backend that we cannot test is likely to break and
may be removed.
See the [testing](#testing) section for more information on integration tests.
### Backend documentation
Add your backend to the docs - you'll need to pick an icon for it from
[fontawesome](http://fontawesome.io/icons/). Keep lists of remotes in
alphabetical order of full name of remote (e.g. `drive` is ordered as
`Google Drive`) but with the local file system last.
Add your backend to the docs. Keep lists of remotes in alphabetical
order of full name of remote (e.g. `drive` is ordered as `Google
Drive`) but with the local file system last.
First add a data file about your backend in
`docs/data/backends/remote.yaml` - this is used to build the overview
tables and the tiering info.
- Create it with: `bin/manage_backends.py create docs/data/backends/remote.yaml`
- Edit it to fill in the blanks. Look at the [tiers docs](https://rclone.org/tiers/).
- Run this command to fill in the features: `bin/manage_backends.py features docs/data/backends/remote.yaml`
Next edit these files:
- `README.md` - main GitHub page
- `docs/content/remote.md` - main docs page (note the backend options are automatically added to this file with `make backenddocs`)
- make sure this has the `autogenerated options` comments in (see your reference backend docs)
- update them in your backend with `bin/make_backend_docs.py remote`
- `docs/content/overview.md` - overview docs - add an entry into the Features table and the Optional Features table.
- `docs/content/remote.md` - main docs page (note the backend options are
automatically added to this file with `make backenddocs`)
- make sure this has the `autogenerated options` comments in (see your
reference backend docs)
- update them in your backend with `bin/make_backend_docs.py remote`
- `docs/content/docs.md` - list of remotes in config section
- `docs/content/_index.md` - front page of rclone.org
- `docs/layouts/chrome/navbar.html` - add it to the website navigation
@@ -501,78 +696,43 @@ in the web browser and the links (internal and external) all work.
## Adding a new s3 provider
It is quite easy to add a new S3 provider to rclone.
You'll need to modify the following files
- `backend/s3/s3.go`
- Add the provider to `providerOption` at the top of the file
- Add endpoints and other config for your provider gated on the provider in `fs.RegInfo`.
- Exclude your provider from generic config questions (eg `region` and `endpoint).
- Add the provider to the `setQuirks` function - see the documentation there.
- `docs/content/s3.md`
- Add the provider at the top of the page.
- Add a section about the provider linked from there.
- Add a transcript of a trial `rclone config` session
- Edit the transcript to remove things which might change in subsequent versions
- **Do not** alter or add to the autogenerated parts of `s3.md`
- **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3`
- `README.md` - this is the home page in github
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
- `docs/content/_index.md` - this is the home page of rclone.org
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
When adding the provider, endpoints, quirks, docs etc keep them in
alphabetical order by `Provider` name, but with `AWS` first and
`Other` last.
Once you've written the docs, run `make serve` and check they look OK
in the web browser and the links (internal and external) all work.
Once you've written the code, test `rclone config` works to your
satisfaction, and check the integration tests work `go test -v -remote
NewS3Provider:`. You may need to adjust the quirks to get them to
pass. Some providers just can't pass the tests with control characters
in the names so if these fail and the provider doesn't support
`urlEncodeListings` in the quirks then ignore them. Note that the
`SetTier` test may also fail on non AWS providers.
For an example of adding an s3 provider see [eb3082a1](https://github.com/rclone/rclone/commit/eb3082a1ebdb76d5625f14cedec3f5154a5e7b10).
[Please see the guide in the S3 backend directory](backend/s3/README.md).
## Writing a plugin
New features (backends, commands) can also be added "out-of-tree", through Go plugins.
Changes will be kept in a dynamically loaded file instead of being compiled into the main binary.
This is useful if you can't merge your changes upstream or don't want to maintain a fork of rclone.
New features (backends, commands) can also be added "out-of-tree", through Go
plugins. Changes will be kept in a dynamically loaded file instead of being
compiled into the main binary. This is useful if you can't merge your changes
upstream or don't want to maintain a fork of rclone.
### Usage
- Naming
- Plugins names must have the pattern `librcloneplugin_KIND_NAME.so`.
- `KIND` should be one of `backend`, `command` or `bundle`.
- Example: A plugin with backend support for PiFS would be called
`librcloneplugin_backend_pifs.so`.
- Loading
- Supported on macOS & Linux as of now. ([Go issue for Windows support](https://github.com/golang/go/issues/19282))
- Supported on rclone v1.50 or greater.
- All plugins in the folder specified by variable `$RCLONE_PLUGIN_PATH` are loaded.
- If this variable doesn't exist, plugin support is disabled.
- Plugins must be compiled against the exact version of rclone to work.
(The rclone used during building the plugin must be the same as the source of rclone)
- Naming
- Plugins names must have the pattern `librcloneplugin_KIND_NAME.so`.
- `KIND` should be one of `backend`, `command` or `bundle`.
- Example: A plugin with backend support for PiFS would be called
`librcloneplugin_backend_pifs.so`.
- Loading
- Supported on macOS & Linux as of now. ([Go issue for Windows support](https://github.com/golang/go/issues/19282))
- Supported on rclone v1.50 or greater.
- All plugins in the folder specified by variable `$RCLONE_PLUGIN_PATH` are loaded.
- If this variable doesn't exist, plugin support is disabled.
- Plugins must be compiled against the exact version of rclone to work.
(The rclone used during building the plugin must be the same as the source
of rclone)
### Building
To turn your existing additions into a Go plugin, move them to an external repository
and change the top-level package name to `main`.
Check `rclone --version` and make sure that the plugin's rclone dependency and host Go version match.
Check `rclone --version` and make sure that the plugin's rclone dependency and
host Go version match.
Then, run `go build -buildmode=plugin -o PLUGIN_NAME.so .` to build the plugin.
[Go reference](https://godoc.org/github.com/rclone/rclone/lib/plugin)
[Minimal example](https://gist.github.com/terorie/21b517ee347828e899e1913efc1d684f)
## Keeping a backend or command out of tree
Rclone was designed to be modular so it is very easy to keep a backend
@@ -585,6 +745,6 @@ add them out of tree.
This may be easier than using a plugin and is supported on all
platforms not just macOS and Linux.
This is explained further in https://github.com/rclone/rclone_out_of_tree_example
This is explained further in <https://github.com/rclone/rclone_out_of_tree_example>
which has an example of an out of tree backend `ram` (which is a
renamed version of the `memory` backend).
+101 -66
View File
@@ -1,103 +1,138 @@
# Maintainers guide for rclone #
# Maintainers guide for rclone
Current active maintainers of rclone are:
| Name | GitHub ID | Specific Responsibilities |
| :--------------- | :---------------- | :-------------------------- |
| Nick Craig-Wood | @ncw | overall project health |
| Stefan Breunig | @breunigs | |
| Ishuah Kariuki | @ishuah | |
| Remus Bunduc | @remusb | cache backend |
| Fabian Möller | @B4dM4n | |
| Alex Chen | @Cnly | onedrive backend |
| Sandeep Ummadi | @sandeepkru | azureblob backend |
| Sebastian Bünger | @buengese | jottacloud, yandex & compress backends |
| Ivan Andreev | @ivandeex | chunker & mailru backends |
| Max Sum | @Max-Sum | union backend |
| Fred | @creativeprojects | seafile backend |
| Caleb Case | @calebcase | storj backend |
| wiserain | @wiserain | pikpak backend |
| albertony | @albertony | |
| Chun-Hung Tseng | @henrybear327 | Proton Drive Backend |
| Hideo Aoyama | @boukendesho | snap packaging |
| nielash | @nielash | bisync |
| Dan McArdle | @dmcardle | gitannex |
| Sam Harrison | @childish-sambino | filescom |
| Name | GitHub ID |
| :--------------- | :---------------- |
| Nick Craig-Wood | @ncw |
| Stefan Breunig | @breunigs |
| Ishuah Kariuki | @ishuah |
| Remus Bunduc | @remusb |
| Fabian Möller | @B4dM4n |
| Alex Chen | @Cnly |
| Sandeep Ummadi | @sandeepkru |
| Sebastian Bünger | @buengese |
| Ivan Andreev | @ivandeex |
| Max Sum | @Max-Sum |
| Fred | @creativeprojects |
| Caleb Case | @calebcase |
| wiserain | @wiserain |
| albertony | @albertony |
| Chun-Hung Tseng | @henrybear327 |
| Hideo Aoyama | @boukendesho |
| nielash | @nielash |
| Dan McArdle | @dmcardle |
| Sam Harrison | @childish-sambino |
| Enduriel | @Enduriel |
**This is a work in progress Draft**
Per-area ownership of code (which backend or command each maintainer looks
after) is defined in [.github/CODEOWNERS](.github/CODEOWNERS), which GitHub uses
to auto-request reviews. Two responsibilities don't map to a code path: @ncw
looks after overall project health, and @boukendesho maintains the snap
packaging.
This is a guide for how to be an rclone maintainer. This is mostly a write-up of what I (@ncw) attempt to do.
## This is a work in progress draft
## Triaging Tickets ##
This is a guide for how to be an rclone maintainer. This is mostly a write-up
of what I (@ncw) attempt to do.
When a ticket comes in it should be triaged. This means it should be classified by adding labels and placed into a milestone. Quite a lot of tickets need a bit of back and forth to determine whether it is a valid ticket so tickets may remain without labels or milestone for a while.
## Triaging Tickets
When a ticket comes in it should be triaged. This means it should be classified
by adding labels and placed into a milestone. Quite a lot of tickets need a bit
of back and forth to determine whether it is a valid ticket so tickets may
remain without labels or milestone for a while.
Rclone uses the labels like this:
* `bug` - a definitely verified bug
* `can't reproduce` - a problem which we can't reproduce
* `doc fix` - a bug in the documentation - if users need help understanding the docs add this label
* `duplicate` - normally close these and ask the user to subscribe to the original
* `enhancement: new remote` - a new rclone backend
* `enhancement` - a new feature
* `FUSE` - to do with `rclone mount` command
* `good first issue` - mark these if you find a small self-contained issue - these get shown to new visitors to the project
* `help` wanted - mark these if you find a self-contained issue - these get shown to new visitors to the project
* `IMPORTANT` - note to maintainers not to forget to fix this for the release
* `maintenance` - internal enhancement, code re-organisation, etc.
* `Needs Go 1.XX` - waiting for that version of Go to be released
* `question` - not a `bug` or `enhancement` - direct to the forum for next time
* `Remote: XXX` - which rclone backend this affects
* `thinking` - not decided on the course of action yet
- `bug` - a definitely verified bug
- `can't reproduce` - a problem which we can't reproduce
- `doc fix` - a bug in the documentation - if users need help understanding the
docs add this label
- `duplicate` - normally close these and ask the user to subscribe to the original
- `enhancement: new remote` - a new rclone backend
- `enhancement` - a new feature
- `FUSE` - to do with `rclone mount` command
- `good first issue` - mark these if you find a small self-contained issue -
these get shown to new visitors to the project
- `help` wanted - mark these if you find a self-contained issue - these get
shown to new visitors to the project
- `IMPORTANT` - note to maintainers not to forget to fix this for the release
- `maintenance` - internal enhancement, code re-organisation, etc.
- `Needs Go 1.XX` - waiting for that version of Go to be released
- `question` - not a `bug` or `enhancement` - direct to the forum for next time
- `Remote: XXX` - which rclone backend this affects
- `thinking` - not decided on the course of action yet
If it turns out to be a bug or an enhancement it should be tagged as such, with the appropriate other tags. Don't forget the "good first issue" tag to give new contributors something easy to do to get going.
If it turns out to be a bug or an enhancement it should be tagged as such, with
the appropriate other tags. Don't forget the "good first issue" tag to give new
contributors something easy to do to get going.
When a ticket is tagged it should be added to a milestone, either the next release, the one after, Soon or Help Wanted. Bugs can be added to the "Known Bugs" milestone if they aren't planned to be fixed or need to wait for something (e.g. the next go release).
When a ticket is tagged it should be added to a milestone, either the next
release, the one after, Soon or Help Wanted. Bugs can be added to the
"Known Bugs" milestone if they aren't planned to be fixed or need to wait for
something (e.g. the next go release).
The milestones have these meanings:
* v1.XX - stuff we would like to fit into this release
* v1.XX+1 - stuff we are leaving until the next release
* Soon - stuff we think is a good idea - waiting to be scheduled for a release
* Help wanted - blue sky stuff that might get moved up, or someone could help with
* Known bugs - bugs waiting on external factors or we aren't going to fix for the moment
- v1.XX - stuff we would like to fit into this release
- v1.XX+1 - stuff we are leaving until the next release
- Soon - stuff we think is a good idea - waiting to be scheduled for a release
- Help wanted - blue sky stuff that might get moved up, or someone could help with
- Known bugs - bugs waiting on external factors or we aren't going to fix for
the moment
Tickets [with no milestone](https://github.com/rclone/rclone/issues?utf8=✓&q=is%3Aissue%20is%3Aopen%20no%3Amile) are good candidates for ones that have slipped between the gaps and need following up.
Tickets [with no milestone](https://github.com/rclone/rclone/issues?utf8=✓&q=is%3Aissue%20is%3Aopen%20no%3Amile)
are good candidates for ones that have slipped between the gaps and need
following up.
## Closing Tickets ##
## Closing Tickets
Close tickets as soon as you can - make sure they are tagged with a release. Post a link to a beta in the ticket with the fix in, asking for feedback.
Close tickets as soon as you can - make sure they are tagged with a release.
Post a link to a beta in the ticket with the fix in, asking for feedback.
## Pull requests ##
## Pull requests
Try to process pull requests promptly!
Merging pull requests on GitHub itself works quite well nowadays so you can squash and rebase or rebase pull requests. rclone doesn't use merge commits. Use the squash and rebase option if you need to edit the commit message.
Merging pull requests on GitHub itself works quite well nowadays so you can
squash and rebase or rebase pull requests. rclone doesn't use merge commits.
Use the squash and rebase option if you need to edit the commit message.
After merging the commit, in your local master branch, do `git pull` then run `bin/update-authors.py` to update the authors file then `git push`.
After merging the commit, in your local master branch, do `git pull` then run
`bin/update-authors.py` to update the authors file then `git push`.
Sometimes pull requests need to be left open for a while - this especially true of contributions of new backends which take a long time to get right.
Sometimes pull requests need to be left open for a while - this especially true
of contributions of new backends which take a long time to get right.
## Merges ##
## Merges
If you are merging a branch locally then do `git merge --ff-only branch-name` to avoid a merge commit. You'll need to rebase the branch if it doesn't merge cleanly.
If you are merging a branch locally then do `git merge --ff-only branch-name` to
avoid a merge commit. You'll need to rebase the branch if it doesn't merge cleanly.
## Release cycle ##
## Release cycle
Rclone aims for a 6-8 week release cycle. Sometimes release cycles take longer if there is something big to merge that didn't stabilize properly or for personal reasons.
Rclone aims for a 6-8 week release cycle. Sometimes release cycles take longer
if there is something big to merge that didn't stabilize properly or for personal
reasons.
High impact regressions should be fixed before the next release.
Near the start of the release cycle, the dependencies should be updated with `make update` to give time for bugs to surface.
Near the start of the release cycle, the dependencies should be updated with
`make update` to give time for bugs to surface.
Towards the end of the release cycle try not to merge anything too big so let things settle down.
Towards the end of the release cycle try not to merge anything too big so let
things settle down.
Follow the instructions in RELEASE.md for making the release. Note that the testing part is the most time-consuming often needing several rounds of test and fix depending on exactly how many new features rclone has gained.
Follow the instructions in RELEASE.md for making the release. Note that the
testing part is the most time-consuming often needing several rounds of test
and fix depending on exactly how many new features rclone has gained.
## Mailing list ##
## Mailing list
There is now an invite-only mailing list for rclone developers `rclone-dev` on google groups.
There is now an invite-only mailing list for rclone developers `rclone-dev` on
google groups.
## TODO ##
## TODO
I should probably make a dev@rclone.org to register with cloud providers.
I should probably make a <dev@rclone.org> to register with cloud providers.
Generated
+49063 -16683
View File
File diff suppressed because it is too large. Load diff
Generated
+29437 -15703
View File
File diff suppressed because it is too large. Load diff
Generated
+16786 -5268
View File
File diff suppressed because it is too large. Load diff
+43 -8
View File
@@ -45,7 +45,7 @@ LINTTAGS=--build-tags "$(GOTAGS)"
endif
LDFLAGS=--ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)"
.PHONY: rclone test_all vars version
.PHONY: rclone test_all vars version fetch-gui fetch-gui-and-commit
rclone:
ifeq ($(GO_OS),windows)
@@ -59,6 +59,12 @@ endif
cp -av rclone`go env GOEXE` `go env GOPATH`/bin/rclone`go env GOEXE`.new
mv -v `go env GOPATH`/bin/rclone`go env GOEXE`.new `go env GOPATH`/bin/rclone`go env GOEXE`
fetch-gui:
$(SHELL) ./bin/fetch-gui-dist.sh
fetch-gui-and-commit:
$(SHELL) ./bin/fetch-gui-dist.sh --commit
test_all:
go install $(LDFLAGS) $(BUILDTAGS) $(BUILD_ARGS) github.com/rclone/rclone/fstest/test_all
@@ -100,6 +106,7 @@ compiletest:
check: rclone
@echo "-- START CODE QUALITY REPORT -------------------------------"
@golangci-lint run $(LINTTAGS) ./...
@bin/markdown-lint
@echo "-- END CODE QUALITY REPORT ---------------------------------"
# Get the build dependencies
@@ -113,21 +120,40 @@ release_dep_linux:
# Update dependencies
showupdates:
@echo "*** Direct dependencies that could be updated ***"
@GO111MODULE=on go list -u -f '{{if (and (not (or .Main .Indirect)) .Update)}}{{.Path}}: {{.Version}} -> {{.Update.Version}}{{end}}' -m all 2> /dev/null
@go list -u -f '{{if (and (not (or .Main .Indirect)) .Update)}}{{.Path}}: {{.Version}} -> {{.Update.Version}}{{end}}' -m all 2> /dev/null
# Update direct dependencies only
updatedirect:
GO111MODULE=on go get -d $$(go list -m -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' all)
GO111MODULE=on go mod tidy
go get $$(go list -m -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' all)
go mod tidy
# Update direct dependencies only but won't update the `go` line in go.mod
updatedirectnoupgrade:
@GO_VERSION=$$(awk '/^go /{print $$2}' go.mod) && \
for mod in $$(go list -m -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' all); do \
cp go.mod go.mod.bak && \
cp go.sum go.sum.bak && \
go get $$mod && \
NEW_VERSION=$$(awk '/^go /{print $$2}' go.mod) && \
if [ "$$NEW_VERSION" != "$$GO_VERSION" ]; then \
echo "SKIPPING $$mod (requires go $$NEW_VERSION)"; \
cp go.mod.bak go.mod; \
cp go.sum.bak go.sum; \
else \
echo "updated $$mod"; \
fi; \
done && \
rm -f go.mod.bak go.sum.bak && \
go mod tidy
# Update direct and indirect dependencies and test dependencies
update:
GO111MODULE=on go get -d -u -t ./...
GO111MODULE=on go mod tidy
go get -u -t ./...
go mod tidy
# Tidy the module dependencies
tidy:
GO111MODULE=on go mod tidy
go mod tidy
doc: rclone.1 MANUAL.html MANUAL.txt rcdocs commanddocs
@@ -144,9 +170,12 @@ MANUAL.txt: MANUAL.md
pandoc -s --from markdown-smart --to plain MANUAL.md -o MANUAL.txt
commanddocs: rclone
go generate ./lib/transform
go generate ./cmd/bisync
-@rmdir -p '$$HOME/.config/rclone'
XDG_CACHE_HOME="" XDG_CONFIG_HOME="" HOME="\$$HOME" USER="\$$USER" rclone gendocs --config=/notfound docs/content/
@[ ! -e '$$HOME' ] || (echo 'Error: created unwanted directory named $$HOME' && exit 1)
go run bin/make_bisync_docs.go ./docs/content/
backenddocs: rclone bin/make_backend_docs.py
-@rmdir -p '$$HOME/.config/rclone'
@@ -213,6 +242,12 @@ beta:
rclone -v copy build/ pub.rclone.org:/$(TAG)
@echo Beta release ready at https://pub.rclone.org/$(TAG)/
privatebeta:
go run bin/cross-compile.go $(BUILD_FLAGS) $(BUILDTAGS) $(BUILD_ARGS) -include '^(darwin|windows|linux)/(arm64|amd64)$$' $(TAG)
rclone -Pv copy build/ private-downloads:/beta/$(TAG)
@echo Private beta release ready at private-downloads:/beta/$(TAG)/
rclone link private-downloads:/beta/$(TAG)
log_since_last_release:
git log $(LAST_TAG)..
@@ -243,7 +278,7 @@ fetch_binaries:
rclone -P sync --exclude "/testbuilds/**" --delete-excluded $(BETA_UPLOAD) build/
serve: website
cd docs && hugo server --logLevel info -w --disableFastRender
cd docs && hugo server --logLevel info -w --disableFastRender --ignoreCache
tag: retag doc
bin/make_changelog.py $(LAST_TAG) $(VERSION) > docs/content/changelog.md.new
+154 -127
View File
@@ -1,6 +1,6 @@
<!-- markdownlint-disable-next-line first-line-heading no-inline-html -->
[<img src="https://rclone.org/img/logo_on_light__horizontal_color.svg" width="50%" alt="rclone logo">](https://rclone.org/#gh-light-mode-only)
<!-- markdownlint-disable-next-line no-inline-html -->
[<img src="https://rclone.org/img/logo_on_dark__horizontal_color.svg" width="50%" alt="rclone logo">](https://rclone.org/#gh-dark-mode-only)
[Website](https://rclone.org) |
@@ -18,102 +18,124 @@
# Rclone
Rclone *("rsync for cloud storage")* is a command-line program to sync files and directories to and from different cloud storage providers.
Rclone *("rsync for cloud storage")* is a command-line program to sync files and
directories to and from different cloud storage providers.
## Storage providers
* 1Fichier [:page_facing_up:](https://rclone.org/fichier/)
* Akamai Netstorage [:page_facing_up:](https://rclone.org/netstorage/)
* Alibaba Cloud (Aliyun) Object Storage System (OSS) [:page_facing_up:](https://rclone.org/s3/#alibaba-oss)
* Amazon S3 [:page_facing_up:](https://rclone.org/s3/)
* ArvanCloud Object Storage (AOS) [:page_facing_up:](https://rclone.org/s3/#arvan-cloud-object-storage-aos)
* Backblaze B2 [:page_facing_up:](https://rclone.org/b2/)
* Box [:page_facing_up:](https://rclone.org/box/)
* Ceph [:page_facing_up:](https://rclone.org/s3/#ceph)
* China Mobile Ecloud Elastic Object Storage (EOS) [:page_facing_up:](https://rclone.org/s3/#china-mobile-ecloud-eos)
* Cloudflare R2 [:page_facing_up:](https://rclone.org/s3/#cloudflare-r2)
* Citrix ShareFile [:page_facing_up:](https://rclone.org/sharefile/)
* DigitalOcean Spaces [:page_facing_up:](https://rclone.org/s3/#digitalocean-spaces)
* Digi Storage [:page_facing_up:](https://rclone.org/koofr/#digi-storage)
* Dreamhost [:page_facing_up:](https://rclone.org/s3/#dreamhost)
* Dropbox [:page_facing_up:](https://rclone.org/dropbox/)
* Enterprise File Fabric [:page_facing_up:](https://rclone.org/filefabric/)
* Fastmail Files [:page_facing_up:](https://rclone.org/webdav/#fastmail-files)
* FileLu [:page_facing_up:](https://rclone.org/filelu/)
* Files.com [:page_facing_up:](https://rclone.org/filescom/)
* FlashBlade [:page_facing_up:](https://rclone.org/s3/#pure-storage-flashblade)
* FTP [:page_facing_up:](https://rclone.org/ftp/)
* GoFile [:page_facing_up:](https://rclone.org/gofile/)
* Google Cloud Storage [:page_facing_up:](https://rclone.org/googlecloudstorage/)
* Google Drive [:page_facing_up:](https://rclone.org/drive/)
* Google Photos [:page_facing_up:](https://rclone.org/googlephotos/)
* HDFS (Hadoop Distributed Filesystem) [:page_facing_up:](https://rclone.org/hdfs/)
* Hetzner Storage Box [:page_facing_up:](https://rclone.org/sftp/#hetzner-storage-box)
* HiDrive [:page_facing_up:](https://rclone.org/hidrive/)
* HTTP [:page_facing_up:](https://rclone.org/http/)
* Huawei Cloud Object Storage Service(OBS) [:page_facing_up:](https://rclone.org/s3/#huawei-obs)
* iCloud Drive [:page_facing_up:](https://rclone.org/iclouddrive/)
* ImageKit [:page_facing_up:](https://rclone.org/imagekit/)
* Internet Archive [:page_facing_up:](https://rclone.org/internetarchive/)
* Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
* IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
* IONOS Cloud [:page_facing_up:](https://rclone.org/s3/#ionos)
* Koofr [:page_facing_up:](https://rclone.org/koofr/)
* Leviia Object Storage [:page_facing_up:](https://rclone.org/s3/#leviia)
* Liara Object Storage [:page_facing_up:](https://rclone.org/s3/#liara-object-storage)
* Linkbox [:page_facing_up:](https://rclone.org/linkbox)
* Linode Object Storage [:page_facing_up:](https://rclone.org/s3/#linode)
* Magalu Object Storage [:page_facing_up:](https://rclone.org/s3/#magalu)
* Mail.ru Cloud [:page_facing_up:](https://rclone.org/mailru/)
* Memset Memstore [:page_facing_up:](https://rclone.org/swift/)
* MEGA [:page_facing_up:](https://rclone.org/mega/)
* MEGA S4 Object Storage [:page_facing_up:](https://rclone.org/s3/#mega)
* Memory [:page_facing_up:](https://rclone.org/memory/)
* Microsoft Azure Blob Storage [:page_facing_up:](https://rclone.org/azureblob/)
* Microsoft Azure Files Storage [:page_facing_up:](https://rclone.org/azurefiles/)
* Microsoft OneDrive [:page_facing_up:](https://rclone.org/onedrive/)
* Minio [:page_facing_up:](https://rclone.org/s3/#minio)
* Nextcloud [:page_facing_up:](https://rclone.org/webdav/#nextcloud)
* OVH [:page_facing_up:](https://rclone.org/swift/)
* Blomp Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
* OpenDrive [:page_facing_up:](https://rclone.org/opendrive/)
* OpenStack Swift [:page_facing_up:](https://rclone.org/swift/)
* Oracle Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
* Oracle Object Storage [:page_facing_up:](https://rclone.org/oracleobjectstorage/)
* Outscale [:page_facing_up:](https://rclone.org/s3/#outscale)
* ownCloud [:page_facing_up:](https://rclone.org/webdav/#owncloud)
* pCloud [:page_facing_up:](https://rclone.org/pcloud/)
* Petabox [:page_facing_up:](https://rclone.org/s3/#petabox)
* PikPak [:page_facing_up:](https://rclone.org/pikpak/)
* Pixeldrain [:page_facing_up:](https://rclone.org/pixeldrain/)
* premiumize.me [:page_facing_up:](https://rclone.org/premiumizeme/)
* put.io [:page_facing_up:](https://rclone.org/putio/)
* Proton Drive [:page_facing_up:](https://rclone.org/protondrive/)
* QingStor [:page_facing_up:](https://rclone.org/qingstor/)
* Qiniu Cloud Object Storage (Kodo) [:page_facing_up:](https://rclone.org/s3/#qiniu)
* Quatrix [:page_facing_up:](https://rclone.org/quatrix/)
* Rackspace Cloud Files [:page_facing_up:](https://rclone.org/swift/)
* RackCorp Object Storage [:page_facing_up:](https://rclone.org/s3/#RackCorp)
* rsync.net [:page_facing_up:](https://rclone.org/sftp/#rsync-net)
* Scaleway [:page_facing_up:](https://rclone.org/s3/#scaleway)
* Seafile [:page_facing_up:](https://rclone.org/seafile/)
* Seagate Lyve Cloud [:page_facing_up:](https://rclone.org/s3/#lyve)
* SeaweedFS [:page_facing_up:](https://rclone.org/s3/#seaweedfs)
* Selectel Object Storage [:page_facing_up:](https://rclone.org/s3/#selectel)
* SFTP [:page_facing_up:](https://rclone.org/sftp/)
* SMB / CIFS [:page_facing_up:](https://rclone.org/smb/)
* StackPath [:page_facing_up:](https://rclone.org/s3/#stackpath)
* Storj [:page_facing_up:](https://rclone.org/storj/)
* SugarSync [:page_facing_up:](https://rclone.org/sugarsync/)
* Synology C2 Object Storage [:page_facing_up:](https://rclone.org/s3/#synology-c2)
* Tencent Cloud Object Storage (COS) [:page_facing_up:](https://rclone.org/s3/#tencent-cos)
* Uloz.to [:page_facing_up:](https://rclone.org/ulozto/)
* Wasabi [:page_facing_up:](https://rclone.org/s3/#wasabi)
* WebDAV [:page_facing_up:](https://rclone.org/webdav/)
* Yandex Disk [:page_facing_up:](https://rclone.org/yandex/)
* Zoho WorkDrive [:page_facing_up:](https://rclone.org/zoho/)
* Zata.ai [:page_facing_up:](https://rclone.org/s3/#Zata)
* The local filesystem [:page_facing_up:](https://rclone.org/local/)
- 1Fichier [:page_facing_up:](https://rclone.org/fichier/)
- Akamai Netstorage [:page_facing_up:](https://rclone.org/netstorage/)
- Alibaba Cloud (Aliyun) Object Storage System (OSS) [:page_facing_up:](https://rclone.org/s3/#alibaba-oss)
- Amazon S3 [:page_facing_up:](https://rclone.org/s3/)
- ArvanCloud Object Storage (AOS) [:page_facing_up:](https://rclone.org/s3/#arvan-cloud-object-storage-aos)
- Bizfly Cloud Simple Storage [:page_facing_up:](https://rclone.org/s3/#bizflycloud)
- Backblaze B2 [:page_facing_up:](https://rclone.org/b2/)
- Box [:page_facing_up:](https://rclone.org/box/)
- Ceph [:page_facing_up:](https://rclone.org/s3/#ceph)
- China Mobile Ecloud Elastic Object Storage (EOS) [:page_facing_up:](https://rclone.org/s3/#china-mobile-ecloud-eos)
- Citrix ShareFile [:page_facing_up:](https://rclone.org/sharefile/)
- Cloudflare R2 [:page_facing_up:](https://rclone.org/s3/#cloudflare-r2)
- Cloudinary [:page_facing_up:](https://rclone.org/cloudinary/)
- Cubbit DS3 [:page_facing_up:](https://rclone.org/s3/#Cubbit)
- DigitalOcean Spaces [:page_facing_up:](https://rclone.org/s3/#digitalocean-spaces)
- Digi Storage [:page_facing_up:](https://rclone.org/koofr/#digi-storage)
- Dreamhost [:page_facing_up:](https://rclone.org/s3/#dreamhost)
- Drime [:page_facing_up:](https://rclone.org/s3/#drime)
- Dropbox [:page_facing_up:](https://rclone.org/dropbox/)
- Enterprise File Fabric [:page_facing_up:](https://rclone.org/filefabric/)
- Exaba [:page_facing_up:](https://rclone.org/s3/#exaba)
- Fastly Object Storage [:page_facing_up:](https://rclone.org/s3/#fastly)
- Fastmail Files [:page_facing_up:](https://rclone.org/webdav/#fastmail-files)
- FileLu [:page_facing_up:](https://rclone.org/filelu/)
- Filen [:page_facing_up:](https://rclone.org/filen/)
- Files.com [:page_facing_up:](https://rclone.org/filescom/)
- FlashBlade [:page_facing_up:](https://rclone.org/s3/#pure-storage-flashblade)
- FTP [:page_facing_up:](https://rclone.org/ftp/)
- GoFile [:page_facing_up:](https://rclone.org/gofile/)
- Google Cloud Storage [:page_facing_up:](https://rclone.org/googlecloudstorage/)
- Google Drive [:page_facing_up:](https://rclone.org/drive/)
- Google Photos [:page_facing_up:](https://rclone.org/googlephotos/)
- HDFS (Hadoop Distributed Filesystem) [:page_facing_up:](https://rclone.org/hdfs/)
- Hetzner Object Storage [:page_facing_up:](https://rclone.org/s3/#hetzner)
- Hetzner Storage Box [:page_facing_up:](https://rclone.org/sftp/#hetzner-storage-box)
- HiDrive [:page_facing_up:](https://rclone.org/hidrive/)
- Hitachi Content Platform (HCP) [:page_facing_up:](https://rclone.org/s3/#hcp)
- HTTP [:page_facing_up:](https://rclone.org/http/)
- Huawei Cloud Object Storage Service(OBS) [:page_facing_up:](https://rclone.org/s3/#huawei-obs)
- Huawei Drive [:page_facing_up:](https://rclone.org/huaweidrive/)
- iCloud Drive [:page_facing_up:](https://rclone.org/iclouddrive/)
- ImageKit [:page_facing_up:](https://rclone.org/imagekit/)
- Internet Archive [:page_facing_up:](https://rclone.org/internetarchive/)
- Internxt [:page_facing_up:](https://rclone.org/internxt/)
- Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
- IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
- Impossible Cloud [:page_facing_up:](https://rclone.org/s3/#impossible-cloud)
- Intercolo Object Storage [:page_facing_up:](https://rclone.org/s3/#intercolo)
- IONOS Cloud [:page_facing_up:](https://rclone.org/s3/#ionos)
- Koofr [:page_facing_up:](https://rclone.org/koofr/)
- Leviia Object Storage [:page_facing_up:](https://rclone.org/s3/#leviia)
- Liara Object Storage [:page_facing_up:](https://rclone.org/s3/#liara-object-storage)
- Linkbox [:page_facing_up:](https://rclone.org/linkbox)
- Linode Object Storage [:page_facing_up:](https://rclone.org/s3/#linode)
- Magalu Object Storage [:page_facing_up:](https://rclone.org/s3/#magalu)
- Mail.ru Cloud [:page_facing_up:](https://rclone.org/mailru/)
- Memset Memstore [:page_facing_up:](https://rclone.org/swift/)
- MEGA [:page_facing_up:](https://rclone.org/mega/)
- MEGA S4 Object Storage [:page_facing_up:](https://rclone.org/s3/#mega)
- Memory [:page_facing_up:](https://rclone.org/memory/)
- Microsoft Azure Blob Storage [:page_facing_up:](https://rclone.org/azureblob/)
- Microsoft Azure Files Storage [:page_facing_up:](https://rclone.org/azurefiles/)
- Microsoft OneDrive [:page_facing_up:](https://rclone.org/onedrive/)
- Minio [:page_facing_up:](https://rclone.org/s3/#minio)
- Nextcloud [:page_facing_up:](https://rclone.org/webdav/#nextcloud)
- Blomp Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
- OpenDrive [:page_facing_up:](https://rclone.org/opendrive/)
- OpenStack Swift [:page_facing_up:](https://rclone.org/swift/)
- Oracle Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
- Oracle Object Storage [:page_facing_up:](https://rclone.org/oracleobjectstorage/)
- Outscale [:page_facing_up:](https://rclone.org/s3/#outscale)
- OVHcloud Object Storage (Swift) [:page_facing_up:](https://rclone.org/swift/)
- OVHcloud Object Storage (S3-compatible) [:page_facing_up:](https://rclone.org/s3/#ovhcloud)
- ownCloud [:page_facing_up:](https://rclone.org/webdav/#owncloud)
- pCloud [:page_facing_up:](https://rclone.org/pcloud/)
- Petabox [:page_facing_up:](https://rclone.org/s3/#petabox)
- PikPak [:page_facing_up:](https://rclone.org/pikpak/)
- Pixeldrain [:page_facing_up:](https://rclone.org/pixeldrain/)
- premiumize.me [:page_facing_up:](https://rclone.org/premiumizeme/)
- put.io [:page_facing_up:](https://rclone.org/putio/)
- Proton Drive [:page_facing_up:](https://rclone.org/protondrive/)
- QingStor [:page_facing_up:](https://rclone.org/qingstor/)
- Qiniu Cloud Object Storage (Kodo) [:page_facing_up:](https://rclone.org/s3/#qiniu)
- Rabata Cloud Storage [:page_facing_up:](https://rclone.org/s3/#Rabata)
- Quatrix [:page_facing_up:](https://rclone.org/quatrix/)
- Rackspace Cloud Files [:page_facing_up:](https://rclone.org/swift/)
- RackCorp Object Storage [:page_facing_up:](https://rclone.org/s3/#RackCorp)
- rsync.net [:page_facing_up:](https://rclone.org/sftp/#rsync-net)
- Scaleway [:page_facing_up:](https://rclone.org/s3/#scaleway)
- Scality (RING / ARTESCA) [:page_facing_up:](https://rclone.org/s3/#scality)
- Seafile [:page_facing_up:](https://rclone.org/seafile/)
- Seagate Lyve Cloud [:page_facing_up:](https://rclone.org/s3/#lyve)
- SeaweedFS [:page_facing_up:](https://rclone.org/s3/#seaweedfs)
- Selectel Object Storage [:page_facing_up:](https://rclone.org/s3/#selectel)
- Servercore Object Storage [:page_facing_up:](https://rclone.org/s3/#servercore)
- SFTP [:page_facing_up:](https://rclone.org/sftp/)
- Shade [:page_facing_up:](https://rclone.org/shade/)
- SMB / CIFS [:page_facing_up:](https://rclone.org/smb/)
- Spectra Logic [:page_facing_up:](https://rclone.org/s3/#spectralogic)
- Storj [:page_facing_up:](https://rclone.org/storj/)
- SugarSync [:page_facing_up:](https://rclone.org/sugarsync/)
- Synology C2 Object Storage [:page_facing_up:](https://rclone.org/s3/#synology-c2)
- Tencent Cloud Object Storage (COS) [:page_facing_up:](https://rclone.org/s3/#tencent-cos)
- Uloz.to [:page_facing_up:](https://rclone.org/ulozto/)
- US3 Object Storage [:page_facing_up:](https://rclone.org/s3/#us3)
- Wasabi [:page_facing_up:](https://rclone.org/s3/#wasabi)
- WebDAV [:page_facing_up:](https://rclone.org/webdav/)
- Yandex Disk [:page_facing_up:](https://rclone.org/yandex/)
- Zadara Object Storage [:page_facing_up:](https://rclone.org/s3/#zadara)
- Zero Services (ZERO-Z3) [:page_facing_up:](https://rclone.org/s3/#zero-z3)
- Zoho WorkDrive [:page_facing_up:](https://rclone.org/zoho/)
- Zata.ai [:page_facing_up:](https://rclone.org/s3/#Zata)
- The local filesystem [:page_facing_up:](https://rclone.org/local/)
Please see [the full list of all storage providers and their features](https://rclone.org/overview/)
@@ -121,50 +143,55 @@ Please see [the full list of all storage providers and their features](https://r
These backends adapt or modify other storage providers
* Alias: rename existing remotes [:page_facing_up:](https://rclone.org/alias/)
* Cache: cache remotes (DEPRECATED) [:page_facing_up:](https://rclone.org/cache/)
* Chunker: split large files [:page_facing_up:](https://rclone.org/chunker/)
* Combine: combine multiple remotes into a directory tree [:page_facing_up:](https://rclone.org/combine/)
* Compress: compress files [:page_facing_up:](https://rclone.org/compress/)
* Crypt: encrypt files [:page_facing_up:](https://rclone.org/crypt/)
* Hasher: hash files [:page_facing_up:](https://rclone.org/hasher/)
* Union: join multiple remotes to work together [:page_facing_up:](https://rclone.org/union/)
- Alias: rename existing remotes [:page_facing_up:](https://rclone.org/alias/)
- Archive: read archive files [:page_facing_up:](https://rclone.org/archive/)
- Cache: cache remotes (DEPRECATED) [:page_facing_up:](https://rclone.org/cache/)
- Chunker: split large files [:page_facing_up:](https://rclone.org/chunker/)
- Combine: combine multiple remotes into a directory tree [:page_facing_up:](https://rclone.org/combine/)
- Compress: compress files [:page_facing_up:](https://rclone.org/compress/)
- Crypt: encrypt files [:page_facing_up:](https://rclone.org/crypt/)
- Hasher: hash files [:page_facing_up:](https://rclone.org/hasher/)
- Union: join multiple remotes to work together [:page_facing_up:](https://rclone.org/union/)
## Features
* MD5/SHA-1 hashes checked at all times for file integrity
* Timestamps preserved on files
* Partial syncs supported on a whole file basis
* [Copy](https://rclone.org/commands/rclone_copy/) mode to just copy new/changed files
* [Sync](https://rclone.org/commands/rclone_sync/) (one way) mode to make a directory identical
* [Bisync](https://rclone.org/bisync/) (two way) to keep two directories in sync bidirectionally
* [Check](https://rclone.org/commands/rclone_check/) mode to check for file hash equality
* Can sync to and from network, e.g. two different cloud accounts
* Optional large file chunking ([Chunker](https://rclone.org/chunker/))
* Optional transparent compression ([Compress](https://rclone.org/compress/))
* Optional encryption ([Crypt](https://rclone.org/crypt/))
* Optional FUSE mount ([rclone mount](https://rclone.org/commands/rclone_mount/))
* Multi-threaded downloads to local disk
* Can [serve](https://rclone.org/commands/rclone_serve/) local or remote files over HTTP/WebDAV/FTP/SFTP/DLNA
- MD5/SHA-1 hashes checked at all times for file integrity
- Timestamps preserved on files
- Partial syncs supported on a whole file basis
- [Copy](https://rclone.org/commands/rclone_copy/) mode to just copy new/changed
files
- [Sync](https://rclone.org/commands/rclone_sync/) (one way) mode to make a directory
identical
- [Bisync](https://rclone.org/bisync/) (two way) to keep two directories in sync
bidirectionally
- [Check](https://rclone.org/commands/rclone_check/) mode to check for file hash
equality
- Can sync to and from network, e.g. two different cloud accounts
- Optional large file chunking ([Chunker](https://rclone.org/chunker/))
- Optional transparent compression ([Compress](https://rclone.org/compress/))
- Optional encryption ([Crypt](https://rclone.org/crypt/))
- Optional FUSE mount ([rclone mount](https://rclone.org/commands/rclone_mount/))
- Multi-threaded downloads to local disk
- Can [serve](https://rclone.org/commands/rclone_serve/) local or remote files
over HTTP/WebDAV/FTP/SFTP/DLNA
## Installation & documentation
Please see the [rclone website](https://rclone.org/) for:
* [Installation](https://rclone.org/install/)
* [Documentation & configuration](https://rclone.org/docs/)
* [Changelog](https://rclone.org/changelog/)
* [FAQ](https://rclone.org/faq/)
* [Storage providers](https://rclone.org/overview/)
* [Forum](https://forum.rclone.org/)
* ...and more
- [Installation](https://rclone.org/install/)
- [Documentation & configuration](https://rclone.org/docs/)
- [Changelog](https://rclone.org/changelog/)
- [FAQ](https://rclone.org/faq/)
- [Storage providers](https://rclone.org/overview/)
- [Forum](https://forum.rclone.org/)
- ...and more
## Downloads
* https://rclone.org/downloads/
- <https://rclone.org/downloads/>
License
-------
## License
This is free software under the terms of the MIT license (check the
[COPYING file](/COPYING) included in this package).
+163 -88
View File
@@ -4,85 +4,80 @@ This file describes how to make the various kinds of releases
## Extra required software for making a release
* [gh the github cli](https://github.com/cli/cli) for uploading packages
* pandoc for making the html and man pages
- [gh the github cli](https://github.com/cli/cli) for uploading packages
- pandoc for making the html and man pages
## Making a release
* git checkout master # see below for stable branch
* git pull # IMPORTANT
* git status - make sure everything is checked in
* Check GitHub actions build for master is Green
* make test # see integration test server or run locally
* make tag
* edit docs/content/changelog.md # make sure to remove duplicate logs from point releases
* make tidy
* make doc
* git status - to check for new man pages - git add them
* git commit -a -v -m "Version v1.XX.0"
* make retag
* git push origin # without --follow-tags so it doesn't push the tag if it fails
* git push --follow-tags origin
* # Wait for the GitHub builds to complete then...
* make fetch_binaries
* make tarball
* make vendorball
* make sign_upload
* make check_sign
* make upload
* make upload_website
* make upload_github
* make startdev # make startstable for stable branch
* # announce with forum post, twitter post, patreon post
- git checkout master # see below for stable branch
- git pull # IMPORTANT
- make fetch-gui-and-commit # bump the embedded GUI if rclone-web has a newer release
- git status - make sure everything is checked in
- Check GitHub actions build for master is Green
- make test # see integration test server or run locally
- make tag
- edit docs/content/changelog.md # duplicate logs from point rels removed automatically
releases
- make tidy
- make doc
- git status - to check for new man pages - git add them
- git commit -a -v -m "Version v1.XX.0"
- make check
- make retag
- git push origin # without --follow-tags so it doesn't push the tag if it fails
- git push --follow-tags origin
- \# Wait for the GitHub builds to complete then...
- make fetch_binaries
- make tarball
- make vendorball
- make sign_upload
- make check_sign
- make upload
- make upload_test_website # check sponsors are correct/unchanged
- make upload_website
- make upload_github
- make startdev # make startstable for stable branch
- \# announce with forum post, twitter post, and post to annoucements list
## Update dependencies
Early in the next release cycle update the dependencies.
* Review any pinned packages in go.mod and remove if possible
* `make updatedirect`
* `make GOTAGS=cmount`
* `make compiletest`
* Fix anything which doesn't compile at this point and commit changes here
* `git commit -a -v -m "build: update all dependencies"`
- Active Pins
- `github.com/gdamore/tcell/v2` v2.9.0 - awaiting fix for [tcell/issues/1124](https://github.com/gdamore/tcell/issues/1124)
- See: [#9539](https://github.com/rclone/rclone/issues/9539)
- Review Active Pins in go.mod and remove if possible
- `make updatedirect`
- `make GOTAGS=cmount`
- `make compiletest`
- Fix anything which doesn't compile at this point and commit changes here
- `git commit -a -v -m "build: update all dependencies"`
If the `make updatedirect` upgrades the version of go in the `go.mod`
go 1.22.0
then go to manual mode. `go1.22` here is the lowest supported version
in the `go.mod`.
If `make updatedirect` added a `toolchain` directive then remove it.
We don't want to force a toolchain on our users. Linux packagers are
often using a version of Go that is a few versions out of date.
```
go list -m -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' all > /tmp/potential-upgrades
go get -d $(cat /tmp/potential-upgrades)
go mod tidy -go=1.22 -compat=1.22
```text
go 1.22.0
```
If the `go mod tidy` fails use the output from it to remove the
package which can't be upgraded from `/tmp/potential-upgrades` when
done
then so this instead `make updatedirectnoupgrade`. This will try each
upgrade individually. Consider making an upstream bug report for any
`SKIPPING` lines as these only support the latest Go version not the
previous version too.
```
git co go.mod go.sum
```
And try again.
If `make updatedirect` or `make updatedirectnoupgrade` added a
`toolchain` directive then remove it. We don't want to force a
toolchain on our users. Linux packagers are often using a version of
Go that is a few versions out of date.
Optionally upgrade the direct and indirect dependencies. This is very
likely to fail if the manual method was used abve - in that case
ignore it as it is too time consuming to fix.
likely to fail if `make updatedirectnoupgrade` was used.
* `make update`
* `make GOTAGS=cmount`
* `make compiletest`
* roll back any updates which didn't compile
* `git commit -a -v --amend`
* **NB** watch out for this changing the default go version in `go.mod`
- `make update`
- `make GOTAGS=cmount`
- `make compiletest`
- roll back any updates which didn't compile
- `git commit -a -v --amend`
- **NB** watch out for this changing the default go version in `go.mod`
Note that `make update` updates all direct and indirect dependencies
and there can occasionally be forwards compatibility problems with
@@ -93,21 +88,91 @@ build.
Once it compiles locally, push it on a test branch and commit fixes
until the tests pass.
### Pseudo versions
Go makes pseudo versions for untagged repos and repos not at a tag.
The pseudo versions on repos that have been tagged before do not get
updated automatically so need manually checking. These can be found with
```console
grep -E '[0-9]{14}-[0-9a-f]{12}' go.mod | grep -v indirect | grep -v 'v0\.0\.0'
```
These will need to be updated manually using the `go get ...@branch`
syntax.
### Major versions
The above procedure will not upgrade major versions, so v2 to v3.
However this tool can show which major versions might need to be
upgraded:
go run github.com/icholy/gomajor@latest list -major
```console
go run github.com/icholy/gomajor@latest list -major
```
Expect API breakage when updating major versions.
## Updating Go
When a new Go stable is released update to it. We support the current
stable Go and the previous release which is in line with the rest of
the Go ecosystem.
These files will need editing:
- `.github/workflows/build.yml` - change current and previous Go versions
- `docs/content/install.md` - change minimum Go version required
- `fs/versioncheck.go` - update minimum Go version required
- `go.mod` - update minimum Go version required
Check it builds
- `make GOTAGS=cmount`
- `make compiletest`
Assuming `go1.XX` is current and `go1.YY` is previous version:
Use `git grep go1.YY` and `git grep go1.YY` to look for opportunities
to remove build tags we no longer need.
Commit with message like this:
```text
build: update to go1.YY and make go1.YY the minimum required version
```
Send to CI and if it passes, merge.
### gofix
Updating the minimum required version of Go is a good opportunity to
run the `go fix` command to modernize Go usage.
This needs to be run for all architectures.
```console
GOOS=linux go fix -tags cmount ./...
GOOS=freebsd go fix -tags cmount ./...
GOOS=windows go fix -tags cmount ./...
GOOS=darwin go fix -tags cmount ./...
```
Examine the diff carefully.
Commit with message
```text
build: modernize Go code with go fix for go1.YY
```
## Tidy beta
At some point after the release run
bin/tidy-beta v1.55
```console
bin/tidy-beta v1.55
```
where the version number is that of a couple ago to remove old beta binaries.
@@ -117,54 +182,64 @@ If rclone needs a point release due to some horrendous bug:
Set vars
* BASE_TAG=v1.XX # e.g. v1.52
* NEW_TAG=${BASE_TAG}.Y # e.g. v1.52.1
* echo $BASE_TAG $NEW_TAG # v1.52 v1.52.1
- BASE_TAG=v1.XX # e.g. v1.52
- NEW_TAG=${BASE_TAG}.Y # e.g. v1.52.1
- echo $BASE_TAG $NEW_TAG # v1.52 v1.52.1
First make the release branch. If this is a second point release then
this will be done already.
* git co -b ${BASE_TAG}-stable ${BASE_TAG}.0
* make startstable
- git co -b ${BASE_TAG}-stable ${BASE_TAG}.0
- make startstable
Now
* git co ${BASE_TAG}-stable
* git cherry-pick any fixes
* make startstable
* Do the steps as above
* git co master
* `#` cherry pick the changes to the changelog - check the diff to make sure it is correct
* git checkout ${BASE_TAG}-stable docs/content/changelog.md
* git commit -a -v -m "Changelog updates from Version ${NEW_TAG}"
* git push
- git co ${BASE_TAG}-stable
- `git cherry-pick -x` any fixes
- make startstable
- Do the steps as above
- git co master
- `#` cherry pick the changes to the changelog - check the diff to make sure it
is correct
- git checkout ${BASE_TAG}-stable docs/content/changelog.md
- git commit -a -v -m "Changelog updates from Version ${NEW_TAG}"
- git push
## Sponsor logos
If updating the website note that the sponsor logos have been moved out of the main repository.
If updating the website note that the sponsor logos have been moved out of the
main repository.
You will need to checkout `/docs/static/img/logos` from https://github.com/rclone/third-party-logos
You will need to checkout `/docs/static/img/logos` from <https://github.com/rclone/third-party-logos>
which is a private repo containing artwork from sponsors.
## Update the website between releases
Create an update website branch based off the last release
git co -b update-website
```console
git co -b update-website
```
If the branch already exists, double check there are no commits that need saving.
Now reset the branch to the last release
git reset --hard v1.64.0
```console
git reset --hard v1.64.0
```
Create the changes, check them in, test with `make serve` then
make upload_test_website
```console
make upload_test_website
```
Check out https://test.rclone.org and when happy
Check out <https://test.rclone.org> and when happy
make upload_website
```console
make upload_website
```
Cherry pick any changes back to master and the stable branch if it is active.
@@ -172,14 +247,14 @@ Cherry pick any changes back to master and the stable branch if it is active.
To do a basic build of rclone's docker image to debug builds locally:
```
```console
docker buildx build --load -t rclone/rclone:testing --progress=plain .
docker run --rm rclone/rclone:testing version
```
To test the multipatform build
```
```console
docker buildx build -t rclone/rclone:testing --progress=plain --platform linux/amd64,linux/386,linux/arm64,linux/arm/v7,linux/arm/v6 .
```
@@ -187,6 +262,6 @@ To make a full build then set the tags correctly and add `--push`
Note that you can't only build one architecture - you need to build them all.
```
```console
docker buildx build --platform linux/amd64,linux/386,linux/arm64,linux/arm/v7,linux/arm/v6 -t rclone/rclone:1.54.1 -t rclone/rclone:1.54 -t rclone/rclone:1 -t rclone/rclone:latest --push .
```
+1 -1
View File
@@ -1 +1 @@
v1.71.0
v1.75.0
+6 -1
View File
@@ -4,6 +4,7 @@ package all
import (
// Active file systems
_ "github.com/rclone/rclone/backend/alias"
_ "github.com/rclone/rclone/backend/archive"
_ "github.com/rclone/rclone/backend/azureblob"
_ "github.com/rclone/rclone/backend/azurefiles"
_ "github.com/rclone/rclone/backend/b2"
@@ -15,11 +16,13 @@ import (
_ "github.com/rclone/rclone/backend/compress"
_ "github.com/rclone/rclone/backend/crypt"
_ "github.com/rclone/rclone/backend/doi"
_ "github.com/rclone/rclone/backend/drime"
_ "github.com/rclone/rclone/backend/drive"
_ "github.com/rclone/rclone/backend/dropbox"
_ "github.com/rclone/rclone/backend/fichier"
_ "github.com/rclone/rclone/backend/filefabric"
_ "github.com/rclone/rclone/backend/filelu"
_ "github.com/rclone/rclone/backend/filen"
_ "github.com/rclone/rclone/backend/filescom"
_ "github.com/rclone/rclone/backend/ftp"
_ "github.com/rclone/rclone/backend/gofile"
@@ -29,9 +32,11 @@ import (
_ "github.com/rclone/rclone/backend/hdfs"
_ "github.com/rclone/rclone/backend/hidrive"
_ "github.com/rclone/rclone/backend/http"
_ "github.com/rclone/rclone/backend/huaweidrive"
_ "github.com/rclone/rclone/backend/iclouddrive"
_ "github.com/rclone/rclone/backend/imagekit"
_ "github.com/rclone/rclone/backend/internetarchive"
_ "github.com/rclone/rclone/backend/internxt"
_ "github.com/rclone/rclone/backend/jottacloud"
_ "github.com/rclone/rclone/backend/koofr"
_ "github.com/rclone/rclone/backend/linkbox"
@@ -54,6 +59,7 @@ import (
_ "github.com/rclone/rclone/backend/s3"
_ "github.com/rclone/rclone/backend/seafile"
_ "github.com/rclone/rclone/backend/sftp"
_ "github.com/rclone/rclone/backend/shade"
_ "github.com/rclone/rclone/backend/sharefile"
_ "github.com/rclone/rclone/backend/sia"
_ "github.com/rclone/rclone/backend/smb"
@@ -62,7 +68,6 @@ import (
_ "github.com/rclone/rclone/backend/swift"
_ "github.com/rclone/rclone/backend/ulozto"
_ "github.com/rclone/rclone/backend/union"
_ "github.com/rclone/rclone/backend/uptobox"
_ "github.com/rclone/rclone/backend/webdav"
_ "github.com/rclone/rclone/backend/yandex"
_ "github.com/rclone/rclone/backend/zoho"
+679
View File
@@ -0,0 +1,679 @@
//go:build !plan9
// Package archive implements a backend to access archive files in a remote
package archive
// FIXME factor common code between backends out - eg VFS initialization
// FIXME can we generalize the VFS handle caching and use it in zip backend
// Factor more stuff out if possible
// Odd stats which are probably coming from the VFS
// * tensorflow.sqfs: 0% /3.074Gi, 204.426Ki/s, 4h22m46s
// FIXME this will perform poorly for unpacking as the VFS Reader is bad
// at multiple streams - need cache mode setting?
import (
"context"
"errors"
"fmt"
"io"
"path"
"strings"
"sync"
"time"
// Import all the required archivers here
_ "github.com/rclone/rclone/backend/archive/squashfs"
_ "github.com/rclone/rclone/backend/archive/zip"
"github.com/rclone/rclone/backend/archive/archiver"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/cache"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/fspath"
"github.com/rclone/rclone/fs/hash"
)
// Register with Fs
func init() {
fsi := &fs.RegInfo{
Name: "archive",
Description: "Read archives",
NewFs: NewFs,
MetadataInfo: &fs.MetadataInfo{
Help: `Any metadata supported by the underlying remote is read and written.`,
},
Options: []fs.Option{{
Name: "remote",
Help: `Remote to wrap to read archives from.
Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
"myremote:bucket" or "myremote:".
If this is left empty, then the archive backend will use the root as
the remote.
This means that you can use :archive:remote:path and it will be
equivalent to setting remote="remote:path".
`,
Required: false,
}},
}
fs.Register(fsi)
}
// Options defines the configuration for this backend
type Options struct {
Remote string `config:"remote"`
}
// Fs represents a archive of upstreams
type Fs struct {
name string // name of this remote
features *fs.Features // optional features
opt Options // options for this Fs
root string // the path we are working on
f fs.Fs // remote we are wrapping
wrapper fs.Fs // fs that wraps us
mu sync.Mutex // protects the below
archives map[string]*archive // the archives we have, by path
}
// A single open archive
type archive struct {
archiver archiver.Archiver // archiver responsible
remote string // path to the archive
prefix string // prefix to add on to listings
root string // root of the archive to remove from listings
mu sync.Mutex // protects the following variables
f fs.Fs // the archive Fs, may be nil
}
// If remote is an archive then return it otherwise return nil
func findArchive(remote string) *archive {
// FIXME use something faster than linear search?
for _, archiver := range archiver.Archivers {
if strings.HasSuffix(remote, archiver.Extension) {
return &archive{
archiver: archiver,
remote: remote,
prefix: remote,
root: "",
}
}
}
return nil
}
// Find an archive buried in remote
func subArchive(remote string) *archive {
archive := findArchive(remote)
if archive != nil {
return archive
}
parent := path.Dir(remote)
if parent == "/" || parent == "." {
return nil
}
return subArchive(parent)
}
// If remote is an archive then return it otherwise return nil
func (f *Fs) findArchive(remote string) (archive *archive) {
archive = findArchive(remote)
if archive != nil {
f.mu.Lock()
f.archives[remote] = archive
f.mu.Unlock()
}
return archive
}
// Instantiate archive if it hasn't been instantiated yet
//
// This is done lazily so that we can list a directory full of
// archives without opening them all.
func (a *archive) init(ctx context.Context, f fs.Fs) (fs.Fs, error) {
a.mu.Lock()
defer a.mu.Unlock()
if a.f != nil {
return a.f, nil
}
newFs, err := a.archiver.New(ctx, f, a.remote, a.prefix, a.root)
if err != nil && err != fs.ErrorIsFile {
return nil, fmt.Errorf("failed to create archive %q: %w", a.remote, err)
}
a.f = newFs
return a.f, nil
}
// NewFs constructs an Fs from the path.
//
// The returned Fs is the actual Fs, referenced by remote in the config
func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (outFs fs.Fs, err error) {
// defer log.Trace(nil, "name=%q, root=%q, m=%v", name, root, m)("f=%+v, err=%v", &outFs, &err)
// Parse config into Options struct
opt := new(Options)
err = configstruct.Set(m, opt)
if err != nil {
return nil, err
}
remote := opt.Remote
origRoot := root
// If remote is empty, use the root instead
if remote == "" {
remote = root
root = ""
}
isDirectory := strings.HasSuffix(remote, "/")
remote = strings.TrimRight(remote, "/")
if remote == "" {
remote = "/"
}
if strings.HasPrefix(remote, name+":") {
return nil, errors.New("can't point archive remote at itself - check the value of the upstreams setting")
}
_ = isDirectory
foundArchive := subArchive(remote)
if foundArchive != nil {
fs.Debugf(nil, "Found archiver for %q remote %q", foundArchive.archiver.Extension, foundArchive.remote)
// Archive path
foundArchive.root = strings.Trim(remote[len(foundArchive.remote):], "/")
// Path to the archive
archiveRemote := remote[:len(foundArchive.remote)]
// Remote is archive leaf name
foundArchive.remote = path.Base(archiveRemote)
foundArchive.prefix = ""
// Point remote to archive file
remote = archiveRemote
}
// Make sure to remove trailing . referring to the current dir
if path.Base(root) == "." {
root = strings.TrimSuffix(root, ".")
}
remotePath := fspath.JoinRootPath(remote, root)
wrappedFs, err := cache.Get(ctx, remotePath)
if err != fs.ErrorIsFile && err != nil {
return nil, fmt.Errorf("failed to make remote %q to wrap: %w", remote, err)
}
f := &Fs{
name: name,
//root: path.Join(remotePath, root),
root: origRoot,
opt: *opt,
f: wrappedFs,
archives: make(map[string]*archive),
}
cache.PinUntilFinalized(f.f, f)
// the features here are ones we could support, and they are
// ANDed with the ones from wrappedFs
f.features = (&fs.Features{
CaseInsensitive: true,
DuplicateFiles: false,
ReadMimeType: true,
WriteMimeType: true,
CanHaveEmptyDirectories: true,
BucketBased: true,
SetTier: true,
GetTier: true,
ReadMetadata: true,
WriteMetadata: true,
UserMetadata: true,
PartialUploads: true,
}).Fill(ctx, f).Mask(ctx, wrappedFs).WrapsFs(f, wrappedFs)
if foundArchive != nil {
fs.Debugf(f, "Root is an archive")
if err != fs.ErrorIsFile {
return nil, fmt.Errorf("expecting to find a file at %q", remote)
}
return foundArchive.init(ctx, f.f)
}
// Correct root if definitely pointing to a file
if err == fs.ErrorIsFile {
f.root = path.Dir(f.root)
if f.root == "." || f.root == "/" {
f.root = ""
}
}
return f, err
}
// Name of the remote (as passed into NewFs)
func (f *Fs) Name() string {
return f.name
}
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
return f.root
}
// String converts this Fs to a string
func (f *Fs) String() string {
return fmt.Sprintf("archive root '%s'", f.root)
}
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
// Rmdir removes the root directory of the Fs object
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return f.f.Rmdir(ctx, dir)
}
// Hashes returns hash.HashNone to indicate remote hashing is unavailable
func (f *Fs) Hashes() hash.Set {
return f.f.Hashes()
}
// Mkdir makes the root directory of the Fs object
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
return f.f.Mkdir(ctx, dir)
}
// Purge all files in the directory
//
// Implement this if you have a way of deleting all the files
// quicker than just running Remove() on the result of List()
//
// Return an error if it doesn't exist
func (f *Fs) Purge(ctx context.Context, dir string) error {
do := f.f.Features().Purge
if do == nil {
return fs.ErrorCantPurge
}
return do(ctx, dir)
}
// Copy src to this remote using server-side copy operations.
//
// This is stored with the remote path given.
//
// It returns the destination Object and a possible error.
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantCopy
func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
do := f.f.Features().Copy
if do == nil {
return nil, fs.ErrorCantCopy
}
// FIXME
// o, ok := src.(*Object)
// if !ok {
// return nil, fs.ErrorCantCopy
// }
return do(ctx, src, remote)
}
// Move src to this remote using server-side move operations.
//
// This is stored with the remote path given.
//
// It returns the destination Object and a possible error.
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantMove
func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
do := f.f.Features().Move
if do == nil {
return nil, fs.ErrorCantMove
}
// FIXME
// o, ok := src.(*Object)
// if !ok {
// return nil, fs.ErrorCantMove
// }
return do(ctx, src, remote)
}
// DirMove moves src, srcRemote to this remote at dstRemote
// using server-side move operations.
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantDirMove
//
// If destination exists then return fs.ErrorDirExists
func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) (err error) {
do := f.f.Features().DirMove
if do == nil {
return fs.ErrorCantDirMove
}
srcFs, ok := src.(*Fs)
if !ok {
fs.Debugf(srcFs, "Can't move directory - not same remote type")
return fs.ErrorCantDirMove
}
return do(ctx, srcFs.f, srcRemote, dstRemote)
}
// ChangeNotify calls the passed function with a path
// that has had changes. If the implementation
// uses polling, it should adhere to the given interval.
// At least one value will be written to the channel,
// specifying the initial value and updated values might
// follow. A 0 Duration should pause the polling.
// The ChangeNotify implementation must empty the channel
// regularly. When the channel gets closed, the implementation
// should stop polling and release resources.
func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), ch <-chan time.Duration) {
do := f.f.Features().ChangeNotify
if do == nil {
return
}
wrappedNotifyFunc := func(path string, entryType fs.EntryType) {
// fs.Debugf(f, "ChangeNotify: path %q entryType %d", path, entryType)
notifyFunc(path, entryType)
}
do(ctx, wrappedNotifyFunc, ch)
}
// DirCacheFlush resets the directory cache - used in testing
// as an optional interface
func (f *Fs) DirCacheFlush() {
do := f.f.Features().DirCacheFlush
if do != nil {
do()
}
}
func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, stream bool, options ...fs.OpenOption) (fs.Object, error) {
var o fs.Object
var err error
if stream {
o, err = f.f.Features().PutStream(ctx, in, src, options...)
} else {
o, err = f.f.Put(ctx, in, src, options...)
}
if err != nil {
return nil, err
}
return o, nil
}
// Put in to the remote path with the modTime given of the given size
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
o, err := f.NewObject(ctx, src.Remote())
switch err {
case nil:
return o, o.Update(ctx, in, src, options...)
case fs.ErrorObjectNotFound:
return f.put(ctx, in, src, false, options...)
default:
return nil, err
}
}
// PutStream uploads to the remote path with the modTime given of indeterminate size
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
o, err := f.NewObject(ctx, src.Remote())
switch err {
case nil:
return o, o.Update(ctx, in, src, options...)
case fs.ErrorObjectNotFound:
return f.put(ctx, in, src, true, options...)
default:
return nil, err
}
}
// About gets quota information from the Fs
func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
do := f.f.Features().About
if do == nil {
return nil, errors.New("not supported by underlying remote")
}
return do(ctx)
}
// Find the Fs for the directory
func (f *Fs) findFs(ctx context.Context, dir string) (subFs fs.Fs, err error) {
f.mu.Lock()
defer f.mu.Unlock()
subFs = f.f
// FIXME should do this with a better datastructure like a prefix tree
// FIXME want to find the longest first otherwise nesting won't work
dirSlash := dir + "/"
for archiverRemote, archive := range f.archives {
subRemote := archiverRemote + "/"
if strings.HasPrefix(dirSlash, subRemote) {
subFs, err = archive.init(ctx, f.f)
if err != nil {
return nil, err
}
break
}
}
return subFs, nil
}
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
//
// dir should be "" to list the root, and should not have
// trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
// defer log.Trace(f, "dir=%q", dir)("entries = %v, err=%v", &entries, &err)
subFs, err := f.findFs(ctx, dir)
if err != nil {
return nil, err
}
entries, err = subFs.List(ctx, dir)
if err != nil {
return nil, err
}
for i, entry := range entries {
// Can only unarchive files
if o, ok := entry.(fs.Object); ok {
remote := o.Remote()
archive := f.findArchive(remote)
if archive != nil {
// Overwrite entry with directory
entries[i] = fs.NewDir(remote, o.ModTime(ctx))
}
}
}
return entries, nil
}
// NewObject creates a new remote archive file object
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
dir := path.Dir(remote)
if dir == "/" || dir == "." {
dir = ""
}
subFs, err := f.findFs(ctx, dir)
if err != nil {
return nil, err
}
o, err := subFs.NewObject(ctx, remote)
if err != nil {
return nil, err
}
return o, nil
}
// Precision is the greatest precision of all the archivers
func (f *Fs) Precision() time.Duration {
return time.Second
}
// Shutdown the backend, closing any background tasks and any
// cached connections.
func (f *Fs) Shutdown(ctx context.Context) error {
if do := f.f.Features().Shutdown; do != nil {
return do(ctx)
}
return nil
}
// PublicLink generates a public link to the remote path (usually readable by anyone)
func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration, unlink bool) (string, error) {
do := f.f.Features().PublicLink
if do == nil {
return "", errors.New("PublicLink not supported")
}
return do(ctx, remote, expire, unlink)
}
// PutUnchecked in to the remote path with the modTime given of the given size
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
//
// May create duplicates or return errors if src already
// exists.
func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
do := f.f.Features().PutUnchecked
if do == nil {
return nil, errors.New("can't PutUnchecked")
}
o, err := do(ctx, in, src, options...)
if err != nil {
return nil, err
}
return o, nil
}
// MergeDirs merges the contents of all the directories passed
// in into the first one and rmdirs the other directories.
func (f *Fs) MergeDirs(ctx context.Context, dirs []fs.Directory) error {
if len(dirs) == 0 {
return nil
}
do := f.f.Features().MergeDirs
if do == nil {
return errors.New("MergeDirs not supported")
}
return do(ctx, dirs)
}
// CleanUp the trash in the Fs
//
// Implement this if you have a way of emptying the trash or
// otherwise cleaning up old versions of files.
func (f *Fs) CleanUp(ctx context.Context) error {
do := f.f.Features().CleanUp
if do == nil {
return errors.New("not supported by underlying remote")
}
return do(ctx)
}
// OpenWriterAt opens with a handle for random access writes
//
// Pass in the remote desired and the size if known.
//
// It truncates any existing object
func (f *Fs) OpenWriterAt(ctx context.Context, remote string, size int64) (fs.WriterAtCloser, error) {
do := f.f.Features().OpenWriterAt
if do == nil {
return nil, fs.ErrorNotImplemented
}
return do(ctx, remote, size)
}
// UnWrap returns the Fs that this Fs is wrapping
func (f *Fs) UnWrap() fs.Fs {
return f.f
}
// WrapFs returns the Fs that is wrapping this Fs
func (f *Fs) WrapFs() fs.Fs {
return f.wrapper
}
// SetWrapper sets the Fs that is wrapping this Fs
func (f *Fs) SetWrapper(wrapper fs.Fs) {
f.wrapper = wrapper
}
// OpenChunkWriter returns the chunk size and a ChunkWriter
//
// Pass in the remote and the src object
// You can also use options to hint at the desired chunk size
func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectInfo, options ...fs.OpenOption) (info fs.ChunkWriterInfo, writer fs.ChunkWriter, err error) {
do := f.f.Features().OpenChunkWriter
if do == nil {
return info, nil, fs.ErrorNotImplemented
}
return do(ctx, remote, src, options...)
}
// UserInfo returns info about the connected user
func (f *Fs) UserInfo(ctx context.Context) (map[string]string, error) {
do := f.f.Features().UserInfo
if do == nil {
return nil, fs.ErrorNotImplemented
}
return do(ctx)
}
// Disconnect the current user
func (f *Fs) Disconnect(ctx context.Context) error {
do := f.f.Features().Disconnect
if do == nil {
return fs.ErrorNotImplemented
}
return do(ctx)
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
_ fs.Purger = (*Fs)(nil)
_ fs.PutStreamer = (*Fs)(nil)
_ fs.Copier = (*Fs)(nil)
_ fs.Mover = (*Fs)(nil)
_ fs.DirMover = (*Fs)(nil)
_ fs.DirCacheFlusher = (*Fs)(nil)
_ fs.ChangeNotifier = (*Fs)(nil)
_ fs.Abouter = (*Fs)(nil)
_ fs.Shutdowner = (*Fs)(nil)
_ fs.PublicLinker = (*Fs)(nil)
_ fs.PutUncheckeder = (*Fs)(nil)
_ fs.MergeDirser = (*Fs)(nil)
_ fs.CleanUpper = (*Fs)(nil)
_ fs.OpenWriterAter = (*Fs)(nil)
_ fs.OpenChunkWriter = (*Fs)(nil)
_ fs.UserInfoer = (*Fs)(nil)
_ fs.Disconnecter = (*Fs)(nil)
// FIXME _ fs.FullObject = (*Object)(nil)
)
+279
View File
@@ -0,0 +1,279 @@
//go:build !plan9
package archive
import (
"bytes"
"context"
"fmt"
"io"
"os"
"os/exec"
"path"
"path/filepath"
"strconv"
"strings"
"testing"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/cache"
"github.com/rclone/rclone/fs/filter"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// FIXME need to test Open with seek
// run - run a shell command
func run(t *testing.T, args ...string) {
cmd := exec.Command(args[0], args[1:]...)
fs.Debugf(nil, "run args = %v", args)
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf(`
----------------------------
Failed to run %v: %v
Command output was:
%s
----------------------------
`, args, err, out)
}
}
// check the dst and src are identical
func checkTree(ctx context.Context, name string, t *testing.T, dstArchive, src string, expectedCount int) {
t.Run(name, func(t *testing.T) {
fs.Debugf(nil, "check %q vs %q", dstArchive, src)
Farchive, err := cache.Get(ctx, dstArchive)
if err != fs.ErrorIsFile {
require.NoError(t, err)
}
Fsrc, err := cache.Get(ctx, src)
if err != fs.ErrorIsFile {
require.NoError(t, err)
}
var matches bytes.Buffer
opt := operations.CheckOpt{
Fdst: Farchive,
Fsrc: Fsrc,
Match: &matches,
}
for _, action := range []string{"Check", "Download"} {
t.Run(action, func(t *testing.T) {
matches.Reset()
if action == "Download" {
assert.NoError(t, operations.CheckDownload(ctx, &opt))
} else {
assert.NoError(t, operations.Check(ctx, &opt))
}
if expectedCount > 0 {
assert.Equal(t, expectedCount, strings.Count(matches.String(), "\n"))
}
})
}
t.Run("NewObject", func(t *testing.T) {
// Check we can run NewObject on all files and read them
assert.NoError(t, operations.ListFn(ctx, Fsrc, func(srcObj fs.Object) {
if t.Failed() {
return
}
remote := srcObj.Remote()
archiveObj, err := Farchive.NewObject(ctx, remote)
require.NoError(t, err, remote)
assert.Equal(t, remote, archiveObj.Remote(), remote)
// Test that the contents are the same
archiveBuf := fstests.ReadObject(ctx, t, archiveObj, -1)
srcBuf := fstests.ReadObject(ctx, t, srcObj, -1)
assert.Equal(t, srcBuf, archiveBuf)
if len(srcBuf) < 81 {
return
}
// Tests that Open works with SeekOption
assert.Equal(t, srcBuf[50:], fstests.ReadObject(ctx, t, archiveObj, -1, &fs.SeekOption{Offset: 50}), "contents differ after seek")
// Tests that Open works with RangeOption
for _, test := range []struct {
ro fs.RangeOption
wantStart, wantEnd int
}{
{fs.RangeOption{Start: 5, End: 15}, 5, 16},
{fs.RangeOption{Start: 80, End: -1}, 80, len(srcBuf)},
{fs.RangeOption{Start: 81, End: 100000}, 81, len(srcBuf)},
{fs.RangeOption{Start: -1, End: 20}, len(srcBuf) - 20, len(srcBuf)}, // if start is omitted this means get the final bytes
// {fs.RangeOption{Start: -1, End: -1}, 0, len(srcBuf)}, - this seems to work but the RFC doesn't define it
} {
got := fstests.ReadObject(ctx, t, archiveObj, -1, &test.ro)
foundAt := strings.Index(srcBuf, got)
help := fmt.Sprintf("%#v failed want [%d:%d] got [%d:%d]", test.ro, test.wantStart, test.wantEnd, foundAt, foundAt+len(got))
assert.Equal(t, srcBuf[test.wantStart:test.wantEnd], got, help)
}
// Test that the modtimes are correct
fstest.AssertTimeEqualWithPrecision(t, remote, srcObj.ModTime(ctx), archiveObj.ModTime(ctx), Farchive.Precision())
// Test that the sizes are correct
assert.Equal(t, srcObj.Size(), archiveObj.Size())
// Test that Strings are OK
assert.Equal(t, srcObj.String(), archiveObj.String())
}))
})
// t.Logf("Fdst ------------- %v", Fdst)
// operations.List(ctx, Fdst, os.Stdout)
// t.Logf("Fsrc ------------- %v", Fsrc)
// operations.List(ctx, Fsrc, os.Stdout)
})
}
// test creating and reading back some archives
//
// Note that this uses rclone and zip as external binaries.
func testArchive(t *testing.T, archiveName string, archiveFn func(t *testing.T, output, input string)) {
ctx := context.Background()
checkFiles := 1000
// create random test input files
inputRoot := t.TempDir()
input := filepath.Join(inputRoot, archiveName)
require.NoError(t, os.Mkdir(input, 0777))
run(t, "rclone", "test", "makefiles", "--files", strconv.Itoa(checkFiles), "--ascii", input)
// Create the archive
output := t.TempDir()
zipFile := path.Join(output, archiveName)
archiveFn(t, zipFile, input)
// Check the archive itself
checkTree(ctx, "Archive", t, ":archive:"+zipFile, input, checkFiles)
// Now check a subdirectory
fis, err := os.ReadDir(input)
require.NoError(t, err)
subDir := "NOT FOUND"
aFile := "NOT FOUND"
for _, fi := range fis {
if fi.IsDir() {
subDir = fi.Name()
} else {
aFile = fi.Name()
}
}
checkTree(ctx, "SubDir", t, ":archive:"+zipFile+"/"+subDir, filepath.Join(input, subDir), 0)
// Now check a single file
fiCtx, fi := filter.AddConfig(ctx)
require.NoError(t, fi.AddRule("+ "+aFile))
require.NoError(t, fi.AddRule("- *"))
checkTree(fiCtx, "SingleFile", t, ":archive:"+zipFile+"/"+aFile, filepath.Join(input, aFile), 0)
// Now check the level above
checkTree(ctx, "Root", t, ":archive:"+output, inputRoot, checkFiles)
// run(t, "cp", "-a", inputRoot, output, "/tmp/test-"+archiveName)
}
// Make sure we have the executable named
func skipIfNoExe(t *testing.T, exeName string) {
_, err := exec.LookPath(exeName)
if err != nil {
t.Skipf("%s executable not installed", exeName)
}
}
// Test creating and reading back some archives
//
// Note that this uses rclone and zip as external binaries.
func TestArchiveZip(t *testing.T) {
fstest.Initialise()
skipIfNoExe(t, "zip")
skipIfNoExe(t, "rclone")
testArchive(t, "test.zip", func(t *testing.T, output, input string) {
oldcwd, err := os.Getwd()
require.NoError(t, err)
require.NoError(t, os.Chdir(input))
defer func() {
require.NoError(t, os.Chdir(oldcwd))
}()
run(t, "zip", "-9r", output, ".")
})
}
// Test creating and reading back some archives
//
// Note that this uses rclone and squashfs as external binaries.
func TestArchiveSquashfs(t *testing.T) {
fstest.Initialise()
skipIfNoExe(t, "mksquashfs")
skipIfNoExe(t, "rclone")
testArchive(t, "test.sqfs", func(t *testing.T, output, input string) {
run(t, "mksquashfs", input, output)
})
}
// TestArchiveSquashfsIssue9004 lists and reads squashfs images that exercise
// two layouts go-diskfs used to choke on (fixed in go-diskfs v1.9.4):
//
// - 1.sqfs: a single empty directory, so the image has no fragment table
// (its fragment-table start holds the "not present" sentinel).
// - 2.sqfs: a small tree whose superblock has the NO_XATTRS flag set while
// inodes still carry a (non-sentinel) xattr index - the shape squashfs-
// tools-ng can emit. Built by packing a two-file tree with xattrs via
// `gensquashfs -x`, then setting the NO_XATTRS superblock flag; the tree
// content is trivial placeholder data.
//
// Both images used to fail to list. Regression test for #9004.
func TestArchiveSquashfsIssue9004(t *testing.T) {
fstest.Initialise()
ctx := context.Background()
testdata, err := filepath.Abs(filepath.Join("squashfs", "testdata"))
require.NoError(t, err)
archiveFor := func(t *testing.T, name string) fs.Fs {
f, err := cache.Get(ctx, ":archive:"+filepath.Join(testdata, name))
require.NoError(t, err)
return f
}
t.Run("EmptyDir", func(t *testing.T) {
// 1.sqfs is a single empty directory - it must list without error.
entries, err := archiveFor(t, "1.sqfs").List(ctx, "")
require.NoError(t, err)
assert.Equal(t, 0, len(entries))
})
t.Run("NoXattrTree", func(t *testing.T) {
f := archiveFor(t, "2.sqfs")
entries, err := f.List(ctx, "")
require.NoError(t, err)
names := make([]string, 0, len(entries))
for _, e := range entries {
names = append(names, path.Base(e.Remote()))
}
assert.Contains(t, names, "alpha")
assert.Contains(t, names, "beta")
// A file in the tree must be readable with its real content.
obj, err := f.NewObject(ctx, "beta/sample.xml")
require.NoError(t, err)
assert.Greater(t, obj.Size(), int64(0))
rc, err := obj.Open(ctx)
require.NoError(t, err)
data, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, int(obj.Size()), len(data))
assert.True(t, bytes.HasPrefix(data, []byte("<?xml")))
})
}
+67
View File
@@ -0,0 +1,67 @@
//go:build !plan9
// Test Archive filesystem interface
package archive_test
import (
"testing"
_ "github.com/rclone/rclone/backend/local"
_ "github.com/rclone/rclone/backend/memory"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
)
var (
unimplementableFsMethods = []string{"ListR", "ListP", "MkdirMetadata", "DirSetModTime"}
// In these tests we receive objects from the underlying remote which don't implement these methods
unimplementableObjectMethods = []string{"GetTier", "ID", "Metadata", "MimeType", "SetTier", "UnWrap", "SetMetadata"}
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
if *fstest.RemoteName == "" {
t.Skip("Skipping as -remote not set")
}
fstests.Run(t, &fstests.Opt{
RemoteName: *fstest.RemoteName,
UnimplementableFsMethods: unimplementableFsMethods,
UnimplementableObjectMethods: unimplementableObjectMethods,
})
}
func TestLocal(t *testing.T) {
if *fstest.RemoteName != "" {
t.Skip("Skipping as -remote set")
}
remote := t.TempDir()
name := "TestArchiveLocal"
fstests.Run(t, &fstests.Opt{
RemoteName: name + ":",
ExtraConfig: []fstests.ExtraConfigItem{
{Name: name, Key: "type", Value: "archive"},
{Name: name, Key: "remote", Value: remote},
},
QuickTestOK: true,
UnimplementableFsMethods: unimplementableFsMethods,
UnimplementableObjectMethods: unimplementableObjectMethods,
})
}
func TestMemory(t *testing.T) {
if *fstest.RemoteName != "" {
t.Skip("Skipping as -remote set")
}
remote := ":memory:"
name := "TestArchiveMemory"
fstests.Run(t, &fstests.Opt{
RemoteName: name + ":",
ExtraConfig: []fstests.ExtraConfigItem{
{Name: name, Key: "type", Value: "archive"},
{Name: name, Key: "remote", Value: remote},
},
QuickTestOK: true,
UnimplementableFsMethods: unimplementableFsMethods,
UnimplementableObjectMethods: unimplementableObjectMethods,
})
}
+7
View File
@@ -0,0 +1,7 @@
// Build for archive for unsupported platforms to stop go complaining
// about "no buildable Go source files "
//go:build plan9
// Package archive implements a backend to access archive files in a remote
package archive
+24
View File
@@ -0,0 +1,24 @@
// Package archiver registers all the archivers
package archiver
import (
"context"
"github.com/rclone/rclone/fs"
)
// Archiver describes an archive package
type Archiver struct {
// New constructs an Fs from the (wrappedFs, remote) with the objects
// prefix with prefix and rooted at root
New func(ctx context.Context, f fs.Fs, remote, prefix, root string) (fs.Fs, error)
Extension string
}
// Archivers is a slice of all registered archivers
var Archivers []Archiver
// Register adds the archivers provided to the list of known archivers
func Register(as ...Archiver) {
Archivers = append(Archivers, as...)
}
+233
View File
@@ -0,0 +1,233 @@
// Package base is a base archive Fs
package base
import (
"context"
"errors"
"fmt"
"io"
"path"
"time"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/vfs"
)
// Fs represents a wrapped fs.Fs
type Fs struct {
f fs.Fs
wrapper fs.Fs
name string
features *fs.Features // optional features
vfs *vfs.VFS
node vfs.Node // archive object
remote string // remote of the archive object
prefix string // position for objects
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
}
var errNotImplemented = errors.New("internal error: method not implemented in archiver")
// New constructs an Fs from the (wrappedFs, remote) with the objects
// prefix with prefix and rooted at root
func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (*Fs, error) {
// FIXME vfs cache?
// FIXME could factor out ReadFileHandle and just use that rather than the full VFS
fs.Debugf(nil, "New: remote=%q, prefix=%q, root=%q", remote, prefix, root)
VFS := vfs.New(ctx, wrappedFs, nil)
node, err := VFS.Stat(remote)
if err != nil {
return nil, fmt.Errorf("failed to find %q archive: %w", remote, err)
}
f := &Fs{
f: wrappedFs,
name: path.Join(fs.ConfigString(wrappedFs), remote),
vfs: VFS,
node: node,
remote: remote,
root: root,
prefix: prefix,
prefixSlash: prefix + "/",
}
// FIXME
// the features here are ones we could support, and they are
// ANDed with the ones from wrappedFs
//
// FIXME some of these need to be forced on - CanHaveEmptyDirectories
f.features = (&fs.Features{
CaseInsensitive: false,
DuplicateFiles: false,
ReadMimeType: false, // MimeTypes not supported with gzip
WriteMimeType: false,
BucketBased: false,
CanHaveEmptyDirectories: true,
}).Fill(ctx, f).Mask(ctx, wrappedFs).WrapsFs(f, wrappedFs)
return f, nil
}
// Name of the remote (as passed into NewFs)
func (f *Fs) Name() string {
return f.name
}
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
return f.root
}
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
// String returns a description of the FS
func (f *Fs) String() string {
return f.name
}
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
//
// dir should be "" to list the root, and should not have
// trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
return nil, errNotImplemented
}
// NewObject finds the Object at remote.
func (f *Fs) NewObject(ctx context.Context, remote string) (o fs.Object, err error) {
return nil, errNotImplemented
}
// Precision of the ModTimes in this Fs
func (f *Fs) Precision() time.Duration {
return time.Second
}
// Mkdir makes the directory (container, bucket)
//
// Shouldn't return an error if it already exists
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
return vfs.EROFS
}
// Rmdir removes the directory (container, bucket) if empty
//
// Return an error if it doesn't exist or isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return vfs.EROFS
}
// Put in to the remote path with the modTime given of the given size
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (o fs.Object, err error) {
return nil, vfs.EROFS
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.None)
}
// UnWrap returns the Fs that this Fs is wrapping
func (f *Fs) UnWrap() fs.Fs {
return f.f
}
// WrapFs returns the Fs that is wrapping this Fs
func (f *Fs) WrapFs() fs.Fs {
return f.wrapper
}
// SetWrapper sets the Fs that is wrapping this Fs
func (f *Fs) SetWrapper(wrapper fs.Fs) {
f.wrapper = wrapper
}
// Object describes an object to be read from the raw zip file
type Object struct {
f *Fs
remote string
}
// Fs returns read only access to the Fs that this object is part of
func (o *Object) Fs() fs.Info {
return o.f
}
// Return a string version
func (o *Object) String() string {
if o == nil {
return "<nil>"
}
return o.Remote()
}
// Remote returns the remote path
func (o *Object) Remote() string {
return o.remote
}
// Size returns the size of the file
func (o *Object) Size() int64 {
return -1
}
// ModTime returns the modification time of the object
//
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
return time.Now()
}
// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
return vfs.EROFS
}
// Storable raturns a boolean indicating if this object is storable
func (o *Object) Storable() bool {
return true
}
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *Object) Hash(ctx context.Context, ht hash.Type) (string, error) {
return "", hash.ErrUnsupported
}
// Open opens the file for read. Call Close() on the returned io.ReadCloser
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
return nil, errNotImplemented
}
// Update in to the object with the modTime given of the given size
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
return vfs.EROFS
}
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
return vfs.EROFS
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
_ fs.UnWrapper = (*Fs)(nil)
_ fs.Wrapper = (*Fs)(nil)
_ fs.Object = (*Object)(nil)
)
+170
View File
@@ -0,0 +1,170 @@
package squashfs
// Could just be using bare object Open with RangeRequest which
// would transfer the minimum amount of data but may be slower.
import (
"errors"
"fmt"
"io/fs"
"os"
"sync"
"github.com/diskfs/go-diskfs/backend"
"github.com/rclone/rclone/vfs"
)
// Cache file handles for accessing the file
type cache struct {
node vfs.Node
fhsMu sync.Mutex
fhs []cacheHandle
}
// A cached file handle
type cacheHandle struct {
offset int64
fh vfs.Handle
}
// Make a new cache
func newCache(node vfs.Node) *cache {
return &cache{
node: node,
}
}
// Get a vfs.Handle from the pool or open one
//
// This tries to find an open file handle which doesn't require seeking.
func (c *cache) open(off int64) (fh vfs.Handle, err error) {
c.fhsMu.Lock()
defer c.fhsMu.Unlock()
if len(c.fhs) > 0 {
// Look for exact match first
for i, cfh := range c.fhs {
if cfh.offset == off {
// fs.Debugf(nil, "CACHE MATCH")
c.fhs = append(c.fhs[:i], c.fhs[i+1:]...)
return cfh.fh, nil
}
}
// fs.Debugf(nil, "CACHE MISS")
// Just take the first one if not found
cfh := c.fhs[0]
c.fhs = c.fhs[1:]
return cfh.fh, nil
}
fh, err = c.node.Open(os.O_RDONLY)
if err != nil {
return nil, fmt.Errorf("failed to open squashfs archive: %w", err)
}
return fh, nil
}
// Close a vfs.Handle or return it to the pool
//
// off should be the offset the file handle would read from without seeking
func (c *cache) close(fh vfs.Handle, off int64) {
c.fhsMu.Lock()
defer c.fhsMu.Unlock()
c.fhs = append(c.fhs, cacheHandle{
offset: off,
fh: fh,
})
}
// ReadAt reads len(p) bytes into p starting at offset off in the underlying
// input source. It returns the number of bytes read (0 <= n <= len(p)) and any
// error encountered.
//
// When ReadAt returns n < len(p), it returns a non-nil error explaining why
// more bytes were not returned. In this respect, ReadAt is stricter than Read.
//
// Even if ReadAt returns n < len(p), it may use all of p as scratch
// space during the call. If some data is available but not len(p) bytes,
// ReadAt blocks until either all the data is available or an error occurs.
// In this respect ReadAt is different from Read.
//
// If the n = len(p) bytes returned by ReadAt are at the end of the input
// source, ReadAt may return either err == EOF or err == nil.
//
// If ReadAt is reading from an input source with a seek offset, ReadAt should
// not affect nor be affected by the underlying seek offset.
//
// Clients of ReadAt can execute parallel ReadAt calls on the same input
// source.
//
// Implementations must not retain p.
func (c *cache) ReadAt(p []byte, off int64) (n int, err error) {
fh, err := c.open(off)
if err != nil {
return n, err
}
defer func() {
c.close(fh, off+int64(len(p)))
}()
// fs.Debugf(nil, "ReadAt(p[%d], off=%d, fh=%p)", len(p), off, fh)
return fh.ReadAt(p, off)
}
var errCacheNotImplemented = errors.New("internal error: squashfs cache doesn't implement method")
// WriteAt method dummy stub to satisfy interface
func (c *cache) WriteAt(p []byte, off int64) (n int, err error) {
return 0, errCacheNotImplemented
}
// Seek method dummy stub to satisfy interface
func (c *cache) Seek(offset int64, whence int) (int64, error) {
return 0, errCacheNotImplemented
}
// Read method dummy stub to satisfy interface
func (c *cache) Read(p []byte) (n int, err error) {
return 0, errCacheNotImplemented
}
func (c *cache) Stat() (fs.FileInfo, error) {
return nil, errCacheNotImplemented
}
// Close the file
func (c *cache) Close() (err error) {
c.fhsMu.Lock()
defer c.fhsMu.Unlock()
// Close any open file handles
for i := range c.fhs {
fh := &c.fhs[i]
newErr := fh.fh.Close()
if err == nil {
err = newErr
}
}
c.fhs = nil
return err
}
// Sys returns OS-specific file for ioctl calls via fd
func (c *cache) Sys() (*os.File, error) {
return nil, errCacheNotImplemented
}
// Writable returns file for read-write operations
func (c *cache) Writable() (backend.WritableFile, error) {
return nil, errCacheNotImplemented
}
// Path returns the path to the underlying storage - not applicable here
func (c *cache) Path() string {
return ""
}
// check interfaces
var _ backend.Storage = (*cache)(nil)
+482
View File
@@ -0,0 +1,482 @@
// Package squashfs implements a squashfs archiver for the archive backend
package squashfs
import (
"context"
"fmt"
"io"
"os"
"path"
"strings"
"time"
"github.com/diskfs/go-diskfs/filesystem/squashfs"
"github.com/rclone/rclone/backend/archive/archiver"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/vfs"
"github.com/rclone/rclone/vfs/vfscommon"
)
func init() {
archiver.Register(archiver.Archiver{
New: New,
Extension: ".sqfs",
})
}
// Fs represents a wrapped fs.Fs
type Fs struct {
f fs.Fs
wrapper fs.Fs
name string
features *fs.Features // optional features
vfs *vfs.VFS
sqfs *squashfs.FileSystem // interface to the squashfs
c *cache
node vfs.Node // squashfs file object - set if reading
remote string // remote of the squashfs file object
prefix string // position for objects
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
}
// recoverParsePanic converts a panic raised while parsing a squashfs image
// into an error assigned through err.
func recoverParsePanic(err *error) {
if r := recover(); r != nil {
*err = fmt.Errorf("invalid or corrupt squashfs image: %v", r)
}
}
// recoveringReadCloser wraps a reader returned by the go-diskfs parser so
// panics raised while reading become errors.
//
// The parser reads file data lazily, so an image which opens cleanly can
// still panic on the first Read.
type recoveringReadCloser struct {
io.ReadCloser
}
func (r recoveringReadCloser) Read(p []byte) (n int, err error) {
defer recoverParsePanic(&err)
return r.ReadCloser.Read(p)
}
// New constructs an Fs from the (wrappedFs, remote) with the objects
// prefix with prefix and rooted at root
func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (_ fs.Fs, err error) {
defer recoverParsePanic(&err)
// FIXME vfs cache?
// FIXME could factor out ReadFileHandle and just use that rather than the full VFS
fs.Debugf(nil, "Squashfs: New: remote=%q, prefix=%q, root=%q", remote, prefix, root)
vfsOpt := vfscommon.Opt
vfsOpt.ReadWait = 0
VFS := vfs.New(ctx, wrappedFs, &vfsOpt)
node, err := VFS.Stat(remote)
if err != nil {
return nil, fmt.Errorf("failed to find %q archive: %w", remote, err)
}
c := newCache(node)
// FIXME blocksize
sqfs, err := squashfs.Read(c, node.Size(), 0, 1024*1024)
if err != nil {
return nil, fmt.Errorf("failed to read squashfs: %w", err)
}
f := &Fs{
f: wrappedFs,
name: path.Join(fs.ConfigString(wrappedFs), remote),
vfs: VFS,
node: node,
sqfs: sqfs,
c: c,
remote: remote,
root: strings.Trim(root, "/"),
prefix: prefix,
prefixSlash: prefix + "/",
}
if prefix == "" {
f.prefixSlash = ""
}
singleObject := false
// Find the directory the root points to
if f.root != "" && !strings.HasSuffix(root, "/") {
native, err := f.toNative("")
if err == nil {
native = strings.TrimRight(native, "/")
_, err := f.newObjectNative(native)
if err == nil {
// If it pointed to a file, find the directory above
f.root = path.Dir(f.root)
if f.root == "." || f.root == "/" {
f.root = ""
}
}
}
}
// FIXME
// the features here are ones we could support, and they are
// ANDed with the ones from wrappedFs
//
// FIXME some of these need to be forced on - CanHaveEmptyDirectories
f.features = (&fs.Features{
CaseInsensitive: false,
DuplicateFiles: false,
ReadMimeType: false, // MimeTypes not supported with gsquashfs
WriteMimeType: false,
BucketBased: false,
CanHaveEmptyDirectories: true,
}).Fill(ctx, f).Mask(ctx, wrappedFs).WrapsFs(f, wrappedFs)
if singleObject {
return f, fs.ErrorIsFile
}
return f, nil
}
// Name of the remote (as passed into NewFs)
func (f *Fs) Name() string {
return f.name
}
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
return f.root
}
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
// String returns a description of the FS
func (f *Fs) String() string {
return fmt.Sprintf("Squashfs %q", f.name)
}
// This turns a remote into a native path in the squashfs starting with a /
func (f *Fs) toNative(remote string) (string, error) {
native := strings.Trim(remote, "/")
if f.prefix == "" {
native = "/" + native
} else if native == f.prefix {
native = "/"
} else if !strings.HasPrefix(native, f.prefixSlash) {
return "", fmt.Errorf("internal error: %q doesn't start with prefix %q", native, f.prefixSlash)
} else {
native = native[len(f.prefix):]
}
if f.root != "" {
native = "/" + f.root + native
}
return native, nil
}
// This turns a native path with a leading / into a path suitable for
// the squashfs library which requires io/fs.ValidPath paths with no
// leading / and "." for the root directory.
func toIOFS(native string) string {
native = strings.Trim(native, "/")
if native == "" {
return "."
}
return native
}
// Turn a (nativeDir, leaf) into a remote
func (f *Fs) fromNative(nativeDir string, leaf string) string {
// fs.Debugf(nil, "nativeDir = %q, leaf = %q, root=%q", nativeDir, leaf, f.root)
dir := nativeDir
if f.root != "" {
dir = strings.TrimPrefix(dir, "/"+f.root)
}
remote := f.prefixSlash + strings.Trim(path.Join(dir, leaf), "/")
// fs.Debugf(nil, "dir = %q, remote=%q", dir, remote)
return remote
}
// Convert a FileInfo into an Object from native dir
func (f *Fs) objectFromFileInfo(nativeDir string, item os.FileInfo) *Object {
return &Object{
fs: f,
remote: f.fromNative(nativeDir, item.Name()),
size: item.Size(),
modTime: item.ModTime(),
}
}
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
//
// dir should be "" to list the root, and should not have
// trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
defer log.Trace(f, "dir=%q", dir)("entries=%v, err=%v", &entries, &err)
defer recoverParsePanic(&err)
nativeDir, err := f.toNative(dir)
if err != nil {
return nil, err
}
items, err := f.sqfs.ReadDir(toIOFS(nativeDir))
if err != nil {
return nil, fmt.Errorf("read squashfs: couldn't read directory: %w", err)
}
entries = make(fs.DirEntries, 0, len(items))
for _, item := range items {
// fs.Debugf(item.Name(), "entry = %#v", item)
var entry fs.DirEntry
if item.IsDir() {
remote := f.fromNative(nativeDir, item.Name())
info, err := item.Info()
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", item.Name(), err)
}
entry = fs.NewDir(remote, info.ModTime())
} else if item.Type().IsRegular() {
info, err := item.Info()
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", item.Name(), err)
}
entry = f.objectFromFileInfo(nativeDir, info)
} else {
fs.Debugf(item.Name(), "FIXME Not regular file - skipping")
continue
}
entries = append(entries, entry)
}
// fs.Debugf(f, "dir=%q, entries=%v", dir, entries)
return entries, nil
}
// newObjectNative finds the object at the native path passed in
func (f *Fs) newObjectNative(nativePath string) (o fs.Object, err error) {
defer recoverParsePanic(&err)
// get the path and filename
dir, leaf := path.Split(nativePath)
dir = strings.TrimRight(dir, "/")
leaf = strings.Trim(leaf, "/")
// FIXME need to detect directory not found
fis, err := f.sqfs.ReadDir(toIOFS(dir))
if err != nil {
return nil, fs.ErrorObjectNotFound
}
for _, fi := range fis {
if fi.Name() == leaf {
if fi.IsDir() {
return nil, fs.ErrorNotAFile
}
info, err := fi.Info()
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", fi.Name(), err)
}
o = f.objectFromFileInfo(dir, info)
break
}
}
if o == nil {
return nil, fs.ErrorObjectNotFound
}
return o, nil
}
// NewObject finds the Object at remote.
func (f *Fs) NewObject(ctx context.Context, remote string) (o fs.Object, err error) {
defer log.Trace(f, "remote=%q", remote)("obj=%v, err=%v", &o, &err)
nativePath, err := f.toNative(remote)
if err != nil {
return nil, err
}
return f.newObjectNative(nativePath)
}
// Precision of the ModTimes in this Fs
func (f *Fs) Precision() time.Duration {
return time.Second
}
// Mkdir makes the directory (container, bucket)
//
// Shouldn't return an error if it already exists
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
return vfs.EROFS
}
// Rmdir removes the directory (container, bucket) if empty
//
// Return an error if it doesn't exist or isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return vfs.EROFS
}
// Put in to the remote path with the modTime given of the given size
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (o fs.Object, err error) {
return nil, vfs.EROFS
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.None)
}
// UnWrap returns the Fs that this Fs is wrapping
func (f *Fs) UnWrap() fs.Fs {
return f.f
}
// WrapFs returns the Fs that is wrapping this Fs
func (f *Fs) WrapFs() fs.Fs {
return f.wrapper
}
// SetWrapper sets the Fs that is wrapping this Fs
func (f *Fs) SetWrapper(wrapper fs.Fs) {
f.wrapper = wrapper
}
// Object describes an object to be read from the raw squashfs file
type Object struct {
fs *Fs
remote string
size int64
modTime time.Time
}
// Fs returns read only access to the Fs that this object is part of
func (o *Object) Fs() fs.Info {
return o.fs
}
// Return a string version
func (o *Object) String() string {
if o == nil {
return "<nil>"
}
return o.Remote()
}
// Turn a squashfs path into a full path for the parent Fs
// func (o *Object) path(remote string) string {
// return path.Join(o.fs.prefix, remote)
// }
// Remote returns the remote path
func (o *Object) Remote() string {
return o.remote
}
// Size returns the size of the file
func (o *Object) Size() int64 {
return o.size
}
// ModTime returns the modification time of the object
//
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
return o.modTime
}
// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
return vfs.EROFS
}
// Storable raturns a boolean indicating if this object is storable
func (o *Object) Storable() bool {
return true
}
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *Object) Hash(ctx context.Context, ht hash.Type) (string, error) {
return "", hash.ErrUnsupported
}
// Open opens the file for read. Call Close() on the returned io.ReadCloser
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
defer recoverParsePanic(&err)
var offset, limit int64 = 0, -1
for _, option := range options {
switch x := option.(type) {
case *fs.SeekOption:
offset = x.Offset
case *fs.RangeOption:
offset, limit = x.Decode(o.Size())
default:
if option.Mandatory() {
fs.Logf(o, "Unsupported mandatory option: %v", option)
}
}
}
remote, err := o.fs.toNative(o.remote)
if err != nil {
return nil, err
}
fs.Debugf(o, "Opening %q", remote)
fh, err := o.fs.sqfs.OpenFile(toIOFS(remote), os.O_RDONLY)
if err != nil {
return nil, err
}
// discard data from start as necessary
if offset > 0 {
_, err = fh.Seek(offset, io.SeekStart)
if err != nil {
return nil, err
}
}
rc = recoveringReadCloser{fh}
// If limited then don't return everything
if limit >= 0 {
fs.Debugf(nil, "limit=%d, offset=%d, options=%v", limit, offset, options)
return readers.NewLimitedReadCloser(rc, limit), nil
}
return rc, nil
}
// Update in to the object with the modTime given of the given size
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
return vfs.EROFS
}
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
return vfs.EROFS
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
_ fs.UnWrapper = (*Fs)(nil)
_ fs.Wrapper = (*Fs)(nil)
_ fs.Object = (*Object)(nil)
)
+25
View File
@@ -0,0 +1,25 @@
package squashfs
import (
"io"
"testing"
"github.com/stretchr/testify/assert"
)
// panickingReader panics on Read, standing in for the go-diskfs parser
// hitting an unchecked slice while reading a malformed image.
type panickingReader struct{}
func (panickingReader) Read([]byte) (int, error) { panic("boom") }
func (panickingReader) Close() error { return nil }
// Test that a panic raised while reading file data becomes an error. The
// parser reads lazily, so an image can open cleanly and only panic here.
func TestRecoveringReadCloser(t *testing.T) {
var rc io.ReadCloser = recoveringReadCloser{panickingReader{}}
_, err := rc.Read(make([]byte, 16))
assert.ErrorContains(t, err, "invalid or corrupt squashfs image")
assert.ErrorContains(t, err, "boom")
}
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,89 @@
//go:build !plan9
package archive
import (
"context"
"encoding/binary"
"os"
"path/filepath"
"testing"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/fs/cache"
"github.com/rclone/rclone/fstest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// squashfsSuperblock builds a minimal 96-byte squashfs superblock with the
// given attacker-controllable fields. It mirrors the on-disk layout the
// go-diskfs parser reads, and is only used to construct deliberately malformed
// images for the tests below.
func squashfsSuperblock(blockSize uint32, blockLog, compression uint16, rootInode, inodeTableStart uint64) []byte {
const u64max = ^uint64(0)
b := make([]byte, 96)
binary.LittleEndian.PutUint32(b[0:], 0x73717368) // "hsqs" magic
binary.LittleEndian.PutUint32(b[4:], 1) // inode count
binary.LittleEndian.PutUint32(b[8:], 0) // modtime
binary.LittleEndian.PutUint32(b[12:], blockSize)
binary.LittleEndian.PutUint32(b[16:], 0) // fragment count
binary.LittleEndian.PutUint16(b[20:], compression)
binary.LittleEndian.PutUint16(b[22:], blockLog)
binary.LittleEndian.PutUint16(b[24:], 0x0210) // no fragments/xattrs
binary.LittleEndian.PutUint16(b[26:], 0) // id count
binary.LittleEndian.PutUint16(b[28:], 4) // version major
binary.LittleEndian.PutUint16(b[30:], 0) // version minor
binary.LittleEndian.PutUint64(b[32:], rootInode)
binary.LittleEndian.PutUint64(b[40:], 96) // bytes used
binary.LittleEndian.PutUint64(b[48:], u64max)
binary.LittleEndian.PutUint64(b[56:], u64max)
binary.LittleEndian.PutUint64(b[64:], inodeTableStart)
binary.LittleEndian.PutUint64(b[72:], inodeTableStart)
binary.LittleEndian.PutUint64(b[80:], u64max)
binary.LittleEndian.PutUint64(b[88:], u64max)
return b
}
// TestArchiveSquashfsMalformed feeds the archive backend squashfs images whose
// superblock/metadata make the go-diskfs parser panic. See GHSA-6jcg-q3wp-x2f4.
func TestArchiveSquashfsMalformed(t *testing.T) {
fstest.Initialise()
ctx := context.Background()
// Variant 1: zero block size reaches an integer division in the parser.
zeroBlock := squashfsSuperblock(0, 0, 1, 0, 96)
// Variant 2: an out-of-range root inode byte offset reaches an unchecked
// slice against an empty, uncompressed metadata block.
oob := append(squashfsSuperblock(65536, 16, 1, 12336, 96), 0x00, 0x80)
binary.LittleEndian.PutUint64(oob[40:], uint64(len(oob)))
for _, tc := range []struct {
name string
image []byte
}{
{"ZeroBlockSize", zeroBlock},
{"OutOfRangeInodeOffset", oob},
} {
t.Run(tc.name, func(t *testing.T) {
// A directory per image: the fs cache keys on the parent
// directory, so sharing one would hand the second subtest a
// cached listing which does not contain its image.
path := filepath.Join(t.TempDir(), tc.name+".sqfs")
require.NoError(t, os.WriteFile(path, tc.image, 0o644))
// The whole point is that this returns rather than panicking and
// killing the test binary (and, in production, the serve daemon).
f, err := cache.Get(ctx, ":archive:"+path)
if err == nil {
// The image parsed, so the panic must surface from a later
// lazily-parsed operation instead.
_, err = f.List(ctx, "")
}
require.Error(t, err, "a malformed squashfs image must error, not panic")
assert.Contains(t, err.Error(), "invalid or corrupt squashfs image",
"error should come from the recover, not an unrelated failure")
})
}
}
+385
View File
@@ -0,0 +1,385 @@
// Package zip implements a zip archiver for the archive backend
package zip
import (
"archive/zip"
"context"
"errors"
"fmt"
"io"
"os"
"path"
"strings"
"time"
"github.com/rclone/rclone/backend/archive/archiver"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/dirtree"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/vfs"
"github.com/rclone/rclone/vfs/vfscommon"
)
func init() {
archiver.Register(archiver.Archiver{
New: New,
Extension: ".zip",
})
}
// Fs represents a wrapped fs.Fs
type Fs struct {
f fs.Fs
wrapper fs.Fs
name string
features *fs.Features // optional features
vfs *vfs.VFS
node vfs.Node // zip file object - set if reading
remote string // remote of the zip file object
prefix string // position for objects
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
dt dirtree.DirTree // read from zipfile
}
// New constructs an Fs from the (wrappedFs, remote) with the objects
// prefix with prefix and rooted at root
func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (fs.Fs, error) {
// FIXME vfs cache?
// FIXME could factor out ReadFileHandle and just use that rather than the full VFS
fs.Debugf(nil, "Zip: New: remote=%q, prefix=%q, root=%q", remote, prefix, root)
vfsOpt := vfscommon.Opt
vfsOpt.ReadWait = 0
VFS := vfs.New(ctx, wrappedFs, &vfsOpt)
node, err := VFS.Stat(remote)
if err != nil {
return nil, fmt.Errorf("failed to find %q archive: %w", remote, err)
}
f := &Fs{
f: wrappedFs,
name: path.Join(fs.ConfigString(wrappedFs), remote),
vfs: VFS,
node: node,
remote: remote,
root: root,
prefix: prefix,
prefixSlash: prefix + "/",
}
// Read the contents of the zip file
singleObject, err := f.readZip()
if err != nil {
return nil, fmt.Errorf("failed to open zip file: %w", err)
}
// FIXME
// the features here are ones we could support, and they are
// ANDed with the ones from wrappedFs
//
// FIXME some of these need to be forced on - CanHaveEmptyDirectories
f.features = (&fs.Features{
CaseInsensitive: false,
DuplicateFiles: false,
ReadMimeType: false, // MimeTypes not supported with gzip
WriteMimeType: false,
BucketBased: false,
CanHaveEmptyDirectories: true,
}).Fill(ctx, f).Mask(ctx, wrappedFs).WrapsFs(f, wrappedFs)
if singleObject {
return f, fs.ErrorIsFile
}
return f, nil
}
// Name of the remote (as passed into NewFs)
func (f *Fs) Name() string {
return f.name
}
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
return f.root
}
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
// String returns a description of the FS
func (f *Fs) String() string {
return fmt.Sprintf("Zip %q", f.name)
}
// readZip the zip file into f
//
// Returns singleObject=true if f.root points to a file
func (f *Fs) readZip() (singleObject bool, err error) {
if f.node == nil {
return singleObject, fs.ErrorDirNotFound
}
size := f.node.Size()
if size < 0 {
return singleObject, errors.New("can't read from zip file with unknown size")
}
r, err := f.node.Open(os.O_RDONLY)
if err != nil {
return singleObject, fmt.Errorf("failed to open zip file: %w", err)
}
zr, err := zip.NewReader(r, size)
if err != nil {
return singleObject, fmt.Errorf("failed to read zip file: %w", err)
}
dt := dirtree.New()
for _, file := range zr.File {
remote := strings.Trim(path.Clean(file.Name), "/")
if remote == "." {
remote = ""
}
remote = path.Join(f.prefix, remote)
if f.root != "" {
// Ignore all files outside the root
if !strings.HasPrefix(remote, f.root) {
continue
}
if remote == f.root {
remote = ""
} else {
remote = strings.TrimPrefix(remote, f.root+"/")
}
}
if strings.HasSuffix(file.Name, "/") {
dir := fs.NewDir(remote, file.Modified)
dt.AddDir(dir)
} else {
if remote == "" {
remote = path.Base(f.root)
singleObject = true
dt = dirtree.New()
}
o := &Object{
f: f,
remote: remote,
fh: &file.FileHeader,
file: file,
}
dt.Add(o)
if singleObject {
break
}
}
}
dt.CheckParents("")
dt.Sort()
f.dt = dt
//fs.Debugf(nil, "dt = %v", dt)
return singleObject, nil
}
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
//
// dir should be "" to list the root, and should not have
// trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
defer log.Trace(f, "dir=%q", dir)("entries=%v, err=%v", &entries, &err)
// _, err = f.strip(dir)
// if err != nil {
// return nil, err
// }
entries, ok := f.dt[dir]
if !ok {
return nil, fs.ErrorDirNotFound
}
fs.Debugf(f, "dir=%q, entries=%v", dir, entries)
return entries, nil
}
// NewObject finds the Object at remote.
func (f *Fs) NewObject(ctx context.Context, remote string) (o fs.Object, err error) {
defer log.Trace(f, "remote=%q", remote)("obj=%v, err=%v", &o, &err)
if f.dt == nil {
return nil, fs.ErrorObjectNotFound
}
_, entry := f.dt.Find(remote)
if entry == nil {
return nil, fs.ErrorObjectNotFound
}
o, ok := entry.(*Object)
if !ok {
return nil, fs.ErrorNotAFile
}
return o, nil
}
// Precision of the ModTimes in this Fs
func (f *Fs) Precision() time.Duration {
return time.Second
}
// Mkdir makes the directory (container, bucket)
//
// Shouldn't return an error if it already exists
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
return vfs.EROFS
}
// Rmdir removes the directory (container, bucket) if empty
//
// Return an error if it doesn't exist or isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return vfs.EROFS
}
// Put in to the remote path with the modTime given of the given size
//
// May create the object even if it returns an error - if so
// will return the object and the error, otherwise will return
// nil and the error
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (o fs.Object, err error) {
return nil, vfs.EROFS
}
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.CRC32)
}
// UnWrap returns the Fs that this Fs is wrapping
func (f *Fs) UnWrap() fs.Fs {
return f.f
}
// WrapFs returns the Fs that is wrapping this Fs
func (f *Fs) WrapFs() fs.Fs {
return f.wrapper
}
// SetWrapper sets the Fs that is wrapping this Fs
func (f *Fs) SetWrapper(wrapper fs.Fs) {
f.wrapper = wrapper
}
// Object describes an object to be read from the raw zip file
type Object struct {
f *Fs
remote string
fh *zip.FileHeader
file *zip.File
}
// Fs returns read only access to the Fs that this object is part of
func (o *Object) Fs() fs.Info {
return o.f
}
// Return a string version
func (o *Object) String() string {
if o == nil {
return "<nil>"
}
return o.Remote()
}
// Remote returns the remote path
func (o *Object) Remote() string {
return o.remote
}
// Size returns the size of the file
func (o *Object) Size() int64 {
return int64(o.fh.UncompressedSize64)
}
// ModTime returns the modification time of the object
//
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
return o.fh.Modified
}
// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
return vfs.EROFS
}
// Storable raturns a boolean indicating if this object is storable
func (o *Object) Storable() bool {
return true
}
// Hash returns the selected checksum of the file
// If no checksum is available it returns ""
func (o *Object) Hash(ctx context.Context, ht hash.Type) (string, error) {
if ht == hash.CRC32 {
// FIXME return empty CRC if writing
if o.f.dt == nil {
return "", nil
}
return fmt.Sprintf("%08x", o.fh.CRC32), nil
}
return "", hash.ErrUnsupported
}
// Open opens the file for read. Call Close() on the returned io.ReadCloser
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
var offset, limit int64 = 0, -1
for _, option := range options {
switch x := option.(type) {
case *fs.SeekOption:
offset = x.Offset
case *fs.RangeOption:
offset, limit = x.Decode(o.Size())
default:
if option.Mandatory() {
fs.Logf(o, "Unsupported mandatory option: %v", option)
}
}
}
rc, err = o.file.Open()
if err != nil {
return nil, err
}
// discard data from start as necessary
if offset > 0 {
_, err = io.CopyN(io.Discard, rc, offset)
if err != nil {
return nil, err
}
}
// If limited then don't return everything
if limit >= 0 {
return readers.NewLimitedReadCloser(rc, limit), nil
}
return rc, nil
}
// Update in to the object with the modTime given of the given size
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
return vfs.EROFS
}
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
return vfs.EROFS
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
_ fs.UnWrapper = (*Fs)(nil)
_ fs.Wrapper = (*Fs)(nil)
_ fs.Object = (*Object)(nil)
)
+169
View File
@@ -0,0 +1,169 @@
//go:build !plan9 && !solaris && !js
// Parallel listing support for azureblob, built on the Apache Arrow
// listing format.
//
// The Arrow listing exposes the server-side startFrom (inclusive) and endBefore
// (exclusive) name-range parameters, which let a listing be sharded by
// blob-name range and listed concurrently.
package azureblob
import (
"context"
"errors"
"fmt"
"sync"
"sync/atomic"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/rclone/rclone/backend/azureblob/arrowlist"
"github.com/rclone/rclone/fs"
"golang.org/x/sync/errgroup"
)
// listArrowParallel lists (containerName, directory) by splitting the blob name
// keyspace into shards and listing them concurrently. opts must already request
// the Arrow format; it is cloned per shard with StartFrom/EndBefore set. It
// returns the number of raw items seen. directory/prefix are already normalised
// by the caller.
func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, prefix string, addContainer bool, opts *arrowlist.ListBlobsHierarchyOptions, delimiter string, fn listFn) (int, error) {
// Parallel listing needs the arrowlist client (server-side endBefore is
// only honoured on the Arrow path) - without it fall back to sequential.
if _, err := f.arrowCntSVC(containerName); err != nil {
fs.Debugf(f, "Not using parallel Arrow listing: %v", err)
return f.listBlobsPager(ctx, containerName, directory, prefix, addContainer, opts, delimiter, fn)
}
// Split the keyspace by a single-case character ladder (see
// arrowLadderBoundaries for why single-case matters).
points := arrowLadderBoundaries(directory, arrowShardTarget(f.opt.ListParallelism))
if len(points) < 1 {
// Couldn't usefully split - list sequentially.
return f.listBlobsPager(ctx, containerName, directory, prefix, addContainer, opts, delimiter, fn)
}
// Build half-open [startFrom, endBefore) shards from the sorted boundary
// points p0..pk: ["", p0), [p0, p1), ..., [pk, "").
type shard struct{ startFrom, endBefore string }
shards := make([]shard, 0, len(points)+1)
prev := ""
for _, p := range points {
shards = append(shards, shard{prev, p})
prev = p
}
shards = append(shards, shard{prev, ""})
// fn (and the list.Helper behind it) is not concurrency-safe, so serialise
// delivery. Only the cheap callback is locked, not the HTTP/decode work.
var fnMu sync.Mutex
safeFn := func(remote string, item *container.BlobItem, isDir bool) error {
fnMu.Lock()
defer fnMu.Unlock()
return fn(remote, item, isDir)
}
// Schedule every shard; SetLimit bounds concurrency, so more shards than
// workers gives free load balancing as workers pick up the next shard.
var total int64
g, gCtx := errgroup.WithContext(ctx)
g.SetLimit(f.opt.ListParallelism)
for _, sh := range shards {
if gCtx.Err() != nil {
break
}
g.Go(func() error {
// Clone opts per shard so concurrent shards don't share StartFrom/EndBefore.
shardOpts := *opts
if sh.startFrom != "" {
startFrom := sh.startFrom
shardOpts.StartFrom = &startFrom
} else {
shardOpts.StartFrom = nil
}
if sh.endBefore != "" {
endBefore := sh.endBefore
shardOpts.EndBefore = &endBefore
} else {
shardOpts.EndBefore = nil
}
n, err := f.listBlobsPager(gCtx, containerName, directory, prefix, addContainer, &shardOpts, delimiter, safeFn)
atomic.AddInt64(&total, int64(n))
return err
})
}
err := g.Wait()
found := int(atomic.LoadInt64(&total))
if isEndBeforeUnsupported(err) {
// The account can't honour the shards' endBefore bounds, so parallel
// listing can't be used on it.
f.arrowXMLFallback.Store(true)
if found == 0 {
// No shard delivered anything yet (the normal case - Arrow listing
// enablement is per account so every shard fails on its first
// page) so it is safe to relist sequentially.
fs.Debugf(f, "Parallel Arrow listing not supported by this account (%v) - listing sequentially", err)
return f.listBlobsPager(ctx, containerName, directory, prefix, addContainer, opts, delimiter, fn)
}
// Some entries were already delivered so a rerun would duplicate them.
return found, fmt.Errorf("parallel listing aborted after partial results (retry, or unset list_parallelism): %w", err)
}
return found, err
}
// isEndBeforeUnsupported reports whether err means the service can't honour
// the endBefore listing bound, so sharded parallel listing must not be used.
//
// This happens two ways: the arrowlist pager's ErrEndBeforeXMLFallback (the
// server answered XML with endBefore set), or the service rejecting the
// endBefore parameter with 400 OperationNotSupportedWithFeatureMissing "The
// requested operation is not allowed as EndBefore parameter support is
// missing on this account" - endBefore is only supported on the Arrow
// listing path of accounts with Blob Listing with Apache Arrow enabled.
func isEndBeforeUnsupported(err error) bool {
if errors.Is(err, arrowlist.ErrEndBeforeXMLFallback) {
return true
}
var respErr *azcore.ResponseError
return errors.As(err, &respErr) && respErr.ErrorCode == "OperationNotSupportedWithFeatureMissing"
}
// arrowShardTarget is the number of ladder boundaries to use for the given
// parallelism (a few per worker for load balancing).
func arrowShardTarget(parallelism int) int {
return parallelism * 4
}
// arrowLadderBoundaries returns up to target sorted ENCODED-space boundaries of
// the form directory+c, used to split the keyspace into shards.
//
// The partition is always correct (no dup/gap) regardless of the alphabet: keys
// whose next byte is outside the alphabet simply fall into an adjacent shard;
// the alphabet only affects balance. A single-case (digit+lowercase) alphabet
// is deliberate: the boundaries are then monotonic under BOTH byte order (the
// listing order) AND the service's case-insensitive startFrom/endBefore
// validation, so server-side endBefore is accepted. (A mixed-case ladder gives
// pairs like [".../Z", ".../a") which are byte-ascending but rejected 400 as
// "endbefore precedes startfrom".) directory is already encoded; the alphabet
// is ASCII (encoder-invariant) and ascending, so the result is sorted.
func arrowLadderBoundaries(directory string, target int) []string {
const alphabet = "0123456789abcdefghijklmnopqrstuvwxyz"
if target < 1 {
return nil
}
if target > len(alphabet) {
target = len(alphabet)
}
points := make([]string, 0, target)
last := -1
for i := 0; i < target; i++ {
idx := i * len(alphabet) / target
if idx == last {
continue
}
last = idx
points = append(points, directory+string(alphabet[idx]))
}
return points
}
+221
View File
@@ -0,0 +1,221 @@
//go:build !plan9 && !solaris && !js
package azureblob
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/rclone/rclone/backend/azureblob/arrowlist"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// testArrowList exercises the experimental Apache Arrow listing.
//
// It is skipped automatically if Arrow listing is not enabled on the test
// account (the server returns XML, or 409 on HNS accounts).
func (f *Fs) testArrowList(t *testing.T) {
ctx := context.Background()
containerName, _ := f.split("probe")
if containerName == "" {
t.Skip("Arrow list test needs a container in the remote root")
}
// withArrow runs fn with UseArrowList temporarily set.
withArrow := func(on bool, fn func()) {
old := f.opt.UseArrowList
f.opt.UseArrowList = on
defer func() { f.opt.UseArrowList = old }()
fn()
}
// withParallel runs fn with ListParallelism temporarily set.
withParallel := func(n int, fn func()) {
old := f.opt.ListParallelism
f.opt.ListParallelism = n
defer func() { f.opt.ListParallelism = old }()
fn()
}
// Create a set of known objects under a unique directory: flat files plus
// one in a subdirectory (so we exercise both the flat and delimited paths).
dir := "arrow-list-" + random.String(8)
// Includes upper-case and digit first-characters: these byte-sort between
// the digit and lowercase ladder boundaries, so they verify the server's
// endBefore filtering is byte-ordered (not case-insensitive, which would
// drop them from a shard and make parallel != sequential).
objects := []string{"a.txt", "b.txt", "c.txt", "d.txt", "e.txt", "sub/x.txt", "Foo.txt", "Zoo.txt", "0num.txt", "9end.txt"}
wantSizes := map[string]int64{}
created := make([]string, 0, len(objects))
for i, n := range objects {
contents := random.String(10 + i)
remote := dir + "/" + n
item := fstest.NewItem(remote, contents, fstest.Time("2001-05-06T04:05:06.499Z"))
_ = fstests.PutTestContents(ctx, t, f, &item, contents, true)
created = append(created, remote)
wantSizes[remote] = int64(len(contents))
}
defer func() {
for _, remote := range created {
if o, err := f.NewObject(ctx, remote); err == nil {
_ = o.Remove(ctx)
}
}
}()
// listing returns remote->size for dir (dirs reported as size -1).
listing := func(recurse bool) (map[string]int64, error) {
got := map[string]int64{}
add := func(entries fs.DirEntries) error {
for _, e := range entries {
if o, ok := e.(fs.Object); ok {
got[e.Remote()] = o.Size()
} else {
got[e.Remote()] = -1
}
}
return nil
}
if recurse {
return got, f.ListR(ctx, dir, add)
}
entries, err := f.List(ctx, dir)
if err != nil {
return nil, err
}
_ = add(entries)
return got, nil
}
// Probe: is Arrow available? If not, skip cleanly.
var probeErr error
withArrow(true, func() { _, probeErr = listing(true) })
if probeErr != nil {
if strings.Contains(probeErr.Error(), "OperationNotSupportedWithFeatureMissing") || strings.Contains(probeErr.Error(), "409") {
t.Skipf("Arrow listing not available on this account: %v", probeErr)
}
require.NoError(t, probeErr)
}
// Arrow listing must equal XML listing, recursively and per-directory
// (including the "sub" subdirectory).
t.Run("EqualsXML", func(t *testing.T) {
for _, recurse := range []bool{true, false} {
var xml, arrow map[string]int64
var err error
withArrow(false, func() { xml, err = listing(recurse) })
require.NoError(t, err)
withArrow(true, func() { arrow, err = listing(recurse) })
require.NoError(t, err)
assert.Equal(t, xml, arrow, "recurse=%v: Arrow listing differs from XML", recurse)
}
// Recursive Arrow listing contains every object with the right size.
var arrow map[string]int64
withArrow(true, func() { arrow, _ = listing(true) })
for remote, size := range wantSizes {
assert.Equal(t, size, arrow[remote], remote)
}
// Single-directory Arrow listing reports "sub" as a directory.
var entries fs.DirEntries
withArrow(true, func() {
var err error
entries, err = f.List(ctx, dir)
require.NoError(t, err)
})
foundSubDir := false
for _, e := range entries {
if e.Remote() == dir+"/sub" {
_, isObj := e.(fs.Object)
foundSubDir = !isObj
}
}
assert.True(t, foundSubDir, "expected %q to be listed as a directory", dir+"/sub")
})
// Parallel listing must equal sequential listing, recursively and
// per-directory, including the "sub" subdirectory.
t.Run("ParallelEqualsSequential", func(t *testing.T) {
for _, recurse := range []bool{true, false} {
var seq, par map[string]int64
var err error
withArrow(true, func() {
withParallel(0, func() { seq, err = listing(recurse) })
require.NoError(t, err)
withParallel(8, func() { par, err = listing(recurse) })
require.NoError(t, err)
})
assert.Equal(t, seq, par, "recurse=%v: parallel listing differs from sequential", recurse)
}
})
// A tiny list_chunk forces each shard to span multiple Arrow pages, so this
// exercises pagination (startFrom+endBefore+marker resent together) inside a
// shard. The "sub/" boundary puts a.txt..e.txt in one shard => >1 page at
// chunk size 2.
t.Run("ParallelMultiPage", func(t *testing.T) {
oldChunk := f.opt.ListChunkSize
f.opt.ListChunkSize = 2
defer func() { f.opt.ListChunkSize = oldChunk }()
var seq, par map[string]int64
var err error
withArrow(true, func() {
withParallel(0, func() { seq, err = listing(true) })
require.NoError(t, err)
withParallel(8, func() { par, err = listing(true) })
require.NoError(t, err)
})
assert.Equal(t, seq, par)
for remote, size := range wantSizes {
assert.Equal(t, size, par[remote], remote)
}
})
}
func TestArrowLadderBoundaries(t *testing.T) {
for _, tc := range []struct {
directory string
target int
}{
{"", 8},
{"dir/", 16},
{"dir/", 100}, // capped at alphabet size
{"dir/", 1},
} {
points := arrowLadderBoundaries(tc.directory, tc.target)
require.NotEmpty(t, points)
assert.LessOrEqual(t, len(points), 62)
for i, p := range points {
assert.True(t, strings.HasPrefix(p, tc.directory), "boundary %q lacks prefix %q", p, tc.directory)
if i > 0 {
assert.Less(t, points[i-1], p, "boundaries not strictly ascending")
}
}
}
assert.Nil(t, arrowLadderBoundaries("dir/", 0))
// alphabet is single-case (digit+lowercase) so boundaries are monotonic
// under both byte order and case-insensitive order
pts := arrowLadderBoundaries("dir/", 36)
for _, p := range pts {
assert.Equal(t, p, strings.ToLower(p), "boundary must be lowercase: %q", p)
}
}
func TestIsEndBeforeUnsupported(t *testing.T) {
assert.True(t, isEndBeforeUnsupported(arrowlist.ErrEndBeforeXMLFallback))
assert.True(t, isEndBeforeUnsupported(fmt.Errorf("wrapped: %w", arrowlist.ErrEndBeforeXMLFallback)))
assert.True(t, isEndBeforeUnsupported(&azcore.ResponseError{ErrorCode: "OperationNotSupportedWithFeatureMissing"}))
assert.False(t, isEndBeforeUnsupported(&azcore.ResponseError{ErrorCode: "ContainerNotFound"}))
assert.False(t, isEndBeforeUnsupported(errors.New("some other error")))
assert.False(t, isEndBeforeUnsupported(nil))
}
+421
View File
@@ -0,0 +1,421 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/arrow/arrow.go in the Azure SDK for Go
// github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at commit
// c6fa341ca22b (branch feature/storage/bifrost), retyped onto the SDK's
// public type aliases so it compiles against the released azblob module.
package arrowlist
import (
"encoding/base64"
"fmt"
"io"
"net/http"
"strconv"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/memory"
)
const (
// ArrowContentType is the Content-Type for Apache Arrow IPC stream responses.
ArrowContentType = "application/vnd.apache.arrow.stream"
// ArrowAcceptHeader is the Accept header value to request Arrow format.
ArrowAcceptHeader = ArrowContentType
// resourceTypeBlobPrefix is the ResourceType value that identifies a virtual directory prefix
// in hierarchy listing responses. In XML, prefixes are separate <BlobPrefix> elements;
// in Arrow, all rows share the same schema and are distinguished by this field value.
resourceTypeBlobPrefix = "blobprefix"
)
// HandleFlatListResponse parses an Arrow IPC stream response into a ContainerClientListBlobFlatSegmentResponse.
func HandleFlatListResponse(resp *http.Response) (container.ListBlobsFlatResponse, error) {
result := container.ListBlobsFlatResponse{}
// Extract response headers
extractResponseHeaders(resp, &result.ClientRequestID, &result.ContentType, &result.Date, &result.RequestID, &result.Version)
// Parse Arrow IPC stream
items, nextMarker, err := parseArrowStream(resp.Body)
if err != nil {
return result, fmt.Errorf("failed to parse Arrow IPC response: %w", err)
}
result.ListBlobsFlatSegmentResponse = container.ListBlobsFlatSegmentResponse{
Segment: &container.BlobFlatListSegment{
BlobItems: items,
},
NextMarker: nextMarker,
}
return result, nil
}
// HandleHierarchyListResponse parses an Arrow IPC stream response into a ContainerClientListBlobHierarchySegmentResponse.
func HandleHierarchyListResponse(resp *http.Response) (container.ListBlobsHierarchyResponse, error) {
result := container.ListBlobsHierarchyResponse{}
extractResponseHeaders(resp, &result.ClientRequestID, &result.ContentType, &result.Date, &result.RequestID, &result.Version)
items, nextMarker, err := parseArrowStream(resp.Body)
if err != nil {
return result, fmt.Errorf("failed to parse Arrow IPC response: %w", err)
}
// Separate blob items from blob prefixes based on ResourceType field.
var blobItems []*container.BlobItem
var blobPrefixes []*container.BlobPrefix
for _, item := range items {
if item.Properties != nil && item.Properties.ResourceType != nil && *item.Properties.ResourceType == resourceTypeBlobPrefix {
blobPrefixes = append(blobPrefixes, &container.BlobPrefix{
Name: item.Name,
Properties: item.Properties,
})
} else {
blobItems = append(blobItems, item)
}
}
result.ListBlobsHierarchySegmentResponse = container.ListBlobsHierarchySegmentResponse{
Segment: &container.BlobHierarchyListSegment{
BlobItems: blobItems,
BlobPrefixes: blobPrefixes,
},
NextMarker: nextMarker,
}
return result, nil
}
func extractResponseHeaders(resp *http.Response, clientRequestID, contentType **string, date **time.Time, requestID, version **string) {
if val := resp.Header.Get("x-ms-client-request-id"); val != "" {
*clientRequestID = &val
}
if val := resp.Header.Get("Content-Type"); val != "" {
*contentType = &val
}
if val := resp.Header.Get("Date"); val != "" {
if t, err := time.Parse(time.RFC1123, val); err == nil {
*date = &t
}
}
if val := resp.Header.Get("x-ms-request-id"); val != "" {
*requestID = &val
}
if val := resp.Header.Get("x-ms-version"); val != "" {
*version = &val
}
}
// parseArrowStream reads an Arrow IPC stream and converts it to BlobItems.
func parseArrowStream(body io.Reader) ([]*container.BlobItem, *string, error) {
reader, err := ipc.NewReader(body, ipc.WithAllocator(memory.DefaultAllocator))
if err != nil {
return nil, nil, fmt.Errorf("failed to create Arrow IPC reader: %w", err)
}
defer reader.Release()
// Extract NextMarker from schema metadata
var nextMarker *string
var numRecords int
if md := reader.Schema().Metadata(); md.Len() > 0 {
if idx := md.FindKey("NextMarker"); idx >= 0 {
val := md.Values()[idx]
if val != "" {
nextMarker = &val
}
}
if idx := md.FindKey("NumberOfRecords"); idx >= 0 {
if n, err := strconv.Atoi(md.Values()[idx]); err == nil {
numRecords = n
}
}
}
// Build column index for the schema
schema := reader.Schema()
colIndex := buildColumnIndex(schema)
// Pre-allocate if we know the count
items := make([]*container.BlobItem, 0, numRecords)
// Iterate over record batches
for reader.Next() {
rec := reader.RecordBatch()
rows := int(rec.NumRows())
for row := 0; row < rows; row++ {
item := &container.BlobItem{
Properties: &container.BlobProperties{},
}
populateBlobItem(item, rec, row, colIndex)
items = append(items, item)
}
}
if err := reader.Err(); err != nil {
return nil, nil, fmt.Errorf("error reading Arrow record batches: %w", err)
}
return items, nextMarker, nil
}
// columnIndex maps column names to their indices in the record batch for O(1) lookup.
type columnIndex map[string]int
func buildColumnIndex(schema *arrow.Schema) columnIndex {
idx := make(columnIndex, len(schema.Fields()))
for i, f := range schema.Fields() {
idx[f.Name] = i
}
return idx
}
// populateBlobItem fills a BlobItem from a single row across all columns.
func populateBlobItem(item *container.BlobItem, rec arrow.RecordBatch, row int, colIdx columnIndex) {
// Top-level BlobItem fields
item.Name = getStringField(rec, row, colIdx, "Name")
item.Deleted = getBoolField(rec, row, colIdx, "Deleted")
item.Snapshot = getStringField(rec, row, colIdx, "Snapshot")
item.VersionID = getStringField(rec, row, colIdx, "VersionId")
item.IsCurrentVersion = getBoolField(rec, row, colIdx, "IsCurrentVersion")
item.HasVersionsOnly = getBoolField(rec, row, colIdx, "HasVersionsOnly")
// Metadata maps
item.Metadata = getMapField(rec, row, colIdx, "Metadata")
item.OrMetadata = getMapField(rec, row, colIdx, "OrMetadata")
// Tags
if tags := getMapField(rec, row, colIdx, "Tags"); tags != nil {
var blobTags []*container.BlobTag
for k, v := range tags {
key := k
blobTags = append(blobTags, &container.BlobTag{Key: &key, Value: v})
}
item.BlobTags = &container.BlobTags{BlobTagSet: blobTags}
}
// BlobProperties fields
p := item.Properties
p.CreationTime = getTimestampField(rec, row, colIdx, "Creation-Time")
p.LastModified = getTimestampField(rec, row, colIdx, "Last-Modified")
p.ETag = getETagField(rec, row, colIdx, "Etag")
p.ContentLength = getInt64Field(rec, row, colIdx, "Content-Length")
p.ContentType = getStringField(rec, row, colIdx, "Content-Type")
p.ContentEncoding = getStringField(rec, row, colIdx, "Content-Encoding")
p.ContentLanguage = getStringField(rec, row, colIdx, "Content-Language")
p.ContentDisposition = getStringField(rec, row, colIdx, "Content-Disposition")
p.CacheControl = getStringField(rec, row, colIdx, "Cache-Control")
p.ContentMD5 = getBase64BytesField(rec, row, colIdx, "Content-MD5")
p.BlobType = getEnumField[container.BlobType](rec, row, colIdx, "BlobType")
p.AccessTier = getEnumField[container.AccessTier](rec, row, colIdx, "AccessTier")
p.AccessTierInferred = getBoolField(rec, row, colIdx, "AccessTierInferred")
p.AccessTierChangeTime = getTimestampField(rec, row, colIdx, "AccessTierChangeTime")
p.LeaseState = getEnumField[lease.StateType](rec, row, colIdx, "LeaseState")
p.LeaseStatus = getEnumField[lease.StatusType](rec, row, colIdx, "LeaseStatus")
p.LeaseDuration = getEnumField[lease.DurationType](rec, row, colIdx, "LeaseDuration")
p.ServerEncrypted = getBoolField(rec, row, colIdx, "ServerEncrypted")
p.CustomerProvidedKeySHA256 = getStringField(rec, row, colIdx, "CustomerProvidedKeySha256")
p.EncryptionScope = getStringField(rec, row, colIdx, "EncryptionScope")
p.IncrementalCopy = getBoolField(rec, row, colIdx, "IncrementalCopy")
p.IsSealed = getBoolField(rec, row, colIdx, "Sealed")
p.ArchiveStatus = getEnumField[container.ArchiveStatus](rec, row, colIdx, "ArchiveStatus")
p.RehydratePriority = getEnumField[blob.RehydratePriority](rec, row, colIdx, "RehydratePriority")
p.CopyID = getStringField(rec, row, colIdx, "CopyId")
p.CopyStatus = getEnumField[blob.CopyStatusType](rec, row, colIdx, "CopyStatus")
p.CopySource = getStringField(rec, row, colIdx, "CopySource")
p.CopyProgress = getStringField(rec, row, colIdx, "CopyProgress")
p.CopyCompletionTime = getTimestampField(rec, row, colIdx, "CopyCompletionTime")
p.CopyStatusDescription = getStringField(rec, row, colIdx, "CopyStatusDescription")
p.DestinationSnapshot = getStringField(rec, row, colIdx, "CopyDestinationSnapshot")
p.ImmutabilityPolicyExpiresOn = getTimestampField(rec, row, colIdx, "ImmutabilityPolicyUntilDate")
p.ImmutabilityPolicyMode = getEnumField[container.ImmutabilityPolicyMode](rec, row, colIdx, "ImmutabilityPolicyMode")
p.LegalHold = getBoolField(rec, row, colIdx, "LegalHold")
p.DeletedTime = getTimestampField(rec, row, colIdx, "DeletedTime")
p.RemainingRetentionDays = getInt32Field(rec, row, colIdx, "RemainingRetentionDays")
p.LastAccessedOn = getTimestampField(rec, row, colIdx, "LastAccessTime")
p.TagCount = getInt32Field(rec, row, colIdx, "TagCount")
p.BlobSequenceNumber = getInt64Field(rec, row, colIdx, "x-ms-blob-sequence-number")
p.ResourceType = getStringField(rec, row, colIdx, "ResourceType")
// Content-CRC64 is a string field in Arrow but not directly mapped to BlobProperties.
// The field exists in the Arrow schema but has no corresponding field in the Go SDK BlobProperties.
// It is silently ignored for forward compatibility.
}
// Field accessor helpers that safely handle missing columns and null values.
func getStringField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *string {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.String); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getBoolField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *bool {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Boolean); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getTimestampField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *time.Time {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Timestamp); ok {
val := arr.Value(row)
// Arrow timestamps are stored as int64 with a unit; convert to time.Time
dt := val.ToTime(arr.DataType().(*arrow.TimestampType).Unit)
return &dt
}
return nil
}
func getInt64Field(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *int64 {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Uint64); ok {
val := int64(arr.Value(row))
return &val
}
if arr, ok := col.(*array.Int64); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getInt32Field(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *int32 {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Uint64); ok {
val := int32(arr.Value(row))
return &val
}
if arr, ok := col.(*array.Int32); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getETagField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *azcore.ETag {
s := getStringField(rec, row, colIdx, name)
if s == nil {
return nil
}
etag := azcore.ETag(*s)
return &etag
}
func getBase64BytesField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) []byte {
s := getStringField(rec, row, colIdx, name)
if s == nil {
return nil
}
decoded, err := base64.StdEncoding.DecodeString(*s)
if err != nil {
return nil
}
return decoded
}
// getEnumField returns a pointer to a string-typed enum value.
type stringEnum interface {
~string
}
func getEnumField[T stringEnum](rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *T {
s := getStringField(rec, row, colIdx, name)
if s == nil {
return nil
}
val := T(*s)
return &val
}
func getMapField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) map[string]*string {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
mapArr, ok := col.(*array.Map)
if !ok {
return nil
}
// Get the offsets for this row's map entries
start := int(mapArr.Offsets()[row])
end := int(mapArr.Offsets()[row+1])
if start == end {
return nil
}
keys := mapArr.Keys().(*array.String)
values := mapArr.Items().(*array.String)
result := make(map[string]*string, end-start)
for i := start; i < end; i++ {
k := keys.Value(i)
if values.IsNull(i) {
result[k] = nil
} else {
v := values.Value(i)
result[k] = &v
}
}
return result
}
+374
View File
@@ -0,0 +1,374 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/arrow/arrow_test.go in the Azure SDK for Go
// github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at commit
// c6fa341ca22b (branch feature/storage/bifrost).
package arrowlist
import (
"bytes"
"io"
"net/http"
"testing"
"github.com/apache/arrow-go/v18/arrow"
arrowArray "github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/stretchr/testify/require"
)
// coreSchema returns a basic Arrow schema for testing with common blob fields.
func coreSchema(md *arrow.Metadata) *arrow.Schema {
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Creation-Time", Type: &arrow.TimestampType{Unit: arrow.Microsecond}, Nullable: true},
{Name: "Last-Modified", Type: &arrow.TimestampType{Unit: arrow.Microsecond}, Nullable: true},
{Name: "BlobType", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "Etag", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "Content-Length", Type: arrow.PrimitiveTypes.Uint64, Nullable: true},
{Name: "Content-Type", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "ServerEncrypted", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
{Name: "AccessTier", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "LeaseState", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "LeaseStatus", Type: arrow.BinaryTypes.String, Nullable: true},
}
return arrow.NewSchema(fields, md)
}
// buildArrowStream constructs a synthetic Arrow IPC stream from a schema and a populate function.
func buildArrowStream(t *testing.T, schema *arrow.Schema, populate func(builder *arrowArray.RecordBuilder)) []byte {
t.Helper()
alloc := memory.DefaultAllocator
var buf bytes.Buffer
w := ipc.NewWriter(&buf, ipc.WithSchema(schema), ipc.WithAllocator(alloc))
builder := arrowArray.NewRecordBuilder(alloc, schema)
defer builder.Release()
populate(builder)
rec := builder.NewRecordBatch()
defer rec.Release()
require.NoError(t, w.Write(rec))
require.NoError(t, w.Close())
return buf.Bytes()
}
func makeHTTPResponse(data []byte, headers map[string]string) *http.Response {
h := http.Header{}
h.Set("Content-Type", ArrowContentType)
for k, v := range headers {
h.Set(k, v)
}
return &http.Response{
StatusCode: http.StatusOK,
Header: h,
Body: io.NopCloser(bytes.NewReader(data)),
}
}
func TestHandleFlatListResponse_BasicParsing(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{"marker123", "2"})
schema := coreSchema(&md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob1.txt")
b.Field(1).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000000000000))
b.Field(2).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000000000000))
b.Field(3).(*arrowArray.StringBuilder).Append("BlockBlob")
b.Field(4).(*arrowArray.StringBuilder).Append("0x1234")
b.Field(5).(*arrowArray.Uint64Builder).Append(1024)
b.Field(6).(*arrowArray.StringBuilder).Append("application/octet-stream")
b.Field(7).(*arrowArray.BooleanBuilder).Append(true)
b.Field(8).(*arrowArray.StringBuilder).Append("Hot")
b.Field(9).(*arrowArray.StringBuilder).Append("available")
b.Field(10).(*arrowArray.StringBuilder).Append("unlocked")
b.Field(0).(*arrowArray.StringBuilder).Append("blob2.txt")
b.Field(1).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000001000000))
b.Field(2).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000001000000))
b.Field(3).(*arrowArray.StringBuilder).Append("BlockBlob")
b.Field(4).(*arrowArray.StringBuilder).Append("0x5678")
b.Field(5).(*arrowArray.Uint64Builder).Append(2048)
b.Field(6).(*arrowArray.StringBuilder).Append("text/plain")
b.Field(7).(*arrowArray.BooleanBuilder).Append(false)
b.Field(8).(*arrowArray.StringBuilder).Append("Cool")
b.Field(9).(*arrowArray.StringBuilder).Append("available")
b.Field(10).(*arrowArray.StringBuilder).Append("unlocked")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.NotNil(t, result.NextMarker)
require.Equal(t, "marker123", *result.NextMarker)
require.Len(t, result.Segment.BlobItems, 2)
item := result.Segment.BlobItems[0]
require.Equal(t, "blob1.txt", *item.Name)
require.Equal(t, "BlockBlob", string(*item.Properties.BlobType))
require.Equal(t, int64(1024), *item.Properties.ContentLength)
require.Equal(t, "Hot", string(*item.Properties.AccessTier))
}
func TestHandleFlatListResponse_EmptyRecordBatch(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{"", "0"})
schema := coreSchema(&md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
// No rows added
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Nil(t, result.NextMarker)
require.Empty(t, result.Segment.BlobItems)
}
func TestHandleFlatListResponse_NullFields(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
schema := coreSchema(&md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
b.Field(1).(*arrowArray.TimestampBuilder).AppendNull()
b.Field(2).(*arrowArray.TimestampBuilder).AppendNull()
b.Field(3).(*arrowArray.StringBuilder).AppendNull()
b.Field(4).(*arrowArray.StringBuilder).AppendNull()
b.Field(5).(*arrowArray.Uint64Builder).AppendNull()
b.Field(6).(*arrowArray.StringBuilder).AppendNull()
b.Field(7).(*arrowArray.BooleanBuilder).AppendNull()
b.Field(8).(*arrowArray.StringBuilder).AppendNull()
b.Field(9).(*arrowArray.StringBuilder).AppendNull()
b.Field(10).(*arrowArray.StringBuilder).AppendNull()
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
item := result.Segment.BlobItems[0]
require.Equal(t, "blob.txt", *item.Name)
require.Nil(t, item.Properties.CreationTime)
require.Nil(t, item.Properties.BlobType)
require.Nil(t, item.Properties.ETag)
require.Nil(t, item.Properties.ContentLength)
require.Nil(t, item.Properties.ContentType)
require.Nil(t, item.Properties.ServerEncrypted)
}
func TestHandleFlatListResponse_UnknownColumnsIgnored(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "FutureField", Type: arrow.BinaryTypes.String, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
b.Field(1).(*arrowArray.StringBuilder).Append("some-future-value")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
require.Equal(t, "blob.txt", *result.Segment.BlobItems[0].Name)
}
func TestHandleFlatListResponse_MultiRecordBatch(t *testing.T) {
alloc := memory.DefaultAllocator
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{"", "3"})
schema := coreSchema(&md)
var buf bytes.Buffer
w := ipc.NewWriter(&buf, ipc.WithSchema(schema), ipc.WithAllocator(alloc))
// First batch: 2 rows
builder := arrowArray.NewRecordBuilder(alloc, schema)
for i := 0; i < 2; i++ {
builder.Field(0).(*arrowArray.StringBuilder).Append("batch1_blob")
for j := 1; j < 11; j++ {
builder.Field(j).AppendNull()
}
}
rec := builder.NewRecordBatch()
require.NoError(t, w.Write(rec))
rec.Release()
builder.Release()
// Second batch: 1 row
builder = arrowArray.NewRecordBuilder(alloc, schema)
builder.Field(0).(*arrowArray.StringBuilder).Append("batch2_blob")
for j := 1; j < 11; j++ {
builder.Field(j).AppendNull()
}
rec = builder.NewRecordBatch()
require.NoError(t, w.Write(rec))
rec.Release()
builder.Release()
require.NoError(t, w.Close())
resp := makeHTTPResponse(buf.Bytes(), nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 3)
}
func TestHandleHierarchyListResponse_SeparatesBlobsAndPrefixes(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "ResourceType", Type: arrow.BinaryTypes.String, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
// A regular blob
b.Field(0).(*arrowArray.StringBuilder).Append("folder/file.txt")
b.Field(1).(*arrowArray.StringBuilder).AppendNull()
// A blob prefix (virtual directory)
b.Field(0).(*arrowArray.StringBuilder).Append("folder/subfolder/")
b.Field(1).(*arrowArray.StringBuilder).Append("blobprefix")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleHierarchyListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
require.Equal(t, "folder/file.txt", *result.Segment.BlobItems[0].Name)
require.Len(t, result.Segment.BlobPrefixes, 1)
require.Equal(t, "folder/subfolder/", *result.Segment.BlobPrefixes[0].Name)
}
func TestHandleFlatListResponse_ResponseHeaders(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
})
resp := makeHTTPResponse(data, map[string]string{
"x-ms-request-id": "req-123",
"x-ms-version": "2026-10-06",
"x-ms-client-request-id": "client-req-456",
"Date": "Mon, 01 Jan 2024 00:00:00 GMT",
})
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Equal(t, "req-123", *result.RequestID)
require.Equal(t, "2026-10-06", *result.Version)
require.Equal(t, "client-req-456", *result.ClientRequestID)
require.NotNil(t, result.Date)
}
func TestHandleFlatListResponse_MapFields(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Metadata", Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Nullable: true},
{Name: "Tags", Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
metaBuilder := b.Field(1).(*arrowArray.MapBuilder)
metaBuilder.Append(true)
metaBuilder.KeyBuilder().(*arrowArray.StringBuilder).Append("key1")
metaBuilder.ItemBuilder().(*arrowArray.StringBuilder).Append("value1")
tagBuilder := b.Field(2).(*arrowArray.MapBuilder)
tagBuilder.Append(true)
tagBuilder.KeyBuilder().(*arrowArray.StringBuilder).Append("env")
tagBuilder.ItemBuilder().(*arrowArray.StringBuilder).Append("prod")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
item := result.Segment.BlobItems[0]
require.NotNil(t, item.Metadata)
require.Equal(t, "value1", *item.Metadata["key1"])
require.NotNil(t, item.BlobTags)
require.Len(t, item.BlobTags.BlobTagSet, 1)
}
func TestHandleFlatListResponse_BooleanFields(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Deleted", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
{Name: "IsCurrentVersion", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
{Name: "ServerEncrypted", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
b.Field(1).(*arrowArray.BooleanBuilder).Append(true)
b.Field(2).(*arrowArray.BooleanBuilder).Append(false)
b.Field(3).(*arrowArray.BooleanBuilder).Append(true)
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
item := result.Segment.BlobItems[0]
require.True(t, *item.Deleted)
require.False(t, *item.IsCurrentVersion)
require.True(t, *item.Properties.ServerEncrypted)
}
func TestHandleFlatListResponse_VersionAndSnapshot(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "VersionId", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "Snapshot", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "IsCurrentVersion", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("versioned-blob.txt")
b.Field(1).(*arrowArray.StringBuilder).Append("2024-01-01T00:00:00.0000000Z")
b.Field(2).(*arrowArray.StringBuilder).Append("snap-123")
b.Field(3).(*arrowArray.BooleanBuilder).Append(true)
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
item := result.Segment.BlobItems[0]
require.Equal(t, "2024-01-01T00:00:00.0000000Z", *item.VersionID)
require.Equal(t, "snap-123", *item.Snapshot)
require.True(t, *item.IsCurrentVersion)
}
+259
View File
@@ -0,0 +1,259 @@
//go:build !plan9 && !solaris && !js
// Package arrowlist implements the experimental "Blob Listing with Apache
// Arrow" feature on top of the released Azure azblob SDK.
//
// The Azure SDK for Go has support for Arrow listing on the unreleased
// feature/storage/bifrost branch (commit c6fa341ca22b) where the listing
// pagers decode Arrow IPC stream responses into the ordinary ListBlobs
// response types. Until that ships in a tagged azblob release this package
// fills the gap: it exposes the same options and pager interface as the
// experimental SDK, built only on the public API of the released SDK plus
// copies of two small internal auth policies (see sharedkey.go and
// challenge_policy.go).
//
// This package is temporary. When Arrow listing ships in a released azblob
// the whole package should be deleted and its callers pointed back at
// container.Client.NewListBlobsHierarchyPager - the option and response
// types here are deliberately source compatible with the SDK's to make that
// a mechanical change.
package arrowlist
import (
"context"
"errors"
"net/http"
"strconv"
"strings"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
)
const (
// moduleName and moduleVersion identify this package in the azcore
// telemetry policy's User-Agent fragment.
moduleName = "github.com/rclone/rclone/backend/azureblob/arrowlist"
moduleVersion = "v0.0.1"
// serviceVersion is the x-ms-version sent with listing requests. Arrow
// responses and the startFrom/endBefore parameters need at least
// 2026-06-06.
serviceVersion = "2026-06-06"
// tokenScope is the OAuth scope for Azure Storage, as used by the SDK's
// own clients.
tokenScope = "https://storage.azure.com/.default"
)
// ErrEndBeforeXMLFallback is returned by the pager when the server answered
// with XML instead of Arrow while EndBefore was set. The service only honours
// endBefore on the Arrow listing path, so an XML page could silently contain
// blobs outside the requested range and cannot be used.
var ErrEndBeforeXMLFallback = errors.New("arrow listing fell back to XML with endBefore set")
// ListBlobsHierarchyOptions provides the configuration for
// Client.NewListBlobsHierarchyPager.
//
// It mirrors the experimental SDK's container.ListBlobsHierarchyOptions:
// the released SDK's options are embedded (providing Include, Marker,
// MaxResults, Prefix and StartFrom by promotion) with the two fields the
// released SDK lacks added alongside.
type ListBlobsHierarchyOptions struct {
container.ListBlobsHierarchyOptions
// EndBefore limits listing to blobs whose full container path is
// lexically before it (exclusive). Only honoured on Arrow responses.
EndBefore *string
// UseArrowFormat requests the response as an Apache Arrow IPC stream.
UseArrowFormat *bool
}
// ClientOptions contains the optional parameters for client creation.
type ClientOptions struct {
azcore.ClientOptions
}
// Client issues Arrow listing requests to a single container.
type Client struct {
endpoint string
azClient *azcore.Client
}
// newClient creates a Client for the container at containerURL sending
// requests through a pipeline with the given auth policy (which may be nil
// for anonymous or SAS access).
func newClient(containerURL string, authPolicy policy.Policy, options *ClientOptions) (*Client, error) {
if options == nil {
options = &ClientOptions{}
}
plOpts := runtime.PipelineOptions{}
if authPolicy != nil {
plOpts.PerRetry = []policy.Policy{authPolicy}
}
azClient, err := azcore.NewClient(moduleName, moduleVersion, plOpts, &options.ClientOptions)
if err != nil {
return nil, err
}
return &Client{
endpoint: containerURL,
azClient: azClient,
}, nil
}
// NewClient creates a Client authenticating with a token credential.
func NewClient(containerURL string, cred azcore.TokenCredential, options *ClientOptions) (*Client, error) {
return newClient(containerURL, NewStorageChallengePolicy(cred, tokenScope, false), options)
}
// NewClientWithNoCredential creates a Client with no authentication, for
// anonymous access or a containerURL carrying a SAS token in its query.
func NewClientWithNoCredential(containerURL string, options *ClientOptions) (*Client, error) {
return newClient(containerURL, nil, options)
}
// NewClientWithSharedKeyCredential creates a Client signing requests with the
// account's shared key.
func NewClientWithSharedKeyCredential(containerURL string, cred *SharedKeyCredential, options *ClientOptions) (*Client, error) {
return newClient(containerURL, NewSharedKeyCredPolicy(cred), options)
}
// NewListBlobsHierarchyPager returns a pager over the container's blobs,
// requesting Arrow responses when o.UseArrowFormat is set and returning the
// same response type as container.Client.NewListBlobsHierarchyPager.
//
// If the service does not support Arrow listing (the feature not enabled on
// the account, or an HNS account) it answers with XML, which is decoded
// transparently - unless EndBefore is set, in which case the fetcher returns
// ErrEndBeforeXMLFallback (see there).
func (c *Client) NewListBlobsHierarchyPager(delimiter string, o *ListBlobsHierarchyOptions) *runtime.Pager[container.ListBlobsHierarchyResponse] {
// Copy the options so advancing the Marker doesn't mutate the caller's struct.
opts := ListBlobsHierarchyOptions{}
if o != nil {
opts = *o
}
useArrow := opts.UseArrowFormat != nil && *opts.UseArrowFormat
return runtime.NewPager(runtime.PagingHandler[container.ListBlobsHierarchyResponse]{
More: func(page container.ListBlobsHierarchyResponse) bool {
return page.NextMarker != nil && len(*page.NextMarker) > 0
},
Fetcher: func(ctx context.Context, page *container.ListBlobsHierarchyResponse) (container.ListBlobsHierarchyResponse, error) {
if page != nil {
opts.Marker = page.NextMarker
}
req, err := c.listCreateRequest(ctx, delimiter, &opts, useArrow)
if err != nil {
return container.ListBlobsHierarchyResponse{}, err
}
resp, err := c.azClient.Pipeline().Do(req)
if err != nil {
return container.ListBlobsHierarchyResponse{}, err
}
if !runtime.HasStatusCode(resp, http.StatusOK) {
return container.ListBlobsHierarchyResponse{}, runtime.NewResponseError(resp)
}
if useArrow && resp.Header.Get("Content-Type") == ArrowContentType {
return HandleHierarchyListResponse(resp)
}
if opts.EndBefore != nil {
return container.ListBlobsHierarchyResponse{}, ErrEndBeforeXMLFallback
}
return handleXMLResponse(resp)
},
})
}
// listCreateRequest builds the ListBlobs request, mirroring the generated
// request builders in the SDK.
func (c *Client) listCreateRequest(ctx context.Context, delimiter string, opts *ListBlobsHierarchyOptions, useArrow bool) (*policy.Request, error) {
req, err := runtime.NewRequest(ctx, http.MethodGet, c.endpoint)
if err != nil {
return nil, err
}
// Reading the existing query first preserves any SAS token in the endpoint.
reqQP := req.Raw().URL.Query()
reqQP.Set("comp", "list")
reqQP.Set("restype", "container")
reqQP.Set("delimiter", delimiter)
if include := formatInclude(opts.Include); include != "" {
reqQP.Set("include", include)
}
if opts.Marker != nil {
reqQP.Set("marker", *opts.Marker)
}
if opts.MaxResults != nil {
reqQP.Set("maxresults", strconv.FormatInt(int64(*opts.MaxResults), 10))
}
if opts.Prefix != nil {
reqQP.Set("prefix", *opts.Prefix)
}
if opts.StartFrom != nil {
reqQP.Set("startFrom", *opts.StartFrom)
}
if useArrow && opts.EndBefore != nil {
reqQP.Set("endBefore", *opts.EndBefore)
}
req.Raw().URL.RawQuery = strings.ReplaceAll(reqQP.Encode(), "+", "%20")
if useArrow {
// Stream the Arrow body rather than buffering it in the pipeline.
runtime.SkipBodyDownload(req)
req.Raw().Header["Accept"] = []string{ArrowAcceptHeader}
} else {
req.Raw().Header["Accept"] = []string{"application/xml"}
}
req.Raw().Header["x-ms-version"] = []string{serviceVersion}
return req, nil
}
// handleXMLResponse decodes an XML ListBlobs response, mirroring the SDK's
// generated ListBlobHierarchySegmentHandleResponse.
func handleXMLResponse(resp *http.Response) (container.ListBlobsHierarchyResponse, error) {
result := container.ListBlobsHierarchyResponse{}
extractResponseHeaders(resp, &result.ClientRequestID, &result.ContentType, &result.Date, &result.RequestID, &result.Version)
if err := runtime.UnmarshalAsXML(resp, &result.ListBlobsHierarchySegmentResponse); err != nil {
return container.ListBlobsHierarchyResponse{}, err
}
return result, nil
}
// formatInclude renders the include datasets as the comma separated query
// parameter value, in the same order as the SDK's unexported
// ListBlobsInclude.format.
func formatInclude(l container.ListBlobsInclude) string {
var include []string
if l.Copy {
include = append(include, "copy")
}
if l.Deleted {
include = append(include, "deleted")
}
if l.DeletedWithVersions {
include = append(include, "deletedwithversions")
}
if l.ImmutabilityPolicy {
include = append(include, "immutabilitypolicy")
}
if l.LegalHold {
include = append(include, "legalhold")
}
if l.Metadata {
include = append(include, "metadata")
}
if l.Snapshots {
include = append(include, "snapshots")
}
if l.Tags {
include = append(include, "tags")
}
if l.UncommittedBlobs {
include = append(include, "uncommittedblobs")
}
if l.Versions {
include = append(include, "versions")
}
if l.Permissions {
include = append(include, "permissions")
}
return strings.Join(include, ",")
}
@@ -0,0 +1,199 @@
//go:build !plan9 && !solaris && !js
package arrowlist
import (
"context"
"encoding/base64"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/apache/arrow-go/v18/arrow"
arrowArray "github.com/apache/arrow-go/v18/arrow/array"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// arrowPage builds a one row Arrow IPC listing page with the given blob name
// and NextMarker.
func arrowPage(t *testing.T, name, nextMarker string) []byte {
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{nextMarker, "1"})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Content-Length", Type: arrow.PrimitiveTypes.Uint64, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
return buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append(name)
b.Field(1).(*arrowArray.Uint64Builder).Append(42)
})
}
func TestPagerArrow(t *testing.T) {
var requests []*http.Request
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests = append(requests, r.Clone(context.Background()))
w.Header().Set("Content-Type", ArrowContentType)
if r.URL.Query().Get("marker") == "" {
_, _ = w.Write(arrowPage(t, "blob1.txt", "marker1"))
} else {
_, _ = w.Write(arrowPage(t, "blob2.txt", ""))
}
}))
defer srv.Close()
// SAS style query on the endpoint must be preserved in requests.
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer?sv=fakesas", nil)
require.NoError(t, err)
opts := &ListBlobsHierarchyOptions{
ListBlobsHierarchyOptions: container.ListBlobsHierarchyOptions{
Include: container.ListBlobsInclude{Metadata: true, Tags: true},
Prefix: to.Ptr("dir/"),
MaxResults: to.Ptr(int32(1000)),
StartFrom: to.Ptr("testcontainer/dir/a"),
},
EndBefore: to.Ptr("testcontainer/dir/n"),
UseArrowFormat: to.Ptr(true),
}
pager := client.NewListBlobsHierarchyPager("/", opts)
var names []string
for pager.More() {
page, err := pager.NextPage(context.Background())
require.NoError(t, err)
assert.Equal(t, ArrowContentType, *page.ContentType)
for _, item := range page.Segment.BlobItems {
names = append(names, *item.Name)
assert.Equal(t, int64(42), *item.Properties.ContentLength)
}
}
assert.Equal(t, []string{"blob1.txt", "blob2.txt"}, names)
require.Len(t, requests, 2)
q := requests[0].URL.Query()
assert.Equal(t, "list", q.Get("comp"))
assert.Equal(t, "container", q.Get("restype"))
assert.Equal(t, "/", q.Get("delimiter"))
assert.Equal(t, "metadata,tags", q.Get("include"))
assert.Equal(t, "dir/", q.Get("prefix"))
assert.Equal(t, "1000", q.Get("maxresults"))
assert.Equal(t, "testcontainer/dir/a", q.Get("startFrom"))
assert.Equal(t, "testcontainer/dir/n", q.Get("endBefore"))
assert.Equal(t, "fakesas", q.Get("sv"))
assert.Empty(t, q.Get("marker"))
assert.Equal(t, ArrowAcceptHeader, requests[0].Header.Get("Accept"))
assert.Equal(t, serviceVersion, requests[0].Header.Get("x-ms-version"))
assert.Equal(t, "marker1", requests[1].URL.Query().Get("marker"))
// The caller's options must not have been mutated by paging.
assert.Nil(t, opts.Marker)
}
const xmlListResponse = `<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://example.com/" ContainerName="testcontainer">
<Blobs>
<Blob>
<Name>file.txt</Name>
<Properties>
<Last-Modified>Mon, 01 Jan 2024 00:00:00 GMT</Last-Modified>
<Etag>0x8D1</Etag>
<Content-Length>7</Content-Length>
<BlobType>BlockBlob</BlobType>
</Properties>
</Blob>
<BlobPrefix>
<Name>subdir/</Name>
</BlobPrefix>
</Blobs>
<NextMarker />
</EnumerationResults>`
func TestPagerXMLFallback(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml")
_, _ = w.Write([]byte(xmlListResponse))
}))
defer srv.Close()
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer", nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
UseArrowFormat: to.Ptr(true),
})
page, err := pager.NextPage(context.Background())
require.NoError(t, err)
assert.False(t, pager.More())
assert.Equal(t, "application/xml", *page.ContentType)
require.Len(t, page.Segment.BlobItems, 1)
assert.Equal(t, "file.txt", *page.Segment.BlobItems[0].Name)
assert.Equal(t, int64(7), *page.Segment.BlobItems[0].Properties.ContentLength)
require.Len(t, page.Segment.BlobPrefixes, 1)
assert.Equal(t, "subdir/", *page.Segment.BlobPrefixes[0].Name)
}
func TestPagerXMLFallbackWithEndBefore(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml")
_, _ = w.Write([]byte(xmlListResponse))
}))
defer srv.Close()
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer", nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
EndBefore: to.Ptr("testcontainer/n"),
UseArrowFormat: to.Ptr(true),
})
_, err = pager.NextPage(context.Background())
require.ErrorIs(t, err, ErrEndBeforeXMLFallback)
}
func TestPagerSharedKeyAuth(t *testing.T) {
var gotAuth, gotDate string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
gotDate = r.Header.Get("x-ms-date")
w.Header().Set("Content-Type", ArrowContentType)
_, _ = w.Write(arrowPage(t, "blob.txt", ""))
}))
defer srv.Close()
cred, err := NewSharedKeyCredential("testaccount", base64.StdEncoding.EncodeToString([]byte("testkey")))
require.NoError(t, err)
client, err := NewClientWithSharedKeyCredential(srv.URL+"/testcontainer", cred, nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
UseArrowFormat: to.Ptr(true),
})
_, err = pager.NextPage(context.Background())
require.NoError(t, err)
assert.True(t, strings.HasPrefix(gotAuth, "SharedKey testaccount:"), "Authorization = %q", gotAuth)
assert.NotEmpty(t, gotDate)
}
func TestPagerResponseError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("x-ms-error-code", string(bloberror.ContainerNotFound))
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer", nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
UseArrowFormat: to.Ptr(true),
})
_, err = pager.NextPage(context.Background())
require.Error(t, err)
assert.True(t, bloberror.HasCode(err, bloberror.ContainerNotFound), "expected ContainerNotFound, got %v", err)
}
@@ -0,0 +1,121 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/shared/challenge_policy.go in the Azure SDK for Go
// github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at v1.8.0. The
// released SDK does not export its storage bearer-challenge policy, which is
// needed for token authentication with the same semantics as the SDK's own
// clients.
package arrowlist
import (
"errors"
"net/http"
"strings"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
)
type storageAuthorizer struct {
scopes []string
tenantID string
}
// NewStorageChallengePolicy returns a bearer token policy for cred which
// handles the storage service's authentication challenges.
func NewStorageChallengePolicy(cred azcore.TokenCredential, audience string, allowHTTP bool) policy.Policy {
s := storageAuthorizer{scopes: []string{audience}}
return runtime.NewBearerTokenPolicy(cred, []string{audience}, &policy.BearerTokenOptions{
AuthorizationHandler: policy.AuthorizationHandler{
OnRequest: s.onRequest,
OnChallenge: s.onChallenge,
},
InsecureAllowCredentialWithHTTP: allowHTTP,
})
}
func (s *storageAuthorizer) onRequest(req *policy.Request, authNZ func(policy.TokenRequestOptions) error) error {
return authNZ(policy.TokenRequestOptions{Scopes: s.scopes})
}
func (s *storageAuthorizer) onChallenge(req *policy.Request, resp *http.Response, authNZ func(policy.TokenRequestOptions) error) error {
// parse the challenge
err := s.parseChallenge(resp)
if err != nil {
return err
}
// TODO: Set tenantID when policy.TokenRequestOptions supports it. https://github.com/Azure/azure-sdk-for-go/issues/19841
return authNZ(policy.TokenRequestOptions{Scopes: s.scopes})
}
type challengePolicyError struct {
err error
}
func (c *challengePolicyError) Error() string {
return c.err.Error()
}
func (*challengePolicyError) NonRetriable() {
// marker method
}
func (c *challengePolicyError) Unwrap() error {
return c.err
}
// parses Tenant ID from auth challenge
// https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/authorize
func parseTenant(url string) string {
if url == "" {
return ""
}
parts := strings.Split(url, "/")
if len(parts) >= 3 {
tenant := parts[3]
tenant = strings.ReplaceAll(tenant, ",", "")
return tenant
}
return ""
}
func (s *storageAuthorizer) parseChallenge(resp *http.Response) error {
authHeader := resp.Header.Get("WWW-Authenticate")
if authHeader == "" {
return &challengePolicyError{err: errors.New("response has no WWW-Authenticate header for challenge authentication")}
}
// Strip down to auth and resource
// Format is "Bearer authorization_uri=\"<site>\" resource_id=\"<site>\""
authHeader = strings.ReplaceAll(authHeader, "Bearer ", "")
parts := strings.Split(authHeader, " ")
vals := map[string]string{}
for _, part := range parts {
subParts := strings.Split(part, "=")
if len(subParts) == 2 {
stripped := strings.ReplaceAll(subParts[1], "\"", "")
stripped = strings.TrimSuffix(stripped, ",")
vals[subParts[0]] = stripped
}
}
s.tenantID = parseTenant(vals["authorization_uri"])
scope := vals["resource_id"]
if scope == "" {
return &challengePolicyError{err: errors.New("could not find a valid resource in the WWW-Authenticate header")}
}
if !strings.HasSuffix(scope, "/.default") {
scope += "/.default"
}
s.scopes = []string{scope}
return nil
}
+318
View File
@@ -0,0 +1,318 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/exported/shared_key_credential.go in the Azure SDK
// for Go github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at v1.8.0,
// with the internal header constants inlined. The released SDK does not
// export its shared key signing policy, so a copy is needed to sign the
// Arrow listing requests this package sends through its own pipeline.
package arrowlist
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"fmt"
"net/http"
"net/url"
"sort"
"strings"
"sync/atomic"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/rclone/rclone/fs"
)
// NewSharedKeyCredential creates an immutable SharedKeyCredential containing the
// storage account's name and either its primary or secondary key.
func NewSharedKeyCredential(accountName string, accountKey string) (*SharedKeyCredential, error) {
c := SharedKeyCredential{accountName: accountName}
if err := c.SetAccountKey(accountKey); err != nil {
return nil, err
}
return &c, nil
}
// SharedKeyCredential contains an account's name and its primary or secondary key.
type SharedKeyCredential struct {
// Only the NewSharedKeyCredential method should set these; all other methods should treat them as read-only
accountName string
accountKey atomic.Value // []byte
}
// AccountName returns the Storage account's name.
func (c *SharedKeyCredential) AccountName() string {
return c.accountName
}
// SetAccountKey replaces the existing account key with the specified account key.
func (c *SharedKeyCredential) SetAccountKey(accountKey string) error {
_bytes, err := base64.StdEncoding.DecodeString(accountKey)
if err != nil {
return fmt.Errorf("decode account key: %w", err)
}
c.accountKey.Store(_bytes)
return nil
}
// ComputeHMACSHA256 generates a hash signature for an HTTP request or for a SAS.
func (c *SharedKeyCredential) computeHMACSHA256(message string) (string, error) {
h := hmac.New(sha256.New, c.accountKey.Load().([]byte))
_, err := h.Write([]byte(message))
return base64.StdEncoding.EncodeToString(h.Sum(nil)), err
}
func (c *SharedKeyCredential) buildStringToSign(req *http.Request) (string, error) {
// https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
headers := req.Header
contentLength := getHeader("Content-Length", headers)
if contentLength == "0" {
contentLength = ""
}
canonicalizedResource, err := c.buildCanonicalizedResource(req.URL)
if err != nil {
return "", err
}
stringToSign := strings.Join([]string{
req.Method,
getHeader("Content-Encoding", headers),
getHeader("Content-Language", headers),
contentLength,
getHeader("Content-MD5", headers),
getHeader("Content-Type", headers),
"", // Empty date because x-ms-date is expected (as per web page above)
getHeader("If-Modified-Since", headers),
getHeader("If-Match", headers),
getHeader("If-None-Match", headers),
getHeader("If-Unmodified-Since", headers),
getHeader("Range", headers),
c.buildCanonicalizedHeader(headers),
canonicalizedResource,
}, "\n")
return stringToSign, nil
}
func getHeader(key string, headers map[string][]string) string {
if headers == nil {
return ""
}
if v, ok := headers[key]; ok {
if len(v) > 0 {
return v[0]
}
}
return ""
}
func getWeightTables() [][]int {
tableLv0 := [...]int{
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x71c, 0x0, 0x71f, 0x721, 0x723, 0x725,
0x0, 0x0, 0x0, 0x72d, 0x803, 0x0, 0x0, 0x733, 0x0, 0xd03, 0xd1a, 0xd1c, 0xd1e,
0xd20, 0xd22, 0xd24, 0xd26, 0xd28, 0xd2a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xe02, 0xe09, 0xe0a, 0xe1a, 0xe21, 0xe23, 0xe25, 0xe2c, 0xe32, 0xe35, 0xe36, 0xe48, 0xe51,
0xe70, 0xe7c, 0xe7e, 0xe89, 0xe8a, 0xe91, 0xe99, 0xe9f, 0xea2, 0xea4, 0xea6, 0xea7, 0xea9,
0x0, 0x0, 0x0, 0x743, 0x744, 0x748, 0xe02, 0xe09, 0xe0a, 0xe1a, 0xe21, 0xe23, 0xe25,
0xe2c, 0xe32, 0xe35, 0xe36, 0xe48, 0xe51, 0xe70, 0xe7c, 0xe7e, 0xe89, 0xe8a, 0xe91, 0xe99,
0xe9f, 0xea2, 0xea4, 0xea6, 0xea7, 0xea9, 0x0, 0x74c, 0x0, 0x750, 0x0,
}
tableLv2 := [...]int{
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8012, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8212, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
}
tables := [][]int{tableLv0[:], tableLv2[:]}
return tables
}
// NewHeaderStringComparer performs a multi-level, weight-based comparison of two strings
func compareHeaders(lhs, rhs string, tables [][]int) int {
currLevel, i, j := 0, 0, 0
n := len(tables)
lhsLen := len(lhs)
rhsLen := len(rhs)
for currLevel < n {
if currLevel == (n-1) && i != j {
if i > j {
return -1
}
if i < j {
return 1
}
return 0
}
var w1, w2 int
// Check bounds before accessing lhs[i]
if i < lhsLen {
w1 = tables[currLevel][lhs[i]]
} else {
w1 = 0x1
}
// Check bounds before accessing rhs[j]
if j < rhsLen {
w2 = tables[currLevel][rhs[j]]
} else {
w2 = 0x1
}
if w1 == 0x1 && w2 == 0x1 {
i = 0
j = 0
currLevel++
} else if w1 == w2 {
i++
j++
} else if w1 == 0 {
i++
} else if w2 == 0 {
j++
} else {
if w1 < w2 {
return -1
}
if w1 > w2 {
return 1
}
return 0
}
}
return 0
}
func (c *SharedKeyCredential) buildCanonicalizedHeader(headers http.Header) string {
cm := map[string][]string{}
for k, v := range headers {
headerName := strings.TrimSpace(strings.ToLower(k))
if strings.HasPrefix(headerName, "x-ms-") {
cm[headerName] = v // NOTE: the value must not have any whitespace around it.
}
}
if len(cm) == 0 {
return ""
}
keys := make([]string, 0, len(cm))
for key := range cm {
keys = append(keys, key)
}
tables := getWeightTables()
// Sort the keys using the custom comparator
sort.Slice(keys, func(i, j int) bool {
return compareHeaders(keys[i], keys[j], tables) < 0
})
ch := bytes.NewBufferString("")
for i, key := range keys {
if i > 0 {
ch.WriteRune('\n')
}
ch.WriteString(key)
ch.WriteRune(':')
ch.WriteString(strings.Join(cm[key], ","))
}
return ch.String()
}
func (c *SharedKeyCredential) buildCanonicalizedResource(u *url.URL) (string, error) {
// https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
cr := bytes.NewBufferString("/")
cr.WriteString(c.accountName)
if len(u.Path) > 0 {
// Any portion of the CanonicalizedResource string that is derived from
// the resource's URI should be encoded exactly as it is in the URI.
// -- https://msdn.microsoft.com/en-gb/library/azure/dd179428.aspx
cr.WriteString(u.EscapedPath())
} else {
// a slash is required to indicate the root path
cr.WriteString("/")
}
// params is a map[string][]string; param name is key; params values is []string
params, err := url.ParseQuery(u.RawQuery) // Returns URL decoded values
if err != nil {
return "", fmt.Errorf("failed to parse query params: %w", err)
}
if len(params) > 0 { // There is at least 1 query parameter
var paramNames []string // We use this to sort the parameter key names
for paramName := range params {
paramNames = append(paramNames, paramName) // paramNames must be lowercase
}
sort.Strings(paramNames)
for _, paramName := range paramNames {
paramValues := params[paramName]
sort.Strings(paramValues)
// Join the sorted key values separated by ','
// Then prepend "keyName:"; then add this string to the buffer
cr.WriteString("\n" + strings.ToLower(paramName) + ":" + strings.Join(paramValues, ","))
}
}
return cr.String(), nil
}
// ComputeHMACSHA256 is a helper for computing the signed string outside of this package.
func ComputeHMACSHA256(cred *SharedKeyCredential, message string) (string, error) {
return cred.computeHMACSHA256(message)
}
// SharedKeyCredPolicy is a pipeline policy that signs each request with the
// account's shared key.
type SharedKeyCredPolicy struct {
cred *SharedKeyCredential
}
// NewSharedKeyCredPolicy creates a SharedKeyCredPolicy signing with cred.
func NewSharedKeyCredPolicy(cred *SharedKeyCredential) *SharedKeyCredPolicy {
return &SharedKeyCredPolicy{cred: cred}
}
// Do signs the request and sends it via the next policy in the pipeline.
func (s *SharedKeyCredPolicy) Do(req *policy.Request) (*http.Response, error) {
// skip adding the authorization header if no SharedKeyCredential was provided.
// this prevents a panic that might be hard to diagnose and allows testing
// against http endpoints that don't require authentication.
if s.cred == nil {
return req.Next()
}
if d := getHeader("x-ms-date", req.Raw().Header); d == "" {
req.Raw().Header.Set("x-ms-date", time.Now().UTC().Format(http.TimeFormat))
}
stringToSign, err := s.cred.buildStringToSign(req.Raw())
if err != nil {
return nil, err
}
signature, err := s.cred.computeHMACSHA256(stringToSign)
if err != nil {
return nil, err
}
authHeader := strings.Join([]string{"SharedKey ", s.cred.AccountName(), ":", signature}, "")
req.Raw().Header.Set("Authorization", authHeader)
response, err := req.Next()
if err != nil && response != nil && response.StatusCode == http.StatusForbidden {
// Service failed to authenticate request, log it
fs.Debugf(nil, "azureblob arrowlist: HTTP Forbidden status, String-to-Sign: %q", stringToSign)
}
return response, err
}
+595
View File
@@ -0,0 +1,595 @@
// Package auth supplies the authentication and client creation for the azure SDK
package auth
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
"os"
"strings"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/obscure"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/lib/env"
)
const (
// Default storage account, key and blob endpoint for emulator support,
// though it is a base64 key checked in here, it is publicly available secret.
emulatorAccount = "devstoreaccount1"
emulatorAccountKey = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
emulatorBlobEndpoint = "http://127.0.0.1:10000/devstoreaccount1"
)
// ConfigOptions is the common authentication options for azure
var ConfigOptions = []fs.Option{{
Name: "account",
Help: `Azure Storage Account Name.
Set this to the Azure Storage Account Name in use.
Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable ` + "`AZURE_STORAGE_ACCOUNT_NAME`" + ` if possible.
`,
Sensitive: true,
}, {
Name: "env_auth",
Help: `Read credentials from runtime (environment variables, CLI or MSI).
See the [authentication docs](/azureblob#authentication) for full info.`,
Default: false,
}, {
Name: "key",
Help: `Storage Account Shared Key.
Leave blank to use SAS URL or Emulator.`,
Sensitive: true,
}, {
Name: "sas_url",
Help: `SAS URL for container level access only.
Leave blank if using account/key or Emulator.`,
Sensitive: true,
}, {
Name: "connection_string",
Help: `Storage Connection String.
Connection string for the storage. Leave blank if using other auth methods.
`,
Sensitive: true,
}, {
Name: "tenant",
Help: `ID of the service principal's tenant. Also called its directory ID.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
`,
Sensitive: true,
}, {
Name: "client_id",
Help: `The ID of the client in use.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
`,
Sensitive: true,
}, {
Name: "client_secret",
Help: `One of the service principal's client secrets
Set this if using
- Service principal with client secret
`,
Sensitive: true,
}, {
Name: "client_certificate_path",
Help: `Path to a PEM or PKCS12 certificate file including the private key.
Set this if using
- Service principal with certificate
`,
}, {
Name: "client_certificate_password",
Help: `Password for the certificate file (optional).
Optionally set this if using
- Service principal with certificate
And the certificate has a password.
`,
IsPassword: true,
}, {
Name: "client_send_certificate_chain",
Help: `Send the certificate chain when using certificate auth.
Specifies whether an authentication request will include an x5c header
to support subject name / issuer based authentication. When set to
true, authentication requests include the x5c header.
Optionally set this if using
- Service principal with certificate
`,
Default: false,
Advanced: true,
}, {
Name: "username",
Help: `User name (usually an email address)
Set this if using
- User with username and password
`,
Advanced: true,
Sensitive: true,
}, {
Name: "password",
Help: `The user's password
Set this if using
- User with username and password
`,
IsPassword: true,
Advanced: true,
}, {
Name: "service_principal_file",
Help: `Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "<name>" \
--role "Storage Blob Data Owner" \
--scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
> azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to blob data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
It may be more convenient to put the credentials directly into the
rclone config file under the ` + "`client_id`, `tenant` and `client_secret`" + `
keys instead of setting ` + "`service_principal_file`" + `.
`,
Advanced: true,
}, {
Name: "disable_instance_discovery",
Help: `Skip requesting Microsoft Entra instance metadata
This should be set true only by applications authenticating in
disconnected clouds, or private clouds such as Azure Stack.
It determines whether rclone requests Microsoft Entra instance
metadata from ` + "`https://login.microsoft.com/`" + ` before
authenticating.
Setting this to true will skip this request, making you responsible
for ensuring the configured authority is valid and trustworthy.
`,
Default: false,
Advanced: true,
}, {
Name: "use_msi",
Help: `Use a managed service identity to authenticate (only works in Azure).
When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
to authenticate to Azure Storage instead of a SAS token or account key.
If the VM(SS) on which this program is running has a system-assigned identity, it will
be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
the user-assigned identity will be used by default. If the resource has multiple user-assigned
identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
msi_client_id, or msi_mi_res_id parameters.`,
Default: false,
Advanced: true,
}, {
Name: "msi_object_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_mi_res_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "msi_client_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_object_id or msi_mi_res_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "msi_mi_res_id",
Help: "Azure resource ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_object_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "use_emulator",
Help: "Uses local storage emulator if provided as 'true'.\n\nLeave blank if using real azure storage endpoint.",
Default: false,
Advanced: true,
}, {
Name: "use_az",
Help: `Use Azure CLI tool az for authentication
Set to use the [Azure CLI tool az](https://learn.microsoft.com/en-us/cli/azure/)
as the sole means of authentication.
Setting this can be useful if you wish to use the az CLI on a host with
a System Managed Identity that you do not want to use.
Don't set env_auth at the same time.
`,
Default: false,
Advanced: true,
}, {
Name: "endpoint",
Help: "Endpoint for the service.\n\nLeave blank normally.",
Advanced: true,
}}
// Options defines the common auth configuration for azure backends
type Options struct {
Account string `config:"account"`
EnvAuth bool `config:"env_auth"`
Key string `config:"key"`
SASURL string `config:"sas_url"`
ConnectionString string `config:"connection_string"`
Tenant string `config:"tenant"`
ClientID string `config:"client_id"`
ClientSecret string `config:"client_secret"`
ClientCertificatePath string `config:"client_certificate_path"`
ClientCertificatePassword string `config:"client_certificate_password"`
ClientSendCertificateChain bool `config:"client_send_certificate_chain"`
Username string `config:"username"`
Password string `config:"password"`
ServicePrincipalFile string `config:"service_principal_file"`
DisableInstanceDiscovery bool `config:"disable_instance_discovery"`
UseMSI bool `config:"use_msi"`
MSIObjectID string `config:"msi_object_id"`
MSIClientID string `config:"msi_client_id"`
MSIResourceID string `config:"msi_mi_res_id"`
UseEmulator bool `config:"use_emulator"`
UseAZ bool `config:"use_az"`
Endpoint string `config:"endpoint"`
}
type servicePrincipalCredentials struct {
AppID string `json:"appId"`
Password string `json:"password"`
Tenant string `json:"tenant"`
}
// parseServicePrincipalCredentials unmarshals a service principal credentials JSON file as generated by az cli.
func parseServicePrincipalCredentials(ctx context.Context, credentialsData []byte) (*servicePrincipalCredentials, error) {
var spCredentials servicePrincipalCredentials
if err := json.Unmarshal(credentialsData, &spCredentials); err != nil {
return nil, fmt.Errorf("error parsing credentials from JSON file: %w", err)
}
// TODO: support certificate credentials
// Validate all fields present
if spCredentials.AppID == "" || spCredentials.Password == "" || spCredentials.Tenant == "" {
return nil, fmt.Errorf("missing fields in credentials file")
}
return &spCredentials, nil
}
// Wrap the http.Transport to satisfy the Transporter interface
type transporter struct {
http.RoundTripper
}
// Make a new transporter
func newTransporter(ctx context.Context) transporter {
// Set the User-Agent to include the Microsoft Partner Network prefix
ctx, ci := fs.AddConfig(ctx)
ci.UserAgent = "APN/1.0 rclone/1.0 rclone/" + strings.TrimPrefix(fs.Version, "v")
return transporter{
RoundTripper: fshttp.NewTransport(ctx),
}
}
// Do sends the HTTP request and returns the HTTP response or error.
func (tr transporter) Do(req *http.Request) (*http.Response, error) {
return tr.RoundTripper.RoundTrip(req)
}
// Transporter returns the policy.Transporter rclone uses for Azure SDK
// clients (an fshttp based transport with the APN user agent), for callers
// which build their own azcore pipelines.
func Transporter(ctx context.Context) policy.Transporter {
return newTransporter(ctx)
}
// NewClientOpts should be passed to configure NewClient
type NewClientOpts[Client, ClientOptions, SharedKeyCredential any] struct {
DefaultBaseURL string // Base URL, eg blob.core.windows.net
Blob bool // set if this is blob storage
RootContainer string // Container that rclone is looking at
NewClient func(serviceURL string, cred azcore.TokenCredential, options *ClientOptions) (*Client, error)
NewClientFromConnectionString func(connectionString string, options *ClientOptions) (*Client, error)
NewClientWithNoCredential func(serviceURL string, options *ClientOptions) (*Client, error)
NewClientWithSharedKeyCredential func(serviceURL string, cred *SharedKeyCredential, options *ClientOptions) (*Client, error)
NewSharedKeyCredential func(accountName, accountKey string) (*SharedKeyCredential, error)
SetClientOptions func(options *ClientOptions, policyClientOptions policy.ClientOptions)
}
// NewClientResult is returned from NewClient
type NewClientResult[Client any] struct {
Client *Client // Client to access the Service
Cred azcore.TokenCredential // how to generate tokens (may be nil)
UsingSharedKeyCred bool // set if using shared key credentials
Anonymous bool // true if anonymous authentication was used
Container string // Container that SAS URL points to
}
// NewClient creates a service client from the rclone options
func NewClient[Client, ClientOptions, SharedKeyCredential any](ctx context.Context, conf NewClientOpts[Client, ClientOptions, SharedKeyCredential], opt *Options) (r NewClientResult[Client], err error) {
var sharedKeyCred *SharedKeyCredential
// Client options specifying our own transport
policyClientOptions := policy.ClientOptions{
Transport: newTransporter(ctx),
}
// Can't do this with generics (yet)
// clientOpt := service.ClientOptions{
// ClientOptions: policyClientOptions,
// }
// So call back to user
var clientOpt ClientOptions
conf.SetClientOptions(&clientOpt, policyClientOptions)
// Here we auth by setting one of cred, sharedKeyCred, client or anonymous
switch {
case opt.EnvAuth:
// Read account from environment if needed
if opt.Account == "" {
opt.Account, _ = os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME")
}
// Read credentials from the environment
options := azidentity.DefaultAzureCredentialOptions{
ClientOptions: policyClientOptions,
DisableInstanceDiscovery: opt.DisableInstanceDiscovery,
}
r.Cred, err = azidentity.NewDefaultAzureCredential(&options)
if err != nil {
return r, fmt.Errorf("create azure environment credential failed: %w", err)
}
case opt.UseEmulator:
if opt.Account == "" {
opt.Account = emulatorAccount
}
if opt.Key == "" {
opt.Key = emulatorAccountKey
}
if opt.Endpoint == "" {
opt.Endpoint = emulatorBlobEndpoint
}
if conf.NewSharedKeyCredential == nil {
return r, errors.New("emulator use not supported")
}
sharedKeyCred, err = conf.NewSharedKeyCredential(opt.Account, opt.Key)
if err != nil {
return r, fmt.Errorf("create new shared key credential for emulator failed: %w", err)
}
case opt.Account != "" && opt.Key != "":
if conf.NewSharedKeyCredential == nil {
return r, errors.New("shared key credentials not supported")
}
sharedKeyCred, err = conf.NewSharedKeyCredential(opt.Account, opt.Key)
if err != nil {
return r, fmt.Errorf("create new shared key credential failed: %w", err)
}
case opt.SASURL != "":
parts, err := sas.ParseURL(opt.SASURL)
if err != nil {
return r, fmt.Errorf("failed to parse SAS URL: %w", err)
}
endpoint := opt.SASURL
r.Container = parts.ContainerName
// Check if we have container level SAS or account level SAS
if conf.Blob && r.Container != "" {
// Container level SAS
if conf.RootContainer != "" && r.Container != conf.RootContainer {
return r, fmt.Errorf("container name in SAS URL (%q) and container provided in command (%q) do not match", r.Container, conf.RootContainer)
}
// Rewrite the endpoint string to be without the container
parts.ContainerName = ""
endpoint = parts.String()
}
r.Client, err = conf.NewClientWithNoCredential(endpoint, &clientOpt)
if err != nil {
return r, fmt.Errorf("unable to create SAS URL client: %w", err)
}
case opt.ConnectionString != "":
r.Client, err = conf.NewClientFromConnectionString(opt.ConnectionString, &clientOpt)
if err != nil {
return r, fmt.Errorf("unable to create connection string client: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.ClientSecret != "":
// Service principal with client secret
options := azidentity.ClientSecretCredentialOptions{
ClientOptions: policyClientOptions,
}
r.Cred, err = azidentity.NewClientSecretCredential(opt.Tenant, opt.ClientID, opt.ClientSecret, &options)
if err != nil {
return r, fmt.Errorf("error creating a client secret credential: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.ClientCertificatePath != "":
// Service principal with certificate
//
// Read the certificate
data, err := os.ReadFile(env.ShellExpand(opt.ClientCertificatePath))
if err != nil {
return r, fmt.Errorf("error reading client certificate file: %w", err)
}
// NewClientCertificateCredential requires at least one *x509.Certificate, and a
// crypto.PrivateKey.
//
// ParseCertificates returns these given certificate data in PEM or PKCS12 format.
// It handles common scenarios but has limitations, for example it doesn't load PEM
// encrypted private keys.
var password []byte
if opt.ClientCertificatePassword != "" {
pw, err := obscure.Reveal(opt.Password)
if err != nil {
return r, fmt.Errorf("certificate password decode failed - did you obscure it?: %w", err)
}
password = []byte(pw)
}
certs, key, err := azidentity.ParseCertificates(data, password)
if err != nil {
return r, fmt.Errorf("failed to parse client certificate file: %w", err)
}
options := azidentity.ClientCertificateCredentialOptions{
ClientOptions: policyClientOptions,
SendCertificateChain: opt.ClientSendCertificateChain,
}
r.Cred, err = azidentity.NewClientCertificateCredential(
opt.Tenant, opt.ClientID, certs, key, &options,
)
if err != nil {
return r, fmt.Errorf("create azure service principal with client certificate credential failed: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.Username != "" && opt.Password != "":
// User with username and password
//nolint:staticcheck // this is deprecated due to Azure policy
options := azidentity.UsernamePasswordCredentialOptions{
ClientOptions: policyClientOptions,
}
password, err := obscure.Reveal(opt.Password)
if err != nil {
return r, fmt.Errorf("user password decode failed - did you obscure it?: %w", err)
}
r.Cred, err = azidentity.NewUsernamePasswordCredential(
opt.Tenant, opt.ClientID, opt.Username, password, &options,
)
if err != nil {
return r, fmt.Errorf("authenticate user with password failed: %w", err)
}
case opt.ServicePrincipalFile != "":
// Loading service principal credentials from file.
loadedCreds, err := os.ReadFile(env.ShellExpand(opt.ServicePrincipalFile))
if err != nil {
return r, fmt.Errorf("error opening service principal credentials file: %w", err)
}
parsedCreds, err := parseServicePrincipalCredentials(ctx, loadedCreds)
if err != nil {
return r, fmt.Errorf("error parsing service principal credentials file: %w", err)
}
options := azidentity.ClientSecretCredentialOptions{
ClientOptions: policyClientOptions,
}
r.Cred, err = azidentity.NewClientSecretCredential(parsedCreds.Tenant, parsedCreds.AppID, parsedCreds.Password, &options)
if err != nil {
return r, fmt.Errorf("error creating a client secret credential: %w", err)
}
case opt.UseMSI:
// Specifying a user-assigned identity. Exactly one of the above IDs must be specified.
// Validate and ensure exactly one is set. (To do: better validation.)
var b2i = map[bool]int{false: 0, true: 1}
set := b2i[opt.MSIClientID != ""] + b2i[opt.MSIObjectID != ""] + b2i[opt.MSIResourceID != ""]
if set > 1 {
return r, errors.New("more than one user-assigned identity ID is set")
}
var options azidentity.ManagedIdentityCredentialOptions
switch {
case opt.MSIClientID != "":
options.ID = azidentity.ClientID(opt.MSIClientID)
case opt.MSIObjectID != "":
// FIXME this doesn't appear to be in the new SDK?
return r, fmt.Errorf("MSI object ID is currently unsupported")
case opt.MSIResourceID != "":
options.ID = azidentity.ResourceID(opt.MSIResourceID)
}
r.Cred, err = azidentity.NewManagedIdentityCredential(&options)
if err != nil {
return r, fmt.Errorf("failed to acquire MSI token: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.MSIClientID != "":
// Workload Identity based authentication
var options azidentity.ManagedIdentityCredentialOptions
options.ID = azidentity.ClientID(opt.MSIClientID)
msiCred, err := azidentity.NewManagedIdentityCredential(&options)
if err != nil {
return r, fmt.Errorf("failed to acquire MSI token: %w", err)
}
getClientAssertions := func(context.Context) (string, error) {
token, err := msiCred.GetToken(context.Background(), policy.TokenRequestOptions{
Scopes: []string{"api://AzureADTokenExchange"},
})
if err != nil {
return "", fmt.Errorf("failed to acquire MSI token: %w", err)
}
return token.Token, nil
}
assertOpts := &azidentity.ClientAssertionCredentialOptions{}
r.Cred, err = azidentity.NewClientAssertionCredential(
opt.Tenant,
opt.ClientID,
getClientAssertions,
assertOpts)
if err != nil {
return r, fmt.Errorf("failed to acquire client assertion token: %w", err)
}
case opt.UseAZ:
var options = azidentity.AzureCLICredentialOptions{}
r.Cred, err = azidentity.NewAzureCLICredential(&options)
if err != nil {
return r, fmt.Errorf("failed to create Azure CLI credentials: %w", err)
}
case opt.Account != "":
// Anonymous access
r.Anonymous = true
default:
return r, errors.New("no authentication method configured")
}
// Make the client if not already created
if r.Client == nil {
// Work out what the endpoint is if it is still unset
if opt.Endpoint == "" {
if opt.Account == "" {
return r, fmt.Errorf("account must be set: can't make service URL")
}
u, err := url.Parse(fmt.Sprintf("https://%s.%s", opt.Account, conf.DefaultBaseURL))
if err != nil {
return r, fmt.Errorf("failed to make azure storage URL from account: %w", err)
}
opt.Endpoint = u.String()
}
if sharedKeyCred != nil {
// Shared key cred
r.Client, err = conf.NewClientWithSharedKeyCredential(opt.Endpoint, sharedKeyCred, &clientOpt)
if err != nil {
return r, fmt.Errorf("create client with shared key failed: %w", err)
}
r.UsingSharedKeyCred = true
} else if r.Cred != nil {
// Azidentity cred
r.Client, err = conf.NewClient(opt.Endpoint, r.Cred, &clientOpt)
if err != nil {
return r, fmt.Errorf("create client failed: %w", err)
}
} else if r.Anonymous {
// Anonymous public access
r.Client, err = conf.NewClientWithNoCredential(opt.Endpoint, &clientOpt)
if err != nil {
return r, fmt.Errorf("create public client failed: %w", err)
}
}
}
if r.Client == nil {
return r, fmt.Errorf("internal error: auth failed to make credentials or client")
}
return r, nil
}
File diff suppressed because it is too large. Load diff
@@ -3,13 +3,22 @@
package azureblob
import (
"bytes"
"compress/gzip"
"context"
"crypto/md5"
"encoding/base64"
"fmt"
"net/http"
"strings"
"testing"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/random"
@@ -145,7 +154,476 @@ func (f *Fs) testWriteUncommittedBlocks(t *testing.T) {
require.NoError(t, dst.Remove(ctx))
}
func gz(t *testing.T, s string) string {
var buf bytes.Buffer
zw := gzip.NewWriter(&buf)
_, err := zw.Write([]byte(s))
require.NoError(t, err)
err = zw.Close()
require.NoError(t, err)
return buf.String()
}
func md5sum(t *testing.T, s string) string {
hash := md5.Sum([]byte(s))
return fmt.Sprintf("%x", hash)
}
func (f *Fs) testGzipEncoding(t *testing.T) {
ctx := context.Background()
original := random.String(1000)
contents := gz(t, original)
item := fstest.NewItem("test-gzip", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
metadata := fs.Metadata{
"content-encoding": "gzip",
"content-type": "text/plain",
}
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/html", metadata)
defer func() {
assert.NoError(t, obj.Remove(ctx))
}()
o := obj.(*Object)
// Test that the gzipped file we uploaded can be
// downloaded with and without decompression
checkDownload := func(wantContents string, wantSize int64, wantHash string) {
gotContents := fstests.ReadObject(ctx, t, o, -1)
assert.Equal(t, wantContents, gotContents)
assert.Equal(t, wantSize, o.Size())
gotHash, err := o.Hash(ctx, hash.MD5)
require.NoError(t, err)
assert.Equal(t, wantHash, gotHash)
}
t.Run("NoDecompress", func(t *testing.T) {
checkDownload(contents, int64(len(contents)), md5sum(t, contents))
})
t.Run("Decompress", func(t *testing.T) {
f.opt.Decompress = true
defer func() {
f.opt.Decompress = false
}()
checkDownload(original, -1, "")
})
}
func (f *Fs) InternalTest(t *testing.T) {
t.Run("Features", f.testFeatures)
t.Run("WriteUncommittedBlocks", f.testWriteUncommittedBlocks)
t.Run("Metadata", f.testMetadataPaths)
t.Run("GzipEncoding", f.testGzipEncoding)
t.Run("ArrowList", f.testArrowList)
}
// helper to read blob properties for an object
func getProps(ctx context.Context, t *testing.T, o fs.Object) *blob.GetPropertiesResponse {
ao := o.(*Object)
props, err := ao.readMetaDataAlways(ctx)
require.NoError(t, err)
return props
}
// helper to assert select headers and user metadata
func assertHeadersAndMetadata(t *testing.T, props *blob.GetPropertiesResponse, want map[string]string, wantUserMeta map[string]string) {
// Headers
get := func(p *string) string {
if p == nil {
return ""
}
return *p
}
if v, ok := want["content-type"]; ok {
assert.Equal(t, v, get(props.ContentType), "content-type")
}
if v, ok := want["cache-control"]; ok {
assert.Equal(t, v, get(props.CacheControl), "cache-control")
}
if v, ok := want["content-disposition"]; ok {
assert.Equal(t, v, get(props.ContentDisposition), "content-disposition")
}
if v, ok := want["content-encoding"]; ok {
assert.Equal(t, v, get(props.ContentEncoding), "content-encoding")
}
if v, ok := want["content-language"]; ok {
assert.Equal(t, v, get(props.ContentLanguage), "content-language")
}
// User metadata (case-insensitive keys from service)
norm := make(map[string]*string, len(props.Metadata))
for kk, vv := range props.Metadata {
norm[strings.ToLower(kk)] = vv
}
for k, v := range wantUserMeta {
pv, ok := norm[strings.ToLower(k)]
if assert.True(t, ok, fmt.Sprintf("missing user metadata key %q", k)) {
if pv == nil {
assert.Equal(t, v, "", k)
} else {
assert.Equal(t, v, *pv, k)
}
} else {
// Log available keys for diagnostics
keys := make([]string, 0, len(props.Metadata))
for kk := range props.Metadata {
keys = append(keys, kk)
}
t.Logf("available user metadata keys: %v", keys)
}
}
}
// helper to read blob tags for an object
func getTagsMap(ctx context.Context, t *testing.T, o fs.Object) map[string]string {
ao := o.(*Object)
blb := ao.getBlobSVC()
resp, err := blb.GetTags(ctx, nil)
require.NoError(t, err)
out := make(map[string]string)
for _, tag := range resp.BlobTagSet {
if tag.Key != nil {
k := *tag.Key
v := ""
if tag.Value != nil {
v = *tag.Value
}
out[k] = v
}
}
return out
}
// Test metadata across different write paths
func (f *Fs) testMetadataPaths(t *testing.T) {
ctx := context.Background()
if testing.Short() {
t.Skip("skipping in short mode")
}
// Common expected metadata and headers
baseMeta := fs.Metadata{
"cache-control": "no-cache",
"content-disposition": "inline",
"content-language": "en-US",
// Note: Don't set content-encoding here to avoid download decoding differences
// We will set a custom user metadata key
"potato": "royal",
// and modtime
"mtime": fstest.Time("2009-05-06T04:05:06.499999999Z").Format(time.RFC3339Nano),
}
// Singlepart upload
t.Run("PutSinglepart", func(t *testing.T) {
// size less than chunk size
contents := random.String(int(f.opt.ChunkSize / 2))
item := fstest.NewItem("meta-single.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
// override content-type via metadata mapping
meta := fs.Metadata{}
meta.Merge(baseMeta)
meta["content-type"] = "text/plain"
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/html", meta)
defer func() { _ = obj.Remove(ctx) }()
props := getProps(ctx, t, obj)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "text/plain",
"cache-control": "no-cache",
"content-disposition": "inline",
"content-language": "en-US",
}, map[string]string{
"potato": "royal",
})
_ = http.StatusOK // keep import for parity but don't inspect RawResponse
})
// Multipart upload
t.Run("PutMultipart", func(t *testing.T) {
// size greater than chunk size to force multipart
contents := random.String(int(f.opt.ChunkSize + 1024))
item := fstest.NewItem("meta-multipart.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{}
meta.Merge(baseMeta)
meta["content-type"] = "application/json"
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/html", meta)
defer func() { _ = obj.Remove(ctx) }()
props := getProps(ctx, t, obj)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "application/json",
"cache-control": "no-cache",
"content-disposition": "inline",
"content-language": "en-US",
}, map[string]string{
"potato": "royal",
})
// Tags: Singlepart upload
t.Run("PutSinglepartTags", func(t *testing.T) {
contents := random.String(int(f.opt.ChunkSize / 2))
item := fstest.NewItem("tags-single.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
"x-ms-tags": "env=dev,team=sync",
}
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/plain", meta)
defer func() { _ = obj.Remove(ctx) }()
tags := getTagsMap(ctx, t, obj)
assert.Equal(t, "dev", tags["env"])
assert.Equal(t, "sync", tags["team"])
})
// Tags: Multipart upload
t.Run("PutMultipartTags", func(t *testing.T) {
contents := random.String(int(f.opt.ChunkSize + 2048))
item := fstest.NewItem("tags-multipart.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
"x-ms-tags": "project=alpha,release=2025-08",
}
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "application/octet-stream", meta)
defer func() { _ = obj.Remove(ctx) }()
tags := getTagsMap(ctx, t, obj)
assert.Equal(t, "alpha", tags["project"])
assert.Equal(t, "2025-08", tags["release"])
})
})
// Singlepart copy with metadata-set mapping; omit content-type to exercise fallback
t.Run("CopySinglepart", func(t *testing.T) {
// create small source
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("meta-copy-single-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "text/plain", nil)
defer func() { _ = srcObj.Remove(ctx) }()
// set mapping via MetadataSet
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"cache-control": "private, max-age=60",
"content-disposition": "attachment; filename=foo.txt",
"content-language": "fr",
// no content-type: should fallback to source
"potato": "maris",
}
// do copy
dstName := "meta-copy-single-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
// content-type should fallback to source (text/plain)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "text/plain",
"cache-control": "private, max-age=60",
"content-disposition": "attachment; filename=foo.txt",
"content-language": "fr",
}, map[string]string{
"potato": "maris",
})
// mtime should be populated on copy when --metadata is used
// and should equal the source ModTime (RFC3339Nano)
// Read user metadata (case-insensitive)
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
// parse and compare times ignoring formatting differences
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// CopySinglepart with only --metadata (no MetadataSet) must inject mtime and preserve src content-type
t.Run("CopySinglepart_MetadataOnly", func(t *testing.T) {
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("meta-copy-single-only-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "text/plain", nil)
defer func() { _ = srcObj.Remove(ctx) }()
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
dstName := "meta-copy-single-only-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "text/plain",
}, map[string]string{})
// Assert mtime injected
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// Multipart copy with metadata-set mapping; omit content-type to exercise fallback
t.Run("CopyMultipart", func(t *testing.T) {
// create large source to force multipart
contents := random.String(int(f.opt.CopyCutoff + 1024))
srcItem := fstest.NewItem("meta-copy-multi-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "application/octet-stream", nil)
defer func() { _ = srcObj.Remove(ctx) }()
// set mapping via MetadataSet
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"cache-control": "max-age=0, no-cache",
// omit content-type to trigger fallback
"content-language": "de",
"potato": "desiree",
}
dstName := "meta-copy-multi-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
// content-type should fallback to source (application/octet-stream)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "application/octet-stream",
"cache-control": "max-age=0, no-cache",
"content-language": "de",
}, map[string]string{
"potato": "desiree",
})
// mtime should be populated on copy when --metadata is used
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// CopyMultipart with only --metadata must inject mtime and preserve src content-type
t.Run("CopyMultipart_MetadataOnly", func(t *testing.T) {
contents := random.String(int(f.opt.CopyCutoff + 2048))
srcItem := fstest.NewItem("meta-copy-multi-only-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "application/octet-stream", nil)
defer func() { _ = srcObj.Remove(ctx) }()
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
dstName := "meta-copy-multi-only-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "application/octet-stream",
}, map[string]string{})
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// Tags: Singlepart copy
t.Run("CopySinglepartTags", func(t *testing.T) {
// create small source
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("tags-copy-single-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "text/plain", nil)
defer func() { _ = srcObj.Remove(ctx) }()
// set mapping via MetadataSet including tags
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"x-ms-tags": "copy=single,mode=test",
}
dstName := "tags-copy-single-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
tags := getTagsMap(ctx2, t, dst)
assert.Equal(t, "single", tags["copy"])
assert.Equal(t, "test", tags["mode"])
})
// Tags: Multipart copy
t.Run("CopyMultipartTags", func(t *testing.T) {
// create large source to force multipart
contents := random.String(int(f.opt.CopyCutoff + 4096))
srcItem := fstest.NewItem("tags-copy-multi-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "application/octet-stream", nil)
defer func() { _ = srcObj.Remove(ctx) }()
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"x-ms-tags": "copy=multi,mode=test",
}
dstName := "tags-copy-multi-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
tags := getTagsMap(ctx2, t, dst)
assert.Equal(t, "multi", tags["copy"])
assert.Equal(t, "test", tags["mode"])
})
// Negative: invalid x-ms-tags must error
t.Run("InvalidXMsTags", func(t *testing.T) {
contents := random.String(32)
item := fstest.NewItem("tags-invalid.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
// construct ObjectInfo with invalid x-ms-tags
buf := strings.NewReader(contents)
// Build obj info with metadata
meta := fs.Metadata{
"x-ms-tags": "badpair-without-equals",
}
// force metadata on
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
obji := object.NewStaticObjectInfo(item.Path, item.ModTime, int64(len(contents)), true, nil, nil)
obji = obji.WithMetadata(meta).WithMimeType("text/plain")
_, err := f.Put(ctx2, buf, obji)
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid tag")
})
}
+102 -495
View File
@@ -29,35 +29,28 @@ import (
"context"
"crypto/md5"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"os"
"path"
"slices"
"strings"
"sync"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azfile/directory"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azfile/file"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azfile/fileerror"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azfile/service"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azfile/share"
"github.com/rclone/rclone/backend/azureblob/auth"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/config/obscure"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/list"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/env"
"github.com/rclone/rclone/lib/readers"
)
@@ -65,6 +58,9 @@ const (
maxFileSize = 4 * fs.Tebi
defaultChunkSize = 4 * fs.Mebi
storageDefaultBaseURL = "file.core.windows.net"
// smbTimePrecision is the precision the server stores
// modtimes with (100 ns FILETIME ticks)
smbTimePrecision = 100 * time.Nanosecond
)
func init() {
@@ -72,199 +68,12 @@ func init() {
Name: "azurefiles",
Description: "Microsoft Azure Files",
NewFs: NewFs,
Options: []fs.Option{{
Name: "account",
Help: `Azure Storage Account Name.
Set this to the Azure Storage Account Name in use.
Leave blank to use SAS URL or connection string, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable ` + "`AZURE_STORAGE_ACCOUNT_NAME`" + ` if possible.
`,
Sensitive: true,
}, {
Options: slices.Concat(auth.ConfigOptions, []fs.Option{{
Name: "share_name",
Help: `Azure Files Share Name.
This is required and is the name of the share to access.
`,
}, {
Name: "env_auth",
Help: `Read credentials from runtime (environment variables, CLI or MSI).
See the [authentication docs](/azurefiles#authentication) for full info.`,
Default: false,
}, {
Name: "key",
Help: `Storage Account Shared Key.
Leave blank to use SAS URL or connection string.`,
Sensitive: true,
}, {
Name: "sas_url",
Help: `SAS URL.
Leave blank if using account/key or connection string.`,
Sensitive: true,
}, {
Name: "connection_string",
Help: `Azure Files Connection String.`,
Sensitive: true,
}, {
Name: "tenant",
Help: `ID of the service principal's tenant. Also called its directory ID.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
`,
Sensitive: true,
}, {
Name: "client_id",
Help: `The ID of the client in use.
Set this if using
- Service principal with client secret
- Service principal with certificate
- User with username and password
`,
Sensitive: true,
}, {
Name: "client_secret",
Help: `One of the service principal's client secrets
Set this if using
- Service principal with client secret
`,
Sensitive: true,
}, {
Name: "client_certificate_path",
Help: `Path to a PEM or PKCS12 certificate file including the private key.
Set this if using
- Service principal with certificate
`,
}, {
Name: "client_certificate_password",
Help: `Password for the certificate file (optional).
Optionally set this if using
- Service principal with certificate
And the certificate has a password.
`,
IsPassword: true,
}, {
Name: "client_send_certificate_chain",
Help: `Send the certificate chain when using certificate auth.
Specifies whether an authentication request will include an x5c header
to support subject name / issuer based authentication. When set to
true, authentication requests include the x5c header.
Optionally set this if using
- Service principal with certificate
`,
Default: false,
Advanced: true,
}, {
Name: "username",
Help: `User name (usually an email address)
Set this if using
- User with username and password
`,
Advanced: true,
Sensitive: true,
}, {
Name: "password",
Help: `The user's password
Set this if using
- User with username and password
`,
IsPassword: true,
Advanced: true,
}, {
Name: "service_principal_file",
Help: `Path to file containing credentials for use with a service principal.
Leave blank normally. Needed only if you want to use a service principal instead of interactive login.
$ az ad sp create-for-rbac --name "<name>" \
--role "Storage Files Data Owner" \
--scopes "/subscriptions/<subscription>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account>/blobServices/default/containers/<container>" \
> azure-principal.json
See ["Create an Azure service principal"](https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli) and ["Assign an Azure role for access to files data"](https://docs.microsoft.com/en-us/azure/storage/common/storage-auth-aad-rbac-cli) pages for more details.
**NB** this section needs updating for Azure Files - pull requests appreciated!
It may be more convenient to put the credentials directly into the
rclone config file under the ` + "`client_id`, `tenant` and `client_secret`" + `
keys instead of setting ` + "`service_principal_file`" + `.
`,
Advanced: true,
}, {
Name: "use_msi",
Help: `Use a managed service identity to authenticate (only works in Azure).
When true, use a [managed service identity](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/)
to authenticate to Azure Storage instead of a SAS token or account key.
If the VM(SS) on which this program is running has a system-assigned identity, it will
be used by default. If the resource has no system-assigned but exactly one user-assigned identity,
the user-assigned identity will be used by default. If the resource has multiple user-assigned
identities, the identity to use must be explicitly specified using exactly one of the msi_object_id,
msi_client_id, or msi_mi_res_id parameters.`,
Default: false,
Advanced: true,
}, {
Name: "msi_object_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_mi_res_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "msi_client_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_object_id or msi_mi_res_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "msi_mi_res_id",
Help: "Azure resource ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_object_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "disable_instance_discovery",
Help: `Skip requesting Microsoft Entra instance metadata
This should be set true only by applications authenticating in
disconnected clouds, or private clouds such as Azure Stack.
It determines whether rclone requests Microsoft Entra instance
metadata from ` + "`https://login.microsoft.com/`" + ` before
authenticating.
Setting this to true will skip this request, making you responsible
for ensuring the configured authority is valid and trustworthy.
`,
Default: false,
Advanced: true,
}, {
Name: "use_az",
Help: `Use Azure CLI tool az for authentication
Set to use the [Azure CLI tool az](https://learn.microsoft.com/en-us/cli/azure/)
as the sole means of authentication.
Setting this can be useful if you wish to use the az CLI on a host with
a System Managed Identity that you do not want to use.
Don't set env_auth at the same time.
`,
Default: false,
Advanced: true,
}, {
Name: "endpoint",
Help: "Endpoint for the service.\n\nLeave blank normally.",
Advanced: true,
}, {
Name: "chunk_size",
Help: `Upload chunk size.
@@ -322,38 +131,18 @@ You will need this much free space in the share as the file will be this size te
encoder.EncodeInvalidUtf8 |
encoder.EncodeCtl | encoder.EncodeDel |
encoder.EncodeDot | encoder.EncodeRightPeriod),
}},
}}),
})
}
// Options defines the configuration for this backend
type Options struct {
Account string `config:"account"`
ShareName string `config:"share_name"`
EnvAuth bool `config:"env_auth"`
Key string `config:"key"`
SASURL string `config:"sas_url"`
ConnectionString string `config:"connection_string"`
Tenant string `config:"tenant"`
ClientID string `config:"client_id"`
ClientSecret string `config:"client_secret"`
ClientCertificatePath string `config:"client_certificate_path"`
ClientCertificatePassword string `config:"client_certificate_password"`
ClientSendCertificateChain bool `config:"client_send_certificate_chain"`
Username string `config:"username"`
Password string `config:"password"`
ServicePrincipalFile string `config:"service_principal_file"`
DisableInstanceDiscovery bool `config:"disable_instance_discovery"`
UseMSI bool `config:"use_msi"`
MSIObjectID string `config:"msi_object_id"`
MSIClientID string `config:"msi_client_id"`
MSIResourceID string `config:"msi_mi_res_id"`
UseAZ bool `config:"use_az"`
Endpoint string `config:"endpoint"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
MaxStreamSize fs.SizeSuffix `config:"max_stream_size"`
UploadConcurrency int `config:"upload_concurrency"`
Enc encoder.MultiEncoder `config:"encoding"`
auth.Options
ShareName string `config:"share_name"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
MaxStreamSize fs.SizeSuffix `config:"max_stream_size"`
UploadConcurrency int `config:"upload_concurrency"`
Enc encoder.MultiEncoder `config:"encoding"`
}
// Fs represents a root directory inside a share. The root directory can be ""
@@ -376,267 +165,33 @@ type Object struct {
contentType string // content type if known
}
// Wrap the http.Transport to satisfy the Transporter interface
type transporter struct {
http.RoundTripper
}
// Make a new transporter
func newTransporter(ctx context.Context) transporter {
return transporter{
RoundTripper: fshttp.NewTransport(ctx),
}
}
// Do sends the HTTP request and returns the HTTP response or error.
func (tr transporter) Do(req *http.Request) (*http.Response, error) {
return tr.RoundTripper.RoundTrip(req)
}
type servicePrincipalCredentials struct {
AppID string `json:"appId"`
Password string `json:"password"`
Tenant string `json:"tenant"`
}
// parseServicePrincipalCredentials unmarshals a service principal credentials JSON file as generated by az cli.
func parseServicePrincipalCredentials(ctx context.Context, credentialsData []byte) (*servicePrincipalCredentials, error) {
var spCredentials servicePrincipalCredentials
if err := json.Unmarshal(credentialsData, &spCredentials); err != nil {
return nil, fmt.Errorf("error parsing credentials from JSON file: %w", err)
}
// TODO: support certificate credentials
// Validate all fields present
if spCredentials.AppID == "" || spCredentials.Password == "" || spCredentials.Tenant == "" {
return nil, fmt.Errorf("missing fields in credentials file")
}
return &spCredentials, nil
}
// Factored out from NewFs so that it can be tested with opt *Options and without m configmap.Mapper
func newFsFromOptions(ctx context.Context, name, root string, opt *Options) (fs.Fs, error) {
// Client options specifying our own transport
policyClientOptions := policy.ClientOptions{
Transport: newTransporter(ctx),
conf := auth.NewClientOpts[service.Client, service.ClientOptions, service.SharedKeyCredential]{
DefaultBaseURL: storageDefaultBaseURL,
NewClient: service.NewClient,
NewClientFromConnectionString: service.NewClientFromConnectionString,
NewClientWithNoCredential: service.NewClientWithNoCredential,
NewClientWithSharedKeyCredential: service.NewClientWithSharedKeyCredential,
NewSharedKeyCredential: service.NewSharedKeyCredential,
SetClientOptions: func(options *service.ClientOptions, policyClientOptions policy.ClientOptions) {
options.ClientOptions = policyClientOptions
// FileRequestIntent is required for TokenCredential (OAuth)
// auth; the SDK omits it for shared-key / SAS.
backup := service.ShareTokenIntentBackup
options.FileRequestIntent = &backup
},
}
backup := service.ShareTokenIntentBackup
clientOpt := service.ClientOptions{
ClientOptions: policyClientOptions,
FileRequestIntent: &backup,
res, err := auth.NewClient(ctx, conf, &opt.Options)
if err != nil {
return nil, err
}
// f.svc = res.Client
// f.cred = res.Cred
// f.sharedKeyCred = res.SharedKeyCred
// f.anonymous = res.Anonymous
// Here we auth by setting one of cred, sharedKeyCred or f.client
var (
cred azcore.TokenCredential
sharedKeyCred *service.SharedKeyCredential
client *service.Client
err error
)
switch {
case opt.EnvAuth:
// Read account from environment if needed
if opt.Account == "" {
opt.Account, _ = os.LookupEnv("AZURE_STORAGE_ACCOUNT_NAME")
}
// Read credentials from the environment
options := azidentity.DefaultAzureCredentialOptions{
ClientOptions: policyClientOptions,
DisableInstanceDiscovery: opt.DisableInstanceDiscovery,
}
cred, err = azidentity.NewDefaultAzureCredential(&options)
if err != nil {
return nil, fmt.Errorf("create azure environment credential failed: %w", err)
}
case opt.Account != "" && opt.Key != "":
sharedKeyCred, err = service.NewSharedKeyCredential(opt.Account, opt.Key)
if err != nil {
return nil, fmt.Errorf("create new shared key credential failed: %w", err)
}
case opt.UseAZ:
var options = azidentity.AzureCLICredentialOptions{}
cred, err = azidentity.NewAzureCLICredential(&options)
fmt.Println(cred)
if err != nil {
return nil, fmt.Errorf("failed to create Azure CLI credentials: %w", err)
}
case opt.SASURL != "":
client, err = service.NewClientWithNoCredential(opt.SASURL, &clientOpt)
if err != nil {
return nil, fmt.Errorf("unable to create SAS URL client: %w", err)
}
case opt.ConnectionString != "":
client, err = service.NewClientFromConnectionString(opt.ConnectionString, &clientOpt)
if err != nil {
return nil, fmt.Errorf("unable to create connection string client: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.ClientSecret != "":
// Service principal with client secret
options := azidentity.ClientSecretCredentialOptions{
ClientOptions: policyClientOptions,
}
cred, err = azidentity.NewClientSecretCredential(opt.Tenant, opt.ClientID, opt.ClientSecret, &options)
if err != nil {
return nil, fmt.Errorf("error creating a client secret credential: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.ClientCertificatePath != "":
// Service principal with certificate
//
// Read the certificate
data, err := os.ReadFile(env.ShellExpand(opt.ClientCertificatePath))
if err != nil {
return nil, fmt.Errorf("error reading client certificate file: %w", err)
}
// NewClientCertificateCredential requires at least one *x509.Certificate, and a
// crypto.PrivateKey.
//
// ParseCertificates returns these given certificate data in PEM or PKCS12 format.
// It handles common scenarios but has limitations, for example it doesn't load PEM
// encrypted private keys.
var password []byte
if opt.ClientCertificatePassword != "" {
pw, err := obscure.Reveal(opt.Password)
if err != nil {
return nil, fmt.Errorf("certificate password decode failed - did you obscure it?: %w", err)
}
password = []byte(pw)
}
certs, key, err := azidentity.ParseCertificates(data, password)
if err != nil {
return nil, fmt.Errorf("failed to parse client certificate file: %w", err)
}
options := azidentity.ClientCertificateCredentialOptions{
ClientOptions: policyClientOptions,
SendCertificateChain: opt.ClientSendCertificateChain,
}
cred, err = azidentity.NewClientCertificateCredential(
opt.Tenant, opt.ClientID, certs, key, &options,
)
if err != nil {
return nil, fmt.Errorf("create azure service principal with client certificate credential failed: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.Username != "" && opt.Password != "":
// User with username and password
//nolint:staticcheck // this is deprecated due to Azure policy
options := azidentity.UsernamePasswordCredentialOptions{
ClientOptions: policyClientOptions,
}
password, err := obscure.Reveal(opt.Password)
if err != nil {
return nil, fmt.Errorf("user password decode failed - did you obscure it?: %w", err)
}
cred, err = azidentity.NewUsernamePasswordCredential(
opt.Tenant, opt.ClientID, opt.Username, password, &options,
)
if err != nil {
return nil, fmt.Errorf("authenticate user with password failed: %w", err)
}
case opt.ServicePrincipalFile != "":
// Loading service principal credentials from file.
loadedCreds, err := os.ReadFile(env.ShellExpand(opt.ServicePrincipalFile))
if err != nil {
return nil, fmt.Errorf("error opening service principal credentials file: %w", err)
}
parsedCreds, err := parseServicePrincipalCredentials(ctx, loadedCreds)
if err != nil {
return nil, fmt.Errorf("error parsing service principal credentials file: %w", err)
}
options := azidentity.ClientSecretCredentialOptions{
ClientOptions: policyClientOptions,
}
cred, err = azidentity.NewClientSecretCredential(parsedCreds.Tenant, parsedCreds.AppID, parsedCreds.Password, &options)
if err != nil {
return nil, fmt.Errorf("error creating a client secret credential: %w", err)
}
case opt.UseMSI:
// Specifying a user-assigned identity. Exactly one of the above IDs must be specified.
// Validate and ensure exactly one is set. (To do: better validation.)
var b2i = map[bool]int{false: 0, true: 1}
set := b2i[opt.MSIClientID != ""] + b2i[opt.MSIObjectID != ""] + b2i[opt.MSIResourceID != ""]
if set > 1 {
return nil, errors.New("more than one user-assigned identity ID is set")
}
var options azidentity.ManagedIdentityCredentialOptions
switch {
case opt.MSIClientID != "":
options.ID = azidentity.ClientID(opt.MSIClientID)
case opt.MSIObjectID != "":
// FIXME this doesn't appear to be in the new SDK?
return nil, fmt.Errorf("MSI object ID is currently unsupported")
case opt.MSIResourceID != "":
options.ID = azidentity.ResourceID(opt.MSIResourceID)
}
cred, err = azidentity.NewManagedIdentityCredential(&options)
if err != nil {
return nil, fmt.Errorf("failed to acquire MSI token: %w", err)
}
case opt.ClientID != "" && opt.Tenant != "" && opt.MSIClientID != "":
// Workload Identity based authentication
var options azidentity.ManagedIdentityCredentialOptions
options.ID = azidentity.ClientID(opt.MSIClientID)
msiCred, err := azidentity.NewManagedIdentityCredential(&options)
if err != nil {
return nil, fmt.Errorf("failed to acquire MSI token: %w", err)
}
getClientAssertions := func(context.Context) (string, error) {
token, err := msiCred.GetToken(context.Background(), policy.TokenRequestOptions{
Scopes: []string{"api://AzureADTokenExchange"},
})
if err != nil {
return "", fmt.Errorf("failed to acquire MSI token: %w", err)
}
return token.Token, nil
}
assertOpts := &azidentity.ClientAssertionCredentialOptions{}
cred, err = azidentity.NewClientAssertionCredential(
opt.Tenant,
opt.ClientID,
getClientAssertions,
assertOpts)
if err != nil {
return nil, fmt.Errorf("failed to acquire client assertion token: %w", err)
}
default:
return nil, errors.New("no authentication method configured")
}
// Make the client if not already created
if client == nil {
// Work out what the endpoint is if it is still unset
if opt.Endpoint == "" {
if opt.Account == "" {
return nil, fmt.Errorf("account must be set: can't make service URL")
}
u, err := url.Parse(fmt.Sprintf("https://%s.%s", opt.Account, storageDefaultBaseURL))
if err != nil {
return nil, fmt.Errorf("failed to make azure storage URL from account: %w", err)
}
opt.Endpoint = u.String()
}
if sharedKeyCred != nil {
// Shared key cred
client, err = service.NewClientWithSharedKeyCredential(opt.Endpoint, sharedKeyCred, &clientOpt)
if err != nil {
return nil, fmt.Errorf("create client with shared key failed: %w", err)
}
} else if cred != nil {
// Azidentity cred
client, err = service.NewClient(opt.Endpoint, cred, &clientOpt)
if err != nil {
return nil, fmt.Errorf("create client failed: %w", err)
}
}
}
if client == nil {
return nil, fmt.Errorf("internal error: auth failed to make credentials or client")
}
shareClient := client.NewShareClient(opt.ShareName)
shareClient := res.Client.NewShareClient(opt.ShareName)
svc := shareClient.NewRootDirectoryClient()
f := &Fs{
shareClient: shareClient,
@@ -698,10 +253,10 @@ func (f *Fs) Features() *fs.Features {
// Precision return the precision of this Fs
//
// One second. FileREST API times are in RFC1123 which in the example shows a precision of seconds
// The SMB last write time properties are ISO 8601 with 100 ns (FILETIME) precision
// Source: https://learn.microsoft.com/en-us/rest/api/storageservices/representation-of-date-time-values-in-headers
func (f *Fs) Precision() time.Duration {
return time.Second
return smbTimePrecision
}
// Hashes returns the supported hash sets.
@@ -844,18 +399,35 @@ func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, opt
//
// This should return ErrDirNotFound if the directory isn't found.
func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
var entries fs.DirEntries
return list.WithListP(ctx, dir, f)
}
// ListP lists the objects and directories of the Fs starting
// from dir non recursively into out.
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) error {
list := list.NewHelper(callback)
subDirClient := f.dirClient(dir)
// Checking whether directory exists
_, err := subDirClient.GetProperties(ctx, nil)
if fileerror.HasCode(err, fileerror.ParentNotFound, fileerror.ResourceNotFound) {
return entries, fs.ErrorDirNotFound
return fs.ErrorDirNotFound
} else if err != nil {
return entries, err
return err
}
var opt = &directory.ListFilesAndDirectoriesOptions{
opt := &directory.ListFilesAndDirectoriesOptions{
Include: directory.ListFilesInclude{
Timestamps: true,
},
@@ -864,7 +436,7 @@ func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
for pager.More() {
resp, err := pager.NextPage(ctx)
if err != nil {
return entries, err
return err
}
for _, directory := range resp.Segment.Directories {
// Name *string `xml:"Name"`
@@ -890,7 +462,10 @@ func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
if directory.Properties.ContentLength != nil {
entry.SetSize(*directory.Properties.ContentLength)
}
entries = append(entries, entry)
err = list.Add(entry)
if err != nil {
return err
}
}
for _, file := range resp.Segment.Files {
leaf := f.opt.Enc.ToStandardPath(*file.Name)
@@ -904,10 +479,13 @@ func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
if file.Properties.LastWriteTime != nil {
entry.modTime = *file.Properties.LastWriteTime
}
entries = append(entries, entry)
err = list.Add(entry)
if err != nil {
return err
}
}
}
return entries, nil
return list.Flush()
}
// ------------------------------------------------------------
@@ -1014,12 +592,19 @@ func (o *Object) SetModTime(ctx context.Context, t time.Time) error {
SMBProperties: &file.SMBProperties{
LastWriteTime: &t,
},
HTTPHeaders: &file.HTTPHeaders{
ContentMD5: o.md5,
ContentType: &o.contentType,
},
}
_, err := o.fileClient().SetHTTPHeaders(ctx, &opt)
if err != nil {
return fmt.Errorf("unable to set modTime: %w", err)
}
o.modTime = t
// Truncate to the precision the server stores the modtime with
// to keep the in-memory modtime identical to the one a fresh
// listing returns.
o.modTime = t.Truncate(smbTimePrecision)
return nil
}
@@ -1190,10 +775,12 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
return fmt.Errorf("update: failed to set properties: %w", err)
}
// Make sure Object is in sync
// Make sure Object is in sync, truncating the modtime to the
// precision the server stores it with so it is identical to the
// one a fresh listing returns
o.size = size
o.md5 = md5Hash
o.modTime = modTime
o.modTime = modTime.Truncate(smbTimePrecision)
o.contentType = contentType
return nil
}
@@ -1310,10 +897,29 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
}
srcURL := srcObj.fileClient().URL()
fc := f.fileClient(remote)
_, err = fc.StartCopyFromURL(ctx, srcURL, &opt)
startCopy, err := fc.StartCopyFromURL(ctx, srcURL, &opt)
if err != nil {
return nil, fmt.Errorf("Copy failed: %w", err)
}
// Poll for completion if necessary
//
// The for loop is never executed for same storage account copies.
copyStatus := startCopy.CopyStatus
var properties file.GetPropertiesResponse
pollTime := 100 * time.Millisecond
for copyStatus != nil && string(*copyStatus) == string(file.CopyStatusTypePending) {
time.Sleep(pollTime)
properties, err = fc.GetProperties(ctx, &file.GetPropertiesOptions{})
if err != nil {
return nil, err
}
copyStatus = properties.CopyStatus
pollTime = min(2*pollTime, time.Second)
}
dstObj, err := f.NewObject(ctx, remote)
if err != nil {
return nil, fmt.Errorf("Copy: NewObject failed: %w", err)
@@ -1428,6 +1034,7 @@ var (
_ fs.DirMover = &Fs{}
_ fs.Copier = &Fs{}
_ fs.OpenWriterAter = &Fs{}
_ fs.ListPer = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
)
+12 -5
View File
@@ -8,6 +8,7 @@ import (
"strings"
"testing"
"github.com/rclone/rclone/backend/azureblob/auth"
"github.com/rclone/rclone/fstest/fstests"
"github.com/stretchr/testify/assert"
)
@@ -28,22 +29,28 @@ func (f *Fs) InternalTestAuth(t *testing.T) {
{
name: "ConnectionString",
options: &Options{
ShareName: shareName,
ConnectionString: "",
ShareName: shareName,
Options: auth.Options{
ConnectionString: "",
},
},
},
{
name: "AccountAndKey",
options: &Options{
ShareName: shareName,
Account: "",
Key: "",
Options: auth.Options{
Account: "",
Key: "",
},
}},
{
name: "SASUrl",
options: &Options{
ShareName: shareName,
SASURL: "",
Options: auth.Options{
SASURL: "",
},
}},
}
+44 -22
View File
@@ -48,6 +48,14 @@ type LifecycleRule struct {
FileNamePrefix string `json:"fileNamePrefix"`
}
// ServerSideEncryption is a configuration object for B2 Server-Side Encryption
type ServerSideEncryption struct {
Mode string `json:"mode"`
Algorithm string `json:"algorithm"` // Encryption algorithm to use
CustomerKey string `json:"customerKey"` // User provided Base64 encoded key that is used by the server to encrypt files
CustomerKeyMd5 string `json:"customerKeyMd5"` // An MD5 hash of the decoded key
}
// Timestamp is a UTC time when this file was uploaded. It is a base
// 10 number of milliseconds since midnight, January 1, 1970 UTC. This
// fits in a 64 bit integer such as the type "long" in the programming
@@ -125,23 +133,32 @@ type File struct {
Info map[string]string `json:"fileInfo"` // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
}
// AuthorizeAccountResponse is as returned from the b2_authorize_account call
type AuthorizeAccountResponse struct {
// StorageAPI is as returned from the b2_authorize_account call
type StorageAPI struct {
AbsoluteMinimumPartSize int `json:"absoluteMinimumPartSize"` // The smallest possible size of a part of a large file.
AccountID string `json:"accountId"` // The identifier for the account.
Allowed struct { // An object (see below) containing the capabilities of this auth token, and any restrictions on using it.
BucketID string `json:"bucketId"` // When present, access is restricted to one bucket.
BucketName string `json:"bucketName"` // When present, name of bucket - may be empty
Capabilities []string `json:"capabilities"` // A list of strings, each one naming a capability the key has.
Buckets []struct { // When present, access is restricted to one or more buckets.
ID string `json:"id"` // ID of bucket
Name string `json:"name"` // When present, name of bucket - may be empty
} `json:"buckets"`
Capabilities []string `json:"capabilities"` // A list of strings, each one naming a capability the key has for every bucket.
NamePrefix any `json:"namePrefix"` // When present, access is restricted to files whose names start with the prefix
} `json:"allowed"`
APIURL string `json:"apiUrl"` // The base URL to use for all API calls except for uploading and downloading files.
AuthorizationToken string `json:"authorizationToken"` // An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
DownloadURL string `json:"downloadUrl"` // The base URL to use for downloading files.
MinimumPartSize int `json:"minimumPartSize"` // DEPRECATED: This field will always have the same value as recommendedPartSize. Use recommendedPartSize instead.
RecommendedPartSize int `json:"recommendedPartSize"` // The recommended size for each part of a large file. We recommend using this part size for optimal upload performance.
}
// AuthorizeAccountResponse is as returned from the b2_authorize_account call
type AuthorizeAccountResponse struct {
AccountID string `json:"accountId"` // The identifier for the account.
AuthorizationToken string `json:"authorizationToken"` // An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
APIs struct { // Supported APIs for this account / key. These are API-dependent JSON objects.
Storage StorageAPI `json:"storageApi"`
} `json:"apiInfo"`
}
// ListBucketsRequest is parameters for b2_list_buckets call
type ListBucketsRequest struct {
AccountID string `json:"accountId"` // The identifier for the account.
@@ -261,10 +278,11 @@ type GetFileInfoRequest struct {
//
// Example: { "src_last_modified_millis" : "1452802803026", "large_file_sha1" : "a3195dc1e7b46a2ff5da4b3c179175b75671e80d", "color": "blue" }
type StartLargeFileRequest struct {
BucketID string `json:"bucketId"` //The ID of the bucket that the file will go in.
Name string `json:"fileName"` // The name of the file. See Files for requirements on file names.
ContentType string `json:"contentType"` // The MIME type of the content of the file, which will be returned in the Content-Type header when downloading the file. Use the Content-Type b2/x-auto to automatically set the stored Content-Type post upload. In the case where a file extension is absent or the lookup fails, the Content-Type is set to application/octet-stream.
Info map[string]string `json:"fileInfo"` // A JSON object holding the name/value pairs for the custom file info.
BucketID string `json:"bucketId"` // The ID of the bucket that the file will go in.
Name string `json:"fileName"` // The name of the file. See Files for requirements on file names.
ContentType string `json:"contentType"` // The MIME type of the content of the file, which will be returned in the Content-Type header when downloading the file. Use the Content-Type b2/x-auto to automatically set the stored Content-Type post upload. In the case where a file extension is absent or the lookup fails, the Content-Type is set to application/octet-stream.
Info map[string]string `json:"fileInfo"` // A JSON object holding the name/value pairs for the custom file info.
ServerSideEncryption *ServerSideEncryption `json:"serverSideEncryption,omitempty"` // A JSON object holding values related to Server-Side Encryption
}
// StartLargeFileResponse is the response to StartLargeFileRequest
@@ -325,21 +343,25 @@ type CancelLargeFileResponse struct {
// CopyFileRequest is as passed to b2_copy_file
type CopyFileRequest struct {
SourceID string `json:"sourceFileId"` // The ID of the source file being copied.
Name string `json:"fileName"` // The name of the new file being created.
Range string `json:"range,omitempty"` // The range of bytes to copy. If not provided, the whole source file will be copied.
MetadataDirective string `json:"metadataDirective,omitempty"` // The strategy for how to populate metadata for the new file: COPY or REPLACE
ContentType string `json:"contentType,omitempty"` // The MIME type of the content of the file (REPLACE only)
Info map[string]string `json:"fileInfo,omitempty"` // This field stores the metadata that will be stored with the file. (REPLACE only)
DestBucketID string `json:"destinationBucketId,omitempty"` // The destination ID of the bucket if set, if not the source bucket will be used
SourceID string `json:"sourceFileId"` // The ID of the source file being copied.
Name string `json:"fileName"` // The name of the new file being created.
Range string `json:"range,omitempty"` // The range of bytes to copy. If not provided, the whole source file will be copied.
MetadataDirective string `json:"metadataDirective,omitempty"` // The strategy for how to populate metadata for the new file: COPY or REPLACE
ContentType string `json:"contentType,omitempty"` // The MIME type of the content of the file (REPLACE only)
Info map[string]string `json:"fileInfo,omitempty"` // This field stores the metadata that will be stored with the file. (REPLACE only)
DestBucketID string `json:"destinationBucketId,omitempty"` // The destination ID of the bucket if set, if not the source bucket will be used
SourceServerSideEncryption *ServerSideEncryption `json:"sourceServerSideEncryption,omitempty"` // A JSON object holding values related to Server-Side Encryption for the source file
DestinationServerSideEncryption *ServerSideEncryption `json:"destinationServerSideEncryption,omitempty"` // A JSON object holding values related to Server-Side Encryption for the destination file
}
// CopyPartRequest is the request for b2_copy_part - the response is UploadPartResponse
type CopyPartRequest struct {
SourceID string `json:"sourceFileId"` // The ID of the source file being copied.
LargeFileID string `json:"largeFileId"` // The ID of the large file the part will belong to, as returned by b2_start_large_file.
PartNumber int64 `json:"partNumber"` // Which part this is (starting from 1)
Range string `json:"range,omitempty"` // The range of bytes to copy. If not provided, the whole source file will be copied.
SourceID string `json:"sourceFileId"` // The ID of the source file being copied.
LargeFileID string `json:"largeFileId"` // The ID of the large file the part will belong to, as returned by b2_start_large_file.
PartNumber int64 `json:"partNumber"` // Which part this is (starting from 1)
Range string `json:"range,omitempty"` // The range of bytes to copy. If not provided, the whole source file will be copied.
SourceServerSideEncryption *ServerSideEncryption `json:"sourceServerSideEncryption,omitempty"` // A JSON object holding values related to Server-Side Encryption for the source file
DestinationServerSideEncryption *ServerSideEncryption `json:"destinationServerSideEncryption,omitempty"` // A JSON object holding values related to Server-Side Encryption for the destination file
}
// UpdateBucketRequest describes a request to modify a B2 bucket
+276 -83
View File
@@ -8,7 +8,9 @@ import (
"bufio"
"bytes"
"context"
"crypto/md5"
"crypto/sha1"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
@@ -53,6 +55,9 @@ const (
nameHeader = "X-Bz-File-Name"
timestampHeader = "X-Bz-Upload-Timestamp"
retryAfterHeader = "Retry-After"
sseAlgorithmHeader = "X-Bz-Server-Side-Encryption-Customer-Algorithm"
sseKeyHeader = "X-Bz-Server-Side-Encryption-Customer-Key"
sseMd5Header = "X-Bz-Server-Side-Encryption-Customer-Key-Md5"
minSleep = 10 * time.Millisecond
maxSleep = 5 * time.Minute
decayConstant = 1 // bigger for slower decay, exponential
@@ -67,7 +72,7 @@ const (
// Globals
var (
errNotWithVersions = errors.New("can't modify or delete files in --b2-versions mode")
errNotWithVersions = errors.New("can't modify files in --b2-versions mode")
errNotWithVersionAt = errors.New("can't modify or delete files in --b2-version-at mode")
)
@@ -252,6 +257,51 @@ See: [rclone backend lifecycle](#lifecycle) for setting lifecycles after bucket
Default: (encoder.Display |
encoder.EncodeBackSlash |
encoder.EncodeInvalidUtf8),
}, {
Name: "sse_customer_algorithm",
Help: "If using SSE-C, the server-side encryption algorithm used when storing this object in B2.",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}, {
Value: "AES256",
Help: "Advanced Encryption Standard (256 bits key length)",
}},
}, {
Name: "sse_customer_key",
Help: `To use SSE-C, you may provide the secret encryption key encoded in a UTF-8 compatible string to encrypt/decrypt your data
Alternatively you can provide --sse-customer-key-base64.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}},
Sensitive: true,
}, {
Name: "sse_customer_key_base64",
Help: `To use SSE-C, you may provide the secret encryption key encoded in Base64 format to encrypt/decrypt your data
Alternatively you can provide --sse-customer-key.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}},
Sensitive: true,
}, {
Name: "sse_customer_key_md5",
Help: `If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
If you leave it blank, this is calculated automatically from the sse_customer_key provided.
`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}},
Sensitive: true,
}},
})
}
@@ -274,6 +324,10 @@ type Options struct {
DownloadAuthorizationDuration fs.Duration `config:"download_auth_duration"`
Lifecycle int `config:"lifecycle"`
Enc encoder.MultiEncoder `config:"encoding"`
SSECustomerAlgorithm string `config:"sse_customer_algorithm"`
SSECustomerKey string `config:"sse_customer_key"`
SSECustomerKeyBase64 string `config:"sse_customer_key_base64"`
SSECustomerKeyMD5 string `config:"sse_customer_key_md5"`
}
// Fs represents a remote b2 server
@@ -504,6 +558,24 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
if opt.Endpoint == "" {
opt.Endpoint = defaultEndpoint
}
if opt.SSECustomerKey != "" && opt.SSECustomerKeyBase64 != "" {
return nil, errors.New("b2: can't use both sse_customer_key and sse_customer_key_base64 at the same time")
} else if opt.SSECustomerKeyBase64 != "" {
// Decode the Base64-encoded key and store it in the SSECustomerKey field
decoded, err := base64.StdEncoding.DecodeString(opt.SSECustomerKeyBase64)
if err != nil {
return nil, fmt.Errorf("b2: Could not decode sse_customer_key_base64: %w", err)
}
opt.SSECustomerKey = string(decoded)
} else {
// Encode the raw key as Base64
opt.SSECustomerKeyBase64 = base64.StdEncoding.EncodeToString([]byte(opt.SSECustomerKey))
}
if opt.SSECustomerKey != "" && opt.SSECustomerKeyMD5 == "" {
// Calculate CustomerKeyMd5 if not supplied
md5sumBinary := md5.Sum([]byte(opt.SSECustomerKey))
opt.SSECustomerKeyMD5 = base64.StdEncoding.EncodeToString(md5sumBinary[:])
}
ci := fs.GetConfig(ctx)
f := &Fs{
name: name,
@@ -535,17 +607,29 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
if err != nil {
return nil, fmt.Errorf("failed to authorize account: %w", err)
}
// If this is a key limited to a single bucket, it must exist already
if f.rootBucket != "" && f.info.Allowed.BucketID != "" {
allowedBucket := f.opt.Enc.ToStandardName(f.info.Allowed.BucketName)
if allowedBucket == "" {
return nil, errors.New("bucket that application key is restricted to no longer exists")
// If this is a key limited to one or more buckets, one of them must exist
// and be ours.
if f.rootBucket != "" && len(f.info.APIs.Storage.Allowed.Buckets) != 0 {
buckets := f.info.APIs.Storage.Allowed.Buckets
var rootFound = false
var rootID string
for _, b := range buckets {
allowedBucket := f.opt.Enc.ToStandardName(b.Name)
if allowedBucket == "" {
fs.Debugf(f, "bucket %q that application key is restricted to no longer exists", b.ID)
continue
}
if allowedBucket == f.rootBucket {
rootFound = true
rootID = b.ID
}
}
if allowedBucket != f.rootBucket {
return nil, fmt.Errorf("you must use bucket %q with this application key", allowedBucket)
if !rootFound {
return nil, fmt.Errorf("you must use bucket(s) %q with this application key", buckets)
}
f.cache.MarkOK(f.rootBucket)
f.setBucketID(f.rootBucket, f.info.Allowed.BucketID)
f.setBucketID(f.rootBucket, rootID)
}
if f.rootBucket != "" && f.rootDirectory != "" {
// Check to see if the (bucket,directory) is actually an existing file
@@ -571,7 +655,7 @@ func (f *Fs) authorizeAccount(ctx context.Context) error {
defer f.authMu.Unlock()
opts := rest.Opts{
Method: "GET",
Path: "/b2api/v1/b2_authorize_account",
Path: "/b2api/v4/b2_authorize_account",
RootURL: f.opt.Endpoint,
UserName: f.opt.Account,
Password: f.opt.Key,
@@ -584,13 +668,13 @@ func (f *Fs) authorizeAccount(ctx context.Context) error {
if err != nil {
return fmt.Errorf("failed to authenticate: %w", err)
}
f.srv.SetRoot(f.info.APIURL+"/b2api/v1").SetHeader("Authorization", f.info.AuthorizationToken)
f.srv.SetRoot(f.info.APIs.Storage.APIURL+"/b2api/v1").SetHeader("Authorization", f.info.AuthorizationToken)
return nil
}
// hasPermission returns if the current AuthorizationToken has the selected permission
func (f *Fs) hasPermission(permission string) bool {
return slices.Contains(f.info.Allowed.Capabilities, permission)
return slices.Contains(f.info.APIs.Storage.Allowed.Capabilities, permission)
}
// getUploadURL returns the upload info with the UploadURL and the AuthorizationToken
@@ -847,7 +931,7 @@ func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *api.File
}
// listDir lists a single directory
func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addBucket bool) (entries fs.DirEntries, err error) {
func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addBucket bool, callback func(fs.DirEntry) error) (err error) {
last := ""
err = f.list(ctx, bucket, directory, prefix, f.rootBucket == "", false, 0, f.opt.Versions, false, func(remote string, object *api.File, isDirectory bool) error {
entry, err := f.itemToDirEntry(ctx, remote, object, isDirectory, &last)
@@ -855,16 +939,16 @@ func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addB
return err
}
if entry != nil {
entries = append(entries, entry)
return callback(entry)
}
return nil
})
if err != nil {
return nil, err
return err
}
// bucket must be present if listing succeeded
f.cache.MarkOK(bucket)
return entries, nil
return nil
}
// listBuckets returns all the buckets to out
@@ -890,14 +974,46 @@ func (f *Fs) listBuckets(ctx context.Context) (entries fs.DirEntries, err error)
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
return list.WithListP(ctx, dir, f)
}
// ListP lists the objects and directories of the Fs starting
// from dir non recursively into out.
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) error {
list := list.NewHelper(callback)
bucket, directory := f.split(dir)
if bucket == "" {
if directory != "" {
return nil, fs.ErrorListBucketRequired
return fs.ErrorListBucketRequired
}
entries, err := f.listBuckets(ctx)
if err != nil {
return err
}
for _, entry := range entries {
err = list.Add(entry)
if err != nil {
return err
}
}
} else {
err := f.listDir(ctx, bucket, directory, f.rootDirectory, f.rootBucket == "", list.Add)
if err != nil {
return err
}
return f.listBuckets(ctx)
}
return f.listDir(ctx, bucket, directory, f.rootDirectory, f.rootBucket == "")
return list.Flush()
}
// ListR lists the objects and directories of the Fs starting
@@ -963,44 +1079,83 @@ type listBucketFn func(*api.Bucket) error
// listBucketsToFn lists the buckets to the function supplied
func (f *Fs) listBucketsToFn(ctx context.Context, bucketName string, fn listBucketFn) error {
var account = api.ListBucketsRequest{
AccountID: f.info.AccountID,
BucketID: f.info.Allowed.BucketID,
}
if bucketName != "" && account.BucketID == "" {
account.BucketName = f.opt.Enc.FromStandardName(bucketName)
responses := make([]api.ListBucketsResponse, len(f.info.APIs.Storage.Allowed.Buckets))[:0]
call := func(id string) error {
var account = api.ListBucketsRequest{
AccountID: f.info.AccountID,
BucketID: id,
}
if bucketName != "" && account.BucketID == "" {
account.BucketName = f.opt.Enc.FromStandardName(bucketName)
}
var response api.ListBucketsResponse
opts := rest.Opts{
Method: "POST",
Path: "/b2_list_buckets",
}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(ctx, &opts, &account, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
}
responses = append(responses, response)
return nil
}
var response api.ListBucketsResponse
opts := rest.Opts{
Method: "POST",
Path: "/b2_list_buckets",
for i := range f.info.APIs.Storage.Allowed.Buckets {
b := &f.info.APIs.Storage.Allowed.Buckets[i]
// Empty names indicate a bucket that no longer exists, this is non-fatal
// for multi-bucket API keys.
if b.Name == "" {
continue
}
// When requesting a specific bucket skip over non-matching names
if bucketName != "" && b.Name != bucketName {
continue
}
err := call(b.ID)
if err != nil {
return err
}
}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(ctx, &opts, &account, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
if len(f.info.APIs.Storage.Allowed.Buckets) == 0 {
err := call("")
if err != nil {
return err
}
}
f.bucketIDMutex.Lock()
f.bucketTypeMutex.Lock()
f._bucketID = make(map[string]string, 1)
f._bucketType = make(map[string]string, 1)
for i := range response.Buckets {
bucket := &response.Buckets[i]
bucket.Name = f.opt.Enc.ToStandardName(bucket.Name)
f.cache.MarkOK(bucket.Name)
f._bucketID[bucket.Name] = bucket.ID
f._bucketType[bucket.Name] = bucket.Type
for ri := range responses {
response := &responses[ri]
for i := range response.Buckets {
bucket := &response.Buckets[i]
bucket.Name = f.opt.Enc.ToStandardName(bucket.Name)
f.cache.MarkOK(bucket.Name)
f._bucketID[bucket.Name] = bucket.ID
f._bucketType[bucket.Name] = bucket.Type
}
}
f.bucketTypeMutex.Unlock()
f.bucketIDMutex.Unlock()
for i := range response.Buckets {
bucket := &response.Buckets[i]
err = fn(bucket)
if err != nil {
return err
for ri := range responses {
response := &responses[ri]
for i := range response.Buckets {
bucket := &response.Buckets[i]
err := fn(bucket)
if err != nil {
return err
}
}
}
return nil
@@ -1403,6 +1558,16 @@ func (f *Fs) copy(ctx context.Context, dstObj *Object, srcObj *Object, newInfo *
Name: f.opt.Enc.FromStandardPath(dstPath),
DestBucketID: destBucketID,
}
if f.opt.SSECustomerKey != "" && f.opt.SSECustomerKeyMD5 != "" {
serverSideEncryptionConfig := api.ServerSideEncryption{
Mode: "SSE-C",
Algorithm: f.opt.SSECustomerAlgorithm,
CustomerKey: f.opt.SSECustomerKeyBase64,
CustomerKeyMd5: f.opt.SSECustomerKeyMD5,
}
request.SourceServerSideEncryption = &serverSideEncryptionConfig
request.DestinationServerSideEncryption = &serverSideEncryptionConfig
}
if newInfo == nil {
request.MetadataDirective = "COPY"
} else {
@@ -1492,7 +1657,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
bucket, bucketPath := f.split(remote)
var RootURL string
if f.opt.DownloadURL == "" {
RootURL = f.info.DownloadURL
RootURL = f.info.APIs.Storage.DownloadURL
} else {
RootURL = f.opt.DownloadURL
}
@@ -1834,15 +1999,16 @@ var _ io.ReadCloser = &openFile{}
func (o *Object) getOrHead(ctx context.Context, method string, options []fs.OpenOption) (resp *http.Response, info *api.File, err error) {
opts := rest.Opts{
Method: method,
Options: options,
NoResponse: method == "HEAD",
Method: method,
Options: options,
NoResponse: method == "HEAD",
ExtraHeaders: map[string]string{},
}
// Use downloadUrl from backblaze if downloadUrl is not set
// otherwise use the custom downloadUrl
if o.fs.opt.DownloadURL == "" {
opts.RootURL = o.fs.info.DownloadURL
opts.RootURL = o.fs.info.APIs.Storage.DownloadURL
} else {
opts.RootURL = o.fs.opt.DownloadURL
}
@@ -1854,6 +2020,11 @@ func (o *Object) getOrHead(ctx context.Context, method string, options []fs.Open
bucket, bucketPath := o.split()
opts.Path += "/file/" + urlEncode(o.fs.opt.Enc.FromStandardName(bucket)) + "/" + urlEncode(o.fs.opt.Enc.FromStandardPath(bucketPath))
}
if o.fs.opt.SSECustomerKey != "" && o.fs.opt.SSECustomerKeyMD5 != "" {
opts.ExtraHeaders[sseAlgorithmHeader] = o.fs.opt.SSECustomerAlgorithm
opts.ExtraHeaders[sseKeyHeader] = o.fs.opt.SSECustomerKeyBase64
opts.ExtraHeaders[sseMd5Header] = o.fs.opt.SSECustomerKeyMD5
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(ctx, resp, err)
@@ -2118,6 +2289,11 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
},
ContentLength: &size,
}
if o.fs.opt.SSECustomerKey != "" && o.fs.opt.SSECustomerKeyMD5 != "" {
opts.ExtraHeaders[sseAlgorithmHeader] = o.fs.opt.SSECustomerAlgorithm
opts.ExtraHeaders[sseKeyHeader] = o.fs.opt.SSECustomerKeyBase64
opts.ExtraHeaders[sseMd5Header] = o.fs.opt.SSECustomerKeyMD5
}
var response api.FileInfo
// Don't retry, return a retry error instead
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
@@ -2192,20 +2368,27 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
return info, nil, err
}
up, err := f.newLargeUpload(ctx, o, nil, src, f.opt.ChunkSize, false, nil, options...)
if err != nil {
return info, nil, err
}
info = fs.ChunkWriterInfo{
ChunkSize: int64(f.opt.ChunkSize),
ChunkSize: up.chunkSize,
Concurrency: o.fs.opt.UploadConcurrency,
//LeavePartsOnError: o.fs.opt.LeavePartsOnError,
}
up, err := f.newLargeUpload(ctx, o, nil, src, f.opt.ChunkSize, false, nil, options...)
return info, up, err
return info, up, nil
}
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
bucket, bucketPath := o.split()
if o.fs.opt.Versions {
return errNotWithVersions
t, path := api.RemoveVersion(bucketPath)
if !t.IsZero() {
return o.fs.deleteByID(ctx, o.id, path)
}
}
if o.fs.opt.VersionAt.IsSet() {
return errNotWithVersionAt
@@ -2228,32 +2411,36 @@ func (o *Object) ID() string {
var lifecycleHelp = fs.CommandHelp{
Name: "lifecycle",
Short: "Read or set the lifecycle for a bucket",
Short: "Read or set the lifecycle for a bucket.",
Long: `This command can be used to read or set the lifecycle for a bucket.
Usage Examples:
To show the current lifecycle rules:
rclone backend lifecycle b2:bucket
` + "```console" + `
rclone backend lifecycle b2:bucket
` + "```" + `
This will dump something like this showing the lifecycle rules.
[
{
"daysFromHidingToDeleting": 1,
"daysFromUploadingToHiding": null,
"daysFromStartingToCancelingUnfinishedLargeFiles": null,
"fileNamePrefix": ""
}
]
` + "```json" + `
[
{
"daysFromHidingToDeleting": 1,
"daysFromUploadingToHiding": null,
"daysFromStartingToCancelingUnfinishedLargeFiles": null,
"fileNamePrefix": ""
}
]
` + "```" + `
If there are no lifecycle rules (the default) then it will just return [].
If there are no lifecycle rules (the default) then it will just return ` + "`[]`" + `.
To reset the current lifecycle rules:
rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=30
rclone backend lifecycle b2:bucket -o daysFromUploadingToHiding=5 -o daysFromHidingToDeleting=1
` + "```console" + `
rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=30
rclone backend lifecycle b2:bucket -o daysFromUploadingToHiding=5 -o daysFromHidingToDeleting=1
` + "```" + `
This will run and then print the new lifecycle rules as above.
@@ -2265,14 +2452,17 @@ the daysFromHidingToDeleting to 1 day. You can enable hard_delete in
the config also which will mean deletions won't cause versions but
overwrites will still cause versions to be made.
rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=1
` + "```console" + `
rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=1
` + "```" + `
See: https://www.backblaze.com/docs/cloud-storage-lifecycle-rules
`,
See: <https://www.backblaze.com/docs/cloud-storage-lifecycle-rules>`,
Opts: map[string]string{
"daysFromHidingToDeleting": "After a file has been hidden for this many days it is deleted. 0 is off.",
"daysFromUploadingToHiding": "This many days after uploading a file is hidden",
"daysFromStartingToCancelingUnfinishedLargeFiles": "Cancels any unfinished large file versions after this many days",
"daysFromHidingToDeleting": `After a file has been hidden for this many days
it is deleted. 0 is off.`,
"daysFromUploadingToHiding": `This many days after uploading a file is hidden.`,
"daysFromStartingToCancelingUnfinishedLargeFiles": `Cancels any unfinished
large file versions after this many days.`,
},
}
@@ -2355,13 +2545,14 @@ max-age, which defaults to 24 hours.
Note that you can use --interactive/-i or --dry-run with this command to see what
it would do.
rclone backend cleanup b2:bucket/path/to/object
rclone backend cleanup -o max-age=7w b2:bucket/path/to/object
` + "```console" + `
rclone backend cleanup b2:bucket/path/to/object
rclone backend cleanup -o max-age=7w b2:bucket/path/to/object
` + "```" + `
Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.
`,
Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.`,
Opts: map[string]string{
"max-age": "Max age of upload to delete",
"max-age": "Max age of upload to delete.",
},
}
@@ -2384,8 +2575,9 @@ var cleanupHiddenHelp = fs.CommandHelp{
Note that you can use --interactive/-i or --dry-run with this command to see what
it would do.
rclone backend cleanup-hidden b2:bucket/path/to/dir
`,
` + "```console" + `
rclone backend cleanup-hidden b2:bucket/path/to/dir
` + "```",
}
func (f *Fs) cleanupHiddenCommand(ctx context.Context, name string, arg []string, opt map[string]string) (out any, err error) {
@@ -2428,6 +2620,7 @@ var (
_ fs.PutStreamer = &Fs{}
_ fs.CleanUpper = &Fs{}
_ fs.ListRer = &Fs{}
_ fs.ListPer = &Fs{}
_ fs.PublicLinker = &Fs{}
_ fs.OpenChunkWriter = &Fs{}
_ fs.Commander = &Fs{}
+33 -1
View File
@@ -22,6 +22,7 @@ import (
"github.com/rclone/rclone/lib/atexit"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/rest"
"github.com/rclone/rclone/lib/transferaccounter"
"golang.org/x/sync/errgroup"
)
@@ -144,6 +145,14 @@ func (f *Fs) newLargeUpload(ctx context.Context, o *Object, in io.Reader, src fs
request.ContentType = newInfo.ContentType
request.Info = newInfo.Info
}
if o.fs.opt.SSECustomerKey != "" && o.fs.opt.SSECustomerKeyMD5 != "" {
request.ServerSideEncryption = &api.ServerSideEncryption{
Mode: "SSE-C",
Algorithm: o.fs.opt.SSECustomerAlgorithm,
CustomerKey: o.fs.opt.SSECustomerKeyBase64,
CustomerKeyMd5: o.fs.opt.SSECustomerKeyMD5,
}
}
opts := rest.Opts{
Method: "POST",
Path: "/b2_start_large_file",
@@ -295,6 +304,12 @@ func (up *largeUpload) WriteChunk(ctx context.Context, chunkNumber int, reader i
ContentLength: &sizeWithHash,
}
if up.o.fs.opt.SSECustomerKey != "" && up.o.fs.opt.SSECustomerKeyMD5 != "" {
opts.ExtraHeaders[sseAlgorithmHeader] = up.o.fs.opt.SSECustomerAlgorithm
opts.ExtraHeaders[sseKeyHeader] = up.o.fs.opt.SSECustomerKeyBase64
opts.ExtraHeaders[sseMd5Header] = up.o.fs.opt.SSECustomerKeyMD5
}
var response api.UploadPartResponse
resp, err := up.f.srv.CallJSON(ctx, &opts, nil, &response)
@@ -334,6 +349,17 @@ func (up *largeUpload) copyChunk(ctx context.Context, part int, partSize int64)
PartNumber: int64(part + 1),
Range: fmt.Sprintf("bytes=%d-%d", offset, offset+partSize-1),
}
if up.o.fs.opt.SSECustomerKey != "" && up.o.fs.opt.SSECustomerKeyMD5 != "" {
serverSideEncryptionConfig := api.ServerSideEncryption{
Mode: "SSE-C",
Algorithm: up.o.fs.opt.SSECustomerAlgorithm,
CustomerKey: up.o.fs.opt.SSECustomerKeyBase64,
CustomerKeyMd5: up.o.fs.opt.SSECustomerKeyMD5,
}
request.SourceServerSideEncryption = &serverSideEncryptionConfig
request.DestinationServerSideEncryption = &serverSideEncryptionConfig
}
var response api.UploadPartResponse
resp, err := up.f.srv.CallJSON(ctx, &opts, &request, &response)
retry, err := up.f.shouldRetry(ctx, resp, err)
@@ -474,10 +500,12 @@ func (up *largeUpload) Copy(ctx context.Context) (err error) {
defer atexit.OnError(&err, func() { _ = up.Abort(ctx) })()
fs.Debugf(up.o, "Starting %s of large file in %d chunks (id %q)", up.what, up.parts, up.id)
var (
account = transferaccounter.Get(ctx)
g, gCtx = errgroup.WithContext(ctx)
remaining = up.size
)
g.SetLimit(up.f.opt.UploadConcurrency)
account.Start()
for part := range up.parts {
// Fail fast, in case an errgroup managed function returns an error
// gCtx is cancelled. There is no point in copying all the other parts.
@@ -489,7 +517,11 @@ func (up *largeUpload) Copy(ctx context.Context) (err error) {
part := part // for the closure
g.Go(func() (err error) {
return up.copyChunk(gCtx, part, reqSize)
err = up.copyChunk(gCtx, part, reqSize)
if err == nil {
account.Add(reqSize)
}
return err
})
remaining -= reqSize
}
+18 -7
View File
@@ -125,10 +125,21 @@ type FolderItems struct {
Offset int `json:"offset"`
Limit int `json:"limit"`
NextMarker *string `json:"next_marker,omitempty"`
Order []struct {
By string `json:"by"`
Direction string `json:"direction"`
} `json:"order"`
// There is some confusion about how this is actually
// returned. The []struct has worked for many years, but in
// https://github.com/rclone/rclone/issues/8776 box was
// returning it returned not as a list. We don't actually use
// this so comment it out.
//
// Order struct {
// By string `json:"by"`
// Direction string `json:"direction"`
// } `json:"order"`
//
// Order []struct {
// By string `json:"by"`
// Direction string `json:"direction"`
// } `json:"order"`
}
// Parent defined the ID of the parent directory
@@ -271,9 +282,9 @@ type User struct {
ModifiedAt time.Time `json:"modified_at"`
Language string `json:"language"`
Timezone string `json:"timezone"`
SpaceAmount int64 `json:"space_amount"`
SpaceUsed int64 `json:"space_used"`
MaxUploadSize int64 `json:"max_upload_size"`
SpaceAmount float64 `json:"space_amount"`
SpaceUsed float64 `json:"space_used"`
MaxUploadSize float64 `json:"max_upload_size"`
Status string `json:"status"`
JobTitle string `json:"job_title"`
Phone string `json:"phone"`
+66 -23
View File
@@ -37,6 +37,7 @@ import (
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/list"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/env"
@@ -86,13 +87,11 @@ func init() {
Description: "Box",
NewFs: NewFs,
Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
jsonFile, ok := m.Get("box_config_file")
boxSubType, boxSubTypeOk := m.Get("box_sub_type")
boxAccessToken, boxAccessTokenOk := m.Get("access_token")
var err error
// If using box config.json, use JWT auth
if ok && boxSubTypeOk && jsonFile != "" && boxSubType != "" {
err = refreshJWTToken(ctx, jsonFile, boxSubType, name, m)
if usesJWTAuth(m) {
err = refreshJWTToken(ctx, name, m)
if err != nil {
return nil, fmt.Errorf("failed to configure token with jwt authentication: %w", err)
}
@@ -113,6 +112,11 @@ func init() {
}, {
Name: "box_config_file",
Help: "Box App config.json location\n\nLeave blank normally." + env.ShellExpandHelp,
}, {
Name: "config_credentials",
Help: "Box App config.json contents.\n\nLeave blank normally.",
Hide: fs.OptionHideBoth,
Sensitive: true,
}, {
Name: "access_token",
Help: "Box App Primary Access Token\n\nLeave blank normally.",
@@ -183,9 +187,17 @@ See: https://developer.box.com/guides/authentication/jwt/as-user/
})
}
func refreshJWTToken(ctx context.Context, jsonFile string, boxSubType string, name string, m configmap.Mapper) error {
jsonFile = env.ShellExpand(jsonFile)
boxConfig, err := getBoxConfig(jsonFile)
func usesJWTAuth(m configmap.Mapper) bool {
jsonFile, okFile := m.Get("box_config_file")
jsonFileCredentials, okCredentials := m.Get("config_credentials")
boxSubType, boxSubTypeOk := m.Get("box_sub_type")
return (okFile || okCredentials) && boxSubTypeOk && (jsonFile != "" || jsonFileCredentials != "") && boxSubType != ""
}
func refreshJWTToken(ctx context.Context, name string, m configmap.Mapper) error {
boxSubType, _ := m.Get("box_sub_type")
boxConfig, err := getBoxConfig(m)
if err != nil {
return fmt.Errorf("get box config: %w", err)
}
@@ -204,12 +216,19 @@ func refreshJWTToken(ctx context.Context, jsonFile string, boxSubType string, na
return err
}
func getBoxConfig(configFile string) (boxConfig *api.ConfigJSON, err error) {
file, err := os.ReadFile(configFile)
if err != nil {
return nil, fmt.Errorf("box: failed to read Box config: %w", err)
func getBoxConfig(m configmap.Mapper) (boxConfig *api.ConfigJSON, err error) {
configFileCredentials, _ := m.Get("config_credentials")
configFileBytes := []byte(configFileCredentials)
if configFileCredentials == "" {
configFile, _ := m.Get("box_config_file")
configFileBytes, err = os.ReadFile(configFile)
if err != nil {
return nil, fmt.Errorf("box: failed to read Box config: %w", err)
}
}
err = json.Unmarshal(file, &boxConfig)
err = json.Unmarshal(configFileBytes, &boxConfig)
if err != nil {
return nil, fmt.Errorf("box: failed to parse Box config: %w", err)
}
@@ -484,15 +503,12 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
f.srv.SetHeader("as-user", f.opt.Impersonate)
}
jsonFile, ok := m.Get("box_config_file")
boxSubType, boxSubTypeOk := m.Get("box_sub_type")
if ts != nil {
// If using box config.json and JWT, renewing should just refresh the token and
// should do so whether there are uploads pending or not.
if ok && boxSubTypeOk && jsonFile != "" && boxSubType != "" {
if usesJWTAuth(m) {
f.tokenRenewer = oauthutil.NewRenew(f.String(), ts, func() error {
err := refreshJWTToken(ctx, jsonFile, boxSubType, name, m)
err := refreshJWTToken(ctx, name, m)
return err
})
f.tokenRenewer.Start()
@@ -705,9 +721,27 @@ OUTER:
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
return list.WithListP(ctx, dir, f)
}
// ListP lists the objects and directories of the Fs starting
// from dir non recursively into out.
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) error {
list := list.NewHelper(callback)
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return nil, err
return err
}
var iErr error
_, err = f.listAll(ctx, directoryID, false, false, true, func(info *api.Item) bool {
@@ -717,14 +751,22 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
f.dirCache.Put(remote, info.ID)
d := fs.NewDir(remote, info.ModTime()).SetID(info.ID)
// FIXME more info from dir?
entries = append(entries, d)
err = list.Add(d)
if err != nil {
iErr = err
return true
}
} else if info.Type == api.ItemTypeFile {
o, err := f.newObjectWithInfo(ctx, remote, info)
if err != nil {
iErr = err
return true
}
entries = append(entries, o)
err = list.Add(o)
if err != nil {
iErr = err
return true
}
}
// Cache some metadata for this Item to help us process events later
@@ -740,12 +782,12 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return false
})
if err != nil {
return nil, err
return err
}
if iErr != nil {
return nil, iErr
return iErr
}
return entries, nil
return list.Flush()
}
// Creates from the parameters passed in a half finished Object which
@@ -1741,6 +1783,7 @@ var (
_ fs.DirCacheFlusher = (*Fs)(nil)
_ fs.PublicLinker = (*Fs)(nil)
_ fs.CleanUpper = (*Fs)(nil)
_ fs.ListPer = (*Fs)(nil)
_ fs.Shutdowner = (*Fs)(nil)
_ fs.Object = (*Object)(nil)
_ fs.IDer = (*Object)(nil)
+11 -6
View File
@@ -503,15 +503,19 @@ func NewFs(ctx context.Context, name, rootPath string, m configmap.Mapper) (fs.F
}
go func() {
// Wait on the timer and the stop channel together so that a stop
// signalled by StopBackgroundRunners is honoured immediately.
timer := time.NewTimer(time.Duration(f.opt.ChunkCleanInterval))
defer timer.Stop()
for {
time.Sleep(time.Duration(f.opt.ChunkCleanInterval))
select {
case <-f.cleanupChan:
fs.Infof(f, "stopping cleanup")
return
default:
case <-timer.C:
fs.Debugf(f, "starting cleanup")
f.CleanUpCache(false)
timer.Reset(time.Duration(f.opt.ChunkCleanInterval))
}
}
}()
@@ -563,9 +567,10 @@ Eg
})
rc.Add(rc.Call{
Path: "cache/stats",
Fn: f.httpStats,
Title: "Get cache stats",
Path: "cache/stats",
NoAuth: true,
Fn: f.httpStats,
Title: "Get cache stats",
Help: `
Show statistics for the cache remote.
`,
@@ -684,7 +689,7 @@ func (f *Fs) rcFetch(ctx context.Context, in rc.Params) (rc.Params, error) {
start, end int64
}
parseChunks := func(ranges string) (crs []chunkRange, err error) {
for _, part := range strings.Split(ranges, ",") {
for part := range strings.SplitSeq(ranges, ",") {
var start, end int64 = 0, math.MaxInt64
switch ints := strings.Split(part, ":"); len(ints) {
case 1:
-1
View File
@@ -1,5 +1,4 @@
//go:build !plan9 && !js
// +build !plan9,!js
package cache
+11 -7
View File
@@ -34,6 +34,13 @@ import (
"github.com/zeebo/blake3"
)
// Pacer settings - tuned to match other rclone backends
const (
minSleep = 10 * time.Millisecond
maxSleep = 2 * time.Second
decayConstant = 2 // bigger for slower decay, exponential
)
// Cloudinary shouldn't have a trailing dot if there is no path
func cldPathDir(somePath string) string {
if somePath == "" || somePath == "." {
@@ -189,7 +196,7 @@ func NewFs(ctx context.Context, name string, root string, m configmap.Mapper) (f
root: root,
opt: *opt,
cld: cld,
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(1000), pacer.MaxSleep(10000), pacer.DecayConstant(2))),
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
srv: rest.NewClient(client),
}
@@ -586,21 +593,18 @@ func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, err
if fserrors.ContextError(ctx, &err) {
return false, err
}
// Cloudinary returns rate-limit windows as "Try again on YYYY-MM-DD HH:MM:SS UTC"
// embedded in the error message - honour it as a retry-after.
if err != nil {
tryAgain := "Try again on "
if idx := strings.Index(err.Error(), tryAgain); idx != -1 {
layout := "2006-01-02 15:04:05 UTC"
dateStr := err.Error()[idx+len(tryAgain) : idx+len(tryAgain)+len(layout)]
timestamp, err2 := time.Parse(layout, dateStr)
if err2 == nil {
if timestamp, err2 := time.Parse(layout, dateStr); err2 == nil {
return true, fserrors.NewErrorRetryAfter(time.Until(timestamp))
}
}
fs.Debugf(nil, "Retrying API error %v", err)
return true, err
}
return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
+6 -5
View File
@@ -187,7 +187,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (outFs fs
g, gCtx := errgroup.WithContext(ctx)
var mu sync.Mutex
for _, upstream := range opt.Upstreams {
upstream := upstream
g.Go(func() (err error) {
equal := strings.IndexRune(upstream, '=')
if equal < 0 {
@@ -241,18 +240,22 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (outFs fs
DirModTimeUpdatesOnWrite: true,
PartialUploads: true,
}).Fill(ctx, f)
canMove := true
canMove, slowHash := true, false
for _, u := range f.upstreams {
features = features.Mask(ctx, u.f) // Mask all upstream fs
if !operations.CanServerSideMove(u.f) {
canMove = false
}
slowHash = slowHash || u.f.Features().SlowHash
}
// We can move if all remotes support Move or Copy
if canMove {
features.Move = f.Move
}
// If any of upstreams are SlowHash, propagate it
features.SlowHash = slowHash
// Enable ListR when upstreams either support ListR or is local
// But not when all upstreams are local
if features.ListR == nil {
@@ -366,7 +369,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (outFs fs
func (f *Fs) multithread(ctx context.Context, fn func(context.Context, *upstream) error) error {
g, gCtx := errgroup.WithContext(ctx)
for _, u := range f.upstreams {
u := u
g.Go(func() (err error) {
return fn(gCtx, u)
})
@@ -633,7 +635,6 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
var uChans []chan time.Duration
for _, u := range f.upstreams {
u := u
if do := u.f.Features().ChangeNotify; do != nil {
ch := make(chan time.Duration)
uChans = append(uChans, ch)
@@ -1103,7 +1104,7 @@ func (o *Object) Remote() string {
newPath, err := o.u.pathAdjustment.do(o.Object.String())
if err != nil {
fs.Errorf(o.Object, "Bad object: %v", err)
return err.Error()
return ""
}
return newPath
}
+130 -188
View File
@@ -2,10 +2,8 @@
package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/base64"
"encoding/binary"
"encoding/hex"
@@ -46,6 +44,7 @@ const (
minCompressionRatio = 1.1
gzFileExt = ".gz"
zstdFileExt = ".zst"
metaFileExt = ".json"
uncompressedFileExt = ".bin"
)
@@ -54,6 +53,7 @@ const (
const (
Uncompressed = 0
Gzip = 2
Zstd = 4
)
var nameRegexp = regexp.MustCompile(`^(.+?)\.([A-Za-z0-9-_]{11})$`)
@@ -66,6 +66,10 @@ func init() {
Value: "gzip",
Help: "Standard gzip compression with fastest parameters.",
},
{
Value: "zstd",
Help: "Zstandard compression — fast modern algorithm offering adjustable speed-to-compression tradeoffs.",
},
}
// Register our remote
@@ -87,17 +91,23 @@ func init() {
Examples: compressionModeOptions,
}, {
Name: "level",
Help: `GZIP compression level (-2 to 9).
Generally -1 (default, equivalent to 5) is recommended.
Levels 1 to 9 increase compression at the cost of speed. Going past 6
generally offers very little return.
Level -2 uses Huffman encoding only. Only use if you know what you
are doing.
Level 0 turns off compression.`,
Default: sgzip.DefaultCompression,
Advanced: true,
Help: `GZIP (levels -2 to 9):
- -2 — Huffman encoding only. Only use if you know what you're doing.
- -1 (default) — recommended; equivalent to level 5.
- 0 — turns off compression.
- 1–9 — increase compression at the cost of speed. Going past 6 generally offers very little return.
ZSTD (levels 0 to 4):
- 0 — turns off compression entirely.
- 1 — fastest compression with the lowest ratio.
- 2 (default) — good balance of speed and compression.
- 3 — better compression, but uses about 2–3x more CPU than the default.
- 4 — best possible compression ratio (highest CPU cost).
Notes:
- Choose GZIP for wide compatibility; ZSTD for better speed/ratio tradeoffs.
- Negative gzip levels: -2 = Huffman-only, -1 = default (≈ level 5).`,
Required: true,
}, {
Name: "ram_cache_limit",
Help: `Some remotes don't allow the upload of files with unknown size.
@@ -112,6 +122,47 @@ this limit will be cached on disk.`,
})
}
// compressionModeHandler defines the interface for handling different compression modes
type compressionModeHandler interface {
// processFileNameGetFileExtension returns the file extension for the given compression mode
processFileNameGetFileExtension(compressionMode int) string
// newObjectGetOriginalSize returns the original file size from the metadata
newObjectGetOriginalSize(meta *ObjectMetadata) (int64, error)
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
isCompressible(r io.Reader, compressionMode int) (bool, error)
// putCompress compresses the input data and uploads it to the remote, returning the new object and its metadata
putCompress(
ctx context.Context,
f *Fs,
in io.Reader,
src fs.ObjectInfo,
options []fs.OpenOption,
mimeType string,
) (fs.Object, *ObjectMetadata, error)
// openGetReadCloser opens a compressed object and returns a ReadCloser in the Open method
openGetReadCloser(
ctx context.Context,
o *Object,
offset int64,
limit int64,
cr chunkedreader.ChunkedReader,
closer io.Closer,
options ...fs.OpenOption,
) (rc io.ReadCloser, err error)
// putUncompressGetNewMetadata returns metadata in the putUncompress method for a specific compression algorithm
putUncompressGetNewMetadata(o fs.Object, mode int, md5 string, mimeType string, sum []byte) (fs.Object, *ObjectMetadata, error)
// This function generates a metadata object for sgzip.GzipMetadata or SzstdMetadata.
// Warning: This function panics if cmeta is not of the expected type.
newMetadata(size int64, mode int, cmeta any, md5 string, mimeType string) *ObjectMetadata
}
// Options defines the configuration for this backend
type Options struct {
Remote string `config:"remote"`
@@ -125,12 +176,13 @@ type Options struct {
// Fs represents a wrapped fs.Fs
type Fs struct {
fs.Fs
wrapper fs.Fs
name string
root string
opt Options
mode int // compression mode id
features *fs.Features // optional features
wrapper fs.Fs
name string
root string
opt Options
mode int // compression mode id
features *fs.Features // optional features
modeHandler compressionModeHandler // compression mode handler
}
// NewFs constructs an Fs from the path, container:path
@@ -167,13 +219,28 @@ func NewFs(ctx context.Context, name, rpath string, m configmap.Mapper) (fs.Fs,
return nil, fmt.Errorf("failed to make remote %s:%q to wrap: %w", wName, remotePath, err)
}
compressionMode := compressionModeFromName(opt.CompressionMode)
var modeHandler compressionModeHandler
switch compressionMode {
case Gzip:
modeHandler = &gzipModeHandler{}
case Zstd:
modeHandler = &zstdModeHandler{}
case Uncompressed:
modeHandler = &uncompressedModeHandler{}
default:
modeHandler = &unknownModeHandler{}
}
// Create the wrapping fs
f := &Fs{
Fs: wrappedFs,
name: name,
root: rpath,
opt: *opt,
mode: compressionModeFromName(opt.CompressionMode),
Fs: wrappedFs,
name: name,
root: rpath,
opt: *opt,
mode: compressionMode,
modeHandler: modeHandler,
}
// Correct root if definitely pointing to a file
if err == fs.ErrorIsFile {
@@ -215,10 +282,13 @@ func NewFs(ctx context.Context, name, rpath string, m configmap.Mapper) (fs.Fs,
return f, err
}
// compressionModeFromName converts a compression mode name to its int representation.
func compressionModeFromName(name string) int {
switch name {
case "gzip":
return Gzip
case "zstd":
return Zstd
default:
return Uncompressed
}
@@ -242,7 +312,7 @@ func base64ToInt64(str string) (int64, error) {
// Processes a file name for a compressed file. Returns the original file name, the extension, and the size of the original file.
// Returns -2 for the original size if the file is uncompressed.
func processFileName(compressedFileName string) (origFileName string, extension string, origSize int64, err error) {
func processFileName(compressedFileName string, modeHandler compressionModeHandler) (origFileName string, extension string, origSize int64, err error) {
// Separate the filename and size from the extension
extensionPos := strings.LastIndex(compressedFileName, ".")
if extensionPos == -1 {
@@ -261,7 +331,8 @@ func processFileName(compressedFileName string) (origFileName string, extension
if err != nil {
return "", "", 0, errors.New("could not decode size")
}
return match[1], gzFileExt, size, nil
ext := modeHandler.processFileNameGetFileExtension(compressionModeFromName(compressedFileName[extensionPos+1:]))
return match[1], ext, size, nil
}
// Generates the file name for a metadata file
@@ -286,11 +357,15 @@ func unwrapMetadataFile(filename string) (string, bool) {
// makeDataName generates the file name for a data file with specified compression mode
func makeDataName(remote string, size int64, mode int) (newRemote string) {
if mode != Uncompressed {
switch mode {
case Gzip:
newRemote = remote + "." + int64ToBase64(size) + gzFileExt
} else {
case Zstd:
newRemote = remote + "." + int64ToBase64(size) + zstdFileExt
default:
newRemote = remote + uncompressedFileExt
}
return newRemote
}
@@ -304,7 +379,7 @@ func (f *Fs) dataName(remote string, size int64, compressed bool) (name string)
// addData parses an object and adds it to the DirEntries
func (f *Fs) addData(entries *fs.DirEntries, o fs.Object) {
origFileName, _, size, err := processFileName(o.Remote())
origFileName, _, size, err := processFileName(o.Remote(), f.modeHandler)
if err != nil {
fs.Errorf(o, "Error on parsing file name: %v", err)
return
@@ -427,8 +502,12 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
if err != nil {
return nil, fmt.Errorf("error decoding metadata: %w", err)
}
size, err := f.modeHandler.newObjectGetOriginalSize(meta)
if err != nil {
return nil, fmt.Errorf("error reading metadata: %w", err)
}
// Create our Object
o, err := f.Fs.NewObject(ctx, makeDataName(remote, meta.CompressionMetadata.Size, meta.Mode))
o, err := f.Fs.NewObject(ctx, makeDataName(remote, size, meta.Mode))
if err != nil {
return nil, err
}
@@ -437,7 +516,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
// checkCompressAndType checks if an object is compressible and determines it's mime type
// returns a multireader with the bytes that were read to determine mime type
func checkCompressAndType(in io.Reader) (newReader io.Reader, compressible bool, mimeType string, err error) {
func checkCompressAndType(in io.Reader, compressionMode int, modeHandler compressionModeHandler) (newReader io.Reader, compressible bool, mimeType string, err error) {
in, wrap := accounting.UnWrap(in)
buf := make([]byte, heuristicBytes)
n, err := in.Read(buf)
@@ -446,7 +525,7 @@ func checkCompressAndType(in io.Reader) (newReader io.Reader, compressible bool,
return nil, false, "", err
}
mime := mimetype.Detect(buf)
compressible, err = isCompressible(bytes.NewReader(buf))
compressible, err = modeHandler.isCompressible(bytes.NewReader(buf), compressionMode)
if err != nil {
return nil, false, "", err
}
@@ -454,26 +533,6 @@ func checkCompressAndType(in io.Reader) (newReader io.Reader, compressible bool,
return wrap(in), compressible, mime.String(), nil
}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func isCompressible(r io.Reader) (bool, error) {
var b bytes.Buffer
w, err := sgzip.NewWriterLevel(&b, sgzip.DefaultCompression)
if err != nil {
return false, err
}
n, err := io.Copy(w, r)
if err != nil {
return false, err
}
err = w.Close()
if err != nil {
return false, err
}
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
// verifyObjectHash verifies the Objects hash
func (f *Fs) verifyObjectHash(ctx context.Context, o fs.Object, hasher *hash.MultiHasher, ht hash.Type) error {
srcHash := hasher.Sums()[ht]
@@ -494,9 +553,9 @@ func (f *Fs) verifyObjectHash(ctx context.Context, o fs.Object, hasher *hash.Mul
type putFn func(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error)
type compressionResult struct {
type compressionResult[T sgzip.GzipMetadata | SzstdMetadata] struct {
err error
meta sgzip.GzipMetadata
meta T
}
// replicating some of operations.Rcat functionality because we want to support remotes without streaming
@@ -537,106 +596,18 @@ func (f *Fs) rcat(ctx context.Context, dstFileName string, in io.ReadCloser, mod
return nil, fmt.Errorf("failed to write temporary local file: %w", err)
}
if _, err = tempFile.Seek(0, 0); err != nil {
return nil, err
return nil, fmt.Errorf("failed to seek temporary local file: %w", err)
}
finfo, err := tempFile.Stat()
if err != nil {
return nil, err
return nil, fmt.Errorf("failed to stat temporary local file: %w", err)
}
return f.Fs.Put(ctx, tempFile, object.NewStaticObjectInfo(dstFileName, modTime, finfo.Size(), false, nil, f.Fs))
}
// Put a compressed version of a file. Returns a wrappable object and metadata.
func (f *Fs) putCompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, options []fs.OpenOption, mimeType string) (fs.Object, *ObjectMetadata, error) {
// Unwrap reader accounting
in, wrap := accounting.UnWrap(in)
// Add the metadata hasher
metaHasher := md5.New()
in = io.TeeReader(in, metaHasher)
// Compress the file
pipeReader, pipeWriter := io.Pipe()
results := make(chan compressionResult)
go func() {
gz, err := sgzip.NewWriterLevel(pipeWriter, f.opt.CompressionLevel)
if err != nil {
results <- compressionResult{err: err, meta: sgzip.GzipMetadata{}}
return
}
_, err = io.Copy(gz, in)
gzErr := gz.Close()
if gzErr != nil {
fs.Errorf(nil, "Failed to close compress: %v", gzErr)
if err == nil {
err = gzErr
}
}
closeErr := pipeWriter.Close()
if closeErr != nil {
fs.Errorf(nil, "Failed to close pipe: %v", closeErr)
if err == nil {
err = closeErr
}
}
results <- compressionResult{err: err, meta: gz.MetaData()}
}()
wrappedIn := wrap(bufio.NewReaderSize(pipeReader, bufferSize)) // Probably no longer needed as sgzip has it's own buffering
// Find a hash the destination supports to compute a hash of
// the compressed data.
ht := f.Fs.Hashes().GetOne()
var hasher *hash.MultiHasher
var err error
if ht != hash.None {
// unwrap the accounting again
wrappedIn, wrap = accounting.UnWrap(wrappedIn)
hasher, err = hash.NewMultiHasherTypes(hash.NewHashSet(ht))
if err != nil {
return nil, nil, err
}
// add the hasher and re-wrap the accounting
wrappedIn = io.TeeReader(wrappedIn, hasher)
wrappedIn = wrap(wrappedIn)
}
// Transfer the data
o, err := f.rcat(ctx, makeDataName(src.Remote(), src.Size(), f.mode), io.NopCloser(wrappedIn), src.ModTime(ctx), options)
//o, err := operations.Rcat(ctx, f.Fs, makeDataName(src.Remote(), src.Size(), f.mode), io.NopCloser(wrappedIn), src.ModTime(ctx))
if err != nil {
if o != nil {
removeErr := o.Remove(ctx)
if removeErr != nil {
fs.Errorf(o, "Failed to remove partially transferred object: %v", err)
}
}
return nil, nil, err
}
// Check whether we got an error during compression
result := <-results
err = result.err
if err != nil {
if o != nil {
removeErr := o.Remove(ctx)
if removeErr != nil {
fs.Errorf(o, "Failed to remove partially compressed object: %v", err)
}
}
return nil, nil, err
}
// Generate metadata
meta := newMetadata(result.meta.Size, f.mode, result.meta, hex.EncodeToString(metaHasher.Sum(nil)), mimeType)
// Check the hashes of the compressed data if we were comparing them
if ht != hash.None && hasher != nil {
err = f.verifyObjectHash(ctx, o, hasher, ht)
if err != nil {
return nil, nil, err
}
}
return o, meta, nil
return f.modeHandler.putCompress(ctx, f, in, src, options, mimeType)
}
// Put an uncompressed version of a file. Returns a wrappable object and metadata.
@@ -680,7 +651,8 @@ func (f *Fs) putUncompress(ctx context.Context, in io.Reader, src fs.ObjectInfo,
if err != nil {
return nil, nil, err
}
return o, newMetadata(o.Size(), Uncompressed, sgzip.GzipMetadata{}, hex.EncodeToString(sum), mimeType), nil
return f.modeHandler.putUncompressGetNewMetadata(o, Uncompressed, hex.EncodeToString(sum), mimeType, sum)
}
// This function will write a metadata struct to a metadata Object for an src. Returns a wrappable metadata object.
@@ -751,7 +723,7 @@ func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options .
o, err := f.NewObject(ctx, src.Remote())
if err == fs.ErrorObjectNotFound {
// Get our file compressibility
in, compressible, mimeType, err := checkCompressAndType(in)
in, compressible, mimeType, err := checkCompressAndType(in, f.mode, f.modeHandler)
if err != nil {
return nil, err
}
@@ -771,7 +743,7 @@ func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, opt
}
found := err == nil
in, compressible, mimeType, err := checkCompressAndType(in)
in, compressible, mimeType, err := checkCompressAndType(in, f.mode, f.modeHandler)
if err != nil {
return nil, err
}
@@ -1090,11 +1062,12 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, duration fs.Duration
// ObjectMetadata describes the metadata for an Object.
type ObjectMetadata struct {
Mode int // Compression mode of the file.
Size int64 // Size of the object.
MD5 string // MD5 hash of the file.
MimeType string // Mime type of the file
CompressionMetadata sgzip.GzipMetadata
Mode int // Compression mode of the file.
Size int64 // Size of the object.
MD5 string // MD5 hash of the file.
MimeType string // Mime type of the file
CompressionMetadataGzip *sgzip.GzipMetadata // Metadata for Gzip compression
CompressionMetadataZstd *SzstdMetadata // Metadata for Zstd compression
}
// Object with external metadata
@@ -1107,17 +1080,6 @@ type Object struct {
meta *ObjectMetadata // Metadata struct for this object (nil if not loaded)
}
// This function generates a metadata object
func newMetadata(size int64, mode int, cmeta sgzip.GzipMetadata, md5 string, mimeType string) *ObjectMetadata {
meta := new(ObjectMetadata)
meta.Size = size
meta.Mode = mode
meta.CompressionMetadata = cmeta
meta.MD5 = md5
meta.MimeType = mimeType
return meta
}
// This function will read the metadata from a metadata object.
func readMetadata(ctx context.Context, mo fs.Object) (meta *ObjectMetadata, err error) {
// Open our meradata object
@@ -1165,7 +1127,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
return o.mo, o.mo.Update(ctx, in, src, options...)
}
in, compressible, mimeType, err := checkCompressAndType(in)
in, compressible, mimeType, err := checkCompressAndType(in, o.meta.Mode, o.f.modeHandler)
if err != nil {
return err
}
@@ -1278,7 +1240,7 @@ func (o *Object) String() string {
// Remote returns the remote path
func (o *Object) Remote() string {
origFileName, _, _, err := processFileName(o.Object.Remote())
origFileName, _, _, err := processFileName(o.Object.Remote(), o.f.modeHandler)
if err != nil {
fs.Errorf(o.f, "Could not get remote path for: %s", o.Object.Remote())
return o.Object.Remote()
@@ -1381,7 +1343,6 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.Read
return o.Object.Open(ctx, options...)
}
// Get offset and limit from OpenOptions, pass the rest to the underlying remote
var openOptions = []fs.OpenOption{&fs.SeekOption{Offset: 0}}
var offset, limit int64 = 0, -1
for _, option := range options {
switch x := option.(type) {
@@ -1389,31 +1350,12 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.Read
offset = x.Offset
case *fs.RangeOption:
offset, limit = x.Decode(o.Size())
default:
openOptions = append(openOptions, option)
}
}
// Get a chunkedreader for the wrapped object
chunkedReader := chunkedreader.New(ctx, o.Object, initialChunkSize, maxChunkSize, chunkStreams)
// Get file handle
var file io.Reader
if offset != 0 {
file, err = sgzip.NewReaderAt(chunkedReader, &o.meta.CompressionMetadata, offset)
} else {
file, err = sgzip.NewReader(chunkedReader)
}
if err != nil {
return nil, err
}
var fileReader io.Reader
if limit != -1 {
fileReader = io.LimitReader(file, limit)
} else {
fileReader = file
}
// Return a ReadCloser
return ReadCloserWrapper{Reader: fileReader, Closer: chunkedReader}, nil
var retCloser io.Closer = chunkedReader
return o.f.modeHandler.openGetReadCloser(ctx, o, offset, limit, chunkedReader, retCloser, options...)
}
// ObjectInfo describes a wrapped fs.ObjectInfo for being the source
+21 -1
View File
@@ -48,7 +48,27 @@ func TestRemoteGzip(t *testing.T) {
opt.ExtraConfig = []fstests.ExtraConfigItem{
{Name: name, Key: "type", Value: "compress"},
{Name: name, Key: "remote", Value: tempdir},
{Name: name, Key: "compression_mode", Value: "gzip"},
{Name: name, Key: "mode", Value: "gzip"},
{Name: name, Key: "level", Value: "-1"},
}
opt.QuickTestOK = true
fstests.Run(t, &opt)
}
// TestRemoteZstd tests ZSTD compression
func TestRemoteZstd(t *testing.T) {
if *fstest.RemoteName != "" {
t.Skip("Skipping as -remote set")
}
tempdir := filepath.Join(os.TempDir(), "rclone-compress-test-zstd")
name := "TestCompressZstd"
opt := defaultOpt
opt.RemoteName = name + ":"
opt.ExtraConfig = []fstests.ExtraConfigItem{
{Name: name, Key: "type", Value: "compress"},
{Name: name, Key: "remote", Value: tempdir},
{Name: name, Key: "mode", Value: "zstd"},
{Name: name, Key: "level", Value: "2"},
}
opt.QuickTestOK = true
fstests.Run(t, &opt)
+207
View File
@@ -0,0 +1,207 @@
package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
"errors"
"io"
"github.com/buengese/sgzip"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/fs/chunkedreader"
"github.com/rclone/rclone/fs/hash"
)
// gzipModeHandler implements compressionModeHandler for gzip
type gzipModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (g *gzipModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
var b bytes.Buffer
var n int64
w, err := sgzip.NewWriterLevel(&b, sgzip.DefaultCompression)
if err != nil {
return false, err
}
n, err = io.Copy(w, r)
if err != nil {
return false, err
}
err = w.Close()
if err != nil {
return false, err
}
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
// newObjectGetOriginalSize returns the original file size from the metadata
func (g *gzipModeHandler) newObjectGetOriginalSize(meta *ObjectMetadata) (int64, error) {
if meta.CompressionMetadataGzip == nil {
return 0, errors.New("missing gzip metadata")
}
return meta.CompressionMetadataGzip.Size, nil
}
// openGetReadCloser opens a compressed object and returns a ReadCloser in the Open method
func (g *gzipModeHandler) openGetReadCloser(
ctx context.Context,
o *Object,
offset int64,
limit int64,
cr chunkedreader.ChunkedReader,
closer io.Closer,
options ...fs.OpenOption,
) (rc io.ReadCloser, err error) {
var file io.Reader
if offset != 0 {
file, err = sgzip.NewReaderAt(cr, o.meta.CompressionMetadataGzip, offset)
} else {
file, err = sgzip.NewReader(cr)
}
if err != nil {
return nil, err
}
var fileReader io.Reader
if limit != -1 {
fileReader = io.LimitReader(file, limit)
} else {
fileReader = file
}
// Return a ReadCloser
return ReadCloserWrapper{Reader: fileReader, Closer: closer}, nil
}
// processFileNameGetFileExtension returns the file extension for the given compression mode
func (g *gzipModeHandler) processFileNameGetFileExtension(compressionMode int) string {
if compressionMode == Gzip {
return gzFileExt
}
return ""
}
// putCompress compresses the input data and uploads it to the remote, returning the new object and its metadata
func (g *gzipModeHandler) putCompress(
ctx context.Context,
f *Fs,
in io.Reader,
src fs.ObjectInfo,
options []fs.OpenOption,
mimeType string,
) (fs.Object, *ObjectMetadata, error) {
// Unwrap reader accounting
in, wrap := accounting.UnWrap(in)
// Add the metadata hasher
metaHasher := md5.New()
in = io.TeeReader(in, metaHasher)
// Compress the file
pipeReader, pipeWriter := io.Pipe()
resultsGzip := make(chan compressionResult[sgzip.GzipMetadata])
go func() {
gz, err := sgzip.NewWriterLevel(pipeWriter, f.opt.CompressionLevel)
if err != nil {
resultsGzip <- compressionResult[sgzip.GzipMetadata]{err: err, meta: sgzip.GzipMetadata{}}
close(resultsGzip)
return
}
_, err = io.Copy(gz, in)
gzErr := gz.Close()
if gzErr != nil && err == nil {
err = gzErr
}
closeErr := pipeWriter.Close()
if closeErr != nil && err == nil {
err = closeErr
}
resultsGzip <- compressionResult[sgzip.GzipMetadata]{err: err, meta: gz.MetaData()}
close(resultsGzip)
}()
wrappedIn := wrap(bufio.NewReaderSize(pipeReader, bufferSize)) // Probably no longer needed as sgzip has it's own buffering
// Find a hash the destination supports to compute a hash of
// the compressed data.
ht := f.Fs.Hashes().GetOne()
var hasher *hash.MultiHasher
var err error
if ht != hash.None {
// unwrap the accounting again
wrappedIn, wrap = accounting.UnWrap(wrappedIn)
hasher, err = hash.NewMultiHasherTypes(hash.NewHashSet(ht))
if err != nil {
return nil, nil, err
}
// add the hasher and re-wrap the accounting
wrappedIn = io.TeeReader(wrappedIn, hasher)
wrappedIn = wrap(wrappedIn)
}
// Transfer the data
o, err := f.rcat(ctx, makeDataName(src.Remote(), src.Size(), f.mode), io.NopCloser(wrappedIn), src.ModTime(ctx), options)
if err != nil {
if o != nil {
if removeErr := o.Remove(ctx); removeErr != nil {
fs.Errorf(o, "Failed to remove partially transferred object: %v", removeErr)
}
}
return nil, nil, err
}
// Check whether we got an error during compression
result := <-resultsGzip
if result.err != nil {
if o != nil {
if removeErr := o.Remove(ctx); removeErr != nil {
fs.Errorf(o, "Failed to remove partially compressed object: %v", removeErr)
}
}
return nil, nil, result.err
}
// Generate metadata
meta := g.newMetadata(result.meta.Size, f.mode, result.meta, hex.EncodeToString(metaHasher.Sum(nil)), mimeType)
// Check the hashes of the compressed data if we were comparing them
if ht != hash.None && hasher != nil {
err = f.verifyObjectHash(ctx, o, hasher, ht)
if err != nil {
return nil, nil, err
}
}
return o, meta, nil
}
// putUncompressGetNewMetadata returns metadata in the putUncompress method for a specific compression algorithm
func (g *gzipModeHandler) putUncompressGetNewMetadata(o fs.Object, mode int, md5 string, mimeType string, sum []byte) (fs.Object, *ObjectMetadata, error) {
return o, g.newMetadata(o.Size(), mode, sgzip.GzipMetadata{}, hex.EncodeToString(sum), mimeType), nil
}
// This function generates a metadata object for sgzip.GzipMetadata or SzstdMetadata.
// Warning: This function panics if cmeta is not of the expected type.
func (g *gzipModeHandler) newMetadata(size int64, mode int, cmeta any, md5 string, mimeType string) *ObjectMetadata {
meta, ok := cmeta.(sgzip.GzipMetadata)
if !ok {
panic("invalid cmeta type: expected sgzip.GzipMetadata")
}
objMeta := new(ObjectMetadata)
objMeta.Size = size
objMeta.Mode = mode
objMeta.CompressionMetadataGzip = &meta
objMeta.CompressionMetadataZstd = nil
objMeta.MD5 = md5
objMeta.MimeType = mimeType
return objMeta
}
+327
View File
@@ -0,0 +1,327 @@
package compress
import (
"context"
"errors"
"io"
"runtime"
"sync"
szstd "github.com/a1ex3/zstd-seekable-format-go/pkg"
"github.com/klauspost/compress/zstd"
)
const szstdChunkSize int = 1 << 20 // 1 MiB chunk size
// SzstdMetadata holds metadata for szstd compressed files.
type SzstdMetadata struct {
BlockSize int // BlockSize is the size of the blocks in the zstd file
Size int64 // Size is the uncompressed size of the file
BlockData []uint32 // BlockData is the block data for the zstd file, used for seeking
}
// SzstdWriter is a writer that compresses data in szstd format.
type SzstdWriter struct {
enc *zstd.Encoder
w szstd.ConcurrentWriter
metadata SzstdMetadata
mu sync.Mutex
}
// NewWriterSzstd creates a new szstd writer with the specified options.
// It initializes the szstd writer with a zstd encoder and returns a pointer to the SzstdWriter.
// The writer can be used to write data in chunks, and it will automatically handle block sizes and metadata.
func NewWriterSzstd(w io.Writer, opts ...zstd.EOption) (*SzstdWriter, error) {
encoder, err := zstd.NewWriter(nil, opts...)
if err != nil {
return nil, err
}
sw, err := szstd.NewWriter(w, encoder)
if err != nil {
if err := encoder.Close(); err != nil {
return nil, err
}
return nil, err
}
return &SzstdWriter{
enc: encoder,
w: sw,
metadata: SzstdMetadata{
BlockSize: szstdChunkSize,
Size: 0,
},
}, nil
}
// Write writes data to the szstd writer in chunks of szstdChunkSize.
// It handles the block size and metadata updates automatically.
func (w *SzstdWriter) Write(p []byte) (int, error) {
if len(p) == 0 {
return 0, nil
}
if w.metadata.BlockData == nil {
numBlocks := (len(p) + w.metadata.BlockSize - 1) / w.metadata.BlockSize
w.metadata.BlockData = make([]uint32, 1, numBlocks+1)
w.metadata.BlockData[0] = 0
}
start := 0
total := len(p)
var writerFunc szstd.FrameSource = func() ([]byte, error) {
if start >= total {
return nil, nil
}
end := min(start+w.metadata.BlockSize, total)
chunk := p[start:end]
size := end - start
w.mu.Lock()
w.metadata.Size += int64(size)
w.mu.Unlock()
start = end
return chunk, nil
}
// write sizes of compressed blocks in the callback
err := w.w.WriteMany(context.Background(), writerFunc,
szstd.WithWriteCallback(func(size uint32) {
w.mu.Lock()
lastOffset := w.metadata.BlockData[len(w.metadata.BlockData)-1]
w.metadata.BlockData = append(w.metadata.BlockData, lastOffset+size)
w.mu.Unlock()
}),
)
if err != nil {
return 0, err
}
return total, nil
}
// Close closes the SzstdWriter and its underlying encoder.
func (w *SzstdWriter) Close() error {
if err := w.w.Close(); err != nil {
return err
}
if err := w.enc.Close(); err != nil {
return err
}
return nil
}
// GetMetadata returns the metadata of the szstd writer.
func (w *SzstdWriter) GetMetadata() SzstdMetadata {
return w.metadata
}
// SzstdReaderAt is a reader that allows random access in szstd compressed data.
type SzstdReaderAt struct {
r szstd.Reader
decoder *zstd.Decoder
metadata *SzstdMetadata
pos int64
mu sync.Mutex
}
// NewReaderAtSzstd creates a new SzstdReaderAt at the specified io.ReadSeeker.
func NewReaderAtSzstd(rs io.ReadSeeker, meta *SzstdMetadata, offset int64, opts ...zstd.DOption) (*SzstdReaderAt, error) {
decoder, err := zstd.NewReader(nil, opts...)
if err != nil {
return nil, err
}
r, err := szstd.NewReader(rs, decoder)
if err != nil {
decoder.Close()
return nil, err
}
sr := &SzstdReaderAt{
r: r,
decoder: decoder,
metadata: meta,
pos: 0,
}
// Set initial position to the provided offset
if _, err := sr.Seek(offset, io.SeekStart); err != nil {
if err := sr.Close(); err != nil {
return nil, err
}
return nil, err
}
return sr, nil
}
// Seek sets the offset for the next Read.
func (s *SzstdReaderAt) Seek(offset int64, whence int) (int64, error) {
s.mu.Lock()
defer s.mu.Unlock()
pos, err := s.r.Seek(offset, whence)
if err == nil {
s.pos = pos
}
return pos, err
}
func (s *SzstdReaderAt) Read(p []byte) (int, error) {
s.mu.Lock()
defer s.mu.Unlock()
n, err := s.r.Read(p)
if err == nil {
s.pos += int64(n)
}
return n, err
}
// ReadAt reads data at the specified offset.
func (s *SzstdReaderAt) ReadAt(p []byte, off int64) (int, error) {
if off < 0 {
return 0, errors.New("invalid offset")
}
if off >= s.metadata.Size {
return 0, io.EOF
}
endOff := min(off+int64(len(p)), s.metadata.Size)
// Find all blocks covered by the range
type blockInfo struct {
index int // Block index
offsetInBlock int64 // Offset within the block for starting reading
bytesToRead int64 // How many bytes to read from this block
}
var blocks []blockInfo
uncompressedOffset := int64(0)
currentOff := off
for i := 0; i < len(s.metadata.BlockData)-1; i++ {
blockUncompressedEnd := min(uncompressedOffset+int64(s.metadata.BlockSize), s.metadata.Size)
if currentOff < blockUncompressedEnd && endOff > uncompressedOffset {
offsetInBlock := max(0, currentOff-uncompressedOffset)
bytesToRead := min(blockUncompressedEnd-uncompressedOffset-offsetInBlock, endOff-currentOff)
blocks = append(blocks, blockInfo{
index: i,
offsetInBlock: offsetInBlock,
bytesToRead: bytesToRead,
})
currentOff += bytesToRead
if currentOff >= endOff {
break
}
}
uncompressedOffset = blockUncompressedEnd
}
if len(blocks) == 0 {
return 0, io.EOF
}
// Parallel block decoding
type decodeResult struct {
index int
data []byte
err error
}
resultCh := make(chan decodeResult, len(blocks))
var wg sync.WaitGroup
sem := make(chan struct{}, runtime.NumCPU())
for _, block := range blocks {
wg.Add(1)
go func(block blockInfo) {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
startOffset := int64(s.metadata.BlockData[block.index])
endOffset := int64(s.metadata.BlockData[block.index+1])
compressedSize := endOffset - startOffset
compressed := make([]byte, compressedSize)
_, err := s.r.ReadAt(compressed, startOffset)
if err != nil && err != io.EOF {
resultCh <- decodeResult{index: block.index, err: err}
return
}
decoded, err := s.decoder.DecodeAll(compressed, nil)
if err != nil {
resultCh <- decodeResult{index: block.index, err: err}
return
}
resultCh <- decodeResult{index: block.index, data: decoded, err: nil}
}(block)
}
go func() {
wg.Wait()
close(resultCh)
}()
// Collect results in block index order
totalRead := 0
results := make(map[int]decodeResult)
expected := len(blocks)
minIndex := blocks[0].index
for res := range resultCh {
results[res.index] = res
for {
if result, ok := results[minIndex]; ok {
if result.err != nil {
return 0, result.err
}
// find the corresponding blockInfo
var blk blockInfo
for _, b := range blocks {
if b.index == result.index {
blk = b
break
}
}
start := blk.offsetInBlock
end := start + blk.bytesToRead
copy(p[totalRead:totalRead+int(blk.bytesToRead)], result.data[start:end])
totalRead += int(blk.bytesToRead)
minIndex++
if minIndex-blocks[0].index >= len(blocks) {
break
}
} else {
break
}
}
if len(results) == expected && minIndex-blocks[0].index >= len(blocks) {
break
}
}
return totalRead, nil
}
// Close closes the SzstdReaderAt and underlying decoder.
func (s *SzstdReaderAt) Close() error {
if err := s.r.Close(); err != nil {
return err
}
s.decoder.Close()
return nil
}
+65
View File
@@ -0,0 +1,65 @@
package compress
import (
"context"
"fmt"
"io"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/chunkedreader"
)
// uncompressedModeHandler implements compressionModeHandler for uncompressed files
type uncompressedModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (u *uncompressedModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
return false, nil
}
// newObjectGetOriginalSize returns the original file size from the metadata
func (u *uncompressedModeHandler) newObjectGetOriginalSize(meta *ObjectMetadata) (int64, error) {
return 0, nil
}
// openGetReadCloser opens a compressed object and returns a ReadCloser in the Open method
func (u *uncompressedModeHandler) openGetReadCloser(
ctx context.Context,
o *Object,
offset int64,
limit int64,
cr chunkedreader.ChunkedReader,
closer io.Closer,
options ...fs.OpenOption,
) (rc io.ReadCloser, err error) {
return o.Object.Open(ctx, options...)
}
// processFileNameGetFileExtension returns the file extension for the given compression mode
func (u *uncompressedModeHandler) processFileNameGetFileExtension(compressionMode int) string {
return ""
}
// putCompress compresses the input data and uploads it to the remote, returning the new object and its metadata
func (u *uncompressedModeHandler) putCompress(
ctx context.Context,
f *Fs,
in io.Reader,
src fs.ObjectInfo,
options []fs.OpenOption,
mimeType string,
) (fs.Object, *ObjectMetadata, error) {
return nil, nil, fmt.Errorf("unsupported compression mode %d", f.mode)
}
// putUncompressGetNewMetadata returns metadata in the putUncompress method for a specific compression algorithm
func (u *uncompressedModeHandler) putUncompressGetNewMetadata(o fs.Object, mode int, md5 string, mimeType string, sum []byte) (fs.Object, *ObjectMetadata, error) {
return nil, nil, fmt.Errorf("unsupported compression mode %d", Uncompressed)
}
// This function generates a metadata object for sgzip.GzipMetadata or SzstdMetadata.
// Warning: This function panics if cmeta is not of the expected type.
func (u *uncompressedModeHandler) newMetadata(size int64, mode int, cmeta any, md5 string, mimeType string) *ObjectMetadata {
return nil
}
+65
View File
@@ -0,0 +1,65 @@
package compress
import (
"context"
"fmt"
"io"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/chunkedreader"
)
// unknownModeHandler implements compressionModeHandler for unknown compression types
type unknownModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (unk *unknownModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
return false, fmt.Errorf("unknown compression mode %d", compressionMode)
}
// newObjectGetOriginalSize returns the original file size from the metadata
func (unk *unknownModeHandler) newObjectGetOriginalSize(meta *ObjectMetadata) (int64, error) {
return 0, nil
}
// openGetReadCloser opens a compressed object and returns a ReadCloser in the Open method
func (unk *unknownModeHandler) openGetReadCloser(
ctx context.Context,
o *Object,
offset int64,
limit int64,
cr chunkedreader.ChunkedReader,
closer io.Closer,
options ...fs.OpenOption,
) (rc io.ReadCloser, err error) {
return nil, fmt.Errorf("unknown compression mode %d", o.meta.Mode)
}
// processFileNameGetFileExtension returns the file extension for the given compression mode
func (unk *unknownModeHandler) processFileNameGetFileExtension(compressionMode int) string {
return ""
}
// putCompress compresses the input data and uploads it to the remote, returning the new object and its metadata
func (unk *unknownModeHandler) putCompress(
ctx context.Context,
f *Fs,
in io.Reader,
src fs.ObjectInfo,
options []fs.OpenOption,
mimeType string,
) (fs.Object, *ObjectMetadata, error) {
return nil, nil, fmt.Errorf("unknown compression mode %d", f.mode)
}
// putUncompressGetNewMetadata returns metadata in the putUncompress method for a specific compression algorithm
func (unk *unknownModeHandler) putUncompressGetNewMetadata(o fs.Object, mode int, md5 string, mimeType string, sum []byte) (fs.Object, *ObjectMetadata, error) {
return nil, nil, fmt.Errorf("unknown compression mode")
}
// This function generates a metadata object for sgzip.GzipMetadata or SzstdMetadata.
// Warning: This function panics if cmeta is not of the expected type.
func (unk *unknownModeHandler) newMetadata(size int64, mode int, cmeta any, md5 string, mimeType string) *ObjectMetadata {
return nil
}
+192
View File
@@ -0,0 +1,192 @@
package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
"errors"
"io"
"github.com/klauspost/compress/zstd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/fs/chunkedreader"
"github.com/rclone/rclone/fs/hash"
)
// zstdModeHandler implements compressionModeHandler for zstd
type zstdModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (z *zstdModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
var b bytes.Buffer
var n int64
w, err := NewWriterSzstd(&b, zstd.WithEncoderLevel(zstd.SpeedDefault))
if err != nil {
return false, err
}
n, err = io.Copy(w, r)
if err != nil {
return false, err
}
err = w.Close()
if err != nil {
return false, err
}
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
// newObjectGetOriginalSize returns the original file size from the metadata
func (z *zstdModeHandler) newObjectGetOriginalSize(meta *ObjectMetadata) (int64, error) {
if meta.CompressionMetadataZstd == nil {
return 0, errors.New("missing zstd metadata")
}
return meta.CompressionMetadataZstd.Size, nil
}
// openGetReadCloser opens a compressed object and returns a ReadCloser in the Open method
func (z *zstdModeHandler) openGetReadCloser(
ctx context.Context,
o *Object,
offset int64,
limit int64,
cr chunkedreader.ChunkedReader,
closer io.Closer,
options ...fs.OpenOption,
) (rc io.ReadCloser, err error) {
var file io.Reader
if offset != 0 {
file, err = NewReaderAtSzstd(cr, o.meta.CompressionMetadataZstd, offset)
} else {
file, err = zstd.NewReader(cr)
}
if err != nil {
return nil, err
}
var fileReader io.Reader
if limit != -1 {
fileReader = io.LimitReader(file, limit)
} else {
fileReader = file
}
// Return a ReadCloser
return ReadCloserWrapper{Reader: fileReader, Closer: closer}, nil
}
// processFileNameGetFileExtension returns the file extension for the given compression mode
func (z *zstdModeHandler) processFileNameGetFileExtension(compressionMode int) string {
if compressionMode == Zstd {
return zstdFileExt
}
return ""
}
// putCompress compresses the input data and uploads it to the remote, returning the new object and its metadata
func (z *zstdModeHandler) putCompress(
ctx context.Context,
f *Fs,
in io.Reader,
src fs.ObjectInfo,
options []fs.OpenOption,
mimeType string,
) (fs.Object, *ObjectMetadata, error) {
// Unwrap reader accounting
in, wrap := accounting.UnWrap(in)
// Add the metadata hasher
metaHasher := md5.New()
in = io.TeeReader(in, metaHasher)
// Compress the file
pipeReader, pipeWriter := io.Pipe()
resultsZstd := make(chan compressionResult[SzstdMetadata])
go func() {
writer, err := NewWriterSzstd(pipeWriter, zstd.WithEncoderLevel(zstd.EncoderLevel(f.opt.CompressionLevel)))
if err != nil {
resultsZstd <- compressionResult[SzstdMetadata]{err: err}
close(resultsZstd)
return
}
_, err = io.Copy(writer, in)
if wErr := writer.Close(); wErr != nil && err == nil {
err = wErr
}
if cErr := pipeWriter.Close(); cErr != nil && err == nil {
err = cErr
}
resultsZstd <- compressionResult[SzstdMetadata]{err: err, meta: writer.GetMetadata()}
close(resultsZstd)
}()
wrappedIn := wrap(bufio.NewReaderSize(pipeReader, bufferSize))
ht := f.Fs.Hashes().GetOne()
var hasher *hash.MultiHasher
var err error
if ht != hash.None {
wrappedIn, wrap = accounting.UnWrap(wrappedIn)
hasher, err = hash.NewMultiHasherTypes(hash.NewHashSet(ht))
if err != nil {
return nil, nil, err
}
wrappedIn = io.TeeReader(wrappedIn, hasher)
wrappedIn = wrap(wrappedIn)
}
o, err := f.rcat(ctx, makeDataName(src.Remote(), src.Size(), f.mode), io.NopCloser(wrappedIn), src.ModTime(ctx), options)
if err != nil {
return nil, nil, err
}
result := <-resultsZstd
if result.err != nil {
if o != nil {
_ = o.Remove(ctx)
}
return nil, nil, result.err
}
// Build metadata using uncompressed size for filename
meta := z.newMetadata(result.meta.Size, f.mode, result.meta, hex.EncodeToString(metaHasher.Sum(nil)), mimeType)
if ht != hash.None && hasher != nil {
err = f.verifyObjectHash(ctx, o, hasher, ht)
if err != nil {
return nil, nil, err
}
}
return o, meta, nil
}
// putUncompressGetNewMetadata returns metadata in the putUncompress method for a specific compression algorithm
func (z *zstdModeHandler) putUncompressGetNewMetadata(o fs.Object, mode int, md5 string, mimeType string, sum []byte) (fs.Object, *ObjectMetadata, error) {
return o, z.newMetadata(o.Size(), mode, SzstdMetadata{}, hex.EncodeToString(sum), mimeType), nil
}
// This function generates a metadata object for sgzip.GzipMetadata or SzstdMetadata.
// Warning: This function panics if cmeta is not of the expected type.
func (z *zstdModeHandler) newMetadata(size int64, mode int, cmeta any, md5 string, mimeType string) *ObjectMetadata {
meta, ok := cmeta.(SzstdMetadata)
if !ok {
panic("invalid cmeta type: expected SzstdMetadata")
}
objMeta := new(ObjectMetadata)
objMeta.Size = size
objMeta.Mode = mode
objMeta.CompressionMetadataGzip = nil
objMeta.CompressionMetadataZstd = &meta
objMeta.MD5 = md5
objMeta.MimeType = mimeType
return objMeta
}
+5 -5
View File
@@ -403,14 +403,14 @@ func (c *Cipher) deobfuscateSegment(ciphertext string) (string, error) {
if ciphertext == "" {
return "", nil
}
pos := strings.Index(ciphertext, ".")
if pos == -1 {
before, after, ok := strings.Cut(ciphertext, ".")
if !ok {
return "", ErrorNotAnEncryptedFile
} // No .
num := ciphertext[:pos]
num := before
if num == "!" {
// No rotation; probably original was not valid unicode
return ciphertext[pos+1:], nil
return after, nil
}
dir, err := strconv.Atoi(num)
if err != nil {
@@ -425,7 +425,7 @@ func (c *Cipher) deobfuscateSegment(ciphertext string) (string, error) {
var result bytes.Buffer
inQuote := false
for _, runeValue := range ciphertext[pos+1:] {
for _, runeValue := range after {
switch {
case inQuote:
_, _ = result.WriteRune(runeValue)
+12 -10
View File
@@ -923,28 +923,30 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
var commandHelp = []fs.CommandHelp{
{
Name: "encode",
Short: "Encode the given filename(s)",
Short: "Encode the given filename(s).",
Long: `This encodes the filenames given as arguments returning a list of
strings of the encoded results.
Usage Example:
Usage examples:
rclone backend encode crypt: file1 [file2...]
rclone rc backend/command command=encode fs=crypt: file1 [file2...]
`,
` + "```console" + `
rclone backend encode crypt: file1 [file2...]
rclone rc backend/command command=encode fs=crypt: file1 [file2...]
` + "```",
},
{
Name: "decode",
Short: "Decode the given filename(s)",
Short: "Decode the given filename(s).",
Long: `This decodes the filenames given as arguments returning a list of
strings of the decoded results. It will return an error if any of the
inputs are invalid.
Usage Example:
Usage examples:
rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
`,
` + "```console" + `
rclone backend decode crypt: encryptedfile1 [encryptedfile2...]
rclone rc backend/command command=decode fs=crypt: encryptedfile1 [encryptedfile2...]
` + "```",
},
}
+16 -12
View File
@@ -77,7 +77,7 @@ The DOI provider can be set when rclone does not automatically recognize a suppo
Name: "doi_resolver_api_url",
Help: `The URL of the DOI resolver API to use.
The DOI resolver can be set for testing or for cases when the the canonical DOI resolver API cannot be used.
The DOI resolver can be set for testing or for cases when the canonical DOI resolver API cannot be used.
Defaults to "https://doi.org/api".`,
Required: false,
@@ -563,21 +563,26 @@ var commandHelp = []fs.CommandHelp{{
Short: "Show metadata about the DOI.",
Long: `This command returns a JSON object with some information about the DOI.
rclone backend medatadata doi:
Usage example:
It returns a JSON object representing metadata about the DOI.
`,
` + "```console" + `
rclone backend metadata doi:
` + "```" + `
It returns a JSON object representing metadata about the DOI.`,
}, {
Name: "set",
Short: "Set command for updating the config parameters.",
Long: `This set command can be used to update the config parameters
for a running doi backend.
Usage Examples:
Usage examples:
rclone backend set doi: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=doi: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=doi: -o doi=NEW_DOI
` + "```console" + `
rclone backend set doi: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=doi: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=doi: -o doi=NEW_DOI
` + "```" + `
The option keys are named as they are in the config file.
@@ -585,8 +590,7 @@ This rebuilds the connection to the doi backend when it is called with
the new parameters. Only new parameters need be passed as the values
will default to those currently in use.
It doesn't return anything.
`,
It doesn't return anything.`,
}}
// Command the backend to run a named command
@@ -598,7 +602,7 @@ It doesn't return anything.
// The result should be capable of being JSON encoded
// If it is a string or a []string it will be shown to the user
// otherwise it will be JSON encoded and shown to the user like that
func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[string]string) (out interface{}, err error) {
func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[string]string) (out any, err error) {
switch name {
case "metadata":
return f.ShowMetadata(ctx)
@@ -625,7 +629,7 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
}
// ShowMetadata returns some metadata about the corresponding DOI
func (f *Fs) ShowMetadata(ctx context.Context) (metadata interface{}, err error) {
func (f *Fs) ShowMetadata(ctx context.Context) (metadata any, err error) {
doiURL, err := url.Parse("https://doi.org/" + f.opt.Doi)
if err != nil {
return nil, err
+2 -2
View File
@@ -18,7 +18,7 @@ type headerLink struct {
}
func parseLinkHeader(header string) (links []headerLink) {
for _, link := range strings.Split(header, ",") {
for link := range strings.SplitSeq(header, ",") {
link = strings.TrimSpace(link)
parsed := parseLink(link)
if parsed != nil {
@@ -30,7 +30,7 @@ func parseLinkHeader(header string) (links []headerLink) {
func parseLink(link string) (parsedLink *headerLink) {
var parts []string
for _, part := range strings.Split(link, ";") {
for part := range strings.SplitSeq(link, ";") {
parts = append(parts, strings.TrimSpace(part))
}
+246
View File
@@ -0,0 +1,246 @@
// Package api has type definitions for drime
//
// Converted from the API docs with help from https://mholt.github.io/json-to-go/
package api
import (
"encoding/json"
"fmt"
"time"
)
// Types of things in Item
const (
ItemTypeFolder = "folder"
)
// User information
type User struct {
Email string `json:"email"`
ID json.Number `json:"id"`
Avatar string `json:"avatar"`
ModelType string `json:"model_type"`
OwnsEntry bool `json:"owns_entry"`
EntryPermissions any `json:"entry_permissions"`
DisplayName string `json:"display_name"`
}
// Permissions for a file
type Permissions struct {
FilesUpdate bool `json:"files.update"`
FilesCreate bool `json:"files.create"`
FilesDownload bool `json:"files.download"`
FilesDelete bool `json:"files.delete"`
}
// Item describes a folder or a file as returned by /drive/file-entries
type Item struct {
ID json.Number `json:"id"`
Name string `json:"name"`
Description any `json:"description"`
FileName string `json:"file_name"`
Mime string `json:"mime"`
Color any `json:"color"`
Backup bool `json:"backup"`
Tracked int `json:"tracked"`
FileSize int64 `json:"file_size"`
UserID json.Number `json:"user_id"`
ParentID json.Number `json:"parent_id"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
DeletedAt any `json:"deleted_at"`
IsDeleted int `json:"is_deleted"`
Path string `json:"path"`
DiskPrefix any `json:"disk_prefix"`
Type string `json:"type"`
Extension any `json:"extension"`
FileHash any `json:"file_hash"`
Public bool `json:"public"`
Thumbnail bool `json:"thumbnail"`
ThumbnailURL any `json:"thumbnail_url"`
WorkspaceID int `json:"workspace_id"`
IsEncrypted int `json:"is_encrypted"`
Iv any `json:"iv"`
VaultID any `json:"vault_id"`
OwnerID int `json:"owner_id"`
Hash string `json:"hash"`
URL string `json:"url"`
Users []User `json:"users"`
Tags []any `json:"tags"`
Permissions Permissions `json:"permissions"`
}
// Listing response
type Listing struct {
CurrentPage int `json:"current_page"`
Data []Item `json:"data"`
From int `json:"from"`
LastPage int `json:"last_page"`
NextPageURL string `json:"next_page_url"`
PerPage int `json:"per_page"`
PrevPageURL string `json:"prev_page_url"`
To int `json:"to"`
Total int `json:"total"`
}
// UploadResponse for a file
type UploadResponse struct {
Status string `json:"status"`
FileEntry Item `json:"fileEntry"`
}
// CreateFolderRequest for a folder
type CreateFolderRequest struct {
Name string `json:"name"`
ParentID json.Number `json:"parentId,omitempty"`
}
// CreateFolderResponse for a folder
type CreateFolderResponse struct {
Status string `json:"status"`
Folder Item `json:"folder"`
}
// Error is returned from drime when things go wrong
type Error struct {
Message string `json:"message"`
}
// Error returns a string for the error and satisfies the error interface
func (e Error) Error() string {
out := fmt.Sprintf("Error %q", e.Message)
return out
}
// Check Error satisfies the error interface
var _ error = (*Error)(nil)
// DeleteRequest is the input to DELETE /file-entries
type DeleteRequest struct {
EntryIDs []string `json:"entryIds"`
DeleteForever bool `json:"deleteForever"`
}
// DeleteResponse is the input to DELETE /file-entries
type DeleteResponse struct {
Status string `json:"status"`
Message string `json:"message"`
Errors map[string]string `json:"errors"`
}
// UpdateItemRequest describes the updates to be done to an item for PUT /file-entries/{id}/
type UpdateItemRequest struct {
Name string `json:"name,omitempty"`
Description string `json:"description,omitempty"`
}
// UpdateItemResponse is returned by PUT /file-entries/{id}/
type UpdateItemResponse struct {
Status string `json:"status"`
FileEntry Item `json:"fileEntry"`
}
// MoveRequest is the input to /file-entries/move
type MoveRequest struct {
EntryIDs []string `json:"entryIds"`
DestinationID string `json:"destinationId"`
}
// MoveResponse is returned by POST /file-entries/move
type MoveResponse struct {
Status string `json:"status"`
Entries []Item `json:"entries"`
}
// CopyRequest is the input to /file-entries/duplicate
type CopyRequest struct {
EntryIDs []string `json:"entryIds"`
DestinationID string `json:"destinationId"`
}
// CopyResponse is returned by POST /file-entries/duplicate
type CopyResponse struct {
Status string `json:"status"`
Entries []Item `json:"entries"`
}
// MultiPartCreateRequest is the input of POST /s3/multipart/create
type MultiPartCreateRequest struct {
Filename string `json:"filename"`
Mime string `json:"mime"`
Size int64 `json:"size"`
Extension string `json:"extension"`
ParentID json.Number `json:"parentId"`
RelativePath string `json:"relativePath"`
WorkspaceID string `json:"workspaceId,omitempty"`
}
// MultiPartCreateResponse is returned by POST /s3/multipart/create
type MultiPartCreateResponse struct {
UploadID string `json:"uploadId"`
Key string `json:"key"`
}
// CompletedPart Type for completed parts when making a multipart upload.
type CompletedPart struct {
ETag string `json:"ETag"`
PartNumber int32 `json:"PartNumber"`
}
// MultiPartGetURLsRequest is the input of POST /s3/multipart/batch-sign-part-urls
type MultiPartGetURLsRequest struct {
UploadID string `json:"uploadId"`
Key string `json:"key"`
PartNumbers []int `json:"partNumbers"`
}
// MultiPartGetURLsResponse is the result of POST /s3/multipart/batch-sign-part-urls
type MultiPartGetURLsResponse struct {
URLs []struct {
URL string `json:"url"`
PartNumber int32 `json:"partNumber"`
} `json:"urls"`
}
// MultiPartCompleteRequest is the input to POST /s3/multipart/complete
type MultiPartCompleteRequest struct {
UploadID string `json:"uploadId"`
Key string `json:"key"`
Parts []CompletedPart `json:"parts"`
}
// MultiPartCompleteResponse is the result of POST /s3/multipart/complete
type MultiPartCompleteResponse struct {
Location string `json:"location"`
}
// MultiPartEntriesRequest is the input to POST /s3/entries
type MultiPartEntriesRequest struct {
ClientMime string `json:"clientMime"`
ClientName string `json:"clientName"`
Filename string `json:"filename"`
Size int64 `json:"size"`
ClientExtension string `json:"clientExtension"`
ParentID json.Number `json:"parentId"`
RelativePath string `json:"relativePath"`
WorkspaceID string `json:"workspaceId,omitempty"`
}
// MultiPartEntriesResponse is the result of POST /s3/entries
type MultiPartEntriesResponse struct {
FileEntry Item `json:"fileEntry"`
}
// MultiPartAbort is the input of POST /s3/multipart/abort
type MultiPartAbort struct {
UploadID string `json:"uploadId"`
Key string `json:"key"`
}
// SpaceUsageResponse is returned by GET /user/space-usage
type SpaceUsageResponse struct {
Used int64 `json:"used"`
Available int64 `json:"available"`
Status string `json:"status"`
SEO any `json:"seo"`
}
File diff suppressed because it is too large. Load diff
+33
View File
@@ -0,0 +1,33 @@
// Drime filesystem interface
package drime
import (
"testing"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest/fstests"
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestDrime:",
NilObject: (*Object)(nil),
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: minChunkSize,
},
})
}
func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
return f.setUploadChunkSize(cs)
}
func (f *Fs) SetUploadCutoff(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
return f.setUploadCutoff(cs)
}
var (
_ fstests.SetUploadChunkSizer = (*Fs)(nil)
_ fstests.SetUploadCutoffer = (*Fs)(nil)
)
+270 -138
View File
@@ -144,6 +144,7 @@ var (
_importFormats map[string][]string // allowed import MIME type conversions
templatesOnce sync.Once // parse link templates only once
_linkTemplates map[string]*template.Template // available link types
gdocsWarnOnce sync.Once // warn once about skipped non-exportable Google documents
)
// rwChoices type for fs.Bits
@@ -191,7 +192,7 @@ func driveScopes(scopesString string) (scopes []string) {
if scopesString == "" {
scopesString = defaultScope
}
for _, scope := range strings.Split(scopesString, ",") {
for scope := range strings.SplitSeq(scopesString, ",") {
scope = strings.TrimSpace(scope)
scopes = append(scopes, scopePrefix+scope)
}
@@ -207,7 +208,10 @@ func driveOAuthOptions() []fs.Option {
opts := []fs.Option{}
for _, opt := range oauthutil.SharedOptions {
if opt.Name == config.ConfigClientID {
opt.Help = "Google Application Client Id\nSetting your own is recommended.\nSee https://rclone.org/drive/#making-your-own-client-id for how to create your own.\nIf you leave this blank, it will use an internal key which is low performance."
opt.Help = "Google Application Client Id\nLeave blank to use rclone's shared client_id, or if you are using a service account.\nThe shared client_id is being retired and will stop working during 2026, so creating your own is now strongly recommended.\nSee https://rclone.org/drive/#making-your-own-client-id for how to create your own."
}
if opt.Name == config.ConfigClientSecret {
opt.Help = "Google Application Client Secret\nLeave blank to use rclone's shared client_id, or if you are using a service account.\nIf you created your own client_id then enter its client secret here."
}
opts = append(opts, opt)
}
@@ -221,7 +225,7 @@ func init() {
Description: "Google Drive",
NewFs: NewFs,
CommandHelp: commandHelp,
Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
Config: func(ctx context.Context, name string, m configmap.Mapper, configIn fs.ConfigIn) (*fs.ConfigOut, error) {
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
@@ -229,7 +233,7 @@ func init() {
return nil, fmt.Errorf("couldn't parse config into struct: %w", err)
}
switch config.State {
switch configIn.State {
case "":
// Fill in the scopes
driveConfig.Scopes = driveScopes(opt.Scope)
@@ -240,24 +244,43 @@ func init() {
}
if opt.ServiceAccountFile == "" && opt.ServiceAccountCredentials == "" && !opt.EnvAuth {
return oauthutil.ConfigOut("teamdrive", &oauthutil.Options{
OAuth2Config: driveConfig,
})
return fs.ConfigGoto("client_id")
}
return fs.ConfigGoto("teamdrive")
case "client_id":
if clientID, _ := m.Get(config.ConfigClientID); clientID != "" {
return fs.ConfigGoto("oauth")
}
return oauthutil.SharedClientIDConfigConfirm("client_id_warning", "Google Drive", "https://rclone.org/drive/#making-your-own-client-id")
case "client_id_warning":
if configIn.Result == "true" {
// Continue using the shared client_id
return fs.ConfigGoto("oauth")
}
return fs.ConfigInput("client_id_set", config.ConfigClientID, "Google Application Client Id")
case "client_id_set":
m.Set(config.ConfigClientID, configIn.Result)
return fs.ConfigInput("client_secret_set", config.ConfigClientSecret, "Google Application Client Secret")
case "client_secret_set":
m.Set(config.ConfigClientSecret, configIn.Result)
return fs.ConfigGoto("oauth")
case "oauth":
return oauthutil.ConfigOut("teamdrive", &oauthutil.Options{
OAuth2Config: driveConfig,
})
case "teamdrive":
if opt.TeamDriveID == "" {
return fs.ConfigConfirm("teamdrive_ok", false, "config_change_team_drive", "Configure this as a Shared Drive (Team Drive)?\n")
}
return fs.ConfigConfirm("teamdrive_change", false, "config_change_team_drive", fmt.Sprintf("Change current Shared Drive (Team Drive) ID %q?\n", opt.TeamDriveID))
case "teamdrive_ok":
if config.Result == "false" {
if configIn.Result == "false" {
m.Set("team_drive", "")
return nil, nil
}
return fs.ConfigGoto("teamdrive_config")
case "teamdrive_change":
if config.Result == "false" {
if configIn.Result == "false" {
return nil, nil
}
return fs.ConfigGoto("teamdrive_config")
@@ -278,14 +301,14 @@ func init() {
return teamDrive.Id, teamDrive.Name
})
case "teamdrive_final":
driveID := config.Result
driveID := configIn.Result
m.Set("team_drive", driveID)
m.Set("root_folder_id", "")
opt.TeamDriveID = driveID
opt.RootFolderID = ""
return nil, nil
}
return nil, fmt.Errorf("unknown state %q", config.State)
return nil, fmt.Errorf("unknown state %q", configIn.State)
},
MetadataInfo: &fs.MetadataInfo{
System: systemMetadataInfo,
@@ -733,6 +756,17 @@ two accounts.
Advanced: true,
Default: rwOff,
Examples: rwExamples,
}, {
Name: "metadata_enforce_expansive_access",
Help: `Whether the request should enforce expansive access rules.
From Feb 2026 this flag will be set by default so this flag can be used for
testing before then.
See: https://developers.google.com/workspace/drive/api/guides/limited-expansive-access
`,
Advanced: true,
Default: false,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
@@ -812,6 +846,7 @@ type Options struct {
MetadataOwner rwChoice `config:"metadata_owner"`
MetadataPermissions rwChoice `config:"metadata_permissions"`
MetadataLabels rwChoice `config:"metadata_labels"`
EnforceExpansiveAccess bool `config:"metadata_enforce_expansive_access"`
Enc encoder.MultiEncoder `config:"encoding"`
EnvAuth bool `config:"env_auth"`
}
@@ -1220,7 +1255,7 @@ func isLinkMimeType(mimeType string) bool {
// into a list of unique extensions with leading "." and a list of associated MIME types
func parseExtensions(extensionsIn ...string) (extensions, mimeTypes []string, err error) {
for _, extensionText := range extensionsIn {
for _, extension := range strings.Split(extensionText, ",") {
for extension := range strings.SplitSeq(extensionText, ",") {
extension = strings.ToLower(strings.TrimSpace(extension))
if extension == "" {
continue
@@ -1290,6 +1325,7 @@ func createOAuthClient(ctx context.Context, opt *Options, name string, m configm
return nil, fmt.Errorf("failed to create client from environment: %w", err)
}
} else {
oauthutil.SharedClientIDWarning(name, "Google Drive", "https://rclone.org/drive/#making-your-own-client-id", m)
oAuthClient, _, err = oauthutil.NewClientWithBaseClient(ctx, name, m, driveConfig, getClient(ctx, opt))
if err != nil {
return nil, fmt.Errorf("failed to create oauth client: %w", err)
@@ -1674,6 +1710,9 @@ func (f *Fs) newObjectWithExportInfo(
// If item MimeType is in the ExportFormats then it is a google doc
if !isDocument {
fs.Debugf(remote, "Ignoring unknown document type %q", info.MimeType)
gdocsWarnOnce.Do(func() {
fs.Logf(remote, "Skipping unexportable google document %q. Use --drive-show-all-gdocs to include them in server side copy and move", info.MimeType)
})
return nil, fs.ErrorObjectNotFound
}
if extension == "" {
@@ -1965,9 +2004,28 @@ func (f *Fs) findImportFormat(ctx context.Context, mimeType string) string {
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
return list.WithListP(ctx, dir, f)
}
// ListP lists the objects and directories of the Fs starting
// from dir non recursively into out.
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) error {
list := list.NewHelper(callback)
entriesAdded := 0
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return nil, err
return err
}
directoryID = actualID(directoryID)
@@ -1979,25 +2037,30 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return true
}
if entry != nil {
entries = append(entries, entry)
err = list.Add(entry)
if err != nil {
iErr = err
return true
}
entriesAdded++
}
return false
})
if err != nil {
return nil, err
return err
}
if iErr != nil {
return nil, iErr
return iErr
}
// If listing the root of a teamdrive and got no entries,
// double check we have access
if f.isTeamDrive && len(entries) == 0 && f.root == "" && dir == "" {
if f.isTeamDrive && entriesAdded == 0 && f.root == "" && dir == "" {
err = f.teamDriveOK(ctx)
if err != nil {
return nil, err
return err
}
}
return entries, nil
return list.Flush()
}
// listREntry is a task to be executed by a litRRunner
@@ -2376,12 +2439,40 @@ func (f *Fs) resolveShortcut(ctx context.Context, item *drive.File) (newItem *dr
return newItem, nil
}
// shortcutLoop reports whether a folder shortcut at remote pointing to
// the directory targetID would create a recursive loop.
//
// This is the case when targetID is the ID of remote's parent or any
// other ancestor directory, as descending into it would list an
// ancestor again forever. The ancestor IDs are looked up in the
// directory cache which is always populated for the parents of remote
// as they are listed before their children.
func (f *Fs) shortcutLoop(remote, targetID string) bool {
for dir := path.Dir(remote); ; dir = path.Dir(dir) {
if dir == "." {
dir = ""
}
if id, ok := f.dirCache.Get(dir); ok && actualID(id) == targetID {
return true
}
if dir == "" {
return false
}
}
}
// itemToDirEntry converts a drive.File to an fs.DirEntry.
// When the drive.File cannot be represented as an fs.DirEntry
// (nil, nil) is returned.
func (f *Fs) itemToDirEntry(ctx context.Context, remote string, item *drive.File) (entry fs.DirEntry, err error) {
switch {
case item.MimeType == driveFolderType:
// A folder shortcut pointing at one of its own ancestors
// would make rclone recurse forever, so drop it.
if isShortcutID(item.Id) && f.shortcutLoop(remote, actualID(item.Id)) {
fs.Errorf(remote, "Ignoring folder shortcut as it points to an ancestor directory creating a loop")
return nil, nil
}
// cache the directory ID for later lookups
f.dirCache.Put(remote, item.Id)
// cache the resource key for later lookups
@@ -2998,11 +3089,18 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
remote = remote[:len(remote)-len(ext)]
}
_, srcParentID, err := srcObj.fs.dirCache.FindPath(ctx, src.Remote(), false)
if err != nil {
return nil, err
// Find the ID of the parent to remove the file from.
var srcParentID string
if len(srcObj.parents) == 1 {
srcParentID = srcObj.parents[0]
} else {
var err error
_, srcParentID, err = srcObj.fs.dirCache.FindPath(ctx, src.Remote(), false)
if err != nil {
return nil, err
}
srcParentID = actualID(srcParentID)
}
srcParentID = actualID(srcParentID)
// Temporary Object under construction
dstInfo, err := f.createFileInfo(ctx, remote, src.ModTime(ctx))
@@ -3068,6 +3166,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
_, err = f.svc.Permissions.Create(id, permission).
Fields("").
SupportsAllDrives(true).
EnforceExpansiveAccess(f.opt.EnforceExpansiveAccess).
Context(ctx).Do()
return f.shouldRetry(ctx, err)
})
@@ -3372,9 +3471,11 @@ func (f *Fs) makeShortcut(ctx context.Context, srcPath string, dstFs *Fs, dstPat
return nil, fmt.Errorf("failed to find source dir: %w", err)
}
isDir = true
} else {
} else if do, ok := srcObj.(fs.IDer); ok {
// source was a file
srcID = srcObj.(*Object).id
srcID = do.ID()
} else {
return nil, fmt.Errorf("unknown source object: %T", srcObj)
}
srcID = actualID(srcID) // link to underlying object not to shortcut
@@ -3640,41 +3741,47 @@ func (f *Fs) rescue(ctx context.Context, dirID string, delete bool) (err error)
var commandHelp = []fs.CommandHelp{{
Name: "get",
Short: "Get command for fetching the drive config parameters",
Long: `This is a get command which will be used to fetch the various drive config parameters
Short: "Get command for fetching the drive config parameters.",
Long: `This is a get command which will be used to fetch the various drive config
parameters.
Usage Examples:
Usage examples:
rclone backend get drive: [-o service_account_file] [-o chunk_size]
rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
`,
` + "```console" + `
rclone backend get drive: [-o service_account_file] [-o chunk_size]
rclone rc backend/command command=get fs=drive: [-o service_account_file] [-o chunk_size]
` + "```",
Opts: map[string]string{
"chunk_size": "show the current upload chunk size",
"service_account_file": "show the current service account file",
"chunk_size": "Show the current upload chunk size.",
"service_account_file": "Show the current service account file.",
},
}, {
Name: "set",
Short: "Set command for updating the drive config parameters",
Long: `This is a set command which will be used to update the various drive config parameters
Short: "Set command for updating the drive config parameters.",
Long: `This is a set command which will be used to update the various drive config
parameters.
Usage Examples:
Usage examples:
rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
`,
` + "```console" + `
rclone backend set drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
rclone rc backend/command command=set fs=drive: [-o service_account_file=sa.json] [-o chunk_size=67108864]
` + "```",
Opts: map[string]string{
"chunk_size": "update the current upload chunk size",
"service_account_file": "update the current service account file",
"chunk_size": "Update the current upload chunk size.",
"service_account_file": "Update the current service account file.",
},
}, {
Name: "shortcut",
Short: "Create shortcuts from files or directories",
Short: "Create shortcuts from files or directories.",
Long: `This command creates shortcuts from files or directories.
Usage:
Usage examples:
rclone backend shortcut drive: source_item destination_shortcut
rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
` + "```console" + `
rclone backend shortcut drive: source_item destination_shortcut
rclone backend shortcut drive: source_item -o target=drive2: destination_shortcut
` + "```" + `
In the first example this creates a shortcut from the "source_item"
which can be a file or a directory to the "destination_shortcut". The
@@ -3684,90 +3791,100 @@ from "drive:"
In the second example this creates a shortcut from the "source_item"
relative to "drive:" to the "destination_shortcut" relative to
"drive2:". This may fail with a permission error if the user
authenticated with "drive2:" can't read files from "drive:".
`,
authenticated with "drive2:" can't read files from "drive:".`,
Opts: map[string]string{
"target": "optional target remote for the shortcut destination",
"target": "Optional target remote for the shortcut destination.",
},
}, {
Name: "drives",
Short: "List the Shared Drives available to this account",
Short: "List the Shared Drives available to this account.",
Long: `This command lists the Shared Drives (Team Drives) available to this
account.
Usage:
Usage example:
rclone backend [-o config] drives drive:
` + "```console" + `
rclone backend [-o config] drives drive:
` + "```" + `
This will return a JSON list of objects like this
This will return a JSON list of objects like this:
[
{
"id": "0ABCDEF-01234567890",
"kind": "drive#teamDrive",
"name": "My Drive"
},
{
"id": "0ABCDEFabcdefghijkl",
"kind": "drive#teamDrive",
"name": "Test Drive"
}
]
` + "```json" + `
[
{
"id": "0ABCDEF-01234567890",
"kind": "drive#teamDrive",
"name": "My Drive"
},
{
"id": "0ABCDEFabcdefghijkl",
"kind": "drive#teamDrive",
"name": "Test Drive"
}
]
` + "```" + `
With the -o config parameter it will output the list in a format
suitable for adding to a config file to make aliases for all the
drives found and a combined drive.
[My Drive]
type = alias
remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
` + "```ini" + `
[My Drive]
type = alias
remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:
[Test Drive]
type = alias
remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
[Test Drive]
type = alias
remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:
[AllDrives]
type = combine
upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
[AllDrives]
type = combine
upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"
` + "```" + `
Adding this to the rclone config file will cause those team drives to
be accessible with the aliases shown. Any illegal characters will be
substituted with "_" and duplicate names will have numbers suffixed.
It will also add a remote called AllDrives which shows all the shared
drives combined into one directory tree.
`,
drives combined into one directory tree.`,
}, {
Name: "untrash",
Short: "Untrash files and directories",
Short: "Untrash files and directories.",
Long: `This command untrashes all the files and directories in the directory
passed in recursively.
Usage:
Usage example:
` + "```console" + `
rclone backend untrash drive:directory
rclone backend --interactive untrash drive:directory subdir
` + "```" + `
This takes an optional directory to trash which make this easier to
use via the API.
rclone backend untrash drive:directory
rclone backend --interactive untrash drive:directory subdir
Use the --interactive/-i or --dry-run flag to see what would be restored before restoring it.
Use the --interactive/-i or --dry-run flag to see what would be restored before
restoring it.
Result:
{
"Untrashed": 17,
"Errors": 0
}
`,
` + "```json" + `
{
"Untrashed": 17,
"Errors": 0
}
` + "```",
}, {
Name: "copyid",
Short: "Copy files by ID",
Long: `This command copies files by ID
Short: "Copy files by ID.",
Long: `This command copies files by ID.
Usage:
Usage examples:
rclone backend copyid drive: ID path
rclone backend copyid drive: ID1 path1 ID2 path2
` + "```console" + `
rclone backend copyid drive: ID path
rclone backend copyid drive: ID1 path1 ID2 path2
` + "```" + `
It copies the drive file with ID given to the path (an rclone path which
will be passed internally to rclone copyto). The ID and path pairs can be
@@ -3780,17 +3897,19 @@ component will be used as the file name.
If the destination is a drive backend then server-side copying will be
attempted if possible.
Use the --interactive/-i or --dry-run flag to see what would be copied before copying.
`,
Use the --interactive/-i or --dry-run flag to see what would be copied before
copying.`,
}, {
Name: "moveid",
Short: "Move files by ID",
Long: `This command moves files by ID
Short: "Move files by ID.",
Long: `This command moves files by ID.
Usage:
Usage examples:
rclone backend moveid drive: ID path
rclone backend moveid drive: ID1 path1 ID2 path2
` + "```console" + `
rclone backend moveid drive: ID path
rclone backend moveid drive: ID1 path1 ID2 path2
` + "```" + `
It moves the drive file with ID given to the path (an rclone path which
will be passed internally to rclone moveto).
@@ -3802,58 +3921,65 @@ component will be used as the file name.
If the destination is a drive backend then server-side moving will be
attempted if possible.
Use the --interactive/-i or --dry-run flag to see what would be moved beforehand.
`,
Use the --interactive/-i or --dry-run flag to see what would be moved beforehand.`,
}, {
Name: "exportformats",
Short: "Dump the export formats for debug purposes",
Short: "Dump the export formats for debug purposes.",
}, {
Name: "importformats",
Short: "Dump the import formats for debug purposes",
Short: "Dump the import formats for debug purposes.",
}, {
Name: "query",
Short: "List files using Google Drive query language",
Long: `This command lists files based on a query
Short: "List files using Google Drive query language.",
Long: `This command lists files based on a query.
Usage:
Usage example:
` + "```console" + `
rclone backend query drive: query
` + "```" + `
rclone backend query drive: query
The query syntax is documented at [Google Drive Search query terms and
operators](https://developers.google.com/drive/api/guides/ref-search-terms).
For example:
rclone backend query drive: "'0ABc9DEFGHIJKLMNop0QRatUVW3X' in parents and name contains 'foo'"
` + "```console" + `
rclone backend query drive: "'0ABc9DEFGHIJKLMNop0QRatUVW3X' in parents and name contains 'foo'"
` + "```" + `
If the query contains literal ' or \ characters, these need to be escaped with
\ characters. "'" becomes "\'" and "\" becomes "\\\", for example to match a
file named "foo ' \.txt":
rclone backend query drive: "name = 'foo \' \\\.txt'"
` + "```console" + `
rclone backend query drive: "name = 'foo \' \\\.txt'"
` + "```" + `
The result is a JSON array of matches, for example:
[
{
"createdTime": "2017-06-29T19:58:28.537Z",
"id": "0AxBe_CDEF4zkGHI4d0FjYko2QkD",
"md5Checksum": "68518d16be0c6fbfab918be61d658032",
"mimeType": "text/plain",
"modifiedTime": "2024-02-02T10:40:02.874Z",
"name": "foo ' \\.txt",
"parents": [
"0BxAe_BCDE4zkFGZpcWJGek0xbzC"
],
"resourceKey": "0-ABCDEFGHIXJQpIGqBJq3MC",
"sha1Checksum": "8f284fa768bfb4e45d076a579ab3905ab6bfa893",
"size": "311",
"webViewLink": "https://drive.google.com/file/d/0AxBe_CDEF4zkGHI4d0FjYko2QkD/view?usp=drivesdk\u0026resourcekey=0-ABCDEFGHIXJQpIGqBJq3MC"
}
]`,
` + "```json" + `
[
{
"createdTime": "2017-06-29T19:58:28.537Z",
"id": "0AxBe_CDEF4zkGHI4d0FjYko2QkD",
"md5Checksum": "68518d16be0c6fbfab918be61d658032",
"mimeType": "text/plain",
"modifiedTime": "2024-02-02T10:40:02.874Z",
"name": "foo ' \\.txt",
"parents": [
"0BxAe_BCDE4zkFGZpcWJGek0xbzC"
],
"resourceKey": "0-ABCDEFGHIXJQpIGqBJq3MC",
"sha1Checksum": "8f284fa768bfb4e45d076a579ab3905ab6bfa893",
"size": "311",
"webViewLink": "https://drive.google.com/file/d/0AxBe_CDEF4zkGHI4d0FjYko2QkD/view?usp=drivesdk\u0026resourcekey=0-ABCDEFGHIXJQpIGqBJq3MC"
}
]
` + "```console",
}, {
Name: "rescue",
Short: "Rescue or delete any orphaned files",
Short: "Rescue or delete any orphaned files.",
Long: `This command rescues or deletes any orphaned files or directories.
Sometimes files can get orphaned in Google Drive. This means that they
@@ -3862,26 +3988,31 @@ are no longer in any folder in Google Drive.
This command finds those files and either rescues them to a directory
you specify or deletes them.
Usage:
This can be used in 3 ways.
First, list all orphaned files
First, list all orphaned files:
rclone backend rescue drive:
` + "```console" + `
rclone backend rescue drive:
` + "```" + `
Second rescue all orphaned files to the directory indicated
Second rescue all orphaned files to the directory indicated:
rclone backend rescue drive: "relative/path/to/rescue/directory"
` + "```console" + `
rclone backend rescue drive: "relative/path/to/rescue/directory"
` + "```" + `
e.g. To rescue all orphans to a directory called "Orphans" in the top level
E.g. to rescue all orphans to a directory called "Orphans" in the top level:
rclone backend rescue drive: Orphans
` + "```console" + `
rclone backend rescue drive: Orphans
` + "```" + `
Third delete all orphaned files to the trash
Third delete all orphaned files to the trash:
rclone backend rescue drive: -o delete
`,
` + "```console" + `
rclone backend rescue drive: -o delete
` + "```",
}}
// Command the backend to run a named command
@@ -4617,6 +4748,7 @@ var (
_ fs.PutUncheckeder = (*Fs)(nil)
_ fs.PublicLinker = (*Fs)(nil)
_ fs.ListRer = (*Fs)(nil)
_ fs.ListPer = (*Fs)(nil)
_ fs.MergeDirser = (*Fs)(nil)
_ fs.DirSetModTimer = (*Fs)(nil)
_ fs.MkdirMetadataer = (*Fs)(nil)
+146
View File
@@ -24,6 +24,7 @@ import (
"github.com/rclone/rclone/fs/sync"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -145,6 +146,34 @@ func TestInternalFindExportFormat(t *testing.T) {
}
}
func TestShortcutLoop(t *testing.T) {
f := &Fs{}
f.dirCache = dircache.New("", "root", f)
// root is FolderX with ID "X"
f.dirCache.Put("", "X")
// a real subfolder
f.dirCache.Put("sub", "sub-id")
// a nested subfolder reached via a shortcut, composite ID
f.dirCache.Put("sub/nested", joinID("nested-id", "shortcut-id"))
for _, test := range []struct {
name string
remote string
targetID string
want bool
}{
{"shortcut to the root itself", "loop", "X", true},
{"shortcut to the parent", "sub/loop", "sub-id", true},
{"shortcut to a grandparent", "sub/nested/loop", "X", true},
{"shortcut to a composite ancestor", "sub/nested/loop", "nested-id", true},
{"shortcut to an unrelated folder", "sub/loop", "other-id", false},
{"shortcut to a sibling", "sub/loop", "sub-id2", false},
} {
got := f.shortcutLoop(test.remote, test.targetID)
assert.Equal(t, test.want, got, test.name)
}
}
func TestMimeTypesToExtension(t *testing.T) {
for mimeType, extension := range _mimeTypeToExtension {
extensions, err := mime.ExtensionsByType(mimeType)
@@ -643,6 +672,121 @@ func (f *Fs) InternalTestAgeQuery(t *testing.T) {
assert.Contains(t, subFs.lastQuery, timeQuery)
}
// TestIntegration/FsMkdir/FsPutFiles/Internal/SingleQuoteFolder
func (f *Fs) InternalTestSingleQuoteFolder(t *testing.T) {
ctx := context.Background()
// Test various folder names containing single quotes
for _, name := range []string{
"'",
"''",
"'a'",
"it's a test",
} {
t.Run(name, func(t *testing.T) {
dir := "singleQuoteTest/" + name
err := f.Mkdir(ctx, dir)
require.NoError(t, err)
defer func() {
err := f.Rmdir(ctx, dir)
assert.NoError(t, err)
}()
entries, err := f.List(ctx, "singleQuoteTest")
require.NoError(t, err)
found := false
for _, entry := range entries {
if entry.Remote() == dir {
found = true
break
}
}
assert.True(t, found, "directory %q not found in listing", name)
})
}
err := f.Rmdir(ctx, "singleQuoteTest")
assert.NoError(t, err)
}
// TestIntegration/FsMkdir/FsPutFiles/Internal/MoveDuplicateParent
//
// Check that moving a file removes it from its real parent even when there
// are two directories with the same name and the directory cache resolves the
// path to the wrong duplicate. Otherwise the file ends up with two parents
// which fails on shared drives with a teamDrivesParentLimit error (#9472).
func (f *Fs) InternalTestMoveDuplicateParent(t *testing.T) {
ctx := context.Background()
if f.Features().Move == nil {
t.Skip("Move not supported")
}
rootID, err := f.dirCache.RootID(ctx, false)
require.NoError(t, err)
const dupName = "dup9472"
const dstName = "dst9472"
const fileName = "michael.txt"
// Create the first "dup9472" directory - this is the one the directory
// cache will resolve the path to.
dirA, err := f.createDir(ctx, rootID, dupName, nil)
require.NoError(t, err)
f.dirCache.Put(dupName, dirA.Id)
// Create a second directory with the SAME name and parent: a duplicate
// the directory cache does not know about.
dirB, err := f.createDir(ctx, rootID, dupName, nil)
require.NoError(t, err)
// Destination directory
dirDst, err := f.createDir(ctx, rootID, dstName, nil)
require.NoError(t, err)
defer func() {
_ = f.delete(ctx, dirA.Id, false)
_ = f.delete(ctx, dirB.Id, false)
_ = f.delete(ctx, dirDst.Id, false)
f.dirCache.Flush()
}()
// Upload a file directly into dirB (the duplicate the cache doesn't know).
createInfo := &drive.File{
Name: fileName,
Parents: []string{dirB.Id},
}
var fileInfo *drive.File
err = f.pacer.Call(func() (bool, error) {
fileInfo, err = f.svc.Files.Create(createInfo).
Media(strings.NewReader("9472 duplicate parent test")).
Fields(f.getFileFields(ctx)).
SupportsAllDrives(true).
Context(ctx).Do()
return f.shouldRetry(ctx, err)
})
require.NoError(t, err)
require.Equal(t, []string{dirB.Id}, fileInfo.Parents)
// Sanity check: the directory cache resolves the path to dirA, not the
// dirB the file actually lives in.
gotDirID, err := f.dirCache.FindDir(ctx, dupName, false)
require.NoError(t, err)
require.Equal(t, dirA.Id, gotDirID, "directory cache should resolve to the wrong duplicate")
// Build the source object pointing at the file in dirB and move it.
srcObj, err := f.newObjectWithInfo(ctx, dupName+"/"+fileName, fileInfo)
require.NoError(t, err)
dstObj, err := f.Move(ctx, srcObj, dstName+"/"+fileName)
require.NoError(t, err)
// The moved file must have exactly one parent: the destination.
movedInfo, err := f.getFile(ctx, dstObj.(fs.IDer).ID(), "id,parents")
require.NoError(t, err)
assert.Equal(t, []string{dirDst.Id}, movedInfo.Parents)
}
func (f *Fs) InternalTest(t *testing.T) {
// These tests all depend on each other so run them as nested tests
t.Run("DocumentImport", func(t *testing.T) {
@@ -662,6 +806,8 @@ func (f *Fs) InternalTest(t *testing.T) {
t.Run("CopyOrMoveID", f.InternalTestCopyOrMoveID)
t.Run("Query", f.InternalTestQuery)
t.Run("AgeQuery", f.InternalTestAgeQuery)
t.Run("SingleQuoteFolder", f.InternalTestSingleQuoteFolder)
t.Run("MoveDuplicateParent", f.InternalTestMoveDuplicateParent)
t.Run("ShouldRetry", f.InternalTestShouldRetry)
}
+2 -2
View File
@@ -149,6 +149,7 @@ func (f *Fs) setPermissions(ctx context.Context, info *drive.File, permissions [
_, err := f.svc.Permissions.Create(info.Id, perm).
SupportsAllDrives(true).
SendNotificationEmail(false).
EnforceExpansiveAccess(f.opt.EnforceExpansiveAccess).
Context(ctx).Do()
return f.shouldRetry(ctx, err)
})
@@ -386,7 +387,6 @@ func (o *baseObject) parseMetadata(ctx context.Context, info *drive.File) (err e
g.SetLimit(o.fs.ci.Checkers)
var mu sync.Mutex // protect the info.Permissions from concurrent writes
for _, permissionID := range info.PermissionIds {
permissionID := permissionID
g.Go(func() error {
// must fetch the team drive ones individually to check the inherited flag
perm, inherited, err := o.fs.getPermission(gCtx, actualID(info.Id), permissionID, !o.fs.isTeamDrive)
@@ -484,6 +484,7 @@ func (f *Fs) setOwner(ctx context.Context, info *drive.File, owner string) (err
SupportsAllDrives(true).
TransferOwnership(true).
// SendNotificationEmail(false). - required apparently!
EnforceExpansiveAccess(f.opt.EnforceExpansiveAccess).
Context(ctx).Do()
return f.shouldRetry(ctx, err)
})
@@ -520,7 +521,6 @@ func (f *Fs) updateMetadata(ctx context.Context, updateInfo *drive.File, meta fs
}
// merge metadata into request and user metadata
for k, v := range meta {
k, v := k, v
// parse a boolean from v and write into out
parseBool := func(out *bool) error {
b, err := strconv.ParseBool(v)
+223 -67
View File
@@ -47,6 +47,7 @@ import (
"github.com/rclone/rclone/fs/config/obscure"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/list"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/lib/batcher"
"github.com/rclone/rclone/lib/encoder"
@@ -149,14 +150,20 @@ var (
// Gets an oauth config with the right scopes
func getOauthConfig(m configmap.Mapper) *oauthutil.Config {
impersonate, _ := m.Get("impersonate")
impersonateAdmin, _ := m.Get("impersonate_admin")
// If not impersonating, use standard scopes
if impersonate, _ := m.Get("impersonate"); impersonate == "" {
if impersonate == "" && impersonateAdmin == "" {
return dropboxConfig
}
// Make a copy of the config
config := *dropboxConfig
// Make a copy of the scopes with extra scopes requires appended
config.Scopes = append(config.Scopes, "members.read", "team_data.member")
config.Scopes = append(config.Scopes, "team_data.member")
if impersonate != "" {
config.Scopes = append(config.Scopes, "members.read")
}
return &config
}
@@ -205,6 +212,29 @@ You will have to use your own App (setting your own client_id and
client_secret) to use this option as currently rclone's default set of
permissions doesn't include "members.read". This can be added once
v1.55 or later is in use everywhere.
`,
Default: "",
Advanced: true,
Sensitive: true,
}, {
Name: "impersonate_admin",
Help: `Team admin ID to use when performing actions as a team administrator.
This sets the Dropbox-API-Select-Admin header with the given team
member ID (for example "dbmid:...").
This takes a team member ID directly rather than an email address.
Note that if you want to use impersonate_admin, you should make sure this
flag is set when running "rclone config" as this will cause rclone to
request the "team_data.member" scope which it won't normally.
Using the "team_data.member" scope will require a Dropbox Team Admin
to approve during the OAuth flow.
You will have to use your own App (setting your own client_id and
client_secret) to use this option as currently rclone's default set of
permissions doesn't include "team_data.member".
`,
Default: "",
Advanced: true,
@@ -237,6 +267,36 @@ See also --dropbox-root-namespace for an alternative way to work with shared
folders.`,
Default: false,
Advanced: true,
}, {
Name: "skip_shared_folders",
Help: `Instructs rclone to skip all shared folders.
When set, any folder that is a shared folder mount point will be
excluded from directory listings, regardless of ownership.
This is useful if you prefer to back up shared folders separately
using a separate remote configured with the shared folder namespace.`,
Default: false,
Advanced: true,
}, {
Name: "skip_unowned_folders",
Help: `Instructs rclone to skip shared folders not owned by the current user.
When set, any folder that is a shared folder mount point and not
owned by the current user will be excluded from directory listings.
This is useful when backing up multiple Dropbox accounts that share
common folders, to avoid duplicating the shared data across accounts.
Note: In Dropbox Business, 'Team Folders' are owned by the Team.
For standard team members, these folders evaluate as 'unowned'
(editor/viewer access) and will be excluded by this flag. To back up
Team Folders, do not use this flag or run the backup using a Team Admin
account.
If --dropbox-skip-shared-folders is also enabled, this flag has no effect.
This makes an extra API call per shared folder mount point.`,
Default: false,
Advanced: true,
}, {
Name: "pacer_min_sleep",
Default: defaultMinSleep,
@@ -301,20 +361,23 @@ will fail to download them.
// Options defines the configuration for this backend
type Options struct {
ChunkSize fs.SizeSuffix `config:"chunk_size"`
Impersonate string `config:"impersonate"`
SharedFiles bool `config:"shared_files"`
SharedFolders bool `config:"shared_folders"`
BatchMode string `config:"batch_mode"`
BatchSize int `config:"batch_size"`
BatchTimeout fs.Duration `config:"batch_timeout"`
AsyncBatch bool `config:"async_batch"`
PacerMinSleep fs.Duration `config:"pacer_min_sleep"`
Enc encoder.MultiEncoder `config:"encoding"`
RootNsid string `config:"root_namespace"`
ExportFormats fs.CommaSepList `config:"export_formats"`
SkipExports bool `config:"skip_exports"`
ShowAllExports bool `config:"show_all_exports"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
Impersonate string `config:"impersonate"`
ImpersonateAdmin string `config:"impersonate_admin"`
SharedFiles bool `config:"shared_files"`
SharedFolders bool `config:"shared_folders"`
SkipSharedFolders bool `config:"skip_shared_folders"`
SkipUnownedFolders bool `config:"skip_unowned_folders"`
BatchMode string `config:"batch_mode"`
BatchSize int `config:"batch_size"`
BatchTimeout fs.Duration `config:"batch_timeout"`
AsyncBatch bool `config:"async_batch"`
PacerMinSleep fs.Duration `config:"pacer_min_sleep"`
Enc encoder.MultiEncoder `config:"encoding"`
RootNsid string `config:"root_namespace"`
ExportFormats fs.CommaSepList `config:"export_formats"`
SkipExports bool `config:"skip_exports"`
ShowAllExports bool `config:"show_all_exports"`
}
// Fs represents a remote dropbox server
@@ -536,6 +599,10 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
cfg.AsMemberID = memberIDs[0].MemberInfo.Profile.MemberProfile.TeamMemberId
}
if opt.ImpersonateAdmin != "" {
cfg.AsAdminID = opt.ImpersonateAdmin
}
f.srv = files.New(cfg)
f.svc = files.New(ucfg)
f.sharing = sharing.New(cfg)
@@ -834,7 +901,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
// listSharedFolders lists all available shared folders mounted and not mounted
// we'll need the id later so we have to return them in original format
func (f *Fs) listSharedFolders(ctx context.Context) (entries fs.DirEntries, err error) {
func (f *Fs) listSharedFolders(ctx context.Context, callback func(fs.DirEntry) error) (err error) {
started := false
var res *sharing.ListFoldersResult
for {
@@ -847,7 +914,7 @@ func (f *Fs) listSharedFolders(ctx context.Context) (entries fs.DirEntries, err
return shouldRetry(ctx, err)
})
if err != nil {
return nil, err
return err
}
started = true
} else {
@@ -859,15 +926,15 @@ func (f *Fs) listSharedFolders(ctx context.Context) (entries fs.DirEntries, err
return shouldRetry(ctx, err)
})
if err != nil {
return nil, fmt.Errorf("list continue: %w", err)
return fmt.Errorf("list continue: %w", err)
}
}
for _, entry := range res.Entries {
leaf := f.opt.Enc.ToStandardName(entry.Name)
d := fs.NewDir(leaf, time.Time{}).SetID(entry.SharedFolderId)
entries = append(entries, d)
err = callback(d)
if err != nil {
return nil, err
return err
}
}
if res.Cursor == "" {
@@ -875,21 +942,25 @@ func (f *Fs) listSharedFolders(ctx context.Context) (entries fs.DirEntries, err
}
}
return entries, nil
return nil
}
// findSharedFolder find the id for a given shared folder name
// somewhat annoyingly there is no endpoint to query a shared folder by it's name
// so our only option is to iterate over all shared folders
func (f *Fs) findSharedFolder(ctx context.Context, name string) (id string, err error) {
entries, err := f.listSharedFolders(ctx)
if err != nil {
return "", err
}
for _, entry := range entries {
errFoundFile := errors.New("found file")
err = f.listSharedFolders(ctx, func(entry fs.DirEntry) error {
if entry.(*fs.Dir).Remote() == name {
return entry.(*fs.Dir).ID(), nil
id = entry.(*fs.Dir).ID()
return errFoundFile
}
return nil
})
if errors.Is(err, errFoundFile) {
return id, nil
} else if err != nil {
return "", err
}
return "", fs.ErrorDirNotFound
}
@@ -908,7 +979,7 @@ func (f *Fs) mountSharedFolder(ctx context.Context, id string) error {
// listReceivedFiles lists shared the user as access to (note this means individual
// files not files contained in shared folders)
func (f *Fs) listReceivedFiles(ctx context.Context) (entries fs.DirEntries, err error) {
func (f *Fs) listReceivedFiles(ctx context.Context, callback func(fs.DirEntry) error) (err error) {
started := false
var res *sharing.ListFilesResult
for {
@@ -921,7 +992,7 @@ func (f *Fs) listReceivedFiles(ctx context.Context) (entries fs.DirEntries, err
return shouldRetry(ctx, err)
})
if err != nil {
return nil, err
return err
}
started = true
} else {
@@ -933,39 +1004,45 @@ func (f *Fs) listReceivedFiles(ctx context.Context) (entries fs.DirEntries, err
return shouldRetry(ctx, err)
})
if err != nil {
return nil, fmt.Errorf("list continue: %w", err)
return fmt.Errorf("list continue: %w", err)
}
}
for _, entry := range res.Entries {
fmt.Printf("%+v\n", entry)
entryPath := entry.Name
o := &Object{
fs: f,
url: entry.PreviewUrl,
remote: entryPath,
modTime: *entry.TimeInvited,
modTime: time.Time(*entry.TimeInvited),
}
if err != nil {
return nil, err
return err
}
err = callback(o)
if err != nil {
return err
}
entries = append(entries, o)
}
if res.Cursor == "" {
break
}
}
return entries, nil
return nil
}
func (f *Fs) findSharedFile(ctx context.Context, name string) (o *Object, err error) {
files, err := f.listReceivedFiles(ctx)
if err != nil {
return nil, err
}
for _, entry := range files {
errFoundFile := errors.New("found file")
err = f.listReceivedFiles(ctx, func(entry fs.DirEntry) error {
if entry.(*Object).remote == name {
return entry.(*Object), nil
o = entry.(*Object)
return errFoundFile
}
return nil
})
if errors.Is(err, errFoundFile) {
return o, nil
} else if err != nil {
return nil, err
}
return nil, fs.ErrorObjectNotFound
}
@@ -980,11 +1057,37 @@ func (f *Fs) findSharedFile(ctx context.Context, name string) (o *Object, err er
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
return list.WithListP(ctx, dir, f)
}
// ListP lists the objects and directories of the Fs starting
// from dir non recursively into out.
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) (err error) {
list := list.NewHelper(callback)
if f.opt.SharedFiles {
return f.listReceivedFiles(ctx)
err := f.listReceivedFiles(ctx, list.Add)
if err != nil {
return err
}
return list.Flush()
}
if f.opt.SharedFolders {
return f.listSharedFolders(ctx)
err := f.listSharedFolders(ctx, list.Add)
if err != nil {
return err
}
return list.Flush()
}
root := f.slashRoot
@@ -1014,7 +1117,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
err = fs.ErrorDirNotFound
}
}
return nil, err
return err
}
started = true
} else {
@@ -1026,7 +1129,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return shouldRetry(ctx, err)
})
if err != nil {
return nil, fmt.Errorf("list continue: %w", err)
return fmt.Errorf("list continue: %w", err)
}
}
for _, entry := range res.Entries {
@@ -1050,15 +1153,41 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
leaf := f.opt.Enc.ToStandardName(path.Base(entryPath))
remote := path.Join(dir, leaf)
if folderInfo != nil {
if folderInfo.SharingInfo != nil && folderInfo.SharingInfo.SharedFolderId != "" {
if f.opt.SkipSharedFolders {
fs.Debugf(remote, "Skipping shared folder")
continue
}
if f.opt.SkipUnownedFolders {
var sfMeta *sharing.SharedFolderMetadata
err = f.pacer.Call(func() (bool, error) {
var apiErr error
sfMeta, apiErr = f.sharing.GetFolderMetadata(sharing.NewGetMetadataArgs(folderInfo.SharingInfo.SharedFolderId))
return shouldRetry(ctx, apiErr)
})
if err != nil {
fs.Errorf(remote, "Failed to get shared folder metadata (defaulting to include): %v", err)
} else if sfMeta != nil && sfMeta.AccessType != nil && sfMeta.AccessType.Tag != sharing.AccessLevelOwner {
fs.Debugf(remote, "Skipping unowned shared folder")
continue
}
}
}
d := fs.NewDir(remote, time.Time{}).SetID(folderInfo.Id)
entries = append(entries, d)
err = list.Add(d)
if err != nil {
return err
}
} else if fileInfo != nil {
o, err := f.newObjectWithInfo(ctx, remote, fileInfo)
if err != nil {
return nil, err
return err
}
if o.(*Object).exportType.listable() {
entries = append(entries, o)
err = list.Add(o)
if err != nil {
return err
}
}
}
}
@@ -1066,7 +1195,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
break
}
}
return entries, nil
return list.Flush()
}
// Put the object
@@ -1138,15 +1267,11 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) (err error)
encRoot := f.opt.Enc.FromStandardPath(root)
if check {
// check directory exists
_, err = f.getDirMetadata(ctx, root)
if err != nil {
return fmt.Errorf("Rmdir: %w", err)
}
// check directory empty
// ListFolder reports a missing path and a path that is a file, so it
// checks that the directory exists and is empty in a single request.
arg := files.NewListFolderArg(encRoot)
arg.Recursive = false
arg.Limit = 1
if root == "/" {
arg.Path = "" // Specify root folder as empty string
}
@@ -1156,9 +1281,20 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) (err error)
return shouldRetry(ctx, err)
})
if err != nil {
switch e := err.(type) {
case files.ListFolderAPIError:
if e.EndpointError != nil && e.EndpointError.Path != nil {
switch e.EndpointError.Path.Tag {
case files.LookupErrorNotFound:
err = fs.ErrorDirNotFound
case files.LookupErrorNotFolder:
err = fs.ErrorIsFile
}
}
}
return fmt.Errorf("Rmdir: %w", err)
}
if len(res.Entries) != 0 {
if len(res.Entries) != 0 || res.HasMore {
return errors.New("directory not empty")
}
}
@@ -1286,6 +1422,16 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
var result *files.RelocationResult
err = f.pacer.Call(func() (bool, error) {
result, err = f.srv.MoveV2(&arg)
switch e := err.(type) {
case files.MoveV2APIError:
// There seems to be a bit of eventual consistency here which causes this to
// fail on just created objects
// See: https://github.com/rclone/rclone/issues/8881
if e.EndpointError != nil && e.EndpointError.FromLookup != nil && e.EndpointError.FromLookup.Tag == files.LookupErrorNotFound {
fs.Debugf(srcObj, "Retrying move on %v error", err)
return true, err
}
}
return shouldRetry(ctx, err)
})
if err != nil {
@@ -1323,7 +1469,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
},
}
if expire < fs.DurationOff {
expiryTime := time.Now().Add(time.Duration(expire)).UTC().Round(time.Second)
expiryTime := dropbox.DBXTime(time.Now().Add(time.Duration(expire)).UTC().Round(time.Second))
createArg.Settings.Expires = &expiryTime
}
@@ -1446,9 +1592,9 @@ func (f *Fs) About(ctx context.Context) (usage *fs.Usage, err error) {
}
}
usage = &fs.Usage{
Total: fs.NewUsageValue(int64(total)), // quota of bytes that can be used
Used: fs.NewUsageValue(int64(used)), // bytes in use
Free: fs.NewUsageValue(int64(total - used)), // bytes which can be uploaded before reaching the quota
Total: fs.NewUsageValue(total), // quota of bytes that can be used
Used: fs.NewUsageValue(used), // bytes in use
Free: fs.NewUsageValue(total - used), // bytes which can be uploaded before reaching the quota
}
return usage, nil
}
@@ -1737,7 +1883,7 @@ func (o *Object) setMetadataForExport(info *files.FileMetadata) {
func (o *Object) setMetadataFromEntry(info *files.FileMetadata) error {
o.id = info.Id
o.bytes = int64(info.Size)
o.modTime = info.ClientModified
o.modTime = time.Time(info.ClientModified)
o.hash = info.ContentHash
if !info.IsDownloadable {
@@ -1896,12 +2042,21 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
// write chunks
in := readers.NewCountingReader(in0)
buf := make([]byte, int(chunkSize))
bufSize := chunkSize
if size >= 0 && size < bufSize {
bufSize = size
}
buf := make([]byte, int(bufSize))
cursor := files.UploadSessionCursor{
SessionId: res.SessionId,
Offset: 0,
}
appendArg := files.UploadSessionAppendArg{Cursor: &cursor}
appendArg := files.UploadSessionAppendArg{
Cursor: &cursor,
// A known-size upload which fits in a single chunk can close the
// session with its only append, saving an empty append request
Close: size >= 0 && size <= chunkSize,
}
for currentChunk := 1; ; currentChunk++ {
cursor.Offset = in.BytesRead()
@@ -2031,7 +2186,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
commitInfo := files.NewCommitInfo(o.fs.opt.Enc.FromStandardPath(o.remotePath()))
commitInfo.Mode.Tag = "overwrite"
// The Dropbox API only accepts timestamps in UTC with second precision.
clientModified := src.ModTime(ctx).UTC().Round(time.Second)
clientModified := dropbox.DBXTime(src.ModTime(ctx).UTC().Round(time.Second))
commitInfo.ClientModified = &clientModified
// Don't attempt to create filenames that are too long
if cErr := checkPathLength(commitInfo.Path); cErr != nil {
@@ -2057,7 +2212,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// This will only happen if we are uploading async batches
if entry == nil {
o.bytes = size
o.modTime = *commitInfo.ClientModified
o.modTime = time.Time(*commitInfo.ClientModified)
o.hash = "" // we don't have this
return nil
}
@@ -2087,6 +2242,7 @@ var (
_ fs.Mover = (*Fs)(nil)
_ fs.PublicLinker = (*Fs)(nil)
_ fs.DirMover = (*Fs)(nil)
_ fs.ListPer = (*Fs)(nil)
_ fs.Abouter = (*Fs)(nil)
_ fs.Shutdowner = &Fs{}
_ fs.Object = (*Object)(nil)
+14 -1
View File
@@ -3,12 +3,25 @@ package api
import "encoding/json"
// MultipartInitResponse represents the response from multipart/init.
type MultipartInitResponse struct {
Status int `json:"status"`
Msg string `json:"msg"`
Result struct {
UploadID string `json:"upload_id"`
SessID string `json:"sess_id"`
Server string `json:"server"`
FolderID int64 `json:"folder_id"`
ObjectPath string `json:"object_path"`
} `json:"result"`
}
// CreateFolderResponse represents the response for creating a folder.
type CreateFolderResponse struct {
Status int `json:"status"`
Msg string `json:"msg"`
Result struct {
FldID interface{} `json:"fld_id"`
FldID any `json:"fld_id"`
} `json:"result"`
}
+39 -36
View File
@@ -21,6 +21,11 @@ import (
"github.com/rclone/rclone/lib/rest"
)
const (
defaultUploadCutoff = fs.SizeSuffix(500 * 1024 * 1024)
defaultChunkSize = fs.SizeSuffix(64 * 1024 * 1024)
)
// Register the backend with Rclone
func init() {
fs.Register(&fs.RegInfo{
@@ -33,6 +38,17 @@ func init() {
Required: true,
Sensitive: true,
},
{
Name: "upload_cutoff",
Help: "Cutoff for switching to chunked upload. Any files larger than this will be uploaded in chunks of chunk_size.",
Default: defaultUploadCutoff,
Advanced: true,
}, {
Name: "chunk_size",
Help: "Chunk size to use for uploading. Used for multipart uploads.",
Default: defaultChunkSize,
Advanced: true,
},
{
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
@@ -72,8 +88,10 @@ func init() {
// Options defines the configuration for the FileLu backend
type Options struct {
Key string `config:"key"`
Enc encoder.MultiEncoder `config:"encoding"`
Key string `config:"key"`
Enc encoder.MultiEncoder `config:"encoding"`
UploadCutoff fs.SizeSuffix `config:"upload_cutoff"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
}
// Fs represents the FileLu file system
@@ -189,7 +207,6 @@ func (f *Fs) Purge(ctx context.Context, dir string) error {
return f.deleteFolder(ctx, fullPath)
}
// List returns a list of files and folders
// List returns a list of files and folders for the given directory
func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
// Compose full path for API call
@@ -205,38 +222,36 @@ func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
return nil, err
}
fldMap := map[string]bool{}
for _, folder := range result.Result.Folders {
fldMap[folder.FldID.String()] = true
if f.root == "" && dir == "" && strings.Contains(folder.Path, "/") {
continue
}
paths := strings.Split(folder.Path, fullPath+"/")
remote := paths[0]
if len(paths) > 1 {
remote = paths[1]
folderPath := strings.Trim(folder.Path, "/")
root := strings.Trim(f.root, "/")
remotePathWithoutRoot := folderPath
if root != "" {
if remotePathWithoutRoot == root {
continue
}
remotePathWithoutRoot = strings.TrimPrefix(remotePathWithoutRoot, root+"/")
}
if strings.Contains(remote, "/") {
remotePathWithoutRoot = strings.TrimPrefix(remotePathWithoutRoot, "/")
if remotePathWithoutRoot == "" {
continue
}
pathsWithoutRoot := strings.Split(folder.Path, "/"+f.root+"/")
remotePathWithoutRoot := pathsWithoutRoot[0]
if len(pathsWithoutRoot) > 1 {
remotePathWithoutRoot = pathsWithoutRoot[1]
}
remotePathWithoutRoot = strings.TrimPrefix(remotePathWithoutRoot, "/")
entries = append(entries, fs.NewDir(remotePathWithoutRoot, time.Now()))
}
for _, file := range result.Result.Files {
if _, ok := fldMap[file.FldID.String()]; ok {
continue
}
remote := path.Join(dir, file.Name)
// trim leading slashes
remote = strings.TrimPrefix(remote, "/")
obj := &Object{
fs: f,
remote: remote,
@@ -250,23 +265,11 @@ func (f *Fs) List(ctx context.Context, dir string) (fs.DirEntries, error) {
// Put uploads a file directly to the destination folder in the FileLu storage system.
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
if src.Size() == 0 {
return nil, fs.ErrorCantUploadEmptyFiles
o := &Object{
fs: f,
remote: src.Remote(),
}
err := f.uploadFile(ctx, in, src.Remote())
if err != nil {
return nil, err
}
newObject := &Object{
fs: f,
remote: src.Remote(),
size: src.Size(),
modTime: src.ModTime(ctx),
}
fs.Infof(f, "Put: Successfully uploaded new file %q", src.Remote())
return newObject, nil
return o, o.Update(ctx, in, src, options...)
}
// Move moves the file to the specified location
+127 -153
View File
@@ -1,9 +1,7 @@
package filelu
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
@@ -16,40 +14,82 @@ import (
"github.com/rclone/rclone/lib/rest"
)
// multipartInit starts a new multipart upload and returns server details.
func (f *Fs) multipartInit(ctx context.Context, folderPath, filename string) (*api.MultipartInitResponse, error) {
opts := rest.Opts{
Method: "GET",
Path: "/multipart/init",
Parameters: url.Values{
"key": {f.opt.Key},
"filename": {filename},
"folder_path": {folderPath},
},
}
var result api.MultipartInitResponse
err := f.pacer.Call(func() (bool, error) {
_, err := f.srv.CallJSON(ctx, &opts, nil, &result)
return fserrors.ShouldRetry(err), err
})
if err != nil {
return nil, err
}
if result.Status != 200 {
return nil, fmt.Errorf("multipart init error: %s", result.Msg)
}
return &result, nil
}
// completeMultipart finalizes the multipart upload on the file server.
func (f *Fs) completeMultipart(ctx context.Context, server string, uploadID string, sessID string, objectPath string) error {
req, err := http.NewRequestWithContext(ctx, "POST", server, nil)
if err != nil {
return err
}
req.Header.Set("X-RC-Upload-Id", uploadID)
req.Header.Set("X-Sess-ID", sessID)
req.Header.Set("X-Object-Path", objectPath)
resp, err := f.client.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != 202 {
body, _ := io.ReadAll(resp.Body)
return fmt.Errorf("completeMultipart failed %d: %s", resp.StatusCode, string(body))
}
return nil
}
// createFolder creates a folder at the specified path.
func (f *Fs) createFolder(ctx context.Context, dirPath string) (*api.CreateFolderResponse, error) {
encodedDir := f.fromStandardPath(dirPath)
apiURL := fmt.Sprintf("%s/folder/create?folder_path=%s&key=%s",
f.endpoint,
url.QueryEscape(encodedDir),
url.QueryEscape(f.opt.Key), // assuming f.opt.Key is the correct field
)
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
if err != nil {
return nil, fmt.Errorf("failed to create request: %w", err)
opts := rest.Opts{
Method: "GET",
Path: "/folder/create",
Parameters: url.Values{
"folder_path": {encodedDir},
"key": {f.opt.Key},
},
}
var resp *http.Response
result := api.CreateFolderResponse{}
err = f.pacer.Call(func() (bool, error) {
var innerErr error
resp, innerErr = f.client.Do(req)
return fserrors.ShouldRetry(innerErr), innerErr
var result api.CreateFolderResponse
err := f.pacer.Call(func() (bool, error) {
_, err := f.srv.CallJSON(ctx, &opts, nil, &result)
return fserrors.ShouldRetry(err), err
})
if err != nil {
return nil, fmt.Errorf("request failed: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
err = json.NewDecoder(resp.Body).Decode(&result)
if err != nil {
return nil, fmt.Errorf("error decoding response: %w", err)
}
if result.Status != 200 {
return nil, fmt.Errorf("error: %s", result.Msg)
}
@@ -61,44 +101,29 @@ func (f *Fs) createFolder(ctx context.Context, dirPath string) (*api.CreateFolde
// getFolderList List both files and folders in a directory.
func (f *Fs) getFolderList(ctx context.Context, path string) (*api.FolderListResponse, error) {
encodedDir := f.fromStandardPath(path)
apiURL := fmt.Sprintf("%s/folder/list?folder_path=%s&key=%s",
f.endpoint,
url.QueryEscape(encodedDir),
url.QueryEscape(f.opt.Key),
)
var body []byte
opts := rest.Opts{
Method: "GET",
Path: "/folder/list",
Parameters: url.Values{
"folder_path": {encodedDir},
"key": {f.opt.Key},
},
}
var response api.FolderListResponse
err := f.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
if err != nil {
return false, fmt.Errorf("failed to create request: %w", err)
}
resp, err := f.client.Do(req)
_, err := f.srv.CallJSON(ctx, &opts, nil, &response)
if err != nil {
return shouldRetry(err), fmt.Errorf("failed to list directory: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
body, err = io.ReadAll(resp.Body)
if err != nil {
return false, fmt.Errorf("error reading response body: %w", err)
}
return shouldRetryHTTP(resp.StatusCode), nil
return false, nil
})
if err != nil {
return nil, err
}
var response api.FolderListResponse
if err := json.NewDecoder(bytes.NewReader(body)).Decode(&response); err != nil {
return nil, fmt.Errorf("error decoding response: %w", err)
}
if response.Status != 200 {
if strings.Contains(response.Msg, "Folder not found") {
return nil, fs.ErrorDirNotFound
@@ -115,42 +140,28 @@ func (f *Fs) getFolderList(ctx context.Context, path string) (*api.FolderListRes
}
return &response, nil
}
// deleteFolder deletes a folder at the specified path.
func (f *Fs) deleteFolder(ctx context.Context, fullPath string) error {
fullPath = f.fromStandardPath(fullPath)
deleteURL := fmt.Sprintf("%s/folder/delete?folder_path=%s&key=%s",
f.endpoint,
url.QueryEscape(fullPath),
url.QueryEscape(f.opt.Key),
)
opts := rest.Opts{
Method: "GET",
Path: "/folder/delete",
Parameters: url.Values{
"folder_path": {fullPath},
"key": {f.opt.Key},
},
}
delResp := api.DeleteFolderResponse{}
err := f.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", deleteURL, nil)
if err != nil {
return false, err
}
resp, err := f.client.Do(req)
_, err := f.srv.CallJSON(ctx, &opts, nil, &delResp)
if err != nil {
return fserrors.ShouldRetry(err), err
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
body, err := io.ReadAll(resp.Body)
if err != nil {
return false, err
}
if err := json.Unmarshal(body, &delResp); err != nil {
return false, fmt.Errorf("error decoding delete response: %w", err)
}
if delResp.Status != 200 {
return false, fmt.Errorf("delete error: %s", delResp.Msg)
}
@@ -167,38 +178,27 @@ func (f *Fs) deleteFolder(ctx context.Context, fullPath string) error {
// getDirectLink of file from FileLu to download.
func (f *Fs) getDirectLink(ctx context.Context, filePath string) (string, int64, error) {
filePath = f.fromStandardPath(filePath)
apiURL := fmt.Sprintf("%s/file/direct_link?file_path=%s&key=%s",
f.endpoint,
url.QueryEscape(filePath),
url.QueryEscape(f.opt.Key),
)
opts := rest.Opts{
Method: "GET",
Path: "/file/direct_link",
Parameters: url.Values{
"file_path": {filePath},
"key": {f.opt.Key},
},
}
result := api.FileDirectLinkResponse{}
err := f.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
if err != nil {
return false, fmt.Errorf("failed to create request: %w", err)
}
resp, err := f.client.Do(req)
err := f.pacer.Call(func() (bool, error) {
_, err := f.srv.CallJSON(ctx, &opts, nil, &result)
if err != nil {
return shouldRetry(err), fmt.Errorf("failed to fetch direct link: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return false, fmt.Errorf("error decoding response: %w", err)
}
if result.Status != 200 {
return false, fmt.Errorf("API error: %s", result.Msg)
}
return shouldRetryHTTP(resp.StatusCode), nil
return false, nil
})
if err != nil {
return "", 0, err
@@ -210,39 +210,31 @@ func (f *Fs) getDirectLink(ctx context.Context, filePath string) (string, int64,
// deleteFile deletes a file based on filePath
func (f *Fs) deleteFile(ctx context.Context, filePath string) error {
filePath = f.fromStandardPath(filePath)
apiURL := fmt.Sprintf("%s/file/remove?file_path=%s&key=%s",
f.endpoint,
url.QueryEscape(filePath),
url.QueryEscape(f.opt.Key),
)
opts := rest.Opts{
Method: "GET",
Path: "/file/remove",
Parameters: url.Values{
"file_path": {filePath},
"key": {f.opt.Key},
},
}
result := api.DeleteFileResponse{}
err := f.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
_, err := f.srv.CallJSON(ctx, &opts, nil, &result)
if err != nil {
return false, fmt.Errorf("failed to create request: %w", err)
}
resp, err := f.client.Do(req)
if err != nil {
return shouldRetry(err), fmt.Errorf("failed to fetch direct link: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return false, fmt.Errorf("error decoding response: %w", err)
return shouldRetry(err), fmt.Errorf("failed to delete file: %w", err)
}
if result.Status != 200 {
return false, fmt.Errorf("API error: %s", result.Msg)
}
return shouldRetryHTTP(resp.StatusCode), nil
return false, nil
})
return err
}
@@ -275,46 +267,28 @@ func (f *Fs) getAccountInfo(ctx context.Context) (*api.AccountInfoResponse, erro
// getFileInfo retrieves file information based on file code
func (f *Fs) getFileInfo(ctx context.Context, fileCode string) (*api.FileInfoResponse, error) {
u, _ := url.Parse(f.endpoint + "/file/info2")
q := u.Query()
q.Set("file_code", fileCode) // raw path — Go handles escaping properly here
q.Set("key", f.opt.Key)
u.RawQuery = q.Encode()
apiURL := f.endpoint + "/file/info2?" + u.RawQuery
opts := rest.Opts{
Method: "GET",
Path: "/file/info2",
Parameters: url.Values{
"file_code": {fileCode},
"key": {f.opt.Key},
},
}
result := api.FileInfoResponse{}
var body []byte
err := f.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
if err != nil {
return false, fmt.Errorf("failed to create request: %w", err)
}
resp, err := f.client.Do(req)
_, err := f.srv.CallJSON(ctx, &opts, nil, &result)
if err != nil {
return shouldRetry(err), fmt.Errorf("failed to fetch file info: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
body, err = io.ReadAll(resp.Body)
if err != nil {
return false, fmt.Errorf("error reading response body: %w", err)
}
return shouldRetryHTTP(resp.StatusCode), nil
return false, nil
})
if err != nil {
return nil, err
}
result := api.FileInfoResponse{}
if err := json.NewDecoder(bytes.NewReader(body)).Decode(&result); err != nil {
return nil, fmt.Errorf("error decoding response: %w", err)
}
if result.Status != 200 || len(result.Result) == 0 {
return nil, fs.ErrorObjectNotFound
+115 -23
View File
@@ -1,6 +1,7 @@
package filelu
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -13,8 +14,108 @@ import (
"strings"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/rest"
)
// multipartUpload uploads a file in fixed-size chunks using the multipart API.
func (f *Fs) multipartUpload(ctx context.Context, in io.Reader, remote string) error {
dir := path.Dir(remote)
if dir == "." {
dir = ""
}
if dir != "" {
_ = f.Mkdir(ctx, dir)
}
folder := strings.Trim(dir, "/")
if folder != "" {
folder = "/" + folder
}
file := path.Base(remote)
initResp, err := f.multipartInit(ctx, folder, file)
if err != nil {
return fmt.Errorf("multipart init failed: %w", err)
}
uploadID := initResp.Result.UploadID
sessID := initResp.Result.SessID
server := initResp.Result.Server
objectPath := initResp.Result.ObjectPath
chunkSize := int(f.opt.ChunkSize)
buf := make([]byte, 0, chunkSize)
tmp := make([]byte, 1024*1024)
partNo := 1
for {
n, errRead := in.Read(tmp)
if n > 0 {
buf = append(buf, tmp[:n]...)
// If buffer reached chunkSize, upload a full part
if len(buf) >= chunkSize {
err = f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, bytes.NewReader(buf))
if err != nil {
return fmt.Errorf("upload part %d failed: %w", partNo, err)
}
partNo++
buf = buf[:0]
}
}
if errRead == io.EOF {
break
}
if errRead != nil {
return fmt.Errorf("read failed: %w", errRead)
}
}
if len(buf) > 0 {
err = f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, bytes.NewReader(buf))
if err != nil {
return fmt.Errorf("upload part %d failed: %w", partNo, err)
}
}
err = f.completeMultipart(ctx, server, uploadID, sessID, objectPath)
if err != nil {
return fmt.Errorf("complete multipart failed: %w", err)
}
return nil
}
// uploadPart sends a single multipart chunk to the upload server.
func (f *Fs) uploadPart(ctx context.Context, server, uploadID, sessID, objectPath string, partNo int, r io.Reader) error {
url := fmt.Sprintf("%s?partNumber=%d&uploadId=%s", server, partNo, uploadID)
req, err := http.NewRequestWithContext(ctx, "PUT", url, r)
if err != nil {
return err
}
req.Header.Set("X-RC-Upload-Id", uploadID)
req.Header.Set("X-RC-Part-No", fmt.Sprintf("%d", partNo))
req.Header.Set("X-Sess-ID", sessID)
req.Header.Set("X-Object-Path", objectPath)
resp, err := f.client.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != 200 {
return fmt.Errorf("uploadPart failed: %s", resp.Status)
}
return nil
}
// uploadFile uploads a file to FileLu
func (f *Fs) uploadFile(ctx context.Context, fileContent io.Reader, fileFullPath string) error {
directory := path.Dir(fileFullPath)
@@ -68,9 +169,15 @@ func (f *Fs) uploadFile(ctx context.Context, fileContent io.Reader, fileFullPath
return nil
}
// getUploadServer gets the upload server URL with proper key authentication
func (f *Fs) getUploadServer(ctx context.Context) (string, string, error) {
apiURL := fmt.Sprintf("%s/upload/server?key=%s", f.endpoint, url.QueryEscape(f.opt.Key))
opts := rest.Opts{
Method: "GET",
Path: "/upload/server",
Parameters: url.Values{
"key": {f.opt.Key},
},
}
var result struct {
Status int `json:"status"`
@@ -80,36 +187,21 @@ func (f *Fs) getUploadServer(ctx context.Context) (string, string, error) {
}
err := f.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
if err != nil {
return false, fmt.Errorf("failed to create request: %w", err)
}
resp, err := f.client.Do(req)
_, err := f.srv.CallJSON(ctx, &opts, nil, &result)
if err != nil {
return shouldRetry(err), fmt.Errorf("failed to get upload server: %w", err)
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return false, fmt.Errorf("error decoding response: %w", err)
}
if result.Status != 200 {
return false, fmt.Errorf("API error: %s", result.Msg)
}
return shouldRetryHTTP(resp.StatusCode), nil
return false, nil
})
if err != nil {
return "", "", err
}
if result.Status != 200 {
return "", "", fmt.Errorf("API error: %s", result.Msg)
}
return result.Result, result.SessID, nil
}
+1 -1
View File
@@ -14,7 +14,7 @@ import (
)
// errFileNotFound represent file not found error
var errFileNotFound error = errors.New("file not found")
var errFileNotFound = errors.New("file not found")
// getFileCode retrieves the file code for a given file path
func (f *Fs) getFileCode(ctx context.Context, filePath string) (string, error) {
+44 -38
View File
@@ -1,9 +1,7 @@
package filelu
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
@@ -16,6 +14,7 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/rest"
)
// Object describes a FileLu object
@@ -88,6 +87,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
}
}
// Wrap the response body to handle offset and count
var reader io.ReadCloser
err = o.fs.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", directLink, nil)
@@ -109,22 +109,25 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
return false, fmt.Errorf("failed to download file: HTTP %d", resp.StatusCode)
}
// Wrap the response body to handle offset and count
currentContents, err := io.ReadAll(resp.Body)
if err != nil {
return false, fmt.Errorf("failed to read response body: %w", err)
if offset > 0 {
_, err = io.CopyN(io.Discard, resp.Body, offset)
if err != nil {
_ = resp.Body.Close()
return false, fmt.Errorf("failed to skip offset: %w", err)
}
}
if offset > 0 {
if offset > int64(len(currentContents)) {
return false, fmt.Errorf("offset %d exceeds file size %d", offset, len(currentContents))
if count > 0 {
reader = struct {
io.Reader
io.Closer
}{
Reader: io.LimitReader(resp.Body, count),
Closer: resp.Body,
}
currentContents = currentContents[offset:]
} else {
reader = resp.Body
}
if count > 0 && count < int64(len(currentContents)) {
currentContents = currentContents[:count]
}
reader = io.NopCloser(bytes.NewReader(currentContents))
return false, nil
})
@@ -137,15 +140,23 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
// Update updates the object with new data
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
if src.Size() <= 0 {
return fs.ErrorCantUploadEmptyFiles
size := src.Size()
if size <= int64(o.fs.opt.UploadCutoff) {
err := o.fs.uploadFile(ctx, in, o.remote)
if err != nil {
return err
}
} else {
fullPath := path.Join(o.fs.root, o.remote)
err := o.fs.multipartUpload(ctx, in, fullPath)
if err != nil {
return fmt.Errorf("failed to upload file: %w", err)
}
}
err := o.fs.uploadFile(ctx, in, o.remote)
if err != nil {
return fmt.Errorf("failed to upload file: %w", err)
}
o.size = src.Size()
o.size = size
o.modTime = src.ModTime(ctx)
return nil
}
@@ -183,8 +194,14 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
return "", fmt.Errorf("no valid file code found in the remote path")
}
apiURL := fmt.Sprintf("%s/file/info?file_code=%s&key=%s",
o.fs.endpoint, url.QueryEscape(fileCode), url.QueryEscape(o.fs.opt.Key))
opts := rest.Opts{
Method: "GET",
Path: "/file/info",
Parameters: url.Values{
"file_code": {fileCode},
"key": {o.fs.opt.Key},
},
}
var result struct {
Status int `json:"status"`
@@ -193,29 +210,18 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
Hash string `json:"hash"`
} `json:"result"`
}
err := o.fs.pacer.Call(func() (bool, error) {
req, err := http.NewRequestWithContext(ctx, "GET", apiURL, nil)
if err != nil {
return false, err
}
resp, err := o.fs.client.Do(req)
_, err := o.fs.srv.CallJSON(ctx, &opts, nil, &result)
if err != nil {
return shouldRetry(err), err
}
defer func() {
if err := resp.Body.Close(); err != nil {
fs.Logf(nil, "Failed to close response body: %v", err)
}
}()
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return false, err
}
return shouldRetryHTTP(resp.StatusCode), nil
return false, nil
})
if err != nil {
return "", err
}
if result.Status != 200 || len(result.Result) == 0 {
return "", fmt.Errorf("error: unable to fetch hash: %s", result.Msg)
}
File diff suppressed because it is too large. Load diff
+14
View File
@@ -0,0 +1,14 @@
package filen
import (
"testing"
"github.com/rclone/rclone/fstest/fstests"
)
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestFilen:",
NilObject: (*Object)(nil),
})
}
+18 -1
View File
@@ -866,7 +866,24 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
return err
}
return o.readMetaData(ctx)
// The server computes the MD5 asynchronously after upload so
// retry reading the metadata for a short time until it appears.
const maxTries = 10
for tries := 1; ; tries++ {
err = o.readMetaData(ctx)
if err != nil {
return err
}
if o.md5 != "" || tries >= maxTries {
break
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(time.Duration(tries) * 100 * time.Millisecond):
}
}
return nil
}
// Remove an object
+90 -37
View File
@@ -163,6 +163,16 @@ Enabled by default. Use 0 to disable.`,
Help: "Disable TLS 1.3 (workaround for FTP servers with buggy TLS)",
Default: false,
Advanced: true,
}, {
Name: "allow_insecure_tls_ciphers",
Help: `Allow insecure TLS ciphers
Setting this flag will allow the usage of the following TLS ciphers in addition to the secure defaults:
- TLS_RSA_WITH_AES_128_GCM_SHA256
`,
Default: false,
Advanced: true,
}, {
Name: "shut_timeout",
Help: "Maximum time to wait for data connection closing status.",
@@ -194,6 +204,12 @@ Example:
Help: `URL for HTTP CONNECT proxy
Set this to a URL for an HTTP proxy which supports the HTTP CONNECT verb.
Supports the format http://user:pass@host:port, http://host:port, http://host.
Example:
http://myUser:myPass@proxyhostname.example.com:8000
`,
Advanced: true,
}, {
@@ -236,29 +252,30 @@ a write only folder.
// Options defines the configuration for this backend
type Options struct {
Host string `config:"host"`
User string `config:"user"`
Pass string `config:"pass"`
Port string `config:"port"`
TLS bool `config:"tls"`
ExplicitTLS bool `config:"explicit_tls"`
TLSCacheSize int `config:"tls_cache_size"`
DisableTLS13 bool `config:"disable_tls13"`
Concurrency int `config:"concurrency"`
SkipVerifyTLSCert bool `config:"no_check_certificate"`
DisableEPSV bool `config:"disable_epsv"`
DisableMLSD bool `config:"disable_mlsd"`
DisableUTF8 bool `config:"disable_utf8"`
WritingMDTM bool `config:"writing_mdtm"`
ForceListHidden bool `config:"force_list_hidden"`
IdleTimeout fs.Duration `config:"idle_timeout"`
CloseTimeout fs.Duration `config:"close_timeout"`
ShutTimeout fs.Duration `config:"shut_timeout"`
AskPassword bool `config:"ask_password"`
Enc encoder.MultiEncoder `config:"encoding"`
SocksProxy string `config:"socks_proxy"`
HTTPProxy string `config:"http_proxy"`
NoCheckUpload bool `config:"no_check_upload"`
Host string `config:"host"`
User string `config:"user"`
Pass string `config:"pass"`
Port string `config:"port"`
TLS bool `config:"tls"`
ExplicitTLS bool `config:"explicit_tls"`
TLSCacheSize int `config:"tls_cache_size"`
DisableTLS13 bool `config:"disable_tls13"`
AllowInsecureTLSCiphers bool `config:"allow_insecure_tls_ciphers"`
Concurrency int `config:"concurrency"`
SkipVerifyTLSCert bool `config:"no_check_certificate"`
DisableEPSV bool `config:"disable_epsv"`
DisableMLSD bool `config:"disable_mlsd"`
DisableUTF8 bool `config:"disable_utf8"`
WritingMDTM bool `config:"writing_mdtm"`
ForceListHidden bool `config:"force_list_hidden"`
IdleTimeout fs.Duration `config:"idle_timeout"`
CloseTimeout fs.Duration `config:"close_timeout"`
ShutTimeout fs.Duration `config:"shut_timeout"`
AskPassword bool `config:"ask_password"`
Enc encoder.MultiEncoder `config:"encoding"`
SocksProxy string `config:"socks_proxy"`
HTTPProxy string `config:"http_proxy"`
NoCheckUpload bool `config:"no_check_upload"`
}
// Fs represents a remote FTP server
@@ -272,6 +289,7 @@ type Fs struct {
user string
pass string
dialAddr string
tlsConf *tls.Config // default TLS client config
poolMu sync.Mutex
pool []*ftp.ServerConn
drain *time.Timer // used to drain the pool when we stop using the connections
@@ -397,9 +415,14 @@ func shouldRetry(ctx context.Context, err error) (bool, error) {
func (f *Fs) tlsConfig() *tls.Config {
var tlsConfig *tls.Config
if f.opt.TLS || f.opt.ExplicitTLS {
tlsConfig = &tls.Config{
ServerName: f.opt.Host,
InsecureSkipVerify: f.opt.SkipVerifyTLSCert,
if f.tlsConf != nil {
tlsConfig = f.tlsConf.Clone()
} else {
tlsConfig = new(tls.Config)
}
tlsConfig.ServerName = f.opt.Host
if f.opt.SkipVerifyTLSCert {
tlsConfig.InsecureSkipVerify = true
}
if f.opt.TLSCacheSize > 0 {
tlsConfig.ClientSessionCache = tls.NewLRUClientSessionCache(f.opt.TLSCacheSize)
@@ -407,6 +430,14 @@ func (f *Fs) tlsConfig() *tls.Config {
if f.opt.DisableTLS13 {
tlsConfig.MaxVersion = tls.VersionTLS12
}
if f.opt.AllowInsecureTLSCiphers {
var ids []uint16
// Read default ciphers
for _, cs := range tls.CipherSuites() {
ids = append(ids, cs.ID)
}
tlsConfig.CipherSuites = append(ids, tls.TLS_RSA_WITH_AES_128_GCM_SHA256)
}
}
return tlsConfig
}
@@ -431,9 +462,7 @@ func (f *Fs) ftpConnection(ctx context.Context) (c *ftp.ServerConn, err error) {
}
}()
baseDialer := fshttp.NewDialer(ctx)
if f.opt.SocksProxy != "" {
conn, err = proxy.SOCKS5Dial(network, address, f.opt.SocksProxy, baseDialer)
} else if f.proxyURL != nil {
if f.opt.SocksProxy != "" || f.proxyURL != nil {
// We need to make the onward connection to f.opt.Host. However the FTP
// library sets the host to the proxy IP after using EPSV or PASV so we need
// to correct that here.
@@ -443,7 +472,11 @@ func (f *Fs) ftpConnection(ctx context.Context) (c *ftp.ServerConn, err error) {
return nil, err
}
dialAddress := net.JoinHostPort(f.opt.Host, dialPort)
conn, err = proxy.HTTPConnectDial(network, dialAddress, f.proxyURL, baseDialer)
if f.opt.SocksProxy != "" {
conn, err = proxy.SOCKS5Dial(network, dialAddress, f.opt.SocksProxy, baseDialer)
} else {
conn, err = proxy.HTTPConnectDial(network, dialAddress, f.proxyURL, baseDialer)
}
} else {
conn, err = baseDialer.Dial(network, address)
}
@@ -604,6 +637,18 @@ func (f *Fs) drainPool(ctx context.Context) (err error) {
return err
}
// commandEncoding hardens the user-configured filename encoding so that it
// can never restore a raw CR or LF.
//
// The FTP control channel is line oriented and the ftp library writes command
// arguments (paths) straight onto it without escaping, so a filename
// containing CR/LF would otherwise be able to inject an independent FTP
// command. CR/LF are therefore always encoded to safe symbols regardless of
// the configured encoding, which is what the default encoding already does.
func commandEncoding(enc encoder.MultiEncoder) encoder.MultiEncoder {
return enc | encoder.EncodeCrLf
}
// NewFs constructs an Fs from the path, container:path
func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs, err error) {
// defer fs.Trace(nil, "name=%q, root=%q", name, root)("fs=%v, err=%v", &ff, &err)
@@ -613,6 +658,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs
if err != nil {
return nil, err
}
opt.Enc = commandEncoding(opt.Enc)
pass := ""
if opt.AskPassword && opt.Pass == "" {
pass = config.GetPassword("FTP server password")
@@ -652,6 +698,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs
dialAddr: dialAddr,
tokens: pacer.NewTokenDispenser(opt.Concurrency),
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
tlsConf: fshttp.NewTransport(ctx).TLSClientConfig,
}
f.features = (&fs.Features{
CanHaveEmptyDirectories: true,
@@ -864,7 +911,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
resultchan := make(chan []*ftp.Entry, 1)
errchan := make(chan error, 1)
go func() {
go func(c *ftp.ServerConn) {
result, err := c.List(f.dirFromStandardPath(path.Join(f.root, dir)))
f.putFtpConnection(&c, err)
if err != nil {
@@ -872,7 +919,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return
}
resultchan <- result
}()
}(c)
// Wait for List for up to Timeout seconds
timer := time.NewTimer(f.ci.TimeoutOrInfinite())
@@ -1201,7 +1248,10 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
path = o.fs.opt.Enc.FromStandardPath(path)
err = c.SetTime(path, modTime.In(time.UTC))
if err == nil && o.info != nil {
o.info.ModTime = modTime
// The server stores modtimes with second precision so
// truncate here too to keep the in-memory modtime identical
// to the one a fresh listing returns.
o.info.ModTime = modTime.Truncate(time.Second)
o.info.precise = true
}
o.fs.putFtpConnection(&c, err)
@@ -1264,7 +1314,7 @@ func (f *ftpReadCloser) Close() error {
// See: https://github.com/rclone/rclone/issues/3445#issuecomment-521654257
if errX := textprotoError(err); errX != nil {
switch errX.Code {
case ftp.StatusTransfertAborted, ftp.StatusFileUnavailable, ftp.StatusAboutToSend:
case ftp.StatusTransfertAborted, ftp.StatusFileUnavailable, ftp.StatusAboutToSend, ftp.StatusRequestedFileActionOK:
err = nil
}
}
@@ -1356,9 +1406,12 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
o.fs.putFtpConnection(&c, nil)
if o.fs.opt.NoCheckUpload {
o.info = &FileInfo{
Name: o.remote,
Size: uint64(src.Size()),
ModTime: src.ModTime(ctx),
Name: o.remote,
Size: uint64(src.Size()),
// The server stores modtimes with second precision so
// truncate here too to keep the in-memory modtime
// identical to the one a fresh listing returns.
ModTime: src.ModTime(ctx).Truncate(time.Second),
precise: true,
IsDir: false,
}
+32 -1
View File
@@ -12,11 +12,34 @@ import (
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/readers"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// A filename containing raw CR/LF must never survive encoding into an FTP
// command argument, whatever encoding the user configured, otherwise it could
// inject an independent FTP command onto the line-oriented control channel.
func TestCommandEncodingNeutralisesCRLF(t *testing.T) {
// Internal Standard-encoded name as produced by e.g. a Unix local
// source, carrying raw CR/LF.
for _, std := range []string{
"victim\r\nDELE other-secret\r\nNOOP",
"victim␍␊DELE other-secret␍␊NOOP",
} {
for _, configured := range []encoder.MultiEncoder{
encoder.EncodeSlash, // "Slash" - omits Ctl and CrLf
encoder.EncodeZero, // "None"
encoder.Display | encoder.EncodeRightSpace, // the FTP default
} {
got := commandEncoding(configured).FromStandardPath(std)
assert.NotContains(t, got, "\r", "encoding %v leaked raw CR: %q", configured, got)
assert.NotContains(t, got, "\n", "encoding %v leaked raw LF: %q", configured, got)
}
}
}
type settings map[string]any
func deriveFs(ctx context.Context, t *testing.T, f fs.Fs, opts settings) fs.Fs {
@@ -52,13 +75,21 @@ func (f *Fs) testUploadTimeout(t *testing.T) {
ci.Timeout = saveTimeout
}()
ci.LowLevelRetries = 1
ci.Timeout = fs.Duration(idleTimeout)
upload := func(concurrency int, shutTimeout time.Duration) (obj fs.Object, err error) {
// Use the saved (long) timeout while NewFs dials the FTP control
// connection and reads the welcome banner. On slow CI servers this
// can take longer than idleTimeout, which would cause a spurious
// i/o timeout before the test can run. The connection's idle
// deadline is captured at dial time, so the data connection
// dialled later inside Put will still get idleTimeout - which is
// what the test is actually trying to exercise.
ci.Timeout = saveTimeout
fixFs := deriveFs(ctx, t, f, settings{
"concurrency": concurrency,
"shut_timeout": shutTimeout,
})
ci.Timeout = fs.Duration(idleTimeout)
// Make test object
fileTime := fstest.Time("2020-03-08T09:30:00.000000000Z")
@@ -252,6 +252,9 @@ Docs: https://cloud.google.com/storage/docs/bucket-policy-only
}, {
Value: "us-east4",
Help: "Northern Virginia",
}, {
Value: "us-east5",
Help: "Ohio",
}, {
Value: "us-west1",
Help: "Oregon",
@@ -343,9 +346,26 @@ can't check the size and hash but the file contents will be decompressed.
Advanced: true,
Default: false,
}, {
Name: "endpoint",
Help: "Endpoint for the service.\n\nLeave blank normally.",
Name: "endpoint",
Help: `Custom endpoint for the storage API. Leave blank to use the provider default.
When using a custom endpoint that includes a subpath (e.g. example.org/custom/endpoint),
the subpath will be ignored during upload operations due to a limitation in the
underlying Google API Go client library.
Download and listing operations will work correctly with the full endpoint path.
If you require subpath support for uploads, avoid using subpaths in your custom
endpoint configuration.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "storage.example.org",
Help: "Specify a custom endpoint",
}, {
Value: "storage.example.org:4443",
Help: "Specifying a custom endpoint with port",
}, {
Value: "storage.example.org:4443/gcs/api",
Help: "Specifying a subpath, see the note, uploads won't use the custom path!",
}},
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
@@ -760,7 +780,7 @@ func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *storage.
}
// listDir lists a single directory
func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addBucket bool) (entries fs.DirEntries, err error) {
func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addBucket bool, callback func(fs.DirEntry) error) (err error) {
// List the objects
err = f.list(ctx, bucket, directory, prefix, addBucket, false, func(remote string, object *storage.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(ctx, remote, object, isDirectory)
@@ -768,16 +788,16 @@ func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addB
return err
}
if entry != nil {
entries = append(entries, entry)
return callback(entry)
}
return nil
})
if err != nil {
return nil, err
return err
}
// bucket must be present if listing succeeded
f.cache.MarkOK(bucket)
return entries, err
return err
}
// listBuckets lists the buckets
@@ -820,14 +840,46 @@ func (f *Fs) listBuckets(ctx context.Context) (entries fs.DirEntries, err error)
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
return list.WithListP(ctx, dir, f)
}
// ListP lists the objects and directories of the Fs starting
// from dir non recursively into out.
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) error {
list := list.NewHelper(callback)
bucket, directory := f.split(dir)
if bucket == "" {
if directory != "" {
return nil, fs.ErrorListBucketRequired
return fs.ErrorListBucketRequired
}
entries, err := f.listBuckets(ctx)
if err != nil {
return err
}
for _, entry := range entries {
err = list.Add(entry)
if err != nil {
return err
}
}
} else {
err := f.listDir(ctx, bucket, directory, f.rootDirectory, f.rootBucket == "", list.Add)
if err != nil {
return err
}
return f.listBuckets(ctx)
}
return f.listDir(ctx, bucket, directory, f.rootDirectory, f.rootBucket == "")
return list.Flush()
}
// ListR lists the objects and directories of the Fs starting
@@ -1099,7 +1151,15 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
remote: remote,
}
rewriteRequest := f.svc.Objects.Rewrite(srcBucket, srcPath, dstBucket, dstPath, nil)
// Set the storage class for the destination object if configured
var dstObject *storage.Object
if f.opt.StorageClass != "" {
dstObject = &storage.Object{
StorageClass: f.opt.StorageClass,
}
}
rewriteRequest := f.svc.Objects.Rewrite(srcBucket, srcPath, dstBucket, dstPath, dstObject)
if !f.opt.BucketPolicyOnly {
rewriteRequest.DestinationPredefinedAcl(f.opt.ObjectACL)
}
@@ -1387,6 +1447,10 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
ContentType: fs.MimeType(ctx, src),
Metadata: metadataFromModTime(modTime),
}
// Set the storage class from config if configured
if o.fs.opt.StorageClass != "" {
object.StorageClass = o.fs.opt.StorageClass
}
// Apply upload options
for _, option := range options {
key, value := option.Header()
@@ -1462,6 +1526,7 @@ var (
_ fs.Copier = &Fs{}
_ fs.PutStreamer = &Fs{}
_ fs.ListRer = &Fs{}
_ fs.ListPer = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
)
+2 -2
View File
@@ -116,8 +116,8 @@ type Date struct {
type DateFilter struct {
Dates []Date `json:"dates,omitempty"`
Ranges []struct {
StartDate Date `json:"startDate,omitempty"`
EndDate Date `json:"endDate,omitempty"`
StartDate Date `json:"startDate"`
EndDate Date `json:"endDate"`
} `json:"ranges,omitempty"`
}
+51 -7
View File
@@ -94,6 +94,23 @@ var (
}
)
// gphotosOAuthOptions returns a copy of oauthutil.SharedOptions with
// the client_id help tailored for Google Photos. It copies the shared
// options rather than mutating the global slice.
func gphotosOAuthOptions() []fs.Option {
opts := []fs.Option{}
for _, opt := range oauthutil.SharedOptions {
if opt.Name == config.ConfigClientID {
opt.Help = "OAuth Client Id.\n\nCreating your own is now strongly recommended.\nIf you leave this blank rclone uses a shared client_id which is being retired and will stop working during 2026.\nSee https://rclone.org/googlephotos/#making-your-own-client-id for how to create your own."
}
if opt.Name == config.ConfigClientSecret {
opt.Help = "OAuth Client Secret.\n\nLeave blank to use rclone's shared client_id.\nIf you created your own client_id then enter its client secret here."
}
opts = append(opts, opt)
}
return opts
}
// Register with Fs
func init() {
fs.Register(&fs.RegInfo{
@@ -101,7 +118,7 @@ func init() {
Prefix: "gphotos",
Description: "Google Photos",
NewFs: NewFs,
Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
Config: func(ctx context.Context, name string, m configmap.Mapper, configIn fs.ConfigIn) (*fs.ConfigOut, error) {
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
@@ -109,7 +126,7 @@ func init() {
return nil, fmt.Errorf("couldn't parse config into struct: %w", err)
}
switch config.State {
switch configIn.State {
case "":
// Fill in the scopes
if opt.ReadOnly {
@@ -117,22 +134,47 @@ func init() {
} else {
oauthConfig.Scopes = scopesReadWrite
}
return oauthutil.ConfigOut("warning", &oauthutil.Options{
return fs.ConfigGoto("client_id")
case "client_id":
if clientID, _ := m.Get(config.ConfigClientID); clientID != "" {
return fs.ConfigGoto("oauth")
}
return oauthutil.SharedClientIDConfigConfirm("client_id_warning", "Google Photos", "https://rclone.org/googlephotos/#making-your-own-client-id")
case "client_id_warning":
if configIn.Result == "true" {
// Continue using the shared client_id
return fs.ConfigGoto("oauth")
}
return fs.ConfigInput("client_id_set", config.ConfigClientID, "OAuth Client Id")
case "client_id_set":
m.Set(config.ConfigClientID, configIn.Result)
return fs.ConfigInput("client_secret_set", config.ConfigClientSecret, "OAuth Client Secret")
case "client_secret_set":
m.Set(config.ConfigClientSecret, configIn.Result)
return fs.ConfigGoto("oauth")
case "oauth":
return oauthutil.ConfigOut("warning1", &oauthutil.Options{
OAuth2Config: oauthConfig,
})
case "warning":
case "warning1":
// Warn the user as required by google photos integration
return fs.ConfigConfirm("warning_done", true, "config_warning", `Warning
return fs.ConfigConfirm("warning2", true, "config_warning", `Warning
IMPORTANT: All media items uploaded to Google Photos with rclone
are stored in full resolution at original quality. These uploads
will count towards storage in your Google Account.`)
case "warning2":
// Warn the user that rclone can no longer download photos it didnt upload from google photos
return fs.ConfigConfirm("warning_done", true, "config_warning", `Warning
IMPORTANT: Due to Google policy changes rclone can now only download photos it uploaded.`)
case "warning_done":
return nil, nil
}
return nil, fmt.Errorf("unknown state %q", config.State)
return nil, fmt.Errorf("unknown state %q", configIn.State)
},
Options: append(append(oauthutil.SharedOptions, []fs.Option{{
Options: append(append(gphotosOAuthOptions(), []fs.Option{{
Name: "read_only",
Default: false,
Help: `Set to make the Google Photos backend read only.
@@ -343,6 +385,8 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
return nil, err
}
oauthutil.SharedClientIDWarning(name, "Google Photos", "https://rclone.org/googlephotos/#making-your-own-client-id", m)
baseClient := fshttp.NewClient(ctx)
oAuthClient, ts, err := oauthutil.NewClientWithBaseClient(ctx, name, m, oauthConfig, baseClient)
if err != nil {
+25 -16
View File
@@ -43,33 +43,42 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
var commandHelp = []fs.CommandHelp{{
Name: "drop",
Short: "Drop cache",
Short: "Drop cache.",
Long: `Completely drop checksum cache.
Usage Example:
rclone backend drop hasher:
`,
Usage example:
` + "```console" + `
rclone backend drop hasher:
` + "```",
}, {
Name: "dump",
Short: "Dump the database",
Long: "Dump cache records covered by the current remote",
Short: "Dump the database.",
Long: "Dump cache records covered by the current remote.",
}, {
Name: "fulldump",
Short: "Full dump of the database",
Long: "Dump all cache records in the database",
Short: "Full dump of the database.",
Long: "Dump all cache records in the database.",
}, {
Name: "import",
Short: "Import a SUM file",
Short: "Import a SUM file.",
Long: `Amend hash cache from a SUM file and bind checksums to files by size/time.
Usage Example:
rclone backend import hasher:subdir md5 /path/to/sum.md5
`,
Usage example:
` + "```console" + `
rclone backend import hasher:subdir md5 /path/to/sum.md5
` + "```",
}, {
Name: "stickyimport",
Short: "Perform fast import of a SUM file",
Short: "Perform fast import of a SUM file.",
Long: `Fill hash cache from a SUM file without verifying file fingerprints.
Usage Example:
rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5
`,
Usage example:
` + "```console" + `
rclone backend stickyimport hasher:subdir md5 remote:path/to/sum.md5
` + "```",
}}
func (f *Fs) dbDump(ctx context.Context, full bool, root string) error {
+54
View File
@@ -69,12 +69,66 @@ func (f *Fs) testUploadFromCrypt(t *testing.T) {
_ = operations.Purge(ctx, f, dirName)
}
func (f *Fs) testUpdateStoresHash(t *testing.T) {
// make a temporary local remote
tempRoot, err := fstest.LocalRemote()
require.NoError(t, err)
defer func() {
_ = os.RemoveAll(tempRoot)
}()
// make a temporary crypt remote as source
ctx := context.Background()
pass := obscure.MustObscure("crypt")
remote := fmt.Sprintf(`:crypt,remote="%s",password="%s":`, tempRoot, pass)
cryptFs, err := fs.NewFs(ctx, remote)
require.NoError(t, err)
const dirName = "update_hash_1"
const fileName = dirName + "/file_update_1"
const longTime = fs.ModTimeNotSupported
hashType := f.keepHashes.GetOne()
// upload initial file to hasher via Put
src1 := putFile(ctx, t, cryptFs, fileName, "initial content")
in1, err := src1.Open(ctx)
require.NoError(t, err)
dst, err := f.Put(ctx, in1, src1)
require.NoError(t, err)
require.NotNil(t, dst)
// verify hash was stored after Put
var hash1 string
if f.opt.MaxAge > 0 {
hash1, err = f.getRawHash(ctx, hashType, fileName, anyFingerprint, longTime)
assert.NoError(t, err)
assert.NotEmpty(t, hash1)
}
// update the file with new content via Update (this is what sync does for replacements)
src2 := putFile(ctx, t, cryptFs, fileName, "updated content")
in2, err := src2.Open(ctx)
require.NoError(t, err)
err = dst.Update(ctx, in2, src2)
require.NoError(t, err)
// verify hash was stored after Update and is different from the original
if f.opt.MaxAge > 0 {
hash2, err := f.getRawHash(ctx, hashType, fileName, anyFingerprint, longTime)
assert.NoError(t, err)
assert.NotEmpty(t, hash2, "hash should be stored after Update")
assert.NotEqual(t, hash1, hash2, "hash should change after Update with different content")
}
_ = operations.Purge(ctx, f, dirName)
}
// InternalTest dispatches all internal tests
func (f *Fs) InternalTest(t *testing.T) {
if !kv.Supported() {
t.Skip("hasher is not supported on this OS")
}
t.Run("UploadFromCrypt", f.testUploadFromCrypt)
t.Run("UpdateStoresHash", f.testUpdateStoresHash)
}
var _ fstests.InternalTester = (*Fs)(nil)
+57 -41
View File
@@ -131,10 +131,62 @@ func (o *Object) updateHashes(ctx context.Context) error {
return nil
}
// wrapInHashingReader wraps in with a hashing reader if needed and
// returns the (possibly wrapped) reader. After the transfer completes
// successfully, call the returned storeHashes function to store the
// computed hashes.
func (f *Fs) wrapInHashingReader(ctx context.Context, in io.Reader, src fs.ObjectInfo) (wrapIn io.Reader, storeHashes func(o *Object)) {
var (
common hash.Set
rehash bool
hashes hashMap
)
if fsrc := src.Fs(); fsrc != nil {
common = fsrc.Hashes().Overlap(f.keepHashes)
// Rehash if source does not have all required hashes or hashing is slow
rehash = fsrc.Features().SlowHash || common != f.keepHashes
}
wrapIn = in
if rehash {
r, err := f.newHashingReader(ctx, in, func(sums hashMap) {
hashes = sums
})
fs.Debugf(src, "Rehash in-fly due to incomplete or slow source set %v (err: %v)", common, err)
if err == nil {
wrapIn = r
} else {
rehash = false
}
}
storeHashes = func(o *Object) {
if !rehash {
hashes = hashMap{}
for _, ht := range common.Array() {
if h, e := src.Hash(ctx, ht); e == nil && h != "" {
hashes[ht] = h
}
}
}
if len(hashes) > 0 {
err := o.putHashes(ctx, hashes)
fs.Debugf(o, "Applied %d source hashes, err: %v", len(hashes), err)
}
}
return wrapIn, storeHashes
}
// Update the object with the given data, time and size.
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
wrapIn, storeHashes := o.f.wrapInHashingReader(ctx, in, src)
_ = o.f.pruneHash(src.Remote())
return o.Object.Update(ctx, in, src, options...)
err := o.Object.Update(ctx, wrapIn, src, options...)
if err != nil {
return err
}
storeHashes(o)
return nil
}
// Remove an object.
@@ -189,51 +241,15 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (r io.ReadC
// Put data into the remote path with given modTime and size
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
var (
o fs.Object
common hash.Set
rehash bool
hashes hashMap
)
if fsrc := src.Fs(); fsrc != nil {
common = fsrc.Hashes().Overlap(f.keepHashes)
// Rehash if source does not have all required hashes or hashing is slow
rehash = fsrc.Features().SlowHash || common != f.keepHashes
}
wrapIn := in
if rehash {
r, err := f.newHashingReader(ctx, in, func(sums hashMap) {
hashes = sums
})
fs.Debugf(src, "Rehash in-fly due to incomplete or slow source set %v (err: %v)", common, err)
if err == nil {
wrapIn = r
} else {
rehash = false
}
}
wrapIn, storeHashes := f.wrapInHashingReader(ctx, in, src)
_ = f.pruneHash(src.Remote())
oResult, err := f.Fs.Put(ctx, wrapIn, src, options...)
o, err = f.wrapObject(oResult, err)
o, err := f.wrapObject(oResult, err)
if err != nil {
return nil, err
}
if !rehash {
hashes = hashMap{}
for _, ht := range common.Array() {
if h, e := src.Hash(ctx, ht); e == nil && h != "" {
hashes[ht] = h
}
}
}
if len(hashes) > 0 {
err := o.(*Object).putHashes(ctx, hashes)
fs.Debugf(o, "Applied %d source hashes, err: %v", len(hashes), err)
}
return o, err
storeHashes(o.(*Object))
return o, nil
}
type hashingReader struct {
+3 -3
View File
@@ -371,9 +371,9 @@ func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
return nil, err
}
return &fs.Usage{
Total: fs.NewUsageValue(int64(info.Capacity)),
Used: fs.NewUsageValue(int64(info.Used)),
Free: fs.NewUsageValue(int64(info.Remaining)),
Total: fs.NewUsageValue(info.Capacity),
Used: fs.NewUsageValue(info.Used),
Free: fs.NewUsageValue(info.Remaining),
}, nil
}
Loaded 100 of 1116 files, more files were not shown because too many files have changed in this diff. Show more