Compare commits

..
119 Commits
Author SHA1 Message Date
Nick Craig-Wood 687d264b68 Version v1.75.1 2026-09-04 17:03:20 +01:00
Nick Craig-Wood 4158f63d2f Start v1.75.1-DEV development 2026-09-04 16:50:38 +01:00
Hakan İSMAİL c47832ad34 serve: fix VFS instance leaks on server startup failures and shutdown
(cherry picked from commit 216d2a8c76)
2026-09-04 16:39:12 +01:00
Nick Craig-Wood 60e96416df rc: silence deprecation lint warning in the old web gui plugin proxy
Raising the minimum Go version to 1.26 made staticcheck flag the
deprecated ReverseProxy.Director. This code is unmaintained and has
been removed on the development branch, so suppress the warning rather
than rewrite it.
2026-09-04 16:34:03 +01:00
Nick Craig-Wood c841c3bba3 build: disable staticcheck SA4023 to fix lint job timeout
The dataflow analysis behind SA4023, new in the staticcheck 0.8.0
bundled with golangci-lint v2.13.0, makes linting large packages more
than 10x slower (89s vs 7s for backend/s3 alone) which took the CI
lint job past its 30 minute limit. golangci-lint no longer enforces
its run timeout during analysis so the job ran until cancelled, and
the cancellation meant the lint cache was never saved, making every
subsequent run cold and guaranteeing the timeout repeated.

The check also produces false positives (eg claiming operations.Delete
never returns nil).

(cherry picked from commit 357c2a2b44)
2026-09-04 16:33:21 +01:00
Nick Craig-Wood 79fbc0842f fshttp: don't send --header values to other hosts on redirect GHSA-486v-q2wf-fp2r CVE-PENDING
The headers set with --header and --header-download are added to
every request by the rclone transport, including redirect hops which
net/http makes to other hosts, so a credential passed with --header
for one host could be sent to any host that server chose to redirect
to.

The transport now walks the redirect chain net/http records on each
redirected request and, once the chain has visited a host other than
the one originally requested, removes the headers rather than adding
them.

Also restore the global --client-cert and --client-key config after
TestCertificates so its temporary files are not used by later tests.

(cherry picked from commit 101c3e342c86ae65c7c743474d7319330f0eb0ca)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 22859b7e69 http: don't leak configured headers to other hosts or over plaintext on redirect GHSA-486v-q2wf-fp2r CVE-PENDING
The headers set with --http-headers are documented for passing
credentials such as Authorization or Cookie. The backend used the
default net/http redirect policy which copies all but a handful of
well known headers to any redirect target, so a redirect from the
configured server to another host would send those credentials to
that host, and a redirect from https to http would send them in
plaintext.

When headers are configured this installs a CheckRedirect function
which:

- removes the configured headers from every hop once the redirect
  chain has left the originally requested host
- refuses a redirect from https to http with an error

(cherry picked from commit 0adc0082dff7ef6ed1597418f85ac691478a6fa5)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 925fb4fb21 rest: add SameHost and check HTTPS downgrades against the original request GHSA-486v-q2wf-fp2r CVE-PENDING
SameHost compares two URLs by host name (case insensitively) and port
(treating the scheme's default port as no port) so redirect policies
can tell a real change of host from a server spelling its own host
differently, e.g. redirecting "https://example.com/" to
"https://EXAMPLE.com:443/".

The HTTPS downgrade check now compares the redirect target against
the original request rather than the previous hop, so a chain which
started on plaintext http, passed through an https server and came
back to http is no longer refused - nothing is being downgraded
relative to what the user asked for. A chain which started on https
and reaches http via any number of hops is still refused.

(cherry picked from commit fc7a64f61c2e4b364b5789f6ea17ad0485d8246e)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 96f298bdec archive: hide any archive entry which escapes the directory being listed GHSA-66hp-wgxq-6f5q
Whether an archive entry name can escape the archive's namespace was
left entirely to each archiver. Enforce it in the archive backend too.

List only passes on direct children of the directory listed and
NewObject only returns the object asked for, so a future archiver
which forgets to validate names cannot expose a traversal to fs/sync
and fs/operations.

(cherry picked from commit a6a7d95e081b91231b49c4004e8a2bff16da4cd8)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 60fb55dc6a archive: fix "directory not found" for archive paths containing "./" or "//" GHSA-66hp-wgxq-6f5q
The path inside the archive was compared against the cleaned entry
names without being cleaned itself, so `archive.zip/sub/./dir` or
`archive.zip/sub//dir` failed to list even though `archive.zip/sub/dir`
worked.

(cherry picked from commit 34636f34079585f3a8f883038483ece58d39b084)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 181ef190ce archive: fix zip entry named "." hiding every other file GHSA-66hp-wgxq-6f5q
A zip containing a file entry whose name refers to the archive's own
root (".", "/" or "") was presented as a single file called "." and
all its other entries disappeared. A file at the root can only be the
archive member the backend was pointed at, so with no root such an
entry is skipped like any other unsafe name.

(cherry picked from commit c9fac578aae7707faf340b58cf45465a5f698cbd)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood ad8cd41c49 archive: reject unsafe entry names when mounting squashfs images GHSA-66hp-wgxq-6f5q
Entry names read from a squashfs directory are not sanitized by
go-diskfs. The squashfs backend joined each leaf name onto its
directory to form the object's remote, so a crafted image could escape
its directory.

Use sanitize.Leaf to skip unsafe entries in List. A "\" is an
ordinary character in a file name on the systems squashfs images are
made on and in an rclone remote path, so it is deliberately not
rejected; making it safe for the destination is the destination
backend's job.

Skipped entries are logged at DEBUG with a single NOTICE count per
listing so a crafted image under a mount cannot flood the log.

(cherry picked from commit 63385c66175141b63bb53b3e42b50908649f8437)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 5dae3adbf5 archive: fix zip subdirectory root matching sibling directories GHSA-66hp-wgxq-6f5q
When a zip archive was mounted at a subdirectory root, readZip used a bare
strings.HasPrefix to decide which entries fell inside the root. This
matched on a raw string prefix rather than a path boundary, so mounting
root "foo" also exposed sibling entries such as "foobar/..." with their
names left uncorrected.

Require a path boundary when filtering by root.

(cherry picked from commit b444227264cee2a9307f03ba90c2a411e7eeb693)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 6507e13d5a archive: fix zip slip path traversal in untrusted zip files GHSA-66hp-wgxq-6f5q CVE-PENDING
The zip backend mounts a zip file as a browsable Fs. Go's archive/zip
does not sanitize entry names, and readZip applied path.Clean but did
not reject a cleaned name that still pointed outside the archive. A
crafted zip could make rclone copy/sync attempt writes outside the
intended destination.

Sanitize entry names with sanitize.Path - the same check used by
rclone archive extract - skipping any entry with a ".." path
component, whether separated by "/" or "\". A backslash is otherwise
kept as an ordinary character in the name, as archive extract does. It
is up to the destination backend to make names safe for its storage.

Skipped entries are logged as a single count per archive so a crafted
archive with many escaping entries cannot flood the log.

(cherry picked from commit 224fe97ac13105094651264a9e16ee3cf41ccf77)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood e4913c9e91 lib/sanitize: factor untrusted path sanitization out of archive extract
Move the archive entry name validation added for CVE-2026-59732 from
cmd/archive/extract into a new lib/sanitize package as sanitize.Path,
so the same check can be shared with the archive backend which mounts
archives as a filesystem.

sanitize.Path keeps the extract semantics - reject any name with a
".." path component, treating both "/" and "\" as separators - and
additionally cleans the name with path.Clean. This corrects two edge
cases in extract: a repeated "./" prefix ("././file.txt") is now fully
stripped rather than only the first, and a bare "." entry is now
treated as the archive root and skipped.

Add sanitize.Leaf, which rejects a name that is empty, ".", ".." or
contains a "/", for checking a single directory entry name read from
an archive.

The names handled are rclone remote paths, in which "/" is the only
separator and "\" an ordinary character, so Leaf does not reject a
backslash: making a name safe for its storage is the destination
backend's job (the local backend encodes "\" on Windows and refuses
paths which escape its root). Path's rejection of ".." between
backslashes is kept as defence in depth for extract.

(cherry picked from commit 4aab7cd450127e902509b4c8e3a1e4f5c12056df)
2026-09-04 16:18:35 +01:00
Nick Craig-Wood 28bf49d66f local: fix panic on Range request past the end of a symlink GHSA-p6m2-r3w9-mpxw CVE-PENDING
With --links/-l, a symlink is served as a .rclonelink object whose
content is the target path. A Range request with a start offset beyond
the target length (e.g. "Range: bytes=99999999999-") reached
openTranslatedLink and sliced the target string at that offset, panicking
with "slice bounds out of range".

Clamp the offset to the target length so an out-of-range start reads
empty, matching how a real file read past EOF behaves.

(cherry picked from commit 3fa32192c20f0b4cfd4f4b07636dc3445026041a)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood 17b0c03338 local: fix btime escaping the root via a planted symlink GHSA-f8g7-2xjc-7mfh CVE-PENDING
The birth-time (btime) write in writeMetadataToFile followed symlinks for
any object that was not a translated link, so under -l/--links a symlink
planted by an untrusted source at the destination path could redirect the
btime write to a target outside the backup destination on OSes where
birth time is settable (Windows).

Use the NOFOLLOW birth-time write whenever translating symlinks, not only
for translated links. It is a no-op on a real file or directory and stops
a planted symlink from being followed out of the destination.

(cherry picked from commit bd86336faac7cfc4e9057add38ec5fd12d762d02)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood a7ab39d3d1 local: fix dir metadata escaping the root through a planted symlink GHSA-f8g7-2xjc-7mfh CVE-PENDING
With -l/--links the local backend faithfully recreates a source ".rclonelink" as
a real symlink at the destination. Directory metadata (chmod/chown/chtimes),
however, was applied with the raw following syscalls
os.Chmod/os.Chown/os.Chtimes rather than through the os.Root sandbox used for
content writes. A Directory is never a translatedLink, so when the destination
path already existed as a symlink planted by an untrusted source, the metadata
was applied through it to a target outside the backup destination.

Route directory metadata through os.Root when translating symlinks, so a planted
symlink can no longer redirect chmod/chown/chtimes out of the destination, while
legitimate in-tree directories are unaffected.

(cherry picked from commit b764ccbd23582f29c611862f11f86dde3544035a)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood 7a594b09c6 serve docker: reject volume names resolving to the base directory itself GHSA-p6vx-hf7p-98j6
An empty or "." volume name joined onto the base directory resolves to the
base directory itself. newVolume does not call validate, so such a name
would mount a remote over the base directory and shadow every other
volume's mountpoint.

Require the resolved mountpoint to be a strict descendant of the base
directory so these degenerate names are refused.

(cherry picked from commit 4765ab020802ebf8ffbc2f2e529a6b23e640bf6f)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood f376c64784 serve docker: re-derive volume mountpoint from name when restoring state GHSA-p6vx-hf7p-98j6
When the plugin restarts it reads its persisted state file and used the
stored mountpoint verbatim. A state file written by an older rclone that
allowed escaping volume names, or one that was tampered with, could point
the mountpoint outside the base directory, so upgrading did not remediate
an already-escaped volume.

Re-derive the mountpoint from the base directory and the volume name on
restore, confined to the base directory, rather than trusting the stored
path.

(cherry picked from commit b4069bc49676e55b9c8843de9a1919f940a585ac)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood afeb26ae74 serve docker: reject volume names that escape the base directory GHSA-p6vx-hf7p-98j6
A Docker VolumeDriver.Create request carries a raw volume name that was
joined onto the base directory with filepath.Join and used verbatim as the
mountpoint. filepath.Join collapses ".." components, so a crafted name such
as "../../../etc/foo" resolved to a host path outside the base directory,
where the plugin then created a directory and mounted the remote.

Confine the mountpoint to the base directory and refuse any name that
resolves outside it.

(cherry picked from commit d9ed70376eb500d23d4d7ed0d6b3db2f3c7242e8)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood c6af0b57c2 serve ftp: fix auth-proxy sessions sharing credentials by username GHSA-c476-6w5q-jw77 CVE-PENDING
When serving FTP with --auth-proxy, the obscured password was cached in a
driver-global map keyed only by the username. Two sessions that logged in
with the same username but different credentials shared one map entry, so a
later login overwrote it and every subsequent operation on the earlier,
still-authenticated session was re-authorized with the later session's
credential and executed against the later session's backend.

Bind the credential to the FTP session by storing the obscured password in
the per-session goftp Session.Data map instead, so each session always
resolves the backend it authenticated for.

(cherry picked from commit 0bc745328dbf4669561d2abd9c81c596fbb0787d)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood 90595f34f2 serve s3: fix auth proxy accepting any request signed with an empty secret GHSA-xwwr-4h3p-r22c CVE-PENDING
With --auth-proxy set and --auth-key unset, serve s3 registered every client
supplied access key ID with an empty secret and verified the SigV4 signature
against that, so anyone could sign a request for an arbitrary access key ID with
an empty secret and be let in. The proxy program was only ever given the access
key ID (as both user and pass) so it had nothing with which to authenticate the
client either.

An S3 client never sends its secret, only a signature made with it, so the
server has to know the secret to check the request. The auth proxy protocol as
been changed to handle this. For serve s3 the proxy program is given just the
access key ID as the user (no pass or public_key) and must return the matching
secret as _secret_access_key in its output. rclone verifies the request's
signature against that secret, refusing the request if the proxy rejects the
access key ID, doesn't return a secret or returns an empty one, or the signature
doesn't match. The secret is only used for this server's own verification and is
never registered with gofakes3, so other serve s3 instances in the same process
don't honour it.

The proxy's answers are cached. If a signature fails against a cached secret the
proxy is consulted again so a rotated secret takes effect immediately - but only
for a signature mismatch, and at most once every 10 seconds per access key ID
and client IP, so a stream of bad signatures can't make the proxy program run
for every request. A rotation never shuts down the cached backend under requests
still using it. A cached answer is checked with the proxy again once it is 5
minutes old even if in constant use, so revoking an access key ID takes effect
within 5 minutes.

This means --auth-key is no longer needed with --auth-proxy: it is ignored and a
warning is given at startup if both are set. The proxy is the source of truth
for both the credentials and the backend they map to. Presigned URLs (credential
in the query string) are now recognised by the proxy middleware too. The auth
proxy docs are added to serve s3.

Note that the serve s3 auth proxy protocol has changed. The proxy program is now
given the access key ID as "user" (it was previously given an MD5 hash of it,
with the access key ID as "pass") and must return the matching secret as
"_secret_access_key".

This needs gofakes3 v0.0.9 for signature.V4SignVerifyWithSecret.

(cherry picked from commit 7306f0668125ecbd1cb9d704f570fbb492a76fb8)
2026-09-04 16:18:19 +01:00
Nick Craig-Wood b3cf0444a2 serve s3: fix each server accepting the --auth-key credentials of all the others
gofakes3 kept the keys given with --auth-key in a store global to the process,
so when more than one serve s3 was running in one rclone (eg started via the rc)
each accepted the others' credentials and a client with the key for one server
could read and write the backend of another.

This updates gofakes3 to v0.0.9 which keeps auth keys per instance and adds a
test that two servers only accept their own keys.

(cherry picked from commit b81c9c892f855095924b89e623f0e4c4e7149168)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 11ff6e49bb serve sftp: fix auth proxy configured via rc being silently ignored GHSA-p569-5gjg-9cmj CVE-PENDING
From v1.70.0, an SFTP server started through the rc serve/start API with
a per-server proxyOpt.AuthProxy decided whether to enable proxy
authentication by checking the process-global proxy.Opt.AuthProxy
instead of the supplied proxyOpt.AuthProxy. In the normal rc case the
global is empty, so the auth proxy was silently ignored: the server
either failed to start with "no authorization found" or authenticated
against the local authorized_keys file instead of routing each login
through the proxy the operator configured.

The serve Provider refactor (f425f8d46) fixed the constructor by building the
provider from the supplied proxyOpt, but the authorized-keys handling in
configure() still consulted the global option. Make it depend on whether
proxy mode is actually active, and add a regression test for the
per-server option.

(cherry picked from commit 929933f5c303d6e7d5ab0cdcbae843862cb15e73)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood bcef98db3f serve s3: fix misleading anonymous access log and add test for auth proxy via rc GHSA-p569-5gjg-9cmj CVE-PENDING
From v1.70.0 until the serve Provider refactor (f425f8d46), an S3 server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server served the fixed
filesystem supplied to serve/start rather than routing each access key
to the backend chosen by the proxy, bypassing the operator's intended
per-key authorization.

The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again, and only
logs "allowing anonymous access" when neither an auth key nor an auth
proxy is configured so the log reflects the effective mode.

(cherry picked from commit 1e8134e83bbdcededfdc3c48f00e52544a775342)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood f18aacf2c7 serve ftp: add test for auth proxy configured via rc GHSA-p569-5gjg-9cmj CVE-PENDING
From v1.70.0 until the serve Provider refactor (f425f8d46), an FTP server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server fell back to its
fixed-backend mode, whose default account accepts user "anonymous" with
any password - a complete authentication bypass.

The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again.

(cherry picked from commit f89737278f27dc233f4dae2cb126a5e8b1980f70)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood ab1f458013 serve s3: reject bogus multipart part sizes in the reorder buffer GHSA-2p48-j3qc-rx9f
The multipart reorder-buffer admission trusted the client-declared part length.
A negative length was accepted, and `buffered + size` could overflow int64 for
a huge declared length, wrapping the running total negative and admitting
further parts past --multipart-streaming-buffer-limit.

Reject a negative length and use the overflow-safe comparison `size <=
bufferLimit - buffered` so an untrusted Content-Length can neither poison nor
overflow the budget.

(cherry picked from commit 337ab7f76ba6ba3731fba04b088bd831511bca04)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 7c1dfd99f3 serve s3: fix memory exhaustion from client-declared multipart part size GHSA-2p48-j3qc-rx9f CVE-PENDING
Streamed multipart UploadPart called Reserve(contentLength) before reading any
body bytes, so the pool immediately allocated one 1 MiB page per MiB of the
client-declared Content-Length (or X-Amz-Decoded-Content-Length). An client
could declare a huge part size, send no body, and force an arbitrarily large
allocation without paying the bandwidth cost of the declared body.

Drop the Reserve so the pool-backed buffer grows a page at a time as the body
is actually read: memory now tracks the bytes received, not the unverified
header.

(cherry picked from commit 8f2ad09941b0d69b67366101d5c2c350ca2a12df)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 57842c5ee4 fs: confine directory listing entries that escape the root GHSA-3vxh-3pcx-9m8q GHSA-38xv-hf3p-h7mq CVE-PENDING
The rclone core does not sanitise ".." in an object's Remote(). Such a name can
arrive from a malicious or buggy backend - an object store permits keys
containing ".." or a leading "/" - and, if acted on, lets a listing or transfer
escape the configured root. A source object named "../../other/x" is copied to
"other/x" outside the destination root, and a crafted listing name surfaces
outside the directory being listed.

Add list.RemoteEscapesRoot, which reports whether a Remote climbs above the
root when joined onto it, and list.RemoveEscaping, which drops and logs such
entries.

Apply RemoveEscaping unconditionally - independent of the include/exclude
filters - at the three per-entry filtering points every listing passes through:
filterDir, walk.listR and walk.walkRDirTree (recursive ListR).
operations.StatJSON calls List and NewObject directly, bypassing those, so it
rejects an escaping remote up front.

This confines every backend at once, so no per-backend change is needed.

(cherry picked from commit 3530367fcdefeb718d9cac14a5147836b448bc73)
2026-09-04 16:18:18 +01:00
Hakan İSMAİL 739403963a serve: refactor VFS and proxy handling into Provider
(cherry picked from commit f425f8d466)
2026-09-04 16:18:18 +01:00
am-at-enrollvb 1bec2133c5 serve: pass the client IP address to the auth proxy - fixes #4499
The auth proxy was only given the user and their password or public
key, so a proxy program had no way to restrict logins to particular
networks, or to record where an authentication attempt came from.

The JSON sent to the program now has a client_ip key holding the bare
IP the client connected from, with the port stripped so IPv6 arrives
as 2001:db8::1 rather than [2001:db8::1]:52344. An IPv4-mapped IPv6
address is reported as plain IPv4 so that a client arriving over a
dual-stack listener still matches IPv4 networks. The key is omitted
when the client has no IP address.

The IP is also mixed into the backend cache key. That is needed as the
program is only run on a cache miss, so a client from a
non-allowlisted address presenting valid credentials within the 5
minute cache lifetime would get a cache hit and be let in without the
program being consulted at all.

(cherry picked from commit 5dd34275dc)
2026-09-04 16:18:18 +01:00
Nick Craig-Wood 64de81e6a0 build: make go1.26 the minimum required version
golang.org/x/crypto v0.56.0, which fixes CVE-2026-78662 and
CVE-2026-56855 in its ssh package, requires go1.26, so rclone can no
longer be built with go1.25.
2026-09-04 14:07:42 +01:00
Nick Craig-Wood 0b187f3d7a build: update golang.org/x/crypto to v0.56.0 to fix CVE-2026-78662 and CVE-2026-56855
CVE-2026-78662: a malicious peer could flood an undecided channel's
incoming requests, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6354)

CVE-2026-56855: a malicious peer could send crafted messages on an
established channel, deadlocking the whole connection in
golang.org/x/crypto/ssh (GO-2026-6355)

(cherry picked from commit f550317590)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 0e5100203c bisync: fix failed transfers of empty files being recorded as synced
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.

(cherry picked from commit 7e17e1b90d)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood ba3d34b9f8 docs: describe how backends should allocate memory
(cherry picked from commit 220fe76192)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 77e6390f5b quatrix: fix chunk upload retries and fix memory leak
Each upload chunk is buffered in a pool.RW from the global memory pool
but was never closed, so its pages were never returned to the pool.

Close the buffer after each chunk is uploaded and on the read error
path.

A chunk that failed with a retryable error was also retried without
rewinding the buffer, so the retry sent an empty body with the original
Content-Length and Content-Range and failed.

Seek the chunk back to the start inside the pacer closure so each
attempt re-sends it in full.

The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.

(cherry picked from commit 2f0228029e)
2026-09-04 14:07:19 +01:00
SillyZir f190b34d95 onedrive: fall back to manual drive ID entry when drive listing fails
When both /me/drives and /me/drive fail during config (for example an
account-level 403 serviceReadOnly "Database Is Read Only"), send the
config state machine to the existing manual drive ID entry state
instead of dead-ending at choose_type with the raw error. The drive
itself remains usable when only the enumeration API is blocked.

Fixes #9794

(cherry picked from commit 03fe2ef794)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 49e7016d12 build: update golang.org/x/crypto to v0.55.0 to fix CVE-2026-56854
CVE-2026-56854: source-address critical option not enforced for
non-public-key auth callbacks in golang.org/x/crypto/ssh (GO-2026-6303)

(cherry picked from commit e5e1ee3e96)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 6240cbb694 crypt: warn about directories with legacy version-like encrypted names
Directory names which look like they have a --b2-versions version
string are now encrypted in full, so directories created by older
rclone (which left the version string in plain text) no longer
decrypt and vanished silently from listings.

DecryptDirName now falls back to the old form for such names so the
directory is listed, and logs the name it needs to be renamed to on
the underlying remote to make it accessible again. Document this in
the crypt docs.

(cherry picked from commit 1583cce1e2)
2026-09-04 14:07:19 +01:00
CAOShurong 66bc465d1a docs: fix dead links in sia and storj backends
(cherry picked from commit 413138f56b)
2026-09-04 14:07:19 +01:00
0rangeSeaW0lf b576cbdf40 internxt: persist rotated token returned by the user info call
The refresh endpoint returns a rotated token with a fresh expiry on
every successful call, but getUserInfo discarded it, so routine use
never extended the stored token's life. Once the stored token aged
out, accounts with 2FA enabled could not recover non-interactively
and required a manual reconnect.

Carry the rotated token out of getUserInfo and persist it in NewFs
via the same jwtToOAuth2Token + oauthutil.PutToken path that
refreshJWTToken uses, keeping f.cfg.Token in sync (same pattern as
refreshOrReLogin).

Fixes #9584

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 66761670da)
2026-09-04 14:07:19 +01:00
TowyTowy 66dba8239f crypt: fix directory names which look like versioned file names
The --b2-versions support added in 3fe2aaf96 strips a version string
from the last segment of a path before encrypting it, so that the
plain text version suffixes which the underlying backend appends to
encrypted file leaf names can be handled. EncryptDirName and
DecryptDirName share that code, so the last segment of a *directory*
name was version stripped too. Only file leaf names are ever given a
version string by the backend - a directory gets a
version-string-like name from the user, and such a name is encrypted
verbatim when it appears as the parent of a file name, so the same
directory ended up with two different encryptions.

Before this change, with a directory whose name matches rclone's
version format, eg dir-v2001-02-03-040506-123:

    rclone copy file.txt crypt:dir-v2001-02-03-040506-123/
    rclone ls crypt:dir-v2001-02-03-040506-123
    # => "directory not found" - the file is invisible to listings
    rclone mkdir crypt:dir-v2001-02-03-040506-123
    # => creates a second directory with the same decrypted name

After this change EncryptDirName and DecryptDirName encrypt directory
names verbatim, so a directory encrypts the same way whether it is
named on its own or as the parent of a file. Version strings are only
added to file names by the underlying backend, so --b2-versions is
unaffected and the existing version tests are untouched.

A directory which was created by the old EncryptDirName will no longer
decrypt and will be reported as undecryptable in listings. Such
directories were already unusable - anything copied into one was
written to a different encrypted directory - so nothing which worked
before is broken by this.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 67b184d6e7)
2026-09-04 14:07:19 +01:00
Anatoly Tarnavsky 92ef010fd2 s3: fix server side copy failing with --s3-no-head-object - fixes #9629
With no_head_object set, NewObject does not read any metadata, so the
destination object returned from a server side copy had a size of 0.
The size check in operations.Copy then failed with "corrupted on
transfer: sizes differ N vs 0" and deleted the newly copied object.
This also broke Move and hence renames through rclone mount.

Populate the destination object's size and MD5 from the source object
when no_head_object is set, as a server side copy produces an object
with identical content.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6df7b8aba1)
2026-09-04 14:07:19 +01:00
Loi Nguyen feb0664b64 dropbox: fix ChangeNotify when the root's case differs from Dropbox's - fixes #9692
Dropbox is case insensitive and the path_display it returns in
change notifications may not match the case of the configured root.
Before this change the root was trimmed with a case sensitive prefix
match, so when the cases differed the full path was passed to the
ChangeNotify callback and the notification was ignored.

This trims the root case insensitively while preserving the display
case of the remaining path.

(cherry picked from commit 4af64270cc)
2026-09-04 14:07:19 +01:00
Sune Mølgaard 61427e9729 serve http: prevent scrolling to the top on page reload - fixes #9771
(cherry picked from commit bdeb95ae01)
2026-09-04 14:07:19 +01:00
Vijay Misal 3d1fd51f61 vfscache: fix log message growing without bound on repeated write errors
Write() overwrote a successful write's nil error with the stale
lastErr returned by kickWaiters() once the downloader had recorded
too many errors. download() then wrapped that stale error again and
stored it back as the new lastErr, so every subsequent write added
another "vfs reader: failed to write to cache file:" prefix - fixes #4998

(cherry picked from commit efa5e8fcc1)
2026-09-04 14:07:19 +01:00
Rayan Salhab 6c44400bf7 accounting: fix bwlimit burst overflow - fixes #9820
Co-authored-by: cyphercodes <cyphercodes@users.noreply.github.com>
(cherry picked from commit 468eccb122)
2026-09-04 14:07:19 +01:00
water 041b766428 fix: do not retry multipart upload chunk on 404 (upload session not found)
(cherry picked from commit 5d1feea7e8)
2026-09-04 14:07:19 +01:00
shaurya 3c505b1e99 docs: fix broken links and wrong s3 directory bucket flag name
Several documentation links pointed at anchors or paths that no longer
resolve, and the S3 directory buckets section named the config option
and flag in the plural, which does not match the backend.

Co-authored-by: shaurya <19599684+no-hup@users.noreply.github.com>
Co-authored-by: no-hup <shauryaj.finance@gmail.com>
(cherry picked from commit 9dbfd9d852)
2026-09-04 14:07:19 +01:00
CAOShurong 6317faccc0 s3: treat UploadPart success without ETag as retryable error
A successful UploadPart whose response carries no ETag header made
WriteChunk panic dereferencing uout.ETag in a debug log line. The part
ETag is required by CompleteMultipartUpload, so an ETag-less 200 is
unusable: return a retryable error from inside the pacer callback so
the chunk is retried instead of crashing the transfer or completing
the upload with a broken part list.

Fixes #9822

Co-authored-by: Shurong Cao <170531907+CAOShurong@users.noreply.github.com>
(cherry picked from commit 660144d311)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 5a490a31c5 docs: update sponsors
(cherry picked from commit c140d36a1f)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood b8470e4cac test_all: pikpak: ignore TestRcatSizeChecksum/Corrupted
Pikpak never returns MD5 for uploads which causes this test to fail.

Perhaps Pikpak should not declare MD5 but that is a bigger decision
being discussed in #9826

(cherry picked from commit 4369d16a1c)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood e7ae39f42d webdav: fix SetModTime failing and hashes missing on Nextcloud
Nextcloud only stores a checksum which is supplied in the OC-Checksum
header of an upload, and discards it again when the modification time
is set with PROPPATCH. Re-sending the checksum in the PROPPATCH (as is
done for ownCloud) is rejected by Nextcloud with 403 Forbidden which
made the whole PROPPATCH fail, so SetModTime returned an error on any
object which had a hash. Uploads from sources without hashes, eg
streamed uploads with `rclone rcat`, were stored with no hash at all.

Use the Nextcloud PATCH extension with the X-Recalculate-Hash header
to have the server calculate and store the SHA1 of an object after a
streamed upload and after setting the modification time. This gives
a server side hash of the stored data which also lets rclone verify
streamed uploads.

(cherry picked from commit f7c510af49)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 43f107e049 pikpak: fix truncated single part uploads reported as ok when source ends early
If the source supplied fewer bytes than its declared size, the single
part upload path accepted the short body and stored a truncated file
recorded with the declared size, reporting a successful upload. The
multipart path already checks for this.

Count the bytes actually read and fail the upload if they do not match
the declared size, which cancels the partially created file.

This was found by the FsPutShortEOF integration test.

(cherry picked from commit 8e744de5e6)
2026-09-04 14:07:19 +01:00
machsix 748eaf9b28 onedrive: fix 403 Forbidden for configuration personal onedrive
(cherry picked from commit 8869a848f2)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 11dd394670 azureblob: fix test which didn't compile
We accidentally merged this commit with non compiling tests.

bee45bccfd azureblob: fix spurious vfs cache corruption errors during chunked reads #9782

(cherry picked from commit d3a71eea36)
2026-09-04 14:07:19 +01:00
Sanjay Kanth A 21483522a6 dropbox: decode received shared-file names - fixes #9707
listSharedFolders already decoded shared-folder names with
f.opt.Enc.ToStandardName, but listReceivedFiles stored the raw name
returned by the Dropbox API unchanged. Names that require encoding
(e.g. a trailing space, which Dropbox itself rejects, so rclone
stores it as "name␠" via EncodeRightSpace) were therefore shown under
their raw, encoded form for received files instead of being decoded
back to the standard name, and findSharedFile could not resolve such
a file by its standard name.

Apply the same ToStandardName conversion listSharedFolders uses.

(cherry picked from commit f3a7aaf635)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood f92262d48c azureblob: fix spurious vfs cache corruption errors during chunked reads - fixes #9782
On a ranged download the metadata decoder stored the response's
Content-Length (the length of the range, not the blob) in the object's
size and only corrected it from the Content-Range total afterwards.
Object.Size() is read concurrently by the VFS cache and chunked reader
while a download is in progress, so with --vfs-read-chunk-size a reader
could observe the chunk length (e.g. 67108864 for 64M chunks) as the
object size. The VFS cache then logged

    vfs cache: cached file (N) is unexpectedly larger than the remote
    object (67108864). The cached file is likely corrupted after an
    unclean shutdown; recovering ...

and truncated the read request against the bogus size, breaking
sequential reads of large blobs with --vfs-cache-mode full.

This applies the Content-Range correction before the size is stored so
the range length is never published as the object size.

(cherry picked from commit bee45bccfd)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood ad87bf6c93 lib/rest: make ParseContentRange public
(cherry picked from commit 2a8d8afd0f)
2026-09-04 14:07:19 +01:00
kingston125 f1c06a047e filelu: fix duplicate root path during multipart folder creation
(cherry picked from commit 6ee1d851ec)
2026-09-04 14:07:19 +01:00
Rohit Behera d6fb148d9b huaweidrive: fix truncated files being uploaded successfully when the source ends early
The multipart upload copied the source into the request buffer without
checking how many bytes it had read, so a source that supplied fewer
bytes than its declared size was accepted by the server and reported as
a success with a truncated file stored.

Count the bytes actually read and fail the upload if they do not match
the declared size.

Signed-off-by: Rohit Behera <126186063+r0h1tb@users.noreply.github.com>
(cherry picked from commit 83b143103c)
2026-09-04 14:07:19 +01:00
Nick Craig-Wood 930de88f69 protondrive: fix files uploaded with v1.75.0 not being readable in the Proton apps
rclone v1.75.0 started creating files in Proton Drive's new
crypto-refresh encryption format, following guidance from Proton that
new file node keys should use the v6/AEAD profile. It turns out the
official Proton web app cannot decrypt files whose node key is a v6
key (but the Android app can), so every file uploaded with v1.75.0 (or
a beta after 2026-07-13) shows 'Item cannot be decrypted' in the web
app, even though rclone itself reads the files fine. Inspecting a file
created by the web app shows Proton itself still creates v4 node keys,
using the new format only for the file content.

New files are now created with the same fully pre-crypto-refresh
format as v1.74.4 (v4 node key, v3 PKESK content key, v1 SEIPD
blocks), which every Proton client can read. Reading files in the new
format still works, new revisions of files which already use the new
content format keep it, and the auxiliary fields (name, node
passphrase, extended attributes, block signatures) are pinned to the
old format regardless of the recipient key's preferences, as Proton
requires.

Files already uploaded with v1.75.0 cannot be repaired in place -
uploading a new revision does not change the file's node key. To make
such a file readable by the Proton apps again, delete it from the
remote and upload it again with a fixed version of rclone.

This updates Proton-API-Bridge to v1.0.5 and go-proton-api to v1.0.4.

See: https://forum.rclone.org/t/proton-drive-unable-to-decrypt/54087
(cherry picked from commit d9aa903358)
2026-09-04 14:07:19 +01:00
Rohit Behera fad0359e6b box: fix truncated files being uploaded successfully when the source ends early
The single-shot upload path sent the source straight to Box as a multipart
body with no Content-Length, so a source that supplied fewer bytes than its
declared size produced a short request that Box accepted and stored, and the
upload was reported as a success.

Count the bytes actually read and fail the upload if they do not match the
declared size. The multipart path already reads each chunk with io.ReadFull
and so already fails in this case.

(cherry picked from commit 64ab1ac322)
2026-09-04 14:07:18 +01:00
Rahman Yilmaz c14e507609 walk: stop directory traversal when the context is cancelled - fixes #9788
The concurrent walker created by walk() only stopped when the callback
returned an error or the whole tree had been listed. Cancelling the
context (for example via the rc job/stop endpoint for an async
operations/size or recursive operations/list call) was therefore
ignored: the checkers kept pulling list jobs from the channel and kept
listing the entire tree, burning CPU and making job cancellation
useless for every backend without a native ListR implementation.

Make every checker select on ctx.Done() so a cancelled walk shuts down
promptly through the existing quit/drain path and reports the context
error. Also check the context between directory read chunks in the
local backend so a single huge directory does not block cancellation.

(cherry picked from commit 5eb5c01e36)
2026-09-04 14:07:18 +01:00
Rohit Behera 0720a194cf yandex: fix truncated files being uploaded successfully when the source ends early
Update already wrapped the source in a counting reader but never looked at the
count, so a source that supplied fewer bytes than its declared size was
uploaded as a chunked request, accepted by the server and reported as a
success with a truncated file stored.

Compare the bytes actually read against the declared size.

(cherry picked from commit 1128693468)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 194d22ce71 build: untap aws/tap to silence homebrew tap trust warnings on macOS
(cherry picked from commit be7f9b38b0)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 7995d87cd5 s3: Mega: update endpoints
(cherry picked from commit ec3a95c279)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 2d261879dd docs: add assigned CVE numbers to the v1.75.0 security advisories in the changelog
Five of the advisories released with v1.75.0 now have CVEs assigned:

- GHSA-45pq-889g-fcgh serve restic path traversal: CVE-2026-71309
- GHSA-xhf4-832v-7xcr lib/proxy CONNECT header OOM: CVE-2026-71310
- GHSA-8c48-q9wj-3w37 ftp command injection: CVE-2026-71311
- GHSA-2m8m-jhrm-w6j2 sftp PowerShell command injection: CVE-2026-71312
- GHSA-7p4m-qxvv-g567 local file name escape: CVE-2026-71313

GHSA-6jcg-q3wp-x2f4 (squashfs) loses its CVE-PENDING marker as GitHub
declined to issue a CVE from the rclone repository - the vulnerable code
is in go-diskfs so any CVE must come from an advisory there.

GHSA-mfvx-7rcj-9m5g (pprof) keeps its CVE-PENDING marker as the CVE
request is still awaiting allocation.

(cherry picked from commit 2c1174af0d)
2026-09-04 14:07:18 +01:00
VXNCXNX 2e0718167f lib/transform: fix panic in truncate_keep_extension
Return error when extension is longer than truncation limit.

(cherry picked from commit c667e53638)
2026-09-04 14:07:18 +01:00
Shantanav Mukherjee 9718ed3fc2 docs: clarify VFS cache age semantics
(cherry picked from commit 4f22d62c66)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 645e1ea5be build: fix multiple CVEs by upgrading to go1.26.6
- CVE-2026-56860: net/url: quadratic complexity in resolvePath
- CVE-2026-56858: html/template: JavaScript regexp context tracking
- CVE-2026-56862: crypto/tls: limit handshake messages accepted post-handshake
- CVE-2026-56853: net/http: apply ReadHeaderTimeout to unencrypted HTTP/2 check
- CVE-2026-56859: encoding/xml: recursion depth guard during decode
- CVE-2026-33818: encoding/asn1: enforce maximum recursion depth
- CVE-2026-46600: net: panic parsing an invalid SVCB or HTTPS RR in dnsmessage
- CVE-2026-39821: net/http: reject ASCII-only Punycode-encoded labels in idna

This also updates the go1.25 test job to go1.25.13 which contains the
same fixes.

(cherry picked from commit f0b210a886)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 40dab7ecdd build: update golang.org/x/image to v0.45.0 to fix CVE-2026-46603
CVE-2026-46603: excessive memory allocation during VP8L decoding

This also updates golang.org/x/text to v0.41.0 as a dependency.

(cherry picked from commit 00593a96fe)
2026-09-04 14:07:18 +01:00
Dave 43a2a92950 vfs/vfscache: fix reader deadlock when the item size drops below the read offset
_dispatchWaiters decided whether a waiter was satisfied by clipping its
range against dls.src.Size(), the size of the fs.Object snapshot taken
when the Downloaders was created. _ensureDownloader decided whether to
start a downloader from Item.FindMissing, which clips against
item.info.Size instead.

When item.info.Size dropped below the offset a waiter was parked on while
the source object still reported the full size, the two disagreed.
_ensureDownloader found nothing missing so it started no downloader, and
_dispatchWaiters found the range absent so it never released the waiter.
Nothing was downloaded and no error was produced, so the error count never
reached maxErrorCount and the waiter was never woken. The reader blocked
forever with nothing logged at any level.

Wake a waiter when FindMissing reports nothing left to download for it as
well as when its data has arrived. Since _ensureDownloader starts a
downloader only when FindMissing is non empty, a waiter with nothing
missing has nothing that could ever wake it.

Fixes #9769

(cherry picked from commit 6e0c71bd27)
2026-09-04 14:07:18 +01:00
Morax 62f8f944e2 lib/rest: validate ranged responses
Add response validation for calls made with Range open options. Verify
Content-Range, Content-Length, response status, and the complete
representation size before a backend accepts the response body.

Return a shared sentinel when a server ignores a partial range so callers
can avoid retrying the same unsupported request.

(cherry picked from commit 69e5aff2a9)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 841ff2d418 crypt: fix hash mismatches with no_data_encryption on backends which check upload hashes
Before this change, when no_data_encryption was set, uploads from
local disk advertised the hash of the encrypted data even though the
data was uploaded unencrypted.

On backends which check upload hashes (eg b2) this made uploads of
small files fail with errors like "Checksum did not match data
received", and made chunked uploads store an incorrect hash so the
files failed their checksum on download with "corrupted on transfer:
SHA1 hashes differ".

See: https://forum.rclone.org/t/sha1-mismatches-on-b2-with-no-data-encryption-true/54121
(cherry picked from commit 5a0b7d6746)
2026-09-04 14:07:18 +01:00
Pastalikek65 33ab81ce6d config: redact env var config values in logs
Before this change the environment variable getters in fs/configmap.go
logged the option value with %q, so a password set via
RCLONE_CONFIG_remote_pass (or RCLONE_remote_pass) was printed in full
to the debug log. Values from the config file were already redacted,
which made the leak easy to miss.

This change routes both getters through fs.RedactOptionValue, which
looks up the option in the backend's option list: options marked
IsPassword or Sensitive log as XXX, unknown options are conservatively
redacted, and --dump auth still shows the value for debugging.

Fixes #5794

(cherry picked from commit adc7f2ebfa)
2026-09-04 14:07:18 +01:00
Dean Chen 632ff74375 docs/mount: mention nfsmount for macOS NFS mounts
The NFS section under Mounting on macOS talked about serve nfs without
pointing at rclone nfsmount, which is the command that actually does the
NFS-based mount on macOS.

Fixes #7869

Signed-off-by: Dean Chen <862469039@qq.com>
(cherry picked from commit c785ff90d7)
2026-09-04 14:07:18 +01:00
Recoordinate 39b926a2ce docs: fix doubled words
(cherry picked from commit cfdc9d0558)
2026-09-04 14:07:18 +01:00
Rodrigo Rodrigues 31f4c78c5e docs: fix typo in drive client_id section
(cherry picked from commit 8b42a38e9d)
2026-09-04 14:07:18 +01:00
Christian De Santis fcb317e226 iclouddrive: fix uploads into an app container failing with 412 - fixes #9729
Writing any file into a third-party app container - the Obsidian, Pages or
Shortcuts folders that iCloud Drive shows alongside your own - failed with

    HTTP error 412 (412 Precondition Failed) returned body:
    "{ ... \"error_code\" : \"VALIDATING_REFERENCE_ERROR\", \"reason\" :
    \"Request has out of order children to be chained but the parents were
    missing\" }"

Reading from those paths worked, and so did creating directories in them, so
the failure looked like a missing parent when the parent was plainly there.

Items in an app container live in a different zone from ordinary iCloud Drive
folders: a folder under Documents has a drive ID like
FOLDER::com.apple.CloudDocs::<uuid>, while the Obsidian container has
FOLDER::iCloud.md.obsidian::documents#o2v. DownloadFile already accounts for
this - it deconstructs the item's own ID and addresses the zone it finds - but
CreateUpload and UpdateFile hardcoded defaultZone, and UpdateFile built the
resulting Drivewsid with a hardcoded com.apple.CloudDocs as well.

So rclone asked Apple to chain the new document to a parent in
com.apple.CloudDocs while the parent lived in iCloud.md.obsidian. The parent
really was missing from the zone being addressed, which is what the error said.

Take the zone from the parent's drive ID instead, the same way the download
path does, and build the new item's ID with ConstructDriveID. Uploads outside
an app container are unaffected: their parents are in com.apple.CloudDocs, so
the derived zone is the value that was previously hardcoded.

Verified against a real remote: files now upload into an Obsidian vault inside
the container and read back correctly with an unpatched binary afterwards.

(cherry picked from commit aba403fa79)
2026-09-04 14:07:18 +01:00
nielash 82a4a63b92 accounting: fix memory leak from stats groups on long-running rcd
Before this change, `NewStats` stored the context it was created from on the
`StatsInfo`. Stats groups are never freed -- they are only evicted once there are
`--max-stats-groups` of them -- so each one kept its context, and everything reachable
from it, alive for the life of the process. As the rc creates a group per call,
that included the call's filters and their compiled regexps.

The context was only ever used to get `ci.StatsFileNameLength` from the config.
`StatsInfo` already stores that same `*fs.ConfigInfo`, read from the same
context in `NewStats`.

This change fixes the issue by passing the stored ci to `transferMap.String` and
dropping the context from `StatsInfo`.

(cherry picked from commit abae66ee1a)
2026-09-04 14:07:18 +01:00
nielash d407018995 accounting: fix memory leak on long-running rcd
Before this change, Transfer.Done closed the account of a completed transfer but
never released it, because it assigned nil to only a local copy of the pointer.
As a result, every completed transfer continued to reference its account.

An Account holds the transfer context and the source reader, and the stats keep
completed transfers around up to `MaxCompletedTransfers` per group, with groups
discarded only at --max-stats-groups. On a long-running `rclone rcd`, this adds up.

It was noticeable when running bisync repeatedly via the rc, where the transfer
context carries `b.WriteResults` (bisync's `LoggerFn`). A `*bisyncRun` holds
Path1 and Path2 listings, which can be quite large, and are not supposed to be
retained between runs. (Naturally they aren't, when running bisync on the
command line -- which is probably why we didn't notice this issue sooner.)

This change fixes the issue by releasing `tr.acc` in Done, instead of the local
copy. `tr.Snapshot` reads the byte counts off the account, so the progress is
recorded on the transfer first. That read happens before taking `tr.mu`, because
`acc.progress()` locks `acc.values.mu`, `checkReadBefore` holds that lock while
calling `StatsInfo.GetBytes`, and `StatsInfo` locks back into `Transfer` in
`Transferred` and `_removeTransfer`.

(cherry picked from commit 71a0932126)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood b5f0ea9b7b serve s3: clean up abandoned multipart uploads after --multipart-expiry
A client which started a multipart upload and vanished without either
completing or aborting it used to hold on to its resources forever.

Incomplete multipart uploads which have had no activity for
--multipart-expiry (default 24h) are now aborted and cleaned up
exactly as if the client had called AbortMultipartUpload, with a
NOTICE logged.

An upload with a part still being received is never expired, and each
completed part restarts the clock. Late operations on an expired
upload fail with NoSuchUpload, as they do on real S3 when a lifecycle
rule has aborted the upload.

Set --multipart-expiry 0 to keep incomplete uploads forever.

(cherry picked from commit 1947e4217c)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood d166eaf635 serve s3: upload all multipart uploads via the VFS
Multipart uploads used to be streamed directly to the remote with their
own PutStream machinery, bypassing the VFS, a design left over from
before the VFS could abandon a streaming write.

They are now written through the VFS exactly like plain object PUTs in
every cache mode. The parts are written, in part-number order, to a
temporary object which is renamed into place server-side on
completion.

With the default --vfs-cache-mode off the parts stream through the VFS
to the remote as they arrive. With --vfs-cache-mode writes or above
they are buffered in the VFS cache and uploaded by its write-back.

User visible changes:

- Multipart uploads now show in rclone's transfer stats and obey
  --bwlimit (previously they bypassed both).
- Remotes without streaming upload support now spool the upload to a
  temporary file on local disk instead of buffering it in memory.
- Multipart uploads are never buffered in memory because of missing
  remote capabilities - only --disable-multipart-streaming does that.
- Remotes that upload atomically now also write to a temporary object
  renamed into place, so an in-progress multipart upload is no longer
  briefly visible under its final key.
- On the few remotes with no server-side move or copy the parts are
  written straight to the final object in all cache modes.
- With --vfs-cache-mode writes, plain PUTs and multipart uploads to the
  same key go through the same cache entry, so an earlier PUT still in
  the write-back window can no longer be written back over a newer
  multipart upload.
- Failed write-backs are retried by the VFS without the client having
  to restart the upload, and completed objects are served from the
  cache for read-after-write.

(cherry picked from commit b4db289d0a)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 62a1df7271 serve s3: reserve the .rclone_temp_ prefix for temporary objects
The temporary objects that uploads are written to before being renamed
into place are now named .rclone_temp_put_* and .rclone_temp_multipart_*,
and the whole .rclone_temp_ prefix is reserved: any object whose name
starts with it is hidden from S3 listings. This gives a single pattern
for cleaning up leftovers from killed servers:

    rclone delete --min-age 24h --include ".rclone_temp_*" remote:path

The .rclone_multipart_upload_* objects rclone v1.75 used are still
hidden from listings so leftovers from an older server stay invisible
to S3 clients.

(cherry picked from commit 0aa90200bd)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood e171f996d6 serve s3: fix modtime not being set when only mtime metadata is supplied on PUT
The mtime metadata fallback was nested inside the X-Amz-Meta-Mtime
branch, so it only ran when X-Amz-Meta-Mtime was present but invalid -
and then set the modtime from the invalid value's failed parse rather
than parsing mtime. An object PUT with only mtime metadata kept the
upload time as its modtime.

Now the two keys are checked independently, as TouchObject already
does.

(cherry picked from commit a3489456de)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 2bf9e0d20e serve s3: fix crash when a multipart upload is aborted while a part is uploading
streamPart did not check whether the upload had been torn down, but
AbortMultipartUpload sets the reorder buffer map to nil, so an abort
arriving while a part body was still being received panicked with an
assignment to a nil map once the part was buffered.

Now a part whose upload has been aborted or completed under it is
rejected with NoSuchUpload.

(cherry picked from commit 531d873bd7)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood afb1daa8fa serve s3: fix failed uploads deleting or corrupting the object at the key - fixes #9718
A PUT which failed part way through removed the object at the
destination key. As well as differing from real S3 (where a failed PUT
never affects the stored object), this raced with the client's
automatic retry of the same PUT: the retry stored the object and
returned 200 OK, then the failed first attempt's cleanup deleted it,
silently losing an acknowledged upload. The interrupted upload could
also be committed as a truncated object, since closing the write handle
gave the streaming upload a clean end of stream.

Now a failed or interrupted PUT never disturbs the object at the key:

- The object at the key is never removed on error.
- On backends where a partial upload is visible at its final name
  (PartialUploads), and when the VFS cache mode is writes or above, the
  upload is written to a temporary object which is renamed into place
  on success and removed on failure, as streamed multipart uploads
  already do. Backends which upload atomically are still streamed
  straight to the destination.
- An interrupted or short body fails the upload via
  WriteFileHandle.CloseWithError instead of committing truncated data,
  and a body which ends cleanly short of its declared size is rejected
  with IncompleteBody.

(cherry picked from commit 84298fc090)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 5775f0430b vfs: add WriteFileHandle.CloseWithError to abandon streaming writes
Closing a streaming write handle sends a clean EOF to the backend
upload, so a writer which knows its data source failed part way through
had no way to stop the truncated file being stored as if it were
complete.

CloseWithError closes the handle failing the upload with the given
reason instead. EOF-like reasons are rewritten so the layers reading
the pipe can't mistake them for a clean end of stream and commit the
partial file.

Needed by serve s3 to abort interrupted PUTs - see #9718

(cherry picked from commit 2f0657c35b)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood d0f00a22d5 internetarchive: fix corrupted files being created when the source ends early
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.

Count the bytes actually read from the source and if they do not match
the declared size return an error.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit 7357fb82a9)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood ad7e0c64af sia: fix corrupted files being created when the source ends early
If the source supplied fewer bytes than its declared size, the upload
request failed but a retry could report success even though the stored
file was truncated, because the retry re-sent an already exhausted
reader.

Count the bytes actually read from the source and if they do not match
the declared size, remove the partially uploaded file and return an
error.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit 5d057549b9)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 674d7a8a70 pikpak: fix truncated files being created when the source ends early
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated file and
reporting a successful upload.

Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit a2baa978db)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 1274301521 multipart: fix chunked uploads storing truncated objects when the source ends early
If the source supplied fewer bytes than its declared size, the
multipart upload was completed anyway, storing a truncated object and
reporting a successful upload.

Check the number of bytes read from the source against the declared
size before finalising and abort the upload with an error if they do
not match.

This affects all backends which use lib/multipart for chunked uploads:
azureblob, b2, drime, internxt, oracleobjectstorage, s3 and shade.

This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.

(cherry picked from commit 5b7cc097e3)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 840755543f filelu: fix truncated files being uploaded successfully when the source ends early
If the source supplied fewer bytes than its declared size, the
truncated file was stored and the upload reported success with the
object claiming the declared size.

Count the bytes actually read from the source and if they do not match
the declared size, remove the truncated file and return an error.

This was found by the new TestRcatSizeShortEOF integration test.

(cherry picked from commit e1bf9405e2)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 2472718b7c azurefiles: fix zero padded files being created when the source ends early
The file is created at the declared size and the data then written
with ranged writes, so if the source supplied fewer bytes than
declared, the remainder of the file was left as zeroes and the upload
reported success.

Count the bytes actually read from the source and if they do not match
the declared size, delete the partially uploaded file (if newly
created) and return an error.

This was found by the new FsPutShortEOF and TestRcatSizeShortEOF
integration tests.

(cherry picked from commit 884b28c203)
2026-09-04 14:07:18 +01:00
nielash a4f1c1558e bisync: fix memory leak when running via the rc
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.

For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.

This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.

(cherry picked from commit 387b4b5e11)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 76d5c49ae6 docs: update sponsor links
(cherry picked from commit 1aa9efef17)
2026-09-04 14:07:18 +01:00
Edward Klesel 9fec073ab7 azureblob: fix Entra ID server-side copy source authentication
When using Microsoft Entra ID credentials, Azure Blob server-side copy
uses a user delegation SAS URL for the private copy source.

The SAS start time was set to the current local time. Azure Storage
validates the copy source from the service side, and small clock
differences can make that SAS appear not yet valid. The service then
returns 403 CannotVerifyCopySource with AuthenticationFailed.

Start the copy-source SAS 15 minutes in the past, matching Microsoft SAS
guidance for clock skew.

(cherry picked from commit e4c7aca6bd)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood a9c4548d92 overview: fix "internal error: no overview data found" on 32 bit architectures - fixes #9723
The precision field in the backend overview YAML files can hold
fs.ModTimeNotSupported (100 years in nanoseconds) which overflows int
on 32 bit platforms, making the YAML for those backends fail to parse
and causing rclone to log 18 internal errors on every invocation.

Use int64 for the precision field and add a test that parses every
embedded backend YAML file so this is caught on 32 bit test runs.

(cherry picked from commit 5629f2668c)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood d038b8b1c9 protondrive: fix corrupted uploads after a retried upload error - fixes #9722
When an upload failed part way through with a retryable error (eg a
502 from the block storage servers) the pacer retried the whole upload
call with the same input stream. The stream had already been partially
consumed, so the retry re-created the upload draft and committed just
the remainder of the stream as a complete file, silently truncating
it. With restic over serve restic this corrupted the repository as the
truncated pack was reported as successfully uploaded.

This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.

(cherry picked from commit a06df7a2de)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 3ceea42329 operations: fix silent truncation of streaming uploads whose source ends early
Uploads through RcatSize with a known size - used by rcat --size, the
rc operations/uploadfile and the serve backends, eg serve restic - did
not check the size of the uploaded object. If the source stream ended
before the declared size worth of data had been read, the truncated
object was reported as a successful upload. This could corrupt data
for callers which trust the result, eg a restic repository accessed
via serve restic (see #9722).

This adds the same size check operations.Copy performs after a copy,
respecting --ignore-size and backends which do not report sizes.

(cherry picked from commit 9b13247ba8)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 6beb4a5bd2 docs: fix width of sponsor images on very big screens
(cherry picked from commit 1318962a96)
2026-09-04 14:07:18 +01:00
Anton Karpov 7d70921bf5 docs: use the --dump form for the obsolete --dump-* flags
The SFTP page said `--dump-auth`, which no longer exists: it became a
value of `--dump`, so the docs asked for a flag rclone would reject.

The same line, and a line in the Swift troubleshooting section, also
used `--dump-headers` and `--dump-bodies`. Those still parse, but
SetFlags logs "--dump-headers is obsolete - please use --dump headers
instead", so the docs were steering readers onto a deprecated form.

The generated flag listings in docs/content/flags.md and
docs/content/commands/rclone.md are left alone: those flags do still
exist, so `--help` output should keep showing them.

(cherry picked from commit 6cb4732cc3)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 76c6992260 compress: fix corrupted objects being created when the source ends early
When the source supplied fewer bytes than its declared size, the
compressed data file was stored under a name containing the declared
size while the metadata recorded the actual number of bytes read.
NewObject looks the data file up by the size in the metadata, so the
resulting object could never be read again, and the upload reported
success.

Check that the number of bytes read matches the declared size after
uploading the data and before writing the metadata, and remove the
data file and return an error if it does not.

This was found by the new FsPutShortEOF integration test.

(cherry picked from commit 18fa445ffc)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood f1297d6298 dropbox: don't retry chunked upload requests when the upload has been cancelled
The append loop retries everything once the upload session has
started, so a cancelled context error was retried through all the low
level retries with exponential backoff before the upload gave up.

(cherry picked from commit e0701daea0)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood 9a112f40c1 dropbox: fix chunked uploads of truncated files never finishing - fixes #9704
A source which returned EOF before supplying as many bytes as it
declared would either commit a truncated file (if the shortfall was
within the final chunk) or loop forever appending empty chunks to the
upload session. Return an error wrapping io.ErrUnexpectedEOF instead.

Note that all dropbox uploads use the chunked upload path with the
default batch_mode of sync, so this affected uploads of every size.

(cherry picked from commit bff17664ad)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood a3dcf7f0ad fstests: check uploads terminate when the source ends before its declared size
A source reader which returns EOF before supplying as many bytes as it
declared should not cause a backend Put to hang, panic or create an
object claiming the declared size.

The layers above detect a truncated transfer by comparing sizes after
the upload, so a backend may either return an error or create a
shorter object.

See #9704

(cherry picked from commit 03f8582a55)
2026-09-04 14:07:18 +01:00
alliasgher 3084177049 smb: reuse the upload connection for SetModTime - fixes #9675
Object.Update held its connection until the deferred putConnection ran at
function exit, so the SetModTime it does at the end of every upload had to take
a second connection from the pool, dialling a whole new SMB session when the
pool was empty. With N transfers in flight the pool grew to roughly 2N sessions
for no reason.

Return the connection as soon as the file is closed. At that point the upload
has succeeded and remove() can no longer be reached, so nothing else needs it,
and SetModTime picks the same connection straight back out of the pool.
putConnection nils the pointer, so the deferred putConnection becomes a no-op
and the connection is not returned twice.

(cherry picked from commit ea9a64c751)
2026-09-04 14:07:18 +01:00
alliasgher e4b778b5e3 drive: don't list trashed files when removing a directory into the trash - fixes #9681
purgeCheck listed a directory's children with includeAll set, so the Drive API
returned trashed children as well as live ones. The only thing that came of
that was the trashedFiles flag, which is used solely to decide whether to trash
the directory or delete it permanently, and when use_trash is on the directory
gets trashed either way.

That made the common case pay for nothing: after trashing N files in a
directory, removing the now empty directory paged through all N trashed entries,
about one API call and pacer delay per 1000, instead of running a single query
that returns nothing.

Ask the server to filter out trashed children when the directory is going to be
trashed anyway. Hard deletes still enumerate them, since there the distinction
decides whether the directory can be removed permanently (#1040), and
--drive-trashed-only is left alone because it deliberately lists trashed items.

(cherry picked from commit 065af89635)
2026-09-04 14:07:18 +01:00
Nick Craig-Wood ee363b1826 build: fix intermittent CI failures by raising the test timeout to 20m
The cmd/gitannex end to end tests routinely take 230-530s and on slow
CI runners the package exceeds the go test default timeout of 10m.

Raise the per-package timeout to 20m in quicktest and racequicktest to
give slow runners headroom.

(cherry picked from commit 01d32dbde5)
2026-09-04 14:07:18 +01:00
Loi Nguyen e7f9c7bc10 smb: test closing failed dial connections
Followup to:

74f9f182aa smb: fix TCP connection leak when connection setup fails - #9678

(cherry picked from commit 23f1dbfe50)
2026-09-04 14:07:18 +01:00
debaditya 10eb7b5b10 dropbox: fix context cancellation (e.g. --max-duration limit) not stopping in-flight requests
This switches the calls to the dropbox SDK over to use the context taking
variants. This allows context cancellation to propagate properly.

Fixes #9688

(cherry picked from commit c97ab80ca1)
2026-09-04 14:07:18 +01:00
Loi Nguyen e0faeeb5d6 vfs: make poll shutdown lifecycle deterministic
(cherry picked from commit 1d03a77172)
2026-09-04 14:07:17 +01:00
Loi Nguyen dcf15b5f9c vfs: test poll interval update lifecycle
(cherry picked from commit d5275c4ebb)
2026-09-04 14:07:17 +01:00
Loi Nguyen c93cc1e1ae vfs: synchronize poll updates with shutdown - fixes #9689
(cherry picked from commit f132aef295)
2026-09-04 14:07:17 +01:00
Loi Nguyen 21c98ff5ff dropbox: preserve Paper export paths on lookup - fixes #9691
Direct lookups of exported Dropbox Paper files retained the
caller-visible extension before export metadata processing appended it
again. Track when metadata was resolved through an export path so the
object keeps the requested remote name while listing behavior remains
unchanged.

(cherry picked from commit 8c8062da78)
2026-09-04 14:07:17 +01:00
Loi Nguyen 5fbc677ed4 lib/batcher: prevent commits racing shutdown - fixes #9687
(cherry picked from commit 7f6207fae2)
2026-09-04 14:07:17 +01:00
SillyZir 00347ecf36 serve nfs: fix EIO when creating symlinks with --vfs-links
The macOS NFS client sends SETATTR after SYMLINK, which arrives as
Chmod/Chown on the link path. These opened the target with vfs.Open,
which follows symlinks - a freshly created symlink usually dangles, so
the open failed with ENOENT, surfaced to the client as NFS3ERR_IO even
though the link was created.

Add path-based VFS.Chmod and VFS.Chown mirroring VFS.Chtimes. They do
not follow symlinks (lstat semantics, matching VFS.Stat) and return
ENOSYS when the node exists, since the VFS stores neither permissions
nor ownership; serve nfs calls them and masks ENOSYS as before.

Fixes #9627

(cherry picked from commit 7804c1b315)
2026-09-04 14:07:17 +01:00
867 changed files with 72782 additions and 102278 deletions

No files matched your search

-25
View File
@@ -1,25 +0,0 @@
{
"permissions": {
"allow": [
"Bash(git status:*)", "Bash(git diff:*)", "Bash(git log:*)", "Bash(git show:*)",
"Bash(git rev-parse:*)", "Bash(git add:*)", "Bash(git commit:*)",
"Bash(go mod download:*)", "Bash(go build:*)", "Bash(go test:*)", "Bash(go vet:*)",
"Bash(gofmt -l:*)",
"Bash(golangci-lint run:*)"
],
"ask": [
"Bash(git push:*)", "Bash(git merge:*)", "Bash(git rebase:*)", "Bash(git switch:*)",
"Bash(git restore:*)", "Bash(git commit --amend:*)", "Bash(rm:*)",
"Bash(go get:*)", "Bash(go mod tidy:*)", "Bash(go install:*)",
"Bash(make:*)", "Bash(docker:*)"
],
"deny": [
"Bash(git push --force:*)", "Bash(git push -f:*)", "Bash(git push --force-with-lease:*)",
"Bash(git reset --hard:*)", "Bash(git clean:*)", "Bash(git add -f:*)", "Bash(git add --force:*)",
"Bash(git commit --no-verify:*)", "Bash(git commit -n:*)",
"Bash(rm -r:*)", "Bash(rm -R:*)", "Bash(rm -rf:*)", "Bash(rm -fr:*)", "Bash(rm -Rf:*)",
"Bash(rm -fR:*)", "Bash(rm --recursive:*)", "Bash(sudo:*)",
"Read(./.env)", "Read(./.env.*)", "Read(./secrets/**)"
]
}
}
-2
View File
@@ -1,2 +0,0 @@
sandbox_mode = "workspace-write"
approval_policy = "on-request"
-20
View File
@@ -1,20 +0,0 @@
# Rendered from docs/PERMISSIONS.md - change both in the same commit.
# Deny
prefix_rule(pattern = ["git", "push", ["--force", "-f", "--force-with-lease"]], decision = "forbidden",
justification = "Force-push rewrites shared history.")
prefix_rule(pattern = ["git", "reset", "--hard"], decision = "forbidden", justification = "Discards work.")
prefix_rule(pattern = ["git", "clean"], decision = "forbidden", justification = "Deletes untracked files.")
prefix_rule(pattern = ["git", "add", ["-f", "--force"]], decision = "forbidden",
justification = "Stages ignored files, e.g. secrets.")
prefix_rule(pattern = ["git", "commit", ["--no-verify", "-n"]], decision = "forbidden", justification = "Skips hooks.")
prefix_rule(pattern = ["rm", ["-r", "-R", "-rf", "-fr", "-Rf", "-fR", "--recursive"]], decision = "forbidden",
justification = "Recursive delete.")
prefix_rule(pattern = ["sudo"], decision = "forbidden", justification = "No privilege escalation.")
# Ask
prefix_rule(pattern = ["git", ["push", "merge", "rebase", "switch", "restore"]], decision = "prompt")
prefix_rule(pattern = ["git", "commit", "--amend"], decision = "prompt")
prefix_rule(pattern = ["rm"], decision = "prompt")
prefix_rule(pattern = ["go", ["get", "install"]], decision = "prompt")
prefix_rule(pattern = ["go", "mod", "tidy"], decision = "prompt")
prefix_rule(pattern = ["make"], decision = "prompt")
prefix_rule(pattern = ["docker"], decision = "prompt")
-20
View File
@@ -1,20 +0,0 @@
{
"tools": {
"allowed": [
"run_shell_command(git status)", "run_shell_command(git diff)", "run_shell_command(git log)",
"run_shell_command(git show)", "run_shell_command(git branch --list)",
"run_shell_command(git add)", "run_shell_command(git commit)",
"run_shell_command(go mod download)", "run_shell_command(go build)",
"run_shell_command(go test)", "run_shell_command(go vet)", "run_shell_command(gofmt -l)",
"run_shell_command(golangci-lint run)"
],
"exclude": [
"run_shell_command(git push --force)", "run_shell_command(git push -f)",
"run_shell_command(git push --force-with-lease)", "run_shell_command(git reset --hard)",
"run_shell_command(git clean)", "run_shell_command(git add -f)", "run_shell_command(git add --force)",
"run_shell_command(git commit --no-verify)", "run_shell_command(git commit -n)",
"run_shell_command(rm -r)", "run_shell_command(rm -R)", "run_shell_command(rm -rf)",
"run_shell_command(rm -fr)", "run_shell_command(rm -Rf)", "run_shell_command(sudo)"
]
}
}
-4
View File
@@ -2,13 +2,9 @@
go.mod text eol=lf
go.sum text eol=lf
# Embedded templates must render the same on every OS
lib/http/templates/** text eol=lf
# Ignore generated files in GitHub language statistics and diffs
/MANUAL.* linguist-generated=true
/rclone.1 linguist-generated=true
cmd/bisync/testdata/*/golden/** linguist-generated=true
# Don't fiddle with the line endings of test data
**/testdata/** -text
-56
View File
@@ -1,56 +0,0 @@
# Fork CI for the Proton Drive backend. Every step is hermetic: the
# tests run against the fake server in backend/protondrive/protontest.
name: ci
on:
push:
pull_request:
env:
# Never pick up a config holding a real Proton remote.
RCLONE_CONFIG: /notfound
jobs:
protondrive:
runs-on: ubuntu-latest
# The race suite takes about 6 minutes on a fast machine and 7 on
# two cores; the slow runner may take several times that.
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Install Go
uses: actions/setup-go@v5
with:
# The line the code was verified with, not go.mod's minimum,
# so govulncheck sees a patched standard library.
go-version: '1.27.x'
# Caches the module and build caches, keyed on go.sum.
cache: true
- name: Build
run: go build ./...
- name: Vet
run: go vet ./backend/protondrive/...
- name: Lint
uses: golangci/golangci-lint-action@v8
with:
version: v2.15.0
args: ./backend/protondrive/...
- name: Test
# Go's default 10 minute limit per test binary is too short
# for the race suite on a slow runner.
run: go test -race -timeout 40m ./backend/protondrive/...
- name: Fuzz
run: |
for target in $(go test -list '^Fuzz' ./backend/protondrive/ | grep '^Fuzz'); do
go test -run '^$' -fuzz "^${target}\$" -fuzztime 20s ./backend/protondrive/
done
- name: Vulnerability check
run: go run golang.org/x/vuln/cmd/govulncheck@v1.9.0 ./backend/protondrive/...
+12 -15
View File
@@ -37,7 +37,7 @@ jobs:
include:
- job_name: linux
os: ubuntu-latest
go: '~1.27.3'
go: '~1.26.6'
gotags: cmount
build_flags: '-include "^linux/"'
check: true
@@ -48,14 +48,14 @@ jobs:
- job_name: linux_386
os: ubuntu-latest
go: '~1.27.3'
go: '~1.26.6'
goarch: 386
gotags: cmount
quicktest: true
- job_name: mac_amd64
os: macos-latest
go: '~1.27.3'
go: '~1.26.6'
gotags: 'cmount'
build_flags: '-include "^darwin/amd64" -cgo'
quicktest: true
@@ -64,14 +64,14 @@ jobs:
- job_name: mac_arm64
os: macos-latest
go: '~1.27.3'
go: '~1.26.6'
gotags: 'cmount'
build_flags: '-include "^darwin/arm64" -cgo -macos-arch arm64 -cgo-cflags=-I/usr/local/include -cgo-ldflags=-L/usr/local/lib'
deploy: true
- job_name: windows
os: windows-latest
go: '~1.27.3'
go: '~1.26.6'
gotags: cmount
cgo: '0'
build_flags: '-include "^windows/"'
@@ -81,14 +81,14 @@ jobs:
- job_name: other_os
os: ubuntu-latest
go: '~1.27.3'
go: '~1.26.6'
build_flags: '-exclude "^(windows/|darwin/|linux/)"'
compile_all: true
deploy: true
- job_name: go1.26
os: ubuntu-latest
go: '~1.26.10'
go: '~1.26.6'
quicktest: true
racequicktest: true
@@ -177,12 +177,9 @@ jobs:
# https://github.com/Homebrew/brew/issues/15621#issuecomment-1619266788
# https://github.com/orgs/Homebrew/discussions/4612#discussioncomment-6319008
unset HOMEBREW_NO_INSTALL_FROM_API
# brew untap fails if the tap isn't installed, which varies between runner images
for tap in homebrew/core homebrew/cask aws/tap; do
if brew tap | grep -qx "$tap"; then
brew untap --force "$tap"
fi
done
brew untap --force homebrew/core
brew untap --force homebrew/cask
brew untap --force aws/tap
brew update
brew install --cask macfuse
brew install git-annex git-annex-remote-rclone
@@ -287,7 +284,7 @@ jobs:
id: setup-go
uses: actions/setup-go@v7
with:
go-version: '~1.27.3'
go-version: '~1.26.6'
check-latest: true
cache: false
@@ -402,7 +399,7 @@ jobs:
id: setup-go
uses: actions/setup-go@v7
with:
go-version: '~1.27.3'
go-version: '~1.26.6'
# Caching is handled explicitly below to share the module cache
# with the other jobs - see the build job for the rationale.
cache: false
-8
View File
@@ -19,12 +19,4 @@ Thumbs.db
__pycache__
.DS_Store
resource_windows_*.syso
/librclone.h
.devcontainer
/cmd/bisync/testdata/bisync_vscode_debuggers_launch.json
.env
.env.*
CLAUDE.local.md
.claude/settings.local.json
.worktrees/
.superpowers/
-15
View File
@@ -20,8 +20,6 @@ The core parts of rclone under `fs` and `vfs` need to work with all backends and
Make sure added tests **actually test the code you have written** and test the intention behind the change. If you are fixing a problem, write the tests first to reproduce the problem before starting on the fix.
Tests in `fs/operations`, `fs/sync`, `cmd/bisync`, `cmd/gitannex` and `vfs` (the definitive list is in `fstest/test_all/config.yaml`) are run on the integration test machine against **all** the backends, so they **must be prepared to run on backends with limited functionality**. Ideally tests, or parts of tests, should be skipped using the `Fs.Features()` flags on backends which can't support them.
## Build and Test Commands
```bash
@@ -134,16 +132,3 @@ Uses golangci-lint v2 with config in `.golangci.yml`. Enabled linters: errcheck,
- Command docs are in the command source code (e.g., `cmd/ls/ls.go`).
- Don't commit autogenerated doc changes from `make backenddocs` or `make commanddocs`.
- Website docs are in `docs/content/` as markdown, built with Hugo (`make serve` to preview).
## Fork Workflow
This fork carries agent tooling that upstream does not. Never include `docs/specs/`, `docs/plans/`, `docs/{ARCHITECTURE,DECISIONS,TODO,PERMISSIONS}.md`, `GEMINI.md`, `.claude/`, `.codex/`, `.gemini/` or this section in a PR to rclone/rclone.
- Required skill: `superpowers` - check at session start, install if missing. There is no product UI here, so `impeccable` is not needed.
- Specs go in `docs/specs/NNN-slug.md` (permanent). Plans go in `docs/plans/NNN-slug.md` and are removed with `git rm` before merge. This overrides the superpowers default paths.
- Paused or in-flight work is listed under "Now" in `docs/TODO.md` with the handoff to read first.
- Non-trivial work: brainstorm → spec → plan → TDD → review. For trivial fixes say "trivial, skipping spec" and proceed.
- Log things noticed but deliberately not fixed in `docs/TODO.md`, record decisions in `docs/DECISIONS.md`, and keep `docs/ARCHITECTURE.md` in step with structural changes - in the same commit as the code.
- `docs/PERMISSIONS.md` is the source for the `.claude/`, `.codex/` and `.gemini/` permission files - change them together.
- Never push, force-push or commit directly to `master`. Ask before adding or upgrading a dependency (`go get`, `go mod tidy`).
- Done means run in this session and seen passing: `go build`, `go test` on the touched packages, and `golangci-lint run` on them.
+1 -5
View File
@@ -35,21 +35,17 @@ RUN echo "**** Print Version Binary ****" && \
# Begin final image
FROM alpine:latest
ARG RCLONE_UID=1009
ARG RCLONE_GID=1009
RUN echo "**** Install Dependencies ****" && \
apk add --no-cache \
ca-certificates \
fuse3 \
mailcap \
tzdata && \
echo "Enable user_allow_other in fuse" && \
echo "user_allow_other" >> /etc/fuse.conf
COPY --from=builder /go/src/github.com/rclone/rclone/rclone /usr/local/bin/
RUN addgroup -g "${RCLONE_GID}" rclone && adduser -u "${RCLONE_UID}" -Ds /bin/sh -G rclone rclone
RUN addgroup -g 1009 rclone && adduser -u 1009 -Ds /bin/sh -G rclone rclone
ENTRYPOINT [ "rclone" ]
-8
View File
@@ -1,8 +0,0 @@
# GEMINI.md
**`AGENTS.md` is the single source of truth for this repository.** Read
`./AGENTS.md` before doing anything else and follow it exactly.
This file exists only so Gemini CLI finds the rules. It contains no project
rules of its own - add those to `AGENTS.md`. If anything here ever conflicts
with `AGENTS.md`, `AGENTS.md` wins.
Generated
+2625 -6888
View File
File diff suppressed because it is too large. Load diff
Generated
+536 -3912
View File
File diff suppressed because it is too large. Load diff
Generated
+597 -4317
View File
File diff suppressed because it is too large. Load diff
+14 -25
View File
@@ -38,9 +38,6 @@ BETA_PATH := $(BRANCH_PATH)$(TAG)$(BETA_SUBDIR)
BETA_URL := https://beta.rclone.org/$(BETA_PATH)/
BETA_UPLOAD_ROOT := beta.rclone.org:
BETA_UPLOAD := $(BETA_UPLOAD_ROOT)/$(BETA_PATH)
# Update the beta site's directory listings after an upload - add
# --changed for the uploaded directories
BETA_INDEX := --config bin/ci.rclone.conf -v index $(BETA_UPLOAD_ROOT) --use-server-modtime --checkers 16 --output index.html=html --output index.json=caddy --header-upload "Cache-Control: public, max-age=60" --exclude /favicon.ico
# Pass in GOTAGS=xyz on the make command line to set build tags
ifdef GOTAGS
BUILDTAGS=-tags "$(GOTAGS)"
@@ -237,7 +234,6 @@ check_sign:
upload:
rclone -P copy build/ downloads.rclone.org:/$(TAG)
rclone lsf build --files-only --include '*.{zip,deb,rpm}' --include version.txt | xargs -i bash -c 'i={}; j="$$i"; [[ $$i =~ (.*)(-v[0-9\.]+-)(.*) ]] && j=$${BASH_REMATCH[1]}-current-$${BASH_REMATCH[3]}; rclone copyto -v "downloads.rclone.org:/$(TAG)/$$i" "downloads.rclone.org:/$$j"'
rclone -P index downloads.rclone.org: --changed /$(TAG)/ --checkers 16 --output index.html=html --output index.json=caddy --header-upload "Cache-Control: public, max-age=60" --exclude /favicon.ico
upload_github:
./bin/upload-github $(TAG)
@@ -270,7 +266,6 @@ ci_upload:
ifeq ($(or $(BRANCH_PATH),$(RELEASE_TAG)),)
./rclone --no-check-dest --config bin/ci.rclone.conf -v copy build/ $(BETA_UPLOAD_ROOT)/test/testbuilds-latest
endif
./rclone $(BETA_INDEX) --changed $(BETA_PATH)/testbuilds/ $(if $(or $(BRANCH_PATH),$(RELEASE_TAG)),,--changed test/testbuilds-latest/)
@echo Beta release ready at $(BETA_URL)/testbuilds
ci_beta:
@@ -280,12 +275,11 @@ ci_beta:
ifeq ($(or $(BRANCH_PATH),$(RELEASE_TAG)),)
rclone --no-check-dest --config bin/ci.rclone.conf -v copy --include '*beta-latest*' --include version.txt build/ $(BETA_UPLOAD_ROOT)$(BETA_SUBDIR)
endif
rclone $(BETA_INDEX) --changed $(BETA_PATH)/ $(if $(or $(BRANCH_PATH),$(RELEASE_TAG)),,$(if $(BETA_SUBDIR),--changed $(BETA_SUBDIR)/))
@echo Beta release ready at $(BETA_URL)
# Fetch the binary builds from GitHub actions
fetch_binaries:
rclone -P sync --exclude "/testbuilds/**" --exclude "/index.{html,json}" --delete-excluded $(BETA_UPLOAD) build/
rclone -P sync --exclude "/testbuilds/**" --delete-excluded $(BETA_UPLOAD) build/
serve: website
cd docs && hugo server --logLevel info -w --disableFastRender --ignoreCache
@@ -302,26 +296,21 @@ retag:
@echo "Version is $(VERSION)"
git tag -f -s -m "Version $(VERSION)" $(VERSION)
# Start development of the next version after a release: the next
# minor version on master or the next patch version on a stable branch.
startdev:
@branch=$$(git rev-parse --abbrev-ref HEAD); \
case "$$branch" in \
master) next="$(NEXT_VERSION)" ;; \
v*-stable) next="$(NEXT_PATCH_VERSION)" ;; \
*) echo "Branch $$branch isn't master or a stable branch" >&2; exit 1 ;; \
esac; \
if ! git rev-parse -q --verify "refs/tags/$(VERSION)" >/dev/null; then \
echo "$(VERSION) hasn't been tagged yet - has development already been started?" >&2; exit 1; \
fi; \
echo "Version is $(VERSION)"; \
echo "Next version is $$next"; \
echo -e "package fs\n\n// VersionTag of rclone\nvar VersionTag = \"$$next\"\n" | gofmt > fs/versiontag.go; \
echo -n "$$next" > docs/layouts/partials/version.html; \
echo "$$next" > VERSION; \
git commit -m "Start $$next-DEV development" fs/versiontag.go VERSION docs/layouts/partials/version.html
@echo "Version is $(VERSION)"
@echo "Next version is $(NEXT_VERSION)"
echo -e "package fs\n\n// VersionTag of rclone\nvar VersionTag = \"$(NEXT_VERSION)\"\n" | gofmt > fs/versiontag.go
echo -n "$(NEXT_VERSION)" > docs/layouts/partials/version.html
echo "$(NEXT_VERSION)" > VERSION
git commit -m "Start $(NEXT_VERSION)-DEV development" fs/versiontag.go VERSION docs/layouts/partials/version.html
startstable: startdev
startstable:
@echo "Version is $(VERSION)"
@echo "Next stable version is $(NEXT_PATCH_VERSION)"
echo -e "package fs\n\n// VersionTag of rclone\nvar VersionTag = \"$(NEXT_PATCH_VERSION)\"\n" | gofmt > fs/versiontag.go
echo -n "$(NEXT_PATCH_VERSION)" > docs/layouts/partials/version.html
echo "$(NEXT_PATCH_VERSION)" > VERSION
git commit -m "Start $(NEXT_PATCH_VERSION)-DEV development" fs/versiontag.go VERSION docs/layouts/partials/version.html
winzip:
zip -9 rclone-$(TAG).zip rclone.exe
-9
View File
@@ -24,25 +24,21 @@ directories to and from different cloud storage providers.
## Storage providers
- 1Fichier [:page_facing_up:](https://rclone.org/fichier/)
- Abrha Storage [:page_facing_up:](https://rclone.org/s3/#abrhastorage)
- Akamai Netstorage [:page_facing_up:](https://rclone.org/netstorage/)
- Alibaba Cloud (Aliyun) Object Storage System (OSS) [:page_facing_up:](https://rclone.org/s3/#alibaba-oss)
- Amazon S3 [:page_facing_up:](https://rclone.org/s3/)
- ArvanCloud Object Storage (AOS) [:page_facing_up:](https://rclone.org/s3/#arvan-cloud-object-storage-aos)
- Autonomys Auto Drive [:page_facing_up:](https://rclone.org/s3/#autonomys)
- Bizfly Cloud Simple Storage [:page_facing_up:](https://rclone.org/s3/#bizflycloud)
- Backblaze B2 [:page_facing_up:](https://rclone.org/b2/)
- Box [:page_facing_up:](https://rclone.org/box/)
- Ceph [:page_facing_up:](https://rclone.org/s3/#ceph)
- China Mobile Ecloud Elastic Object Storage (EOS) [:page_facing_up:](https://rclone.org/s3/#china-mobile-ecloud-eos)
- Citrix ShareFile [:page_facing_up:](https://rclone.org/sharefile/)
- Clever Cloud Cellar [:page_facing_up:](https://rclone.org/s3/#clever-cloud-cellar)
- Cloudflare R2 [:page_facing_up:](https://rclone.org/s3/#cloudflare-r2)
- Cloudinary [:page_facing_up:](https://rclone.org/cloudinary/)
- Cubbit DS3 [:page_facing_up:](https://rclone.org/s3/#Cubbit)
- DigitalOcean Spaces [:page_facing_up:](https://rclone.org/s3/#digitalocean-spaces)
- Digi Storage [:page_facing_up:](https://rclone.org/koofr/#digi-storage)
- dosya.dev [:page_facing_up:](https://rclone.org/dosya/)
- Dreamhost [:page_facing_up:](https://rclone.org/s3/#dreamhost)
- Drime [:page_facing_up:](https://rclone.org/s3/#drime)
- Dropbox [:page_facing_up:](https://rclone.org/dropbox/)
@@ -74,11 +70,8 @@ directories to and from different cloud storage providers.
- Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
- IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
- Impossible Cloud [:page_facing_up:](https://rclone.org/s3/#impossible-cloud)
- Infomaniak Swiss Backup (S3) [:page_facing_up:](https://rclone.org/s3/#infomaniak)
- Infomaniak Swiss Backup (Swift) [:page_facing_up:](https://rclone.org/swift/)
- Intercolo Object Storage [:page_facing_up:](https://rclone.org/s3/#intercolo)
- IONOS Cloud [:page_facing_up:](https://rclone.org/s3/#ionos)
- kDrive [:page_facing_up:](https://rclone.org/kdrive/)
- Koofr [:page_facing_up:](https://rclone.org/koofr/)
- Leviia Object Storage [:page_facing_up:](https://rclone.org/s3/#leviia)
- Liara Object Storage [:page_facing_up:](https://rclone.org/s3/#liara-object-storage)
@@ -94,7 +87,6 @@ directories to and from different cloud storage providers.
- Microsoft Azure Files Storage [:page_facing_up:](https://rclone.org/azurefiles/)
- Microsoft OneDrive [:page_facing_up:](https://rclone.org/onedrive/)
- Minio [:page_facing_up:](https://rclone.org/s3/#minio)
- Neon Object Storage [:page_facing_up:](https://rclone.org/s3/#neon)
- Nextcloud [:page_facing_up:](https://rclone.org/webdav/#nextcloud)
- Blomp Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
- OpenDrive [:page_facing_up:](https://rclone.org/opendrive/)
@@ -118,7 +110,6 @@ directories to and from different cloud storage providers.
- Quatrix [:page_facing_up:](https://rclone.org/quatrix/)
- Rackspace Cloud Files [:page_facing_up:](https://rclone.org/swift/)
- RackCorp Object Storage [:page_facing_up:](https://rclone.org/s3/#RackCorp)
- RelAix.Storage [:page_facing_up:](https://rclone.org/s3/#relaix)
- rsync.net [:page_facing_up:](https://rclone.org/sftp/#rsync-net)
- Scaleway [:page_facing_up:](https://rclone.org/s3/#scaleway)
- Scality (RING / ARTESCA) [:page_facing_up:](https://rclone.org/s3/#scality)
+22 -81
View File
@@ -6,41 +6,27 @@ This file describes how to make the various kinds of releases
- [gh the github cli](https://github.com/cli/cli) for uploading packages
- pandoc for making the html and man pages
- hugo for building the website
- golangci-lint and docker for `make check` (markdownlint runs in docker)
- a gpg key for signing the tag and the checksums
- rclone remotes `beta.rclone.org:`, `downloads.rclone.org:`,
`www.rclone.org:` and `test-rclone-org:`
- the sponsor logos - see [Sponsor logos](#sponsor-logos)
## Making a release
- Enable the [release freeze rule](https://github.com/rclone/rclone/settings/rules/24751862)
- git checkout master # see below for stable branch
- git pull # IMPORTANT
- make fetch-gui-and-commit # bump the embedded GUI if rclone-web has a newer release
- git status - make sure everything is checked in
- Check GitHub actions build for master is Green
- Check the integration tests at <https://integration.rclone.org>
- cat VERSION # check this is the version being released
- make tag # signed tag on HEAD + new changelog entry + docs
- edit docs/content/changelog.md # entries already in point releases are
removed automatically
- make test # see integration test server or run locally
- make tag
- edit docs/content/changelog.md # duplicate logs from point rels removed automatically
releases
- make tidy
- make doc # again to pick up the changelog edits
- git status - to check for new command docs in docs/content/commands - git add them
- make doc
- git status - to check for new man pages - git add them
- git commit -a -v -m "Version v1.XX.0"
- make check # if this fails, fix, git commit --amend and run it again
- make retag # moves the tag onto the Version commit
- git push origin # without --follow-tags so it doesn't push the tag yet
- Wait for the GitHub actions build of master to complete and check it is
green - this is the first time CI sees any commits which were only local,
such as security fixes, so it must pass before the tag is pushed
- if it fails: fix, git commit, make retag, git push origin and wait again
- make check
- make retag
- git push origin # without --follow-tags so it doesn't push the tag if it fails
- git push --follow-tags origin
- \# Wait for the GitHub builds of the tag to complete then...
- \# Don't commit or push anything until after make upload_github as these steps
use the tag on HEAD to find the version
- \# Wait for the GitHub builds to complete then...
- make fetch_binaries
- make tarball
- make vendorball
@@ -49,31 +35,19 @@ This file describes how to make the various kinds of releases
- make upload
- make upload_test_website # check sponsors are correct/unchanged
- make upload_website
- make upload_github # this triggers the docker plugin and winget workflows
- Check the docker image, docker plugin and winget GitHub actions succeeded
- make startdev # works out the next version for master or a stable branch
- git push
- Disable the release freeze rule
- Release any security advisories
- request a CVE for each advisory if not done already
- set the patched version on each advisory and publish it
- when the CVEs are assigned replace `CVE-PENDING` in the changelog on
master and the stable branch then `make upload_website`
- \# announce with forum post, twitter post, and post to announcements list
- make upload_github
- make startdev # make startstable for stable branch
- \# announce with forum post, twitter post, and post to annoucements list
## Update dependencies
Early in the next release cycle update the dependencies.
- Active Pins
- `github.com/coreos/go-systemd/v22` v22.6.0 - v22.7.0 fails to compile on netbsd
- `github.com/hanwen/go-fuse/v2` v2.11.0 - v2.12.0 fails to compile on 32
bit linux, awaiting fix for
[go-fuse/issues/628](https://github.com/hanwen/go-fuse/issues/628)
- `github.com/gdamore/tcell/v2` v2.9.0 - awaiting fix for [tcell/issues/1124](https://github.com/gdamore/tcell/issues/1124)
- See: [#9539](https://github.com/rclone/rclone/issues/9539)
- Review Active Pins in go.mod and remove if possible
- `make updatedirect`
- The update ignores the pins so put them back, eg
`go get github.com/coreos/go-systemd/v22@v22.6.0` then `go mod tidy`
- `make GOTAGS=cmount`
- `make compiletest`
- Fix anything which doesn't compile at this point and commit changes here
@@ -216,53 +190,20 @@ First make the release branch. If this is a second point release then
this will be done already.
- git co -b ${BASE_TAG}-stable ${BASE_TAG}.0
- make startdev # sets the version to ${BASE_TAG}.1
- git push -u origin ${BASE_TAG}-stable
- make startstable
Now
- Enable release freeze rule
- git co master
- git pull --rebase
- make fetch-gui-and-commit # update the GUI on master first
- git push
- git co ${BASE_TAG}-stable
- git pull
- cat VERSION # check this is ${NEW_TAG} - if not run make startdev
- `git cherry-pick -x` any fixes, including the GUI update if there was one
- for security releases consider updating the Go patch version in
`.github/workflows/build.yml` and dependencies with CVE fixes
- fixes may depend on commits which weren't picked so push to CI
and check it is green before tagging
- Do the steps in "Making a release" above, except
- stay on ${BASE_TAG}-stable - don't checkout master or run
make fetch-gui-and-commit
- check the GitHub actions build for ${BASE_TAG}-stable not master
- run the integration tests on ${BASE_TAG}-stable as described below
- the release freeze rule is already enabled - leave it until the end
- `git cherry-pick -x` any fixes
- make startstable
- Do the steps as above
- git co master
- git pull --rebase
- `#` cherry pick the changes to the changelog - check the diff to make sure it
is correct
- git checkout ${BASE_TAG}-stable docs/content/changelog.md
- git diff --cached # check only the ${NEW_TAG} entry has changed
- git commit -v -m "Changelog updates from Version ${NEW_TAG}" docs/content/changelog.md
- git commit -a -v -m "Changelog updates from Version ${NEW_TAG}"
- git push
- Disable release freeze rule
Point releases are only made for the latest release. Once v1.(XX+1).0
is out the ${BASE_TAG}-stable branch is abandoned.
### Running the integration tests on the stable branch
The integration tests at <https://integration.rclone.org> test master.
To test the stable branch push it to the testing server and run the
tests there in `screen` as they take a while.
```console
git push -f rclone@rclone-testing:private-rclone.git ${BASE_TAG}-stable
ssh rclone@rclone-testing
screen
cd ~/integration-test && go run integration-test.go -repo /home/rclone/private-rclone.git -branch v1.XX-stable
```
## Sponsor logos
+1 -1
View File
@@ -1 +1 @@
v1.77.0
v1.75.1
-2
View File
@@ -16,7 +16,6 @@ import (
_ "github.com/rclone/rclone/backend/compress"
_ "github.com/rclone/rclone/backend/crypt"
_ "github.com/rclone/rclone/backend/doi"
_ "github.com/rclone/rclone/backend/dosya"
_ "github.com/rclone/rclone/backend/drime"
_ "github.com/rclone/rclone/backend/drive"
_ "github.com/rclone/rclone/backend/dropbox"
@@ -39,7 +38,6 @@ import (
_ "github.com/rclone/rclone/backend/internetarchive"
_ "github.com/rclone/rclone/backend/internxt"
_ "github.com/rclone/rclone/backend/jottacloud"
_ "github.com/rclone/rclone/backend/kdrive"
_ "github.com/rclone/rclone/backend/koofr"
_ "github.com/rclone/rclone/backend/linkbox"
_ "github.com/rclone/rclone/backend/local"
+9 -14
View File
@@ -36,7 +36,6 @@ import (
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/fspath"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/sanitize"
)
// Register with Fs
@@ -194,12 +193,8 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (outFs fs
}
// Path to the archive
archiveRemote := remote[:len(foundArchive.remote)]
// Remote is archive leaf name. This must understand remote
// names as the archive may be in the root, e.g. "remote:file.zip"
_, foundArchive.remote, err = fspath.Split(archiveRemote)
if err != nil {
return nil, fmt.Errorf("failed to parse archive path %q: %w", archiveRemote, err)
}
// Remote is archive leaf name
foundArchive.remote = path.Base(archiveRemote)
foundArchive.prefix = ""
// Point remote to archive file
remote = archiveRemote
@@ -534,14 +529,14 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
// isDirectChild reports whether remote names an entry directly in dir
// ("" being the root), with no ".." or other components in between.
func isDirectChild(dir, remote string) bool {
if dir != "" {
var ok bool
remote, ok = strings.CutPrefix(remote, dir+"/")
if !ok {
return false
}
if remote == "" || remote == "." || remote == ".." || strings.HasSuffix(remote, "/") {
return false
}
return sanitize.Leaf(remote) == nil
parent := path.Dir(remote)
if parent == "." || parent == "/" {
parent = ""
}
return parent == dir && path.Clean(remote) == remote
}
// NewObject creates a new remote archive file object
-183
View File
@@ -12,19 +12,16 @@ import (
"os/exec"
"path"
"path/filepath"
"sort"
"strconv"
"strings"
"testing"
_ "github.com/rclone/rclone/backend/alias"
"github.com/rclone/rclone/backend/archive/archiver"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/cache"
"github.com/rclone/rclone/fs/filter"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/sync"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/stretchr/testify/assert"
@@ -178,18 +175,6 @@ func testArchive(t *testing.T, archiveName string, archiveFn func(t *testing.T,
checkTree(ctx, "SubDir", t, ":archive:"+zipFile+"/"+subDir, filepath.Join(input, subDir), 0)
// Now check a single file
//
// The filter below applies to both sides, so check first that the
// archive exposes only that file without it
Fsingle, err := cache.Get(ctx, ":archive:"+zipFile+"/"+aFile)
require.NoError(t, err)
entries, err := Fsingle.List(ctx, "")
require.NoError(t, err)
require.Len(t, entries, 1)
_, err = Fsingle.List(ctx, subDir)
assert.Equal(t, fs.ErrorDirNotFound, err)
_, err = Fsingle.NewObject(ctx, subDir)
assert.Equal(t, fs.ErrorObjectNotFound, err)
fiCtx, fi := filter.AddConfig(ctx)
require.NoError(t, fi.AddRule("+ "+aFile))
require.NoError(t, fi.AddRule("- *"))
@@ -324,50 +309,6 @@ func TestArchiveUncleanRoot(t *testing.T) {
}
}
// TestArchiveInRemoteRoot checks that an archive in the root of a
// remote, with no "/" between the remote name and the archive name, can
// be read, as can a path within it.
func TestArchiveInRemoteRoot(t *testing.T) {
fstest.Initialise()
ctx := context.Background()
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
w, err := zw.Create("sub/a.txt")
require.NoError(t, err)
_, err = w.Write([]byte("data"))
require.NoError(t, err)
require.NoError(t, zw.Close())
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "test.zip"), buf.Bytes(), 0600))
t.Chdir(dir)
for _, test := range []struct {
name string
remote string
}{
{name: "plain", remote: ":local:test.zip"},
{name: "slash in remote name", remote: ":alias,remote='" + dir + "':test.zip"},
} {
for _, sub := range []struct {
path string
want string
}{
{path: "", want: "sub"},
{path: "/sub", want: "a.txt"},
} {
t.Run(test.name+sub.path, func(t *testing.T) {
f, err := cache.Get(ctx, ":archive:"+test.remote+sub.path)
require.NoError(t, err)
entries, err := f.List(ctx, "")
require.NoError(t, err)
require.Len(t, entries, 1)
assert.Equal(t, sub.want, entries[0].Remote())
})
}
}
}
// escapingObject is an object whose remote is not where it was asked for.
type escapingObject struct {
fs.Object
@@ -438,127 +379,3 @@ func TestArchiveEscapingArchiver(t *testing.T) {
_, err = f.NewObject(ctx, "test.escaping/file.txt")
assert.ErrorIs(t, err, fs.ErrorObjectNotFound)
}
// TestIsDirectChild checks the guard which decides whether an entry
// returned by an archiver belongs directly in the directory listed.
func TestIsDirectChild(t *testing.T) {
for _, test := range []struct {
dir, remote string
want bool
}{
{"", "a.txt", true},
{"", "/a.txt", false},
{"", "a.txt/", false},
{"", "../a.txt", false},
{"", "sub/a.txt", false},
{"d", "d/a.txt", true},
{"d", "d", false},
{"d", "d/", false},
{"d", "d//a.txt", false},
{"d", "d/../a.txt", false},
{"d", "dd/a.txt", false},
{"d", "a.txt", false},
} {
assert.Equal(t, test.want, isDirectChild(test.dir, test.remote), "dir=%q remote=%q", test.dir, test.remote)
}
}
// A symlink in a squashfs image is skipped unless -l/--links is in use,
// as it is on the other backends. With the flag it is exposed under its
// name with the link suffix and reads back as its target.
//
// Note that this uses mksquashfs as an external binary.
func TestArchiveSquashfsSymlink(t *testing.T) {
fstest.Initialise()
skipIfNoExe(t, "mksquashfs")
ctx := context.Background()
dir := t.TempDir()
input := filepath.Join(dir, "input")
require.NoError(t, os.Mkdir(input, 0777))
require.NoError(t, os.WriteFile(filepath.Join(input, "file.txt"), []byte("hello"), 0600))
require.NoError(t, os.Symlink("file.txt", filepath.Join(input, "link.txt")))
// a regular file which happens to carry the link suffix must stay
// readable under the flag, not be taken for a symlink
require.NoError(t, os.WriteFile(filepath.Join(input, "note"+fs.LinkSuffix), []byte("plain"), 0600))
image := filepath.Join(dir, "test.sqfs")
run(t, "mksquashfs", input, image)
remotes := func(t *testing.T, f fs.Fs) []string {
entries, err := f.List(ctx, "")
require.NoError(t, err)
names := make([]string, 0, len(entries))
for _, entry := range entries {
names = append(names, entry.Remote())
}
sort.Strings(names)
return names
}
read := func(t *testing.T, f fs.Fs, remote string) string {
o, err := f.NewObject(ctx, remote)
require.NoError(t, err)
rc, err := o.Open(ctx)
require.NoError(t, err)
contents, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, o.Size(), int64(len(contents)))
return string(contents)
}
f, err := fs.NewFs(ctx, ":archive:"+image)
require.NoError(t, err)
assert.Equal(t, []string{"file.txt", "note" + fs.LinkSuffix}, remotes(t, f))
ctx, ci := fs.AddConfig(ctx)
ci.Links = true
f, err = fs.NewFs(ctx, ":archive:"+image)
require.NoError(t, err)
assert.Equal(t, []string{"file.txt", "link.txt" + fs.LinkSuffix, "note" + fs.LinkSuffix}, remotes(t, f))
assert.Equal(t, "file.txt", read(t, f, "link.txt"+fs.LinkSuffix))
assert.Equal(t, "plain", read(t, f, "note"+fs.LinkSuffix))
}
// A hostile archive must not be able to write outside the destination.
// A symlink pointing above the root with a member inside it is the
// classic way to try it, and the refusal comes from the local backend,
// so pin it from end to end here rather than trusting it stays.
func TestArchiveZipSymlinkEscape(t *testing.T) {
fstest.Initialise()
ctx := context.Background()
ctx, ci := fs.AddConfig(ctx)
ci.Links = true
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
hdr := &zip.FileHeader{Name: "a"}
hdr.SetMode(os.ModeSymlink | 0777)
w, err := zw.CreateHeader(hdr)
require.NoError(t, err)
_, err = w.Write([]byte("../outside"))
require.NoError(t, err)
w, err = zw.Create("a/pwned.txt")
require.NoError(t, err)
_, err = w.Write([]byte("pwned"))
require.NoError(t, err)
require.NoError(t, zw.Close())
dir := t.TempDir()
zipPath := filepath.Join(dir, "evil.zip")
require.NoError(t, os.WriteFile(zipPath, buf.Bytes(), 0600))
dst := filepath.Join(dir, "dst")
require.NoError(t, os.Mkdir(dst, 0777))
fsrc, err := fs.NewFs(ctx, ":archive:"+zipPath)
require.NoError(t, err)
fdst, err := fs.NewFs(ctx, dst)
require.NoError(t, err)
assert.Error(t, sync.CopyDir(ctx, fdst, fsrc, false))
_, err = os.Stat(filepath.Join(dir, "outside", "pwned.txt"))
assert.True(t, os.IsNotExist(err), "the archive wrote outside the destination")
}
+2 -93
View File
@@ -42,8 +42,6 @@ type Fs struct {
prefix string // position for objects
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
file string // remote of the single file root points at, if any
links bool // read symlinks as .rclonelink files
}
// recoverParsePanic converts a panic raised while parsing a squashfs image
@@ -100,7 +98,6 @@ func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (_ f
c: c,
remote: remote,
root: strings.Trim(root, "/"),
links: fs.GetConfig(ctx).Links,
prefix: prefix,
prefixSlash: prefix + "/",
}
@@ -118,13 +115,10 @@ func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (_ f
_, err := f.newObjectNative(native)
if err == nil {
// If it pointed to a file, find the directory above
// and remember the file so that only it is listed
f.root = path.Dir(f.root)
if f.root == "." || f.root == "/" {
f.root = ""
}
f.file = f.prefixSlash + path.Base(native)
singleObject = true
}
}
}
@@ -156,9 +150,7 @@ func (f *Fs) Name() string {
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
// Include the file so the fs cache can tell this Fs apart from
// the one for the directory containing it
return path.Join(f.root, f.file)
return f.root
}
// Features returns the optional features of this Fs
@@ -222,30 +214,6 @@ func (f *Fs) objectFromFileInfo(nativeDir string, item os.FileInfo) *Object {
}
}
// Convert a symlink entry into an Object holding its target
func (f *Fs) objectFromLink(nativeDir string, item os.FileInfo, target string) *Object {
return &Object{
fs: f,
remote: f.fromNative(nativeDir, item.Name()) + fs.LinkSuffix,
size: int64(len(target)),
modTime: item.ModTime(),
isLink: true,
link: target,
}
}
// readlink returns the target of the symlink the info describes.
//
// A symlink may carry an empty target, so the caller decides what is a
// symlink from the mode rather than from the target being set.
func readlink(info os.FileInfo) (string, error) {
st, ok := info.Sys().(*squashfs.StatT)
if !ok {
return "", fmt.Errorf("squashfs entry %q has no stat information", info.Name())
}
return st.LinkTarget, nil
}
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
@@ -259,10 +227,6 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
defer log.Trace(f, "dir=%q", dir)("entries=%v, err=%v", &entries, &err)
defer recoverParsePanic(&err)
if f.file != "" && dir != "" {
return nil, fs.ErrorDirNotFound
}
nativeDir, err := f.toNative(dir)
if err != nil {
return nil, err
@@ -275,7 +239,6 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
entries = make(fs.DirEntries, 0, len(items))
skipped := 0
skippedLinks := 0
for _, item := range items {
// fs.Debugf(item.Name(), "entry = %#v", item)
// Skip entry names that aren't safe
@@ -298,27 +261,10 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, fmt.Errorf("error reading item %q: %w", item.Name(), err)
}
entry = f.objectFromFileInfo(nativeDir, info)
} else if item.Type()&os.ModeSymlink != 0 {
if !f.links {
skippedLinks++
continue
}
info, err := item.Info()
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", item.Name(), err)
}
target, err := readlink(info)
if err != nil {
return nil, err
}
entry = f.objectFromLink(nativeDir, info, target)
} else {
fs.Debugf(item.Name(), "FIXME Not regular file - skipping")
continue
}
if f.file != "" && entry.Remote() != f.file {
continue
}
entries = append(entries, entry)
}
@@ -327,9 +273,6 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
if skipped > 0 {
fs.Logf(f, "Skipped %d squashfs entries in %q whose names escape the archive", skipped, dir)
}
if skippedLinks > 0 {
fs.Logf(f, "Skipped %d symlinks in %q - use -l/--links to read them", skippedLinks, dir)
}
// fs.Debugf(f, "dir=%q, entries=%v", dir, entries)
return entries, nil
@@ -351,17 +294,7 @@ func (f *Fs) newObjectNative(nativePath string) (o fs.Object, err error) {
}
for _, fi := range fis {
// Name the entry as List does, so that the two agree on
// which names exist
isLink := fi.Type()&os.ModeSymlink != 0
if isLink && !f.links {
continue
}
name := fi.Name()
if isLink {
name += fs.LinkSuffix
}
if name == leaf {
if fi.Name() == leaf {
if fi.IsDir() {
return nil, fs.ErrorNotAFile
}
@@ -369,14 +302,6 @@ func (f *Fs) newObjectNative(nativePath string) (o fs.Object, err error) {
if err != nil {
return nil, fmt.Errorf("error reading item %q: %w", fi.Name(), err)
}
if isLink {
target, err := readlink(info)
if err != nil {
return nil, err
}
o = f.objectFromLink(dir, info, target)
break
}
o = f.objectFromFileInfo(dir, info)
break
}
@@ -391,9 +316,6 @@ func (f *Fs) newObjectNative(nativePath string) (o fs.Object, err error) {
func (f *Fs) NewObject(ctx context.Context, remote string) (o fs.Object, err error) {
defer log.Trace(f, "remote=%q", remote)("obj=%v, err=%v", &o, &err)
if f.file != "" && remote != f.file {
return nil, fs.ErrorObjectNotFound
}
nativePath, err := f.toNative(remote)
if err != nil {
return nil, err
@@ -455,8 +377,6 @@ type Object struct {
remote string
size int64
modTime time.Time
isLink bool // whether this is a symlink
link string // target of the symlink
}
// Fs returns read only access to the Fs that this object is part of
@@ -528,17 +448,6 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.Read
}
}
if o.isLink {
// Clamp offset into range to avoid panic
if offset < 0 {
offset = 0
}
if offset > int64(len(o.link)) {
offset = int64(len(o.link))
}
return readers.NewLimitedReadCloser(io.NopCloser(strings.NewReader(o.link[offset:])), limit), nil
}
remote, err := o.fs.toNative(o.remote)
if err != nil {
return nil, err
+10 -27
View File
@@ -9,7 +9,6 @@ import (
"io"
"os"
"path"
"slices"
"strings"
"time"
@@ -43,7 +42,6 @@ type Fs struct {
prefix string // position for objects
prefixSlash string // position for objects with a slash on
root string // position to read from within the archive
links bool // read symlinks as .rclonelink files
dt dirtree.DirTree // read from zipfile
}
@@ -68,7 +66,6 @@ func New(ctx context.Context, wrappedFs fs.Fs, remote, prefix, root string) (fs.
node: node,
remote: remote,
root: root,
links: fs.GetConfig(ctx).Links,
prefix: prefix,
prefixSlash: prefix + "/",
}
@@ -140,7 +137,6 @@ func (f *Fs) readZip() (singleObject bool, err error) {
}
dt := dirtree.New()
skipped := 0
skippedLinks := 0
for _, file := range zr.File {
// Skip entries whose name escapes the archive's own namespace
remote, err := sanitize.Path(file.Name)
@@ -148,22 +144,6 @@ func (f *Fs) readZip() (singleObject bool, err error) {
skipped++
continue
}
isDir := strings.HasSuffix(file.Name, "/")
// A file entry whose last component is "", "." or ".." names
// a directory rather than a file, so skip
if !isDir && sanitize.Leaf(path.Base(file.Name)) != nil {
skipped++
continue
}
if !isDir && file.Mode()&os.ModeSymlink != 0 {
if !f.links {
skippedLinks++
continue
}
// The member holds the link target, so it reads back
// like a .rclonelink file
remote += fs.LinkSuffix
}
remote = path.Join(f.prefix, remote)
if f.root != "" {
// Ignore all files outside the root, requiring a path
@@ -178,12 +158,19 @@ func (f *Fs) readZip() (singleObject bool, err error) {
remote = strings.TrimPrefix(remote, f.root+"/")
}
}
if isDir {
if strings.HasSuffix(file.Name, "/") {
dir := fs.NewDir(remote, file.Modified)
dt.AddDir(dir)
} else {
if remote == "" {
// A file at the root is the archive member f.root points at
// A file at the root itself can only be the
// archive member f.root points at - with no root
// it is a crafted name for the archive's own
// directory, which can't be a file
if f.root == "" {
skipped++
continue
}
remote = path.Base(f.root)
singleObject = true
dt = dirtree.New()
@@ -203,9 +190,6 @@ func (f *Fs) readZip() (singleObject bool, err error) {
if skipped > 0 {
fs.Logf(f, "Skipped %d zip entries which escape the archive", skipped)
}
if skippedLinks > 0 {
fs.Logf(f, "Skipped %d symlinks - use -l/--links to read them", skippedLinks)
}
dt.CheckParents("")
dt.Sort()
f.dt = dt
@@ -233,8 +217,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, fs.ErrorDirNotFound
}
fs.Debugf(f, "dir=%q, entries=%v", dir, entries)
// Return a copy as callers may filter the listing in place
return slices.Clone(entries), nil
return entries, nil
}
// NewObject finds the Object at remote.
+6 -103
View File
@@ -4,8 +4,6 @@ import (
"archive/zip"
"bytes"
"context"
"fmt"
"io"
"os"
"path/filepath"
"sort"
@@ -37,26 +35,6 @@ func writeZip(t *testing.T, dir, name string, names ...string) string {
return name
}
// writeZipSymlink builds a zip holding file and a symlink entry link
// pointing at target, and returns its leaf name.
func writeZipSymlink(t *testing.T, dir, name, file, link, target string) string {
var buf bytes.Buffer
zw := zip.NewWriter(&buf)
w, err := zw.Create(file)
require.NoError(t, err)
_, err = w.Write([]byte("data for " + file))
require.NoError(t, err)
hdr := &zip.FileHeader{Name: link}
hdr.SetMode(os.ModeSymlink | 0777)
w, err = zw.CreateHeader(hdr)
require.NoError(t, err)
_, err = w.Write([]byte(target))
require.NoError(t, err)
require.NoError(t, zw.Close())
require.NoError(t, os.WriteFile(filepath.Join(dir, name), buf.Bytes(), 0600))
return name
}
// allRemotes returns every Object remote in the mounted archive's dirtree.
func allRemotes(t *testing.T, f fs.Fs) []string {
zf, ok := f.(*Fs)
@@ -127,12 +105,10 @@ func TestReadZipRootBoundary(t *testing.T) {
assert.Equal(t, []string{"a.txt"}, remotes)
}
// A file entry whose name refers to a directory (".", "/", "./", "" or
// "sub/.") must be skipped, not turn the archive or the directory
// mounted as its root into a single file which hides every other
// entry, or become an object named for the archive itself when the
// archive is found by listing its directory.
func TestReadZipDirNamedFileEntry(t *testing.T) {
// A file entry whose name refers to the archive's own root (".", "/",
// "./" or "") must be skipped, not turn the whole archive into a single
// file which hides every other entry.
func TestReadZipRootNamedEntry(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
name := writeZip(t, dir, "dot.zip",
@@ -141,87 +117,14 @@ func TestReadZipDirNamedFileEntry(t *testing.T) {
"./",
"",
"good.txt",
"sub/",
"sub/.",
"sub/good.txt",
)
localFs, err := cache.Get(ctx, dir)
require.NoError(t, err)
for _, test := range []struct {
prefix, root string
want []string
}{
{"", "", []string{"good.txt", "sub/good.txt"}},
{"sub/dot.zip", "", []string{"sub/dot.zip/good.txt", "sub/dot.zip/sub/good.txt"}},
{"", "sub", []string{"good.txt"}},
} {
t.Run(fmt.Sprintf("prefix=%q,root=%q", test.prefix, test.root), func(t *testing.T) {
f, err := New(ctx, localFs, name, test.prefix, test.root)
require.NoError(t, err)
assert.Equal(t, test.want, allRemotes(t, f))
})
}
}
// Listings are served from a cache which must survive callers
// filtering the returned slice in place, as fs/list does.
func TestListCacheNotAliased(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
name := writeZip(t, dir, "test.zip", "a.txt", "b.txt", "c.txt")
localFs, err := cache.Get(ctx, dir)
require.NoError(t, err)
f, err := New(ctx, localFs, name, "", "")
require.NoError(t, err)
entries, err := f.List(ctx, "")
require.NoError(t, err)
require.Len(t, entries, 3)
// Compact in place, dropping the first entry
copy(entries, entries[1:])
entries, err = f.List(ctx, "")
require.NoError(t, err)
var remotes []string
for _, entry := range entries {
remotes = append(remotes, entry.Remote())
}
assert.Equal(t, []string{"a.txt", "b.txt", "c.txt"}, remotes)
}
// A symlink stored in the archive is skipped unless -l/--links is in
// use, as it is on the other backends. With the flag it is exposed
// under its name with the link suffix and reads back as its target.
func TestReadZipSymlink(t *testing.T) {
ctx := context.Background()
dir := t.TempDir()
name := writeZipSymlink(t, dir, "links.zip", "file.txt", "link.txt", "file.txt")
localFs, err := cache.Get(ctx, dir)
require.NoError(t, err)
f, err := New(ctx, localFs, name, "", "")
require.NoError(t, err)
assert.Equal(t, []string{"file.txt"}, allRemotes(t, f))
ctx, ci := fs.AddConfig(ctx)
ci.Links = true
f, err = New(ctx, localFs, name, "", "")
require.NoError(t, err)
assert.Equal(t, []string{"file.txt", "link.txt" + fs.LinkSuffix}, allRemotes(t, f))
o, err := f.NewObject(ctx, "link.txt"+fs.LinkSuffix)
require.NoError(t, err)
assert.Equal(t, int64(len("file.txt")), o.Size())
rc, err := o.Open(ctx)
require.NoError(t, err)
target, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, "file.txt", string(target))
remotes := allRemotes(t, f)
assert.Equal(t, []string{"good.txt"}, remotes)
}
+22 -15
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
// Parallel listing support for azureblob, built on the Apache Arrow
// listing format.
@@ -18,20 +18,24 @@ import (
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/rclone/rclone/backend/azureblob/arrowlist"
"github.com/rclone/rclone/fs"
"golang.org/x/sync/errgroup"
)
// arrowContentType is the Content-Type of an Apache Arrow IPC stream listing
// response, used to tell whether the service honoured an Arrow request.
const arrowContentType = "application/vnd.apache.arrow.stream"
// listArrowParallel lists (containerName, directory) by splitting the blob name
// keyspace into shards and listing them concurrently. opts must already request
// the Arrow format; it is cloned per shard with StartFrom/EndBefore set. It
// returns the number of raw items seen. directory/prefix are already normalised
// by the caller.
func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, prefix string, addContainer bool, opts *container.ListBlobsHierarchyOptions, delimiter string, fn listFn) (int, error) {
func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, prefix string, addContainer bool, opts *arrowlist.ListBlobsHierarchyOptions, delimiter string, fn listFn) (int, error) {
// Parallel listing needs the arrowlist client (server-side endBefore is
// only honoured on the Arrow path) - without it fall back to sequential.
if _, err := f.arrowCntSVC(containerName); err != nil {
fs.Debugf(f, "Not using parallel Arrow listing: %v", err)
return f.listBlobsPager(ctx, containerName, directory, prefix, addContainer, opts, delimiter, fn)
}
// Split the keyspace by a single-case character ladder (see
// arrowLadderBoundaries for why single-case matters).
points := arrowLadderBoundaries(directory, arrowShardTarget(f.opt.ListParallelism))
@@ -62,7 +66,7 @@ func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, pr
// Schedule every shard; SetLimit bounds concurrency, so more shards than
// workers gives free load balancing as workers pick up the next shard.
var total atomic.Int64
var total int64
g, gCtx := errgroup.WithContext(ctx)
g.SetLimit(f.opt.ListParallelism)
for _, sh := range shards {
@@ -85,12 +89,12 @@ func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, pr
shardOpts.EndBefore = nil
}
n, err := f.listBlobsPager(gCtx, containerName, directory, prefix, addContainer, &shardOpts, delimiter, safeFn)
total.Add(int64(n))
atomic.AddInt64(&total, int64(n))
return err
})
}
err := g.Wait()
found := int(total.Load())
found := int(atomic.LoadInt64(&total))
if isEndBeforeUnsupported(err) {
// The account can't honour the shards' endBefore bounds, so parallel
// listing can't be used on it.
@@ -111,13 +115,16 @@ func (f *Fs) listArrowParallel(ctx context.Context, containerName, directory, pr
// isEndBeforeUnsupported reports whether err means the service can't honour
// the endBefore listing bound, so sharded parallel listing must not be used.
//
// The service rejects the endBefore parameter with 400
// OperationNotSupportedWithFeatureMissing "The requested operation is not
// allowed as EndBefore parameter support is missing on this account" -
// endBefore is only supported on the Arrow listing path of accounts with
// Blob Listing with Apache Arrow enabled. (The SDK pager would silently
// accept an XML page with endBefore set, but the service never sends one.)
// This happens two ways: the arrowlist pager's ErrEndBeforeXMLFallback (the
// server answered XML with endBefore set), or the service rejecting the
// endBefore parameter with 400 OperationNotSupportedWithFeatureMissing "The
// requested operation is not allowed as EndBefore parameter support is
// missing on this account" - endBefore is only supported on the Arrow
// listing path of accounts with Blob Listing with Apache Arrow enabled.
func isEndBeforeUnsupported(err error) bool {
if errors.Is(err, arrowlist.ErrEndBeforeXMLFallback) {
return true
}
var respErr *azcore.ResponseError
return errors.As(err, &respErr) && respErr.ErrorCode == "OperationNotSupportedWithFeatureMissing"
}
+4 -2
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
package azureblob
@@ -10,6 +10,7 @@ import (
"testing"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/rclone/rclone/backend/azureblob/arrowlist"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
@@ -211,8 +212,9 @@ func TestArrowLadderBoundaries(t *testing.T) {
}
func TestIsEndBeforeUnsupported(t *testing.T) {
assert.True(t, isEndBeforeUnsupported(arrowlist.ErrEndBeforeXMLFallback))
assert.True(t, isEndBeforeUnsupported(fmt.Errorf("wrapped: %w", arrowlist.ErrEndBeforeXMLFallback)))
assert.True(t, isEndBeforeUnsupported(&azcore.ResponseError{ErrorCode: "OperationNotSupportedWithFeatureMissing"}))
assert.True(t, isEndBeforeUnsupported(fmt.Errorf("wrapped: %w", &azcore.ResponseError{ErrorCode: "OperationNotSupportedWithFeatureMissing"})))
assert.False(t, isEndBeforeUnsupported(&azcore.ResponseError{ErrorCode: "ContainerNotFound"}))
assert.False(t, isEndBeforeUnsupported(errors.New("some other error")))
assert.False(t, isEndBeforeUnsupported(nil))
+421
View File
@@ -0,0 +1,421 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/arrow/arrow.go in the Azure SDK for Go
// github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at commit
// c6fa341ca22b (branch feature/storage/bifrost), retyped onto the SDK's
// public type aliases so it compiles against the released azblob module.
package arrowlist
import (
"encoding/base64"
"fmt"
"io"
"net/http"
"strconv"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
"github.com/apache/arrow-go/v18/arrow"
"github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/memory"
)
const (
// ArrowContentType is the Content-Type for Apache Arrow IPC stream responses.
ArrowContentType = "application/vnd.apache.arrow.stream"
// ArrowAcceptHeader is the Accept header value to request Arrow format.
ArrowAcceptHeader = ArrowContentType
// resourceTypeBlobPrefix is the ResourceType value that identifies a virtual directory prefix
// in hierarchy listing responses. In XML, prefixes are separate <BlobPrefix> elements;
// in Arrow, all rows share the same schema and are distinguished by this field value.
resourceTypeBlobPrefix = "blobprefix"
)
// HandleFlatListResponse parses an Arrow IPC stream response into a ContainerClientListBlobFlatSegmentResponse.
func HandleFlatListResponse(resp *http.Response) (container.ListBlobsFlatResponse, error) {
result := container.ListBlobsFlatResponse{}
// Extract response headers
extractResponseHeaders(resp, &result.ClientRequestID, &result.ContentType, &result.Date, &result.RequestID, &result.Version)
// Parse Arrow IPC stream
items, nextMarker, err := parseArrowStream(resp.Body)
if err != nil {
return result, fmt.Errorf("failed to parse Arrow IPC response: %w", err)
}
result.ListBlobsFlatSegmentResponse = container.ListBlobsFlatSegmentResponse{
Segment: &container.BlobFlatListSegment{
BlobItems: items,
},
NextMarker: nextMarker,
}
return result, nil
}
// HandleHierarchyListResponse parses an Arrow IPC stream response into a ContainerClientListBlobHierarchySegmentResponse.
func HandleHierarchyListResponse(resp *http.Response) (container.ListBlobsHierarchyResponse, error) {
result := container.ListBlobsHierarchyResponse{}
extractResponseHeaders(resp, &result.ClientRequestID, &result.ContentType, &result.Date, &result.RequestID, &result.Version)
items, nextMarker, err := parseArrowStream(resp.Body)
if err != nil {
return result, fmt.Errorf("failed to parse Arrow IPC response: %w", err)
}
// Separate blob items from blob prefixes based on ResourceType field.
var blobItems []*container.BlobItem
var blobPrefixes []*container.BlobPrefix
for _, item := range items {
if item.Properties != nil && item.Properties.ResourceType != nil && *item.Properties.ResourceType == resourceTypeBlobPrefix {
blobPrefixes = append(blobPrefixes, &container.BlobPrefix{
Name: item.Name,
Properties: item.Properties,
})
} else {
blobItems = append(blobItems, item)
}
}
result.ListBlobsHierarchySegmentResponse = container.ListBlobsHierarchySegmentResponse{
Segment: &container.BlobHierarchyListSegment{
BlobItems: blobItems,
BlobPrefixes: blobPrefixes,
},
NextMarker: nextMarker,
}
return result, nil
}
func extractResponseHeaders(resp *http.Response, clientRequestID, contentType **string, date **time.Time, requestID, version **string) {
if val := resp.Header.Get("x-ms-client-request-id"); val != "" {
*clientRequestID = &val
}
if val := resp.Header.Get("Content-Type"); val != "" {
*contentType = &val
}
if val := resp.Header.Get("Date"); val != "" {
if t, err := time.Parse(time.RFC1123, val); err == nil {
*date = &t
}
}
if val := resp.Header.Get("x-ms-request-id"); val != "" {
*requestID = &val
}
if val := resp.Header.Get("x-ms-version"); val != "" {
*version = &val
}
}
// parseArrowStream reads an Arrow IPC stream and converts it to BlobItems.
func parseArrowStream(body io.Reader) ([]*container.BlobItem, *string, error) {
reader, err := ipc.NewReader(body, ipc.WithAllocator(memory.DefaultAllocator))
if err != nil {
return nil, nil, fmt.Errorf("failed to create Arrow IPC reader: %w", err)
}
defer reader.Release()
// Extract NextMarker from schema metadata
var nextMarker *string
var numRecords int
if md := reader.Schema().Metadata(); md.Len() > 0 {
if idx := md.FindKey("NextMarker"); idx >= 0 {
val := md.Values()[idx]
if val != "" {
nextMarker = &val
}
}
if idx := md.FindKey("NumberOfRecords"); idx >= 0 {
if n, err := strconv.Atoi(md.Values()[idx]); err == nil {
numRecords = n
}
}
}
// Build column index for the schema
schema := reader.Schema()
colIndex := buildColumnIndex(schema)
// Pre-allocate if we know the count
items := make([]*container.BlobItem, 0, numRecords)
// Iterate over record batches
for reader.Next() {
rec := reader.RecordBatch()
rows := int(rec.NumRows())
for row := 0; row < rows; row++ {
item := &container.BlobItem{
Properties: &container.BlobProperties{},
}
populateBlobItem(item, rec, row, colIndex)
items = append(items, item)
}
}
if err := reader.Err(); err != nil {
return nil, nil, fmt.Errorf("error reading Arrow record batches: %w", err)
}
return items, nextMarker, nil
}
// columnIndex maps column names to their indices in the record batch for O(1) lookup.
type columnIndex map[string]int
func buildColumnIndex(schema *arrow.Schema) columnIndex {
idx := make(columnIndex, len(schema.Fields()))
for i, f := range schema.Fields() {
idx[f.Name] = i
}
return idx
}
// populateBlobItem fills a BlobItem from a single row across all columns.
func populateBlobItem(item *container.BlobItem, rec arrow.RecordBatch, row int, colIdx columnIndex) {
// Top-level BlobItem fields
item.Name = getStringField(rec, row, colIdx, "Name")
item.Deleted = getBoolField(rec, row, colIdx, "Deleted")
item.Snapshot = getStringField(rec, row, colIdx, "Snapshot")
item.VersionID = getStringField(rec, row, colIdx, "VersionId")
item.IsCurrentVersion = getBoolField(rec, row, colIdx, "IsCurrentVersion")
item.HasVersionsOnly = getBoolField(rec, row, colIdx, "HasVersionsOnly")
// Metadata maps
item.Metadata = getMapField(rec, row, colIdx, "Metadata")
item.OrMetadata = getMapField(rec, row, colIdx, "OrMetadata")
// Tags
if tags := getMapField(rec, row, colIdx, "Tags"); tags != nil {
var blobTags []*container.BlobTag
for k, v := range tags {
key := k
blobTags = append(blobTags, &container.BlobTag{Key: &key, Value: v})
}
item.BlobTags = &container.BlobTags{BlobTagSet: blobTags}
}
// BlobProperties fields
p := item.Properties
p.CreationTime = getTimestampField(rec, row, colIdx, "Creation-Time")
p.LastModified = getTimestampField(rec, row, colIdx, "Last-Modified")
p.ETag = getETagField(rec, row, colIdx, "Etag")
p.ContentLength = getInt64Field(rec, row, colIdx, "Content-Length")
p.ContentType = getStringField(rec, row, colIdx, "Content-Type")
p.ContentEncoding = getStringField(rec, row, colIdx, "Content-Encoding")
p.ContentLanguage = getStringField(rec, row, colIdx, "Content-Language")
p.ContentDisposition = getStringField(rec, row, colIdx, "Content-Disposition")
p.CacheControl = getStringField(rec, row, colIdx, "Cache-Control")
p.ContentMD5 = getBase64BytesField(rec, row, colIdx, "Content-MD5")
p.BlobType = getEnumField[container.BlobType](rec, row, colIdx, "BlobType")
p.AccessTier = getEnumField[container.AccessTier](rec, row, colIdx, "AccessTier")
p.AccessTierInferred = getBoolField(rec, row, colIdx, "AccessTierInferred")
p.AccessTierChangeTime = getTimestampField(rec, row, colIdx, "AccessTierChangeTime")
p.LeaseState = getEnumField[lease.StateType](rec, row, colIdx, "LeaseState")
p.LeaseStatus = getEnumField[lease.StatusType](rec, row, colIdx, "LeaseStatus")
p.LeaseDuration = getEnumField[lease.DurationType](rec, row, colIdx, "LeaseDuration")
p.ServerEncrypted = getBoolField(rec, row, colIdx, "ServerEncrypted")
p.CustomerProvidedKeySHA256 = getStringField(rec, row, colIdx, "CustomerProvidedKeySha256")
p.EncryptionScope = getStringField(rec, row, colIdx, "EncryptionScope")
p.IncrementalCopy = getBoolField(rec, row, colIdx, "IncrementalCopy")
p.IsSealed = getBoolField(rec, row, colIdx, "Sealed")
p.ArchiveStatus = getEnumField[container.ArchiveStatus](rec, row, colIdx, "ArchiveStatus")
p.RehydratePriority = getEnumField[blob.RehydratePriority](rec, row, colIdx, "RehydratePriority")
p.CopyID = getStringField(rec, row, colIdx, "CopyId")
p.CopyStatus = getEnumField[blob.CopyStatusType](rec, row, colIdx, "CopyStatus")
p.CopySource = getStringField(rec, row, colIdx, "CopySource")
p.CopyProgress = getStringField(rec, row, colIdx, "CopyProgress")
p.CopyCompletionTime = getTimestampField(rec, row, colIdx, "CopyCompletionTime")
p.CopyStatusDescription = getStringField(rec, row, colIdx, "CopyStatusDescription")
p.DestinationSnapshot = getStringField(rec, row, colIdx, "CopyDestinationSnapshot")
p.ImmutabilityPolicyExpiresOn = getTimestampField(rec, row, colIdx, "ImmutabilityPolicyUntilDate")
p.ImmutabilityPolicyMode = getEnumField[container.ImmutabilityPolicyMode](rec, row, colIdx, "ImmutabilityPolicyMode")
p.LegalHold = getBoolField(rec, row, colIdx, "LegalHold")
p.DeletedTime = getTimestampField(rec, row, colIdx, "DeletedTime")
p.RemainingRetentionDays = getInt32Field(rec, row, colIdx, "RemainingRetentionDays")
p.LastAccessedOn = getTimestampField(rec, row, colIdx, "LastAccessTime")
p.TagCount = getInt32Field(rec, row, colIdx, "TagCount")
p.BlobSequenceNumber = getInt64Field(rec, row, colIdx, "x-ms-blob-sequence-number")
p.ResourceType = getStringField(rec, row, colIdx, "ResourceType")
// Content-CRC64 is a string field in Arrow but not directly mapped to BlobProperties.
// The field exists in the Arrow schema but has no corresponding field in the Go SDK BlobProperties.
// It is silently ignored for forward compatibility.
}
// Field accessor helpers that safely handle missing columns and null values.
func getStringField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *string {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.String); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getBoolField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *bool {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Boolean); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getTimestampField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *time.Time {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Timestamp); ok {
val := arr.Value(row)
// Arrow timestamps are stored as int64 with a unit; convert to time.Time
dt := val.ToTime(arr.DataType().(*arrow.TimestampType).Unit)
return &dt
}
return nil
}
func getInt64Field(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *int64 {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Uint64); ok {
val := int64(arr.Value(row))
return &val
}
if arr, ok := col.(*array.Int64); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getInt32Field(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *int32 {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
if arr, ok := col.(*array.Uint64); ok {
val := int32(arr.Value(row))
return &val
}
if arr, ok := col.(*array.Int32); ok {
val := arr.Value(row)
return &val
}
return nil
}
func getETagField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *azcore.ETag {
s := getStringField(rec, row, colIdx, name)
if s == nil {
return nil
}
etag := azcore.ETag(*s)
return &etag
}
func getBase64BytesField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) []byte {
s := getStringField(rec, row, colIdx, name)
if s == nil {
return nil
}
decoded, err := base64.StdEncoding.DecodeString(*s)
if err != nil {
return nil
}
return decoded
}
// getEnumField returns a pointer to a string-typed enum value.
type stringEnum interface {
~string
}
func getEnumField[T stringEnum](rec arrow.RecordBatch, row int, colIdx columnIndex, name string) *T {
s := getStringField(rec, row, colIdx, name)
if s == nil {
return nil
}
val := T(*s)
return &val
}
func getMapField(rec arrow.RecordBatch, row int, colIdx columnIndex, name string) map[string]*string {
idx, ok := colIdx[name]
if !ok {
return nil
}
col := rec.Column(idx)
if col.IsNull(row) {
return nil
}
mapArr, ok := col.(*array.Map)
if !ok {
return nil
}
// Get the offsets for this row's map entries
start := int(mapArr.Offsets()[row])
end := int(mapArr.Offsets()[row+1])
if start == end {
return nil
}
keys := mapArr.Keys().(*array.String)
values := mapArr.Items().(*array.String)
result := make(map[string]*string, end-start)
for i := start; i < end; i++ {
k := keys.Value(i)
if values.IsNull(i) {
result[k] = nil
} else {
v := values.Value(i)
result[k] = &v
}
}
return result
}
+374
View File
@@ -0,0 +1,374 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/arrow/arrow_test.go in the Azure SDK for Go
// github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at commit
// c6fa341ca22b (branch feature/storage/bifrost).
package arrowlist
import (
"bytes"
"io"
"net/http"
"testing"
"github.com/apache/arrow-go/v18/arrow"
arrowArray "github.com/apache/arrow-go/v18/arrow/array"
"github.com/apache/arrow-go/v18/arrow/ipc"
"github.com/apache/arrow-go/v18/arrow/memory"
"github.com/stretchr/testify/require"
)
// coreSchema returns a basic Arrow schema for testing with common blob fields.
func coreSchema(md *arrow.Metadata) *arrow.Schema {
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Creation-Time", Type: &arrow.TimestampType{Unit: arrow.Microsecond}, Nullable: true},
{Name: "Last-Modified", Type: &arrow.TimestampType{Unit: arrow.Microsecond}, Nullable: true},
{Name: "BlobType", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "Etag", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "Content-Length", Type: arrow.PrimitiveTypes.Uint64, Nullable: true},
{Name: "Content-Type", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "ServerEncrypted", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
{Name: "AccessTier", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "LeaseState", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "LeaseStatus", Type: arrow.BinaryTypes.String, Nullable: true},
}
return arrow.NewSchema(fields, md)
}
// buildArrowStream constructs a synthetic Arrow IPC stream from a schema and a populate function.
func buildArrowStream(t *testing.T, schema *arrow.Schema, populate func(builder *arrowArray.RecordBuilder)) []byte {
t.Helper()
alloc := memory.DefaultAllocator
var buf bytes.Buffer
w := ipc.NewWriter(&buf, ipc.WithSchema(schema), ipc.WithAllocator(alloc))
builder := arrowArray.NewRecordBuilder(alloc, schema)
defer builder.Release()
populate(builder)
rec := builder.NewRecordBatch()
defer rec.Release()
require.NoError(t, w.Write(rec))
require.NoError(t, w.Close())
return buf.Bytes()
}
func makeHTTPResponse(data []byte, headers map[string]string) *http.Response {
h := http.Header{}
h.Set("Content-Type", ArrowContentType)
for k, v := range headers {
h.Set(k, v)
}
return &http.Response{
StatusCode: http.StatusOK,
Header: h,
Body: io.NopCloser(bytes.NewReader(data)),
}
}
func TestHandleFlatListResponse_BasicParsing(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{"marker123", "2"})
schema := coreSchema(&md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob1.txt")
b.Field(1).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000000000000))
b.Field(2).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000000000000))
b.Field(3).(*arrowArray.StringBuilder).Append("BlockBlob")
b.Field(4).(*arrowArray.StringBuilder).Append("0x1234")
b.Field(5).(*arrowArray.Uint64Builder).Append(1024)
b.Field(6).(*arrowArray.StringBuilder).Append("application/octet-stream")
b.Field(7).(*arrowArray.BooleanBuilder).Append(true)
b.Field(8).(*arrowArray.StringBuilder).Append("Hot")
b.Field(9).(*arrowArray.StringBuilder).Append("available")
b.Field(10).(*arrowArray.StringBuilder).Append("unlocked")
b.Field(0).(*arrowArray.StringBuilder).Append("blob2.txt")
b.Field(1).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000001000000))
b.Field(2).(*arrowArray.TimestampBuilder).Append(arrow.Timestamp(1700000001000000))
b.Field(3).(*arrowArray.StringBuilder).Append("BlockBlob")
b.Field(4).(*arrowArray.StringBuilder).Append("0x5678")
b.Field(5).(*arrowArray.Uint64Builder).Append(2048)
b.Field(6).(*arrowArray.StringBuilder).Append("text/plain")
b.Field(7).(*arrowArray.BooleanBuilder).Append(false)
b.Field(8).(*arrowArray.StringBuilder).Append("Cool")
b.Field(9).(*arrowArray.StringBuilder).Append("available")
b.Field(10).(*arrowArray.StringBuilder).Append("unlocked")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.NotNil(t, result.NextMarker)
require.Equal(t, "marker123", *result.NextMarker)
require.Len(t, result.Segment.BlobItems, 2)
item := result.Segment.BlobItems[0]
require.Equal(t, "blob1.txt", *item.Name)
require.Equal(t, "BlockBlob", string(*item.Properties.BlobType))
require.Equal(t, int64(1024), *item.Properties.ContentLength)
require.Equal(t, "Hot", string(*item.Properties.AccessTier))
}
func TestHandleFlatListResponse_EmptyRecordBatch(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{"", "0"})
schema := coreSchema(&md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
// No rows added
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Nil(t, result.NextMarker)
require.Empty(t, result.Segment.BlobItems)
}
func TestHandleFlatListResponse_NullFields(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
schema := coreSchema(&md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
b.Field(1).(*arrowArray.TimestampBuilder).AppendNull()
b.Field(2).(*arrowArray.TimestampBuilder).AppendNull()
b.Field(3).(*arrowArray.StringBuilder).AppendNull()
b.Field(4).(*arrowArray.StringBuilder).AppendNull()
b.Field(5).(*arrowArray.Uint64Builder).AppendNull()
b.Field(6).(*arrowArray.StringBuilder).AppendNull()
b.Field(7).(*arrowArray.BooleanBuilder).AppendNull()
b.Field(8).(*arrowArray.StringBuilder).AppendNull()
b.Field(9).(*arrowArray.StringBuilder).AppendNull()
b.Field(10).(*arrowArray.StringBuilder).AppendNull()
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
item := result.Segment.BlobItems[0]
require.Equal(t, "blob.txt", *item.Name)
require.Nil(t, item.Properties.CreationTime)
require.Nil(t, item.Properties.BlobType)
require.Nil(t, item.Properties.ETag)
require.Nil(t, item.Properties.ContentLength)
require.Nil(t, item.Properties.ContentType)
require.Nil(t, item.Properties.ServerEncrypted)
}
func TestHandleFlatListResponse_UnknownColumnsIgnored(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "FutureField", Type: arrow.BinaryTypes.String, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
b.Field(1).(*arrowArray.StringBuilder).Append("some-future-value")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
require.Equal(t, "blob.txt", *result.Segment.BlobItems[0].Name)
}
func TestHandleFlatListResponse_MultiRecordBatch(t *testing.T) {
alloc := memory.DefaultAllocator
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{"", "3"})
schema := coreSchema(&md)
var buf bytes.Buffer
w := ipc.NewWriter(&buf, ipc.WithSchema(schema), ipc.WithAllocator(alloc))
// First batch: 2 rows
builder := arrowArray.NewRecordBuilder(alloc, schema)
for i := 0; i < 2; i++ {
builder.Field(0).(*arrowArray.StringBuilder).Append("batch1_blob")
for j := 1; j < 11; j++ {
builder.Field(j).AppendNull()
}
}
rec := builder.NewRecordBatch()
require.NoError(t, w.Write(rec))
rec.Release()
builder.Release()
// Second batch: 1 row
builder = arrowArray.NewRecordBuilder(alloc, schema)
builder.Field(0).(*arrowArray.StringBuilder).Append("batch2_blob")
for j := 1; j < 11; j++ {
builder.Field(j).AppendNull()
}
rec = builder.NewRecordBatch()
require.NoError(t, w.Write(rec))
rec.Release()
builder.Release()
require.NoError(t, w.Close())
resp := makeHTTPResponse(buf.Bytes(), nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 3)
}
func TestHandleHierarchyListResponse_SeparatesBlobsAndPrefixes(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "ResourceType", Type: arrow.BinaryTypes.String, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
// A regular blob
b.Field(0).(*arrowArray.StringBuilder).Append("folder/file.txt")
b.Field(1).(*arrowArray.StringBuilder).AppendNull()
// A blob prefix (virtual directory)
b.Field(0).(*arrowArray.StringBuilder).Append("folder/subfolder/")
b.Field(1).(*arrowArray.StringBuilder).Append("blobprefix")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleHierarchyListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
require.Equal(t, "folder/file.txt", *result.Segment.BlobItems[0].Name)
require.Len(t, result.Segment.BlobPrefixes, 1)
require.Equal(t, "folder/subfolder/", *result.Segment.BlobPrefixes[0].Name)
}
func TestHandleFlatListResponse_ResponseHeaders(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
})
resp := makeHTTPResponse(data, map[string]string{
"x-ms-request-id": "req-123",
"x-ms-version": "2026-10-06",
"x-ms-client-request-id": "client-req-456",
"Date": "Mon, 01 Jan 2024 00:00:00 GMT",
})
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Equal(t, "req-123", *result.RequestID)
require.Equal(t, "2026-10-06", *result.Version)
require.Equal(t, "client-req-456", *result.ClientRequestID)
require.NotNil(t, result.Date)
}
func TestHandleFlatListResponse_MapFields(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Metadata", Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Nullable: true},
{Name: "Tags", Type: arrow.MapOf(arrow.BinaryTypes.String, arrow.BinaryTypes.String), Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
metaBuilder := b.Field(1).(*arrowArray.MapBuilder)
metaBuilder.Append(true)
metaBuilder.KeyBuilder().(*arrowArray.StringBuilder).Append("key1")
metaBuilder.ItemBuilder().(*arrowArray.StringBuilder).Append("value1")
tagBuilder := b.Field(2).(*arrowArray.MapBuilder)
tagBuilder.Append(true)
tagBuilder.KeyBuilder().(*arrowArray.StringBuilder).Append("env")
tagBuilder.ItemBuilder().(*arrowArray.StringBuilder).Append("prod")
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
require.Len(t, result.Segment.BlobItems, 1)
item := result.Segment.BlobItems[0]
require.NotNil(t, item.Metadata)
require.Equal(t, "value1", *item.Metadata["key1"])
require.NotNil(t, item.BlobTags)
require.Len(t, item.BlobTags.BlobTagSet, 1)
}
func TestHandleFlatListResponse_BooleanFields(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Deleted", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
{Name: "IsCurrentVersion", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
{Name: "ServerEncrypted", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("blob.txt")
b.Field(1).(*arrowArray.BooleanBuilder).Append(true)
b.Field(2).(*arrowArray.BooleanBuilder).Append(false)
b.Field(3).(*arrowArray.BooleanBuilder).Append(true)
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
item := result.Segment.BlobItems[0]
require.True(t, *item.Deleted)
require.False(t, *item.IsCurrentVersion)
require.True(t, *item.Properties.ServerEncrypted)
}
func TestHandleFlatListResponse_VersionAndSnapshot(t *testing.T) {
md := arrow.NewMetadata([]string{"NextMarker"}, []string{""})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "VersionId", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "Snapshot", Type: arrow.BinaryTypes.String, Nullable: true},
{Name: "IsCurrentVersion", Type: arrow.FixedWidthTypes.Boolean, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
data := buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append("versioned-blob.txt")
b.Field(1).(*arrowArray.StringBuilder).Append("2024-01-01T00:00:00.0000000Z")
b.Field(2).(*arrowArray.StringBuilder).Append("snap-123")
b.Field(3).(*arrowArray.BooleanBuilder).Append(true)
})
resp := makeHTTPResponse(data, nil)
result, err := HandleFlatListResponse(resp)
require.NoError(t, err)
item := result.Segment.BlobItems[0]
require.Equal(t, "2024-01-01T00:00:00.0000000Z", *item.VersionID)
require.Equal(t, "snap-123", *item.Snapshot)
require.True(t, *item.IsCurrentVersion)
}
+259
View File
@@ -0,0 +1,259 @@
//go:build !plan9 && !solaris && !js
// Package arrowlist implements the experimental "Blob Listing with Apache
// Arrow" feature on top of the released Azure azblob SDK.
//
// The Azure SDK for Go has support for Arrow listing on the unreleased
// feature/storage/bifrost branch (commit c6fa341ca22b) where the listing
// pagers decode Arrow IPC stream responses into the ordinary ListBlobs
// response types. Until that ships in a tagged azblob release this package
// fills the gap: it exposes the same options and pager interface as the
// experimental SDK, built only on the public API of the released SDK plus
// copies of two small internal auth policies (see sharedkey.go and
// challenge_policy.go).
//
// This package is temporary. When Arrow listing ships in a released azblob
// the whole package should be deleted and its callers pointed back at
// container.Client.NewListBlobsHierarchyPager - the option and response
// types here are deliberately source compatible with the SDK's to make that
// a mechanical change.
package arrowlist
import (
"context"
"errors"
"net/http"
"strconv"
"strings"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
)
const (
// moduleName and moduleVersion identify this package in the azcore
// telemetry policy's User-Agent fragment.
moduleName = "github.com/rclone/rclone/backend/azureblob/arrowlist"
moduleVersion = "v0.0.1"
// serviceVersion is the x-ms-version sent with listing requests. Arrow
// responses and the startFrom/endBefore parameters need at least
// 2026-06-06.
serviceVersion = "2026-06-06"
// tokenScope is the OAuth scope for Azure Storage, as used by the SDK's
// own clients.
tokenScope = "https://storage.azure.com/.default"
)
// ErrEndBeforeXMLFallback is returned by the pager when the server answered
// with XML instead of Arrow while EndBefore was set. The service only honours
// endBefore on the Arrow listing path, so an XML page could silently contain
// blobs outside the requested range and cannot be used.
var ErrEndBeforeXMLFallback = errors.New("arrow listing fell back to XML with endBefore set")
// ListBlobsHierarchyOptions provides the configuration for
// Client.NewListBlobsHierarchyPager.
//
// It mirrors the experimental SDK's container.ListBlobsHierarchyOptions:
// the released SDK's options are embedded (providing Include, Marker,
// MaxResults, Prefix and StartFrom by promotion) with the two fields the
// released SDK lacks added alongside.
type ListBlobsHierarchyOptions struct {
container.ListBlobsHierarchyOptions
// EndBefore limits listing to blobs whose full container path is
// lexically before it (exclusive). Only honoured on Arrow responses.
EndBefore *string
// UseArrowFormat requests the response as an Apache Arrow IPC stream.
UseArrowFormat *bool
}
// ClientOptions contains the optional parameters for client creation.
type ClientOptions struct {
azcore.ClientOptions
}
// Client issues Arrow listing requests to a single container.
type Client struct {
endpoint string
azClient *azcore.Client
}
// newClient creates a Client for the container at containerURL sending
// requests through a pipeline with the given auth policy (which may be nil
// for anonymous or SAS access).
func newClient(containerURL string, authPolicy policy.Policy, options *ClientOptions) (*Client, error) {
if options == nil {
options = &ClientOptions{}
}
plOpts := runtime.PipelineOptions{}
if authPolicy != nil {
plOpts.PerRetry = []policy.Policy{authPolicy}
}
azClient, err := azcore.NewClient(moduleName, moduleVersion, plOpts, &options.ClientOptions)
if err != nil {
return nil, err
}
return &Client{
endpoint: containerURL,
azClient: azClient,
}, nil
}
// NewClient creates a Client authenticating with a token credential.
func NewClient(containerURL string, cred azcore.TokenCredential, options *ClientOptions) (*Client, error) {
return newClient(containerURL, NewStorageChallengePolicy(cred, tokenScope, false), options)
}
// NewClientWithNoCredential creates a Client with no authentication, for
// anonymous access or a containerURL carrying a SAS token in its query.
func NewClientWithNoCredential(containerURL string, options *ClientOptions) (*Client, error) {
return newClient(containerURL, nil, options)
}
// NewClientWithSharedKeyCredential creates a Client signing requests with the
// account's shared key.
func NewClientWithSharedKeyCredential(containerURL string, cred *SharedKeyCredential, options *ClientOptions) (*Client, error) {
return newClient(containerURL, NewSharedKeyCredPolicy(cred), options)
}
// NewListBlobsHierarchyPager returns a pager over the container's blobs,
// requesting Arrow responses when o.UseArrowFormat is set and returning the
// same response type as container.Client.NewListBlobsHierarchyPager.
//
// If the service does not support Arrow listing (the feature not enabled on
// the account, or an HNS account) it answers with XML, which is decoded
// transparently - unless EndBefore is set, in which case the fetcher returns
// ErrEndBeforeXMLFallback (see there).
func (c *Client) NewListBlobsHierarchyPager(delimiter string, o *ListBlobsHierarchyOptions) *runtime.Pager[container.ListBlobsHierarchyResponse] {
// Copy the options so advancing the Marker doesn't mutate the caller's struct.
opts := ListBlobsHierarchyOptions{}
if o != nil {
opts = *o
}
useArrow := opts.UseArrowFormat != nil && *opts.UseArrowFormat
return runtime.NewPager(runtime.PagingHandler[container.ListBlobsHierarchyResponse]{
More: func(page container.ListBlobsHierarchyResponse) bool {
return page.NextMarker != nil && len(*page.NextMarker) > 0
},
Fetcher: func(ctx context.Context, page *container.ListBlobsHierarchyResponse) (container.ListBlobsHierarchyResponse, error) {
if page != nil {
opts.Marker = page.NextMarker
}
req, err := c.listCreateRequest(ctx, delimiter, &opts, useArrow)
if err != nil {
return container.ListBlobsHierarchyResponse{}, err
}
resp, err := c.azClient.Pipeline().Do(req)
if err != nil {
return container.ListBlobsHierarchyResponse{}, err
}
if !runtime.HasStatusCode(resp, http.StatusOK) {
return container.ListBlobsHierarchyResponse{}, runtime.NewResponseError(resp)
}
if useArrow && resp.Header.Get("Content-Type") == ArrowContentType {
return HandleHierarchyListResponse(resp)
}
if opts.EndBefore != nil {
return container.ListBlobsHierarchyResponse{}, ErrEndBeforeXMLFallback
}
return handleXMLResponse(resp)
},
})
}
// listCreateRequest builds the ListBlobs request, mirroring the generated
// request builders in the SDK.
func (c *Client) listCreateRequest(ctx context.Context, delimiter string, opts *ListBlobsHierarchyOptions, useArrow bool) (*policy.Request, error) {
req, err := runtime.NewRequest(ctx, http.MethodGet, c.endpoint)
if err != nil {
return nil, err
}
// Reading the existing query first preserves any SAS token in the endpoint.
reqQP := req.Raw().URL.Query()
reqQP.Set("comp", "list")
reqQP.Set("restype", "container")
reqQP.Set("delimiter", delimiter)
if include := formatInclude(opts.Include); include != "" {
reqQP.Set("include", include)
}
if opts.Marker != nil {
reqQP.Set("marker", *opts.Marker)
}
if opts.MaxResults != nil {
reqQP.Set("maxresults", strconv.FormatInt(int64(*opts.MaxResults), 10))
}
if opts.Prefix != nil {
reqQP.Set("prefix", *opts.Prefix)
}
if opts.StartFrom != nil {
reqQP.Set("startFrom", *opts.StartFrom)
}
if useArrow && opts.EndBefore != nil {
reqQP.Set("endBefore", *opts.EndBefore)
}
req.Raw().URL.RawQuery = strings.ReplaceAll(reqQP.Encode(), "+", "%20")
if useArrow {
// Stream the Arrow body rather than buffering it in the pipeline.
runtime.SkipBodyDownload(req)
req.Raw().Header["Accept"] = []string{ArrowAcceptHeader}
} else {
req.Raw().Header["Accept"] = []string{"application/xml"}
}
req.Raw().Header["x-ms-version"] = []string{serviceVersion}
return req, nil
}
// handleXMLResponse decodes an XML ListBlobs response, mirroring the SDK's
// generated ListBlobHierarchySegmentHandleResponse.
func handleXMLResponse(resp *http.Response) (container.ListBlobsHierarchyResponse, error) {
result := container.ListBlobsHierarchyResponse{}
extractResponseHeaders(resp, &result.ClientRequestID, &result.ContentType, &result.Date, &result.RequestID, &result.Version)
if err := runtime.UnmarshalAsXML(resp, &result.ListBlobsHierarchySegmentResponse); err != nil {
return container.ListBlobsHierarchyResponse{}, err
}
return result, nil
}
// formatInclude renders the include datasets as the comma separated query
// parameter value, in the same order as the SDK's unexported
// ListBlobsInclude.format.
func formatInclude(l container.ListBlobsInclude) string {
var include []string
if l.Copy {
include = append(include, "copy")
}
if l.Deleted {
include = append(include, "deleted")
}
if l.DeletedWithVersions {
include = append(include, "deletedwithversions")
}
if l.ImmutabilityPolicy {
include = append(include, "immutabilitypolicy")
}
if l.LegalHold {
include = append(include, "legalhold")
}
if l.Metadata {
include = append(include, "metadata")
}
if l.Snapshots {
include = append(include, "snapshots")
}
if l.Tags {
include = append(include, "tags")
}
if l.UncommittedBlobs {
include = append(include, "uncommittedblobs")
}
if l.Versions {
include = append(include, "versions")
}
if l.Permissions {
include = append(include, "permissions")
}
return strings.Join(include, ",")
}
@@ -0,0 +1,199 @@
//go:build !plan9 && !solaris && !js
package arrowlist
import (
"context"
"encoding/base64"
"net/http"
"net/http/httptest"
"strings"
"testing"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/apache/arrow-go/v18/arrow"
arrowArray "github.com/apache/arrow-go/v18/arrow/array"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// arrowPage builds a one row Arrow IPC listing page with the given blob name
// and NextMarker.
func arrowPage(t *testing.T, name, nextMarker string) []byte {
md := arrow.NewMetadata([]string{"NextMarker", "NumberOfRecords"}, []string{nextMarker, "1"})
fields := []arrow.Field{
{Name: "Name", Type: arrow.BinaryTypes.String, Nullable: false},
{Name: "Content-Length", Type: arrow.PrimitiveTypes.Uint64, Nullable: true},
}
schema := arrow.NewSchema(fields, &md)
return buildArrowStream(t, schema, func(b *arrowArray.RecordBuilder) {
b.Field(0).(*arrowArray.StringBuilder).Append(name)
b.Field(1).(*arrowArray.Uint64Builder).Append(42)
})
}
func TestPagerArrow(t *testing.T) {
var requests []*http.Request
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests = append(requests, r.Clone(context.Background()))
w.Header().Set("Content-Type", ArrowContentType)
if r.URL.Query().Get("marker") == "" {
_, _ = w.Write(arrowPage(t, "blob1.txt", "marker1"))
} else {
_, _ = w.Write(arrowPage(t, "blob2.txt", ""))
}
}))
defer srv.Close()
// SAS style query on the endpoint must be preserved in requests.
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer?sv=fakesas", nil)
require.NoError(t, err)
opts := &ListBlobsHierarchyOptions{
ListBlobsHierarchyOptions: container.ListBlobsHierarchyOptions{
Include: container.ListBlobsInclude{Metadata: true, Tags: true},
Prefix: to.Ptr("dir/"),
MaxResults: to.Ptr(int32(1000)),
StartFrom: to.Ptr("testcontainer/dir/a"),
},
EndBefore: to.Ptr("testcontainer/dir/n"),
UseArrowFormat: to.Ptr(true),
}
pager := client.NewListBlobsHierarchyPager("/", opts)
var names []string
for pager.More() {
page, err := pager.NextPage(context.Background())
require.NoError(t, err)
assert.Equal(t, ArrowContentType, *page.ContentType)
for _, item := range page.Segment.BlobItems {
names = append(names, *item.Name)
assert.Equal(t, int64(42), *item.Properties.ContentLength)
}
}
assert.Equal(t, []string{"blob1.txt", "blob2.txt"}, names)
require.Len(t, requests, 2)
q := requests[0].URL.Query()
assert.Equal(t, "list", q.Get("comp"))
assert.Equal(t, "container", q.Get("restype"))
assert.Equal(t, "/", q.Get("delimiter"))
assert.Equal(t, "metadata,tags", q.Get("include"))
assert.Equal(t, "dir/", q.Get("prefix"))
assert.Equal(t, "1000", q.Get("maxresults"))
assert.Equal(t, "testcontainer/dir/a", q.Get("startFrom"))
assert.Equal(t, "testcontainer/dir/n", q.Get("endBefore"))
assert.Equal(t, "fakesas", q.Get("sv"))
assert.Empty(t, q.Get("marker"))
assert.Equal(t, ArrowAcceptHeader, requests[0].Header.Get("Accept"))
assert.Equal(t, serviceVersion, requests[0].Header.Get("x-ms-version"))
assert.Equal(t, "marker1", requests[1].URL.Query().Get("marker"))
// The caller's options must not have been mutated by paging.
assert.Nil(t, opts.Marker)
}
const xmlListResponse = `<?xml version="1.0" encoding="utf-8"?>
<EnumerationResults ServiceEndpoint="https://example.com/" ContainerName="testcontainer">
<Blobs>
<Blob>
<Name>file.txt</Name>
<Properties>
<Last-Modified>Mon, 01 Jan 2024 00:00:00 GMT</Last-Modified>
<Etag>0x8D1</Etag>
<Content-Length>7</Content-Length>
<BlobType>BlockBlob</BlobType>
</Properties>
</Blob>
<BlobPrefix>
<Name>subdir/</Name>
</BlobPrefix>
</Blobs>
<NextMarker />
</EnumerationResults>`
func TestPagerXMLFallback(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml")
_, _ = w.Write([]byte(xmlListResponse))
}))
defer srv.Close()
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer", nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
UseArrowFormat: to.Ptr(true),
})
page, err := pager.NextPage(context.Background())
require.NoError(t, err)
assert.False(t, pager.More())
assert.Equal(t, "application/xml", *page.ContentType)
require.Len(t, page.Segment.BlobItems, 1)
assert.Equal(t, "file.txt", *page.Segment.BlobItems[0].Name)
assert.Equal(t, int64(7), *page.Segment.BlobItems[0].Properties.ContentLength)
require.Len(t, page.Segment.BlobPrefixes, 1)
assert.Equal(t, "subdir/", *page.Segment.BlobPrefixes[0].Name)
}
func TestPagerXMLFallbackWithEndBefore(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/xml")
_, _ = w.Write([]byte(xmlListResponse))
}))
defer srv.Close()
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer", nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
EndBefore: to.Ptr("testcontainer/n"),
UseArrowFormat: to.Ptr(true),
})
_, err = pager.NextPage(context.Background())
require.ErrorIs(t, err, ErrEndBeforeXMLFallback)
}
func TestPagerSharedKeyAuth(t *testing.T) {
var gotAuth, gotDate string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
gotDate = r.Header.Get("x-ms-date")
w.Header().Set("Content-Type", ArrowContentType)
_, _ = w.Write(arrowPage(t, "blob.txt", ""))
}))
defer srv.Close()
cred, err := NewSharedKeyCredential("testaccount", base64.StdEncoding.EncodeToString([]byte("testkey")))
require.NoError(t, err)
client, err := NewClientWithSharedKeyCredential(srv.URL+"/testcontainer", cred, nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
UseArrowFormat: to.Ptr(true),
})
_, err = pager.NextPage(context.Background())
require.NoError(t, err)
assert.True(t, strings.HasPrefix(gotAuth, "SharedKey testaccount:"), "Authorization = %q", gotAuth)
assert.NotEmpty(t, gotDate)
}
func TestPagerResponseError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("x-ms-error-code", string(bloberror.ContainerNotFound))
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
client, err := NewClientWithNoCredential(srv.URL+"/testcontainer", nil)
require.NoError(t, err)
pager := client.NewListBlobsHierarchyPager("/", &ListBlobsHierarchyOptions{
UseArrowFormat: to.Ptr(true),
})
_, err = pager.NextPage(context.Background())
require.Error(t, err)
assert.True(t, bloberror.HasCode(err, bloberror.ContainerNotFound), "expected ContainerNotFound, got %v", err)
}
@@ -0,0 +1,121 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/shared/challenge_policy.go in the Azure SDK for Go
// github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at v1.8.0. The
// released SDK does not export its storage bearer-challenge policy, which is
// needed for token authentication with the same semantics as the SDK's own
// clients.
package arrowlist
import (
"errors"
"net/http"
"strings"
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
)
type storageAuthorizer struct {
scopes []string
tenantID string
}
// NewStorageChallengePolicy returns a bearer token policy for cred which
// handles the storage service's authentication challenges.
func NewStorageChallengePolicy(cred azcore.TokenCredential, audience string, allowHTTP bool) policy.Policy {
s := storageAuthorizer{scopes: []string{audience}}
return runtime.NewBearerTokenPolicy(cred, []string{audience}, &policy.BearerTokenOptions{
AuthorizationHandler: policy.AuthorizationHandler{
OnRequest: s.onRequest,
OnChallenge: s.onChallenge,
},
InsecureAllowCredentialWithHTTP: allowHTTP,
})
}
func (s *storageAuthorizer) onRequest(req *policy.Request, authNZ func(policy.TokenRequestOptions) error) error {
return authNZ(policy.TokenRequestOptions{Scopes: s.scopes})
}
func (s *storageAuthorizer) onChallenge(req *policy.Request, resp *http.Response, authNZ func(policy.TokenRequestOptions) error) error {
// parse the challenge
err := s.parseChallenge(resp)
if err != nil {
return err
}
// TODO: Set tenantID when policy.TokenRequestOptions supports it. https://github.com/Azure/azure-sdk-for-go/issues/19841
return authNZ(policy.TokenRequestOptions{Scopes: s.scopes})
}
type challengePolicyError struct {
err error
}
func (c *challengePolicyError) Error() string {
return c.err.Error()
}
func (*challengePolicyError) NonRetriable() {
// marker method
}
func (c *challengePolicyError) Unwrap() error {
return c.err
}
// parses Tenant ID from auth challenge
// https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/oauth2/authorize
func parseTenant(url string) string {
if url == "" {
return ""
}
parts := strings.Split(url, "/")
if len(parts) >= 3 {
tenant := parts[3]
tenant = strings.ReplaceAll(tenant, ",", "")
return tenant
}
return ""
}
func (s *storageAuthorizer) parseChallenge(resp *http.Response) error {
authHeader := resp.Header.Get("WWW-Authenticate")
if authHeader == "" {
return &challengePolicyError{err: errors.New("response has no WWW-Authenticate header for challenge authentication")}
}
// Strip down to auth and resource
// Format is "Bearer authorization_uri=\"<site>\" resource_id=\"<site>\""
authHeader = strings.ReplaceAll(authHeader, "Bearer ", "")
parts := strings.Split(authHeader, " ")
vals := map[string]string{}
for _, part := range parts {
subParts := strings.Split(part, "=")
if len(subParts) == 2 {
stripped := strings.ReplaceAll(subParts[1], "\"", "")
stripped = strings.TrimSuffix(stripped, ",")
vals[subParts[0]] = stripped
}
}
s.tenantID = parseTenant(vals["authorization_uri"])
scope := vals["resource_id"]
if scope == "" {
return &challengePolicyError{err: errors.New("could not find a valid resource in the WWW-Authenticate header")}
}
if !strings.HasSuffix(scope, "/.default") {
scope += "/.default"
}
s.scopes = []string{scope}
return nil
}
+318
View File
@@ -0,0 +1,318 @@
//go:build !plan9 && !solaris && !js
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
//
// Adapted from internal/exported/shared_key_credential.go in the Azure SDK
// for Go github.com/Azure/azure-sdk-for-go/sdk/storage/azblob at v1.8.0,
// with the internal header constants inlined. The released SDK does not
// export its shared key signing policy, so a copy is needed to sign the
// Arrow listing requests this package sends through its own pipeline.
package arrowlist
import (
"bytes"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"fmt"
"net/http"
"net/url"
"sort"
"strings"
"sync/atomic"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/rclone/rclone/fs"
)
// NewSharedKeyCredential creates an immutable SharedKeyCredential containing the
// storage account's name and either its primary or secondary key.
func NewSharedKeyCredential(accountName string, accountKey string) (*SharedKeyCredential, error) {
c := SharedKeyCredential{accountName: accountName}
if err := c.SetAccountKey(accountKey); err != nil {
return nil, err
}
return &c, nil
}
// SharedKeyCredential contains an account's name and its primary or secondary key.
type SharedKeyCredential struct {
// Only the NewSharedKeyCredential method should set these; all other methods should treat them as read-only
accountName string
accountKey atomic.Value // []byte
}
// AccountName returns the Storage account's name.
func (c *SharedKeyCredential) AccountName() string {
return c.accountName
}
// SetAccountKey replaces the existing account key with the specified account key.
func (c *SharedKeyCredential) SetAccountKey(accountKey string) error {
_bytes, err := base64.StdEncoding.DecodeString(accountKey)
if err != nil {
return fmt.Errorf("decode account key: %w", err)
}
c.accountKey.Store(_bytes)
return nil
}
// ComputeHMACSHA256 generates a hash signature for an HTTP request or for a SAS.
func (c *SharedKeyCredential) computeHMACSHA256(message string) (string, error) {
h := hmac.New(sha256.New, c.accountKey.Load().([]byte))
_, err := h.Write([]byte(message))
return base64.StdEncoding.EncodeToString(h.Sum(nil)), err
}
func (c *SharedKeyCredential) buildStringToSign(req *http.Request) (string, error) {
// https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
headers := req.Header
contentLength := getHeader("Content-Length", headers)
if contentLength == "0" {
contentLength = ""
}
canonicalizedResource, err := c.buildCanonicalizedResource(req.URL)
if err != nil {
return "", err
}
stringToSign := strings.Join([]string{
req.Method,
getHeader("Content-Encoding", headers),
getHeader("Content-Language", headers),
contentLength,
getHeader("Content-MD5", headers),
getHeader("Content-Type", headers),
"", // Empty date because x-ms-date is expected (as per web page above)
getHeader("If-Modified-Since", headers),
getHeader("If-Match", headers),
getHeader("If-None-Match", headers),
getHeader("If-Unmodified-Since", headers),
getHeader("Range", headers),
c.buildCanonicalizedHeader(headers),
canonicalizedResource,
}, "\n")
return stringToSign, nil
}
func getHeader(key string, headers map[string][]string) string {
if headers == nil {
return ""
}
if v, ok := headers[key]; ok {
if len(v) > 0 {
return v[0]
}
}
return ""
}
func getWeightTables() [][]int {
tableLv0 := [...]int{
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x71c, 0x0, 0x71f, 0x721, 0x723, 0x725,
0x0, 0x0, 0x0, 0x72d, 0x803, 0x0, 0x0, 0x733, 0x0, 0xd03, 0xd1a, 0xd1c, 0xd1e,
0xd20, 0xd22, 0xd24, 0xd26, 0xd28, 0xd2a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0xe02, 0xe09, 0xe0a, 0xe1a, 0xe21, 0xe23, 0xe25, 0xe2c, 0xe32, 0xe35, 0xe36, 0xe48, 0xe51,
0xe70, 0xe7c, 0xe7e, 0xe89, 0xe8a, 0xe91, 0xe99, 0xe9f, 0xea2, 0xea4, 0xea6, 0xea7, 0xea9,
0x0, 0x0, 0x0, 0x743, 0x744, 0x748, 0xe02, 0xe09, 0xe0a, 0xe1a, 0xe21, 0xe23, 0xe25,
0xe2c, 0xe32, 0xe35, 0xe36, 0xe48, 0xe51, 0xe70, 0xe7c, 0xe7e, 0xe89, 0xe8a, 0xe91, 0xe99,
0xe9f, 0xea2, 0xea4, 0xea6, 0xea7, 0xea9, 0x0, 0x74c, 0x0, 0x750, 0x0,
}
tableLv2 := [...]int{
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8012, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8212, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0,
}
tables := [][]int{tableLv0[:], tableLv2[:]}
return tables
}
// NewHeaderStringComparer performs a multi-level, weight-based comparison of two strings
func compareHeaders(lhs, rhs string, tables [][]int) int {
currLevel, i, j := 0, 0, 0
n := len(tables)
lhsLen := len(lhs)
rhsLen := len(rhs)
for currLevel < n {
if currLevel == (n-1) && i != j {
if i > j {
return -1
}
if i < j {
return 1
}
return 0
}
var w1, w2 int
// Check bounds before accessing lhs[i]
if i < lhsLen {
w1 = tables[currLevel][lhs[i]]
} else {
w1 = 0x1
}
// Check bounds before accessing rhs[j]
if j < rhsLen {
w2 = tables[currLevel][rhs[j]]
} else {
w2 = 0x1
}
if w1 == 0x1 && w2 == 0x1 {
i = 0
j = 0
currLevel++
} else if w1 == w2 {
i++
j++
} else if w1 == 0 {
i++
} else if w2 == 0 {
j++
} else {
if w1 < w2 {
return -1
}
if w1 > w2 {
return 1
}
return 0
}
}
return 0
}
func (c *SharedKeyCredential) buildCanonicalizedHeader(headers http.Header) string {
cm := map[string][]string{}
for k, v := range headers {
headerName := strings.TrimSpace(strings.ToLower(k))
if strings.HasPrefix(headerName, "x-ms-") {
cm[headerName] = v // NOTE: the value must not have any whitespace around it.
}
}
if len(cm) == 0 {
return ""
}
keys := make([]string, 0, len(cm))
for key := range cm {
keys = append(keys, key)
}
tables := getWeightTables()
// Sort the keys using the custom comparator
sort.Slice(keys, func(i, j int) bool {
return compareHeaders(keys[i], keys[j], tables) < 0
})
ch := bytes.NewBufferString("")
for i, key := range keys {
if i > 0 {
ch.WriteRune('\n')
}
ch.WriteString(key)
ch.WriteRune(':')
ch.WriteString(strings.Join(cm[key], ","))
}
return ch.String()
}
func (c *SharedKeyCredential) buildCanonicalizedResource(u *url.URL) (string, error) {
// https://docs.microsoft.com/en-us/rest/api/storageservices/authentication-for-the-azure-storage-services
cr := bytes.NewBufferString("/")
cr.WriteString(c.accountName)
if len(u.Path) > 0 {
// Any portion of the CanonicalizedResource string that is derived from
// the resource's URI should be encoded exactly as it is in the URI.
// -- https://msdn.microsoft.com/en-gb/library/azure/dd179428.aspx
cr.WriteString(u.EscapedPath())
} else {
// a slash is required to indicate the root path
cr.WriteString("/")
}
// params is a map[string][]string; param name is key; params values is []string
params, err := url.ParseQuery(u.RawQuery) // Returns URL decoded values
if err != nil {
return "", fmt.Errorf("failed to parse query params: %w", err)
}
if len(params) > 0 { // There is at least 1 query parameter
var paramNames []string // We use this to sort the parameter key names
for paramName := range params {
paramNames = append(paramNames, paramName) // paramNames must be lowercase
}
sort.Strings(paramNames)
for _, paramName := range paramNames {
paramValues := params[paramName]
sort.Strings(paramValues)
// Join the sorted key values separated by ','
// Then prepend "keyName:"; then add this string to the buffer
cr.WriteString("\n" + strings.ToLower(paramName) + ":" + strings.Join(paramValues, ","))
}
}
return cr.String(), nil
}
// ComputeHMACSHA256 is a helper for computing the signed string outside of this package.
func ComputeHMACSHA256(cred *SharedKeyCredential, message string) (string, error) {
return cred.computeHMACSHA256(message)
}
// SharedKeyCredPolicy is a pipeline policy that signs each request with the
// account's shared key.
type SharedKeyCredPolicy struct {
cred *SharedKeyCredential
}
// NewSharedKeyCredPolicy creates a SharedKeyCredPolicy signing with cred.
func NewSharedKeyCredPolicy(cred *SharedKeyCredential) *SharedKeyCredPolicy {
return &SharedKeyCredPolicy{cred: cred}
}
// Do signs the request and sends it via the next policy in the pipeline.
func (s *SharedKeyCredPolicy) Do(req *policy.Request) (*http.Response, error) {
// skip adding the authorization header if no SharedKeyCredential was provided.
// this prevents a panic that might be hard to diagnose and allows testing
// against http endpoints that don't require authentication.
if s.cred == nil {
return req.Next()
}
if d := getHeader("x-ms-date", req.Raw().Header); d == "" {
req.Raw().Header.Set("x-ms-date", time.Now().UTC().Format(http.TimeFormat))
}
stringToSign, err := s.cred.buildStringToSign(req.Raw())
if err != nil {
return nil, err
}
signature, err := s.cred.computeHMACSHA256(stringToSign)
if err != nil {
return nil, err
}
authHeader := strings.Join([]string{"SharedKey ", s.cred.AccountName(), ":", signature}, "")
req.Raw().Header.Set("Authorization", authHeader)
response, err := req.Next()
if err != nil && response != nil && response.StatusCode == http.StatusForbidden {
// Service failed to authenticate request, log it
fs.Debugf(nil, "azureblob arrowlist: HTTP Forbidden status, String-to-Sign: %q", stringToSign)
}
return response, err
}
+158 -220
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
// Package azureblob provides an interface to the Microsoft Azure blob object storage system
package azureblob
@@ -19,7 +19,6 @@ import (
"path"
"slices"
"sort"
"strconv"
"strings"
"sync"
"sync/atomic"
@@ -27,12 +26,15 @@ import (
"github.com/Azure/azure-sdk-for-go/sdk/azcore"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/bloberror"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/container"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/sas"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service"
"github.com/rclone/rclone/backend/azureblob/arrowlist"
"github.com/rclone/rclone/backend/azureblob/auth"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/chunksize"
@@ -240,38 +242,40 @@ avoid the time out.`,
Advanced: true,
}, {
Name: "use_arrow_list",
Help: `Use the Apache Arrow listing format.
Help: `Use the experimental Apache Arrow listing format.
If set, directory listings are fetched using the ListBlobs Apache
Arrow response format instead of XML. Arrow responses are smaller and
much cheaper to parse, making listings of large containers several
times faster. Combine with "list_parallelism" for the biggest gains.
Arrow response format instead of XML. This can be faster for very
large containers.
"Blob Listing with Apache Arrow" is in public preview at Microsoft and
is only supported on flat namespace accounts. On accounts with a
hierarchical namespace (ADLS Gen2), or where the feature is otherwise
unavailable, the server returns XML and the listing transparently
falls back to the normal XML path (logged at debug level).`,
This is EXPERIMENTAL and requires the "Blob Listing with Apache Arrow"
preview feature to be enabled on the storage account. It is NOT
supported on accounts with a hierarchical namespace (ADLS Gen2) -
those return a 409 error. If the feature is not enabled the server
returns XML and the listing transparently falls back to the normal XML
path (logged at debug level).
Not supported with connection_string auth - falls back to normal
listing.`,
Default: false,
Advanced: true,
Hide: fs.OptionHideBoth,
}, {
Name: "list_parallelism",
Help: `Number of parallel shards to list a directory with.
If set greater than 1, the blob name keyspace of each directory is
split into ranges which are listed concurrently using the Arrow
startFrom/endBefore range parameters. This can dramatically speed up
listing containers with millions of objects, for both recursive
(ListR) and single directory listings. Speed keeps improving up to a
parallelism of around 30.
EXPERIMENTAL. If set greater than 1, the blob name keyspace of each
directory is split into this many ranges which are listed concurrently
using the Arrow startFrom/endBefore range parameters. This can
dramatically speed up listing containers with millions of objects, for
both recursive (ListR) and single directory listings.
This has no effect unless "use_arrow_list" is also set, as Arrow is the
only listing path that supports server-side name ranges. If the
account does not support range listing (e.g. it has a hierarchical
namespace) the listing falls back to sequential. The default of 0 (or
1) lists sequentially.`,
only listing path that supports server-side name ranges. The default of
0 (or 1) lists sequentially.`,
Default: 0,
Advanced: true,
Hide: fs.OptionHideBoth,
}, {
Name: "access_tier",
Help: `Access tier of blob: hot, cool, cold or archive.
@@ -365,22 +369,7 @@ creates an empty object ending with "/", to persist the folder.
This object also has the metadata "` + dirMetaKey + ` = ` + dirMetaValue + `" to conform to
the Microsoft standard.
This is not needed on storage accounts with a hierarchical namespace
(see the hns option) as these always support empty directories.
`,
}, {
Name: "hns",
Default: fs.Tristate{},
Advanced: true,
Help: `Set if the storage account has a hierarchical namespace.
Storage accounts with hierarchical namespace enabled (Azure Data Lake
Storage Gen2) have real directories which can be empty.
If this is unset then rclone will read it from the storage account
and save it in the config file.
`,
}, {
Name: "no_check_container",
Help: `If set, don't attempt to check the container exists or create it.
@@ -448,7 +437,6 @@ type Options struct {
Enc encoder.MultiEncoder `config:"encoding"`
PublicAccess string `config:"public_access"`
DirectoryMarkers bool `config:"directory_markers"`
HNS fs.Tristate `config:"hns"`
NoCheckContainer bool `config:"no_check_container"`
NoHeadObject bool `config:"no_head_object"`
DeleteSnapshots string `config:"delete_snapshots"`
@@ -462,8 +450,10 @@ type Fs struct {
opt Options // parsed config options
ci *fs.ConfigInfo // global config
features *fs.Features // optional features
cntSVCcacheMu sync.Mutex // mutex to protect cntSVCcache
cntSVCcacheMu sync.Mutex // mutex to protect cntSVCcache and arrowCntSVCcache
cntSVCcache map[string]*container.Client // reference to containerClient per container
arrowCntSVCcache map[string]*arrowlist.Client // reference to arrowlist client per container
arrowClientOpts *arrowlist.ClientOptions // client options for the arrowlist clients
arrowXMLFallback atomic.Bool // set once the server answers XML so parallel listing is not retried
svc *service.Client // client to access azblob
cred azcore.TokenCredential // how to generate tokens (may be nil)
@@ -472,7 +462,6 @@ type Fs struct {
rootContainer string // container part of root (if any)
rootDirectory string // directory part of root (if any)
isLimited bool // if limited to one container
hns bool // if the storage account has a hierarchical namespace
cache *bucket.Cache // cache for container creation status
pacer *fs.Pacer // To pace and retry the API calls
uploadToken *pacer.TokenDispenser // control concurrency
@@ -539,7 +528,7 @@ func parsePath(path string) (root string) {
// relative to f.root
func (f *Fs) split(rootRelativePath string) (containerName, containerPath string) {
containerName, containerPath = bucket.Split(bucket.Join(f.root, rootRelativePath))
if f.dirMarkers() && strings.HasSuffix(containerPath, "//") {
if f.opt.DirectoryMarkers && strings.HasSuffix(containerPath, "//") {
containerPath = containerPath[:len(containerPath)-1]
}
return f.opt.Enc.FromStandardName(containerName), f.opt.Enc.FromStandardPath(containerPath)
@@ -587,7 +576,8 @@ func (f *Fs) shouldRetry(ctx context.Context, err error) (bool, error) {
if fserrors.ContextError(ctx, &err) {
return false, err
}
if storageErr, ok := errors.AsType[*azcore.ResponseError](err); ok {
var storageErr *azcore.ResponseError
if errors.As(err, &storageErr) {
// General errors from:
// https://learn.microsoft.com/en-us/rest/api/storageservices/common-rest-api-error-codes
// Blob specific errors from:
@@ -682,14 +672,15 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
ci := fs.GetConfig(ctx)
f := &Fs{
name: name,
opt: *opt,
ci: ci,
pacer: fs.NewPacer(ctx, pacer.NewS3(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
uploadToken: pacer.NewTokenDispenser(ci.Transfers),
copyToken: pacer.NewTokenDispenser(opt.CopyTotalConcurrency),
cache: bucket.NewCache(),
cntSVCcache: make(map[string]*container.Client, 1),
name: name,
opt: *opt,
ci: ci,
pacer: fs.NewPacer(ctx, pacer.NewS3(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
uploadToken: pacer.NewTokenDispenser(ci.Transfers),
copyToken: pacer.NewTokenDispenser(opt.CopyTotalConcurrency),
cache: bucket.NewCache(),
cntSVCcache: make(map[string]*container.Client, 1),
arrowCntSVCcache: make(map[string]*arrowlist.Client, 1),
}
f.publicAccess = container.PublicAccessType(opt.PublicAccess)
f.setRoot(root)
@@ -706,6 +697,10 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
ServerSideAcrossConfigs: true,
DoubleSlash: true,
}).Fill(ctx, f)
if opt.DirectoryMarkers {
f.features.CanHaveEmptyDirectories = true
fs.Debugf(f, "Using directory markers")
}
conf := auth.NewClientOpts[service.Client, service.ClientOptions, service.SharedKeyCredential]{
DefaultBaseURL: storageDefaultBaseURL,
@@ -732,34 +727,21 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
f.usingSharedKeyCred = res.UsingSharedKeyCred
f.anonymous = res.Anonymous
// Client options for the arrowlist clients, using the same transport and
// gzip policy as the SDK clients built above.
f.arrowClientOpts = &arrowlist.ClientOptions{
ClientOptions: azcore.ClientOptions{
Transport: auth.Transporter(ctx),
PerCallPolicies: []policy.Policy{setAcceptEncodingGzip{}},
},
}
// if using Container level SAS put the container client into the cache
if opt.SASURL != "" && res.Container != "" {
_ = f.cntSVC(res.Container)
f.isLimited = true
}
if opt.HNS.Valid {
f.hns = opt.HNS.Value
} else if !f.anonymous {
containerName := f.rootContainer
if containerName == "" {
containerName = res.Container
}
f.hns, err = f.readHNS(ctx, containerName)
if err != nil {
fs.Debugf(f, "Failed to read whether storage account has a hierarchical namespace - assuming not: %v", err)
} else {
m.Set("hns", strconv.FormatBool(f.hns))
}
}
if f.hns {
fs.Debugf(f, "Storage account has a hierarchical namespace")
}
if f.dirMarkers() {
f.features.CanHaveEmptyDirectories = true
fs.Debugf(f, "Using directory markers")
}
if f.rootContainer != "" && f.rootDirectory != "" {
// Check to see if the (container,directory) is actually an existing file
oldRoot := f.root
@@ -780,39 +762,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
return f, nil
}
// readHNS reads whether the storage account has a hierarchical namespace
//
// If containerName is set then the container client is used which works
// with credentials limited to that container.
func (f *Fs) readHNS(ctx context.Context, containerName string) (hns bool, err error) {
var isHNS *bool
err = f.pacer.Call(func() (bool, error) {
var err error
if containerName != "" {
var resp container.GetAccountInfoResponse
resp, err = f.cntSVC(containerName).GetAccountInfo(ctx, nil)
isHNS = resp.IsHierarchicalNamespaceEnabled
} else {
var resp service.GetAccountInfoResponse
resp, err = f.svc.GetAccountInfo(ctx, nil)
isHNS = resp.IsHierarchicalNamespaceEnabled
}
return f.shouldRetry(ctx, err)
})
if err != nil {
return false, err
}
if isHNS == nil {
return false, errors.New("not returned by server")
}
return *isHNS, nil
}
// dirMarkers returns true if directories are persisted with directory markers
func (f *Fs) dirMarkers() bool {
return f.opt.DirectoryMarkers || f.hns
}
// return the container client for the container passed in
func (f *Fs) cntSVC(containerName string) (containerClient *container.Client) {
f.cntSVCcacheMu.Lock()
@@ -825,6 +774,44 @@ func (f *Fs) cntSVC(containerName string) (containerClient *container.Client) {
return containerClient
}
// errArrowAuthUnsupported is returned by arrowCntSVC when the configured
// credentials can't be used for Arrow listing.
var errArrowAuthUnsupported = errors.New("credentials not supported for Arrow listing (connection_string auth is not supported)")
// return the arrowlist client for the container passed in
//
// Returns errArrowAuthUnsupported if the configured credentials can't be
// reused for the arrowlist client's pipeline.
func (f *Fs) arrowCntSVC(containerName string) (client *arrowlist.Client, err error) {
// The container URL includes any SAS token in its query
url := f.cntSVC(containerName).URL()
f.cntSVCcacheMu.Lock()
defer f.cntSVCcacheMu.Unlock()
if client, ok := f.arrowCntSVCcache[containerName]; ok {
return client, nil
}
switch {
case f.usingSharedKeyCred:
// Covers account+key and the emulator (auth fills in Account/Key)
var cred *arrowlist.SharedKeyCredential
cred, err = arrowlist.NewSharedKeyCredential(f.opt.Account, f.opt.Key)
if err == nil {
client, err = arrowlist.NewClientWithSharedKeyCredential(url, cred, f.arrowClientOpts)
}
case f.cred != nil:
client, err = arrowlist.NewClient(url, f.cred, f.arrowClientOpts)
case f.anonymous || f.opt.SASURL != "":
client, err = arrowlist.NewClientWithNoCredential(url, f.arrowClientOpts)
default:
return nil, errArrowAuthUnsupported
}
if err != nil {
return nil, err
}
f.arrowCntSVCcache[containerName] = client
return client, nil
}
// Return an Object from a path
//
// If it can't be found it returns the error fs.ErrorObjectNotFound.
@@ -850,12 +837,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *contain
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
o, err := f.newObjectWithInfo(ctx, remote, nil)
if err == fs.ErrorNotAFile {
// The blob is a directory
return nil, fs.ErrorObjectNotFound
}
return o, err
return f.newObjectWithInfo(ctx, remote, nil)
}
// getBlobSVC creates a blob client
@@ -924,15 +906,15 @@ func mapMetadataToAzure(meta map[string]string, logf func(string, ...any)) (head
lowerKey := strings.ToLower(k)
switch lowerKey {
case "cache-control":
headers.BlobCacheControl = new(v)
headers.BlobCacheControl = pString(v)
case "content-disposition":
headers.BlobContentDisposition = new(v)
headers.BlobContentDisposition = pString(v)
case "content-encoding":
headers.BlobContentEncoding = new(v)
headers.BlobContentEncoding = pString(v)
case "content-language":
headers.BlobContentLanguage = new(v)
headers.BlobContentLanguage = pString(v)
case "content-type":
headers.BlobContentType = new(v)
headers.BlobContentType = pString(v)
case "x-ms-tags":
parsed, perr := parseXMsTags(v)
if perr != nil {
@@ -1169,33 +1151,18 @@ func isDirectoryMarker(size int64, metadata map[string]*string, remote string) b
if endsWithSlash || remote == "" {
return true
}
return hasDirMetadata(metadata)
}
return false
}
// Returns whether the metadata marks the blob as a directory
func hasDirMetadata(metadata map[string]*string) bool {
// Note that metadata with hdi_isfolder = true seems to be a
// defacto standard for marking blobs as directories.
// Note also that the metadata hasn't been normalised to lower case yet
for k, v := range metadata {
if v != nil && strings.EqualFold(k, dirMetaKey) && *v == dirMetaValue {
return true
// Note that metadata with hdi_isfolder = true seems to be a
// defacto standard for marking blobs as directories.
// Note also that the metadata hasn't been normalised to lower case yet
for k, v := range metadata {
if v != nil && strings.EqualFold(k, dirMetaKey) && *v == dirMetaValue {
return true
}
}
}
return false
}
// Returns whether props are for a directory on a storage account with
// a hierarchical namespace
//
// These ignore a trailing "/" on the name so a file can be returned
// when looking for a directory.
func isHNSDirectory(props *blob.GetPropertiesResponse) bool {
return props.ContentLength != nil && *props.ContentLength == 0 && hasDirMetadata(props.Metadata)
}
// listFn is called from list to handle an object
type listFn func(remote string, object *container.BlobItem, isDirectory bool) error
@@ -1221,25 +1188,27 @@ func (f *Fs) list(ctx context.Context, containerName, directory, prefix string,
delimiter = "/"
}
opts := &container.ListBlobsHierarchyOptions{
// Copy, Metadata, Snapshots, UncommittedBlobs, Deleted, Tags, Versions, LegalHold, ImmutabilityPolicy, DeletedWithVersions bool
Include: container.ListBlobsInclude{
Copy: false,
Metadata: true,
Snapshots: false,
UncommittedBlobs: false,
Deleted: false,
opts := &arrowlist.ListBlobsHierarchyOptions{
ListBlobsHierarchyOptions: container.ListBlobsHierarchyOptions{
// Copy, Metadata, Snapshots, UncommittedBlobs, Deleted, Tags, Versions, LegalHold, ImmutabilityPolicy, DeletedWithVersions bool
Include: container.ListBlobsInclude{
Copy: false,
Metadata: true,
Snapshots: false,
UncommittedBlobs: false,
Deleted: false,
},
Prefix: &directory,
MaxResults: &maxResults,
},
Prefix: &directory,
MaxResults: &maxResults,
}
// Request the Apache Arrow listing format. The SDK pager requests an
// Arrow IPC stream and decodes it, falling back to XML if the account
// doesn't have Arrow listing enabled. Skip the maxResults==1 probe
// (isEmpty) which doesn't benefit.
// Experimental: request the Apache Arrow listing format. The arrowlist
// pager requests an Arrow IPC stream and decodes it, falling back to XML
// if the account doesn't have Arrow listing enabled. Skip the
// maxResults==1 probe (isEmpty) which doesn't benefit.
useArrow := f.opt.UseArrowList && maxResults != 1
if useArrow {
opts.ResponseFormat = container.StorageResponseFormatArrow
opts.UseArrowFormat = to.Ptr(true)
}
var foundItems int
@@ -1254,18 +1223,15 @@ func (f *Fs) list(ctx context.Context, containerName, directory, prefix string,
if err != nil {
return err
}
if f.dirMarkers() && foundItems == 0 && directory != "" {
if f.opt.DirectoryMarkers && foundItems == 0 && directory != "" {
// Determine whether the directory exists or not by whether it has a marker
props, err := f.readMetaData(ctx, containerName, directory)
_, err := f.readMetaData(ctx, containerName, directory)
if err != nil {
if err == fs.ErrorObjectNotFound {
return fs.ErrorDirNotFound
}
return err
}
if f.hns && !isHNSDirectory(props) {
return fs.ErrorDirNotFound
}
}
return nil
}
@@ -1273,11 +1239,25 @@ func (f *Fs) list(ctx context.Context, containerName, directory, prefix string,
// listBlobsPager runs the hierarchy listing described by opts, calling fn for
// each blob and subdirectory, and returns the number of raw items seen.
// delimiter selects flat (recurse) vs hierarchical listing. If opts requests
// the Apache Arrow format and the service answers with XML (Arrow listing
// not enabled) a debug message is logged.
func (f *Fs) listBlobsPager(ctx context.Context, containerName, directory, prefix string, addContainer bool, opts *container.ListBlobsHierarchyOptions, delimiter string, fn listFn) (foundItems int, err error) {
useArrow := opts.ResponseFormat == container.StorageResponseFormatArrow
pager := f.cntSVC(containerName).NewListBlobsHierarchyPager(delimiter, opts)
// the Apache Arrow format the listing goes through the arrowlist pager
// (falling back to the SDK pager if the credentials don't support it); if the
// service then answers with XML (Arrow listing not enabled) a debug message
// is logged.
func (f *Fs) listBlobsPager(ctx context.Context, containerName, directory, prefix string, addContainer bool, opts *arrowlist.ListBlobsHierarchyOptions, delimiter string, fn listFn) (foundItems int, err error) {
useArrow := opts.UseArrowFormat != nil && *opts.UseArrowFormat
var pager *runtime.Pager[container.ListBlobsHierarchyResponse]
if useArrow {
arrowSVC, err := f.arrowCntSVC(containerName)
if err != nil {
fs.Debugf(f, "Not using Arrow listing: %v", err)
useArrow = false
} else {
pager = arrowSVC.NewListBlobsHierarchyPager(delimiter, opts)
}
}
if pager == nil {
pager = f.cntSVC(containerName).NewListBlobsHierarchyPager(delimiter, &opts.ListBlobsHierarchyOptions)
}
checkedArrow := false
for pager.More() {
var response container.ListBlobsHierarchyResponse
@@ -1293,7 +1273,7 @@ func (f *Fs) listBlobsPager(ctx context.Context, containerName, directory, prefi
// but not accelerated.
if useArrow && !checkedArrow && err == nil {
checkedArrow = true
if response.ContentType == nil || !strings.HasPrefix(*response.ContentType, arrowContentType) {
if response.ContentType == nil || !strings.HasPrefix(*response.ContentType, arrowlist.ArrowContentType) {
fs.Debugf(f, "Apache Arrow listing requested but server returned XML - Blob Listing with Apache Arrow may not be enabled on this account")
}
}
@@ -1318,13 +1298,6 @@ func (f *Fs) listBlobsPager(ctx context.Context, containerName, directory, prefi
fs.Debugf(f, "Nil name received")
continue
}
// Storage accounts with a hierarchical namespace return
// the file "dir" when listing "dir/"
if !strings.HasPrefix(*file.Name, directory) {
fs.Debugf(f, "Ignoring %q not in directory %q", *file.Name, directory)
foundItems--
continue
}
remote := f.opt.Enc.ToStandardPath(*file.Name)
if !strings.HasPrefix(remote, prefix) {
fs.Debugf(f, "Odd name received %q", remote)
@@ -1586,7 +1559,7 @@ type listContainerFn func(Name string, LastModified time.Time) error
func (f *Fs) listContainersToFn(fn listContainerFn) error {
max := int32(f.opt.ListChunkSize)
pager := f.svc.NewListContainersPager(&service.ListContainersOptions{
Include: service.ListContainersInclude{Metadata: true},
Include: service.ListContainersInclude{Metadata: true, Deleted: true},
MaxResults: &max,
})
ctx := context.Background()
@@ -1636,7 +1609,7 @@ func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, opt
// Create directory marker file and parents
func (f *Fs) createDirectoryMarker(ctx context.Context, container, dir string) error {
if !f.dirMarkers() || container == "" {
if !f.opt.DirectoryMarkers || container == "" {
return nil
}
@@ -1693,11 +1666,6 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
// mkdirParent creates the parent bucket/directory if it doesn't exist
func (f *Fs) mkdirParent(ctx context.Context, remote string) error {
if f.hns {
// Parent directories are created by the upload
container, _ := f.split(remote)
return f.makeContainer(ctx, container)
}
remote, _ = strings.CutSuffix(remote, "/")
dir := path.Dir(remote)
if dir == "/" || dir == "." {
@@ -1803,9 +1771,6 @@ func (f *Fs) deleteContainer(ctx context.Context, containerName string) error {
// Returns an error if it isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
container, directory := f.split(dir)
if f.hns && container != "" && directory != "" {
return f.rmdirHNS(ctx, container, directory)
}
// Remove directory marker file
if f.opt.DirectoryMarkers && container != "" && directory != "" {
o := &Object{
@@ -1828,39 +1793,6 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return f.deleteContainer(ctx, container)
}
// rmdirHNS removes the directory on a storage account with a
// hierarchical namespace
//
// The directory must be deleted without a trailing "/" and the server
// refuses to delete it if it isn't empty.
func (f *Fs) rmdirHNS(ctx context.Context, container, directory string) error {
directory, _ = strings.CutSuffix(directory, "/")
// Check it is a directory as deleting a file would succeed
props, err := f.readMetaData(ctx, container, directory)
if err == fs.ErrorObjectNotFound {
return fs.ErrorDirNotFound
} else if err != nil {
return err
}
if !isHNSDirectory(props) {
return fs.ErrorIsFile
}
blb := f.getBlobSVC(container, directory)
err = f.pacer.Call(func() (bool, error) {
_, err := blb.Delete(ctx, nil)
return f.shouldRetry(ctx, err)
})
if storageErr, ok := errors.AsType[*azcore.ResponseError](err); ok {
if storageErr.ErrorCode == "DirectoryIsNotEmpty" {
return fs.ErrorDirectoryNotEmpty
}
if storageErr.StatusCode == http.StatusNotFound {
return fs.ErrorDirNotFound
}
}
return err
}
// Precision of the remote
func (f *Fs) Precision() time.Duration {
return time.Nanosecond
@@ -2699,6 +2631,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
return downloadResponse.Body, nil
}
// Converts a string into a pointer to a string
func pString(s string) *string {
return &s
}
// readSeekCloser joins an io.Reader and an io.Seeker and provides a no-op io.Closer
type readSeekCloser struct {
io.Reader
@@ -2853,7 +2790,8 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
// isInvalidBlockOrBlob looks for the InvalidBlockOrBlob error in err
// returning true if it is found
func isInvalidBlockOrBlob(err error) bool {
if storageErr, ok := errors.AsType[*azcore.ResponseError](err); ok {
var storageErr *azcore.ResponseError
if errors.As(err, &storageErr) {
return storageErr.ErrorCode == string(bloberror.InvalidBlobOrBlock)
}
return false
@@ -3225,7 +3163,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
// Start with default content-type based on source
ui.httpHeaders = blob.HTTPHeaders{
BlobContentType: new(fs.MimeType(ctx, src)),
BlobContentType: pString(fs.MimeType(ctx, src)),
}
// Apply mapped metadata/headers/tags if requested
@@ -3271,15 +3209,15 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
o.tags[parts[0]] = parts[1]
}
case "cache-control":
ui.httpHeaders.BlobCacheControl = new(value)
ui.httpHeaders.BlobCacheControl = pString(value)
case "content-disposition":
ui.httpHeaders.BlobContentDisposition = new(value)
ui.httpHeaders.BlobContentDisposition = pString(value)
case "content-encoding":
ui.httpHeaders.BlobContentEncoding = new(value)
ui.httpHeaders.BlobContentEncoding = pString(value)
case "content-language":
ui.httpHeaders.BlobContentLanguage = new(value)
ui.httpHeaders.BlobContentLanguage = pString(value)
case "content-type":
ui.httpHeaders.BlobContentType = new(value)
ui.httpHeaders.BlobContentType = pString(value)
}
}
+1 -13
View File
@@ -1,4 +1,4 @@
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
package azureblob
@@ -325,14 +325,6 @@ func assertHeadersAndMetadata(t *testing.T, props *blob.GetPropertiesResponse, w
}
// helper to read blob tags for an object
// skipIfHNS skips the test if the storage account has a hierarchical
// namespace as these don't support blob tags
func skipIfHNS(t *testing.T, f *Fs) {
if f.hns {
t.Skip("blob tags not supported with hierarchical namespace")
}
}
func getTagsMap(ctx context.Context, t *testing.T, o fs.Object) map[string]string {
ao := o.(*Object)
blb := ao.getBlobSVC()
@@ -418,7 +410,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Singlepart upload
t.Run("PutSinglepartTags", func(t *testing.T) {
skipIfHNS(t, f)
contents := random.String(int(f.opt.ChunkSize / 2))
item := fstest.NewItem("tags-single.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
@@ -434,7 +425,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Multipart upload
t.Run("PutMultipartTags", func(t *testing.T) {
skipIfHNS(t, f)
contents := random.String(int(f.opt.ChunkSize + 2048))
item := fstest.NewItem("tags-multipart.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
@@ -622,7 +612,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Singlepart copy
t.Run("CopySinglepartTags", func(t *testing.T) {
skipIfHNS(t, f)
// create small source
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("tags-copy-single-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
@@ -648,7 +637,6 @@ func (f *Fs) testMetadataPaths(t *testing.T) {
// Tags: Multipart copy
t.Run("CopyMultipartTags", func(t *testing.T) {
skipIfHNS(t, f)
// create large source to force multipart
contents := random.String(int(f.opt.CopyCutoff + 4096))
srcItem := fstest.NewItem("tags-copy-multi-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
+1 -1
View File
@@ -1,6 +1,6 @@
// Test AzureBlob filesystem interface
//go:build !plan9 && !js
//go:build !plan9 && !solaris && !js
package azureblob
+1 -1
View File
@@ -1,7 +1,7 @@
// Build for azureblob for unsupported platforms to stop go complaining
// about "no buildable Go source files "
//go:build plan9 || js
//go:build plan9 || solaris || js
// Package azureblob provides an interface to the Microsoft Azure blob object storage system
package azureblob
+13 -8
View File
@@ -79,7 +79,7 @@ This is required and is the name of the share to access.
Help: `Upload chunk size.
Note that this is stored in memory and there may be up to
"--transfers" * "--azurefiles-upload-concurrency" chunks stored at once
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.`,
Default: defaultChunkSize,
Advanced: true,
@@ -95,7 +95,7 @@ links and these uploads do not fully utilize your bandwidth, then
increasing this may help to speed up the transfers.
Note that chunks are stored in memory and there may be up to
"--transfers" * "--azurefiles-upload-concurrency" chunks stored at once
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
in memory.`,
Default: 16,
Advanced: true,
@@ -650,6 +650,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
return resp.Body, nil
}
// Returns a pointer to t - useful for returning pointers to constants
func ptr[T any](t T) *T {
return &t
}
var warnStreamUpload sync.Once
// Update the object with the contents of the io.Reader, modTime, size and MD5 hash
@@ -813,8 +818,8 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
return nil, fmt.Errorf("Move: mkParentDir failed: %w", err)
}
opt := file.RenameOptions{
IgnoreReadOnly: new(true),
ReplaceIfExists: new(true),
IgnoreReadOnly: ptr(true),
ReplaceIfExists: ptr(true),
}
dstAbsPath := f.absPath(remote)
fc := srcObj.fileClient()
@@ -859,8 +864,8 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
}
opt := directory.RenameOptions{
IgnoreReadOnly: new(false),
ReplaceIfExists: new(false),
IgnoreReadOnly: ptr(false),
ReplaceIfExists: ptr(false),
}
dstAbsPath := dstFs.absPath(dstRemote)
dirClient := srcFs.dirClient(srcRemote)
@@ -899,8 +904,8 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
ChangeTime: file.SourceCopyFileChangeTime{},
CreationTime: file.SourceCopyFileCreationTime{},
LastWriteTime: file.SourceCopyFileLastWriteTime{},
PermissionCopyMode: new(file.PermissionCopyModeTypeSource),
IgnoreReadOnly: new(true),
PermissionCopyMode: ptr(file.PermissionCopyModeTypeSource),
IgnoreReadOnly: ptr(true),
},
}
srcURL := srcObj.fileClient().URL()
+3 -27
View File
@@ -191,11 +191,6 @@ If the custom endpoint rewrites the requests for authentication,
e.g., in Cloudflare Workers, this header needs to be handled properly.
Leave blank if you want to use the endpoint provided by Backblaze.
Rclone will not follow a redirect from this endpoint which downgrades
HTTPS to HTTP, and will not send the "Authorization" header or the
SSE-C key headers once a redirect has left the host and port of this
endpoint.
The URL provided here SHOULD have the protocol and SHOULD NOT have
a trailing slash or specify the /file/bucket subpath as rclone will
request files with "{download_url}/file/{bucket_name}/{path}".
@@ -277,7 +272,7 @@ See: [rclone backend lifecycle](#lifecycle) for setting lifecycles after bucket
Name: "sse_customer_key",
Help: `To use SSE-C, you may provide the secret encryption key encoded in a UTF-8 compatible string to encrypt/decrypt your data
Alternatively you can provide --b2-sse-customer-key-base64.`,
Alternatively you can provide --sse-customer-key-base64.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -288,7 +283,7 @@ Alternatively you can provide --b2-sse-customer-key-base64.`,
Name: "sse_customer_key_base64",
Help: `To use SSE-C, you may provide the secret encryption key encoded in Base64 format to encrypt/decrypt your data
Alternatively you can provide --b2-sse-customer-key.`,
Alternatively you can provide --sse-customer-key.`,
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -534,25 +529,6 @@ func (f *Fs) setRoot(root string) {
f.rootBucket, f.rootDirectory = bucket.Split(f.root)
}
// redirectSecretHeaders are the request headers carrying secrets that
// must not be forwarded when a redirect leaves the original host.
var redirectSecretHeaders = []string{
"Authorization",
sseAlgorithmHeader,
sseKeyHeader,
sseMd5Header,
}
// newClient makes the http client used for all B2 requests.
//
// The download_url option sends the requests to a host the user does
// not control so it strips headers on cross host redirects.
func newClient(ctx context.Context) *http.Client {
client := fshttp.NewClient(ctx)
client.CheckRedirect = rest.StripHeadersOnCrossHostRedirectFn(redirectSecretHeaders...)
return client
}
// NewFs constructs an Fs from the path, bucket:path
func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
// Parse config into Options struct
@@ -605,7 +581,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
name: name,
opt: *opt,
ci: ci,
srv: rest.NewClient(newClient(ctx)).SetErrorHandler(errorHandler),
srv: rest.NewClient(fshttp.NewClient(ctx)).SetErrorHandler(errorHandler),
cache: bucket.NewCache(),
_bucketID: make(map[string]string, 1),
_bucketType: make(map[string]string, 1),
-59
View File
@@ -4,8 +4,6 @@ import (
"context"
"crypto/sha1"
"fmt"
"net/http"
"net/http/httptest"
"path"
"sort"
"strings"
@@ -619,60 +617,3 @@ func (f *Fs) InternalTest(t *testing.T) {
}
var _ fstests.InternalTester = (*Fs)(nil)
// redirectTestHeaders is a deliberately literal copy of
// redirectSecretHeaders so that a header dropped from the production
// list fails TestRedirectSecretHeaders rather than silently losing
// test coverage.
var redirectTestHeaders = map[string]string{
"Authorization": "secret-token",
"X-Bz-Server-Side-Encryption-Customer-Algorithm": "AES256",
"X-Bz-Server-Side-Encryption-Customer-Key": "secret-key",
"X-Bz-Server-Side-Encryption-Customer-Key-Md5": "secret-md5",
}
func TestRedirectSecretHeaders(t *testing.T) {
want := make([]string, 0, len(redirectTestHeaders))
for header := range redirectTestHeaders {
want = append(want, header)
}
sort.Strings(want)
got := append([]string(nil), redirectSecretHeaders...)
sort.Strings(got)
assert.Equal(t, want, got)
}
// redirectTestRequest makes a GET request to rawURL carrying the
// secret headers plus a harmless marker header.
func redirectTestRequest(t *testing.T, rawURL string) *http.Request {
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, rawURL, nil)
require.NoError(t, err)
for header, value := range redirectTestHeaders {
req.Header.Set(header, value)
}
req.Header.Set("X-Bz-Test-Mode", "marker")
return req
}
func TestClientStripsSecretHeadersOnCrossHostRedirect(t *testing.T) {
client := newClient(context.Background())
redirectServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
for header := range redirectTestHeaders {
assert.Empty(t, r.Header.Get(header), "%s should have been stripped", header)
}
assert.Equal(t, "marker", r.Header.Get("X-Bz-Test-Mode"))
w.WriteHeader(http.StatusOK)
}))
defer redirectServer.Close()
initialServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, redirectServer.URL, http.StatusTemporaryRedirect)
}))
defer initialServer.Close()
resp, err := client.Do(redirectTestRequest(t, initialServer.URL))
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
assert.NoError(t, resp.Body.Close())
}
+10 -11
View File
@@ -27,7 +27,7 @@ import (
"sync/atomic"
"time"
"github.com/golang-jwt/jwt/v5"
"github.com/golang-jwt/jwt/v4"
"github.com/rclone/rclone/backend/box/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
@@ -77,10 +77,7 @@ var (
)
type boxCustomClaims struct {
jwt.RegisteredClaims
// Audience overrides RegisteredClaims.Audience so it is sent as
// a single string rather than an array of strings
Audience string `json:"aud,omitempty"`
jwt.StandardClaims
BoxSubType string `json:"box_sub_type,omitempty"`
}
@@ -226,7 +223,7 @@ func getBoxConfig(m configmap.Mapper) (boxConfig *api.ConfigJSON, err error) {
if configFileCredentials == "" {
configFile, _ := m.Get("box_config_file")
configFileBytes, err = os.ReadFile(env.ShellExpand(configFile))
configFileBytes, err = os.ReadFile(configFile)
if err != nil {
return nil, fmt.Errorf("box: failed to read Box config: %w", err)
}
@@ -246,13 +243,15 @@ func getClaims(boxConfig *api.ConfigJSON, boxSubType string) (claims *boxCustomC
}
claims = &boxCustomClaims{
RegisteredClaims: jwt.RegisteredClaims{
ID: val,
//lint:ignore SA1019 since we need to use jwt.StandardClaims even if deprecated in jwt-go v4 until a more permanent solution is ready in time before jwt-go v5 where it is removed entirely
//nolint:staticcheck // Don't include staticcheck when running golangci-lint to avoid SA1019
StandardClaims: jwt.StandardClaims{
Id: val,
Issuer: boxConfig.BoxAppSettings.ClientID,
Subject: boxConfig.EnterpriseID,
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Second * 45)),
Audience: tokenURL,
ExpiresAt: time.Now().Add(time.Second * 45).Unix(),
},
Audience: tokenURL,
BoxSubType: boxSubType,
}
return claims, nil
@@ -1462,7 +1461,7 @@ func (f *Fs) changeNotifyRunner(ctx context.Context, notifyFunc func(string, fs.
// Item in the cache has the same or newer SequenceID than
// this event. Ignore this event, it must be old.
f.itemMetaCacheMu.Unlock()
fs.Debugf(f, "%s ignored due to old SequenceID (%d)", eventDetails, itemMeta.SequenceID)
fs.Debugf(f, "%s ignored due to old SequenceID (%q)", eventDetails, itemMeta.SequenceID)
continue
}
-118
View File
@@ -1,118 +0,0 @@
package box
import (
"crypto"
"crypto/rand"
"crypto/rsa"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/rclone/rclone/backend/box/api"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/lib/jwtutil"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// decodeJWTPart decodes one base64url encoded part of a JWT into a map
func decodeJWTPart(t *testing.T, part string) map[string]any {
data, err := base64.RawURLEncoding.DecodeString(part)
require.NoError(t, err)
var result map[string]any
require.NoError(t, json.Unmarshal(data, &result))
return result
}
// Test the JWT assertion sent to the token endpoint is in the format Box expects
func TestJWTAssertion(t *testing.T) {
privateKey, err := rsa.GenerateKey(rand.Reader, 2048)
require.NoError(t, err)
boxConfig := &api.ConfigJSON{
BoxAppSettings: api.AppSettings{
ClientID: "client-id",
ClientSecret: "client-secret",
AppAuth: api.AppAuth{
PublicKeyID: "public-key-id",
},
},
EnterpriseID: "enterprise-id",
}
var form map[string][]string
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "POST", r.Method)
assert.NoError(t, r.ParseForm())
form = r.PostForm
_, _ = w.Write([]byte(`{"access_token":"access-token","token_type":"bearer","expires_in":3600}`))
}))
defer ts.Close()
claims, err := getClaims(boxConfig, "enterprise")
require.NoError(t, err)
m := configmap.Simple{}
start := time.Now()
err = jwtutil.Config("box", "test", ts.URL, *claims, getSigningHeaders(boxConfig), getQueryParams(boxConfig), privateKey, m, ts.Client())
require.NoError(t, err)
// Check the token was stored
assert.Contains(t, m["token"], `"access_token":"access-token"`)
// Check the form parameters
assert.Equal(t, []string{"urn:ietf:params:oauth:grant-type:jwt-bearer"}, form["grant_type"])
assert.Equal(t, []string{"client-id"}, form["client_id"])
assert.Equal(t, []string{"client-secret"}, form["client_secret"])
require.Len(t, form["assertion"], 1)
parts := strings.Split(form["assertion"][0], ".")
require.Len(t, parts, 3)
// Check the signature independently of the JWT library
signature, err := base64.RawURLEncoding.DecodeString(parts[2])
require.NoError(t, err)
hash := sha256.Sum256([]byte(parts[0] + "." + parts[1]))
require.NoError(t, rsa.VerifyPKCS1v15(&privateKey.PublicKey, crypto.SHA256, hash[:], signature))
// Check the header
assert.Equal(t, map[string]any{
"alg": "RS256",
"typ": "JWT",
"kid": "public-key-id",
}, decodeJWTPart(t, parts[0]))
// Check the claims - aud must be a string, not an array
payload := decodeJWTPart(t, parts[1])
exp, ok := payload["exp"].(float64)
require.True(t, ok, "exp should be a number")
assert.InDelta(t, float64(start.Add(45*time.Second).Unix()), exp, 5)
jti, ok := payload["jti"].(string)
require.True(t, ok, "jti should be a string")
assert.Len(t, jti, 40)
delete(payload, "exp")
delete(payload, "jti")
assert.Equal(t, map[string]any{
"aud": tokenURL,
"iss": "client-id",
"sub": "enterprise-id",
"box_sub_type": "enterprise",
}, payload)
}
// Test box_config_file has ~ and environment variables expanded
func TestGetBoxConfigShellExpand(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(dir, "config.json"), []byte(`{"enterpriseID":"enterprise-id"}`), 0600))
t.Setenv("RCLONE_TEST_BOX_CONFIG_DIR", dir)
m := configmap.Simple{
"box_config_file": "$RCLONE_TEST_BOX_CONFIG_DIR/config.json",
}
boxConfig, err := getBoxConfig(m)
require.NoError(t, err)
assert.Equal(t, "enterprise-id", boxConfig.EnterpriseID)
}
+15 -23
View File
@@ -3,6 +3,7 @@
package box
import (
"bytes"
"context"
"crypto/sha1"
"encoding/base64"
@@ -19,7 +20,6 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/lib/atexit"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/rest"
)
@@ -55,15 +55,9 @@ func sha1Digest(digest []byte) string {
}
// uploadPart uploads a part in an upload session
//
// chunk must contain exactly the part's data and is read from the start
// on each attempt.
func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, totalSize int64, chunk io.ReadSeeker, wrap accounting.WrapFn, options ...fs.OpenOption) (response *api.UploadPartResponse, err error) {
hasher := sha1.New()
chunkSize, err := io.Copy(hasher, chunk)
if err != nil {
return nil, err
}
func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, totalSize int64, chunk []byte, wrap accounting.WrapFn, options ...fs.OpenOption) (response *api.UploadPartResponse, err error) {
chunkSize := int64(len(chunk))
sha1sum := sha1.Sum(chunk)
opts := rest.Opts{
Method: "PUT",
Path: "/files/upload_sessions/" + SessionID,
@@ -73,16 +67,12 @@ func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, total
ContentRange: fmt.Sprintf("bytes %d-%d/%d", offset, offset+chunkSize-1, totalSize),
Options: options,
ExtraHeaders: map[string]string{
"Digest": sha1Digest(hasher.Sum(nil)),
"Digest": sha1Digest(sha1sum[:]),
},
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
_, err = chunk.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
opts.Body = wrap(chunk)
opts.Body = wrap(bytes.NewReader(chunk))
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &response)
return shouldRetry(ctx, resp, err)
})
@@ -223,25 +213,27 @@ outer:
reqSize := min(remaining, chunkSize)
// Read the chunk into memory from the global pool, hashing
// it for the whole file hash (must be done sequentially)
rw := multipart.NewRW().Reserve(reqSize)
_, err = io.CopyN(rw, io.TeeReader(in, hash), reqSize)
// Make a block of memory
buf := make([]byte, reqSize)
// Read the chunk
_, err = io.ReadFull(in, buf)
if err != nil {
_ = rw.Close()
err = fmt.Errorf("multipart upload failed to read source: %w", err)
break outer
}
// Make the global hash (must be done sequentially)
_, _ = hash.Write(buf)
// Transfer the chunk
wg.Add(1)
o.fs.uploadToken.Get()
go func(part int, position int64) {
defer wg.Done()
defer o.fs.uploadToken.Put()
defer func() { _ = rw.Close() }()
fs.Debugf(o, "Uploading part %d/%d offset %v/%v part size %v", part+1, session.TotalParts, fs.SizeSuffix(position), fs.SizeSuffix(size), fs.SizeSuffix(chunkSize))
partResponse, err := o.uploadPart(ctx, session.ID, position, size, rw, wrap, options...)
partResponse, err := o.uploadPart(ctx, session.ID, position, size, buf, wrap, options...)
if err != nil {
err = fmt.Errorf("multipart upload failed to upload part: %w", err)
select {
+6 -30
View File
@@ -801,12 +801,9 @@ func (f *Fs) processEntries(ctx context.Context, origEntries fs.DirEntries, dirP
fs.Debugf(f, "skip orphan data chunk %q", remote)
break
}
if mainObject == nil && !f.useMeta && !isSpecial {
if mainObject == nil && !f.useMeta {
// this is the "nometa" case
// create dummy chunked object without metadata
//
// not for temporary or control chunks as they may be all
// that is left of an interrupted upload
mainObject = f.newObject(mainRemote, nil, nil)
byRemote[mainRemote] = mainObject
if !badEntry[mainRemote] {
@@ -1502,21 +1499,11 @@ func (c *chunkingReader) rollback(ctx context.Context, metaObject fs.Object) {
func (f *Fs) removeOldChunks(ctx context.Context, remote string) {
oldFsObject, err := f.NewObject(ctx, remote)
if err == nil {
removeStaleChunks(ctx, oldFsObject.(*Object), nil)
}
}
func removeStaleChunks(ctx context.Context, oldObject *Object, currentChunks []fs.Object) {
currentRemotes := make(map[string]struct{}, len(currentChunks))
for _, chunk := range currentChunks {
currentRemotes[chunk.Remote()] = struct{}{}
}
for _, chunk := range oldObject.chunks {
if _, found := currentRemotes[chunk.Remote()]; found {
continue
}
if err := chunk.Remove(ctx); err != nil {
fs.Errorf(chunk, "Failed to remove old chunk: %v", err)
oldObject := oldFsObject.(*Object)
for _, chunk := range oldObject.chunks {
if err := chunk.Remove(ctx); err != nil {
fs.Errorf(chunk, "Failed to remove old chunk: %v", err)
}
}
}
}
@@ -1697,20 +1684,12 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
// metadata format which might involve unsupported chunk types.
return nil, fmt.Errorf("can't %s this file: %w", opName, err)
}
// Snapshot destination chunks before its metadata is replaced.
var oldObject *Object
if oldFsObject, err := f.NewObject(ctx, remote); err == nil {
oldObject = oldFsObject.(*Object)
}
if !o.isComposite() {
fs.Debugf(o, "%s non-chunked object...", opName)
oResult, err := do(ctx, o.mainChunk(), remote) // chain operation to a single wrapped chunk
if err != nil {
return nil, err
}
if oldObject != nil {
removeStaleChunks(ctx, oldObject, nil)
}
return f.newObject("", oResult, nil), nil
}
@@ -1776,9 +1755,6 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
silentlyRemove(ctx, newObj)
return nil, err
}
if oldObject != nil {
removeStaleChunks(ctx, oldObject, newChunks)
}
return newObj, nil
}
+1 -122
View File
@@ -11,7 +11,6 @@ import (
"strings"
"testing"
_ "github.com/rclone/rclone/backend/memory"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/fspath"
@@ -44,7 +43,7 @@ func testPutLarge(t *testing.T, f *Fs, kilobytes int) {
type settings map[string]any
func deriveFs(ctx context.Context, t *testing.T, f fs.Fs, path string, opts settings) fs.Fs {
fsName, _, _ := strings.Cut(f.Name(), "{") // strip off hash
fsName := strings.Split(f.Name(), "{")[0] // strip off hash
configMap := configmap.Simple{}
for key, val := range opts {
configMap[key] = fmt.Sprintf("%v", val)
@@ -628,22 +627,6 @@ func testChunkNumberOverflow(t *testing.T, f *Fs) {
_ = file.Remove(ctx)
}
// testOrphanTempChunks checks that temporary chunks left behind by an
// interrupted upload are not listed and don't break the listing
func testOrphanTempChunks(t *testing.T, f *Fs) {
const dir = "orphan"
ctx := context.Background()
defer func() {
_ = operations.Purge(ctx, f.base, dir)
}()
modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
item := fstest.Item{Path: f.makeChunkName(path.Join(dir, "interrupted"), 0, "", "abc123"), ModTime: modTime}
fstests.PutTestContents(ctx, t, f.base, &item, "data", true)
fstest.CheckListingWithRoot(t, f, dir, nil, nil, f.Precision())
}
func testMetadataInput(t *testing.T, f *Fs) {
const minChunkForTest = 50
if f.opt.ChunkSize < minChunkForTest {
@@ -878,104 +861,6 @@ func testChunkerServerSideMove(t *testing.T, f *Fs) {
_ = operations.Purge(ctx, f.base, dir)
}
func testChunkerServerSideOverwrite(t *testing.T, moveFs *Fs) {
ctx := context.Background()
const dir = "serveroverwrite"
remote := fmt.Sprintf(":chunker,remote=':memory:%s',chunk_size=3B:", dir)
fsResult, err := fs.NewFs(ctx, remote)
require.NoError(t, err)
copyFs, ok := fsResult.(*Fs)
require.True(t, ok)
oldMoveChunkSize := moveFs.opt.ChunkSize
moveFs.opt.ChunkSize = fs.SizeSuffix(3)
defer func() {
moveFs.opt.ChunkSize = oldMoveChunkSize
_ = operations.Purge(ctx, copyFs.base, "")
_ = operations.Purge(ctx, moveFs.base, dir)
}()
type operation struct {
name string
fs *Fs
enabled bool
do copyMoveFn
}
operations := []operation{
{name: "Copy", fs: copyFs, enabled: copyFs.Features().Copy != nil, do: copyFs.Copy},
{name: "Move", fs: moveFs, enabled: moveFs.Features().Move != nil, do: moveFs.Move},
}
testCases := []struct {
name string
sourceContents string
targetContents string
}{
{name: "SmallOverComposite", sourceContents: "hi", targetContents: "abcdef"},
{name: "FewerChunks", sourceContents: "abcd", targetContents: "abcdefghi"},
}
transactionModes := []struct {
name string
useNoRename bool
}{
{name: "Rename"},
{name: "NoRename", useNoRename: true},
}
for _, operation := range operations {
t.Run(operation.name, func(t *testing.T) {
if !operation.enabled {
t.Skipf("wrapped remote does not support server-side %s", strings.ToLower(operation.name))
}
for _, transactionMode := range transactionModes {
t.Run(transactionMode.name, func(t *testing.T) {
f := operation.fs
if transactionMode.useNoRename && !f.useMeta {
t.Skip("Can't test norename transactions without metadata")
}
oldUseNoRename := f.useNoRename
f.useNoRename = transactionMode.useNoRename
defer func() { f.useNoRename = oldUseNoRename }()
for _, testCase := range testCases {
t.Run(testCase.name, func(t *testing.T) {
prefix := path.Join(dir, strings.ToLower(operation.name), strings.ToLower(transactionMode.name), strings.ToLower(testCase.name))
source := testPutFile(ctx, t, f, prefix+"-source", testCase.sourceContents, "source", true)
target := testPutFile(ctx, t, f, prefix+"-target", testCase.targetContents, "target", true)
targetObject := target.(*Object)
require.NotEmpty(t, targetObject.chunks)
oldChunks := append([]fs.Object(nil), targetObject.chunks...)
result, err := operation.do(ctx, source, target.Remote())
require.NoError(t, err)
reader, err := result.Open(ctx)
require.NoError(t, err)
contents, err := io.ReadAll(reader)
require.NoError(t, err)
require.NoError(t, reader.Close())
assert.Equal(t, testCase.sourceContents, string(contents))
resultObject := result.(*Object)
currentChunks := make(map[string]struct{}, len(resultObject.chunks))
for _, chunk := range resultObject.chunks {
currentChunks[chunk.Remote()] = struct{}{}
}
removed := 0
for _, chunk := range oldChunks {
if _, found := currentChunks[chunk.Remote()]; found {
continue
}
_, err := f.base.NewObject(ctx, chunk.Remote())
assert.Equal(t, fs.ErrorObjectNotFound, err)
removed++
}
assert.Positive(t, removed)
})
}
})
}
})
}
}
// Test that md5all creates metadata even for small files
func testMD5AllSlow(t *testing.T, f *Fs) {
ctx := context.Background()
@@ -1043,15 +928,9 @@ func (f *Fs) InternalTest(t *testing.T) {
t.Run("BackwardsCompatibility", func(t *testing.T) {
testBackwardsCompatibility(t, f)
})
t.Run("OrphanTempChunks", func(t *testing.T) {
testOrphanTempChunks(t, f)
})
t.Run("ChunkerServerSideMove", func(t *testing.T) {
testChunkerServerSideMove(t, f)
})
t.Run("ChunkerServerSideOverwrite", func(t *testing.T) {
testChunkerServerSideOverwrite(t, f)
})
t.Run("MD5AllSlow", func(t *testing.T) {
testMD5AllSlow(t, f)
})
+2 -2
View File
@@ -472,8 +472,8 @@ func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options .
if updateOptions, ok := option.(*api.UpdateOptions); ok {
if updateOptions.PublicID != "" {
updateObject = true
params.Overwrite = new(true)
params.Invalidate = new(true)
params.Overwrite = SDKApi.Bool(true)
params.Invalidate = SDKApi.Bool(true)
params.PublicID = updateOptions.PublicID
params.ResourceType = updateOptions.ResourceType
params.Type = SDKApi.DeliveryType(updateOptions.DeliveryType)
+5 -51
View File
@@ -31,7 +31,6 @@ import (
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/lib/multipart"
)
// Globals
@@ -564,19 +563,15 @@ type compressionResult[T sgzip.GzipMetadata | SzstdMetadata] struct {
func (f *Fs) rcat(ctx context.Context, dstFileName string, in io.ReadCloser, modTime time.Time, options []fs.OpenOption) (o fs.Object, err error) {
// cache small files in memory and do normal upload
rw := multipart.NewRW()
defer fs.CheckClose(rw, &err)
_, err = io.CopyN(rw, in, int64(f.opt.RAMCacheLimit))
if err == io.EOF {
src := object.NewStaticObjectInfo(dstFileName, modTime, rw.Size(), false, nil, f.Fs)
return f.Fs.Put(ctx, rw, src, options...)
} else if err != nil {
return nil, err
buf := make([]byte, f.opt.RAMCacheLimit)
if n, err := io.ReadFull(in, buf); err == io.EOF || err == io.ErrUnexpectedEOF {
src := object.NewStaticObjectInfo(dstFileName, modTime, int64(len(buf[:n])), false, nil, f.Fs)
return f.Fs.Put(ctx, bytes.NewBuffer(buf[:n]), src, options...)
}
// Need to include what we already read
in = &ReadCloserWrapper{
Reader: io.MultiReader(rw, in),
Reader: io.MultiReader(bytes.NewReader(buf), in),
Closer: in,
}
@@ -1085,35 +1080,6 @@ type ObjectMetadata struct {
CompressionMetadataZstd *SzstdMetadata // Metadata for Zstd compression
}
// validate checks metadata read from the wrapped remote before it is used to
// construct a compressed reader.
func (meta *ObjectMetadata) validate() error {
if meta.Mode != Gzip {
return nil
}
if meta.CompressionMetadataGzip == nil {
return errors.New("missing gzip metadata")
}
if meta.CompressionMetadataGzip.BlockSize <= 0 {
return fmt.Errorf("invalid gzip block size %d", meta.CompressionMetadataGzip.BlockSize)
}
if meta.Size != meta.CompressionMetadataGzip.Size {
return errors.New("gzip metadata size does not match object size")
}
if meta.Size < 0 {
return errors.New("invalid gzip object size")
}
blockSize := int64(meta.CompressionMetadataGzip.BlockSize)
blocks := meta.Size / blockSize
if meta.Size%blockSize != 0 {
blocks++
}
if int64(len(meta.CompressionMetadataGzip.BlockData)) < blocks {
return errors.New("gzip block data is incomplete")
}
return nil
}
// Object with external metadata
type Object struct {
fs.Object // Wraps around data object for this object
@@ -1137,9 +1103,6 @@ func readMetadata(ctx context.Context, mo fs.Object) (meta *ObjectMetadata, err
if err = jr.Decode(meta); err != nil {
return nil, err
}
if err = meta.validate(); err != nil {
return nil, err
}
return meta, nil
}
@@ -1157,15 +1120,6 @@ func (o *Object) Remove(ctx context.Context) error {
return objErr
}
// countingDiscard is an io.Writer which discards its input and counts
// the bytes written
type countingDiscard int64
func (c *countingDiscard) Write(p []byte) (int, error) {
*c += countingDiscard(len(p))
return len(p), nil
}
// ReadCloserWrapper combines a Reader and a Closer to a ReadCloser
type ReadCloserWrapper struct {
io.Reader
-89
View File
@@ -6,8 +6,6 @@ import (
"path/filepath"
"testing"
"github.com/buengese/sgzip"
_ "github.com/rclone/rclone/backend/drive"
_ "github.com/rclone/rclone/backend/local"
_ "github.com/rclone/rclone/backend/s3"
@@ -16,93 +14,6 @@ import (
"github.com/rclone/rclone/fstest/fstests"
)
func TestObjectMetadataValidate(t *testing.T) {
tests := []struct {
name string
meta ObjectMetadata
want bool
}{
{
name: "nil gzip metadata",
meta: ObjectMetadata{Mode: Gzip},
want: true,
},
{
name: "zero block size",
meta: ObjectMetadata{
Mode: Gzip,
Size: 1024,
CompressionMetadataGzip: &sgzip.GzipMetadata{
BlockSize: 0,
Size: 1024,
BlockData: []uint32{0},
},
},
want: true,
},
{
name: "negative block size",
meta: ObjectMetadata{
Mode: Gzip,
Size: 1024,
CompressionMetadataGzip: &sgzip.GzipMetadata{
BlockSize: -1,
Size: 1024,
BlockData: []uint32{0},
},
},
want: true,
},
{
name: "mismatched sizes",
meta: ObjectMetadata{
Mode: Gzip,
Size: 1024,
CompressionMetadataGzip: &sgzip.GzipMetadata{
BlockSize: 512,
Size: 2048,
BlockData: []uint32{0, 1},
},
},
want: true,
},
{
name: "short block data",
meta: ObjectMetadata{
Mode: Gzip,
Size: 1025,
CompressionMetadataGzip: &sgzip.GzipMetadata{
BlockSize: 512,
Size: 1025,
BlockData: []uint32{0, 1},
},
},
want: true,
},
{
name: "valid",
meta: ObjectMetadata{
Mode: Gzip,
Size: 1025,
CompressionMetadataGzip: &sgzip.GzipMetadata{
BlockSize: 512,
Size: 1025,
BlockData: []uint32{0, 1, 2},
},
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
err := test.meta.validate()
if (err != nil) != test.want {
t.Fatalf("validate() error = %v, want error: %v", err, test.want)
}
})
}
}
var defaultOpt = fstests.Opt{
RemoteName: "TestCompress:",
NilObject: (*Object)(nil),
+3 -2
View File
@@ -2,6 +2,7 @@ package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
@@ -22,7 +23,7 @@ type gzipModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (g *gzipModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
var b countingDiscard
var b bytes.Buffer
var n int64
w, err := sgzip.NewWriterLevel(&b, sgzip.DefaultCompression)
if err != nil {
@@ -36,7 +37,7 @@ func (g *gzipModeHandler) isCompressible(r io.Reader, compressionMode int) (bool
if err != nil {
return false, err
}
ratio := float64(n) / float64(b)
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
-137
View File
@@ -1,137 +0,0 @@
package compress
import (
"bytes"
"context"
"crypto/rand"
"io"
"testing"
"time"
_ "github.com/rclone/rclone/backend/memory"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// putRecorder wraps an fs.Fs and records what rcat hands to Put and
// PutStream. It reads each body twice when it is seekable to model a
// backend retrying an upload.
type putRecorder struct {
fs.Fs
t *testing.T
stream bool // expose PutStream from the wrapped Fs
size int64
seekable bool
body []byte
streamed bool
}
func (r *putRecorder) Features() *fs.Features {
features := *r.Fs.Features()
features.PutStream = nil
if r.stream {
features.PutStream = r.PutStream
}
return &features
}
func (r *putRecorder) record(in io.Reader, src fs.ObjectInfo) []byte {
r.size = src.Size()
body, err := io.ReadAll(in)
require.NoError(r.t, err)
if seeker, ok := in.(io.Seeker); ok {
r.seekable = true
_, err = seeker.Seek(0, io.SeekStart)
require.NoError(r.t, err)
again, err := io.ReadAll(in)
require.NoError(r.t, err)
assert.Equal(r.t, body, again, "retried body should be identical")
}
r.body = body
return body
}
func (r *putRecorder) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
body := r.record(in, src)
return r.Fs.Put(ctx, bytes.NewReader(body), object.NewStaticObjectInfo(src.Remote(), src.ModTime(ctx), int64(len(body)), true, nil, r.Fs), options...)
}
func (r *putRecorder) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
r.streamed = true
return r.Put(ctx, in, src, options...)
}
// TestRcat checks the three upload paths rcat takes: small files buffered
// in memory and uploaded with a known size and a seekable body, large
// files streamed when the wrapped remote supports it, and large files
// spooled to a temporary file when it doesn't. Pool pages taken for the
// buffer must be returned on every path.
func TestRcat(t *testing.T) {
ctx := context.Background()
const limit = 1024
for _, test := range []struct {
name string
size int
stream bool
wantSize int64
}{
{name: "Small", size: limit / 2, stream: true, wantSize: limit / 2},
{name: "Empty", size: 0, stream: true, wantSize: 0},
{name: "ExactLimit", size: limit, stream: true, wantSize: -1},
{name: "LargeStream", size: 3 * limit, stream: true, wantSize: -1},
{name: "LargeSpool", size: 3 * limit, stream: false, wantSize: 3 * limit},
} {
t.Run(test.name, func(t *testing.T) {
remote, err := fs.NewFs(ctx, ":memory:rcat-"+test.name)
require.NoError(t, err)
rec := &putRecorder{Fs: remote, t: t, stream: test.stream}
f := &Fs{Fs: rec}
f.opt.RAMCacheLimit = limit
content := []byte(random.String(test.size))
inUse := pool.Global().InUse()
o, err := f.rcat(ctx, "file", io.NopCloser(bytes.NewReader(content)), time.Now(), nil)
require.NoError(t, err)
assert.Equal(t, inUse, pool.Global().InUse(), "pool pages not returned")
assert.Equal(t, test.wantSize, rec.size)
assert.Equal(t, content, rec.body)
assert.Equal(t, test.stream && test.size >= limit, rec.streamed)
if test.size < limit {
assert.True(t, rec.seekable, "small file body should be seekable")
}
rc, err := o.Open(ctx)
require.NoError(t, err)
got, err := io.ReadAll(rc)
require.NoError(t, err)
require.NoError(t, rc.Close())
assert.Equal(t, content, got)
})
}
}
// TestIsCompressible checks the compressibility heuristic on data at
// both ends of the scale for every compression mode which has one.
func TestIsCompressible(t *testing.T) {
compressible := bytes.Repeat([]byte("compress me "), 4096)
incompressible := make([]byte, len(compressible))
_, err := io.ReadFull(rand.Reader, incompressible)
require.NoError(t, err)
for name, handler := range map[string]compressionModeHandler{
"gzip": &gzipModeHandler{},
"zstd": &zstdModeHandler{},
} {
t.Run(name, func(t *testing.T) {
ok, err := handler.isCompressible(bytes.NewReader(compressible), 0)
require.NoError(t, err)
assert.True(t, ok)
ok, err = handler.isCompressible(bytes.NewReader(incompressible), 0)
require.NoError(t, err)
assert.False(t, ok)
})
}
}
+3 -2
View File
@@ -2,6 +2,7 @@ package compress
import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
@@ -22,7 +23,7 @@ type zstdModeHandler struct{}
// isCompressible checks the compression ratio of the provided data and returns true if the ratio exceeds
// the configured threshold
func (z *zstdModeHandler) isCompressible(r io.Reader, compressionMode int) (bool, error) {
var b countingDiscard
var b bytes.Buffer
var n int64
w, err := NewWriterSzstd(&b, zstd.WithEncoderLevel(zstd.SpeedDefault))
if err != nil {
@@ -36,7 +37,7 @@ func (z *zstdModeHandler) isCompressible(r io.Reader, compressionMode int) (bool
if err != nil {
return false, err
}
ratio := float64(n) / float64(b)
ratio := float64(n) / float64(b.Len())
return ratio > minCompressionRatio, nil
}
+5 -5
View File
@@ -118,8 +118,8 @@ func (mode NameEncryptionMode) String() (out string) {
return out
}
// FileNameEncoding is the interface for the encoding methods dealing with encrypted file names
type FileNameEncoding interface {
// fileNameEncoding are the encoding methods dealing with encrypted file names
type fileNameEncoding interface {
EncodeToString(src []byte) string
DecodeString(s string) ([]byte, error)
}
@@ -154,7 +154,7 @@ func (caseInsensitiveBase32Encoding) DecodeString(s string) ([]byte, error) {
}
// NewNameEncoding creates a NameEncoding from a string
func NewNameEncoding(s string) (enc FileNameEncoding, err error) {
func NewNameEncoding(s string) (enc fileNameEncoding, err error) {
s = strings.ToLower(s)
switch s {
case "base32":
@@ -176,7 +176,7 @@ type Cipher struct {
nameTweak [nameCipherBlockSize]byte // used to tweak the name crypto
block gocipher.Block
mode NameEncryptionMode
fileNameEnc FileNameEncoding
fileNameEnc fileNameEncoding
buffers sync.Pool // encrypt/decrypt buffers
cryptoRand io.Reader // read crypto random numbers from here
dirNameEncrypt bool
@@ -185,7 +185,7 @@ type Cipher struct {
}
// newCipher initialises the cipher. If salt is "" then it uses a built in salt val
func newCipher(mode NameEncryptionMode, password, salt string, dirNameEncrypt bool, enc FileNameEncoding) (*Cipher, error) {
func newCipher(mode NameEncryptionMode, password, salt string, dirNameEncrypt bool, enc fileNameEncoding) (*Cipher, error) {
c := &Cipher{
mode: mode,
fileNameEnc: enc,
+1 -1
View File
@@ -1197,7 +1197,7 @@ func (o *ObjectInfo) Hash(ctx context.Context, hash hash.Type) (string, error) {
return "", nil
}
// if this is wrapping a local object then we work out the hash
if srcObj != nil && srcObj.Fs().Features().IsLocal {
if srcObj.Fs().Features().IsLocal {
// Read the data and encrypt it to calculate the hash
fs.Debugf(o, "Computing %v hash of encrypted source", hash)
return o.f.computeHashWithNonce(ctx, o.nonce, srcObj, hash)
-10
View File
@@ -93,16 +93,6 @@ func testObjectInfo(t *testing.T, f *Fs, wrap bool) {
assert.Equal(t, fmt.Sprintf("%x", wantHash), gotHash)
}
// Test ObjectInfo.Hash doesn't panic when the source is an
// fs.OverrideRemote which wraps an fs.ObjectInfo that isn't an fs.Object
func TestObjectInfoHashOverrideRemoteNotObject(t *testing.T) {
src := object.NewStaticObjectInfo("file", time.Now(), 3, true, nil, nil)
oi := (&Fs{}).newObjectInfo(fs.NewOverrideRemote(src, "new_remote"), nonce{})
gotHash, err := oi.Hash(context.Background(), hash.MD5)
require.NoError(t, err)
assert.Equal(t, "", gotHash)
}
func testComputeHash(t *testing.T, f *Fs) {
var (
contents = random.String(100)
-259
View File
@@ -1,259 +0,0 @@
// Package api has type definitions for dosya.dev
package api
import "fmt"
// Response is the base response wrapper
type Response struct {
OK bool `json:"ok"`
Error string `json:"error,omitempty"`
}
// Error is an error response from the API.
//
// StatusCode and Status are filled in from the HTTP response. The API
// sets ErrorCode on the errors a client is expected to act on rather
// than just report, e.g. "concurrent_upload_limit".
type Error struct {
StatusCode int `json:"-"`
Status string `json:"-"`
OK bool `json:"ok"`
Message string `json:"error"`
ErrorCode string `json:"error_code,omitempty"`
MaxConcurrentUploads int `json:"max_concurrent_uploads,omitempty"`
}
// Error satisfies the error interface
func (e *Error) Error() string {
out := fmt.Sprintf("HTTP error %d (%s)", e.StatusCode, e.Status)
if e.Message != "" {
out += ": " + e.Message
}
if e.ErrorCode != "" {
out += " (" + e.ErrorCode + ")"
}
return out
}
// FileItem represents a file from the dosya API
type FileItem struct {
ID string `json:"id"`
Name string `json:"name"`
SizeBytes int64 `json:"size_bytes"`
MimeType string `json:"mime_type"`
Extension string `json:"extension"`
Region string `json:"region"`
CreatedAt float64 `json:"created_at"`
UpdatedAt float64 `json:"updated_at"`
FolderID *string `json:"folder_id"`
IsSynced int `json:"is_synced"`
// ContentHash is the lowercase-hex SHA-256 of the file's bytes, or "" when
// the server has none (e.g. a file uploaded as an R2 multipart, whose parts
// are never re-read to hash the whole object - the same limitation S3 has).
ContentHash string `json:"content_hash"`
}
// FolderItem represents a folder from the dosya API
type FolderItem struct {
ID string `json:"id"`
Name string `json:"name"`
ParentID *string `json:"parent_id"`
CreatedAt float64 `json:"created_at"`
FileCount int `json:"file_count"`
IsSynced int `json:"is_synced"`
}
// ListResponse is the response from GET /api/files
type ListResponse struct {
Response
Files []FileItem `json:"files"`
Folders []FolderItem `json:"folders"`
Pagination Pagination `json:"pagination"`
}
// Pagination describes which page of files a ListResponse holds
type Pagination struct {
Page int `json:"page"`
PerPage int `json:"per_page"`
TotalFiles int `json:"total_files"`
TotalPages int `json:"total_pages"`
}
// CreateFolderRequest is the request body for POST /api/folders
type CreateFolderRequest struct {
WorkspaceID string `json:"workspace_id"`
ParentID *string `json:"parent_id"`
Name string `json:"name"`
}
// CreateFolderResponse is the response from POST /api/folders
type CreateFolderResponse struct {
Response
Folder struct {
ID string `json:"id"`
Name string `json:"name"`
ParentID *string `json:"parent_id"`
Workspace string `json:"workspace_id"`
} `json:"folder"`
}
// DeleteFolderResponse is the response from DELETE /api/folders/:id
type DeleteFolderResponse struct {
Response
Permanent bool `json:"permanent"`
Complete bool `json:"complete"`
Remaining int `json:"remaining"`
FilesAffected int `json:"files_affected"`
FoldersRemoved int `json:"folders_removed"`
}
// RenameFolderRequest is the request body for PUT /api/folders/:id/rename
type RenameFolderRequest struct {
Name string `json:"name"`
}
// MoveFolderRequest is the request body for PUT /api/folders/:id/move
type MoveFolderRequest struct {
ParentID *string `json:"parent_id"`
Name string `json:"name,omitempty"`
}
// MoveFolderResponse is the response from PUT /api/folders/:id/move
type MoveFolderResponse struct {
Response
Name string `json:"name"`
}
// UploadInitRequest is the request body for POST /api/upload/init
type UploadInitRequest struct {
WorkspaceID string `json:"workspace_id"`
FileName string `json:"file_name"`
FileSize int64 `json:"file_size"`
MimeType string `json:"mime_type"`
Region string `json:"region,omitempty"`
FolderID *string `json:"folder_id"`
FileID *string `json:"file_id,omitempty"`
}
// UploadInitResponse is the response from POST /api/upload/init
type UploadInitResponse struct {
Response
SessionID string `json:"session_id"`
UploadURL string `json:"upload_url"`
WorkspaceID string `json:"workspace_id"`
FileName string `json:"file_name"`
FileSize int64 `json:"file_size"`
MimeType string `json:"mime_type"`
Extension string `json:"extension"`
Region string `json:"region"`
Resumable *struct {
PartSize int64 `json:"part_size"`
TotalParts int `json:"total_parts"`
PartUploadURL string `json:"part_upload_url"`
CompleteURL string `json:"complete_url"`
StatusURL string `json:"status_url"`
} `json:"resumable"`
}
// UploadCompleteResponse is the response from PUT /api/upload/:sessionId or POST /api/upload/:sessionId/complete
type UploadCompleteResponse struct {
Response
File struct {
ID string `json:"id"`
Name string `json:"name"`
SizeBytes int64 `json:"size_bytes"`
MimeType string `json:"mime_type"`
Extension string `json:"extension"`
Region string `json:"region"`
Version int `json:"version"`
CreatedAt float64 `json:"created_at"`
// SHA-256 of the uploaded bytes for a single-shot (small-file) upload;
// "" for a multipart upload, which the server does not hash.
ContentHash string `json:"content_hash"`
} `json:"file"`
}
// UploadPartResponse is the response from PUT /api/upload/:sessionId/part/:partNumber
type UploadPartResponse struct {
Response
PartNumber int `json:"part_number"`
ETag string `json:"etag"`
BytesUploaded int64 `json:"bytes_uploaded"`
PartsCompleted int `json:"parts_completed"`
TotalParts int `json:"total_parts"`
}
// DeleteFileResponse is the response from DELETE /api/files/:id
type DeleteFileResponse struct {
Response
Permanent bool `json:"permanent"`
}
// RenameFileRequest is the request body for PUT /api/files/:id/rename
type RenameFileRequest struct {
Name string `json:"name"`
}
// RenameFileResponse is the response from PUT /api/files/:id/rename
type RenameFileResponse struct {
Response
Name string `json:"name"`
}
// MoveFileRequest is the request body for PUT /api/files/:id/move
type MoveFileRequest struct {
FolderID *string `json:"folder_id"`
Name *string `json:"name,omitempty"`
}
// CopyFileRequest is the request body for POST /api/files/:id/copy
type CopyFileRequest struct {
FolderID *string `json:"folder_id"`
Name string `json:"name,omitempty"`
}
// CopyFileResponse is the response from POST /api/files/:id/copy
type CopyFileResponse struct {
Response
FileID string `json:"file_id"`
Name string `json:"name"`
}
// ShareLinkRequest is the request body for POST /api/files/:id/share
type ShareLinkRequest struct {
ExpiresInDays *int `json:"expires_in_days,omitempty"`
}
// ShareLinkResponse is the response from POST /api/files/:id/share
type ShareLinkResponse struct {
Response
Link struct {
ID string `json:"id"`
Token string `json:"token"`
URL string `json:"url"`
ExpiresAt *int64 `json:"expires_at"`
CreatedAt float64 `json:"created_at"`
} `json:"link"`
}
// Workspace is one entry of the workspace list
type Workspace struct {
ID string `json:"id"`
Name string `json:"name"`
}
// WorkspaceListResponse is the response from GET /api/workspaces
type WorkspaceListResponse struct {
Response
Workspaces []Workspace `json:"workspaces"`
}
// WorkspaceInfoResponse is the response from GET /api/workspaces/:id
type WorkspaceInfoResponse struct {
Response
Storage struct {
Used int64 `json:"used"`
Total int64 `json:"total"`
Free int64 `json:"free"`
} `json:"storage"`
}
-772
View File
@@ -1,772 +0,0 @@
// Package dosya provides an interface to the dosya.dev cloud storage system.
package dosya
import (
"context"
"errors"
"fmt"
"io"
"strings"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/random"
"github.com/rclone/rclone/lib/rest"
)
const (
rootID = "root"
defaultBaseURL = "https://api.dosya.dev"
minSleep = 100 * time.Millisecond
maxSleep = 2 * time.Second
decayConstant = 2
attackConstant = 0
)
func init() {
fs.Register(&fs.RegInfo{
Name: "dosya",
Description: "dosya.dev",
NewFs: NewFs,
Options: []fs.Option{{
Help: "Your API Key, get it from https://dosya.dev/profile under \"API keys\".",
Name: "api_key",
Sensitive: true,
Required: true,
}, {
Name: "workspace_id",
Help: `Your workspace ID.
Leave this blank to use the one workspace the API key can reach, which
is what a key restricted to a single workspace does. If the key can
reach several, rclone will refuse and list their IDs for you to choose
from.`,
}, {
Help: "API base URL.",
Name: "api_url",
Default: defaultBaseURL,
Advanced: true,
}, {
Name: "upload_concurrency",
Help: `Concurrency for multipart uploads.
This is the number of parts of the same file that are uploaded
concurrently. The first part is always uploaded on its own, because
the server creates the upload when it receives it.
Each part in flight is held in memory, so memory use grows with
this value times the part size the server chooses.
If you are uploading large files over high-speed links and these
uploads do not fully utilize your bandwidth, then increasing this
may help to speed up the transfers.`,
Default: 4,
Advanced: true,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
Advanced: true,
Default: (encoder.Display |
encoder.EncodeBackSlash |
encoder.EncodeSingleQuote |
encoder.EncodeDoubleQuote |
encoder.EncodeLtGt |
encoder.EncodeCtl |
encoder.EncodeLeftSpace |
encoder.EncodeRightSpace |
encoder.EncodeInvalidUtf8),
}},
})
}
// Options defines the configuration for this backend
type Options struct {
APIKey string `config:"api_key"`
WorkspaceID string `config:"workspace_id"`
APIURL string `config:"api_url"`
UploadConcurrency int `config:"upload_concurrency"`
Enc encoder.MultiEncoder `config:"encoding"`
}
// Fs represents a remote dosya.dev filesystem
type Fs struct {
name string
root string
opt Options
features *fs.Features
rest *rest.Client
pacer *fs.Pacer
dirCache *dircache.DirCache
}
// Object represents a dosya.dev file
type Object struct {
fs *Fs
remote string
file api.FileItem
}
// findWorkspace fills in the workspace ID when the config leaves it blank.
//
// The ID is not shown anywhere in the dosya.dev web interface, so requiring
// it meant asking people to find it in a developer console. A key restricted
// to one workspace can only mean that one, and an account with a single
// workspace has only one possible answer; anything else genuinely has to be
// chosen, so list the choices in the error.
func (f *Fs) findWorkspace(ctx context.Context) error {
workspaces, err := f.listWorkspaces(ctx)
if err != nil {
return err
}
switch len(workspaces) {
case 0:
return errors.New("no workspaces found for this API key")
case 1:
f.opt.WorkspaceID = workspaces[0].ID
fs.Debugf(f, "Using workspace %q (%s)", workspaces[0].Name, workspaces[0].ID)
return nil
}
var choices strings.Builder
for _, w := range workspaces {
_, _ = fmt.Fprintf(&choices, "\n %s (%s)", w.ID, w.Name)
}
return fmt.Errorf("this API key can reach %d workspaces, so workspace_id must be set to one of:%s",
len(workspaces), choices.String())
}
// FindLeaf finds a directory of name leaf in the folder with ID pathID
func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
listing, err := f.listFilesAndFolders(ctx, pathID)
if err != nil {
return "", false, err
}
for _, folder := range listing.Folders {
if f.opt.Enc.ToStandardName(folder.Name) == leaf {
return folder.ID, true, nil
}
}
return "", false, nil
}
// CreateDir makes a directory with pathID as parent and name leaf
func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string, err error) {
name := f.opt.Enc.FromStandardName(leaf)
resp, err := f.createFolder(ctx, name, pathID)
if err != nil {
return "", err
}
return resp.Folder.ID, nil
}
// Name of the remote (as passed into NewFs)
func (f *Fs) Name() string {
return f.name
}
// Root of the remote (as passed into NewFs)
func (f *Fs) Root() string {
return f.root
}
// String returns a description of the FS
func (f *Fs) String() string {
return fmt.Sprintf("dosya.dev root '%s'", f.root)
}
// Precision of the ModTimes in this Fs
func (f *Fs) Precision() time.Duration {
// Modification times are stored and returned to the second (the API
// keeps them as Unix seconds, sent on upload via the source-mtime header).
return time.Second
}
// Hashes returns the supported hash types of the filesystem
func (f *Fs) Hashes() hash.Set {
// The server computes and stores the SHA-256 of a single-shot upload's
// bytes; multipart uploads have none (see Object.Hash).
return hash.NewHashSet(hash.SHA256)
}
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
// NewFs makes a new Fs object from the path
func NewFs(ctx context.Context, name string, root string, config configmap.Mapper) (fs.Fs, error) {
opt := new(Options)
err := configstruct.Set(config, opt)
if err != nil {
return nil, err
}
root = strings.Trim(root, "/")
f := &Fs{
name: name,
root: root,
opt: *opt,
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant), pacer.AttackConstant(attackConstant))),
}
// No DuplicateFiles: an upload that names no existing file but whose
// (workspace, folder, name) is already taken becomes a new VERSION of
// that file rather than a second row, so a name is unique within a
// folder and the same upload twice returns the same file id.
f.features = (&fs.Features{
CaseInsensitive: false,
CanHaveEmptyDirectories: true,
ReadMimeType: true,
}).Fill(ctx, f)
client := fshttp.NewClient(ctx)
f.rest = rest.NewClient(client).SetRoot(strings.TrimSuffix(opt.APIURL, "/")).SetErrorHandler(errorHandler)
f.rest.SetHeader("Authorization", "Bearer "+f.opt.APIKey)
if f.opt.WorkspaceID == "" {
err = f.findWorkspace(ctx)
if err != nil {
return nil, err
}
}
f.dirCache = dircache.New(root, rootID, f)
// Find the current root
err = f.dirCache.FindRoot(ctx, false)
if err != nil {
// Assume it is a file
newRoot, remote := dircache.SplitPath(root)
tempF := *f
tempF.dirCache = dircache.New(newRoot, rootID, &tempF)
tempF.root = newRoot
// Make new Fs which is the parent
err = tempF.dirCache.FindRoot(ctx, false)
if err != nil {
// No root so return old f
return f, nil
}
_, err := tempF.NewObject(ctx, remote)
if err != nil {
if err == fs.ErrorObjectNotFound {
// File doesn't exist so return old f
return f, nil
}
return nil, err
}
f.features.Fill(ctx, &tempF)
f.dirCache = tempF.dirCache
f.root = tempF.root
return f, fs.ErrorIsFile
}
return f, nil
}
// List the objects and directories in dir into entries
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return nil, err
}
listing, err := f.listFilesAndFolders(ctx, directoryID)
if err != nil {
return nil, err
}
for _, folder := range listing.Folders {
folderName := f.opt.Enc.ToStandardName(folder.Name)
fullPath := getRemote(dir, folderName)
modTime := time.Unix(int64(folder.CreatedAt), 0)
d := fs.NewDir(fullPath, modTime).SetID(folder.ID)
entries = append(entries, d)
f.dirCache.Put(fullPath, folder.ID)
}
for _, file := range listing.Files {
fileName := f.opt.Enc.ToStandardName(file.Name)
remote := getRemote(dir, fileName)
o := &Object{
fs: f,
remote: remote,
file: file,
}
entries = append(entries, o)
}
return entries, nil
}
// NewObject finds the Object at remote
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
leaf, directoryID, err := f.dirCache.FindPath(ctx, remote, false)
if err != nil {
if err == fs.ErrorDirNotFound {
return nil, fs.ErrorObjectNotFound
}
return nil, err
}
listing, err := f.listFilesAndFolders(ctx, directoryID)
if err != nil {
return nil, err
}
for _, file := range listing.Files {
if f.opt.Enc.ToStandardName(file.Name) == leaf {
return &Object{
fs: f,
remote: remote,
file: file,
}, nil
}
}
return nil, fs.ErrorObjectNotFound
}
// Put uploads a new file
func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
existingObj, err := f.NewObject(ctx, src.Remote())
switch err {
case nil:
return existingObj, existingObj.Update(ctx, in, src, options...)
case fs.ErrorObjectNotFound:
return f.put(ctx, in, src, options...)
default:
return nil, err
}
}
// put uploads src as a new file
func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
remote := src.Remote()
_, directoryID, err := f.dirCache.FindPath(ctx, remote, true)
if err != nil {
return nil, err
}
modTime := src.ModTime(ctx)
resp, err := f.uploadFile(ctx, in, remote, src.Size(), modTime, fs.MimeType(ctx, src), directoryID, nil)
if err != nil {
return nil, err
}
return &Object{
fs: f,
remote: remote,
file: fileFromUpload(resp, modTime),
}, nil
}
// fileFromUpload builds the FileItem for the object an upload just created.
// The single-shot upload response omits updated_at, so the modification time
// is taken from what was sent (modTime) rather than defaulting to the create
// time - it is the time the server stored via the source-mtime header. The
// SHA-256 comes straight from the response (empty for a multipart upload).
func fileFromUpload(resp *api.UploadCompleteResponse, modTime time.Time) api.FileItem {
updatedAt := resp.File.CreatedAt
if !modTime.IsZero() {
updatedAt = float64(modTime.Unix())
}
return api.FileItem{
ID: resp.File.ID,
Name: resp.File.Name,
SizeBytes: resp.File.SizeBytes,
MimeType: resp.File.MimeType,
Extension: resp.File.Extension,
Region: resp.File.Region,
CreatedAt: resp.File.CreatedAt,
UpdatedAt: updatedAt,
ContentHash: resp.File.ContentHash,
}
}
// Mkdir makes the directory (container, bucket)
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
_, err := f.dirCache.FindDir(ctx, dir, true)
return err
}
// Rmdir removes the directory (container, bucket) if empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return err
}
// The server checks emptiness itself (?if_empty=1) and answers
// fs.ErrorDirectoryNotEmpty's 409 without a separate listing call.
err = f.removeFolder(ctx, directoryID, true)
if err != nil {
return err
}
f.dirCache.FlushDir(dir)
return nil
}
// Move src to this remote using server-side move operations
func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
srcObj, ok := src.(*Object)
if !ok {
fs.Debugf(src, "Can't move - not same remote type")
return nil, fs.ErrorCantMove
}
srcLeaf := f.opt.Enc.ToStandardName(srcObj.file.Name)
_, srcDirID, err := srcObj.fs.dirCache.FindPath(ctx, srcObj.remote, false)
if err != nil {
return nil, err
}
// Find destination directory
dstLeaf, dstDirID, err := f.dirCache.FindPath(ctx, remote, true)
if err != nil {
return nil, err
}
needsMove := srcDirID != dstDirID
needsRename := srcLeaf != dstLeaf
// Moving and renaming in the same request (rather than a moveFile call
// followed by a separate renameFile call) matters when both are needed:
// a two-step move-then-rename can transiently place the file's CURRENT
// name into a destination folder that already holds an unrelated file
// with that name, even though the FINAL name is free - the server only
// allows one active name per folder, so that intermediate state 409s.
if needsMove || needsRename {
newName := ""
if needsRename {
newName = f.opt.Enc.FromStandardName(dstLeaf)
}
err = f.moveFile(ctx, srcObj.file.ID, dstDirID, newName)
if err != nil {
return nil, err
}
}
// Return new object. file.Name must reflect the destination leaf, not
// the source's - a caller that reuses this object as the src of a
// SECOND Move (e.g. operations.RemoveExisting's restore-on-error path)
// recomputes srcLeaf from file.Name; a stale name there makes that
// second move see srcLeaf == dstLeaf and skip the rename entirely,
// silently leaving the object under the wrong name.
newFile := srcObj.file
newFile.Name = f.opt.Enc.FromStandardName(dstLeaf)
return &Object{
fs: f,
remote: remote,
file: newFile,
}, nil
}
// DirMove moves src, srcRemote to this remote at dstRemote
func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error {
srcFs, ok := src.(*Fs)
if !ok {
fs.Debugf(srcFs, "Can't move directory - not same remote type")
return fs.ErrorCantDirMove
}
srcID, srcDirectoryID, _, dstDirectoryID, dstLeaf, err := f.dirCache.DirMove(ctx, srcFs.dirCache, srcFs.root, srcRemote, f.root, dstRemote)
if err != nil {
return err
}
srcLeaf := srcRemote
if idx := strings.LastIndex(srcRemote, "/"); idx >= 0 {
srcLeaf = srcRemote[idx+1:]
}
needsRename := srcLeaf != dstLeaf
newName := ""
if needsRename {
newName = f.opt.Enc.FromStandardName(dstLeaf)
}
// Move folder to new parent only if parent changed. The new name goes
// in the same request, as for files in Move: a move followed by a
// separate rename can collide with an unrelated folder sharing the
// CURRENT name in the destination even though the final name is free.
if srcDirectoryID != dstDirectoryID {
renamed, err := f.moveFolder(ctx, srcID, dstDirectoryID, newName)
if err != nil {
return fmt.Errorf("couldn't move directory: %w", err)
}
if renamed {
needsRename = false
}
}
if needsRename {
err = f.renameFolder(ctx, srcID, newName)
if err != nil {
return fmt.Errorf("couldn't rename directory: %w", err)
}
}
srcFs.dirCache.FlushDir(srcRemote)
return nil
}
// Copy src to this remote using server-side copy operations
func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
srcObj, ok := src.(*Object)
if !ok {
fs.Debugf(src, "Can't copy - not same remote type")
return nil, fs.ErrorCantCopy
}
dstLeaf, dstDirID, err := f.dirCache.FindPath(ctx, remote, true)
if err != nil {
return nil, err
}
// The copy API creates a new row named after the source file when
// copying across folders, which collides with one already at the
// destination. Copy under a temporary name, then delete the old
// destination, then move the copy onto the name - so a copy that fails
// leaves the destination as it was. Deleting it up front lost it
// (prior art: backend/box).
existingObj, err := f.NewObject(ctx, remote)
switch err {
case nil:
tempRemote := remote + "-rclone-copy-" + random.String(8)
fs.Debugf(remote, "dst already exists, copying to temp name %v", tempRemote)
tempObj, err := f.Copy(ctx, src, tempRemote)
if err != nil {
return nil, err
}
fs.Debugf(tempRemote, "moving to real name %v", remote)
if err := existingObj.Remove(ctx); err != nil {
return nil, fmt.Errorf("couldn't remove existing destination file: %w", err)
}
return f.Move(ctx, tempObj, remote)
case fs.ErrorObjectNotFound:
// nothing at the destination, so copy straight onto the name
default:
return nil, err
}
dstName := f.opt.Enc.FromStandardName(dstLeaf)
resp, err := f.copyFile(ctx, srcObj.file.ID, dstDirID, dstName)
if err != nil {
return nil, fmt.Errorf("couldn't copy file: %w", err)
}
// Servers that predate the name field ignore it and name the copy after
// the source, or "Copy of <original>" in the same folder. Rename to the
// requested destination name if it differs.
if resp.Name != dstName {
err = f.renameFile(ctx, resp.FileID, dstName)
if err != nil {
return nil, fmt.Errorf("couldn't rename copied file: %w", err)
}
resp.Name = dstName
}
// A server-side copy is byte-identical to the source and the API
// preserves the source's modification time and content hash onto the
// copy, so the returned object must report the SOURCE's modtime and hash,
// not the copy's wall-clock time. Reading "now" here made the object
// rclone holds in memory disagree with the one read back from the remote
// (its Fingerprint check compares the two exactly).
return &Object{
fs: f,
remote: remote,
file: api.FileItem{
ID: resp.FileID,
Name: resp.Name,
SizeBytes: srcObj.file.SizeBytes,
MimeType: srcObj.file.MimeType,
Extension: srcObj.file.Extension,
Region: srcObj.file.Region,
CreatedAt: srcObj.file.CreatedAt,
UpdatedAt: srcObj.file.UpdatedAt,
ContentHash: srcObj.file.ContentHash,
},
}, nil
}
// Purge deletes all the files and the container
func (f *Fs) Purge(ctx context.Context, dir string) error {
directoryID, err := f.dirCache.FindDir(ctx, dir, false)
if err != nil {
return err
}
// DELETE /api/folders/:id?permanent=1 recursively deletes all contents
err = f.removeFolder(ctx, directoryID, false)
if err != nil {
return err
}
f.dirCache.FlushDir(dir)
return nil
}
// PublicLink generates a public link to the remote path
func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration, unlink bool) (string, error) {
o, err := f.NewObject(ctx, remote)
if err != nil {
return "", err
}
obj := o.(*Object)
resp, err := f.createShareLink(ctx, obj.file.ID, expire)
if err != nil {
return "", err
}
return resp.Link.URL, nil
}
// DirCacheFlush resets the directory cache - used in testing as an
// optional interface
func (f *Fs) DirCacheFlush() {
f.dirCache.ResetRoot()
}
// About gets quota information from the Fs
func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
info, err := f.getWorkspaceInfo(ctx)
if err != nil {
return nil, err
}
usage := &fs.Usage{
Used: fs.NewUsageValue(info.Storage.Used),
Total: fs.NewUsageValue(info.Storage.Total),
Free: fs.NewUsageValue(info.Storage.Free),
}
return usage, nil
}
// ------------------------------------------------------------
// Object methods
// ------------------------------------------------------------
// String returns a description of the Object
func (o *Object) String() string {
return o.remote
}
// Remote returns the remote path
func (o *Object) Remote() string {
return o.remote
}
// ModTime returns the modification date of the file
func (o *Object) ModTime(ctx context.Context) time.Time {
if o.file.UpdatedAt > 0 {
return time.Unix(int64(o.file.UpdatedAt), 0)
}
return time.Unix(int64(o.file.CreatedAt), 0)
}
// Size returns the size of the file
func (o *Object) Size() int64 {
return o.file.SizeBytes
}
// Fs returns read only access to the Fs that this object is part of
func (o *Object) Fs() fs.Info {
return o.fs
}
// Hash returns the selected checksum of the file
func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
if t != hash.SHA256 {
return "", hash.ErrUnsupported
}
// May be "" - a multipart-uploaded object carries no server-side hash,
// which rclone treats as "hash unavailable for this object" (as it does
// for S3 multipart objects).
return o.file.ContentHash, nil
}
// Storable says whether this object can be stored
func (o *Object) Storable() bool {
return true
}
// SetModTime sets the metadata on the object to set the modification date
func (o *Object) SetModTime(ctx context.Context, t time.Time) error {
return fs.ErrorCantSetModTime
}
// Open opens the file for read
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadCloser, error) {
fs.FixRangeOption(options, o.file.SizeBytes)
return o.fs.downloadFile(ctx, o.file.ID, options)
}
// Update replaces the file content
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
// Find the directory ID for this file
_, directoryID, err := o.fs.dirCache.FindPath(ctx, o.remote, false)
if err != nil {
return err
}
// Upload as new version of existing file
fileID := o.file.ID
modTime := src.ModTime(ctx)
resp, err := o.fs.uploadFile(ctx, in, o.remote, src.Size(), modTime, fs.MimeType(ctx, src), directoryID, &fileID)
if err != nil {
return err
}
// Update object metadata
o.file = fileFromUpload(resp, modTime)
return nil
}
// Remove removes this object
func (o *Object) Remove(ctx context.Context) error {
return o.fs.deleteFile(ctx, o.file.ID)
}
// MimeType of an Object if known, "" otherwise
func (o *Object) MimeType(ctx context.Context) string {
return o.file.MimeType
}
// ID returns the ID of the Object if known, or "" if not
func (o *Object) ID() string {
return o.file.ID
}
func getRemote(dir, fileName string) string {
if dir == "" {
return fileName
}
return dir + "/" + fileName
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
_ fs.Mover = (*Fs)(nil)
_ fs.DirMover = (*Fs)(nil)
_ fs.Copier = (*Fs)(nil)
_ fs.Purger = (*Fs)(nil)
_ fs.PublicLinker = (*Fs)(nil)
_ fs.Abouter = (*Fs)(nil)
_ fs.DirCacheFlusher = (*Fs)(nil)
_ dircache.DirCacher = (*Fs)(nil)
_ fs.Object = (*Object)(nil)
_ fs.MimeTyper = (*Object)(nil)
_ fs.IDer = (*Object)(nil)
)
-880
View File
@@ -1,880 +0,0 @@
package dosya
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"path"
"strconv"
"strings"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/sync/errgroup"
)
// sourceMTimeHeader is the request header the upload endpoints read to set a
// file's modification time (Unix seconds); see lib/upload/source-timestamps.ts.
const sourceMTimeHeader = "x-dosya-source-mtime"
// modTimeHeaders returns the ExtraHeaders that carry a source modification time
// on an upload, or nil when there is nothing plausible to send. The server
// clamps implausible values itself, but a zero time is never worth a header.
func modTimeHeaders(modTime time.Time) map[string]string {
if modTime.IsZero() {
return nil
}
return map[string]string{sourceMTimeHeader: strconv.FormatInt(modTime.Unix(), 10)}
}
// workspaceHintHeader names the workspace a by-id request is about. The API
// maps an id to its workspace through an index the workspace fills about a
// second after a write, so an id this client was handed moments ago can still
// miss; the hint lets the server look for it in the named workspace instead.
// It only locates: every route still checks membership against the workspace
// it resolved, and a workspace-pinned key's hint is ignored unless it matches.
const workspaceHintHeader = "X-Dosya-Workspace"
// byIDHeaders returns the ExtraHeaders every /api/files/:id and
// /api/folders/:id call carries, or nil when no workspace is configured.
func (f *Fs) byIDHeaders() map[string]string {
if f.opt.WorkspaceID == "" {
return nil
}
return map[string]string{workspaceHintHeader: f.opt.WorkspaceID}
}
// retryErrorCodes is a slice of error codes that we will retry
var retryErrorCodes = []int{
429, // Too Many Requests
500, // Internal Server Error
502, // Bad Gateway
503, // Service Unavailable
504, // Gateway Timeout
}
// shouldRetry returns a boolean as to whether this resp and err
// deserve to be retried
func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, error) {
if fserrors.ContextError(ctx, &err) {
return false, err
}
retry := fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes)
if retry {
if d, ok := retryAfter(resp); ok {
// The API answers 503 + Retry-After while a workspace is being
// moved between storage homes, and Cloudflare sends it with 429.
// The pacer's own backoff tops out at maxSleep, far short of that.
fs.Debugf(nil, "Server asked to retry after %v", d)
err = pacer.RetryAfterError(err, d)
}
}
return retry, err
}
// maxRetryAfter caps how long a single Retry-After is honoured for
const maxRetryAfter = 5 * time.Minute
// retryAfter reads a 429 or 503 response's Retry-After header, which the API
// sends as whole seconds
func retryAfter(resp *http.Response) (time.Duration, bool) {
if resp == nil || (resp.StatusCode != http.StatusTooManyRequests && resp.StatusCode != http.StatusServiceUnavailable) {
return 0, false
}
seconds, err := strconv.Atoi(strings.TrimSpace(resp.Header.Get("Retry-After")))
if err != nil || seconds <= 0 {
return 0, false
}
return min(time.Duration(seconds)*time.Second, maxRetryAfter), true
}
// errorHandler decodes a non-2xx response into an *api.Error so callers can
// act on the machine-readable error_code rather than the message text.
func errorHandler(resp *http.Response) error {
body, err := rest.ReadBody(resp)
if err != nil {
return fmt.Errorf("error reading error out of body: %w", err)
}
apiErr := &api.Error{
StatusCode: resp.StatusCode,
Status: resp.Status,
}
if json.Unmarshal(body, apiErr) != nil || apiErr.Message == "" {
// Not the API's JSON shape (e.g. an edge response that never
// reached the API) - keep the body text so it isn't lost.
apiErr.ErrorCode = ""
apiErr.Message = strings.TrimSpace(string(body))
}
return apiErr
}
// The workspace caps the number of in-flight uploads per user. The API
// refuses an upload/init past that cap with a 400 carrying
// error_code "concurrent_upload_limit", and documents it as an error to
// wait on rather than give up on: slots free as uploads finish, and the
// server marks sessions idle for 10 minutes as failed, so waiting up to
// that long guarantees progress unless the workspace really is saturated.
// A 400 is deliberately outside the pacer's retry codes, so the wait
// lives in initUpload instead of shouldRetry.
var (
concurrentUploadRetryDelay = 5 * time.Second
concurrentUploadWaitMax = 10 * time.Minute
)
// abortUploadTimeout bounds the abort of a session the client is giving up on,
// which runs outside the transfer's context so a cancelled transfer still
// sends it
const abortUploadTimeout = 10 * time.Second
// isConcurrentUploadLimit reports whether err is the API refusing an
// upload because the workspace's concurrent upload limit is reached
func isConcurrentUploadLimit(err error) bool {
var apiErr *api.Error
return errors.As(err, &apiErr) && apiErr.ErrorCode == "concurrent_upload_limit"
}
// listPageSize is the largest page GET /api/files will serve
const listPageSize = 500
// listFilesAndFolders lists files and folders in a directory
//
// The API pages files (at most listPageSize per page) but returns every
// folder on each page, so the files from all pages are gathered and the
// folders are taken from the first page only. Pages are ordered by name,
// which the API tie-breaks on id, so offsets are stable between requests.
func (f *Fs) listFilesAndFolders(ctx context.Context, folderID string) (*api.ListResponse, error) {
var listing *api.ListResponse
for page := 1; ; page++ {
opts := rest.Opts{
Method: "GET",
Path: "/api/files",
Parameters: map[string][]string{
"workspace_id": {f.opt.WorkspaceID},
"per_page": {strconv.Itoa(listPageSize)},
"page": {strconv.Itoa(page)},
"sort": {"name"},
"dir": {"asc"},
},
}
if folderID != "" && folderID != rootID {
opts.Parameters["folder_id"] = []string{folderID}
}
var result api.ListResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't list files: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
if listing == nil {
listing = &result
} else {
listing.Files = append(listing.Files, result.Files...)
}
if page >= result.Pagination.TotalPages || len(result.Files) == 0 {
return listing, nil
}
}
}
// createFolder creates a folder
func (f *Fs) createFolder(ctx context.Context, name string, parentID string) (*api.CreateFolderResponse, error) {
request := api.CreateFolderRequest{
WorkspaceID: f.opt.WorkspaceID,
Name: name,
}
if parentID != "" && parentID != rootID {
request.ParentID = &parentID
}
opts := rest.Opts{
Method: "POST",
Path: "/api/folders",
}
var result api.CreateFolderResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't create folder: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// removeFolder deletes a folder
//
// ?permanent=1 trashes and purges a live folder in one request on today's
// API. ifEmpty also sends ?if_empty=1, which asks the server to refuse with
// HTTP 409 "Folder is not empty" instead of purging a folder that still
// holds a live file or a live child folder; that 409 is turned into
// fs.ErrorDirectoryNotEmpty here rather than treated as a generic API
// error, and it is never retried (409 is not one of retryErrorCodes). A big
// subtree is purged in bounded passes (202, complete=false), so keep
// calling until the purge is complete. An older server that ignores
// permanent=1 only trashes on the first call (answering permanent=false),
// so the loop's second pass sends the plain DELETE that purges it.
func (f *Fs) removeFolder(ctx context.Context, folderID string, ifEmpty bool) error {
parameters := map[string][]string{"permanent": {"1"}}
if ifEmpty {
parameters["if_empty"] = []string{"1"}
}
opts := rest.Opts{
Method: "DELETE",
Path: "/api/folders/" + folderID,
Parameters: parameters,
ExtraHeaders: f.byIDHeaders(),
}
for pass := 0; ; pass++ {
var result api.DeleteFolderResponse
var statusCode int
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
if resp != nil {
statusCode = resp.StatusCode
}
return shouldRetry(ctx, resp, err)
})
if err != nil {
if ifEmpty && statusCode == http.StatusConflict {
return fs.ErrorDirectoryNotEmpty
}
if pass == 0 {
return fmt.Errorf("couldn't remove folder: %w", err)
}
return fmt.Errorf("couldn't permanently remove folder: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
if result.Permanent && result.Complete {
return nil
}
if pass >= maxFolderPurgePasses {
return fmt.Errorf("couldn't permanently remove folder: purge still incomplete after %d passes", pass)
}
}
}
// maxFolderPurgePasses bounds the DELETE calls spent purging one folder
const maxFolderPurgePasses = 1000
// renameFolder renames a folder
func (f *Fs) renameFolder(ctx context.Context, folderID string, newName string) error {
request := api.RenameFolderRequest{Name: newName}
opts := rest.Opts{
Method: "PUT",
Path: "/api/folders/" + folderID + "/rename",
ExtraHeaders: f.byIDHeaders(),
}
var result api.Response
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't rename folder: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
return nil
}
// moveFolder moves a folder to a new parent
func (f *Fs) moveFolder(ctx context.Context, folderID string, newParentID string, newName string) (renamed bool, err error) {
request := api.MoveFolderRequest{Name: newName}
if newParentID != "" && newParentID != rootID {
request.ParentID = &newParentID
}
opts := rest.Opts{
Method: "PUT",
Path: "/api/folders/" + folderID + "/move",
ExtraHeaders: f.byIDHeaders(),
}
var result api.MoveFolderResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return false, fmt.Errorf("couldn't move folder: %w", err)
}
if !result.OK {
return false, fmt.Errorf("API error: %s", result.Error)
}
// Servers that predate the name field ignore it and answer without a
// name, leaving the caller to rename separately
return newName != "" && result.Name == newName, nil
}
// deleteFile deletes a file
//
// ?permanent=1 trashes and purges it in one request on today's API. An
// older server that ignores the flag only soft-deletes it, answering
// permanent=false, so a second DELETE on the now soft-deleted file purges
// it from the database and R2, as it always has.
func (f *Fs) deleteFile(ctx context.Context, fileID string) error {
opts := rest.Opts{
Method: "DELETE",
Path: "/api/files/" + fileID,
Parameters: map[string][]string{"permanent": {"1"}},
ExtraHeaders: f.byIDHeaders(),
}
var result api.DeleteFileResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't delete file: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
// Answered into its own value so a field the second response omits
// can't be read off the first.
if !result.Permanent {
var permanent api.DeleteFileResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &permanent)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't permanently delete file: %w", err)
}
if !permanent.OK {
return fmt.Errorf("API error: %s", permanent.Error)
}
}
return nil
}
// renameFile renames a file
func (f *Fs) renameFile(ctx context.Context, fileID string, newName string) error {
request := api.RenameFileRequest{Name: newName}
opts := rest.Opts{
Method: "PUT",
Path: "/api/files/" + fileID + "/rename",
ExtraHeaders: f.byIDHeaders(),
}
var result api.RenameFileResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't rename file: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
return nil
}
// moveFile moves a file to a new folder and, if newName is non-empty,
// renames it in the same request
func (f *Fs) moveFile(ctx context.Context, fileID string, folderID string, newName string) error {
request := api.MoveFileRequest{}
if folderID != "" && folderID != rootID {
request.FolderID = &folderID
}
if newName != "" {
request.Name = &newName
}
opts := rest.Opts{
Method: "PUT",
Path: "/api/files/" + fileID + "/move",
ExtraHeaders: f.byIDHeaders(),
}
var result api.Response
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return fmt.Errorf("couldn't move file: %w", err)
}
if !result.OK {
return fmt.Errorf("API error: %s", result.Error)
}
return nil
}
// copyFile copies a file to a folder
func (f *Fs) copyFile(ctx context.Context, fileID string, folderID string, name string) (*api.CopyFileResponse, error) {
request := api.CopyFileRequest{Name: name}
if folderID != "" && folderID != rootID {
request.FolderID = &folderID
}
opts := rest.Opts{
Method: "POST",
Path: "/api/files/" + fileID + "/copy",
ExtraHeaders: f.byIDHeaders(),
}
var result api.CopyFileResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't copy file: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// downloadFile downloads a file by its ID
//
// The download endpoint answers 302 with a presigned R2 URL in Location, so
// this is two calls. Both go through the rest client, which is built on
// fshttp, so --timeout, --contimeout, --ca-cert, --no-check-certificate,
// --user-agent, --bind, --dump and the proxy settings apply to downloads as
// they do to every other call, and errorHandler and shouldRetry decide the
// errors and the retries uniformly.
func (f *Fs) downloadFile(ctx context.Context, fileID string, options []fs.OpenOption) (io.ReadCloser, error) {
// NoRedirect stops the client following the 302 with the API's
// Authorization header attached. A 302 is not a 2xx, so Call reports it
// as an error (having read and closed the body); the Location header
// is what actually matters, and finding it clears the error, as in
// backend/linkbox.
opts := rest.Opts{
Method: "GET",
Path: "/api/files/" + fileID + "/download",
ExtraHeaders: f.byIDHeaders(),
NoRedirect: true,
}
var downloadURL string
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.Call(ctx, &opts)
if resp != nil {
if location := resp.Header.Get("Location"); location != "" {
downloadURL, err = location, nil
}
}
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't get download URL: %w", err)
}
if downloadURL == "" {
return nil, errors.New("no download URL in redirect response")
}
// Fetch the object from the presigned URL. It carries its own
// credentials and R2 refuses a request that also has an Authorization
// header, so the client-level bearer token is removed by setting the
// header empty: the rest client skips a header with an empty value.
opts = rest.Opts{
Method: "GET",
RootURL: downloadURL,
Options: options,
ExtraHeaders: map[string]string{"Authorization": ""},
}
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
var err error
resp, err = f.rest.Call(ctx, &opts)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't download file: %w", err)
}
return resp.Body, nil
}
// uploadSmallFile uploads a file smaller than 10MB in a single request
func (f *Fs) uploadSmallFile(ctx context.Context, in io.Reader, sessionID string, size int64, modTime time.Time) (*api.UploadCompleteResponse, error) {
opts := rest.Opts{
Method: "PUT",
Path: "/api/upload/" + sessionID,
Body: in,
ContentLength: &size,
ContentType: "application/octet-stream",
ExtraHeaders: modTimeHeaders(modTime),
}
// The body is a stream which can't be rewound, so a low level retry
// would resend whatever is left of it - nothing - and the server
// would store that as an empty file. Don't retry here; rclone's high
// level retries redo the whole transfer instead.
var result api.UploadCompleteResponse
err := f.pacer.CallNoRetry(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't upload file: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// uploadPart uploads a single part of a multipart upload
func (f *Fs) uploadPart(ctx context.Context, part []byte, sessionID string, partNumber int) (*api.UploadPartResponse, error) {
size := int64(len(part))
opts := rest.Opts{
Method: "PUT",
Path: "/api/upload/" + sessionID + "/part/" + strconv.Itoa(partNumber),
ContentLength: &size,
ContentType: "application/octet-stream",
}
var result api.UploadPartResponse
err := f.pacer.Call(func() (bool, error) {
// A fresh reader per attempt so a retry resends the whole part
opts.Body = bytes.NewReader(part)
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't upload part %d: %w", partNumber, err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// completeUpload finalizes a multipart upload
func (f *Fs) completeUpload(ctx context.Context, sessionID string, modTime time.Time) (*api.UploadCompleteResponse, error) {
opts := rest.Opts{
Method: "POST",
Path: "/api/upload/" + sessionID + "/complete",
ExtraHeaders: modTimeHeaders(modTime),
}
var result api.UploadCompleteResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't complete upload: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// initUpload initializes a new upload session
func (f *Fs) initUpload(ctx context.Context, fileName string, fileSize int64, mimeType string, folderID string, fileID *string) (*api.UploadInitResponse, error) {
request := api.UploadInitRequest{
WorkspaceID: f.opt.WorkspaceID,
FileName: fileName,
FileSize: fileSize,
MimeType: mimeType,
FileID: fileID,
}
if folderID != "" && folderID != rootID {
request.FolderID = &folderID
}
opts := rest.Opts{
Method: "POST",
Path: "/api/upload/init",
}
var result api.UploadInitResponse
deadline := time.Now().Add(concurrentUploadWaitMax)
for {
result = api.UploadInitResponse{}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err == nil {
break
}
if !isConcurrentUploadLimit(err) || time.Now().After(deadline) {
return nil, fmt.Errorf("couldn't init upload: %w", err)
}
fs.Debugf(f, "Upload slots in use, waiting %v before retrying %q: %v", concurrentUploadRetryDelay, fileName, err)
select {
case <-ctx.Done():
return nil, fmt.Errorf("couldn't init upload: %w", ctx.Err())
case <-time.After(concurrentUploadRetryDelay):
}
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// uploadParts reads the source part by part and uploads the parts.
//
// Part 1 goes on its own: the server creates the R2 multipart upload (and a
// new file's id and key) when it receives the first part, so parts sent
// before that has finished would each start an upload of their own. The
// rest go out opt.UploadConcurrency at a time. A part is only read from the
// source once a buffer is free, so at most that many parts are held in
// memory, and a short source is still found by the reader before the part
// it could not fill is sent.
func (f *Fs) uploadParts(ctx context.Context, in *readers.CountingReader, sessionID string, size, partSize int64, totalParts int) error {
readPart := func(buf []byte, partNum int) ([]byte, error) {
currentPartSize := partSize
if partNum == totalParts {
currentPartSize = size - partSize*int64(totalParts-1)
}
part := buf[:currentPartSize]
if _, err := io.ReadFull(in, part); err != nil {
if err == io.EOF || err == io.ErrUnexpectedEOF {
return nil, fmt.Errorf("expected %d bytes in input, but got %d: %w", size, in.BytesRead(), io.ErrUnexpectedEOF)
}
return nil, fmt.Errorf("failed to read part %d: %w", partNum, err)
}
return part, nil
}
concurrency := max(f.opt.UploadConcurrency, 1)
buffers := make(chan []byte, concurrency)
buffers <- make([]byte, partSize)
allocated := 1
first := <-buffers
part, err := readPart(first, 1)
if err != nil {
return err
}
if _, err := f.uploadPart(ctx, part, sessionID, 1); err != nil {
return err
}
buffers <- first
// A failed read has to take the parts already in flight down with it,
// or they upload to completion while the error waits on Wait.
ctx, cancel := context.WithCancel(ctx)
defer cancel()
g, gCtx := errgroup.WithContext(ctx)
var readErr error
for partNum := 2; partNum <= totalParts; partNum++ {
// Take a free buffer, allocating one while under the limit.
var buf []byte
select {
case buf = <-buffers:
default:
if allocated < concurrency {
buf = make([]byte, partSize)
allocated++
} else {
select {
case buf = <-buffers:
case <-gCtx.Done():
}
}
}
if buf == nil {
break // a part failed; its error comes back from Wait
}
part, err := readPart(buf, partNum)
if err != nil {
readErr = err
cancel()
break
}
g.Go(func() error {
defer func() { buffers <- buf }()
_, err := f.uploadPart(gCtx, part, sessionID, partNum)
return err
})
}
err = g.Wait()
if readErr != nil {
return readErr
}
return err
}
// abortUpload retires an upload session the client is giving up on.
//
// Best effort: the server reclaims an abandoned session itself, so a failure
// here - including an older server with no such route - costs the slot the
// wait it would have cost anyway and is not worth failing an upload over.
// Sent on a context of its own so a cancelled transfer still gives its slot
// back rather than leaving it to the sweep.
func (f *Fs) abortUpload(ctx context.Context, sessionID string) {
ctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), abortUploadTimeout)
defer cancel()
opts := rest.Opts{
Method: "DELETE",
Path: "/api/upload/" + sessionID,
NoResponse: true,
}
err := f.pacer.CallNoRetry(func() (bool, error) {
resp, err := f.rest.Call(ctx, &opts)
return shouldRetry(ctx, resp, err)
})
if err != nil {
fs.Debugf(f, "Couldn't abort upload session %q: %v", sessionID, err)
}
}
// uploadFile handles the full upload flow (small or multipart)
func (f *Fs) uploadFile(ctx context.Context, in io.Reader, remote string, size int64, modTime time.Time, mimeType string, folderID string, fileID *string) (resp *api.UploadCompleteResponse, err error) {
// Both refusals live here so a new file and a new version of an existing
// one answer the same way: upload/init needs a byte count up front, and
// the server stores no zero-length object.
if size < 0 {
return nil, errors.New("can't upload files with unknown size")
}
if size == 0 {
return nil, fs.ErrorCantUploadEmptyFiles
}
leaf := f.opt.Enc.FromStandardName(path.Base(remote))
initResp, err := f.initUpload(ctx, leaf, size, mimeType, folderID, fileID)
if err != nil {
return nil, err
}
// A session abandoned here holds one of the workspace's upload slots
// until the server's sweep retires it ten minutes later, and rclone's
// high-level retry opens a fresh session rather than resuming this one -
// so without this, a few failed attempts use up every slot.
defer func() {
if err != nil {
f.abortUpload(ctx, initResp.SessionID)
}
}()
// Count what the source actually supplies, so a source which ends
// early is reported as an error rather than stored as a truncated
// file.
counter := readers.NewCountingReader(in)
// Small file upload (single PUT, no resumable)
if initResp.Resumable == nil {
resp, err = f.uploadSmallFile(ctx, counter, initResp.SessionID, size, modTime)
if err != nil {
return nil, err
}
if got := int64(counter.BytesRead()); got != size {
return nil, fmt.Errorf("expected %d bytes in input, but got %d: %w", size, got, io.ErrUnexpectedEOF)
}
return resp, nil
}
// Multipart upload. Each part is read fully into memory before it is
// sent, so a short source is caught before the part goes out and a
// part which has to be retried is resent with all of its bytes.
partSize := initResp.Resumable.PartSize
totalParts := initResp.Resumable.TotalParts
// Both bounds matter. partSize*(totalParts-1) >= size means a part would
// have nothing to carry; size > partSize*totalParts means the last part
// is bigger than partSize, and readPart would slice past its buffer and
// panic.
if partSize <= 0 || totalParts <= 0 || partSize*int64(totalParts-1) >= size || size > partSize*int64(totalParts) {
return nil, fmt.Errorf("invalid multipart layout from server: %d parts of %d bytes for %d bytes", totalParts, partSize, size)
}
if err = f.uploadParts(ctx, counter, initResp.SessionID, size, partSize, totalParts); err != nil {
return nil, err
}
return f.completeUpload(ctx, initResp.SessionID, modTime)
}
// createShareLink creates a public share link for a file
func (f *Fs) createShareLink(ctx context.Context, fileID string, expire fs.Duration) (*api.ShareLinkResponse, error) {
request := api.ShareLinkRequest{}
if expire < fs.DurationOff {
days := int(time.Duration(expire).Hours() / 24)
if days < 1 {
days = 1
}
request.ExpiresInDays = &days
}
opts := rest.Opts{
Method: "POST",
Path: "/api/files/" + fileID + "/share",
ExtraHeaders: f.byIDHeaders(),
}
var result api.ShareLinkResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, &request, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't create share link: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
// getWorkspaceInfo fetches workspace storage info
// listWorkspaces lists the workspaces this credential can reach. An API key
// pinned to a workspace sees exactly one: its own.
func (f *Fs) listWorkspaces(ctx context.Context) ([]api.Workspace, error) {
opts := rest.Opts{
Method: "GET",
Path: "/api/workspaces",
}
var result api.WorkspaceListResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't list workspaces: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return result.Workspaces, nil
}
func (f *Fs) getWorkspaceInfo(ctx context.Context) (*api.WorkspaceInfoResponse, error) {
opts := rest.Opts{
Method: "GET",
Path: "/api/workspaces/" + f.opt.WorkspaceID,
}
var result api.WorkspaceInfoResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, fmt.Errorf("couldn't get workspace info: %w", err)
}
if !result.OK {
return nil, fmt.Errorf("API error: %s", result.Error)
}
return &result, nil
}
-235
View File
@@ -1,235 +0,0 @@
package dosya
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"testing"
"time"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/pacer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestShouldRetryHonoursRetryAfter(t *testing.T) {
ctx := context.Background()
for _, tc := range []struct {
name string
status int
header string
want time.Duration
ok bool
}{
{"503 during a workspace move", http.StatusServiceUnavailable, "30", 30 * time.Second, true},
{"429 from the edge", http.StatusTooManyRequests, "7", 7 * time.Second, true},
{"capped", http.StatusServiceUnavailable, "86400", maxRetryAfter, true},
{"missing header", http.StatusServiceUnavailable, "", 0, false},
{"HTTP date is not seconds", http.StatusServiceUnavailable, "Wed, 21 Oct 2015 07:28:00 GMT", 0, false},
{"502 ignores the header", http.StatusBadGateway, "30", 0, false},
} {
t.Run(tc.name, func(t *testing.T) {
resp := jsonResp(tc.status, `{"ok":false,"error":"busy"}`)
if tc.header != "" {
resp.Header.Set("Retry-After", tc.header)
}
retry, err := shouldRetry(ctx, resp, errorHandler(resp))
assert.True(t, retry)
d, ok := pacer.IsRetryAfter(err)
assert.Equal(t, tc.ok, ok)
if tc.ok {
assert.Equal(t, tc.want, d)
}
})
}
}
func TestShouldRetryDoesNotRetryConflict(t *testing.T) {
resp := jsonResp(http.StatusConflict, `{"ok":false,"error":"A folder with this name already exists"}`)
resp.Header.Set("Retry-After", "30")
retry, err := shouldRetry(context.Background(), resp, errorHandler(resp))
assert.False(t, retry)
_, ok := pacer.IsRetryAfter(err)
assert.False(t, ok)
}
func TestRemoveFolderPurgesPermanently(t *testing.T) {
// a big subtree purges in bounded passes: 202 complete=false repeats
// the same request until complete=true
answers := []struct {
status int
body string
}{
{http.StatusAccepted, `{"ok":true,"permanent":true,"complete":false,"remaining":1}`},
{http.StatusOK, `{"ok":true,"permanent":true,"complete":true,"remaining":0}`},
}
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
require.Equal(t, "DELETE", r.Method)
require.Equal(t, "/api/folders/fld_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "", r.URL.Query().Get("if_empty"))
require.Less(t, calls, len(answers), "DELETE sent after the purge completed")
a := answers[calls]
calls++
return jsonResp(a.status, a.body), nil
}))
require.NoError(t, f.removeFolder(context.Background(), "fld_1", false))
assert.Equal(t, len(answers), calls)
}
func TestRemoveFolderStopsOnPurgeError(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
return jsonResp(http.StatusForbidden, `{"ok":false,"error":"This folder contains a locked file and cannot be purged"}`), nil
}))
err := f.removeFolder(context.Background(), "fld_1", false)
require.Error(t, err)
assert.Contains(t, err.Error(), "locked file")
assert.Equal(t, 1, calls)
}
func TestRemoveFolderIfEmptyRejectsNonEmptyWithoutRetry(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "/api/folders/fld_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "1", r.URL.Query().Get("if_empty"))
return jsonResp(http.StatusConflict, `{"ok":false,"error":"Folder is not empty"}`), nil
}))
err := f.removeFolder(context.Background(), "fld_1", true)
assert.ErrorIs(t, err, fs.ErrorDirectoryNotEmpty)
assert.Equal(t, 1, calls, "a 409 must not be retried by the pacer")
}
func TestRemoveFolderIfEmptyPurgesEmptyFolderInOneCall(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "1", r.URL.Query().Get("if_empty"))
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true,"complete":true}`), nil
}))
require.NoError(t, f.removeFolder(context.Background(), "fld_1", true))
assert.Equal(t, 1, calls)
}
func TestRmdirMakesOneRequest(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "DELETE", r.Method)
assert.Equal(t, "/api/folders/fld_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
assert.Equal(t, "1", r.URL.Query().Get("if_empty"))
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true,"complete":true}`), nil
}))
f.dirCache = dircache.New("", rootID, f)
require.NoError(t, f.dirCache.FindRoot(context.Background(), false))
f.dirCache.Put("sub", "fld_1")
require.NoError(t, f.Rmdir(context.Background(), "sub"))
assert.Equal(t, 1, calls, "Rmdir must not list the directory before removing it")
}
func TestRmdirMapsNonEmptyConflict(t *testing.T) {
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
return jsonResp(http.StatusConflict, `{"ok":false,"error":"Folder is not empty"}`), nil
}))
f.dirCache = dircache.New("", rootID, f)
require.NoError(t, f.dirCache.FindRoot(context.Background(), false))
f.dirCache.Put("sub", "fld_1")
err := f.Rmdir(context.Background(), "sub")
assert.ErrorIs(t, err, fs.ErrorDirectoryNotEmpty)
}
func TestDeleteFileSendsPermanentFlag(t *testing.T) {
for _, tc := range []struct {
name string
permanent bool
wantCalls int
}{
{"today's API purges in one call", true, 1},
{"an older server ignores the flag and still gets a second DELETE", false, 2},
} {
t.Run(tc.name, func(t *testing.T) {
calls := 0
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
calls++
assert.Equal(t, "DELETE", r.Method)
assert.Equal(t, "/api/files/fil_1", r.URL.Path)
assert.Equal(t, "1", r.URL.Query().Get("permanent"))
return jsonResp(http.StatusOK, fmt.Sprintf(`{"ok":true,"permanent":%t}`, tc.permanent)), nil
}))
require.NoError(t, f.deleteFile(context.Background(), "fil_1"))
assert.Equal(t, tc.wantCalls, calls)
})
}
}
// decodeBody reads a JSON request body into a map
func decodeBody(t *testing.T, r *http.Request) map[string]any {
b, err := io.ReadAll(r.Body)
require.NoError(t, err)
var m map[string]any
require.NoError(t, json.Unmarshal(b, &m))
return m
}
func TestMoveFolderSendsName(t *testing.T) {
for _, tc := range []struct {
name string
newName string
respBody string
wantRenamed bool
}{
{"server honours name", "dst", `{"ok":true,"name":"dst"}`, true},
{"older server ignores name", "dst", `{"ok":true}`, false},
{"no rename requested", "", `{"ok":true,"name":"src"}`, false},
} {
t.Run(tc.name, func(t *testing.T) {
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
assert.Equal(t, "/api/folders/fld_1/move", r.URL.Path)
body := decodeBody(t, r)
assert.Equal(t, "fld_parent", body["parent_id"])
if tc.newName == "" {
assert.NotContains(t, body, "name")
} else {
assert.Equal(t, tc.newName, body["name"])
}
return jsonResp(http.StatusOK, tc.respBody), nil
}))
renamed, err := f.moveFolder(context.Background(), "fld_1", "fld_parent", tc.newName)
require.NoError(t, err)
assert.Equal(t, tc.wantRenamed, renamed)
})
}
}
func TestCopyFileSendsName(t *testing.T) {
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
assert.Equal(t, "/api/files/fil_1/copy", r.URL.Path)
body := decodeBody(t, r)
assert.Equal(t, "fld_dst", body["folder_id"])
assert.Equal(t, "final.txt", body["name"])
return jsonResp(http.StatusCreated, `{"ok":true,"file_id":"fil_2","name":"final.txt"}`), nil
}))
resp, err := f.copyFile(context.Background(), "fil_1", "fld_dst", "final.txt")
require.NoError(t, err)
assert.Equal(t, "fil_2", resp.FileID)
assert.Equal(t, "final.txt", resp.Name)
}
-171
View File
@@ -1,171 +0,0 @@
package dosya
import (
"context"
"io"
"net/http"
"strconv"
"strings"
"testing"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestFsAdvertisesSHA256AndSecondPrecision(t *testing.T) {
f := &Fs{}
assert.Equal(t, time.Second, f.Precision())
assert.True(t, f.Hashes().Contains(hash.SHA256), "SHA-256 must be advertised")
// Exactly SHA-256, nothing else.
assert.Equal(t, hash.NewHashSet(hash.SHA256), f.Hashes())
}
func TestObjectHashReturnsContentHash(t *testing.T) {
o := &Object{file: api.FileItem{ContentHash: "abc123"}}
got, err := o.Hash(context.Background(), hash.SHA256)
require.NoError(t, err)
assert.Equal(t, "abc123", got)
// A multipart object with no server-side hash yields "" and no error,
// exactly as an S3 multipart object does.
empty := &Object{file: api.FileItem{ContentHash: ""}}
got, err = empty.Hash(context.Background(), hash.SHA256)
require.NoError(t, err)
assert.Equal(t, "", got)
// Any other algorithm is unsupported.
_, err = o.Hash(context.Background(), hash.MD5)
assert.ErrorIs(t, err, hash.ErrUnsupported)
}
func TestObjectModTimeUsesUpdatedAt(t *testing.T) {
mt := time.Date(2003, 2, 3, 4, 5, 6, 0, time.UTC)
o := &Object{file: api.FileItem{UpdatedAt: float64(mt.Unix()), CreatedAt: 1}}
assert.True(t, mt.Equal(o.ModTime(context.Background())))
}
func TestModTimeHeaders(t *testing.T) {
assert.Nil(t, modTimeHeaders(time.Time{}), "a zero time sends no header")
mt := time.Date(2003, 2, 3, 4, 5, 6, 499, time.UTC)
h := modTimeHeaders(mt)
require.NotNil(t, h)
assert.Equal(t, strconv.FormatInt(mt.Unix(), 10), h[sourceMTimeHeader])
}
func TestFileFromUploadKeepsModTimeAndHash(t *testing.T) {
resp := &api.UploadCompleteResponse{}
resp.File.ID = "fil_1"
resp.File.Name = "x.txt"
resp.File.SizeBytes = 10
resp.File.CreatedAt = 1_000
resp.File.ContentHash = "deadbeef"
mt := time.Date(2004, 3, 3, 4, 5, 6, 0, time.UTC)
item := fileFromUpload(resp, mt)
assert.Equal(t, float64(mt.Unix()), item.UpdatedAt, "modtime is what we sent, not the create time")
assert.Equal(t, "deadbeef", item.ContentHash)
assert.Equal(t, float64(1_000), item.CreatedAt)
// A zero modtime falls back to the create time.
item = fileFromUpload(resp, time.Time{})
assert.Equal(t, float64(1_000), item.UpdatedAt)
}
// TestUploadSendsSourceMTimeHeader drives a real single-PUT upload through a
// fake transport and asserts the source-mtime header actually rides on the PUT.
func TestUploadSendsSourceMTimeHeader(t *testing.T) {
var putMTime string
rt := rtFunc(func(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_, _ = io.ReadAll(r.Body) // drain so the upload's counting reader advances
}
switch {
case r.Method == "POST" && r.URL.Path == "/api/upload/init":
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1"}`), nil
case r.Method == "PUT" && r.URL.Path == "/api/upload/upl_1":
putMTime = r.Header.Get(sourceMTimeHeader)
return jsonResp(http.StatusCreated, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":5,"content_hash":"h"}}`), nil
}
t.Fatalf("unexpected %s %s", r.Method, r.URL.Path)
return nil, nil
})
f := newFakeFs(rt)
mt := time.Date(2003, 2, 3, 4, 5, 6, 0, time.UTC)
resp, err := f.uploadFile(context.Background(), strings.NewReader("hello"), "file.txt", 5, mt, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, "h", resp.File.ContentHash)
assert.Equal(t, strconv.FormatInt(mt.Unix(), 10), putMTime, "the PUT must carry the source-mtime header")
}
// TestMultipartCompleteSendsSourceMTimeHeader asserts the header also rides on
// the completion request for a multipart upload (the server stores the modtime
// there, not on the individual parts).
func TestMultipartCompleteSendsSourceMTimeHeader(t *testing.T) {
var completeMTime string
rt := rtFunc(func(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_, _ = io.ReadAll(r.Body)
}
switch {
case r.URL.Path == "/api/upload/init":
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1","resumable":{"part_size":4,"total_parts":3}}`), nil
case strings.HasPrefix(r.URL.Path, "/api/upload/upl_1/part/"):
return jsonResp(http.StatusOK, `{"ok":true}`), nil
case r.URL.Path == "/api/upload/upl_1/complete":
completeMTime = r.Header.Get(sourceMTimeHeader)
return jsonResp(http.StatusOK, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":10}}`), nil
}
t.Fatalf("unexpected %s %s", r.Method, r.URL.Path)
return nil, nil
})
f := newFakeFs(rt)
mt := time.Date(2003, 2, 3, 4, 5, 6, 0, time.UTC)
_, err := f.uploadFile(context.Background(), strings.NewReader("0123456789"), "file.txt", 10, mt, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, strconv.FormatInt(mt.Unix(), 10), completeMTime, "the complete call must carry the source-mtime header")
}
// TestCopyReturnsSourceModTimeAndHash guards the fingerprint of the object a
// server-side Copy returns: it must report the SOURCE's modtime and hash (the
// copy is byte-identical and the API preserves both), not the copy's wall
// clock. Reading "now" here made rclone's in-memory object disagree with the
// one read back from the remote and failed FsCopy's fingerprint check.
func TestCopyReturnsSourceModTimeAndHash(t *testing.T) {
rt := rtFunc(func(r *http.Request) (*http.Response, error) {
switch {
case r.Method == "GET" && r.URL.Path == "/api/files":
// NewObject's existence probe on the destination: nothing there.
return jsonResp(http.StatusOK, `{"ok":true,"files":[],"folders":[]}`), nil
case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/copy"):
return jsonResp(http.StatusOK, `{"ok":true,"file_id":"fil_copy","name":"f-copy.txt"}`), nil
}
t.Fatalf("unexpected %s %s", r.Method, r.URL.Path)
return nil, nil
})
f := newFakeFs(rt)
f.dirCache = dircache.New("", rootID, f)
mt := time.Date(2001, 2, 3, 4, 5, 10, 0, time.UTC)
ct := time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC)
src := &Object{fs: f, remote: "orig.txt", file: api.FileItem{
ID: "fil_src", Name: "orig.txt", SizeBytes: 100, MimeType: "text/plain",
CreatedAt: float64(ct.Unix()), UpdatedAt: float64(mt.Unix()), ContentHash: "srchash",
}}
dst, err := f.Copy(context.Background(), src, "f-copy.txt")
require.NoError(t, err)
assert.True(t, mt.Equal(dst.ModTime(context.Background())), "copy keeps the source modtime")
assert.Equal(t, int64(100), dst.Size())
got, err := dst.Hash(context.Background(), hash.SHA256)
require.NoError(t, err)
assert.Equal(t, "srchash", got, "copy keeps the source hash")
}
-156
View File
@@ -1,156 +0,0 @@
package dosya
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
"github.com/rclone/rclone/backend/dosya/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const concurrentLimitBody = `{"ok":false,"error":"You have 5 uploads in progress. Wait for them to finish before starting more.","error_code":"concurrent_upload_limit","max_concurrent_uploads":5}`
func TestErrorHandlerDecodesAPIError(t *testing.T) {
resp := &http.Response{
StatusCode: http.StatusBadRequest,
Status: "400 Bad Request",
Body: io.NopCloser(strings.NewReader(concurrentLimitBody)),
}
err := errorHandler(resp)
var apiErr *api.Error
require.ErrorAs(t, err, &apiErr)
assert.Equal(t, http.StatusBadRequest, apiErr.StatusCode)
assert.Equal(t, "concurrent_upload_limit", apiErr.ErrorCode)
assert.Equal(t, 5, apiErr.MaxConcurrentUploads)
assert.Contains(t, err.Error(), "uploads in progress")
assert.Contains(t, err.Error(), "concurrent_upload_limit")
}
func TestErrorHandlerNonJSONBody(t *testing.T) {
// e.g. an edge (Cloudflare) response that never reached the API
resp := &http.Response{
StatusCode: http.StatusTooManyRequests,
Status: "429 Too Many Requests",
Body: io.NopCloser(strings.NewReader("error code: 1015\n")),
}
err := errorHandler(resp)
var apiErr *api.Error
require.ErrorAs(t, err, &apiErr)
assert.Equal(t, http.StatusTooManyRequests, apiErr.StatusCode)
assert.Equal(t, "", apiErr.ErrorCode)
assert.Contains(t, err.Error(), "429")
assert.Contains(t, err.Error(), "error code: 1015")
}
func TestShouldRetryAPIErrors(t *testing.T) {
ctx := context.Background()
// The concurrent upload limit is handled by initUpload's slot wait,
// not by the pacer's low level retries.
limitErr := &api.Error{StatusCode: http.StatusBadRequest, ErrorCode: "concurrent_upload_limit"}
retry, err := shouldRetry(ctx, &http.Response{StatusCode: http.StatusBadRequest}, limitErr)
assert.False(t, retry)
assert.Equal(t, limitErr, err)
retry, _ = shouldRetry(ctx, &http.Response{StatusCode: http.StatusTooManyRequests}, &api.Error{StatusCode: http.StatusTooManyRequests})
assert.True(t, retry)
retry, _ = shouldRetry(ctx, &http.Response{StatusCode: http.StatusBadRequest}, &api.Error{StatusCode: http.StatusBadRequest, ErrorCode: "something_else"})
assert.False(t, retry)
}
// swapWaitTimings shortens the slot wait timings for a test and returns a
// function which restores them
func swapWaitTimings(delay, max time.Duration) func() {
oldDelay, oldMax := concurrentUploadRetryDelay, concurrentUploadWaitMax
concurrentUploadRetryDelay, concurrentUploadWaitMax = delay, max
return func() { concurrentUploadRetryDelay, concurrentUploadWaitMax = oldDelay, oldMax }
}
// newTestFs makes an Fs which talks to srv
func newTestFs(srv *httptest.Server) *Fs {
ctx := context.Background()
f := &Fs{
name: "test",
opt: Options{WorkspaceID: "ws_test"},
pacer: fs.NewPacer(ctx, pacer.NewDefault(
pacer.MinSleep(time.Millisecond),
pacer.MaxSleep(time.Millisecond),
pacer.DecayConstant(decayConstant),
pacer.AttackConstant(attackConstant),
)),
}
f.rest = rest.NewClient(srv.Client()).SetRoot(srv.URL).SetErrorHandler(errorHandler)
return f
}
// limitServer answers the first n init calls with the concurrent upload
// limit error and the rest with success
func limitServer(t *testing.T, n int32, calls *atomic.Int32) *httptest.Server {
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "/api/upload/init", r.URL.Path)
w.Header().Set("Content-Type", "application/json")
if calls.Add(1) <= n {
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(concurrentLimitBody))
return
}
w.WriteHeader(http.StatusCreated)
_, _ = w.Write([]byte(`{"ok":true}`))
}))
}
func TestInitUploadWaitsForFreeSlot(t *testing.T) {
defer swapWaitTimings(5*time.Millisecond, 10*time.Second)()
var calls atomic.Int32
srv := limitServer(t, 2, &calls)
defer srv.Close()
f := newTestFs(srv)
resp, err := f.initUpload(context.Background(), "file.txt", 10, "text/plain", "", nil)
require.NoError(t, err)
assert.True(t, resp.OK)
assert.Equal(t, int32(3), calls.Load())
}
func TestInitUploadGivesUpWhenSlotsStayFull(t *testing.T) {
defer swapWaitTimings(5*time.Millisecond, 40*time.Millisecond)()
var calls atomic.Int32
srv := limitServer(t, 1<<30, &calls)
defer srv.Close()
f := newTestFs(srv)
_, err := f.initUpload(context.Background(), "file.txt", 10, "text/plain", "", nil)
require.Error(t, err)
var apiErr *api.Error
require.ErrorAs(t, err, &apiErr)
assert.Equal(t, "concurrent_upload_limit", apiErr.ErrorCode)
assert.Contains(t, err.Error(), "couldn't init upload")
assert.GreaterOrEqual(t, calls.Load(), int32(2), "should have retried at least once")
}
func TestInitUploadStopsWaitingWhenContextCancelled(t *testing.T) {
defer swapWaitTimings(time.Minute, time.Hour)()
var calls atomic.Int32
srv := limitServer(t, 1<<30, &calls)
defer srv.Close()
f := newTestFs(srv)
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
defer cancel()
start := time.Now()
_, err := f.initUpload(ctx, "file.txt", 10, "text/plain", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, context.DeadlineExceeded)
assert.Less(t, time.Since(start), 5*time.Second)
}
-68
View File
@@ -1,68 +0,0 @@
package dosya
import (
"context"
"fmt"
"net/http"
"strconv"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeListServer serves GET /api/files the way the API does: files paged at
// per_page, every folder repeated on each page
func fakeListServer(t *testing.T, totalFiles int, pages *[]int) rtFunc {
return func(r *http.Request) (*http.Response, error) {
require.Equal(t, "/api/files", r.URL.Path)
q := r.URL.Query()
assert.Equal(t, "name", q.Get("sort"))
assert.Equal(t, "asc", q.Get("dir"))
page, _ := strconv.Atoi(q.Get("page"))
perPage, _ := strconv.Atoi(q.Get("per_page"))
require.Positive(t, page)
require.Positive(t, perPage)
*pages = append(*pages, page)
var files []string
for i := (page - 1) * perPage; i < page*perPage && i < totalFiles; i++ {
files = append(files, fmt.Sprintf(`{"id":"fil_%d","name":"f%05d.txt","size":1}`, i, i))
}
totalPages := max(1, (totalFiles+perPage-1)/perPage)
body := fmt.Sprintf(`{"ok":true,"files":[%s],"folders":[{"id":"fld_1","name":"sub"}],`+
`"pagination":{"page":%d,"per_page":%d,"total_files":%d,"total_pages":%d}}`,
strings.Join(files, ","), page, perPage, totalFiles, totalPages)
return jsonResp(http.StatusOK, body), nil
}
}
func TestListFilesAndFoldersFollowsPages(t *testing.T) {
var pages []int
f := newFakeFs(fakeListServer(t, 2*listPageSize+7, &pages))
listing, err := f.listFilesAndFolders(context.Background(), "fld_parent")
require.NoError(t, err)
assert.Equal(t, []int{1, 2, 3}, pages)
require.Len(t, listing.Files, 2*listPageSize+7)
assert.Equal(t, "f01006.txt", listing.Files[len(listing.Files)-1].Name)
// folders repeat on every page and must not be duplicated
assert.Len(t, listing.Folders, 1)
}
func TestListFilesAndFoldersSinglePage(t *testing.T) {
for _, total := range []int{0, 1, listPageSize} {
t.Run(strconv.Itoa(total), func(t *testing.T) {
var pages []int
f := newFakeFs(fakeListServer(t, total, &pages))
listing, err := f.listFilesAndFolders(context.Background(), rootID)
require.NoError(t, err)
assert.Equal(t, []int{1}, pages)
assert.Len(t, listing.Files, total)
})
}
}
-153
View File
@@ -1,153 +0,0 @@
package dosya
import (
"context"
"fmt"
"io"
"net/http"
"strings"
"sync"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// parallelUploadServer answers the multipart endpoints without holding a
// lock across a request, so parts really do overlap, and records what the
// ordering rules below are about: how many parts were in flight at once, and
// whether any part started before part 1 had finished.
//
// Part 1 matters because the API creates the R2 multipart upload (and, for a
// new file, its id and key) on the first part it sees. Parts that race ahead
// of part 1 would each create their own upload, and only one of those would
// be completed.
type parallelUploadServer struct {
t *testing.T
resumable string
partDelay time.Duration
failPart int // answer this part with a 400, 0 for none
mu sync.Mutex
inFlight int
maxFlight int
part1Done bool
earlyParts []int // parts that started before part 1 finished
parts map[int]string
completes int
aborts int
}
func (s *parallelUploadServer) roundTrip(r *http.Request) (*http.Response, error) {
var body []byte
if r.Body != nil {
var err error
body, err = io.ReadAll(r.Body)
require.NoError(s.t, err)
}
path := r.URL.Path
switch {
case r.Method == "POST" && path == "/api/upload/init":
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1","resumable":`+s.resumable+`}`), nil
case r.Method == "PUT" && strings.HasPrefix(path, "/api/upload/upl_1/part/"):
var n int
_, err := fmt.Sscanf(strings.TrimPrefix(path, "/api/upload/upl_1/part/"), "%d", &n)
require.NoError(s.t, err)
s.mu.Lock()
if n != 1 && !s.part1Done {
s.earlyParts = append(s.earlyParts, n)
}
s.inFlight++
s.maxFlight = max(s.maxFlight, s.inFlight)
s.mu.Unlock()
time.Sleep(s.partDelay)
s.mu.Lock()
s.inFlight--
if n == 1 {
s.part1Done = true
}
s.parts[n] = string(body)
fail := n == s.failPart
s.mu.Unlock()
if fail {
return jsonResp(http.StatusBadRequest, `{"ok":false,"error":"part refused"}`), nil
}
return jsonResp(http.StatusCreated, fmt.Sprintf(`{"ok":true,"part_number":%d,"etag":"e%d"}`, n, n)), nil
case r.Method == "POST" && path == "/api/upload/upl_1/complete":
s.mu.Lock()
s.completes++
s.mu.Unlock()
return jsonResp(http.StatusOK, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":20}}`), nil
case r.Method == "DELETE" && path == "/api/upload/upl_1":
s.mu.Lock()
s.aborts++
s.mu.Unlock()
return jsonResp(http.StatusOK, `{"ok":true}`), nil
}
s.t.Fatalf("unexpected request %s %s", r.Method, path)
return nil, nil
}
func newParallelServer(t *testing.T, resumable string) *parallelUploadServer {
return &parallelUploadServer{t: t, resumable: resumable, partDelay: 50 * time.Millisecond, parts: map[int]string{}}
}
const tenParts = `{"part_size":2,"total_parts":10}`
func TestUploadMultipartSendsPartsConcurrently(t *testing.T) {
srv := newParallelServer(t, tenParts)
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 4
content := "aabbccddeeffgghhiijj"
_, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, 4, srv.maxFlight, "parts after the first should use the whole concurrency")
assert.Empty(t, srv.earlyParts, "no part may start before part 1 has finished")
assert.Equal(t, 1, srv.completes)
for i := 1; i <= 10; i++ {
assert.Equal(t, content[2*(i-1):2*i], srv.parts[i], "part %d carries its own bytes", i)
}
}
func TestUploadMultipartConcurrencyOneIsSequential(t *testing.T) {
srv := newParallelServer(t, tenParts)
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 1
_, err := f.uploadFile(context.Background(), strings.NewReader("aabbccddeeffgghhiijj"), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, 1, srv.maxFlight)
assert.Equal(t, 1, srv.completes)
}
func TestUploadMultipartFailedPartIsNotCompleted(t *testing.T) {
srv := newParallelServer(t, tenParts)
srv.failPart = 6
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 4
_, err := f.uploadFile(context.Background(), strings.NewReader("aabbccddeeffgghhiijj"), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.Contains(t, err.Error(), "part 6")
assert.Equal(t, 0, srv.completes, "an upload with a failed part must not be completed")
}
func TestUploadMultipartParallelRejectsShortSource(t *testing.T) {
// 20 bytes declared, 15 supplied: the short read is found by the
// reader, whichever parts are already in flight, and nothing completes.
srv := newParallelServer(t, tenParts)
f := newFakeFs(rtFunc(srv.roundTrip))
f.opt.UploadConcurrency = 4
_, err := f.uploadFile(context.Background(), strings.NewReader("aabbccddeeffggh"), "file.txt", 20, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
assert.Equal(t, 0, srv.completes)
assert.Empty(t, srv.parts[8], "a part the source never filled must not be sent")
}
-15
View File
@@ -1,15 +0,0 @@
// Test Dosya filesystem interface
package dosya
import (
"testing"
"github.com/rclone/rclone/fstest/fstests"
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestDosya:",
})
}
-247
View File
@@ -1,247 +0,0 @@
package dosya
import (
"context"
"fmt"
"io"
"net/http"
"strings"
"sync"
"testing"
"time"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// rtFunc adapts a function into an http.RoundTripper so the upload path
// can be driven without a network. Unlike a real server it always reads
// the whole request body and answers, which is exactly the misbehaviour
// (accepting a short body as a complete upload) these tests guard against.
type rtFunc func(*http.Request) (*http.Response, error)
func (f rtFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
func jsonResp(status int, body string) *http.Response {
return &http.Response{
StatusCode: status,
Status: fmt.Sprintf("%d %s", status, http.StatusText(status)),
Header: http.Header{"Content-Type": {"application/json"}},
Body: io.NopCloser(strings.NewReader(body)),
ContentLength: int64(len(body)),
}
}
func newFakeFs(rt http.RoundTripper) *Fs {
ctx := context.Background()
f := &Fs{
name: "test",
opt: Options{WorkspaceID: "ws_test"},
pacer: fs.NewPacer(ctx, pacer.NewDefault(
pacer.MinSleep(time.Millisecond),
pacer.MaxSleep(time.Millisecond),
pacer.DecayConstant(decayConstant),
pacer.AttackConstant(attackConstant),
)),
}
f.rest = rest.NewClient(&http.Client{Transport: rt}).SetRoot("https://fake.invalid").SetErrorHandler(errorHandler)
return f
}
// fakeUploadServer records what an upload sends and lets a test inject
// failures per request
type fakeUploadServer struct {
t *testing.T
mu sync.Mutex
resumable string // JSON for the init "resumable" field, "" for a single PUT
puts [][]byte // bodies of PUT /api/upload/<id>
parts map[int][][]byte // bodies of PUT /api/upload/<id>/part/<n>, per part, per attempt
completes int
aborts int // DELETE /api/upload/<id>
failFirst map[string]bool // path -> answer the first request with a 502
seenBefore map[string]bool
}
func newFakeUploadServer(t *testing.T, resumable string) *fakeUploadServer {
return &fakeUploadServer{
t: t,
resumable: resumable,
parts: map[int][][]byte{},
failFirst: map[string]bool{},
seenBefore: map[string]bool{},
}
}
func (s *fakeUploadServer) roundTrip(r *http.Request) (*http.Response, error) {
s.mu.Lock()
defer s.mu.Unlock()
var body []byte
if r.Body != nil {
var err error
body, err = io.ReadAll(r.Body)
require.NoError(s.t, err)
}
path := r.URL.Path
// Bodies are recorded for every attempt, including the ones answered
// with a 502, so tests can see exactly what each attempt carried.
fail := s.failFirst[path] && !s.seenBefore[path]
s.seenBefore[path] = true
bad := jsonResp(http.StatusBadGateway, `{"ok":false,"error":"bad gateway"}`)
switch {
case r.Method == "POST" && path == "/api/upload/init":
if s.resumable != "" {
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1","resumable":`+s.resumable+`}`), nil
}
return jsonResp(http.StatusCreated, `{"ok":true,"session_id":"upl_1"}`), nil
case r.Method == "PUT" && path == "/api/upload/upl_1":
s.puts = append(s.puts, body)
if fail {
return bad, nil
}
return jsonResp(http.StatusCreated, fmt.Sprintf(`{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":%d}}`, len(body))), nil
case r.Method == "PUT" && strings.HasPrefix(path, "/api/upload/upl_1/part/"):
var n int
_, err := fmt.Sscanf(strings.TrimPrefix(path, "/api/upload/upl_1/part/"), "%d", &n)
require.NoError(s.t, err)
s.parts[n] = append(s.parts[n], body)
if fail {
return bad, nil
}
return jsonResp(http.StatusOK, fmt.Sprintf(`{"ok":true,"part_number":%d,"bytes_uploaded":%d}`, n, len(body))), nil
case r.Method == "POST" && path == "/api/upload/upl_1/complete":
s.completes++
return jsonResp(http.StatusOK, `{"ok":true,"file":{"id":"fil_1","name":"file.txt","size_bytes":10}}`), nil
case r.Method == "DELETE" && path == "/api/upload/upl_1":
s.aborts++
return jsonResp(http.StatusOK, `{"ok":true}`), nil
}
s.t.Fatalf("unexpected request %s %s", r.Method, path)
return nil, nil
}
func TestUploadSmallSendsWholeBody(t *testing.T) {
srv := newFakeUploadServer(t, "")
f := newFakeFs(rtFunc(srv.roundTrip))
content := strings.Repeat("a", 100)
resp, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 100, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, "fil_1", resp.File.ID)
require.Len(t, srv.puts, 1)
assert.Equal(t, content, string(srv.puts[0]))
}
func TestUploadSmallRejectsShortSource(t *testing.T) {
// The source declares 100 bytes but only supplies 60. A server which
// accepts that as a complete upload must not turn into a success.
srv := newFakeUploadServer(t, "")
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader(strings.Repeat("a", 60)), "file.txt", 100, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
assert.Contains(t, err.Error(), "expected 100 bytes in input, but got 60")
}
func TestUploadSmallDoesNotResendDrainedBody(t *testing.T) {
// A retried PUT would resend an already-consumed reader, i.e. an
// empty body, which the server would store as an empty file. The
// single PUT therefore must not be retried at the low level.
srv := newFakeUploadServer(t, "")
srv.failFirst["/api/upload/upl_1"] = true
f := newFakeFs(rtFunc(srv.roundTrip))
content := strings.Repeat("a", 100)
_, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 100, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
require.Len(t, srv.puts, 1, "the PUT must not be retried with a drained body")
assert.Equal(t, content, string(srv.puts[0]))
}
const threeParts = `{"part_size":4,"total_parts":3}`
func TestUploadMultipartAssemblesAllBytes(t *testing.T) {
srv := newFakeUploadServer(t, threeParts)
f := newFakeFs(rtFunc(srv.roundTrip))
content := "0123456789"
resp, err := f.uploadFile(context.Background(), strings.NewReader(content), "file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, "fil_1", resp.File.ID)
assert.Equal(t, 1, srv.completes)
require.Len(t, srv.parts, 3)
assert.Equal(t, "0123", string(srv.parts[1][0]))
assert.Equal(t, "4567", string(srv.parts[2][0]))
assert.Equal(t, "89", string(srv.parts[3][0]))
}
func TestUploadMultipartRejectsShortSource(t *testing.T) {
// 10 bytes declared, 8 supplied: the last part must never be sent
// and the upload must not be completed.
srv := newFakeUploadServer(t, threeParts)
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader("01234567"), "file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.ErrorIs(t, err, io.ErrUnexpectedEOF)
assert.Contains(t, err.Error(), "expected 10 bytes in input, but got 8")
assert.Empty(t, srv.parts[3], "short final part must not be uploaded")
assert.Equal(t, 0, srv.completes, "upload must not be completed")
}
func TestUploadMultipartRetriesPartWithFullBody(t *testing.T) {
// A part which fails transiently must be resent with all its bytes,
// not with whatever is left of a consumed reader.
srv := newFakeUploadServer(t, threeParts)
srv.failFirst["/api/upload/upl_1/part/2"] = true
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader("0123456789"), "file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
require.Len(t, srv.parts[2], 2, "part 2 should have been sent twice")
assert.Equal(t, "4567", string(srv.parts[2][0]))
assert.Equal(t, "4567", string(srv.parts[2][1]))
assert.Equal(t, "89", string(srv.parts[3][0]))
assert.Equal(t, 1, srv.completes)
}
// An upload that fails after init has to hand its session back: the session
// holds one of the workspace's upload slots for ten minutes otherwise, and
// rclone's high-level retry opens a new one rather than resuming it.
func TestUploadAbortsTheSessionOnFailure(t *testing.T) {
for _, tc := range []struct {
name string
resumable string
content string
size int64
}{
{"small file short source", "", "0123", 10},
{"multipart short source", `{"part_size":5,"total_parts":4}`, "aabbccddeeffggh", 20},
} {
t.Run(tc.name, func(t *testing.T) {
srv := newFakeUploadServer(t, tc.resumable)
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader(tc.content),
"file.txt", tc.size, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.Error(t, err)
assert.Equal(t, 1, srv.aborts)
})
}
}
// A successful upload must not abort anything
func TestUploadDoesNotAbortOnSuccess(t *testing.T) {
srv := newFakeUploadServer(t, "")
f := newFakeFs(rtFunc(srv.roundTrip))
_, err := f.uploadFile(context.Background(), strings.NewReader(strings.Repeat("a", 10)),
"file.txt", 10, time.Time{}, "text/plain; charset=utf-8", "", nil)
require.NoError(t, err)
assert.Equal(t, 0, srv.aborts)
}
@@ -1,73 +0,0 @@
package dosya
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Every by-id call names the workspace, so the API can find an id it minted
// moments ago before its file_home index row lands.
func TestByIDCallsSendWorkspaceHint(t *testing.T) {
seen := map[string]string{}
f := newFakeFs(rtFunc(func(r *http.Request) (*http.Response, error) {
seen[r.Method+" "+r.URL.Path] = r.Header.Get("X-Dosya-Workspace")
switch r.URL.Path {
case "/api/folders/fld_1":
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true,"complete":true}`), nil
case "/api/files/fil_1":
return jsonResp(http.StatusOK, `{"ok":true,"permanent":true}`), nil
case "/api/folders/fld_1/move":
return jsonResp(http.StatusOK, `{"ok":true,"name":"n"}`), nil
default:
return jsonResp(http.StatusOK, `{"ok":true}`), nil
}
}))
ctx := context.Background()
require.NoError(t, f.removeFolder(ctx, "fld_1", false))
require.NoError(t, f.renameFolder(ctx, "fld_1", "n"))
_, err := f.moveFolder(ctx, "fld_1", "fld_2", "n")
require.NoError(t, err)
require.NoError(t, f.deleteFile(ctx, "fil_1"))
require.NoError(t, f.renameFile(ctx, "fil_1", "n"))
require.NoError(t, f.moveFile(ctx, "fil_1", "fld_2", "n"))
_, err = f.copyFile(ctx, "fil_1", "fld_2", "n")
require.NoError(t, err)
_, err = f.createShareLink(ctx, "fil_1", 0)
require.NoError(t, err)
for call, header := range seen {
assert.Equal(t, "ws_test", header, call)
}
assert.Len(t, seen, 8)
}
// The download's API request carries the hint; the presigned R2 fetch must not.
func TestDownloadSendsWorkspaceHintOnlyToTheAPI(t *testing.T) {
var apiHint, blobHint string
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/api/files/fil_1/download" {
apiHint = r.Header.Get("X-Dosya-Workspace")
http.Redirect(w, r, srv.URL+"/blob", http.StatusFound)
return
}
blobHint = r.Header.Get("X-Dosya-Workspace")
_, _ = w.Write([]byte("bytes"))
}))
defer srv.Close()
f := newTestFs(srv)
f.opt.APIURL = srv.URL
rc, err := f.downloadFile(context.Background(), "fil_1", nil)
require.NoError(t, err)
_ = rc.Close()
assert.Equal(t, "ws_test", apiHint)
assert.Equal(t, "", blobHint)
}
-52
View File
@@ -1,52 +0,0 @@
package dosya
import (
"context"
"net/http"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// fakeWorkspaceServer answers GET /api/workspaces with the given body
func fakeWorkspaceServer(t *testing.T, body string, calls *int) rtFunc {
return func(r *http.Request) (*http.Response, error) {
require.Equal(t, "/api/workspaces", r.URL.Path)
*calls++
return jsonResp(http.StatusOK, body), nil
}
}
func TestFindWorkspaceAdoptsTheOnlyOne(t *testing.T) {
calls := 0
f := newFakeFs(fakeWorkspaceServer(t, `{"ok":true,"workspaces":[{"id":"ws_only","name":"Mine"}]}`, &calls))
f.opt.WorkspaceID = ""
require.NoError(t, f.findWorkspace(context.Background()))
assert.Equal(t, "ws_only", f.opt.WorkspaceID)
assert.Equal(t, 1, calls)
}
func TestFindWorkspaceListsTheChoices(t *testing.T) {
calls := 0
f := newFakeFs(fakeWorkspaceServer(t, `{"ok":true,"workspaces":[`+
`{"id":"ws_one","name":"One"},{"id":"ws_two","name":"Two"}]}`, &calls))
f.opt.WorkspaceID = ""
err := f.findWorkspace(context.Background())
require.Error(t, err)
// the ids have to be in the message: they are the only place the user
// can read them, which is the whole point of listing them
assert.Contains(t, err.Error(), "ws_one (One)")
assert.Contains(t, err.Error(), "ws_two (Two)")
assert.Empty(t, f.opt.WorkspaceID)
}
func TestFindWorkspaceWithNone(t *testing.T) {
calls := 0
f := newFakeFs(fakeWorkspaceServer(t, `{"ok":true,"workspaces":[]}`, &calls))
f.opt.WorkspaceID = ""
require.ErrorContains(t, f.findWorkspace(context.Background()), "no workspaces found")
}
+18 -42
View File
@@ -321,13 +321,10 @@ func (f *Fs) getItem(ctx context.Context, id string, dirID string, leaf string)
}
return false
})
if err != nil {
return nil, err
}
if !found {
return nil, fs.ErrorObjectNotFound
}
return info, nil
return info, err
}
// errorHandler parses a non 2xx error response into an error
@@ -549,11 +546,6 @@ OUTER:
if err != nil {
return found, fmt.Errorf("couldn't list files: %w", err)
}
// Drime won't page past a fixed number of entries (20,000 for
// rclone) and serves the last page it allows again instead
if result.CurrentPage != page {
return found, errors.New("couldn't list files: directory has more entries than drime will list")
}
for _, item := range result.Data {
if item.Type == api.ItemTypeFolder {
if filesOnly {
@@ -700,27 +692,13 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
// deleteObject removes an object by ID
func (f *Fs) deleteObject(ctx context.Context, id string) error {
if f.opt.HardDelete {
// Deleting forever doesn't invalidate the server's cache of
// the parent folder listing, so the entry keeps being listed
// for about a second. Moving to the trash first does.
err := f.deleteEntry(ctx, id, false)
if err != nil {
return err
}
}
return f.deleteEntry(ctx, id, f.opt.HardDelete)
}
// deleteEntry moves the entry with ID to the trash or deletes it forever
func (f *Fs) deleteEntry(ctx context.Context, id string, deleteForever bool) error {
opts := rest.Opts{
Method: "POST",
Path: "/file-entries/delete",
}
request := api.DeleteRequest{
EntryIDs: []string{id},
DeleteForever: deleteForever,
DeleteForever: f.opt.HardDelete,
}
var result api.DeleteResponse
err := f.pacer.Call(func() (bool, error) {
@@ -1062,28 +1040,17 @@ func (f *Fs) copy(ctx context.Context, id, newDirID string) (item *api.Item, err
}
// copy and rename a file or folder to directoryID with leaf
//
// If existing is set it is removed once the copy has succeeded.
func (f *Fs) copyTo(ctx context.Context, srcID, dstLeaf, dstDirectoryID string, existing fs.Object) (info *api.Item, err error) {
func (f *Fs) copyTo(ctx context.Context, srcID, srcLeaf, dstLeaf, dstDirectoryID string) (info *api.Item, err error) {
// Can have duplicates so don't have to be careful here
// Copy to dstDirectoryID first
info, err = f.copy(ctx, srcID, dstDirectoryID)
if err != nil {
return nil, err
}
// The server names the copy "name (1)" if the destination
// directory already has an entry called "name" and refuses to
// rename it to "name" until that entry is removed.
if existing != nil {
fs.Debugf(existing, "Server side copy: removing existing object after successful copy")
err = existing.Remove(ctx)
if err != nil {
return nil, err
}
}
// Rename if required
if info.Name != f.opt.Enc.FromStandardName(dstLeaf) {
if srcLeaf != dstLeaf {
info, err = f.rename(ctx, info.ID.String(), dstLeaf)
if err != nil {
return nil, err
@@ -1107,6 +1074,8 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (dst fs.Obj
fs.Debugf(src, "Can't copy - not same remote type")
return nil, fs.ErrorCantCopy
}
srcLeaf := path.Base(srcObj.remote)
srcPath := srcObj.fs.rootSlash() + srcObj.remote
dstPath := f.rootSlash() + remote
if srcPath == dstPath {
@@ -1115,8 +1084,15 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (dst fs.Obj
// Find existing object
existingObj, err := f.NewObject(ctx, remote)
if err != nil {
existingObj = nil
if err == nil {
defer func() {
// Don't remove existing object if returning an error
if err != nil {
return
}
fs.Debugf(existingObj, "Server side copy: removing existing object after successful copy")
err = existingObj.Remove(ctx)
}()
}
// Create temporary object
@@ -1126,7 +1102,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (dst fs.Obj
}
// Copy the object
info, err := f.copyTo(ctx, srcObj.id, dstLeaf, dstDirectoryID, existingObj)
info, err := f.copyTo(ctx, srcObj.id, srcLeaf, dstLeaf, dstDirectoryID)
if err != nil {
return nil, err
}
-158
View File
@@ -1,158 +0,0 @@
package drime
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strconv"
"sync/atomic"
"testing"
"time"
"github.com/rclone/rclone/backend/drime/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const (
testPerPage = 5 // entries per page the test server returns
testLimit = 20 // entries the test server will page through
)
// listingServer serves /drive/file-entries from entries the way Drime
// does.
//
// It returns testPerPage entries per page and won't page past testLimit
// entries. Asking for a later page returns the last page it allows
// again, with last_page one beyond it.
//
// It counts the requests made in requests.
func listingServer(t *testing.T, entries []api.Item, requests *atomic.Int64) *httptest.Server {
t.Helper()
return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
assert.Equal(t, "/drive/file-entries", r.URL.Path)
params := r.URL.Query()
page, err := strconv.Atoi(params.Get("page"))
require.NoError(t, err)
maxPage := testLimit / testPerPage
currentPage := min(page, maxPage)
start := min((currentPage-1)*testPerPage, len(entries))
end := min(start+testPerPage, len(entries))
lastPage := currentPage
if page > maxPage || end < len(entries) {
lastPage = currentPage + 1
}
w.Header().Set("Content-Type", "application/json")
require.NoError(t, json.NewEncoder(w).Encode(map[string]any{
"current_page": currentPage,
"last_page": lastPage,
"per_page": testPerPage,
"total": (currentPage + 1) * testPerPage,
"data": entries[start:end],
}))
}))
}
// makeEntries makes n file entries
func makeEntries(n int) []api.Item {
entries := make([]api.Item, n)
for i := range entries {
entries[i] = api.Item{
ID: json.Number(strconv.Itoa(1000 + i)),
Name: fmt.Sprintf("file%05d", i),
Type: "text",
}
}
return entries
}
// testFs makes an Fs which uses server as its API
func testFs(t *testing.T, server *httptest.Server) *Fs {
t.Helper()
f := &Fs{
opt: Options{ListChunk: 1000},
srv: rest.NewClient(server.Client()).SetRoot(server.URL),
}
f.pacer = fs.NewPacer(context.Background(), pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant)))
f.srv.SetErrorHandler(errorHandler)
return f
}
// listNames lists a directory with listAll and returns the names found
func listNames(ctx context.Context, f *Fs) (names []string, err error) {
_, err = f.listAll(ctx, "1", false, false, "", func(item *api.Item) bool {
names = append(names, item.Name)
return false
})
return names, err
}
// TestListAll checks that listAll returns every entry exactly once
func TestListAll(t *testing.T) {
for _, test := range []struct {
name string
entries int
}{
{name: "Empty", entries: 0},
{name: "OnePage", entries: 3},
{name: "ExactPages", entries: 10},
{name: "PartialLastPage", entries: 13},
{name: "AtLimit", entries: testLimit},
} {
t.Run(test.name, func(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var requests atomic.Int64
entries := makeEntries(test.entries)
server := listingServer(t, entries, &requests)
defer server.Close()
f := testFs(t, server)
names, err := listNames(ctx, f)
require.NoError(t, err)
var want []string
for _, entry := range entries {
want = append(want, entry.Name)
}
assert.ElementsMatch(t, want, names)
})
}
}
// TestListAllTooManyEntries checks that listAll returns an error for a
// directory with more entries than the server will list, rather than
// looping forever.
func TestListAllTooManyEntries(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
var requests atomic.Int64
server := listingServer(t, makeEntries(2*testLimit), &requests)
defer server.Close()
f := testFs(t, server)
_, err := listNames(ctx, f)
require.Error(t, err)
assert.NoError(t, ctx.Err(), "listing didn't finish")
assert.LessOrEqual(t, requests.Load(), int64(testLimit/testPerPage+1))
}
// TestGetItemListError checks that getItem returns listing errors
// rather than fs.ErrorObjectNotFound.
func TestGetItemListError(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, `{"message":"access denied"}`, http.StatusForbidden)
}))
defer server.Close()
f := testFs(t, server)
_, err := f.getItem(context.Background(), "1000", "1", "file00000")
require.Error(t, err)
assert.NotErrorIs(t, err, fs.ErrorObjectNotFound)
assert.Contains(t, err.Error(), "access denied")
}
+9 -22
View File
@@ -568,7 +568,7 @@ recommended usage is using the flag form --drive-size-as-quota when
doing rclone ls/lsl/lsf/lsjson/etc only.
If you do use this flag for syncing (not recommended) then you will
need to use --ignore-size also.`,
need to use --ignore size also.`,
Advanced: true,
Hide: fs.OptionHideConfigurator,
}, {
@@ -2298,9 +2298,15 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
wg.Wait()
// if the input channel overflowed add the collected entries to the channel now
for len(overflow) > 0 {
if !refillListRInput(&mu, &in, &overflow, &wg) {
return
mu.Lock()
// only fill half of the channel to prevent entries being put into overflow again
l := min(len(overflow), listRInputBuffer/2)
wg.Add(l)
for _, d := range overflow[:l] {
in <- d
}
overflow = overflow[l:]
mu.Unlock()
// wait again for the completion of all directories
wg.Wait()
@@ -2348,25 +2354,6 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
return nil
}
// refillListRInput moves queued directory jobs back onto in.
// It returns false when a listing error has already closed in.
// Sending on that nil channel would block forever while holding mu.
func refillListRInput(mu *sync.Mutex, in *chan listREntry, overflow *[]listREntry, wg *sync.WaitGroup) bool {
mu.Lock()
defer mu.Unlock()
if *in == nil {
return false
}
// only fill half of the channel to prevent entries being put into overflow again
l := min(len(*overflow), listRInputBuffer/2)
wg.Add(l)
for _, d := range (*overflow)[:l] {
*in <- d
}
*overflow = (*overflow)[l:]
return true
}
const shortcutSeparator = '\t'
// joinID adds an actual drive ID to the shortcut ID it came from
-152
View File
@@ -8,14 +8,10 @@ import (
"fmt"
"io"
"mime"
"net/http"
"net/http/httptest"
"os"
"path"
"path/filepath"
"strings"
stdsync "sync"
"sync/atomic"
"testing"
"time"
@@ -29,113 +25,13 @@ import (
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"google.golang.org/api/drive/v3"
"google.golang.org/api/googleapi"
"google.golang.org/api/option"
)
func newPermissionTestFs(t *testing.T, handler http.Handler) *Fs {
t.Helper()
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
service, err := drive.NewService(
context.Background(),
option.WithHTTPClient(server.Client()),
option.WithEndpoint(server.URL+"/"),
)
require.NoError(t, err)
return &Fs{
svc: service,
pacer: fs.NewPacer(context.Background(), pacer.NewGoogleDrive()),
permissionsMu: new(stdsync.Mutex),
permissions: make(map[string]*drive.Permission),
}
}
func TestInternalGetPermissionConcurrent(t *testing.T) {
const requestCount = 2
arrived := make(chan string, requestCount)
release := make(chan struct{})
var releaseOnce stdsync.Once
releaseRequests := func() {
releaseOnce.Do(func() { close(release) })
}
defer releaseRequests()
f := newPermissionTestFs(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
permissionID := path.Base(r.URL.Path)
arrived <- permissionID
<-release
w.Header().Set("Content-Type", "application/json")
_, err := fmt.Fprintf(w, `{"id":%q,"role":"reader","type":"user"}`, permissionID)
assert.NoError(t, err)
}))
type permissionResult struct {
permission *drive.Permission
inherited bool
err error
}
results := make(chan permissionResult, requestCount)
permissionIDs := []string{"permission-1", "permission-2"}
for _, permissionID := range permissionIDs {
go func() {
permission, inherited, err := f.getPermission(context.Background(), "file-id", permissionID, false)
results <- permissionResult{permission: permission, inherited: inherited, err: err}
}()
}
arrivedIDs := make(map[string]bool, requestCount)
timer := time.NewTimer(time.Second)
defer timer.Stop()
for len(arrivedIDs) < requestCount {
select {
case permissionID := <-arrived:
arrivedIDs[permissionID] = true
case <-timer.C:
releaseRequests()
t.Fatalf("timed out waiting for concurrent permission requests: got %d of %d", len(arrivedIDs), requestCount)
}
}
releaseRequests()
assert.Equal(t, map[string]bool{"permission-1": true, "permission-2": true}, arrivedIDs)
returnedIDs := make(map[string]bool, requestCount)
for range requestCount {
result := <-results
require.NoError(t, result.err)
require.NotNil(t, result.permission)
assert.False(t, result.inherited)
returnedIDs[result.permission.Id] = true
}
assert.Equal(t, arrivedIDs, returnedIDs)
}
func TestInternalGetPermissionCacheHit(t *testing.T) {
var requests atomic.Int32
f := newPermissionTestFs(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
permissionID := path.Base(r.URL.Path)
w.Header().Set("Content-Type", "application/json")
_, err := fmt.Fprintf(w, `{"id":%q,"role":"reader","type":"user"}`, permissionID)
assert.NoError(t, err)
}))
first, inherited, err := f.getPermission(context.Background(), "file-id", "permission-1", true)
require.NoError(t, err)
assert.False(t, inherited)
second, inherited, err := f.getPermission(context.Background(), "file-id", "permission-1", true)
require.NoError(t, err)
assert.False(t, inherited)
assert.Same(t, first, second)
assert.Equal(t, int32(1), requests.Load())
}
func TestDriveScopes(t *testing.T) {
for _, test := range []struct {
in string
@@ -916,51 +812,3 @@ func (f *Fs) InternalTest(t *testing.T) {
}
var _ fstests.InternalTester = (*Fs)(nil)
func TestRefillListRInputStopsWhenInputClosed(t *testing.T) {
mu := stdsync.Mutex{}
wg := stdsync.WaitGroup{}
in := make(chan listREntry, listRInputBuffer)
overflow := []listREntry{{id: "folder", path: "folder"}}
mu.Lock()
close(in)
in = nil
mu.Unlock()
done := make(chan bool, 1)
go func() {
done <- refillListRInput(&mu, &in, &overflow, &wg)
}()
select {
case ok := <-done:
require.False(t, ok, "refill continued after the input channel was closed")
case <-time.After(time.Second):
t.Fatal("refill blocked after the input channel was closed")
}
require.Len(t, overflow, 1)
finished := make(chan struct{})
go func() {
wg.Wait()
close(finished)
}()
select {
case <-finished:
case <-time.After(time.Second):
t.Fatal("refill incremented the wait group after the input channel was closed")
}
}
func TestRefillListRInputSendsQueuedDirectory(t *testing.T) {
mu := stdsync.Mutex{}
wg := stdsync.WaitGroup{}
in := make(chan listREntry, listRInputBuffer)
overflow := []listREntry{{id: "folder", path: "folder"}}
require.True(t, refillListRInput(&mu, &in, &overflow, &wg), "refill stopped while the input channel was open")
require.Empty(t, overflow)
got := <-in
require.Equal(t, listREntry{id: "folder", path: "folder"}, got)
wg.Done()
}
+2 -4
View File
@@ -106,10 +106,10 @@ var permissionsFields = googleapi.Field(strings.Join([]string{
// getPermission returns permissions for the fileID and permissionID passed in
func (f *Fs) getPermission(ctx context.Context, fileID, permissionID string, useCache bool) (perm *drive.Permission, inherited bool, err error) {
f.permissionsMu.Lock()
defer f.permissionsMu.Unlock()
if useCache {
f.permissionsMu.Lock()
perm = f.permissions[permissionID]
f.permissionsMu.Unlock()
if perm != nil {
return perm, false, nil
}
@@ -131,9 +131,7 @@ func (f *Fs) getPermission(ctx context.Context, fileID, permissionID string, use
cleanPermission(perm)
// cache the permission
f.permissionsMu.Lock()
f.permissions[permissionID] = perm
f.permissionsMu.Unlock()
return perm, inherited, err
}
+18 -30
View File
@@ -11,6 +11,7 @@
package drive
import (
"bytes"
"context"
"encoding/json"
"fmt"
@@ -21,7 +22,6 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/readers"
"google.golang.org/api/drive/v3"
"google.golang.org/api/googleapi"
@@ -112,13 +112,8 @@ func (f *Fs) Upload(ctx context.Context, in io.Reader, size int64, contentType,
}
// Make an http.Request for the range passed in
//
// The body is wrapped in readers.NoCloser so the transport can't upgrade it
// to an io.Closer and close it after each attempt — the chunk buffer is
// pool-backed, so an early close would free its pages and a retry would then
// read a dead buffer. The upload loop owns the buffer's lifetime.
func (rx *resumableUpload) makeRequest(ctx context.Context, start int64, body io.ReadSeeker, reqSize int64) *http.Request {
req, _ := http.NewRequestWithContext(ctx, "POST", rx.URI, readers.NoCloser(body))
req, _ := http.NewRequestWithContext(ctx, "POST", rx.URI, body)
req.ContentLength = reqSize
totalSize := "*"
if rx.ContentLength >= 0 {
@@ -173,40 +168,37 @@ func (rx *resumableUpload) Upload(ctx context.Context) (*drive.File, error) {
start := int64(0)
var StatusCode int
var err error
chunkSize := int64(rx.f.opt.ChunkSize)
buf := make([]byte, int(rx.f.opt.ChunkSize))
for finished := false; !finished; {
reqSize := chunkSize
var reqSize int64
var chunk io.ReadSeeker
if rx.ContentLength >= 0 {
// If size known use repeatable reader for smoother bwlimit
if start >= rx.ContentLength {
break
}
reqSize = min(rx.ContentLength-start, chunkSize)
}
// Buffer the chunk in memory from the global pool so reads are
// repeatable for retries
rw := multipart.NewRW()
var n int64
n, err = io.CopyN(rw, rx.Media, reqSize)
if rx.ContentLength < 0 {
reqSize = min(rx.ContentLength-start, int64(rx.f.opt.ChunkSize))
chunk = readers.NewRepeatableLimitReaderBuffer(rx.Media, buf, reqSize)
} else {
// If size unknown read into buffer
var n int
n, err = readers.ReadFill(rx.Media, buf)
if err == io.EOF {
// Send the last chunk with the correct ContentLength
// otherwise Google doesn't know we've finished
rx.ContentLength = start + n
rx.ContentLength = start + int64(n)
finished = true
err = nil
} else if err != nil {
return nil, err
}
reqSize = n
}
if err != nil {
_ = rw.Close()
return nil, err
reqSize = int64(n)
chunk = bytes.NewReader(buf[:reqSize])
}
// Transfer the chunk
err = rx.f.pacer.Call(func() (bool, error) {
fs.Debugf(rx.remote, "Sending chunk %d length %d", start, reqSize)
StatusCode, err = rx.transferChunk(ctx, start, rw, reqSize)
StatusCode, err = rx.transferChunk(ctx, start, chunk, reqSize)
again, err := rx.f.shouldRetry(ctx, err)
if StatusCode == statusResumeIncomplete || StatusCode == http.StatusCreated || StatusCode == http.StatusOK {
again = false
@@ -214,13 +206,9 @@ func (rx *resumableUpload) Upload(ctx context.Context) (*drive.File, error) {
}
return again, err
})
closeErr := rw.Close()
if err != nil {
return nil, err
}
if closeErr != nil {
return nil, closeErr
}
start += reqSize
}
-57
View File
@@ -1,57 +0,0 @@
package drive
import (
"bytes"
"context"
"io"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/pacer"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestResumableUploadRetry checks that a chunk which fails with a 5xx is
// retried successfully. The chunk buffer is pool-backed and implements
// io.Closer, so if it reaches the http transport unwrapped the transport
// closes it after the failed attempt, returning its pages to the pool, and
// the retry then reads a freed buffer.
func TestResumableUploadRetry(t *testing.T) {
content := bytes.Repeat([]byte("resumable"), 512)
var attempts atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(r.Body)
require.NoError(t, err)
if attempts.Add(1) == 1 {
w.WriteHeader(http.StatusInternalServerError)
return
}
assert.Equal(t, content, body, "retried chunk should re-send the full chunk")
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"id":"fake-id","name":"remote"}`))
}))
defer server.Close()
f := &Fs{
pacer: fs.NewPacer(context.Background(), pacer.NewGoogleDrive()),
client: server.Client(),
}
f.opt.ChunkSize = fs.SizeSuffix(len(content))
rx := &resumableUpload{
f: f,
remote: "remote",
URI: server.URL,
Media: bytes.NewReader(content),
MediaType: "application/octet-stream",
ContentLength: int64(len(content)),
}
info, err := rx.Upload(context.Background())
require.NoError(t, err)
require.NotNil(t, info)
assert.Equal(t, "fake-id", info.Id)
assert.Equal(t, int32(2), attempts.Load(), "expected exactly one failed attempt and one retry")
}
+18 -28
View File
@@ -51,7 +51,6 @@ import (
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/lib/batcher"
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/readers"
@@ -634,8 +633,13 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
return f, nil // our root it empty so we probably want to list shared folders
}
dir := path.Dir(f.root)
if dir == "." {
dir = f.root
}
// root is not empty so we have find the right shared folder if it exists
id, err := f.findSharedFolder(ctx, sharedFolderName(f.root))
id, err := f.findSharedFolder(ctx, dir)
if err != nil {
// if we didn't find the specified shared folder we have to bail out here
return nil, err
@@ -943,21 +947,13 @@ func (f *Fs) listSharedFolders(ctx context.Context, callback func(fs.DirEntry) e
return nil
}
// sharedFolderName returns the shared folder name in root, which is its first
// path component. root must be the trimmed root as produced by setRoot (no
// leading slash).
func sharedFolderName(root string) string {
name, _, _ := strings.Cut(root, "/")
return name
}
// findSharedFolder find the id for a given shared folder name
// somewhat annoyingly there is no endpoint to query a shared folder by it's name
// so our only option is to iterate over all shared folders
func (f *Fs) findSharedFolder(ctx context.Context, name string) (id string, err error) {
errFoundFile := errors.New("found file")
err = f.listSharedFolders(ctx, func(entry fs.DirEntry) error {
if strings.EqualFold(entry.(*fs.Dir).Remote(), name) {
if entry.(*fs.Dir).Remote() == name {
id = entry.(*fs.Dir).ID()
return errFoundFile
}
@@ -1039,7 +1035,7 @@ func (f *Fs) listReceivedFiles(ctx context.Context, callback func(fs.DirEntry) e
func (f *Fs) findSharedFile(ctx context.Context, name string) (o *Object, err error) {
errFoundFile := errors.New("found file")
err = f.listReceivedFiles(ctx, func(entry fs.DirEntry) error {
if strings.EqualFold(entry.(*Object).remote, name) {
if entry.(*Object).remote == name {
o = entry.(*Object)
return errFoundFile
}
@@ -2083,6 +2079,11 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
// write chunks
in := readers.NewCountingReader(in0)
bufSize := chunkSize
if size >= 0 && size < bufSize {
bufSize = size
}
buf := make([]byte, int(bufSize))
cursor := files.UploadSessionCursor{
SessionId: res.SessionId,
Offset: 0,
@@ -2102,21 +2103,14 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
fs.Debugf(o, "Uploading chunk %d/%d", currentChunk, chunks)
}
// Buffer the chunk in memory from the global pool for retries
rw := multipart.NewRW()
var n int64
n, err = io.CopyN(rw, in, chunkSize)
if err != nil && err != io.EOF {
_ = rw.Close()
return nil, err
}
chunk := readers.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
skip := int64(0)
err = o.fs.pacer.Call(func() (bool, error) {
// seek to the start in case this is a retry
if _, err = rw.Seek(skip, io.SeekStart); err != nil {
if _, err = chunk.Seek(skip, io.SeekStart); err != nil {
return false, err
}
err = o.fs.srv.UploadSessionAppendV2Context(ctx, &appendArg, rw)
err = o.fs.srv.UploadSessionAppendV2Context(ctx, &appendArg, chunk)
// after session is started, we retry everything
if err != nil {
// Check for incorrect offset error and retry with new offset
@@ -2128,10 +2122,10 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
what := fmt.Sprintf("incorrect offset error received: sent %d, need %d, skip %d", cursor.Offset, correctOffset, skip)
if skip < 0 {
return false, fmt.Errorf("can't seek backwards to correct offset: %s", what)
} else if skip == n {
} else if skip == chunkSize {
fs.Debugf(o, "%s: chunk received OK - continuing", what)
return false, nil
} else if skip > n {
} else if skip > chunkSize {
// This error should never happen
return false, fmt.Errorf("can't seek forwards by more than a chunk to correct offset: %s", what)
}
@@ -2148,13 +2142,9 @@ func (o *Object) uploadChunked(ctx context.Context, in0 io.Reader, commitInfo *f
}
return err != nil, err
})
closeErr := rw.Close()
if err != nil {
return nil, err
}
if closeErr != nil {
return nil, closeErr
}
if size >= 0 {
// Check for sources which truncate early
expected := min(uint64(currentChunk)*uint64(chunkSize), uint64(size))
-60
View File
@@ -188,21 +188,6 @@ func TestInternalCheckPathLength(t *testing.T) {
}
}
func TestInternalSharedFolderName(t *testing.T) {
for _, test := range []struct {
root string
want string
}{
{root: "", want: ""},
{root: "SharedFolder", want: "SharedFolder"},
{root: "SharedFolder/subdir", want: "SharedFolder"},
{root: "SharedFolder/subdir/deeper", want: "SharedFolder"},
{root: "SharedFolder/subdir/deeper/deepest", want: "SharedFolder"},
} {
assert.Equal(t, test.want, sharedFolderName(test.root), test.root)
}
}
func TestPaperExportRemote(t *testing.T) {
ctx := context.Background()
info := &files.FileMetadata{
@@ -490,48 +475,3 @@ func TestFindSharedFileResolvesDecodedName(t *testing.T) {
require.NoError(t, err)
assert.Equal(t, encodedName, o.remote)
}
// sharedFoldersHandler serves a single shared folder from list_folders.
func sharedFoldersHandler(name, id string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.URL.Path, "list_folders") {
http.NotFound(w, r)
return
}
resp := sharing.ListFoldersResult{
Entries: []*sharing.SharedFolderMetadata{{Name: name, SharedFolderId: id}},
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(resp)
}
}
// The Dropbox backend advertises CaseInsensitive: true, so a shared folder
// lookup must match regardless of the case of the requested name.
func TestInternalFindSharedFolderCaseInsensitive(t *testing.T) {
f := newSharingTestFs(t, sharedFoldersHandler("TestFolder", "folder-id"))
for _, name := range []string{"TestFolder", "testfolder", "TESTFOLDER"} {
id, err := f.findSharedFolder(context.Background(), name)
require.NoError(t, err, name)
assert.Equal(t, "folder-id", id, name)
}
_, err := f.findSharedFolder(context.Background(), "no-such-folder")
assert.ErrorIs(t, err, fs.ErrorDirNotFound)
}
// The Dropbox backend advertises CaseInsensitive: true, so a received file
// lookup must match regardless of the case of the requested name.
func TestInternalFindSharedFileCaseInsensitive(t *testing.T) {
f := newSharingTestFs(t, receivedFilesHandler(t, "TestFile.txt"))
for _, name := range []string{"TestFile.txt", "testfile.txt", "TESTFILE.TXT"} {
o, err := f.findSharedFile(context.Background(), name)
require.NoError(t, err, name)
assert.Equal(t, "TestFile.txt", o.remote, name)
}
_, err := f.findSharedFile(context.Background(), "no-such-file.txt")
assert.ErrorIs(t, err, fs.ErrorObjectNotFound)
}
+2 -5
View File
@@ -16,16 +16,13 @@ import (
// multipartInit starts a new multipart upload and returns server details.
func (f *Fs) multipartInit(ctx context.Context, folderPath, filename string) (*api.MultipartInitResponse, error) {
encodedFolderPath := f.fromStandardPath(folderPath)
encodedFilename := f.fromStandardPath(filename)
opts := rest.Opts{
Method: "GET",
Path: "/multipart/init",
Parameters: url.Values{
"key": {f.opt.Key},
"filename": {encodedFilename},
"folder_path": {encodedFolderPath},
"filename": {filename},
"folder_path": {folderPath},
},
}
+54 -56
View File
@@ -1,6 +1,7 @@
package filelu
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -10,11 +11,9 @@ import (
"net/http"
"net/url"
"path"
"strconv"
"strings"
"github.com/rclone/rclone/fs"
rclonemultipart "github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/rest"
)
@@ -48,32 +47,40 @@ func (f *Fs) multipartUpload(ctx context.Context, in io.Reader, remote string) e
server := initResp.Result.Server
objectPath := initResp.Result.ObjectPath
chunkSize := int64(f.opt.ChunkSize)
if chunkSize <= 0 {
return fmt.Errorf("multipart upload: chunk_size must be positive: %v", f.opt.ChunkSize)
}
for partNo := 1; ; partNo++ {
// Buffer the part in memory from the global pool so it can be
// re-sent on retry
rw := rclonemultipart.NewRW()
n, err := io.CopyN(rw, in, chunkSize)
if err != nil && err != io.EOF {
_ = rw.Close()
return fmt.Errorf("read failed: %w", err)
}
chunkSize := int(f.opt.ChunkSize)
buf := make([]byte, 0, chunkSize)
tmp := make([]byte, 1024*1024)
partNo := 1
for {
n, errRead := in.Read(tmp)
if n > 0 {
uploadErr := f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, rw, n)
if uploadErr != nil {
_ = rw.Close()
return fmt.Errorf("upload part %d failed: %w", partNo, uploadErr)
buf = append(buf, tmp[:n]...)
// If buffer reached chunkSize, upload a full part
if len(buf) >= chunkSize {
err = f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, bytes.NewReader(buf))
if err != nil {
return fmt.Errorf("upload part %d failed: %w", partNo, err)
}
partNo++
buf = buf[:0]
}
}
if closeErr := rw.Close(); closeErr != nil {
return closeErr
}
if err == io.EOF {
if errRead == io.EOF {
break
}
if errRead != nil {
return fmt.Errorf("read failed: %w", errRead)
}
}
if len(buf) > 0 {
err = f.uploadPart(ctx, server, uploadID, sessID, objectPath, partNo, bytes.NewReader(buf))
if err != nil {
return fmt.Errorf("upload part %d failed: %w", partNo, err)
}
}
err = f.completeMultipart(ctx, server, uploadID, sessID, objectPath)
@@ -84,40 +91,31 @@ func (f *Fs) multipartUpload(ctx context.Context, in io.Reader, remote string) e
return nil
}
// uploadPart sends a single multipart chunk of size bytes to the upload server.
func (f *Fs) uploadPart(ctx context.Context, server, uploadID, sessID, objectPath string, partNo int, r io.ReadSeeker, size int64) error {
opts := rest.Opts{
Method: "PUT",
RootURL: server,
Parameters: url.Values{
"partNumber": {strconv.Itoa(partNo)},
"uploadId": {uploadID},
},
Body: r,
ContentLength: &size,
ExtraHeaders: map[string]string{
"X-RC-Upload-Id": uploadID,
"X-RC-Part-No": strconv.Itoa(partNo),
"X-Sess-ID": sessID,
"X-Object-Path": objectPath,
},
NoResponse: true,
// uploadPart sends a single multipart chunk to the upload server.
func (f *Fs) uploadPart(ctx context.Context, server, uploadID, sessID, objectPath string, partNo int, r io.Reader) error {
url := fmt.Sprintf("%s?partNumber=%d&uploadId=%s", server, partNo, uploadID)
req, err := http.NewRequestWithContext(ctx, "PUT", url, r)
if err != nil {
return err
}
return f.pacer.Call(func() (bool, error) {
// rewind the pooled buffer so each attempt sends the whole chunk
_, err := r.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
resp, err := f.srv.Call(ctx, &opts)
if err != nil {
if resp != nil {
return shouldRetryHTTP(resp.StatusCode), fmt.Errorf("uploadPart failed: %w", err)
}
return shouldRetry(err), err
}
return false, nil
})
req.Header.Set("X-RC-Upload-Id", uploadID)
req.Header.Set("X-RC-Part-No", fmt.Sprintf("%d", partNo))
req.Header.Set("X-Sess-ID", sessID)
req.Header.Set("X-Object-Path", objectPath)
resp, err := f.client.Do(req)
if err != nil {
return err
}
defer func() { _ = resp.Body.Close() }()
if resp.StatusCode != 200 {
return fmt.Errorf("uploadPart failed: %s", resp.Status)
}
return nil
}
// uploadFile uploads a file to FileLu
-29
View File
@@ -1,29 +0,0 @@
package filelu
import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fstest/fstests"
)
// SetUploadChunkSize changes the configured chunk size, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.ChunkSize = f.opt.ChunkSize, cs
return old, nil
}
// SetUploadCutoff changes the configured upload cutoff, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadCutoff(cutoff fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.UploadCutoff = f.opt.UploadCutoff, cutoff
return old, nil
}
var (
_ fstests.SetUploadChunkSizer = (*Fs)(nil)
_ fstests.SetUploadCutoffer = (*Fs)(nil)
)
-3
View File
@@ -12,8 +12,5 @@ func TestIntegration(t *testing.T) {
RemoteName: "TestFileLu:",
NilObject: nil,
SkipInvalidUTF8: true,
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: 1,
},
})
}
+11 -14
View File
@@ -19,7 +19,6 @@ import (
"github.com/Files-com/files-sdk-go/v3/file"
file_migration "github.com/Files-com/files-sdk-go/v3/filemigration"
"github.com/Files-com/files-sdk-go/v3/folder"
files_sdk_lib "github.com/Files-com/files-sdk-go/v3/lib"
"github.com/Files-com/files-sdk-go/v3/session"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
@@ -177,13 +176,6 @@ func shouldRetry(ctx context.Context, err error) (bool, error) {
}
}
// Errors from the upload storage servers are of this type
var httpErr files_sdk_lib.ResponseError
if errors.As(err, &httpErr) && slices.Contains(retryErrorCodes, httpErr.StatusCode) {
fs.Debugf(nil, "Retrying HTTP error %v", err)
return true, err
}
return fserrors.ShouldRetry(err), err
}
@@ -353,7 +345,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
}
for it.Next() {
item := new(it.File())
item := ptr(it.File())
remote := f.opt.Enc.ToStandardPath(item.DisplayName)
remote = path.Join(dir, remote)
if remote == dir {
@@ -424,7 +416,7 @@ func (f *Fs) mkdir(ctx context.Context, path string) error {
params := files_sdk.FolderCreateParams{
Path: path,
MkdirParents: new(true),
MkdirParents: ptr(true),
}
err := f.pacer.Call(func() (bool, error) {
@@ -466,7 +458,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
params := files_sdk.FileDeleteParams{
Path: path,
Recursive: new(!check),
Recursive: ptr(!check),
}
err := f.pacer.Call(func() (bool, error) {
@@ -537,7 +529,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (dstObj fs.
params := files_sdk.FileCopyParams{
Path: srcPath,
Destination: dstPath,
Overwrite: new(true),
Overwrite: ptr(true),
}
var action files_sdk.FileAction
@@ -680,7 +672,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
Paths: []string{f.absPath(remote)},
}
if expire < fs.DurationOff {
params.ExpiresAt = new(time.Now().Add(time.Duration(expire)))
params.ExpiresAt = ptr(time.Now().Add(time.Duration(expire)))
}
var bundle files_sdk.Bundle
@@ -842,6 +834,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
return
}
// Returns a pointer to t - useful for returning pointers to constants
func ptr[T any](t T) *T {
return &t
}
func isFolderNotEmpty(err error) bool {
var re files_sdk.ResponseError
ok := errors.As(err, &re)
@@ -861,7 +858,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
file.UploadWithProvidedMtime(src.ModTime(ctx)),
}
err := o.fs.pacer.CallNoRetry(func() (bool, error) {
err := o.fs.pacer.Call(func() (bool, error) {
err := o.fs.fileClient.Upload(uploadOpts...)
return shouldRetry(ctx, err)
})
+14 -32
View File
@@ -584,9 +584,8 @@ func (f *Fs) getFtpConnection(ctx context.Context) (c *ftp.ServerConn, err error
//
// It nils the pointed to connection out so it can't be reused
//
// if err is not nil then it discards the connection unless the error
// carries an FTP reply code, in which case it checks the connection
// is alive using a NOOP request
// if err is not nil then it checks the connection is alive using a
// NOOP request
func (f *Fs) putFtpConnection(pc **ftp.ServerConn, err error) {
if f.opt.Concurrency > 0 {
defer f.tokens.Put()
@@ -600,17 +599,14 @@ func (f *Fs) putFtpConnection(pc **ftp.ServerConn, err error) {
}
*pc = nil
if err != nil {
// An error without an FTP reply code means the control
// connection can no longer be trusted, so discard it.
if textprotoError(err) == nil {
fs.Debugf(f, "Discarding connection after non-FTP error: %v", err)
_ = c.Quit()
return
}
if nopErr := c.NoOp(); nopErr != nil {
fs.Debugf(f, "Connection failed, closing: %v", nopErr)
_ = c.Quit()
return
// If not a regular FTP error code then check the connection
if tpErr := textprotoError(err); tpErr != nil {
nopErr := c.NoOp()
if nopErr != nil {
fs.Debugf(f, "Connection failed, closing: %v", nopErr)
_ = c.Quit()
return
}
}
}
f.poolMu.Lock()
@@ -1269,18 +1265,15 @@ func (o *Object) Storable() bool {
// ftpReadCloser implements io.ReadCloser for FTP objects.
type ftpReadCloser struct {
rc io.ReadCloser
c *ftp.ServerConn
f *Fs
err error // errors found during read
closed bool // set by Close so a second call is a no-op
read int64 // bytes returned by Read
rc io.ReadCloser
c *ftp.ServerConn
f *Fs
err error // errors found during read
}
// Read bytes into p
func (f *ftpReadCloser) Read(p []byte) (n int, err error) {
n, err = f.rc.Read(p)
f.read += int64(n)
if err != nil && err != io.EOF {
f.err = err // store any errors for Close to examine
}
@@ -1289,10 +1282,6 @@ func (f *ftpReadCloser) Read(p []byte) (n int, err error) {
// Close the FTP reader and return the connection to the pool
func (f *ftpReadCloser) Close() error {
if f.closed {
return nil
}
f.closed = true
var err error
errchan := make(chan error, 1)
go func() {
@@ -1316,7 +1305,6 @@ func (f *ftpReadCloser) Close() error {
// if errors while reading or closing, dump the connection
if err != nil || f.err != nil {
_ = f.c.Quit()
f.c = nil // never pool a connection after Quit
f.f.putFtpConnection(nil, nil)
} else {
f.f.putFtpConnection(&f.c, nil)
@@ -1328,12 +1316,6 @@ func (f *ftpReadCloser) Close() error {
switch errX.Code {
case ftp.StatusTransfertAborted, ftp.StatusFileUnavailable, ftp.StatusAboutToSend, ftp.StatusRequestedFileActionOK:
err = nil
case ftp.StatusCanNotOpenDataConnection:
// Egnyte answers 425 when a RETR is closed before the first
// read; a transfer cut after data was read stays an error
if f.read == 0 {
err = nil
}
}
}
return err
+1 -164
View File
@@ -1,19 +1,12 @@
package ftp
import (
"bufio"
"context"
"errors"
"fmt"
"io"
"net"
"net/textproto"
"strings"
"sync"
"testing"
"time"
"github.com/jlaffaye/ftp"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/object"
@@ -50,7 +43,7 @@ func TestCommandEncodingNeutralisesCRLF(t *testing.T) {
type settings map[string]any
func deriveFs(ctx context.Context, t *testing.T, f fs.Fs, opts settings) fs.Fs {
fsName, _, _ := strings.Cut(f.Name(), "{") // strip off hash
fsName := strings.Split(f.Name(), "{")[0] // strip off hash
configMap := configmap.Simple{}
for key, val := range opts {
configMap[key] = fmt.Sprintf("%v", val)
@@ -151,159 +144,3 @@ func (f *Fs) InternalTest(t *testing.T) {
}
var _ fstests.InternalTester = (*Fs)(nil)
// fakeFTPServer is a minimal FTP control-connection server which answers
// every command with 200 OK.
type fakeFTPServer struct {
ln net.Listener
mu sync.Mutex
conn net.Conn
}
func startFakeFTPServer(t *testing.T) *fakeFTPServer {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
s := &fakeFTPServer{ln: ln}
go s.serve()
t.Cleanup(func() { s.close() })
return s
}
func (s *fakeFTPServer) serve() {
conn, err := s.ln.Accept()
if err != nil {
return
}
s.mu.Lock()
s.conn = conn
s.mu.Unlock()
defer func() { _ = conn.Close() }()
_, _ = fmt.Fprint(conn, "220 fake FTP server\r\n")
scanner := bufio.NewScanner(conn)
for scanner.Scan() {
_, _ = fmt.Fprint(conn, "200 OK\r\n")
}
}
// killConnection closes the server side of the control connection,
// simulating a server that dropped an idle connection.
func (s *fakeFTPServer) killConnection() {
s.mu.Lock()
defer s.mu.Unlock()
if s.conn != nil {
_ = s.conn.Close()
}
}
func (s *fakeFTPServer) close() {
s.mu.Lock()
defer s.mu.Unlock()
if s.conn != nil {
_ = s.conn.Close()
}
_ = s.ln.Close()
}
// TestPutFtpConnectionDiscardsDeadConnection checks that a connection
// returned with a raw network error is closed and never re-pooled, while
// a connection returned with a plain FTP reply-code error is probed with
// NOOP and pooled while the probe succeeds.
func TestPutFtpConnectionDiscardsDeadConnection(t *testing.T) {
t.Run("network error discards", func(t *testing.T) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
f := &Fs{opt: Options{}}
f.putFtpConnection(&c, errors.New("unexpected EOF"))
assert.Nil(t, c)
assert.Empty(t, f.pool)
})
t.Run("FTP reply error with live connection pools", func(t *testing.T) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
f := &Fs{opt: Options{}}
f.putFtpConnection(&c, &textproto.Error{Code: 550, Msg: "not found"})
assert.Nil(t, c)
require.Len(t, f.pool, 1)
assert.NoError(t, f.pool[0].NoOp())
_ = f.pool[0].Quit()
})
t.Run("FTP reply error with dead connection discards", func(t *testing.T) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
server.killConnection()
// Let the FIN propagate so the NOOP probe observes the dead connection.
time.Sleep(100 * time.Millisecond)
f := &Fs{opt: Options{}}
f.putFtpConnection(&c, &textproto.Error{Code: 550, Msg: "not found"})
assert.Nil(t, c)
assert.Empty(t, f.pool)
})
}
// fakeDataReader stands in for the RETR data stream of ftpReadCloser.
type fakeDataReader struct {
data []byte
closeErr error
}
func (r *fakeDataReader) Read(p []byte) (int, error) {
if len(r.data) == 0 {
return 0, io.EOF
}
n := copy(p, r.data)
r.data = r.data[n:]
return n, nil
}
func (r *fakeDataReader) Close() error { return r.closeErr }
// TestFtpReadCloserClose checks that an early-closed RETR answered with 425
// is masked only when nothing was read, that a connection which got QUIT is
// never pooled, and that a second Close does not touch the pool.
func TestFtpReadCloserClose(t *testing.T) {
err425 := &textproto.Error{Code: ftp.StatusCanNotOpenDataConnection, Msg: "Can't open data connection."}
newReader := func(t *testing.T, data []byte, closeErr error) (*ftpReadCloser, *Fs) {
server := startFakeFTPServer(t)
c, err := ftp.Dial(server.ln.Addr().String())
require.NoError(t, err)
f := &Fs{opt: Options{}}
return &ftpReadCloser{rc: &fakeDataReader{data: data, closeErr: closeErr}, c: c, f: f}, f
}
t.Run("425 before any read is masked and the connection is not pooled", func(t *testing.T) {
rc, f := newReader(t, nil, err425)
assert.NoError(t, rc.Close())
assert.Empty(t, f.pool)
assert.NoError(t, rc.Close())
assert.Empty(t, f.pool)
})
t.Run("425 after data was read stays an error", func(t *testing.T) {
rc, f := newReader(t, []byte("partial"), err425)
buf := make([]byte, 4)
n, err := rc.Read(buf)
require.NoError(t, err)
require.Equal(t, 4, n)
err = rc.Close()
require.Error(t, err)
assert.Equal(t, ftp.StatusCanNotOpenDataConnection, textprotoError(err).Code)
assert.Empty(t, f.pool)
})
t.Run("clean close pools the connection once", func(t *testing.T) {
rc, f := newReader(t, []byte("data"), nil)
_, err := io.ReadAll(rc)
require.NoError(t, err)
assert.NoError(t, rc.Close())
require.Len(t, f.pool, 1)
assert.NoError(t, rc.Close())
assert.Len(t, f.pool, 1)
_ = f.pool[0].Quit()
})
}
+2 -1
View File
@@ -194,7 +194,8 @@ var retryErrorCodes = []int{
// Return true if the api error has the status given
func isAPIErr(err error, status string) bool {
if apiErr, ok := errors.AsType[api.Error](err); ok {
var apiErr api.Error
if errors.As(err, &apiErr) {
return apiErr.Status == status
}
return false
-44
View File
@@ -122,49 +122,6 @@ func (f *Fs) testUpdateStoresHash(t *testing.T) {
_ = operations.Purge(ctx, f, dirName)
}
func (f *Fs) testSetModTimeKeepsHash(t *testing.T) {
ctx := context.Background()
const dirName = "setmodtime_hash_1"
const fileName = dirName + "/file_setmodtime_1"
const longTime = fs.ModTimeNotSupported
hashType := f.keepHashes.GetOne()
// upload a file to hasher via Put
src := putFile(ctx, t, f, fileName, "setmodtime content")
dst, err := f.NewObject(ctx, fileName)
require.NoError(t, err)
require.NotNil(t, dst)
_ = src
// verify hash was stored after Put
if f.opt.MaxAge <= 0 {
t.Skip("hash cache is disabled")
}
hash1, err := f.getRawHash(ctx, hashType, fileName, anyFingerprint, longTime)
require.NoError(t, err)
require.NotEmpty(t, hash1)
// change only the modification time, as operations does when it
// syncs the modtime of an otherwise unchanged destination file
newTime := fstest.Time("2002-03-04T05:06:07.499999999Z")
require.NoError(t, dst.SetModTime(ctx, newTime))
// the cached hash must survive a pure modtime change
hash2, err := f.getRawHash(ctx, hashType, fileName, anyFingerprint, longTime)
assert.NoError(t, err)
assert.Equal(t, hash1, hash2, "hash should be kept after SetModTime")
// and it must be found via the fingerprint-checked lookup too
dst2, err := f.NewObject(ctx, fileName)
require.NoError(t, err)
hashVal, err := dst2.Hash(ctx, hashType)
assert.NoError(t, err)
assert.Equal(t, hash1, hashVal, "hash should be returned after SetModTime")
_ = operations.Purge(ctx, f, dirName)
}
// InternalTest dispatches all internal tests
func (f *Fs) InternalTest(t *testing.T) {
if !kv.Supported() {
@@ -172,7 +129,6 @@ func (f *Fs) InternalTest(t *testing.T) {
}
t.Run("UploadFromCrypt", f.testUploadFromCrypt)
t.Run("UpdateStoresHash", f.testUpdateStoresHash)
t.Run("SetModTimeKeepsHash", f.testSetModTimeKeepsHash)
}
var _ fstests.InternalTester = (*Fs)(nil)
-28
View File
@@ -206,34 +206,6 @@ func (op *kvPut) Do(ctx context.Context, b kv.Bucket) (err error) {
return err
}
// kvUpdateFingerprint: rewrite the fingerprint of an existing record
type kvUpdateFingerprint struct {
key string
oldFp string
newFp string
age time.Duration
}
func (op *kvUpdateFingerprint) Do(ctx context.Context, b kv.Bucket) error {
data := b.Get([]byte(op.key))
if len(data) == 0 {
return nil
}
var r hashRecord
if err := r.decode(op.key, data); err != nil {
return nil
}
if r.Fp != op.oldFp || time.Since(r.Created) > op.age {
return nil
}
r.Fp = op.newFp
data, err := r.encode(op.key)
if err != nil {
return err
}
return b.Put([]byte(op.key), data)
}
// kvDump: dump the database.
// Note: long dump can cause concurrent operations to fail.
type kvDump struct {
+5 -35
View File
@@ -56,23 +56,6 @@ func (o *Object) putHashes(ctx context.Context, rawHashes hashMap) error {
return o.f.putRawHashes(ctx, key, fp, hashes)
}
// updateFingerprint rewrites the fingerprint of the cached hash
// record for remote from oldFp to newFp, keeping the hashes. It is
// used when only the modification time of an object changes, so the
// cached hashes remain valid under the new fingerprint.
func (f *Fs) updateFingerprint(ctx context.Context, remote, oldFp, newFp string) error {
if oldFp == "" || newFp == "" || oldFp == newFp || f.opt.MaxAge <= 0 {
return nil
}
key := path.Join(f.Fs.Root(), remote)
return f.db.Do(true, &kvUpdateFingerprint{
key: key,
oldFp: oldFp,
newFp: newFp,
age: time.Duration(f.opt.MaxAge),
})
}
// set hashes for a path without any validation
func (f *Fs) putRawHashes(ctx context.Context, key, fp string, hashes operations.HashSums) error {
return f.db.Do(true, &kvPut{
@@ -213,27 +196,14 @@ func (o *Object) Remove(ctx context.Context) error {
}
// SetModTime sets the modification time of the file.
//
// When the fingerprint includes the modtime, the contents have not
// changed, so the cached hashes are carried over to the fingerprint
// for the new modtime. Otherwise the cache entry is pruned when the
// modtime changes so that touching a file will trigger checksum
// recalculation even on backends that don't provide modTime with
// fingerprint.
// Also prunes the cache entry when modtime changes so that
// touching a file will trigger checksum recalculation even
// on backends that don't provide modTime with fingerprint.
func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error {
if mtime == o.Object.ModTime(ctx) {
return o.Object.SetModTime(ctx, mtime)
}
if !o.f.fpTime {
if mtime != o.Object.ModTime(ctx) {
_ = o.f.pruneHash(o.Remote())
return o.Object.SetModTime(ctx, mtime)
}
oldFp := o.fingerprint(ctx)
if err := o.Object.SetModTime(ctx, mtime); err != nil {
return err
}
_ = o.f.updateFingerprint(ctx, o.Remote(), oldFp, o.fingerprint(ctx))
return nil
return o.Object.SetModTime(ctx, mtime)
}
// Open opens the file for read.
+61 -82
View File
@@ -10,6 +10,7 @@ package hidrive
// to be resolved prior to execution with resolvePath(...).
import (
"bytes"
"context"
"errors"
"io"
@@ -23,9 +24,8 @@ import (
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/ranges"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/sync/errgroup"
"golang.org/x/sync/semaphore"
@@ -454,17 +454,17 @@ func (f *Fs) deleteObject(ctx context.Context, path string) error {
}
// createFile creates a file at the given path
// with the content of the buffer.
// with the content of the io.ReadSeeker.
// This guarantees that existing files will not be overwritten.
// The maximum size of the content is limited by MaximumUploadBytes.
// The caller remains responsible for closing the buffer.
// The io.ReadSeeker should be resettable by seeking to its start.
// If modTime is not the zero time instant,
// it will be set as the file's modification time after the operation.
//
// This returns fs.ErrorDirNotFound
// if the parent directory of the file is not found.
// This returns ErrorFileExists if a file already exists at the specified path.
func (f *Fs) createFile(ctx context.Context, path string, content *pool.RW, modTime time.Time, onExist OnExistAction) (*api.HiDriveObject, error) {
func (f *Fs) createFile(ctx context.Context, path string, content io.ReadSeeker, modTime time.Time, onExist OnExistAction) (*api.HiDriveObject, error) {
parameters := api.NewQueryParameters()
parameters.SetFileInDirectory(path)
if onExist == AutoNameOnExist {
@@ -479,14 +479,12 @@ func (f *Fs) createFile(ctx context.Context, path string, content *pool.RW, modT
}
}
contentLength := content.Size()
opts := rest.Opts{
Method: "POST",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
ContentLength: &contentLength,
Parameters: parameters.Values,
Method: "POST",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
Parameters: parameters.Values,
}
var result api.HiDriveObject
@@ -512,16 +510,16 @@ func (f *Fs) createFile(ctx context.Context, path string, content *pool.RW, modT
}
// overwriteFile updates the content of the file at the given path
// with the content of the buffer.
// with the content of the io.ReadSeeker.
// If the file does not exist it will be created.
// The maximum size of the content is limited by MaximumUploadBytes.
// The caller remains responsible for closing the buffer.
// The io.ReadSeeker should be resettable by seeking to its start.
// If modTime is not the zero time instant,
// it will be set as the file's modification time after the operation.
//
// This returns fs.ErrorDirNotFound
// if the parent directory of the file is not found.
func (f *Fs) overwriteFile(ctx context.Context, path string, content *pool.RW, modTime time.Time) (*api.HiDriveObject, error) {
func (f *Fs) overwriteFile(ctx context.Context, path string, content io.ReadSeeker, modTime time.Time) (*api.HiDriveObject, error) {
parameters := api.NewQueryParameters()
parameters.SetFileInDirectory(path)
@@ -533,14 +531,12 @@ func (f *Fs) overwriteFile(ctx context.Context, path string, content *pool.RW, m
}
}
contentLength := content.Size()
opts := rest.Opts{
Method: "PUT",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
ContentLength: &contentLength,
Parameters: parameters.Values,
Method: "PUT",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
Parameters: parameters.Values,
}
var result api.HiDriveObject
@@ -638,18 +634,11 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
}
// Read a chunk of data.
var (
chunkReader *pool.RW
bytesRead int64
)
chunkReader, bytesRead, readErr = readerForChunk(content, int64(chunkSize), true)
if bytesRead < int64(chunkSize) {
chunkReader, bytesRead, readErr := readerForChunk(content, chunkSize)
if bytesRead < chunkSize {
startMoreTransfers = false
}
if readErr != nil || bytesRead <= 0 {
if chunkReader != nil {
_ = chunkReader.Close()
}
break
}
@@ -659,12 +648,11 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
// After this upload is done,
// signal that another transfer can be started.
defer transferSemaphore.Release(1)
defer func() { _ = chunkReader.Close() }()
uploadErr := f.patchFile(gCtx, path, chunkReader, chunkOffset, zeroTime)
uploadErr := f.patchFile(gCtx, path, cachedReader(chunkReader), chunkOffset, zeroTime)
if uploadErr == nil {
// Remember successfully written chunks.
okChunksMu.Lock()
okChunks = append(okChunks, ranges.Range{Pos: int64(chunkOffset), Size: bytesRead})
okChunks = append(okChunks, ranges.Range{Pos: int64(chunkOffset), Size: int64(bytesRead)})
okChunksMu.Unlock()
fs.Debugf(f, "Done uploading chunk of size %v at offset %v.", bytesRead, chunkOffset)
} else {
@@ -708,10 +696,10 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
// starting at the given offset.
//
// Replaces the file contents starting from the given byte offset
// with the content of the buffer.
// with the content of the io.ReadSeeker.
// If the offset is beyond the file end, the file is extended up to the offset.
// The maximum size of the update is limited by MaximumUploadBytes.
// The caller remains responsible for closing the buffer.
// The io.ReadSeeker should be resettable by seeking to its start.
// If modTime is not the zero time instant,
// it will be set as the file's modification time after the operation.
//
@@ -723,7 +711,7 @@ func (f *Fs) updateFileChunked(ctx context.Context, path string, content io.Read
// usually expand the holes into regions of 0-bytes.
//
// This returns fs.ErrorObjectNotFound if the object is not found.
func (f *Fs) patchFile(ctx context.Context, path string, content *pool.RW, offset uint64, modTime time.Time) error {
func (f *Fs) patchFile(ctx context.Context, path string, content io.ReadSeeker, offset uint64, modTime time.Time) error {
parameters := api.NewQueryParameters()
parameters.SetPath(path)
parameters.Set("offset", strconv.FormatUint(offset, 10))
@@ -735,15 +723,13 @@ func (f *Fs) patchFile(ctx context.Context, path string, content *pool.RW, offse
}
}
contentLength := content.Size()
opts := rest.Opts{
Method: "PATCH",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
ContentLength: &contentLength,
Parameters: parameters.Values,
NoResponse: true,
Method: "PATCH",
Path: "/file",
Body: content,
ContentType: "application/octet-stream",
Parameters: parameters.Values,
NoResponse: true,
}
var resp *http.Response
@@ -853,48 +839,41 @@ func createHiDriveScopes(role string, access string) []string {
return []string{}
}
// unwrapAccounting splits any transfer accounting off reader,
// returning the raw stream and the accounting (nil if there is none),
// so the stream can be buffered and accounted when the buffer is uploaded.
func unwrapAccounting(reader io.Reader) (unwrapped io.Reader, acc *accounting.Account) {
// Any kind of accounter re-wraps into the one type UnWrapAccounting
// knows how to take the *Account out of.
unwrapped, wrap := accounting.UnWrap(reader)
_, acc = accounting.UnWrapAccounting(wrap(unwrapped))
return unwrapped, acc
}
// readerForChunk reads up to length bytes from reader into a buffer
// from the global memory pool and returns it
// along with the number of bytes that have been read from reader.
// Reaching the end of reader early is not an error.
//
// If accountUpload is set, any accounting on reader is applied
// when the buffer is read instead, so the upload is what gets accounted.
//
// This must not be set for a buffer which creates or replaces a file
// in a single request. Accounting can fail part way through the upload,
// for example when --max-transfer is reached, and HiDrive keeps
// what it has received of an aborted request as the file.
//
// The caller must Close the buffer when done with it.
func readerForChunk(reader io.Reader, length int64, accountUpload bool) (chunkReader *pool.RW, bytesRead int64, err error) {
chunkReader = multipart.NewRW()
if accountUpload {
var acc *accounting.Account
reader, acc = unwrapAccounting(reader)
if acc != nil {
chunkReader.SetAccounting(acc.AccountRead)
}
// cachedReader returns a version of the reader that caches its contents and
// can therefore be reset using Seek.
func cachedReader(reader io.Reader) io.ReadSeeker {
bytesReader, ok := reader.(*bytes.Reader)
if ok {
return bytesReader
}
bytesRead, err = io.CopyN(chunkReader, reader, length)
if err == io.EOF {
repeatableReader, ok := reader.(*readers.RepeatableReader)
if ok {
return repeatableReader
}
return readers.NewRepeatableReader(reader)
}
// readerForChunk reads a chunk of bytes from reader (after handling any accounting).
// Returns a new io.Reader (chunkReader) for that chunk
// and the number of bytes that have been read from reader.
func readerForChunk(reader io.Reader, length int) (chunkReader io.Reader, bytesRead int, err error) {
// Unwrap any accounting from the input if present.
reader, wrap := accounting.UnWrap(reader)
// Read a chunk of data.
buffer := make([]byte, length)
bytesRead, err = io.ReadFull(reader, buffer)
if err == io.EOF || err == io.ErrUnexpectedEOF {
err = nil
}
if err != nil {
_ = chunkReader.Close()
return nil, bytesRead, err
}
return chunkReader, bytesRead, nil
// Truncate unused capacity.
buffer = buffer[:bytesRead]
// Use wrap to put any accounting back for chunkReader.
return wrap(bytes.NewReader(buffer)), bytesRead, nil
}
+4 -17
View File
@@ -30,7 +30,6 @@ import (
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/rest"
)
@@ -515,14 +514,8 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
// (i.e. everything up to the cutoff) in the first request,
// avoids files being created on upload failure for small files.
// (As opposed to creating an empty file and then uploading the content.)
//
// Only buffer as much as the source declares it has,
// so small files do not cost the whole cutoff in memory.
prefixSize := int64(f.opt.UploadCutoff)
if size := src.Size(); size >= 0 && size < prefixSize {
prefixSize = size
}
cutoffReader, bytesRead, err := readerForChunk(in, prefixSize, false)
tmpReader, bytesRead, err := readerForChunk(in, int(f.opt.UploadCutoff))
cutoffReader := cachedReader(tmpReader)
if err != nil {
return nil, err
}
@@ -547,7 +540,6 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
}
return false, createErr
})
_ = cutoffReader.Close()
if err != nil {
return nil, err
@@ -564,7 +556,7 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
}
// If there is more left to write, o.Update needs to skip ahead.
// Use a fs.SeekOption with the current offset to do this.
options = append(options, &fs.SeekOption{Offset: bytesRead})
options = append(options, &fs.SeekOption{Offset: int64(bytesRead)})
err = o.Update(ctx, in, src, options...)
if err == nil {
@@ -979,12 +971,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// Metadata should be updated even if the upload fails.
info, metaErr = o.fs.fetchMetadataForPath(ctx, resolvedPath, api.HiDriveObjectWithMetadataFields)
} else {
var content *pool.RW
content, _, err = readerForChunk(in, src.Size(), false)
if err == nil {
info, err = o.fs.overwriteFile(ctx, resolvedPath, content, modTime)
_ = content.Close()
}
info, err = o.fs.overwriteFile(ctx, resolvedPath, cachedReader(in), modTime)
metaErr = err
}
+41 -54
View File
@@ -63,8 +63,7 @@ The headers are only sent to the host in the configured URL. If the
server redirects to another host (including a subdomain or a different
port) the headers are not sent to it, or to any further hop in that
redirect chain. When headers are set, a redirect from https to http is
refused as it would send them in cleartext, even with
--allow-https-downgrade.`,
refused as it would send them in cleartext.`,
Default: fs.CommaSepList{},
Advanced: true,
}, {
@@ -101,30 +100,6 @@ that directory listings are much quicker, but rclone won't have the times or
sizes of any files, and some files that don't exist may be in the listing.`,
Default: false,
Advanced: true,
}, {
Name: "no_head_size",
Help: `Set the reported file size for --http-no-head.
When HEAD requests are disabled, rclone normally reports an unknown file size.
Set this option to a non-negative value to report that size for every file,
including 0. Leave it unset (the default, shown as "off") to keep reporting
an unknown size.
The configured size is authoritative. If it does not match the actual file
size, transfers may fail their size check and retry unless --ignore-size is
used. It can also make multi-thread transfers request incorrect ranges and
make mounted files unreadable past the reported size.`,
Default: fs.SizeSuffix(-1),
Advanced: true,
}, {
Name: "no_head_time",
Help: `Set a default modification time for --http-no-head.
When HEAD requests are disabled, rclone normally reports an unset modification
time. Set this option to report the specified time for every file instead.
Changing the configured time later makes every file appear modified to a sync.`,
Default: fs.Time{},
Advanced: true,
}, {
Name: "no_escape",
Help: "Do not escape URL metacharacters in path names.",
@@ -170,13 +145,11 @@ var systemMetadataInfo = map[string]fs.MetadataHelp{
// Options defines the configuration for this backend
type Options struct {
Endpoint string `config:"url"`
NoSlash bool `config:"no_slash"`
NoHead bool `config:"no_head"`
NoHeadSize fs.SizeSuffix `config:"no_head_size"`
NoHeadTime fs.Time `config:"no_head_time"`
Headers fs.CommaSepList `config:"headers"`
NoEscape bool `config:"no_escape"`
Endpoint string `config:"url"`
NoSlash bool `config:"no_slash"`
NoHead bool `config:"no_head"`
Headers fs.CommaSepList `config:"headers"`
NoEscape bool `config:"no_escape"`
}
// Fs stores the interface to the remote HTTP files
@@ -310,15 +283,14 @@ func (f *Fs) httpConnection(ctx context.Context, opt *Options) (isFile bool, err
if err != nil {
return false, err
}
u, err := rest.URLJoinRoot(base, f.root)
u, err := rest.URLJoin(base, rest.URLPathEscape(f.root))
if err != nil {
return false, err
}
client := fshttp.NewClient(ctx)
// Without configured headers keep the default policy so
// --allow-https-downgrade can follow public mirrors that
// redirect from https to http.
// Without configured headers keep the default policy which
// public mirrors that redirect from https to http rely on.
if len(opt.Headers) > 0 {
client.CheckRedirect = checkRedirect(opt)
}
@@ -350,9 +322,7 @@ func (f *Fs) httpConnection(ctx context.Context, opt *Options) (isFile bool, err
// the host specified in the config file.
func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
// Parse config into Options struct
opt := &Options{
NoHeadSize: fs.SizeSuffix(-1),
}
opt := new(Options)
err := configstruct.Set(m, opt)
if err != nil {
return nil, err
@@ -556,20 +526,40 @@ func addHeaders(req *http.Request, opt *Options) {
// rest.ErrHTTPSDowngrade and strips the configured headers when the
// redirect chain has left the originally requested host at any point.
func checkRedirect(opt *Options) func(req *http.Request, via []*http.Request) error {
headers := make([]string, 0, len(opt.Headers)/2)
for i := 0; i < len(opt.Headers); i += 2 {
headers = append(headers, opt.Headers[i])
}
stripHeaders := rest.StripHeadersOnCrossHostRedirectFn(headers...)
return func(req *http.Request, via []*http.Request) error {
err := stripHeaders(req, via)
if errors.Is(err, rest.ErrHTTPSDowngrade) {
err = fmt.Errorf("%w (the configured headers would be sent to the plaintext target)", err)
if err := rest.RefuseHTTPSDowngradeRedirectFn(req, via); err != nil {
if errors.Is(err, rest.ErrHTTPSDowngrade) {
err = fmt.Errorf("%w (the configured headers would be sent to the plaintext target)", err)
}
return err
}
return err
if redirectLeavesHost(req, via) {
for i := 0; i < len(opt.Headers); i += 2 {
req.Header.Del(opt.Headers[i])
}
}
return nil
}
}
// redirectLeavesHost reports whether any hop in the redirect chain
// via plus the pending request req is to a different host from the
// original request via[0].
//
// net/http copies the headers afresh from the original request for
// every hop, so once the chain has visited another host the headers
// must be stripped from every subsequent hop, even one back to the
// original host, as the other host chose the URL.
func redirectLeavesHost(req *http.Request, via []*http.Request) bool {
origin := via[0].URL
for _, hop := range via {
if !rest.SameHost(hop.URL, origin) {
return true
}
}
return !rest.SameHost(req.URL, origin)
}
// Adds the configured headers to the request if any
func (f *Fs) addHeaders(req *http.Request) {
addHeaders(req, &f.opt)
@@ -603,7 +593,7 @@ func (f *Fs) readDir(ctx context.Context, dir string) (names []string, err error
return nil, fmt.Errorf("failed to readDir: %w", err)
}
contentType, _, _ := strings.Cut(res.Header.Get("Content-Type"), ";")
contentType := strings.SplitN(res.Header.Get("Content-Type"), ";", 2)[0]
switch contentType {
case "text/html":
names, err = parse(u, res.Body)
@@ -747,11 +737,8 @@ func (o *Object) url() string {
// head sends a HEAD request to update info fields in the Object
func (o *Object) head(ctx context.Context) error {
if o.fs.opt.NoHead {
o.size = int64(o.fs.opt.NoHeadSize)
o.size = -1
o.modTime = timeUnset
if o.fs.opt.NoHeadTime.IsSet() {
o.modTime = time.Time(o.fs.opt.NoHeadTime)
}
o.contentType = fs.MimeType(ctx, o)
return nil
}
+4 -96
View File
@@ -314,54 +314,6 @@ func TestOpen(t *testing.T) {
}
}
func TestNoHeadMetadataDefaults(t *testing.T) {
tests := []struct {
name string
size string
modTime string
wantSize int64
wantModTime time.Time
}{
{
name: "defaults",
wantSize: -1,
wantModTime: timeUnset,
}, {
name: "zero size",
size: "0B",
wantSize: 0,
wantModTime: timeUnset,
}, {
name: "configured metadata",
size: "100B",
modTime: "2024-08-05T07:36:07Z",
wantSize: 100,
wantModTime: time.Date(2024, 8, 5, 7, 36, 7, 0, time.UTC),
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
m := prepareServer(t)
m.Set("no_head", "true")
if test.size != "" {
m.Set("no_head_size", test.size)
}
if test.modTime != "" {
m.Set("no_head_time", test.modTime)
}
f, err := NewFs(context.Background(), remoteName, "", m)
require.NoError(t, err)
o, err := f.NewObject(context.Background(), "four/under four.txt")
require.NoError(t, err)
assert.Equal(t, test.wantSize, o.Size())
assert.Equal(t, test.wantModTime, o.ModTime(context.Background()))
})
}
}
func TestMimeType(t *testing.T) {
f := prepare(t)
@@ -688,7 +640,7 @@ func TestRedirectRefusesHTTPSDowngrade(t *testing.T) {
// open makes a remote pointing at ts with or without headers
// and opens file.txt on it
open := func(t *testing.T, withHeaders, allowDowngrade bool) (io.ReadCloser, error) {
open := func(t *testing.T, withHeaders bool) (io.ReadCloser, error) {
configfile.Install()
m := configmap.Simple{
"type": "http",
@@ -700,7 +652,6 @@ func TestRedirectRefusesHTTPSDowngrade(t *testing.T) {
}
ctx, ci := fs.AddConfig(context.Background())
ci.InsecureSkipVerify = true
ci.AllowHTTPSDowngrade = allowDowngrade
f, err := NewFs(ctx, remoteName, "", m)
require.NoError(t, err)
@@ -712,25 +663,13 @@ func TestRedirectRefusesHTTPSDowngrade(t *testing.T) {
t.Run("WithHeaders", func(t *testing.T) {
// Headers could leak so the downgrade must be refused
_, err := open(t, true, false)
require.ErrorIs(t, err, rest.ErrHTTPSDowngrade)
})
t.Run("WithHeadersAllowDowngrade", func(t *testing.T) {
// Headers could leak so the downgrade is refused even when allowed
_, err := open(t, true, true)
_, err := open(t, true)
require.ErrorIs(t, err, rest.ErrHTTPSDowngrade)
})
t.Run("WithoutHeaders", func(t *testing.T) {
// The transport refuses downgrades by default
_, err := open(t, false, false)
require.ErrorIs(t, err, rest.ErrHTTPSDowngrade)
})
t.Run("WithoutHeadersAllowDowngrade", func(t *testing.T) {
// Nothing to leak so the downgrade is followed when allowed
fd, err := open(t, false, true)
// Nothing to leak so the downgrade is followed
fd, err := open(t, false)
require.NoError(t, err)
data, err := io.ReadAll(fd)
require.NoError(t, err)
@@ -821,34 +760,3 @@ func TestRedirectStripsHeadersOnHostChange(t *testing.T) {
assert.Empty(t, got.Values(headers[i]), "header %q leaked to redirect target", headers[i])
}
}
func TestRootCannotChangeHost(t *testing.T) {
// A root starting with "//" is a network-path reference under the
// URL standard. It must not be able to move the remote (and the
// configured headers) to a host other than the configured one.
var requests int
other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
_, _ = w.Write([]byte("hello"))
}))
defer other.Close()
m := prepareServer(t)
otherHost := strings.TrimPrefix(other.URL, "http://")
for _, root := range []string{
"//" + otherHost + "/",
"//" + otherHost + "/one%.txt",
"//" + otherHost,
"//user:pass@" + otherHost + "/",
} {
f, err := NewFs(context.Background(), remoteName, root, m)
assert.ErrorContains(t, err, "must not change the scheme, host or user", root)
assert.Nil(t, f, root)
}
assert.Equal(t, 0, requests, "request sent to the other host")
// A same-origin absolute path must still work
f, err := NewFs(context.Background(), remoteName, "/", m)
require.NoError(t, err)
assert.Equal(t, m["url"]+"/", f.String())
}
+70 -65
View File
@@ -37,44 +37,44 @@ type FileList struct {
// File represents a file or folder in Huawei Drive
type File struct {
ID string `json:"id"`
FileName string `json:"fileName"`
OriginalFilename string `json:"originalFilename,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
Category string `json:"category"`
Size int64 `json:"size,omitempty"`
SHA256 string `json:"sha256,omitempty"`
CreatedTime time.Time `json:"createdTime"`
EditedTime time.Time `json:"editedTime"`
EditedByMeTime time.Time `json:"editedByMeTime"`
ParentFolder []string `json:"parentFolder,omitempty"`
Owners []User `json:"owners"`
LastEditor User `json:"lastEditor"`
Permissions []Permission `json:"permissions"`
PermissionIDs []string `json:"permissionIds"`
Capabilities FileCapabilities `json:"capabilities"`
OwnedByMe bool `json:"ownedByMe"`
EditedByMe bool `json:"editedByMe"`
ViewedByMe bool `json:"viewedByMe"`
HasShared bool `json:"hasShared"`
Recycled bool `json:"recycled"`
DirectlyRecycled bool `json:"directlyRecycled"`
Favorite bool `json:"favorite"`
ExistThumbnail bool `json:"existThumbnail"`
ThumbnailVersion int64 `json:"thumbnailVersion"`
IconDownloadLink string `json:"iconDownloadLink,omitempty"`
ContentDownloadLink string `json:"contentDownloadLink,omitempty"`
ContentVersion string `json:"contentVersion,omitempty"`
LastHistoryVersionID string `json:"lastHistoryVersionId,omitempty"`
OccupiedSpace int64 `json:"occupiedSpace,omitempty"`
Version int64 `json:"version"`
WritersHasSharePermission bool `json:"writersHasSharePermission"`
WriterHasCopyPermission bool `json:"writerHasCopyPermission"`
Containers []string `json:"containers"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
ContentExtras *ContentExtras `json:"contentExtras,omitempty"`
ID string `json:"id"`
FileName string `json:"fileName"`
OriginalFilename string `json:"originalFilename,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
Category string `json:"category"`
Size int64 `json:"size,omitempty"`
SHA256 string `json:"sha256,omitempty"`
CreatedTime time.Time `json:"createdTime"`
EditedTime time.Time `json:"editedTime"`
EditedByMeTime time.Time `json:"editedByMeTime,omitempty"`
ParentFolder []string `json:"parentFolder,omitempty"`
Owners []User `json:"owners"`
LastEditor User `json:"lastEditor"`
Permissions []Permission `json:"permissions"`
PermissionIDs []string `json:"permissionIds"`
Capabilities FileCapabilities `json:"capabilities"`
OwnedByMe bool `json:"ownedByMe"`
EditedByMe bool `json:"editedByMe"`
ViewedByMe bool `json:"viewedByMe"`
HasShared bool `json:"hasShared"`
Recycled bool `json:"recycled"`
DirectlyRecycled bool `json:"directlyRecycled"`
Favorite bool `json:"favorite"`
ExistThumbnail bool `json:"existThumbnail"`
ThumbnailVersion int64 `json:"thumbnailVersion"`
IconDownloadLink string `json:"iconDownloadLink,omitempty"`
ContentDownloadLink string `json:"contentDownloadLink,omitempty"`
ContentVersion string `json:"contentVersion,omitempty"`
LastHistoryVersionID string `json:"lastHistoryVersionId,omitempty"`
OccupiedSpace int64 `json:"occupiedSpace,omitempty"`
Version int64 `json:"version"`
WritersHasSharePermission bool `json:"writersHasSharePermission"`
WriterHasCopyPermission bool `json:"writerHasCopyPermission"`
Containers []string `json:"containers"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
ContentExtras *ContentExtras `json:"contentExtras,omitempty"`
}
// IsDir returns true if the file is a directory
@@ -125,41 +125,41 @@ type Thumbnail struct {
// CreateFolderRequest represents request to create a folder
type CreateFolderRequest struct {
FileName string `json:"fileName"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
ParentFolder []string `json:"parentFolder,omitempty"`
Favorite bool `json:"favorite,omitempty"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
AutoRename int `json:"autoRename,omitempty"`
FileName string `json:"fileName"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType"`
ParentFolder []string `json:"parentFolder,omitempty"`
Favorite bool `json:"favorite,omitempty"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
AutoRename int `json:"autoRename,omitempty"`
}
// UpdateFileRequest represents request to update a file
type UpdateFileRequest struct {
FileName string `json:"fileName,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType,omitempty"`
Favorite *bool `json:"favorite,omitempty"`
Recycled *bool `json:"recycled,omitempty"`
OriginalFilename string `json:"originalFilename,omitempty"`
WriterHasCopyPermission *bool `json:"writerHasCopyPermission,omitempty"`
WritersHasSharePermission *bool `json:"writersHasSharePermission,omitempty"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
AddParentFolder []string `json:"addParentFolder,omitempty"`
RemoveParentFolder []string `json:"removeParentFolder,omitempty"`
CreatedTime *time.Time `json:"createdTime,omitempty"`
EditedTime *time.Time `json:"editedTime,omitempty"`
FileName string `json:"fileName,omitempty"`
Description string `json:"description,omitempty"`
MimeType string `json:"mimeType,omitempty"`
Favorite *bool `json:"favorite,omitempty"`
Recycled *bool `json:"recycled,omitempty"`
OriginalFilename string `json:"originalFilename,omitempty"`
WriterHasCopyPermission *bool `json:"writerHasCopyPermission,omitempty"`
WritersHasSharePermission *bool `json:"writersHasSharePermission,omitempty"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
AddParentFolder []string `json:"addParentFolder,omitempty"`
RemoveParentFolder []string `json:"removeParentFolder,omitempty"`
CreatedTime *time.Time `json:"createdTime,omitempty"`
EditedTime *time.Time `json:"editedTime,omitempty"`
}
// CopyFileRequest represents request to copy a file
type CopyFileRequest struct {
FileName string `json:"fileName,omitempty"`
MimeType string `json:"mimeType,omitempty"`
ParentFolder []string `json:"parentFolder,omitempty"`
Properties map[string]any `json:"properties,omitempty"`
AppSettings map[string]any `json:"appSettings,omitempty"`
FileName string `json:"fileName,omitempty"`
MimeType string `json:"mimeType,omitempty"`
ParentFolder []string `json:"parentFolder,omitempty"`
Properties map[string]interface{} `json:"properties,omitempty"`
AppSettings map[string]interface{} `json:"appSettings,omitempty"`
}
// ResumeUploadInitResponse represents response to resume upload init
@@ -224,6 +224,11 @@ var GlobalDomains = map[string]bool{
"drive.cloud.hicloud.com": true,
}
// BoolPtr returns a pointer to a bool value (helper for UpdateFileRequest)
func BoolPtr(b bool) *bool {
return &b
}
// StartCursor represents response to Changes.getStartCursor endpoint
type StartCursor struct {
Category string `json:"category"` // "drive#startCursor"
+34 -59
View File
@@ -33,7 +33,6 @@ import (
"github.com/rclone/rclone/lib/encoder"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/pool"
"github.com/rclone/rclone/lib/readers"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/oauth2"
@@ -50,10 +49,6 @@ const (
defaultChunkSize = 8 * fs.Mebi
)
// jsonContentType is sent with JSON request bodies as the API doesn't
// decode non-ASCII file names in them as UTF-8 unless told the charset.
const jsonContentType = "application/json; charset=UTF-8"
// OAuth2 configuration
var oauthConfig = &oauthutil.Config{
Scopes: []string{
@@ -551,7 +546,6 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
var info *api.File
@@ -1020,7 +1014,6 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
copyReq := api.CopyFileRequest{
@@ -1038,7 +1031,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
fs.Debugf(f, "Copy: failed to get metadata options: %v", err)
}
if len(mergedMeta) > 0 {
copyReq.Properties = make(map[string]any)
copyReq.Properties = make(map[string]interface{})
for key, value := range mergedMeta {
switch key {
case "content-type", "sha256", "btime", "mtime", "utime",
@@ -1205,10 +1198,9 @@ func (f *Fs) updateFile(ctx context.Context, fileID string, parameters url.Value
}
parameters.Set("fields", "*")
opts := rest.Opts{
Method: "PATCH",
Path: "/files/" + fileID,
Parameters: parameters,
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
Method: "PATCH",
Path: "/files/" + fileID,
Parameters: parameters,
}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(ctx, &opts, req, &info)
@@ -1678,7 +1670,6 @@ func (o *Object) uploadEmpty(ctx context.Context, leaf, directoryID string, src
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
var info *api.File
@@ -1730,7 +1721,7 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
}
// Create metadata
metadata := map[string]any{
metadata := map[string]interface{}{
"fileName": leaf,
"mimeType": mimeType,
}
@@ -1750,10 +1741,9 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
metadata["autoRename"] = 3
}
// Build the multipart body in pooled memory so it can be re-sent on retry
rw := pool.NewRW(pool.Global())
defer fs.CheckClose(rw, &err)
writer := multipart.NewWriter(rw)
// Create multipart form
var buf bytes.Buffer
writer := multipart.NewWriter(&buf)
// Add metadata part - use exact format from Huawei docs
metadataWriter, err := writer.CreatePart(textproto.MIMEHeader{
@@ -1799,17 +1789,15 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
return fmt.Errorf("failed to close multipart writer: %w", err)
}
bodySize := rw.Size()
bodyBytes := buf.Bytes()
opts := rest.Opts{
Method: "POST",
Parameters: url.Values{
"uploadType": []string{"multipart"},
"fields": []string{"*"},
},
ContentType: fmt.Sprintf("multipart/related; boundary=%s", writer.Boundary()),
ContentLength: &bodySize,
RootURL: o.fs.uploadURL,
Body: rw,
ContentType: fmt.Sprintf("multipart/related; boundary=%s", writer.Boundary()),
RootURL: o.fs.uploadURL,
}
if o.id != "" {
@@ -1823,10 +1811,7 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, direct
var info *api.File
err = o.fs.pacer.Call(func() (bool, error) {
_, err := rw.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
opts.Body = bytes.NewReader(bodyBytes)
resp, err := srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(ctx, resp, err)
})
@@ -1854,7 +1839,6 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{
"Content-Type": jsonContentType,
"X-Upload-Content-Length": strconv.FormatInt(size, 10),
},
RootURL: o.fs.uploadURL,
@@ -1888,7 +1872,7 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
}
// Prepare metadata for resumable upload initialization
metadata := map[string]any{
metadata := map[string]interface{}{
"fileName": leaf,
}
@@ -1921,26 +1905,25 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
}
// Now upload the content in chunks
// Note 256k is the minimum chunk size and 64M the maximum
// single upload size according to the Huawei Drive API
chunkSize := min(max(int64(o.fs.opt.ChunkSize), 256*1024), 64*1024*1024)
chunkSize := int64(o.fs.opt.ChunkSize)
if chunkSize < 256*1024 {
chunkSize = 256 * 1024 // Minimum chunk size according to Huawei Drive API
}
if chunkSize > 64*1024*1024 {
chunkSize = 64 * 1024 * 1024 // Maximum single upload size
}
buf := make([]byte, chunkSize)
var offset int64
for offset < size {
// Buffer the chunk in pooled memory so it can be re-sent on retry
rw := pool.NewRW(pool.Global())
n, err := io.CopyN(rw, in, min(size-offset, chunkSize))
if err != nil {
// io.EOF here means the source ran out before the declared size
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
_ = rw.Close()
n, err := io.ReadFull(in, buf)
if err != nil && err != io.ErrUnexpectedEOF && err != io.EOF {
return fmt.Errorf("failed to read chunk at offset %d: %w", offset, err)
}
end := offset + n - 1
chunk := buf[:n]
end := offset + int64(n) - 1
// Upload this chunk using the original client with OAuth authentication
chunkOpts := rest.Opts{
@@ -1948,20 +1931,17 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
RootURL: location,
Path: "", // Empty path since RootURL contains the full URL
ExtraHeaders: map[string]string{
"Content-Range": fmt.Sprintf("bytes %d-%d/%d", offset, end, size),
"Content-Type": mimeType,
"Content-Range": fmt.Sprintf("bytes %d-%d/%d", offset, end, size),
"Content-Length": strconv.Itoa(n),
"Content-Type": mimeType,
},
ContentLength: &n,
Body: rw,
}
var info *api.File
var chunkResp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
_, err := rw.Seek(0, io.SeekStart)
if err != nil {
return false, err
}
chunkOpts.Body = bytes.NewReader(chunk)
var err error
chunkResp, err = srv.CallJSON(ctx, &chunkOpts, nil, &info)
// 308 means continue uploading (incomplete)
if chunkResp != nil && chunkResp.StatusCode == 308 {
@@ -1975,15 +1955,11 @@ func (o *Object) uploadResume(ctx context.Context, in io.Reader, leaf, directory
}
return shouldRetry(ctx, chunkResp, err)
})
closeErr := rw.Close()
if err != nil {
return fmt.Errorf("failed to upload chunk at offset %d: %w", offset, err)
}
if closeErr != nil {
return closeErr
}
offset += n
offset += int64(n)
// If we got file info back, we're done
if info != nil && info.ID != "" {
@@ -2114,8 +2090,8 @@ func (o *Object) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
// Prepare the update request payload
updateReq := api.UpdateFileRequest{
Properties: make(map[string]any),
AppSettings: make(map[string]any),
Properties: make(map[string]interface{}),
AppSettings: make(map[string]interface{}),
}
// Process metadata and separate into properties and app settings
@@ -2126,7 +2102,7 @@ func (o *Object) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
updateReq.Description = value
case "favorite":
if favorite, err := strconv.ParseBool(value); err == nil {
updateReq.Favorite = new(favorite)
updateReq.Favorite = api.BoolPtr(favorite)
}
case "content-type":
// Allow setting/overriding MIME type
@@ -2154,7 +2130,6 @@ func (o *Object) SetMetadata(ctx context.Context, metadata fs.Metadata) error {
Parameters: url.Values{
"fields": []string{"*"},
},
ExtraHeaders: map[string]string{"Content-Type": jsonContentType},
}
var info *api.File
-34
View File
@@ -19,49 +19,15 @@ import (
"github.com/rclone/rclone/lib/encoder"
)
// Limits on the chunk size accepted by the resumable upload API
const (
minChunkSize = 256 * fs.Kibi
maxChunkSize = 64 * fs.Mebi
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestHuaweiDrive:",
NilObject: (*Object)(nil),
SkipBadWindowsCharacters: true,
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: minChunkSize,
MaxChunkSize: maxChunkSize,
CeilChunkSize: fstests.NextPowerOfTwo,
},
})
}
// SetUploadChunkSize changes the configured chunk size, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.ChunkSize = f.opt.ChunkSize, cs
return old, nil
}
// SetUploadCutoff changes the configured upload cutoff, returning the old value.
//
// It is only called by the integration tests while no transfer is in progress.
func (f *Fs) SetUploadCutoff(cutoff fs.SizeSuffix) (fs.SizeSuffix, error) {
var old fs.SizeSuffix
old, f.opt.UploadCutoff = f.opt.UploadCutoff, cutoff
return old, nil
}
var (
_ fstests.SetUploadChunkSizer = (*Fs)(nil)
_ fstests.SetUploadCutoffer = (*Fs)(nil)
)
// TestNewFs tests the NewFs constructor
func TestNewFs(t *testing.T) {
ctx := context.Background()
+4 -3
View File
@@ -8,7 +8,6 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"net/http"
"os"
"path"
@@ -1618,7 +1617,9 @@ func (ps *PhotosService) GetLibraryAlbumCounts(ctx context.Context) (map[string]
if err := ps.requestForArea(ctx, area, "internal/records/query/batch", map[string]any{"batch": batch}, &response); err != nil {
return nil, fmt.Errorf("failed to get library album counts: %w", err)
}
maps.Copy(counts, response.toCounts(order))
for k, v := range response.toCounts(order) {
counts[k] = v
}
}
return counts, nil
}
@@ -2031,7 +2032,7 @@ func (album *Album) fetchPhotosParallel(ctx context.Context, totalPhotos int64)
sem := make(chan struct{}, workers)
var wg sync.WaitGroup
for i := range numPartitions {
for i := 0; i < numPartitions; i++ {
wg.Add(1)
go func(idx int) {
defer wg.Done()
+2 -2
View File
@@ -1191,7 +1191,7 @@ func TestFlushCaches_NoPendingDeltaRace(t *testing.T) {
done := make(chan struct{})
go func() {
defer close(done)
for range 100 {
for i := 0; i < 100; i++ {
lib.deltaMu.Lock()
lib.pendingDelta = &deltaPayload{
records: []json.RawMessage{json.RawMessage(`{"recordName":"m1","recordType":"CPLMaster"}`)},
@@ -1201,7 +1201,7 @@ func TestFlushCaches_NoPendingDeltaRace(t *testing.T) {
lib.applyPendingDelta(context.Background())
}
}()
for range 100 {
for i := 0; i < 100; i++ {
ps.FlushCaches()
// Re-add the library since FlushCaches clears it
ps.mu.Lock()
+5 -3
View File
@@ -718,7 +718,7 @@ func (s *Session) GetAuthState(ctx context.Context) (*AuthStateResponse, error)
Path: "",
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
RootURL: authEndpoint,
ContentLength: new(int64(0)),
ContentLength: int64Ptr(0),
}
// Use srv.Call directly to capture the raw response body for debugging
resp, err := s.srv.Call(ctx, &opts)
@@ -828,7 +828,7 @@ func (s *Session) TrustSession(ctx context.Context) error {
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
RootURL: authEndpoint,
NoResponse: true,
ContentLength: new(int64(0)),
ContentLength: int64Ptr(0),
}
_, err := s.Request(ctx, opts, nil, nil)
@@ -846,7 +846,7 @@ func (s *Session) ValidateSession(ctx context.Context) error {
Path: "/validate",
ExtraHeaders: s.GetHeaders(map[string]string{}),
RootURL: setupEndpoint,
ContentLength: new(int64(0)),
ContentLength: int64Ptr(0),
}
_, err := s.Request(ctx, opts, nil, &s.AccountInfo)
if err != nil {
@@ -904,6 +904,8 @@ func GetCommonHeaders(overwrite map[string]string) map[string]string {
return headers
}
func int64Ptr(v int64) *int64 { return &v }
// NewSession creates a new Session instance with default values
func NewSession() *Session {
session := &Session{
+1 -2
View File
@@ -92,7 +92,7 @@ func (f *Fs) findItem(ctx context.Context, dir string) (item *api.DriveItem, fou
item, resp, err = f.service.GetItemByPath(ctx, path.Join(f.root, dir))
return shouldRetry(ctx, resp, err)
}); err != nil {
if item == nil && resp != nil && resp.StatusCode == 404 {
if item == nil && resp.StatusCode == 404 {
return nil, false, nil
}
return nil, false, err
@@ -694,7 +694,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
}
f.features = (&fs.Features{
CaseInsensitive: true, // names are matched with EqualFold, see findLeafItem
CanHaveEmptyDirectories: true,
PartialUploads: false,
}).Fill(ctx, f)
+6 -2
View File
@@ -376,7 +376,8 @@ func (f *PhotosFs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut
return "", false, nil
}
if libraryName, ok := strings.CutPrefix(pathID, "lib:"); ok {
if strings.HasPrefix(pathID, "lib:") {
libraryName := strings.TrimPrefix(pathID, "lib:")
libraries, err := photosService.GetLibraries(ctx)
if err != nil {
return "", false, err
@@ -735,7 +736,10 @@ func (f *PhotosFs) ListR(ctx context.Context, dir string, callback fs.ListRCallb
}
close(jobCh)
workers := min(len(jobs), fs.GetConfig(ctx).Checkers)
workers := fs.GetConfig(ctx).Checkers
if len(jobs) < workers {
workers = len(jobs)
}
errs := make(chan error, workers)
for range workers {
-2
View File
@@ -77,8 +77,6 @@ func (ik *ImageKit) Upload(ctx context.Context, file io.Reader, param UploadPara
if err != nil {
return nil, nil, fmt.Errorf("failed to make multipart upload: %w", err)
}
// Stop the form writer reading file if the request fails before sending it all
defer func() { _ = formReader.Close() }()
opts := rest.Opts{
Method: "POST",
+2 -2
View File
@@ -477,7 +477,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) (err error) {
FolderPath: f.EncodePath(path.Join(f.root, dir)),
})
if res != nil && res.StatusCode == http.StatusNotFound {
if res.StatusCode == http.StatusNotFound {
return false, fs.ErrorDirNotFound
}
@@ -502,7 +502,7 @@ func (f *Fs) Purge(ctx context.Context, dir string) (err error) {
FolderPath: f.EncodePath(remote),
})
if res != nil && res.StatusCode == http.StatusNotFound {
if res.StatusCode == http.StatusNotFound {
return false, fs.ErrorDirNotFound
}
@@ -634,27 +634,6 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (_ fs.Objec
"x-archive-filemeta-rclone-update-track": updateTracker,
}
mdata, err := fs.GetMetadataOptions(ctx, f, src, fs.MetadataAsOpenOptions(ctx))
if err != nil {
return nil, err
}
if mdata != nil {
// IA ignores file metadata headers with the COPY directive.
headers["x-amz-metadata-directive"] = "REPLACE"
}
for key, value := range mdata {
key = strings.ToLower(key)
if key == "mtime" {
key = "rclone-mtime"
} else if strings.HasPrefix(key, "rclone-") {
// Keep the new copy tracker and mapped mtime, not the source internals.
continue
} else if _, readOnly := roMetadataKey[key]; readOnly {
continue
}
headers["x-archive-filemeta-"+key] = value
}
// make a PUT request at (IAS3)/:item/:path without body
var resp *http.Response
opts := rest.Opts{
Loaded 100 of 867 files, more files were not shown because too many files have changed in this diff. Show more