Add the TestProtonDriveFake init.d script, which builds and starts
protonfake on port 28642 and passes on the session it prints, and a
test_all entry for it, so fs/operations, fs/sync, vfs and cmd/bisync
run against the fake server. Five tests that need a second Fs of the
same account to see what the first wrote are ignored, as each Fs
caches separately, and cmd/gitannex is skipped as for every init.d
remote; both are in docs/TODO.md.
Add a Gitea Actions workflow that builds, vets, lints, race-tests,
fuzzes and vulnerability-checks the backend, and re-verify the agent
permission table on the host toolchain.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Apply metadata overrides and the metadata mapper before queuing a
server-side copy. Preserve the new update tracker and read-only file
attributes, and return metadata errors before sending a PUT.
Fixes#7901
Assisted-by: OpenAI Codex
Co-authored-by: david.castro <david.castro@refinery89.com>
The 5 chunker remotes with 50 byte chunks split each test file into
many objects which made them expensive to run against AWS S3. Run
them against the TestS3Minio docker server instead and keep
TestChunkerS3 with the default chunk size on AWS.
If a remote has no start script but wraps another remote via its
"remote" config value, start the server for the wrapped remote
instead. This lets wrapping backends be tested against the docker
test servers, e.g. chunker over TestS3Minio.
test_all now honours RCLONE_CONFIG in the same way as the tests it
runs, so it finds the same wrapping remotes.
Their fixed ports, 39980 and 38081, are in Linux's ephemeral range
(32768-60999), so a socket from another concurrent test could hold
them and stop the server starting. This made all 5 tries of the Sia
fs/operations tests fail with "address already in use".
Seahub throttles anonymous API calls, which includes logging in, to 60
per minute. When cmd/bisync and backend/seafile ran concurrently they
exceeded this and every new remote failed with "failed to
authenticate" until the end of the retries.
The fauria/vsftpd image limits each IP to 50 connections. When
cmd/bisync and backend/ftp ran concurrently against the shared server
the backend tests were refused with 421 "There are too many
connections from your internet address" on every retry.
Linkbox renames an upload to name(1) when its name is the start of an
existing file's name, which breaks TestCaseInsensitiveMoveWithoutServerSideMove.
Internxt listings lag behind directory renames and deletes for longer than
the list retries allow, which breaks TestDirRename and
TestDirRenameParentEntries.
Remotes made by RandomRemote aren't in the fs cache so they were never
shut down, leaving any pooled connections open until they timed out.
On TestFTPProftpd this used up the server's 30 connection limit part way
through the fs/operations tests, making all the following tests fail
with EOF when connecting.
Linkbox converts file extensions to lower case on upload, and renames an
uploaded file to name(1) if its name is the start of the name of a file
already in the directory, e.g. "apple" next to "applebanana". Document
both and ignore the TestTransform cases which they break.
GCS can't store CR or LF in object names and the S3 provider doesn't
encode them. These cases ran once the bisync pre-flight check was fixed
to stop skipping them.
Before this change, `cmd/bisync` was excluded on `TestS3Rclone`
(`rclone serve s3`), one of the backends `108504963` excluded because the
tests always failed there at the time. After this change, they run there
again: with `test_all` on Linux, every case passes or skips, with or
without the preceding changes to the test harness.
Add an integration test which starts a listener on localhost, makes
the remote with roots naming that listener and checks that nothing
connects to it. The listener replies with junk so a backend which does
connect fails at once rather than retrying a closed connection.
FsOpenChunkWriter wrote three chunks of its own fixed sizes whatever
ChunkWriterInfo.ChunkSize the backend returned. That happened to be
the size the s3 backend uses by default, so the test passed there, but
a chunk writer which works out where a chunk goes from its number needs
every chunk but the last to be exactly the size it asked for - as a
multi-thread copy writes them - and the test could neither exercise nor
catch that.
Cut the file into chunks of info.ChunkSize, the last one short, still
written out of order. Where the backend lets the test set its chunk
size, ask for 5 MiB first so the file takes several chunks. Where it
doesn't and the chunks leave the file in fewer than three, start again
with a file of two full chunks and a short one, for chunks up to
128 MiB.
Add a new backend for dosya.dev, a cloud file storage, sharing, and
synchronization platform with workspace-based multi-tenancy.
Features:
- Server-side copy, move, rename (files and directories)
- Purge (recursive directory delete)
- Recursive listing (ListR) for --fast-list
- Multipart upload for large files (>10MB)
- Public link sharing via rclone link
- About (workspace storage quota)
- MIME type detection
- Empty directory support
- Workspace-based multi-tenancy (one remote per workspace)
- Download via presigned R2 URLs with Range header support
Co-authored-by: firatkaya <firat@netiket.com.tr>
Files.com now treats the replacement characters rclone uses for / and
\ in file names (／ and ＼) as if they were the originals, and the
names ． and ．． as if they were . and .., and rejects them all with
"Invalid path". There is no other way of encoding these names, so
document the restriction and ignore the integration tests for them.
Koofr refuses to download the HTML listings written by the index tests
with "FileBlocked: File download restricted due to possible dangerous
content" and OpenDrive refuses with 403 Forbidden, so the tests can't
read back what they wrote. The index tests which write other formats
still run.
MinIO no longer publishes images on quay.io (nor Docker Hub or ghcr.io)
so pulling quay.io/minio/minio returns "unauthorized". pgsty/minio is a
maintained community fork with the same entrypoint layout.
The start function was run inside an if condition, where bash ignores
errexit, so a failed docker run was not noticed. The script then
printed its connection details anyway and the test spent 100 seconds
trying to connect before failing with a message that hid the real
error.
Run start in a subshell with errexit on and check its status
explicitly so the failure is reported immediately with docker's
error message.
The minio/minio repository has been removed from Docker Hub so the
TestS3Minio test server could not be started and every Linux CI run
failed. quay.io/minio/minio still serves the final release so use
that instead.
OCI rejects a multipart completion request with no parts. Probe unknown-size
streams through a buffered reader and use a regular upload when the stream is
empty. Peek preserves a non-empty stream for the selected upload path.
OCI Object Storage rejects NUL, carriage return and line feed in
object names. NUL is already encoded by rclone, while carriage returns
and line feeds need EncodeCrLf so rclone can represent those names
without issuing invalid OCI requests.
https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/managingobjects.htm
Add an FsPutRetry integration test which uses the fshttp fault injector
to fail the first request carrying upload data, before it is sent, with
either a synthetic 500 or a connection reset, and then checks that Put
either succeeds with the correct contents or fails cleanly without
leaving a corrupt object behind.
It runs a single part upload and, for backends implementing
SetUploadChunkSizer, a chunked one. This catches backends which retry
an upload from a buffer which has already been consumed or returned to
the memory pool. Backends which don't use HTTP skip the test.
Also check in FsPutRetry and the large upload tests that the number of
buffers in use in the global memory pool is unchanged after an upload,
to catch backends which leak pooled buffers.
The sftp backend could be used as the source of a multi-thread copy -
each chunk of a download is read on its own connection - but not as the
destination, because it did not implement OpenWriterAt. Uploading a
single large file was therefore limited to one connection while
downloading the same file was not, capping upload throughput well below
the link speed on high latency connections.
This implements OpenWriterAt for the sftp backend using lib/filepool. A
small pool of open write handles, each backed by its own connection,
lets the core write the chunks of a large file concurrently over several
connections. The file is created and truncated once up front so every
chunk offset is valid before the concurrent writes start.
Multi-thread uploads are off by default and turned on with the new
--sftp-multithread-upload flag, since many sftp servers only accept
sequential writes and would fail large uploads otherwise. Even when
enabled they fall back to a single connection when
--sftp-disable-concurrent-writes is set (a server that can't take
out-of-order packets on one handle won't take several handles either) or
when --sftp-connections caps the pool (the per-file fan-out would
otherwise deadlock waiting on it).
The OpenSSH test server (TestSFTPOpenssh) enables
--sftp-multithread-upload so the feature is exercised in CI against a
real server.
Tested against OpenSSH: a single large upload over a high latency link
went from ~12 MB/s to ~90 MB/s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Splainte <r.wycke@hotmail.fr>
Pikpak never returns MD5 for uploads which causes this test to fail.
Perhaps Pikpak should not declare MD5 but that is a bigger decision
being discussed in #9826
A source reader which returns EOF before supplying as many bytes as it
declared should not cause a backend Put to hang, panic or create an
object claiming the declared size.
The layers above detect a truncated transfer by comparing sizes after
the upload, so a backend may either return an error or create a
shorter object.
See #9704
Every write operation waits for the server's processing tasks which
currently take minutes each, so the full test suite takes much longer
than the timeout. Running with -short skips the FsEncoding subtests
(which are already on the ignore list) and the chunked upload tests,
removing a large number of these waits.
The old rclone-integration-test item has accumulated years of catalog
task churn and IA now deprioritises and periodically holds its tasks,
which makes the tests wait on the server's processing tasks for a
very long time. Start again with a fresh item (created with noindex
set so the test files stay out of IA's search index).
ImageKit can't store zero length files. The test only skips itself
when the failure is reported synchronously, but with --vfs-cache-mode
writes the writeback happens asynchronously and deliberately swallows
the error, so the test fails on the missing files. Other backends
which can't store empty files already ignore this test.
Cloudinary's search API which NewObject uses is eventually
consistent, so the --compare-dest lookup of a just uploaded file
misses it and the test fails. TestCopyFileCompareDest is already
ignored for the same reason.
Layers which wrap backends, such as the hasher backend and the VFS
cache, use fs.Fingerprint to detect whether an object has changed.
This only works if the object a backend hands back from a write
operation is identical to the object a fresh listing returns. If a
backend keeps the source's nanosecond precision modtime in memory
while the server stores milliseconds, or omits a hash the server
knows, every fingerprint comparison fails and cached hashes or files
are silently invalidated.
The existing tests compared modtimes within f.Precision() so they
could not detect these sub-precision divergences.
Add a checkFingerprint helper which asserts the fast and slow
fs.Fingerprint of the object returned from a write operation exactly
match those of the object read back from the remote, and wire it into
all the write paths which return or mutate an object: Put (all
variants), PutStream, large and streamed uploads, Update, SetModTime,
server side Copy and Move, and the metadata mutating Copy/Move
subtests.
Rework the ObjectOpenFingerprint test to use the same helper against
a fully read reference object, with subtests reporting which kind of
read is being checked.
Add TestObjectOpenFingerprint which reads an object's fast and slow
fingerprints, then opens it in several ways - a full read, a seek read
and ranged reads - and after a refresh, checking the fingerprint never
changes.
TestS3Minio brings up a minio container via the fstest/testserver
framework, which exec's bash init.d scripts that shell out to docker.
This is not available on all platforms - Windows has no POSIX shell to
run the scripts, and macOS CI runners have no docker daemon - which
caused the build to fail there.
Add testy.SkipUnlessDocker to detect whether the framework can run and
skip the test when it cannot.
The bisync normalization test relies on uploading distinct NFC and NFD
versions of the same filename and on the backend supporting in-place
modtime updates. Dropbox normalizes unicode server-side (NFD -> NFC)
and can't set modtime in place, so the test inevitably takes a
different code path on Dropbox and the log diverges from the golden
output without any functional difference.
