Compare commits

...
Author SHA1 Message Date
albertony 25bf29db20 build: bump DavidAnson/markdownlint-cli2-action from 22 to 23 2026-04-04 21:45:47 +02:00
albertony 1680e76ff5 docs: fix markdownlint issue MD013/line-length Line length 2026-04-04 21:44:48 +02:00
albertony 962cf6d53b docs: fix markdownlint issue MD060/table-column-style 2026-04-04 21:44:45 +02:00
dependabot[bot] fde3a4e2ee build: bump DavidAnson/markdownlint-cli2-action from 20 to 22
Bumps [DavidAnson/markdownlint-cli2-action](https://github.com/davidanson/markdownlint-cli2-action) from 20 to 22.
- [Release notes](https://github.com/davidanson/markdownlint-cli2-action/releases)
- [Commits](https://github.com/davidanson/markdownlint-cli2-action/compare/v20...v22)

---
updated-dependencies:
- dependency-name: DavidAnson/markdownlint-cli2-action
  dependency-version: '22'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-04-04 21:43:15 +02:00
albertony 1f3770a57f docs: fix markdown issues in mount docs 2026-04-03 17:30:56 +01:00
Clément Notin 829b06d39b docs: fix header level for metadata option 2026-04-03 14:32:24 +02:00
Nick Craig-Wood 0db3e7a2a0 vfs: fix slow nfs serve by adding --vfs-handle-caching
Add a configurable grace period (default 5s) that delays closing file
handles and downloaders when the last handle closes. If a new handle
opens within the grace period, it reuses the existing resources.

This fixes 40x performance degradation with serve nfs vs serve sftp
caused by go-nfs opening/reading/closing on every NFS READ RPC, which
destroyed read-ahead prefetch before it could accumulate.

The grace period only applies to non-dirty files so that writeback
proceeds immediately on close.

Fixes #9251
2026-04-03 11:10:50 +01:00
Nick Craig-Wood 654df91375 Add Xiangzhe to contributors 2026-04-03 11:10:50 +01:00
Nick Craig-Wood f079361a01 Add Mike GIllan to contributors 2026-04-03 11:10:50 +01:00
Ross Smith II 3677b7ff77 fix(docs): Fix link to not be language specific 2026-04-02 17:57:02 +01:00
Mike GIllan 4a00a4dc4b iclouddrive: lowercase Apple ID for SRP authentication
Apple IDs are case-insensitive, but the SRP proof computation (M1)
hashes the username client-side. The old plaintext signin let the
server normalize the case, but with SRP the client must match.
Lowercase the Apple ID before use so mixed-case IDs authenticate
correctly.

Reported-by: ArturKlauser
2026-04-02 17:52:56 +01:00
Xiangzhe 2610beb18d iclouddrive: use dynamic origin for SRP auth headers
This fixes China mainland iCloud authentication by deriving the Origin
and Referer headers from authEndpoint instead of hardcoding idmsa.apple.com.

Fixes compatibility with PR #8818 (China region support) and PR #9209
(SRP authentication).

Signed-off-by: Xiangzhe <xiangzhedev@gmail.com>
2026-04-02 17:52:56 +01:00
Mike GIllan 35e4f60548 iclouddrive: replace plaintext signin with SRP authentication
Apple has deprecated the legacy /appleauth/auth/signin endpoint and
now blocks it, causing "Invalid Session Token" errors for all users
when their trust token expires. The browser login flow now requires
SRP (Secure Remote Password), a cryptographic handshake that never
transmits the password.

Replace Session.SignIn() with a multi-step SRP-6a flow:
1. authStart - initialize session at /authorize/signin
2. authFederate - submit account name to /federate
3. authSRPInit - exchange client public value for salt/B at /signin/init
4. authSRPComplete - send M1/M2 proofs to /signin/complete

The SRP implementation uses the RFC 5054 2048-bit group with SHA-256
and Apple's NoUserNameInX variant. Password derivation supports both
s2k and s2k_fo protocols via SHA-256 + PBKDF2.

The 2FA and trust token flow is unchanged. Auth headers for all
idmsa.apple.com requests now include X-Apple-Auth-Attributes,
X-Apple-Frame-Id, and use Origin/Referer of https://idmsa.apple.com.

Fixes #8587
2026-04-02 17:52:56 +01:00
Nick Craig-Wood 839a3b92e0 docs: modernize rclone.org site design
- replace Bootstrap/jQuery with purpose-built CSS and JS
- remove backend icons from navbar and content pages
- replace remaining FontAwesome icons with inline SVGs, remove FontAwesome
- modernize CSS styling for menus, typography, cards, tables, and code blocks
- add copy-to-clipboard buttons on code blocks using SVG icon
- move TOC to left sidebar with responsive overlay drawer
- add sticky header, top scrollbar and first column for wide tables
- add left/right arrow buttons to scrollable tables
- hide homepage logo on mobile
- make wide menus with filter for Commands and Storage Systems
- add dark mode support based on browser preference
- fix CSS/JS cache busting to use build time
2026-04-02 16:32:54 +01:00
Nick Craig-Wood 492e0bd616 Add Andriy Senyshyn to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood d4894db765 Add Claude Opus 4.6 to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 8d09f16681 Add jinyu.han to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 550889c096 Add jinkeyuu to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 896961404b Add lif to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 5eabfc74f4 Add BizaNator to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 787d033376 Add Patrick Farrell to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood db258d404a Add Jason to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 3cdf680a15 Add ZRHan to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood bc57110920 Add Andrew Furman to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 133eae4d5b Add Andriy Senyshyn to contributors 2026-04-02 16:32:54 +01:00
Nick Craig-Wood 24c40463eb Add Bhagyashreek8 to contributors 2026-04-02 16:32:54 +01:00
jinkeyuuandjinyu.han e9fddaabeb s3: add UCloud Object Storage provider (#9230)
Co-authored-by: jinyu.han <jinyu.han@ucloud.cn>
2026-03-31 11:45:40 +01:00
lif c49015552c bisync: fix handling of unreadable lockfiles - fixes #9290
Lockfiles with invalid JSON content caused bisync to fail permanently
because lockFileIsExpired() logged the decode error but still fell
through to the "valid lock file" path with zero-value TimeExpires.

Now when a JSON decode error is detected:
- If --max-lock is set (< basicallyforever): treat garbled lockfile as
  expired, mark listings failed, and proceed (safe assumption: the
  previous bisync run crashed and left garbage).
- If --max-lock is not set (default): log a clear error telling the
  user the lockfile needs manual inspection, and return false.
2026-03-31 10:56:28 +01:00
BizaNatorandClaude Opus 4.6 ffdd043b95 librclone/ctest: add Windows support and fix memory management
Make ctest build and run on Windows in addition to Linux/macOS:

- Add OS detection in Makefile using ifeq ($(OS),Windows_NT)
- Use .lib extension and .exe suffix on Windows
- Link Windows system libraries (winmm, ws2_32, ole32)
- Remove unused dlfcn.h include that prevented compilation on Windows

Fix memory management to use RcloneFreeString instead of free for
strings returned by RcloneRPC, as documented in the librclone README.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 10:51:15 +01:00
Chris 434edba275 s3: fix regression where PutObject fails with non-seekable readers
Commit a3e1312d accidentally replaced io.NopCloser(in) with a bare
io.Reader when assigning req.Body in uploadSinglepartPutObject.

rclone wraps upload readers in an accounting.Account for progress
tracking. When the AWS SDK calls Seek on the body, Account.Seek does
a type assert on the inner reader. With a bare io.Reader the type
is unexpected and causes:

  operation error S3: PutObject, serialization failed: internal error:
  Seek not implemented for io.nopCloser

With io.NopCloser(in) the type assert works correctly for both seekable
and non-seekable readers.

Restore io.NopCloser(in) to wrap the reader correctly in all cases.

Verified by running both before (regression confirmed) and after (fix
confirmed):
  go test ./backend/s3/... ./fs/operations/... -remote TestS3:
2026-03-31 10:22:27 +01:00
Enduriel da7a3d8d64 filen: update SDK version
- increase timeout for http requests
- only use a single URL per request type
2026-03-30 22:56:32 +01:00
dependabot[bot] 2362b4ca50 build(deps): bump golang.org/x/image from 0.36.0 to 0.38.0
Bumps [golang.org/x/image](https://github.com/golang/image) from 0.36.0 to 0.38.0.
- [Commits](https://github.com/golang/image/compare/v0.36.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/image
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-30 22:55:35 +01:00
kapitainsky 5e3d783b3e docs: note macOS 10.15 (Catalina) support with version v1.70.3
due to min golang requirements macOS Catalina (10.15) can not run newer rclone versions
2026-03-28 12:29:53 +00:00
Bjoern Franke 7a63990df2 Add OVHcloud storage classes
Added OVHcloud storage classes according to https://help.ovhcloud.com/csm/en-ie-public-cloud-storage-s3-choosing-right-storage-class?id=kb_article_view&sysparm_article=KB0047293 and https://help.ovhcloud.com/csm/en-ie-public-cloud-storage-s3-choosing-right-storage-class?id=kb_article_view&sysparm_article=KB0047293
2026-03-25 21:48:56 +00:00
Patrick Farrell 7ca667d35d local: remove fadvise calls that cause spinlock contention
Remove the POSIX_FADV_DONTNEED and POSIX_FADV_SEQUENTIAL calls
from the local backend. The DONTNEED calls cause severe spinlock
contention on parallel file systems (and any system with many
concurrent transfers), because each call triggers per-page cache
teardown under a global lock.

Observed on a 256-core system running rclone with 64 parallel
transfers over Lustre: 69% of all CPU cycles were spent in
kernel spinlock contention from the fadvise path, with effective
throughput well below hardware capability.

The kernel's own page reclaim (kswapd) handles eviction more
efficiently from a single context. Since rclone does not always
read files sequentially (e.g. multipart uploads rewind and
re-read blocks), FADV_SEQUENTIAL was also not reliably correct.

This is consistent with the non-Linux behavior (which never
called fadvise) and with restic's decision to remove identical
code (restic/restic#670).

Fixes #7886
2026-03-24 10:17:36 +00:00
Nick Craig-Wood 1e22af8358 Changelog updates from Version v1.73.3 2026-03-23 23:03:59 +00:00
dependabot[bot] 589d45a0bf build(deps): bump github.com/buger/jsonparser from 1.1.1 to 1.1.2
Bumps [github.com/buger/jsonparser](https://github.com/buger/jsonparser) from 1.1.1 to 1.1.2.
- [Release notes](https://github.com/buger/jsonparser/releases)
- [Commits](https://github.com/buger/jsonparser/compare/v1.1.1...v1.1.2)

---
updated-dependencies:
- dependency-name: github.com/buger/jsonparser
  dependency-version: 1.1.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-23 17:05:35 +00:00
albertony d5deec3534 docs/jottacloud: fix broken link 2026-03-20 23:33:27 +01:00
Jason 53c19e04ba docs: clarify Filen password change requires updating both password and API key in rclone config 2026-03-20 15:26:20 +00:00
Jason e7fadc7e3d docs: note that Filen API key changes on password change 2026-03-20 11:08:59 +00:00
dependabot[bot] 7dfc5b3076 build(deps): bump google.golang.org/grpc from 1.79.1 to 1.79.3
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.79.1 to 1.79.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.79.1...v1.79.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-version: 1.79.3
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-19 11:02:37 +00:00
ZRHan 523a29a4e9 webdav: request only required properties in listAll to improve performance
This PR optimizes the PROPFIND requests in the webdav backend to only ask for
the specific properties rclone actually needs.

Currently, the generic webdav backend sends an empty XML body during directory
listing (listAll), which causes the server to fall back to allprops by default.
This forces the server to return properties we never use, such as
getcontenttype.

Fetching getcontenttype can be a very expensive operation on the server side.
For instance, in the official golang.org/x/net/webdav library, determining the
content type requires the server to open the file and read the first 500 bytes.

For a directory with 1,300 files in my environment, rclone ls time dropped from
~30s to ~4s (as fast as native ls).

This only applies to the other vendor for backwards compatibility which could be
expanded.
2026-03-16 17:29:45 +00:00
Chris a3e1312d9d s3: fix Content-MD5 for Object Lock uploads and add GCS quirk
AWS S3 requires Content-MD5 for PutObject with Object Lock parameters.
Since rclone passes a non-seekable io.Reader, the SDK cannot compute
checksums automatically. Buffer the body and compute MD5 manually for
singlepart PutObject and presigned request uploads when Object Lock
parameters are set. Multipart uploads are unaffected as Object Lock
headers go on CreateMultipartUpload which has no body.

Add object_lock_supported provider quirk (default true) to allow
skipping Object Lock integration tests on providers with incomplete
S3 API support. Set to false for GCS which uses non-standard
x-goog-bypass-governance-retention header and doesn't implement
PutObjectLegalHold/GetObjectLegalHold.

Add Multipart and Presigned subtests to Object Lock integration tests
to cover all three upload paths.

Fixes #9199
2026-03-14 22:18:43 +00:00
Marco Ferretti e987d4f351 s3: add multi tenant support for Cubbit 2026-03-14 22:15:47 +00:00
Andrew Furman 639bd88959 lib/rest: fix URLPathEscapeAll breaking WebDAV servers (eg nzbdav) with strict path matching
URLPathEscapeAll was only passing [A-Za-z0-9/] through unencoded, causing
it to percent-encode RFC 3986 unreserved characters (-, ., _, ~). Per RFC
3986 §2.3, unreserved characters MUST NOT be percent-encoded, and a URI
that unnecessarily encodes them is not equivalent to one that does not.

Servers that perform strict path matching without normalising
percent-encoded characters will reject the over-encoded form with a 404.

Before: /files/my-report.pdf → /files/my%2Dreport%2Epdf
After:  /files/my-report.pdf → /files/my-report.pdf

Reserved characters (spaces, semicolons, colons, etc.) continue to be
encoded as before.
2026-03-12 21:58:22 +00:00
Andriy SenyshynandAndriy Senyshyn bfa2fbb78e copyurl: fix ignored --upload-headers and --download-headers
Co-authored-by: Andriy Senyshyn <andriy.senyshyn@kiteworks.com>
2026-03-12 17:30:13 +00:00
Bhagyashreek8 69ccbacf30 s3: IBM COS: provide ibm_iam_endpoint as a configurable param for IBM IAM-based auth 2026-03-12 10:04:16 +00:00
Nick Craig-Wood 5d6690eb20 list: fix nil pointer panic in Sorter when temp file creation fails
When extsort.Strings() cannot create temporary files (e.g. due to
apparmor restrictions or permission denied), it returns a nil sorter
with the error on errChan. The code then called Sort() on the nil
sorter, causing a panic.

Check for nil sorter and return the error instead of panicking.

Fixes #9244
2026-03-12 10:00:56 +00:00
Nick Craig-Wood c7b8654b17 docs: update RELEASE procedure to avoid mistakes 2026-03-11 11:26:16 +00:00
Nick Craig-Wood 906b27d8b9 Add Billy Hughes to contributors 2026-03-11 11:26:16 +00:00
Billy Hughes e233f8b0a7 accounting: Add deletedDirs stat to core/stats help output 2026-03-09 18:43:19 +00:00
Jan-Philipp ReßlerandJan-Philipp Reßler 17a547a980 docs: added text to the label showing version-introduced info
Co-authored-by: Jan-Philipp Reßler <xodarap@xodarap.de>
2026-03-09 18:35:20 +00:00
Nick Craig-Wood cdb3ecb0fb Changelog updates from Version v1.73.2 2026-03-06 20:43:59 +00:00
Nick Craig-Wood 2d7f84576a fs/log: fix data race on OutputHandler.format field
The format field was read in Handle() without synchronization while
setFormat() could write it concurrently from InitLogging(). This
caused a data race detected by the race detector, failing
TestListBucketsAuthProxy in cmd/serve/s3.

Fix by protecting all access to format with the existing mutex.
2026-03-06 14:41:23 +00:00
dependabot[bot] ab234dcfd7 build(deps): bump docker/build-push-action from 6 to 7
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6 to 7.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](https://github.com/docker/build-push-action/compare/v6...v7)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-06 09:49:05 +00:00
dependabot[bot] c5bcfbac8f build(deps): bump docker/setup-buildx-action from 3 to 4
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](https://github.com/docker/setup-buildx-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-06 09:48:25 +00:00
dependabot[bot] 88ed8e2b43 build(deps): bump docker/metadata-action from 5 to 6
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](https://github.com/docker/metadata-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-06 09:47:33 +00:00
dependabot[bot] b097454f83 build(deps): bump docker/setup-qemu-action from 3 to 4
Bumps [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) from 3 to 4.
- [Release notes](https://github.com/docker/setup-qemu-action/releases)
- [Commits](https://github.com/docker/setup-qemu-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/setup-qemu-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-05 12:36:44 +00:00
dependabot[bot] 2f890114b6 build(deps): bump docker/login-action from 3 to 4
Bumps [docker/login-action](https://github.com/docker/login-action) from 3 to 4.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-05 12:36:13 +00:00
nielash 78a7d9bb8e bisync: update changelog 2026-03-03 16:13:00 -05:00
nielash 72c561d209 bisync: auto-generate rc help docs
This adds a go generate ./cmd/bisync command to autogenerate the bisync rc docs,
including the list of params.
2026-03-03 16:13:00 -05:00
nielash b864c4f9c9 bisync: add more structured info to rc output
This adds a few handy bits of info, like the session name, workdir location, and
listing file paths, to the rc output.
2026-03-03 16:13:00 -05:00
nielash bb78eb8ab2 bisync: add missing rc params - fixes #7799
This adds 11 previously-missing rc params for newer bisync features.

It also makes optional parameters truly optional. (Previously, callers were
required to supply every single one, even if using the default value.)
2026-03-03 16:13:00 -05:00
165 changed files with 4227 additions and 1027 deletions

No files matched your search

+1 -1
View File
@@ -283,7 +283,7 @@ jobs:
run: govulncheck ./...
- name: Check Markdown format
uses: DavidAnson/markdownlint-cli2-action@v20
uses: DavidAnson/markdownlint-cli2-action@v23
with:
globs: |
CONTRIBUTING.md
@@ -100,7 +100,7 @@ jobs:
- name: Extract Metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
env:
DOCKER_METADATA_ANNOTATIONS_LEVELS: manifest,manifest-descriptor # Important for digest annotation (used by Github packages)
with:
@@ -122,10 +122,10 @@ jobs:
type=raw,value=beta,enable={{is_default_branch}}
- name: Setup QEMU
uses: docker/setup-qemu-action@v3
uses: docker/setup-qemu-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Load Go Build Cache for Docker
id: go-cache
@@ -148,7 +148,7 @@ jobs:
skip-extraction: ${{ steps.go-cache.outputs.cache-hit }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
# This is the user that triggered the Workflow. In this case, it will
@@ -159,7 +159,7 @@ jobs:
- name: Build and Publish Image Digest
id: build
uses: docker/build-push-action@v6
uses: docker/build-push-action@v7
with:
file: Dockerfile
context: .
@@ -210,7 +210,7 @@ jobs:
- name: Extract Metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
uses: docker/metadata-action@v6
env:
DOCKER_METADATA_ANNOTATIONS_LEVELS: index
with:
@@ -261,16 +261,16 @@ jobs:
env.write(f"ANNOTATIONS={annotations_string}\n")
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
uses: docker/setup-buildx-action@v4
- name: Login to Docker Hub
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ghcr.io
# This is the user that triggered the Workflow. In this case, it will
+7 -7
View File
@@ -412,8 +412,8 @@ the source file in the `Help:` field:
- The `backenddocs` make target runs the Python script `bin/make_backend_docs.py`,
and you can also run this directly, optionally with the name of a backend
as argument to only update the docs for a specific backend.
- **Do not** commit the updated Markdown files. This operation is run as part of
the release process. Since any manual changes in the autogenerated sections
- **Do not** commit the updated Markdown files. This operation is run as part
of the release process. Since any manual changes in the autogenerated sections
of the Markdown files will then be lost, we have a pull request check that
reports error for any changes within the autogenerated sections. Should you
have done manual changes outside of the autogenerated sections they must be
@@ -580,7 +580,8 @@ remote or an fs.
make sure we can encode any path name and `rclone info` to help determine the
encodings needed
- `rclone purge -v TestRemote:rclone-info`
- `rclone test info --all --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info`
- `rclone test info --all --remote-encoding None -vv --write-json remote.json
TestRemote:rclone-info`
- `go run cmd/test/info/internal/build_csv/main.go -o remote.csv remote.json`
- open `remote.csv` in a spreadsheet and examine
@@ -627,10 +628,9 @@ See the [testing](#testing) section for more information on integration tests.
### Backend documentation
Add your backend to the docs - you'll need to pick an icon for it from
[fontawesome](http://fontawesome.io/icons/). Keep lists of remotes in
alphabetical order of full name of remote (e.g. `drive` is ordered as
`Google Drive`) but with the local file system last.
Add your backend to the docs. Keep lists of remotes in alphabetical
order of full name of remote (e.g. `drive` is ordered as `Google
Drive`) but with the local file system last.
First add a data file about your backend in
`docs/data/backends/remote.yaml` - this is used to build the overview
+20 -20
View File
@@ -2,27 +2,27 @@
Current active maintainers of rclone are:
| Name | GitHub ID | Specific Responsibilities |
| :--------------- | :---------------- | :-------------------------- |
| Nick Craig-Wood | @ncw | overall project health |
| Stefan Breunig | @breunigs | |
| Ishuah Kariuki | @ishuah | |
| Remus Bunduc | @remusb | cache backend |
| Fabian Möller | @B4dM4n | |
| Alex Chen | @Cnly | onedrive backend |
| Sandeep Ummadi | @sandeepkru | azureblob backend |
| Name | GitHub ID | Specific Responsibilities |
| :--------------- | :---------------- | :------------------------------------- |
| Nick Craig-Wood | @ncw | overall project health |
| Stefan Breunig | @breunigs | |
| Ishuah Kariuki | @ishuah | |
| Remus Bunduc | @remusb | cache backend |
| Fabian Möller | @B4dM4n | |
| Alex Chen | @Cnly | onedrive backend |
| Sandeep Ummadi | @sandeepkru | azureblob backend |
| Sebastian Bünger | @buengese | jottacloud, yandex & compress backends |
| Ivan Andreev | @ivandeex | chunker & mailru backends |
| Max Sum | @Max-Sum | union backend |
| Fred | @creativeprojects | seafile backend |
| Caleb Case | @calebcase | storj backend |
| wiserain | @wiserain | pikpak backend |
| albertony | @albertony | |
| Chun-Hung Tseng | @henrybear327 | Proton Drive Backend |
| Hideo Aoyama | @boukendesho | snap packaging |
| nielash | @nielash | bisync |
| Dan McArdle | @dmcardle | gitannex |
| Sam Harrison | @childish-sambino | filescom |
| Ivan Andreev | @ivandeex | chunker & mailru backends |
| Max Sum | @Max-Sum | union backend |
| Fred | @creativeprojects | seafile backend |
| Caleb Case | @calebcase | storj backend |
| wiserain | @wiserain | pikpak backend |
| albertony | @albertony | |
| Chun-Hung Tseng | @henrybear327 | Proton Drive Backend |
| Hideo Aoyama | @boukendesho | snap packaging |
| nielash | @nielash | bisync |
| Dan McArdle | @dmcardle | gitannex |
| Sam Harrison | @childish-sambino | filescom |
## This is a work in progress draft
+1
View File
@@ -146,6 +146,7 @@ MANUAL.txt: MANUAL.md
commanddocs: rclone
go generate ./lib/transform
go generate ./cmd/bisync
-@rmdir -p '$$HOME/.config/rclone'
XDG_CACHE_HOME="" XDG_CONFIG_HOME="" HOME="\$$HOME" USER="\$$USER" rclone gendocs --config=/notfound docs/content/
@[ ! -e '$$HOME' ] || (echo 'Error: created unwanted directory named $$HOME' && exit 1)
+1
View File
@@ -123,6 +123,7 @@ directories to and from different cloud storage providers.
- Synology C2 Object Storage [:page_facing_up:](https://rclone.org/s3/#synology-c2)
- Tencent Cloud Object Storage (COS) [:page_facing_up:](https://rclone.org/s3/#tencent-cos)
- Uloz.to [:page_facing_up:](https://rclone.org/ulozto/)
- US3 Object Storage [:page_facing_up:](https://rclone.org/s3/#us3)
- Wasabi [:page_facing_up:](https://rclone.org/s3/#wasabi)
- WebDAV [:page_facing_up:](https://rclone.org/webdav/)
- Yandex Disk [:page_facing_up:](https://rclone.org/yandex/)
+2 -1
View File
@@ -32,10 +32,11 @@ This file describes how to make the various kinds of releases
- make sign_upload
- make check_sign
- make upload
- make upload_test_website # check sponsors are correct/unchanged
- make upload_website
- make upload_github
- make startdev # make startstable for stable branch
- \# announce with forum post, twitter post, patreon post
- \# announce with forum post, twitter post, and post to annoucements list
## Update dependencies
+12 -5
View File
@@ -118,12 +118,19 @@ func (c *Client) Authenticate(ctx context.Context) error {
fs.Debugf("icloud", "Authenticating as %s\n", c.appleID)
err := c.Session.SignIn(ctx, c.appleID, c.password)
if err != nil {
return err
}
if err == nil {
err = c.Session.AuthWithToken(ctx)
if err == nil && c.sessionSaveCallback != nil {
c.sessionSaveCallback(c.Session)
}
// If 2FA is required, don't try AuthWithToken yet — the caller
// must complete 2FA first, then call AuthWithToken.
if c.Session.Requires2FA() {
return nil
}
err = c.Session.AuthWithToken(ctx)
if err == nil && c.sessionSaveCallback != nil {
c.sessionSaveCallback(c.Session)
}
return err
}
+321 -44
View File
@@ -2,6 +2,7 @@ package api
import (
"context"
"encoding/base64"
"fmt"
"maps"
"net/http"
@@ -9,11 +10,15 @@ import (
"slices"
"strings"
"github.com/google/uuid"
"github.com/oracle/oci-go-sdk/v65/common"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/lib/rest"
)
const iCloudUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3.1 Safari/605.1.15"
// Session represents an iCloud session
type Session struct {
SessionToken string `json:"session_token"`
@@ -22,10 +27,22 @@ type Session struct {
AccountCountry string `json:"account_country"`
TrustToken string `json:"trust_token"`
ClientID string `json:"client_id"`
AuthAttributes string `json:"auth_attributes"`
FrameID string `json:"frame_id"`
Cookies []*http.Cookie `json:"cookies"`
AccountInfo AccountInfo `json:"account_info"`
srv *rest.Client `json:"-"`
srv *rest.Client `json:"-"`
needs2FA bool `json:"-"` // set when SRP signin returns 409
}
// srpInitResponse is the server response from /auth/signin/init
type srpInitResponse struct {
Iteration int `json:"iteration"`
Salt string `json:"salt"`
Protocol string `json:"protocol"`
B string `json:"b"`
C string `json:"c"`
}
// String returns the session as a string
@@ -57,24 +74,222 @@ func (s *Session) Request(ctx context.Context, opts rest.Opts, request any, resp
if val := resp.Header.Get("scnt"); val != "" {
s.Scnt = val
}
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
s.AuthAttributes = val
}
return resp, nil
}
// Requires2FA returns true if the session requires 2FA
func (s *Session) Requires2FA() bool {
return s.AccountInfo.DsInfo.HsaVersion == 2 && s.AccountInfo.HsaChallengeRequired
if s.needs2FA {
return true
}
return s.AccountInfo.DsInfo != nil && s.AccountInfo.DsInfo.HsaVersion == 2 && s.AccountInfo.HsaChallengeRequired
}
// SignIn signs in the session
// SignIn performs SRP-based authentication against Apple's idmsa endpoint.
func (s *Session) SignIn(ctx context.Context, appleID, password string) error {
// Apple's SRP implementation expects a lowercase account name.
// The old plaintext flow didn't need this because the server normalized
// it, but SRP uses the username in client-side proof computation (M1).
appleID = strings.ToLower(appleID)
// Step 1: Initialize the auth session
if err := s.authStart(ctx); err != nil {
return fmt.Errorf("authStart: %w", err)
}
// Step 2: Federate (submit account name)
if err := s.authFederate(ctx, appleID); err != nil {
return fmt.Errorf("authFederate: %w", err)
}
// Step 3: SRP init — send client public value A, get salt + B
client := newSRPClient()
aBase64 := base64.StdEncoding.EncodeToString(client.getABytes())
initResp, err := s.authSRPInit(ctx, aBase64, appleID)
if err != nil {
return fmt.Errorf("authSRPInit: %w", err)
}
// Decode server values
serverB, err := base64.StdEncoding.DecodeString(initResp.B)
if err != nil {
return fmt.Errorf("decode B: %w", err)
}
salt, err := base64.StdEncoding.DecodeString(initResp.Salt)
if err != nil {
return fmt.Errorf("decode salt: %w", err)
}
// Step 4: Derive password key and process the SRP challenge
derivedKey, err := derivePassword(password, salt, initResp.Iteration, initResp.Protocol)
if err != nil {
return fmt.Errorf("derivePassword: %w", err)
}
client.processChallenge([]byte(appleID), derivedKey, salt, serverB)
// Step 5: Complete — send M1, M2 proofs
m1Base64 := base64.StdEncoding.EncodeToString(client.M1)
m2Base64 := base64.StdEncoding.EncodeToString(client.M2)
if err := s.authSRPComplete(ctx, appleID, m1Base64, m2Base64, initResp.C); err != nil {
return fmt.Errorf("authSRPComplete: %w", err)
}
return nil
}
// authStart initializes the SRP auth session by hitting the authorize/signin endpoint.
func (s *Session) authStart(ctx context.Context) error {
if s.FrameID == "" {
s.FrameID = strings.ToLower(uuid.New().String())
}
frameTag := "auth-" + s.FrameID
params := url.Values{}
params.Set("frame_id", frameTag)
params.Set("language", "en_US")
params.Set("skVersion", "7")
params.Set("iframeId", frameTag)
params.Set("client_id", s.ClientID)
params.Set("redirect_uri", "https://www.icloud.com")
params.Set("response_type", "code")
params.Set("response_mode", "web_message")
params.Set("state", frameTag)
params.Set("authVersion", "latest")
opts := rest.Opts{
Method: "GET",
Path: "/authorize/signin",
Parameters: params,
ExtraHeaders: map[string]string{
"Accept": "*/*",
"User-Agent": iCloudUserAgent,
},
RootURL: authEndpoint,
NoResponse: true,
}
resp, err := s.srv.Call(ctx, &opts)
if err != nil {
return err
}
_ = resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("authStart: unexpected status %s", resp.Status)
}
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
s.AuthAttributes = val
}
if val := resp.Header.Get("scnt"); val != "" {
s.Scnt = val
}
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
s.SessionID = val
}
return nil
}
// authFederate submits the account name to Apple's federate endpoint.
func (s *Session) authFederate(ctx context.Context, accountName string) error {
values := map[string]any{
"accountName": accountName,
"rememberMe": true,
}
body, err := IntoReader(values)
if err != nil {
return err
}
opts := rest.Opts{
Method: "POST",
Path: "/federate",
Parameters: url.Values{"isRememberMeEnabled": {"true"}},
ExtraHeaders: s.getSRPAuthHeaders(),
RootURL: authEndpoint,
Body: body,
NoResponse: true,
}
resp, err := s.srv.Call(ctx, &opts)
if err != nil {
return err
}
_ = resp.Body.Close()
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
s.AuthAttributes = val
}
if val := resp.Header.Get("scnt"); val != "" {
s.Scnt = val
}
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
s.SessionID = val
}
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("authFederate: unexpected status %s", resp.Status)
}
return nil
}
// authSRPInit sends the client's public value A to the server and retrieves
// the salt, server public value B, iteration count, protocol, and challenge.
func (s *Session) authSRPInit(ctx context.Context, aBase64, accountName string) (*srpInitResponse, error) {
values := map[string]any{
"a": aBase64,
"accountName": accountName,
"protocols": []string{"s2k", "s2k_fo"},
}
body, err := IntoReader(values)
if err != nil {
return nil, err
}
opts := rest.Opts{
Method: "POST",
Path: "/signin/init",
ExtraHeaders: s.getSRPAuthHeaders(),
RootURL: authEndpoint,
Body: body,
}
var initResp srpInitResponse
resp, err := s.srv.CallJSON(ctx, &opts, nil, &initResp)
if err != nil {
return nil, err
}
if val := resp.Header.Get("scnt"); val != "" {
s.Scnt = val
}
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
s.SessionID = val
}
return &initResp, nil
}
// authSRPComplete sends the SRP proofs M1 and M2 to complete authentication.
// Returns nil on success (200 or 409/2FA needed).
func (s *Session) authSRPComplete(ctx context.Context, accountName, m1Base64, m2Base64, c string) error {
trustTokens := []string{}
if s.TrustToken != "" {
trustTokens = []string{s.TrustToken}
}
values := map[string]any{
"accountName": appleID,
"password": password,
"accountName": accountName,
"m1": m1Base64,
"m2": m2Base64,
"c": c,
"rememberMe": true,
"trustTokens": trustTokens,
}
@@ -82,21 +297,100 @@ func (s *Session) SignIn(ctx context.Context, appleID, password string) error {
if err != nil {
return err
}
opts := rest.Opts{
Method: "POST",
Path: "/signin",
Parameters: url.Values{},
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
Path: "/signin/complete",
Parameters: url.Values{"isRememberMeEnabled": {"true"}},
ExtraHeaders: s.getSRPAuthHeaders(),
RootURL: authEndpoint,
IgnoreStatus: true, // need to handle 409 for hsa2
IgnoreStatus: true,
NoResponse: true,
Body: body,
}
opts.Parameters.Set("isRememberMeEnabled", "true")
_, err = s.Request(ctx, opts, nil, nil)
return err
resp, err := s.srv.Call(ctx, &opts)
if err != nil {
return err
}
_ = resp.Body.Close()
// Extract updated headers
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
s.AuthAttributes = val
}
if val := resp.Header.Get("X-Apple-Session-Token"); val != "" {
s.SessionToken = val
}
if val := resp.Header.Get("scnt"); val != "" {
s.Scnt = val
}
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
s.SessionID = val
}
if val := resp.Header.Get("X-Apple-ID-Account-Country"); val != "" {
s.AccountCountry = val
}
if val := resp.Header.Get("X-Apple-TwoSV-Trust-Token"); val != "" {
s.TrustToken = val
}
switch resp.StatusCode {
case http.StatusOK:
fs.Debugf("icloud", "SRP sign in successful")
return nil
case http.StatusConflict:
// 409 = 2FA required, this is expected
fs.Debugf("icloud", "SRP sign in requires 2FA")
s.needs2FA = true
return nil
case http.StatusForbidden:
return fmt.Errorf("sign in failed: incorrect username or password")
default:
return fmt.Errorf("sign in failed: %s", resp.Status)
}
}
// getAuthOrigin returns the origin URL for auth requests.
// Supports both global (idmsa.apple.com) and China (idmsa.apple.com.cn) endpoints.
func getAuthOrigin() string {
return strings.TrimSuffix(authEndpoint, "/appleauth/auth")
}
// getSRPAuthHeaders returns headers needed for SRP auth requests.
func (s *Session) getSRPAuthHeaders() map[string]string {
frameTag := "auth-" + s.FrameID
authOrigin := getAuthOrigin()
headers := map[string]string{
"Accept": "application/json",
"Content-Type": "application/json",
"User-Agent": iCloudUserAgent,
"Origin": authOrigin,
"Referer": authOrigin + "/",
"X-Apple-Widget-Key": s.ClientID,
"X-Apple-OAuth-Client-Id": s.ClientID,
"X-Apple-OAuth-Client-Type": "firstPartyAuth",
"X-Apple-OAuth-Redirect-URI": "https://www.icloud.com",
"X-Apple-OAuth-Require-Grant-Code": "true",
"X-Apple-OAuth-Response-Mode": "web_message",
"X-Apple-OAuth-Response-Type": "code",
"X-Apple-OAuth-State": frameTag,
"X-Apple-Frame-Id": frameTag,
"X-Requested-With": "XMLHttpRequest",
"X-Apple-Mandate-Security-Upgrade": "0",
"X-Apple-I-Require-UE": "true",
"X-Apple-I-FD-Client-Info": `{"U":"` + iCloudUserAgent + `","L":"en-US","Z":"GMT-05:00","V":"1.1","F":""}`,
}
if s.AuthAttributes != "" {
headers["X-Apple-Auth-Attributes"] = s.AuthAttributes
}
if s.Scnt != "" {
headers["scnt"] = s.Scnt
}
if s.SessionID != "" {
headers["X-Apple-ID-Session-Id"] = s.SessionID
}
return headers
}
// AuthWithToken authenticates the session
@@ -135,14 +429,10 @@ func (s *Session) Validate2FACode(ctx context.Context, code string) error {
return err
}
headers := s.GetAuthHeaders(map[string]string{})
headers["scnt"] = s.Scnt
headers["X-Apple-ID-Session-Id"] = s.SessionID
opts := rest.Opts{
Method: "POST",
Path: "/verify/trusteddevice/securitycode",
ExtraHeaders: headers,
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
RootURL: authEndpoint,
Body: body,
NoResponse: true,
@@ -162,14 +452,10 @@ func (s *Session) Validate2FACode(ctx context.Context, code string) error {
// TrustSession trusts the session
func (s *Session) TrustSession(ctx context.Context) error {
headers := s.GetAuthHeaders(map[string]string{})
headers["scnt"] = s.Scnt
headers["X-Apple-ID-Session-Id"] = s.SessionID
opts := rest.Opts{
Method: "GET",
Path: "/2sv/trust",
ExtraHeaders: headers,
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
RootURL: authEndpoint,
NoResponse: true,
ContentLength: common.Int64(0),
@@ -201,25 +487,9 @@ func (s *Session) ValidateSession(ctx context.Context) error {
}
// GetAuthHeaders returns the authentication headers for the session.
//
// It takes an `overwrite` map[string]string parameter which allows
// overwriting the default headers. It returns a map[string]string.
// Used for 2FA validation and trust requests to idmsa.apple.com.
func (s *Session) GetAuthHeaders(overwrite map[string]string) map[string]string {
headers := map[string]string{
"Accept": "application/json",
"Content-Type": "application/json",
"X-Apple-OAuth-Client-Id": s.ClientID,
"X-Apple-OAuth-Client-Type": "firstPartyAuth",
"X-Apple-OAuth-Redirect-URI": "https://www.icloud.com",
"X-Apple-OAuth-Require-Grant-Code": "true",
"X-Apple-OAuth-Response-Mode": "web_message",
"X-Apple-OAuth-Response-Type": "code",
"X-Apple-OAuth-State": s.ClientID,
"X-Apple-Widget-Key": s.ClientID,
"Origin": homeEndpoint,
"Referer": fmt.Sprintf("%s/", homeEndpoint),
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0",
}
headers := s.getSRPAuthHeaders()
maps.Copy(headers, overwrite)
return headers
}
@@ -248,7 +518,7 @@ func GetCommonHeaders(overwrite map[string]string) map[string]string {
"Content-Type": "application/json",
"Origin": baseEndpoint,
"Referer": fmt.Sprintf("%s/", baseEndpoint),
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0",
"User-Agent": iCloudUserAgent,
}
maps.Copy(headers, overwrite)
return headers
@@ -281,9 +551,16 @@ func GetCookiesForDomain(url *url.URL, cookies []*http.Cookie) ([]*http.Cookie,
// NewSession creates a new Session instance with default values.
func NewSession() *Session {
session := &Session{}
session.srv = rest.NewClient(fshttp.NewClient(context.Background())).SetRoot(baseEndpoint)
//session.ClientID = "auth-" + uuid.New().String()
session := &Session{
FrameID: strings.ToLower(uuid.New().String()),
}
httpClient := fshttp.NewClient(context.Background())
if tr, ok := httpClient.Transport.(*fshttp.Transport); ok {
tr.SetRequestFilter(func(req *http.Request) {
req.Header.Set("User-Agent", iCloudUserAgent)
})
}
session.srv = rest.NewClient(httpClient).SetRoot(baseEndpoint)
return session
}
+229
View File
@@ -0,0 +1,229 @@
package api
import (
"crypto/pbkdf2"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"fmt"
"hash"
"math/big"
)
// RFC 5054 2048-bit group parameters for SRP-6a (g=2, SHA-256).
var (
srpN = mustParseBigHex(
"AC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC3192943DB56050" +
"A37329CBB4A099ED8193E0757767A13DD52312AB4B03310DCD7F48A9DA04FD50" +
"E8083969EDB767B0CF6095179A163AB3661A05FBD5FAAAE82918A9962F0B93B8" +
"55F97993EC975EEAA80D740ADBF4FF747359D041D5C33EA71D281E446B14773B" +
"CA97B43A23FB801676BD207A436C6481F1D2B9078717461A5B9D32E688F87748" +
"544523B524B0D57D5EA77A2775D2ECFA032CFBDBF52FB37861602790" +
"04E57AE6AF874E7303CE53299CCC041C7BC308D82A5698F3A8D0C38271AE35F8" +
"E9DBFBB694B5C803D89F7AE435DE236D525F54759B65E372FCD68EF20FA7111F" +
"9E4AFF73")
srpG = big.NewInt(2)
srpNLenBytes = 2048 / 8
srpHashFunc = sha256.New
)
func mustParseBigHex(s string) *big.Int {
b, ok := new(big.Int).SetString(s, 16)
if !ok {
panic("srp: bad hex constant for N")
}
return b
}
// srpClient implements the client side of SRP-6a for Apple's iCloud auth.
type srpClient struct {
a *big.Int // client secret
A *big.Int // client public value
k *big.Int // multiplier
M1 []byte
M2 []byte
K []byte
}
// newSRPClient generates a random 32-byte secret and computes the public value A.
func newSRPClient() *srpClient {
secret := make([]byte, 32)
if _, err := rand.Read(secret); err != nil {
panic(fmt.Sprintf("srp: rand.Read failed: %v", err))
}
a := new(big.Int).SetBytes(secret)
A := new(big.Int).Exp(srpG, a, srpN)
k := getMultiplier()
return &srpClient{
a: a,
A: A,
k: k,
}
}
// getABytes returns the padded public value A.
func (c *srpClient) getABytes() []byte {
return padToN(c.A)
}
// processChallenge computes the session key and proof values from the server challenge.
// username is the Apple ID, derivedKey is the output of derivePassword, salt and B come
// from the server's init response (raw bytes, already base64-decoded).
func (c *srpClient) processChallenge(username, derivedKey, salt, serverB []byte) {
B := new(big.Int).SetBytes(serverB)
// Validate B
if B.Cmp(big.NewInt(0)) <= 0 || B.Cmp(srpN) >= 0 {
panic("srp: invalid server-supplied B, must be 1..N-1")
}
x := calculateX(salt, derivedKey)
u := calculateU(c.A, B)
S := calculateS(c.k, x, c.a, B, u)
c.K = calculateK(S)
aBytes := padToN(c.A)
bBytes := padToN(B)
c.M1 = calculateM1(username, salt, aBytes, bBytes, c.K)
c.M2 = calculateM2(aBytes, c.M1, c.K)
}
// derivePassword performs Apple's password key derivation.
// For "s2k": PBKDF2(SHA256(password), salt, iterations, 32, SHA256)
// For "s2k_fo": PBKDF2(hex(SHA256(password)), salt, iterations, 32, SHA256)
func derivePassword(password string, salt []byte, iterations int, protocol string) ([]byte, error) {
passHash := sha256.Sum256([]byte(password))
var passInput string
switch protocol {
case "s2k_fo":
passInput = hex.EncodeToString(passHash[:])
default: // "s2k"
passInput = string(passHash[:])
}
return pbkdf2.Key(sha256.New, passInput, salt, iterations, 32)
}
// padToN pads a big.Int to the SRP group size (256 bytes for 2048-bit).
func padToN(n *big.Int) []byte {
b := n.Bytes()
if len(b) >= srpNLenBytes {
return b
}
padded := make([]byte, srpNLenBytes)
copy(padded[srpNLenBytes-len(b):], b)
return padded
}
func srpHash(data []byte) []byte {
h := srpHashFunc()
h.Write(data)
return h.Sum(nil)
}
func hashToInt(h hash.Hash) *big.Int {
return new(big.Int).SetBytes(h.Sum(nil))
}
// getMultiplier computes k = H(N | pad(g))
func getMultiplier() *big.Int {
h := srpHashFunc()
nBytes := srpN.Bytes()
gBytes := srpG.Bytes()
// pad g to same length as N
for len(gBytes) < len(nBytes) {
gBytes = append([]byte{0}, gBytes...)
}
h.Write(nBytes)
h.Write(gBytes)
return hashToInt(h)
}
// calculateX computes x = H(salt | H(":" | password))
// Apple's variant: NoUserNameInX — username is omitted from the inner hash.
// The "password" here is actually the derived key from derivePassword.
func calculateX(salt, derivedKey []byte) *big.Int {
h := srpHashFunc()
// Apple omits username (NoUserNameInX), only writes ":" + derivedKey
h.Write([]byte(":"))
h.Write(derivedKey)
digest := h.Sum(nil)
h2 := srpHashFunc()
h2.Write(salt)
h2.Write(digest)
return new(big.Int).SetBytes(h2.Sum(nil))
}
// calculateU computes u = H(pad(A) | pad(B))
func calculateU(A, B *big.Int) *big.Int {
h := srpHashFunc()
h.Write(padToN(A))
h.Write(padToN(B))
return hashToInt(h)
}
// calculateS computes the client-side shared secret:
// S = (B - k * g^x) ^ (a + u*x) mod N
func calculateS(k, x, a, B, u *big.Int) []byte {
// g^x mod N
gx := new(big.Int).Exp(srpG, x, srpN)
// k * g^x
kgx := new(big.Int).Mul(k, gx)
// B - k*g^x
diff := new(big.Int).Sub(B, kgx)
// u*x
ux := new(big.Int).Mul(u, x)
// a + u*x
exp := new(big.Int).Add(a, ux)
// (B - k*g^x) ^ (a + u*x) mod N
S := new(big.Int).Exp(diff, exp, srpN)
// Ensure positive
S.Mod(S, srpN)
return padToN(S)
}
// calculateK computes K = H(S)
func calculateK(S []byte) []byte {
return srpHash(S)
}
// calculateM1 computes the client proof (Apple/iCloud variant):
// M1 = H(H(g) XOR H(N) | H(username) | salt | A | B | K)
func calculateM1(username, salt, A, B, K []byte) []byte {
hg := srpHash(padToN(srpG))
hn := srpHash(srpN.Bytes())
hi := srpHash(username)
hxor := make([]byte, len(hg))
for i := range hg {
hxor[i] = hg[i] ^ hn[i]
}
h := srpHashFunc()
h.Write(hxor)
h.Write(hi)
h.Write(salt)
h.Write(A)
h.Write(B)
h.Write(K)
return h.Sum(nil)
}
// calculateM2 computes the server proof: M2 = H(A | M1 | K)
func calculateM2(A, M1, K []byte) []byte {
h := srpHashFunc()
h.Write(A)
h.Write(M1)
h.Write(K)
return h.Sum(nil)
}
+379
View File
@@ -0,0 +1,379 @@
package api
import (
"crypto/sha256"
"encoding/hex"
"math/big"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestPadToN(t *testing.T) {
t.Run("pads small value to 256 bytes", func(t *testing.T) {
result := padToN(big.NewInt(1))
assert.Equal(t, srpNLenBytes, len(result))
assert.Equal(t, byte(0), result[0])
assert.Equal(t, byte(1), result[srpNLenBytes-1])
})
t.Run("pads zero to 256 bytes", func(t *testing.T) {
result := padToN(big.NewInt(0))
assert.Equal(t, srpNLenBytes, len(result))
for _, b := range result {
assert.Equal(t, byte(0), b)
}
})
t.Run("preserves value that fills N", func(t *testing.T) {
// Use N itself — already 256 bytes
result := padToN(srpN)
assert.Equal(t, srpNLenBytes, len(result))
assert.Equal(t, srpN.Bytes(), result)
})
t.Run("round-trips through big.Int", func(t *testing.T) {
original := big.NewInt(123456789)
padded := padToN(original)
recovered := new(big.Int).SetBytes(padded)
assert.Equal(t, 0, original.Cmp(recovered))
})
}
func TestGetMultiplier(t *testing.T) {
k := getMultiplier()
// k must be non-zero and less than N
assert.True(t, k.Sign() > 0, "multiplier must be positive")
assert.True(t, k.Cmp(srpN) < 0, "multiplier must be less than N")
// k is deterministic
k2 := getMultiplier()
assert.Equal(t, 0, k.Cmp(k2), "multiplier must be deterministic")
// k = H(N | pad(g)) — verify by manual computation
h := srpHashFunc()
nBytes := srpN.Bytes()
gBytes := srpG.Bytes()
for len(gBytes) < len(nBytes) {
gBytes = append([]byte{0}, gBytes...)
}
h.Write(nBytes)
h.Write(gBytes)
expected := new(big.Int).SetBytes(h.Sum(nil))
assert.Equal(t, 0, expected.Cmp(k), "multiplier must equal H(N|pad(g))")
}
func TestDerivePassword(t *testing.T) {
salt := []byte("test_salt_value!")
password := "test_password"
t.Run("s2k produces 32-byte key", func(t *testing.T) {
key, err := derivePassword(password, salt, 1000, "s2k")
require.NoError(t, err)
assert.Equal(t, 32, len(key))
})
t.Run("s2k_fo produces 32-byte key", func(t *testing.T) {
key, err := derivePassword(password, salt, 1000, "s2k_fo")
require.NoError(t, err)
assert.Equal(t, 32, len(key))
})
t.Run("s2k and s2k_fo produce different keys", func(t *testing.T) {
key1, err := derivePassword(password, salt, 1000, "s2k")
require.NoError(t, err)
key2, err := derivePassword(password, salt, 1000, "s2k_fo")
require.NoError(t, err)
assert.NotEqual(t, key1, key2, "s2k and s2k_fo must produce different keys")
})
t.Run("deterministic output", func(t *testing.T) {
key1, err := derivePassword(password, salt, 1000, "s2k")
require.NoError(t, err)
key2, err := derivePassword(password, salt, 1000, "s2k")
require.NoError(t, err)
assert.Equal(t, key1, key2)
})
t.Run("different passwords produce different keys", func(t *testing.T) {
key1, err := derivePassword("password1", salt, 1000, "s2k")
require.NoError(t, err)
key2, err := derivePassword("password2", salt, 1000, "s2k")
require.NoError(t, err)
assert.NotEqual(t, key1, key2)
})
t.Run("s2k uses raw SHA256 bytes", func(t *testing.T) {
// Verify that s2k uses SHA256(password) as the PBKDF2 input.
// Two passwords with the same SHA256 hash would produce the same key,
// while two passwords with different SHA256 hashes must produce different keys.
key1, err := derivePassword("hello", salt, 10, "s2k")
require.NoError(t, err)
key2, err := derivePassword("world", salt, 10, "s2k")
require.NoError(t, err)
assert.NotEqual(t, key1, key2)
})
t.Run("s2k_fo uses hex-encoded SHA256", func(t *testing.T) {
// For s2k_fo, the PBKDF2 input is hex(SHA256(password)), which is
// a 64-character ASCII string, not raw 32-byte hash.
key, err := derivePassword("hello", salt, 10, "s2k_fo")
require.NoError(t, err)
assert.Equal(t, 32, len(key))
})
}
func TestCalculateX(t *testing.T) {
salt := []byte("some_salt")
derivedKey := []byte("some_derived_key_value_32_bytes!")
x := calculateX(salt, derivedKey)
// x must be positive and less than the hash output space
assert.True(t, x.Sign() > 0, "x must be positive")
// x is deterministic
x2 := calculateX(salt, derivedKey)
assert.Equal(t, 0, x.Cmp(x2), "x must be deterministic")
// Verify manually: x = H(salt | H(":" | derivedKey))
h1 := srpHashFunc()
h1.Write([]byte(":"))
h1.Write(derivedKey)
inner := h1.Sum(nil)
h2 := srpHashFunc()
h2.Write(salt)
h2.Write(inner)
expected := new(big.Int).SetBytes(h2.Sum(nil))
assert.Equal(t, 0, expected.Cmp(x), "x must equal H(salt | H(\":\" | derivedKey))")
}
func TestCalculateU(t *testing.T) {
A := big.NewInt(12345)
B := big.NewInt(67890)
u := calculateU(A, B)
assert.True(t, u.Sign() > 0, "u must be positive")
// u is deterministic
u2 := calculateU(A, B)
assert.Equal(t, 0, u.Cmp(u2))
// u changes when inputs change
u3 := calculateU(A, big.NewInt(99999))
assert.NotEqual(t, 0, u.Cmp(u3), "u must change when B changes")
// Verify: u = H(pad(A) | pad(B))
h := srpHashFunc()
h.Write(padToN(A))
h.Write(padToN(B))
expected := new(big.Int).SetBytes(h.Sum(nil))
assert.Equal(t, 0, expected.Cmp(u))
}
func TestCalculateK(t *testing.T) {
S := []byte("some_shared_secret_value_32bytes")
K := calculateK(S)
// K is a SHA-256 hash, so 32 bytes
assert.Equal(t, sha256.Size, len(K))
// K is deterministic
K2 := calculateK(S)
assert.Equal(t, K, K2)
// Verify: K = H(S)
expected := srpHash(S)
assert.Equal(t, expected, K)
}
func TestCalculateM1(t *testing.T) {
username := []byte("test@example.com")
salt := []byte("test_salt")
A := make([]byte, srpNLenBytes)
B := make([]byte, srpNLenBytes)
K := make([]byte, 32)
// Fill with non-zero values
A[srpNLenBytes-1] = 0x42
B[srpNLenBytes-1] = 0x43
K[0] = 0x44
m1 := calculateM1(username, salt, A, B, K)
// M1 is a SHA-256 hash
assert.Equal(t, sha256.Size, len(m1))
// M1 is deterministic
m1b := calculateM1(username, salt, A, B, K)
assert.Equal(t, m1, m1b)
// M1 changes when username changes
m1c := calculateM1([]byte("other@example.com"), salt, A, B, K)
assert.NotEqual(t, m1, m1c)
// Verify: M1 = H(H(g) XOR H(N) | H(username) | salt | A | B | K)
hg := srpHash(padToN(srpG))
hn := srpHash(srpN.Bytes())
hi := srpHash(username)
hxor := make([]byte, len(hg))
for i := range hg {
hxor[i] = hg[i] ^ hn[i]
}
h := srpHashFunc()
h.Write(hxor)
h.Write(hi)
h.Write(salt)
h.Write(A)
h.Write(B)
h.Write(K)
expected := h.Sum(nil)
assert.Equal(t, expected, m1)
}
func TestCalculateM2(t *testing.T) {
A := make([]byte, srpNLenBytes)
M1 := make([]byte, 32)
K := make([]byte, 32)
A[srpNLenBytes-1] = 0x01
M1[0] = 0x02
K[0] = 0x03
m2 := calculateM2(A, M1, K)
assert.Equal(t, sha256.Size, len(m2))
// Verify: M2 = H(A | M1 | K)
h := srpHashFunc()
h.Write(A)
h.Write(M1)
h.Write(K)
assert.Equal(t, h.Sum(nil), m2)
}
func TestNewSRPClient(t *testing.T) {
client := newSRPClient()
// A must be non-zero and in range [1, N-1]
assert.True(t, client.A.Sign() > 0, "A must be positive")
assert.True(t, client.A.Cmp(srpN) < 0, "A must be less than N")
// A = g^a mod N
expectedA := new(big.Int).Exp(srpG, client.a, srpN)
assert.Equal(t, 0, expectedA.Cmp(client.A), "A must equal g^a mod N")
// getABytes returns padded A
aBytes := client.getABytes()
assert.Equal(t, srpNLenBytes, len(aBytes))
recoveredA := new(big.Int).SetBytes(aBytes)
assert.Equal(t, 0, client.A.Cmp(recoveredA))
// Two clients should have different secrets (probabilistic, but 2^256 collision chance)
client2 := newSRPClient()
assert.NotEqual(t, 0, client.a.Cmp(client2.a), "two clients must have different secrets")
}
func TestProcessChallenge(t *testing.T) {
// Use a fixed secret for deterministic testing
fixedSecret := make([]byte, 32)
fixedSecret[31] = 0x42 // small non-zero value
a := new(big.Int).SetBytes(fixedSecret)
A := new(big.Int).Exp(srpG, a, srpN)
k := getMultiplier()
client := &srpClient{
a: a,
A: A,
k: k,
}
username := []byte("testuser@apple.com")
derivedKey := make([]byte, 32)
for i := range derivedKey {
derivedKey[i] = byte(i)
}
salt := []byte("fixed_salt_for_test!")
// Compute a valid B: B = (k*v + g^b) mod N where v = g^x mod N
x := calculateX(salt, derivedKey)
v := new(big.Int).Exp(srpG, x, srpN)
bSecret := make([]byte, 32)
bSecret[31] = 0x99
b := new(big.Int).SetBytes(bSecret)
// B = (k*v + g^b) mod N
kv := new(big.Int).Mul(k, v)
gb := new(big.Int).Exp(srpG, b, srpN)
B := new(big.Int).Add(kv, gb)
B.Mod(B, srpN)
serverB := padToN(B)
client.processChallenge(username, derivedKey, salt, serverB)
// M1, M2, K must be populated
assert.NotNil(t, client.M1)
assert.NotNil(t, client.M2)
assert.NotNil(t, client.K)
assert.Equal(t, sha256.Size, len(client.M1))
assert.Equal(t, sha256.Size, len(client.M2))
assert.Equal(t, sha256.Size, len(client.K))
// M1 and M2 must be different
assert.NotEqual(t, client.M1, client.M2)
// Results must be deterministic — run again with same inputs
client2 := &srpClient{
a: new(big.Int).Set(a),
A: new(big.Int).Set(A),
k: new(big.Int).Set(k),
}
client2.processChallenge(username, derivedKey, salt, serverB)
assert.Equal(t, client.M1, client2.M1, "M1 must be deterministic")
assert.Equal(t, client.M2, client2.M2, "M2 must be deterministic")
assert.Equal(t, client.K, client2.K, "K must be deterministic")
// Verify the server could also compute S and get the same K.
// Server computes: S_server = (A * v^u)^b mod N
u := calculateU(A, B)
vu := new(big.Int).Exp(v, u, srpN)
avu := new(big.Int).Mul(A, vu)
avu.Mod(avu, srpN)
sServer := new(big.Int).Exp(avu, b, srpN)
sServer.Mod(sServer, srpN)
kServer := calculateK(padToN(sServer))
assert.Equal(t, client.K, kServer, "client and server must derive the same session key")
}
func TestSRPGroupParameters(t *testing.T) {
// Verify the 2048-bit prime starts with known hex prefix
nHex := hex.EncodeToString(srpN.Bytes())
assert.True(t, len(nHex) > 0)
assert.Equal(t, "ac6bdb41", nHex[:8], "N must start with known prefix")
// g must be 2
assert.Equal(t, int64(2), srpG.Int64())
// N byte length must match expected
assert.Equal(t, 256, srpNLenBytes)
assert.Equal(t, 256, len(srpN.Bytes()))
}
func TestSRPHash(t *testing.T) {
data := []byte("hello world")
hash := srpHash(data)
// Must be SHA-256 output
assert.Equal(t, sha256.Size, len(hash))
// Must equal standard sha256
expected := sha256.Sum256(data)
assert.Equal(t, expected[:], hash)
}
-12
View File
@@ -1,12 +0,0 @@
//go:build !linux
package local
import (
"io"
"os"
)
func newFadviseReadCloser(o *Object, f *os.File, offset, limit int64) io.ReadCloser {
return f
}
-165
View File
@@ -1,165 +0,0 @@
//go:build linux
package local
import (
"io"
"os"
"github.com/rclone/rclone/fs"
"golang.org/x/sys/unix"
)
// fadvise provides means to automate freeing pages in kernel page cache for
// a given file descriptor as the file is sequentially processed (read or
// written).
//
// When copying a file to a remote backend all the file content is read by
// kernel and put to page cache to make future reads faster.
// This causes memory pressure visible in both memory usage and CPU consumption
// and can even cause OOM errors in applications consuming large amounts memory.
//
// In case of an upload to a remote backend, there is no benefits from caching.
//
// fadvise would orchestrate calling POSIX_FADV_DONTNEED
//
// POSIX_FADV_DONTNEED attempts to free cached pages associated
// with the specified region. This is useful, for example, while
// streaming large files. A program may periodically request the
// kernel to free cached data that has already been used, so that
// more useful cached pages are not discarded instead.
//
// Requests to discard partial pages are ignored. It is
// preferable to preserve needed data than discard unneeded data.
// If the application requires that data be considered for
// discarding, then offset and len must be page-aligned.
//
// The implementation may attempt to write back dirty pages in
// the specified region, but this is not guaranteed. Any
// unwritten dirty pages will not be freed. If the application
// wishes to ensure that dirty pages will be released, it should
// call fsync(2) or fdatasync(2) first.
type fadvise struct {
o *Object
fd int
lastPos int64
curPos int64
windowSize int64
freePagesCh chan offsetLength
doneCh chan struct{}
}
type offsetLength struct {
offset int64
length int64
}
const (
defaultAllowPages = 32
defaultWorkerQueueSize = 64
)
func newFadvise(o *Object, fd int, offset int64) *fadvise {
f := &fadvise{
o: o,
fd: fd,
lastPos: offset,
curPos: offset,
windowSize: int64(os.Getpagesize()) * defaultAllowPages,
freePagesCh: make(chan offsetLength, defaultWorkerQueueSize),
doneCh: make(chan struct{}),
}
go f.worker()
return f
}
// sequential configures readahead strategy in Linux kernel.
//
// Under Linux, POSIX_FADV_NORMAL sets the readahead window to the
// default size for the backing device; POSIX_FADV_SEQUENTIAL doubles
// this size, and POSIX_FADV_RANDOM disables file readahead entirely.
func (f *fadvise) sequential(limit int64) bool {
l := int64(0)
if limit > 0 {
l = limit
}
if err := unix.Fadvise(f.fd, f.curPos, l, unix.FADV_SEQUENTIAL); err != nil {
fs.Debugf(f.o, "fadvise sequential failed on file descriptor %d: %s", f.fd, err)
return false
}
return true
}
func (f *fadvise) next(n int) {
f.curPos += int64(n)
f.freePagesIfNeeded()
}
func (f *fadvise) freePagesIfNeeded() {
if f.curPos >= f.lastPos+f.windowSize {
f.freePages()
}
}
func (f *fadvise) freePages() {
f.freePagesCh <- offsetLength{f.lastPos, f.curPos - f.lastPos}
f.lastPos = f.curPos
}
func (f *fadvise) worker() {
for p := range f.freePagesCh {
if err := unix.Fadvise(f.fd, p.offset, p.length, unix.FADV_DONTNEED); err != nil {
fs.Debugf(f.o, "fadvise dontneed failed on file descriptor %d: %s", f.fd, err)
}
}
close(f.doneCh)
}
func (f *fadvise) wait() {
close(f.freePagesCh)
<-f.doneCh
}
type fadviseReadCloser struct {
*fadvise
inner io.ReadCloser
}
// newFadviseReadCloser wraps os.File so that reading from that file would
// remove already consumed pages from kernel page cache.
// In addition to that it instructs kernel to double the readahead window to
// make sequential reads faster.
// See also fadvise.
func newFadviseReadCloser(o *Object, f *os.File, offset, limit int64) io.ReadCloser {
r := fadviseReadCloser{
fadvise: newFadvise(o, int(f.Fd()), offset),
inner: f,
}
// If syscall failed it's likely that the subsequent syscalls to that
// file descriptor would also fail. In that case return the provided os.File
// pointer.
if !r.sequential(limit) {
r.wait()
return f
}
return r
}
func (f fadviseReadCloser) Read(p []byte) (n int, err error) {
n, err = f.inner.Read(p)
f.next(n)
return
}
func (f fadviseReadCloser) Close() error {
f.freePages()
f.wait()
return f.inner.Close()
}
+2 -2
View File
@@ -1168,7 +1168,7 @@ func (o *Object) Hash(ctx context.Context, r hash.Type) (string, error) {
var fd *os.File
fd, err = file.Open(o.path)
if fd != nil {
in = newFadviseReadCloser(o, fd, 0, 0)
in = fd
}
} else {
in, err = o.openTranslatedLink(0, -1)
@@ -1374,7 +1374,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
if err != nil {
return
}
wrappedFd := readers.NewLimitedReadCloser(newFadviseReadCloser(o, fd, offset, limit), limit)
wrappedFd := readers.NewLimitedReadCloser(fd, limit)
if offset != 0 {
// seek the object
_, err = fd.Seek(offset, io.SeekStart)
+6 -5
View File
@@ -16,11 +16,12 @@ type Authenticator interface {
}
// IbmIamSigner is a structure for signing requests using IBM IAM.
// Requeres APIKey and Resource InstanceID
// Requires APIKey and Resource InstanceID
type IbmIamSigner struct {
APIKey string
InstanceID string
Auth Authenticator
APIKey string
InstanceID string
IAMEndpoint string
Auth Authenticator
}
// SignHTTP signs requests using IBM IAM token.
@@ -29,7 +30,7 @@ func (signer *IbmIamSigner) SignHTTP(ctx context.Context, credentials aws.Creden
if signer.Auth != nil {
authenticator = signer.Auth
} else {
authenticator = &core.IamAuthenticator{ApiKey: signer.APIKey}
authenticator = &core.IamAuthenticator{ApiKey: signer.APIKey, URL: signer.IAMEndpoint}
}
token, err := authenticator.GetToken()
if err != nil {
+5 -3
View File
@@ -23,14 +23,16 @@ func TestSignHTTP(t *testing.T) {
apiKey := "mock-api-key"
instanceID := "mock-instance-id"
token := "mock-iam-token"
iamEndpoint := "https://iamendpoint.com"
mockAuth := &MockAuthenticator{
Token: token,
Error: nil,
}
signer := &IbmIamSigner{
APIKey: apiKey,
InstanceID: instanceID,
Auth: mockAuth,
APIKey: apiKey,
InstanceID: instanceID,
IAMEndpoint: iamEndpoint,
Auth: mockAuth,
}
req, err := http.NewRequest("GET", "https://example.com", nil)
if err != nil {
+1
View File
@@ -4,6 +4,7 @@ region:
eu-west-1: Europe West
endpoint:
s3.cubbit.eu: Cubbit DS3 Object Storage endpoint
s3.{tenant_name}.cubbit.eu: Multi-tenant endpoint - replace {tenant_name} with your actual tenant name. Do not select this directly
acl: {}
bucket_acl: true
quirks:
+4
View File
@@ -18,3 +18,7 @@ quirks:
# See: https://issuetracker.google.com/issues/323465186
# So make cutoff very large which it does seem to support
copy_cutoff: 9223372036854775807
# GCS S3 API doesn't fully support Object Lock:
# - Uses x-goog-bypass-governance-retention instead of x-amz-bypass-governance-retention
# - PutObjectLegalHold / GetObjectLegalHold not implemented
object_lock_supported: false
+1
View File
@@ -117,6 +117,7 @@ acl:
This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
ibm_api_key: true
ibm_resource_instance_id: true
ibm_iam_endpoint: true
bucket_acl: true
quirks:
list_version: 1
+10
View File
@@ -32,5 +32,15 @@ endpoint:
s3.us-east-va.io.cloud.ovh.us: OVHcloud Vint Hill, Virginia, USA
s3.us-west-or.io.cloud.ovh.us: OVHcloud Hillsboro, Oregon, USA
s3.rbx-archive.io.cloud.ovh.net: OVHcloud Roubaix, France (Cold Archive)
storage_class:
'': Default
EXPRESS_ONEZONE: High Performance storage class
STANDARD: Standard storage class
STANDARD_IA: Standard Infrequent Access storage class
ONEZONE_IA: Standard Infrequent Access storage class
GLACIER: Active Archive storage class
GLACIER_IR: Active Archive storage class
DEEP_ARCHIVE: Cold Archive storage class
INTELLIGENT_TIERING: Standard storage class
acl: {}
bucket_acl: true
+44
View File
@@ -0,0 +1,44 @@
name: US3
description: US3 Object Storage
endpoint:
s3-cn-bj.ufileos.com: North China (Beijing 2)
s3-cn-wlcb.ufileos.com: North China (Ulanqab)
s3-cn-sh2.ufileos.com: Shanghai
s3-cn-gd.ufileos.com: South China (Guangzhou)
s3-hk.ufileos.com: Hongkong
s3-us-ca.ufileos.com: United States (Los Angeles)
s3-sg.ufileos.com: Singapore
s3-idn-jakarta.ufileos.com: Indonesia (Jakarta)
s3-tw-tp.ufileos.com: Taiwan (Taipei)
s3-afr-nigeria.ufileos.com: Nigeria (Lagos)
s3-bra-saopaulo.ufileos.com: Brazil (São Paulo)
s3-uae-dubai.ufileos.com: United Arab Emirates (Dubai)
s3-ge-fra.ufileos.com: Germany (Frankfurt)
s3-vn-sng.ufileos.com: Vietnam (Ho Chi Minh City)
s3-us-ws.ufileos.com: United States (Washington)
s3-ind-mumbai.ufileos.com: India (Mumbai)
s3-kr-seoul.ufileos.com: Seoul (South Korea)
s3-jpn-tky.ufileos.com: Japan (Tokyo)
s3-th-bkk.ufileos.com: Bangkok (Thailand)
s3-uk-london.ufileos.com: United Kingdom (London)
s3-rus-mosc.ufileos.com: Russia (Moscow)
s3-cn-guiyang1.ufileos.com: Guiyang 1
s3-pk-khi.ufileos.com: Pakistan
acl:
private: |-
Owner gets FULL_CONTROL.
No one else has access rights (default).
public-read: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ access.
storage_class:
'': Default
STANDARD: Standard storage class
ARCHIVE: Archive storage mode
STANDARD_IA: Infrequent access storage mode
bucket_acl: true
quirks:
list_version: 1
use_multipart_etag: false
use_already_exists: false
list_url_encode: true
+4
View File
@@ -33,6 +33,7 @@ type Quirks struct {
UseXID *bool `yaml:"use_x_id,omitempty"`
SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"`
EtagIsNotMD5 *bool `yaml:"etag_is_not_md5,omitempty"`
ObjectLockSupported *bool `yaml:"object_lock_supported,omitempty"`
CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"`
MaxUploadParts *int `yaml:"max_upload_parts,omitempty"`
MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"`
@@ -52,6 +53,7 @@ type Provider struct {
// other
IBMApiKey bool `yaml:"ibm_api_key,omitempty"`
IBMResourceInstanceID bool `yaml:"ibm_resource_instance_id,omitempty"`
IBMIAMEndpoint bool `yaml:"ibm_iam_endpoint,omitempty"`
// advanced
BucketACL bool `yaml:"bucket_acl,omitempty"`
@@ -224,6 +226,8 @@ func constructProviders(options fs.Options, providerMap map[string]*Provider) st
addBool(opt, p, p.IBMApiKey)
case "ibm_resource_instance_id":
addBool(opt, p, p.IBMResourceInstanceID)
case "ibm_iam_endpoint":
addBool(opt, p, p.IBMIAMEndpoint)
case "leave_parts_on_error":
addBool(opt, p, p.LeavePartsOnError)
case "sts_endpoint":
+49 -1
View File
@@ -4,6 +4,7 @@ package s3
//go:generate go run gen_setfrom.go -o setfrom.go
import (
"bytes"
"context"
"crypto/md5"
"crypto/tls"
@@ -829,6 +830,10 @@ use |-vv| to see the debug level logs.
}, {
Name: "ibm_resource_instance_id",
Help: "IBM service instance id",
}, {
Name: "ibm_iam_endpoint",
Help: "IBM IAM Endpoint to use for authentication.\n\nLeave blank to use the default public endpoint.",
Advanced: true,
}, {
Name: "object_lock_mode",
Help: `Object Lock mode to apply when uploading or copying objects.
@@ -930,6 +935,18 @@ PutObjectRetention and PutObjectLegalHold API calls after the upload completes.
This adds extra API calls per object, so only enable if your provider requires it.`,
Default: false,
Advanced: true,
}, {
Name: "object_lock_supported",
Help: `Whether the provider supports S3 Object Lock.
This should be true, false or left unset to use the default for the provider.
Set to false for providers that don't fully support the S3 Object Lock API
(e.g. GCS which uses non-standard headers for bypass governance retention
and doesn't implement Legal Hold via the S3 API).
`,
Default: fs.Tristate{},
Advanced: true,
},
}}))
}
@@ -1107,6 +1124,7 @@ type Options struct {
DirectoryBucket bool `config:"directory_bucket"`
IBMAPIKey string `config:"ibm_api_key"`
IBMInstanceID string `config:"ibm_resource_instance_id"`
IBMIAMEndpoint string `config:"ibm_iam_endpoint"`
UseXID fs.Tristate `config:"use_x_id"`
SignAcceptEncoding fs.Tristate `config:"sign_accept_encoding"`
ObjectLockMode string `config:"object_lock_mode"`
@@ -1115,6 +1133,7 @@ type Options struct {
BypassGovernanceRetention bool `config:"bypass_governance_retention"`
BucketObjectLockEnabled bool `config:"bucket_object_lock_enabled"`
ObjectLockSetAfterUpload bool `config:"object_lock_set_after_upload"`
ObjectLockSupported fs.Tristate `config:"object_lock_supported"`
}
// Fs represents a remote s3 server
@@ -1541,7 +1560,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
fs.Debugf(nil, "Using v2 auth")
if opt.Provider == "IBMCOS" && opt.IBMAPIKey != "" && opt.IBMInstanceID != "" {
options = append(options, func(s3Opt *s3.Options) {
s3Opt.HTTPSignerV4 = &IbmIamSigner{APIKey: opt.IBMAPIKey, InstanceID: opt.IBMInstanceID}
s3Opt.HTTPSignerV4 = &IbmIamSigner{APIKey: opt.IBMAPIKey, InstanceID: opt.IBMInstanceID, IAMEndpoint: opt.IBMIAMEndpoint}
})
} else {
options = append(options, func(s3Opt *s3.Options) {
@@ -1721,6 +1740,7 @@ func setQuirks(opt *Options, provider *Provider) {
set(&opt.UseUnsignedPayload, true, provider.Quirks.UseUnsignedPayload)
set(&opt.UseXID, true, provider.Quirks.UseXID)
set(&opt.SignAcceptEncoding, true, provider.Quirks.SignAcceptEncoding)
set(&opt.ObjectLockSupported, true, provider.Quirks.ObjectLockSupported)
}
// setRoot changes the root of the Fs
@@ -4603,8 +4623,31 @@ func (o *Object) uploadMultipart(ctx context.Context, src fs.ObjectInfo, in io.R
return wantETag, gotETag, versionID, s3cw.ui, nil
}
// bufferForObjectLockMD5 buffers the body and computes Content-MD5 when
// Object Lock parameters are set on the request. AWS S3 requires Content-MD5
// for PutObject with Object Lock params and cannot compute it automatically
// from a non-seekable io.Reader.
// See: https://github.com/aws/aws-sdk-go-v2/discussions/2960
func bufferForObjectLockMD5(req *s3.PutObjectInput, in io.Reader) (io.Reader, error) {
if req.ObjectLockMode == "" && req.ObjectLockRetainUntilDate == nil && req.ObjectLockLegalHoldStatus == "" {
return in, nil
}
buf, err := io.ReadAll(in)
if err != nil {
return nil, fmt.Errorf("failed to read body for Content-MD5: %w", err)
}
md5sum := md5.Sum(buf)
md5base64 := base64.StdEncoding.EncodeToString(md5sum[:])
req.ContentMD5 = &md5base64
return bytes.NewReader(buf), nil
}
// Upload a single part using PutObject
func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
in, err = bufferForObjectLockMD5(req, in)
if err != nil {
return etag, lastModified, nil, err
}
req.Body = io.NopCloser(in)
var options = []func(*s3.Options){}
if o.fs.opt.UseUnsignedPayload.Value {
@@ -4635,6 +4678,11 @@ func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjec
// Upload a single part using a presigned request
func (o *Object) uploadSinglepartPresignedRequest(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
// Content-MD5 must be set before signing so it's included in the presigned URL.
in, err = bufferForObjectLockMD5(req, in)
if err != nil {
return etag, lastModified, nil, err
}
// Create the presigned request
putReq, err := s3.NewPresignClient(o.fs.c).PresignPutObject(ctx, req, s3.WithPresignExpires(15*time.Minute))
if err != nil {
+52
View File
@@ -499,6 +499,9 @@ func (f *Fs) InternalTestVersions(t *testing.T) {
}
func (f *Fs) InternalTestObjectLock(t *testing.T) {
if !f.opt.ObjectLockSupported.Value {
t.Skip("Object Lock not supported by this provider (quirk object_lock_supported = false)")
}
ctx := context.Background()
// Create a temporary bucket with Object Lock enabled to test on.
@@ -721,6 +724,55 @@ func (f *Fs) InternalTestObjectLock(t *testing.T) {
assert.WithinDuration(t, retainUntilDate, gotRetainDate, time.Second)
assert.Equal(t, "ON", gotMetadata["object-lock-legal-hold-status"])
})
t.Run("Multipart", func(t *testing.T) {
// Force multipart upload by setting a very low cutoff
oldCutoff := f.opt.UploadCutoff
f.opt.UploadCutoff = fs.SizeSuffix(1)
f.opt.ObjectLockMode = "GOVERNANCE"
f.opt.ObjectLockRetainUntilDate = retainUntilDate.Format(time.RFC3339)
defer func() {
f.opt.UploadCutoff = oldCutoff
f.opt.ObjectLockMode = ""
f.opt.ObjectLockRetainUntilDate = ""
}()
contents := random.String(100)
item := fstest.NewItem("test-object-lock-multipart", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
defer func() {
removeLocked(t, obj)
}()
o := obj.(*Object)
gotMetadata, err := o.Metadata(ctx)
require.NoError(t, err)
assert.Equal(t, "GOVERNANCE", gotMetadata["object-lock-mode"])
})
t.Run("Presigned", func(t *testing.T) {
// Use presigned request upload path
f.opt.UsePresignedRequest = true
f.opt.ObjectLockMode = "GOVERNANCE"
f.opt.ObjectLockRetainUntilDate = retainUntilDate.Format(time.RFC3339)
defer func() {
f.opt.UsePresignedRequest = false
f.opt.ObjectLockMode = ""
f.opt.ObjectLockRetainUntilDate = ""
}()
contents := random.String(100)
item := fstest.NewItem("test-object-lock-presigned", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
defer func() {
removeLocked(t, obj)
}()
o := obj.(*Object)
gotMetadata, err := o.Metadata(ctx)
require.NoError(t, err)
assert.Equal(t, "GOVERNANCE", gotMetadata["object-lock-mode"])
})
}
func (f *Fs) InternalTest(t *testing.T) {
+19 -1
View File
@@ -225,6 +225,7 @@ type Fs struct {
hasOCMD5 bool // set if can use owncloud style checksums for MD5
hasOCSHA1 bool // set if can use owncloud style checksums for SHA1
hasMESHA1 bool // set if can use fastmail style checksums for SHA1
useStandardProps bool // set if should use standard props for PROPFIND
ntlmAuthMu sync.Mutex // mutex to serialize NTLM auth roundtrips
chunksUploadURL string // upload URL for nextcloud chunked
canChunk bool // set if nextcloud and nextcloud_chunk_size is set
@@ -352,7 +353,10 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string, depth string)
}
if f.hasOCMD5 || f.hasOCSHA1 {
opts.Body = bytes.NewBuffer(owncloudProps)
} else if f.useStandardProps {
opts.Body = bytes.NewBuffer(standardProps)
}
// Note: According to WebDAV RFC 4918, empty PROPFIND body defaults to allprop
var result api.Multistatus
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
@@ -711,6 +715,7 @@ func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
f.precision = time.Second
f.useOCMtime = true
case "other":
f.useStandardProps = true
default:
fs.Debugf(f, "Unknown vendor %q", vendor)
}
@@ -759,13 +764,23 @@ var owncloudProps = []byte(`<?xml version="1.0"?>
<d:getlastmodified />
<d:getcontentlength />
<d:resourcetype />
<d:getcontenttype />
<oc:checksums />
<oc:permissions />
</d:prop>
</d:propfind>
`)
var standardProps = []byte(`<?xml version="1.0"?>
<d:propfind xmlns:d="DAV:">
<d:prop>
<d:displayname/>
<d:getlastmodified/>
<d:getcontentlength/>
<d:resourcetype/>
</d:prop>
</d:propfind>
`)
// list the objects into the function supplied
//
// If directories is set it only sends directories
@@ -787,7 +802,10 @@ func (f *Fs) listAll(ctx context.Context, dir string, directoriesOnly bool, file
}
if f.hasOCMD5 || f.hasOCSHA1 {
opts.Body = bytes.NewBuffer(owncloudProps)
} else if f.useStandardProps {
opts.Body = bytes.NewBuffer(standardProps)
}
// Note: According to WebDAV RFC 4918, empty PROPFIND body defaults to `allprop`
var result api.Multistatus
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
+4 -3
View File
@@ -127,14 +127,14 @@ func init() {
// and the Command line syntax section of docs/content/bisync.md (it doesn't update automatically)
flags.BoolVarP(cmdFlags, &Opt.Resync, "resync", "1", Opt.Resync, "Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first.", "")
flags.FVarP(cmdFlags, &Opt.ResyncMode, "resync-mode", "", "During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.)", "")
flags.BoolVarP(cmdFlags, &Opt.CheckAccess, "check-access", "", Opt.CheckAccess, makeHelp("Ensure expected {CHECKFILE} files are found on both Path1 and Path2 filesystems, else abort."), "")
flags.StringVarP(cmdFlags, &Opt.CheckFilename, "check-filename", "", Opt.CheckFilename, makeHelp("Filename for --check-access (default: {CHECKFILE})"), "")
flags.BoolVarP(cmdFlags, &Opt.CheckAccess, "check-access", "", Opt.CheckAccess, MakeHelp("Ensure expected {CHECKFILE} files are found on both Path1 and Path2 filesystems, else abort."), "")
flags.StringVarP(cmdFlags, &Opt.CheckFilename, "check-filename", "", Opt.CheckFilename, MakeHelp("Filename for --check-access (default: {CHECKFILE})"), "")
flags.BoolVarP(cmdFlags, &Opt.Force, "force", "", Opt.Force, "Bypass --max-delete safety check and run the sync. Consider using with --verbose", "")
flags.FVarP(cmdFlags, &Opt.CheckSync, "check-sync", "", "Controls comparison of final listings: true|false|only (default: true)", "")
flags.BoolVarP(cmdFlags, &Opt.CreateEmptySrcDirs, "create-empty-src-dirs", "", Opt.CreateEmptySrcDirs, "Sync creation and deletion of empty directories. (Not compatible with --remove-empty-dirs)", "")
flags.BoolVarP(cmdFlags, &Opt.RemoveEmptyDirs, "remove-empty-dirs", "", Opt.RemoveEmptyDirs, "Remove ALL empty directories at the final cleanup step.", "")
flags.StringVarP(cmdFlags, &Opt.FiltersFile, "filters-file", "", Opt.FiltersFile, "Read filtering patterns from a file", "")
flags.StringVarP(cmdFlags, &Opt.Workdir, "workdir", "", Opt.Workdir, makeHelp("Use custom working dir - useful for testing. (default: {WORKDIR})"), "")
flags.StringVarP(cmdFlags, &Opt.Workdir, "workdir", "", Opt.Workdir, MakeHelp("Use custom working dir - useful for testing. (default: {WORKDIR})"), "")
flags.StringVarP(cmdFlags, &Opt.BackupDir1, "backup-dir1", "", Opt.BackupDir1, "--backup-dir for Path1. Must be a non-overlapping path on the same remote.", "")
flags.StringVarP(cmdFlags, &Opt.BackupDir2, "backup-dir2", "", Opt.BackupDir2, "--backup-dir for Path2. Must be a non-overlapping path on the same remote.", "")
flags.StringVarP(cmdFlags, &Opt.DebugName, "debugname", "", Opt.DebugName, "Debug by tracking one file at various points throughout a bisync run (when -v or -vv)", "")
@@ -153,6 +153,7 @@ func init() {
flags.StringVarP(cmdFlags, &Opt.ConflictSuffixFlag, "conflict-suffix", "", Opt.ConflictSuffixFlag, "Suffix to use when renaming a --conflict-loser. Can be either one string or two comma-separated strings to assign different suffixes to Path1/Path2. (default: 'conflict')", "")
_ = cmdFlags.MarkHidden("debugname")
_ = cmdFlags.MarkHidden("localtime")
addRC()
}
// bisync command definition
+65 -53
View File
@@ -1,65 +1,77 @@
package bisync
//go:build none
// Create the help text for the rc
//
// Run with go generate ./cmd/bisync (defined in rc.go)
package main
import (
"strconv"
"fmt"
"os"
"strings"
"github.com/muesli/reflow/wordwrap"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/cmd/bisync"
"github.com/rclone/rclone/fs"
"github.com/spf13/pflag"
"golang.org/x/text/cases"
"golang.org/x/text/language"
)
func makeHelp(help string) string {
replacer := strings.NewReplacer(
"|", "`",
"{MAXDELETE}", strconv.Itoa(DefaultMaxDelete),
"{CHECKFILE}", DefaultCheckFilename,
// "{WORKDIR}", DefaultWorkdir,
)
return replacer.Replace(help)
// Output the help to stdout
func main() {
out := os.Stdout
if len(os.Args) > 1 {
var err error
out, err = os.Create(os.Args[1])
if err != nil {
fs.Fatalf(nil, "Open output failed: %v", err)
}
defer out.Close()
}
fmt.Fprintf(out, "<!--- Docs generated by help.go - use go generate to rebuild - DO NOT EDIT --->\n\n")
fmt.Fprint(out, RcHelp())
}
var shortHelp = `Perform bidirectional synchronization between two paths.`
var rcHelp = makeHelp(`This takes the following parameters
- path1 - a remote directory string e.g. |drive:path1|
- path2 - a remote directory string e.g. |drive:path2|
- dryRun - dry-run mode
- resync - performs the resync run
- checkAccess - abort if {CHECKFILE} files are not found on both filesystems
- checkFilename - file name for checkAccess (default: {CHECKFILE})
- maxDelete - abort sync if percentage of deleted files is above
this threshold (default: {MAXDELETE})
- force - Bypass maxDelete safety check and run the sync
- checkSync - |true| by default, |false| disables comparison of final listings,
|only| will skip sync, only compare listings from the last run
- createEmptySrcDirs - Sync creation and deletion of empty directories.
(Not compatible with --remove-empty-dirs)
- removeEmptyDirs - remove empty directories at the final cleanup step
- filtersFile - read filtering patterns from a file
- ignoreListingChecksum - Do not use checksums for listings
- resilient - Allow future runs to retry after certain less-serious errors, instead of requiring resync.
- workdir - server directory for history files (default: |~/.cache/rclone/bisync|)
- backupdir1 - --backup-dir for Path1. Must be a non-overlapping path on the same remote.
- backupdir2 - --backup-dir for Path2. Must be a non-overlapping path on the same remote.
- noCleanup - retain working files
// RcHelp returns the rc help
func RcHelp() string {
return wordwrap.String(bisync.MakeHelp(`This takes the following parameters:
- path1 (required) - (string) a remote directory string e.g. ||drive:path1||
- path2 (required) - (string) a remote directory string e.g. ||drive:path2||
- dryRun - (bool) dry-run mode
`+GenerateParams()+`
See [bisync command help](https://rclone.org/commands/rclone_bisync/)
and [full bisync description](https://rclone.org/bisync/)
for more information.`)
for more information.
`), 76)
}
var longHelp = shortHelp + makeHelp(`
// example: "create-empty-src-dirs" -> "createEmptySrcDirs"
func toCamel(s string) string {
split := strings.Split(s, "-")
builder := strings.Builder{}
for i, word := range split {
if i == 0 { // first word always all lowercase
builder.WriteString(strings.ToLower(word))
continue
}
builder.WriteString(cases.Title(language.AmericanEnglish).String(word))
}
return builder.String()
}
[Bisync](https://rclone.org/bisync/) provides a
bidirectional cloud sync solution in rclone.
It retains the Path1 and Path2 filesystem listings from the prior run.
On each successive run it will:
- list files on Path1 and Path2, and check for changes on each side.
Changes include |New|, |Newer|, |Older|, and |Deleted| files.
- Propagate changes on Path1 to Path2, and vice-versa.
Bisync is considered an **advanced command**, so use with care.
Make sure you have read and understood the entire [manual](https://rclone.org/bisync)
(especially the [Limitations](https://rclone.org/bisync/#limitations) section)
before using, or data loss can result. Questions can be asked in the
[Rclone Forum](https://forum.rclone.org/).
See [full bisync description](https://rclone.org/bisync/) for details.`)
// GenerateParams automatically generates the param list from commandDefinition.Flags
func GenerateParams() string {
builder := strings.Builder{}
fn := func(flag *pflag.Flag) {
if flag.Hidden {
return
}
builder.WriteString(fmt.Sprintf("- %s - (%s) %s \n", toCamel(flag.Name), flag.Value.Type(), flag.Usage))
}
commandDefinition, _, _ := cmd.Root.Find([]string{"bisync"})
commandDefinition.Flags().VisitAll(fn)
return builder.String()
}
+12 -1
View File
@@ -98,15 +98,26 @@ func (b *bisyncRun) lockFileIsExpired() bool {
rdf, err := os.Open(b.lockFile)
b.handleErr(b.lockFile, "error reading lock file", err, true, true)
dec := json.NewDecoder(rdf)
var decodeErr error
for {
if err := dec.Decode(&b.lockFileOpt.data); err != nil {
if err != io.EOF {
fs.Errorf(b.lockFile, "err: %v", err)
decodeErr = err
}
break
}
}
b.handleErr(b.lockFile, "error closing file", rdf.Close(), true, true)
if decodeErr != nil {
if b.opt.MaxLock < basicallyforever {
fs.Infof(b.lockFile, Color(terminal.YellowFg, "Lock file is unreadable (decode error: %v) and --max-lock is set. Treating as expired."), decodeErr)
markFailed(b.listing1)
markFailed(b.listing2)
return true
}
fs.Errorf(b.lockFile, Color(terminal.RedFg, "Lock file exists, but contents are unreadable. (decode error: %v)"), decodeErr)
return false
}
if !b.lockFileOpt.data.TimeExpires.IsZero() && b.lockFileOpt.data.TimeExpires.Before(time.Now()) {
fs.Infof(b.lockFile, Color(terminal.GreenFg, "Lock file found, but it expired at %v. Will delete it and proceed."), b.lockFileOpt.data.TimeExpires)
markFailed(b.listing1) // listing is untrusted so force revert to prior (if --recover) or create new ones (if --resync)
+80
View File
@@ -0,0 +1,80 @@
package bisync
import (
"encoding/json"
"os"
"path/filepath"
"testing"
"time"
"github.com/rclone/rclone/fs"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func newTestLockfileBisyncRun(t *testing.T, lockContent string, maxLock fs.Duration) *bisyncRun {
t.Helper()
dir := t.TempDir()
lockPath := filepath.Join(dir, "test.lck")
require.NoError(t, os.WriteFile(lockPath, []byte(lockContent), 0600))
listing1 := filepath.Join(dir, "listing1")
listing2 := filepath.Join(dir, "listing2")
require.NoError(t, os.WriteFile(listing1, []byte(""), 0600))
require.NoError(t, os.WriteFile(listing2, []byte(""), 0600))
return &bisyncRun{
lockFile: lockPath,
opt: &Options{MaxLock: maxLock},
listing1: listing1,
listing2: listing2,
}
}
func TestLockfileIsExpired_UnreadableWithMaxLock(t *testing.T) {
b := newTestLockfileBisyncRun(t, "not json!!!", fs.Duration(5*time.Minute))
assert.True(t, b.lockFileIsExpired(), "unreadable lockfile with --max-lock set should be treated as expired")
}
func TestLockfileIsExpired_UnreadableWithoutMaxLock(t *testing.T) {
b := newTestLockfileBisyncRun(t, "not json!!!", basicallyforever)
assert.False(t, b.lockFileIsExpired(), "unreadable lockfile without --max-lock should not be treated as expired")
}
func TestLockfileIsExpired_ValidExpired(t *testing.T) {
data := struct {
Session string
PID string
TimeRenewed time.Time
TimeExpires time.Time
}{
Session: "test",
PID: "12345",
TimeRenewed: time.Now().Add(-10 * time.Minute),
TimeExpires: time.Now().Add(-5 * time.Minute),
}
content, err := json.Marshal(data)
require.NoError(t, err)
b := newTestLockfileBisyncRun(t, string(content), fs.Duration(5*time.Minute))
assert.True(t, b.lockFileIsExpired(), "valid lockfile with past expiry should be expired")
}
func TestLockfileIsExpired_ValidNotExpired(t *testing.T) {
data := struct {
Session string
PID string
TimeRenewed time.Time
TimeExpires time.Time
}{
Session: "test",
PID: "12345",
TimeRenewed: time.Now(),
TimeExpires: time.Now().Add(10 * time.Minute),
}
content, err := json.Marshal(data)
require.NoError(t, err)
b := newTestLockfileBisyncRun(t, string(content), fs.Duration(5*time.Minute))
assert.False(t, b.lockFileIsExpired(), "valid lockfile with future expiry should not be expired")
}
+124 -25
View File
@@ -1,16 +1,22 @@
//go:generate go run help.go rc.md
package bisync
import (
"context"
_ "embed"
"errors"
"log"
"path/filepath"
"strconv"
"strings"
"github.com/rclone/rclone/cmd/bisync/bilib"
"github.com/rclone/rclone/fs"
fslog "github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/fs/rc"
)
func init() {
func addRC() {
rc.Add(rc.Call{
Path: "sync/bisync",
AuthRequired: true,
@@ -20,6 +26,42 @@ func init() {
})
}
//go:embed rc.md
var rcHelp string
var shortHelp = `Perform bidirectional synchronization between two paths.`
var longHelp = shortHelp + MakeHelp(`
[Bisync](https://rclone.org/bisync/) provides a
bidirectional cloud sync solution in rclone.
It retains the Path1 and Path2 filesystem listings from the prior run.
On each successive run it will:
- list files on Path1 and Path2, and check for changes on each side.
Changes include ||New||, ||Newer||, ||Older||, and ||Deleted|| files.
- Propagate changes on Path1 to Path2, and vice-versa.
Bisync is considered an **advanced command**, so use with care.
Make sure you have read and understood the entire [manual](https://rclone.org/bisync)
(especially the [Limitations](https://rclone.org/bisync/#limitations) section)
before using, or data loss can result. Questions can be asked in the
[Rclone Forum](https://forum.rclone.org/).
See [full bisync description](https://rclone.org/bisync/) for details.
`)
// MakeHelp replaces some dynamic variables for the help docs
func MakeHelp(help string) string {
replacer := strings.NewReplacer(
"||", "`",
"{MAXDELETE}", strconv.Itoa(DefaultMaxDelete),
"{CHECKFILE}", DefaultCheckFilename,
"{WORKDIR}", DefaultWorkdir,
)
return replacer.Replace(help)
}
func rcBisync(ctx context.Context, in rc.Params) (out rc.Params, err error) {
opt := &Options{}
octx, ci := fs.AddConfig(ctx)
@@ -41,56 +83,86 @@ func rcBisync(ctx context.Context, in rc.Params) (out rc.Params, err error) {
}
if opt.Resync, err = in.GetBool("resync"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("resync", "optional parameter is missing. using default value: %v", opt.Resync)
}
if opt.CheckAccess, err = in.GetBool("checkAccess"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("checkAccess", "optional parameter is missing. using default value: %v", opt.CheckAccess)
}
if opt.Force, err = in.GetBool("force"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("force", "optional parameter is missing. using default value: %v", opt.Force)
}
if opt.CreateEmptySrcDirs, err = in.GetBool("createEmptySrcDirs"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("createEmptySrcDirs", "optional parameter is missing. using default value: %v", opt.CreateEmptySrcDirs)
}
if opt.RemoveEmptyDirs, err = in.GetBool("removeEmptyDirs"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("removeEmptyDirs", "optional parameter is missing. using default value: %v", opt.RemoveEmptyDirs)
}
if opt.NoCleanup, err = in.GetBool("noCleanup"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("noCleanup", "optional parameter is missing. using default value: %v", opt.NoCleanup)
}
if opt.IgnoreListingChecksum, err = in.GetBool("ignoreListingChecksum"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("ignoreListingChecksum", "optional parameter is missing. using default value: %v", opt.IgnoreListingChecksum)
}
if opt.Resilient, err = in.GetBool("resilient"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("resilient", "optional parameter is missing. using default value: %v", opt.Resilient)
}
if opt.CheckFilename, err = in.GetString("checkFilename"); rc.NotErrParamNotFound(err) {
return
opt.CheckFilename = DefaultCheckFilename
fs.Debugf("checkFilename", "optional parameter is missing. using default value: %v", opt.CheckFilename)
}
if opt.FiltersFile, err = in.GetString("filtersFile"); rc.NotErrParamNotFound(err) {
return
fs.Debugf("filtersFile", "optional parameter is missing. using default value: %v", opt.FiltersFile)
}
if opt.Workdir, err = in.GetString("workdir"); rc.NotErrParamNotFound(err) {
return
// "" sets correct default later
fs.Debugf("workdir", "optional parameter is missing. using default value: %v", opt.Workdir)
}
if opt.BackupDir1, err = in.GetString("backupdir1"); rc.NotErrParamNotFound(err) {
return
if opt.BackupDir1, err = in.GetString("backupDir1"); rc.NotErrParamNotFound(err) {
// we accept an alternate capitalization here for backward compatibility.
if opt.BackupDir1, err = in.GetString("backupdir1"); rc.NotErrParamNotFound(err) {
fs.Debugf("backupDir1", "optional parameter is missing. using default value: %v", opt.BackupDir1)
}
}
if opt.BackupDir2, err = in.GetString("backupdir2"); rc.NotErrParamNotFound(err) {
return
if opt.BackupDir2, err = in.GetString("backupDir2"); rc.NotErrParamNotFound(err) {
// we accept an alternate capitalization here for backward compatibility.
if opt.BackupDir2, err = in.GetString("backupdir2"); rc.NotErrParamNotFound(err) {
fs.Debugf("backupDir2", "optional parameter is missing. using default value: %v", opt.BackupDir2)
}
}
checkSync, err := in.GetString("checkSync")
if rc.NotErrParamNotFound(err) {
if err = setEnum(in, "checkSync", "true", opt.CheckSync.Set); err != nil {
return nil, err
}
if checkSync == "" {
checkSync = "true"
}
if err := opt.CheckSync.Set(checkSync); err != nil {
if err = setEnum(in, "resyncMode", opt.ResyncMode.String(), opt.ResyncMode.Set); err != nil {
return nil, err
}
if err = setEnum(in, "conflictResolve", opt.ConflictResolve.String(), opt.ConflictResolve.Set); err != nil {
return nil, err
}
if err = setEnum(in, "conflictLoser", opt.ConflictLoser.String(), opt.ConflictLoser.Set); err != nil {
return nil, err
}
if opt.ConflictSuffixFlag, err = in.GetString("conflictSuffix"); rc.NotErrParamNotFound(err) {
fs.Debugf("conflictSuffix", "optional parameter is missing. using default value: %v", opt.ConflictSuffixFlag)
}
if opt.Recover, err = in.GetBool("recover"); rc.NotErrParamNotFound(err) {
fs.Debugf("recover", "optional parameter is missing. using default value: %v", opt.Recover)
}
if opt.CompareFlag, err = in.GetString("compare"); rc.NotErrParamNotFound(err) {
fs.Debugf("compare", "optional parameter is missing. using default value: %v", opt.CompareFlag)
}
if opt.Compare.NoSlowHash, err = in.GetBool("noSlowHash"); rc.NotErrParamNotFound(err) {
fs.Debugf("noSlowHash", "optional parameter is missing. using default value: %v", opt.Compare.NoSlowHash)
}
if opt.Compare.SlowHashSyncOnly, err = in.GetBool("slowHashSyncOnly"); rc.NotErrParamNotFound(err) {
fs.Debugf("slowHashSyncOnly", "optional parameter is missing. using default value: %v", opt.Compare.SlowHashSyncOnly)
}
if opt.Compare.DownloadHash, err = in.GetBool("downloadHash"); rc.NotErrParamNotFound(err) {
fs.Debugf("downloadHash", "optional parameter is missing. using default value: %v", opt.Compare.DownloadHash)
}
if opt.MaxLock, err = in.GetFsDuration("maxLock"); rc.NotErrParamNotFound(err) {
opt.MaxLock = 0
fs.Debugf("maxLock", "optional parameter is missing. using default value: %v", opt.MaxLock)
}
fs1, err := rc.GetFsNamed(octx, in, "path1")
if err != nil {
@@ -105,6 +177,33 @@ func rcBisync(ctx context.Context, in rc.Params) (out rc.Params, err error) {
output := bilib.CaptureOutput(func() {
err = Bisync(octx, fs1, fs2, opt)
})
workDir, _ := filepath.Abs(DefaultWorkdir)
if opt.Workdir != "" {
workDir, _ = filepath.Abs(opt.Workdir)
}
basePath := bilib.BasePath(ctx, workDir, fs1, fs2)
_, _ = log.Writer().Write(output)
return rc.Params{"output": string(output)}, err
return rc.Params{
"output": string(output),
"session": bilib.SessionName(fs1, fs2),
"workDir": workDir,
"basePath": basePath,
"listing1": basePath + ".path1.lst",
"listing2": basePath + ".path2.lst",
"logFile": fslog.Opt.File,
}, err
}
func setEnum(in rc.Params, name string, defaultVal string, set func(s string) error) error {
v, err := in.GetString(name)
if rc.NotErrParamNotFound(err) || v == "" {
v = defaultVal
fs.Debugf(name, "optional parameter is missing. using default value: %v", v)
}
if err := set(v); err != nil {
return err
}
return nil
}
+61
View File
@@ -0,0 +1,61 @@
<!--- Docs generated by help.go - use go generate to rebuild - DO NOT EDIT --->
This takes the following parameters:
- path1 (required) - (string) a remote directory string e.g. `drive:path1`
- path2 (required) - (string) a remote directory string e.g. `drive:path2`
- dryRun - (bool) dry-run mode
- backupDir1 - (string) --backup-dir for Path1. Must be a non-overlapping path on
the same remote.
- backupDir2 - (string) --backup-dir for Path2. Must be a non-overlapping path on
the same remote.
- checkAccess - (bool) Ensure expected RCLONE_TEST files are found on both
Path1 and Path2 filesystems, else abort.
- checkFilename - (string) Filename for --check-access (default: RCLONE_TEST)
- checkSync - (string) Controls comparison of final listings: true|false|only
(default: true)
- compare - (string) Comma-separated list of bisync-specific compare options ex.
'size,modtime,checksum' (default: 'size,modtime')
- conflictLoser - (ConflictLoserAction) Action to take on the loser of a sync
conflict (when there is a winner) or on both files (when there is no
winner): , num, pathname, delete (default: num)
- conflictResolve - (string) Automatically resolve conflicts by preferring the
version that is: none, path1, path2, newer, older, larger, smaller (default:
none)
- conflictSuffix - (string) Suffix to use when renaming a --conflict-loser. Can
be either one string or two comma-separated strings to assign different
suffixes to Path1/Path2. (default: 'conflict')
- createEmptySrcDirs - (bool) Sync creation and deletion of empty directories.
(Not compatible with --remove-empty-dirs)
- downloadHash - (bool) Compute hash by downloading when otherwise
unavailable. (warning: may be slow and use lots of data!)
- filtersFile - (string) Read filtering patterns from a file
- force - (bool) Bypass --max-delete safety check and run the sync. Consider
using with --verbose
- ignoreListingChecksum - (bool) Do not use checksums for listings (add --ignore-
checksum to additionally skip post-copy checksum checks)
- maxLock - (Duration) Consider lock files older than this to be expired
(default: 0 (never expire)) (minimum: 2m)
- noCleanup - (bool) Retain working files (useful for troubleshooting and
testing).
- noSlowHash - (bool) Ignore listing checksums only on backends where they are
slow
- recover - (bool) Automatically recover from interruptions without requiring --
resync.
- removeEmptyDirs - (bool) Remove ALL empty directories at the final cleanup
step.
- resilient - (bool) Allow future runs to retry after certain less-serious
errors, instead of requiring --resync.
- resync - (bool) Performs the resync run. Equivalent to --resync-mode path1.
Consider using --verbose or --dry-run first.
- resyncMode - (string) During resync, prefer the version that is: path1,
path2, newer, older, larger, smaller (default: path1 if --resync, otherwise
none for no resync.)
- slowHashSyncOnly - (bool) Ignore slow checksums for listings and deltas, but
still consider them during sync calls.
- workdir - (string) Use custom working dir - useful for testing. (default:
~/.cache/rclone/bisync)
See [bisync command help](https://rclone.org/commands/rclone_bisync/)
and [full bisync description](https://rclone.org/bisync/)
for more information.
+16 -14
View File
@@ -1,7 +1,8 @@
Rclone @ allows Linux, FreeBSD, macOS and Windows to
mount any of Rclone's cloud storage systems as a file system with FUSE.
First set up your remote using `rclone config`. Check it works with `rclone ls` etc.
First set up your remote using `rclone config`. Check it works with `rclone ls`
etc.
On Linux and macOS, you can run mount in either foreground or background (aka
daemon) mode. Mount runs in foreground mode by default. Use the `--daemon` flag
@@ -29,8 +30,8 @@ The following examples will mount to an automatically assigned drive,
to specific drive letter `X:`, to path `C:\path\parent\mount`
(where parent directory or drive must exist, and mount must **not** exist,
and is not supported when [mounting as a network drive](#mounting-modes-on-windows)),
and the last example will mount as network share `\\cloud\remote` and map it to an
automatically assigned drive:
and the last example will mount as network share `\\cloud\remote` and map it to
an automatically assigned drive:
```console
rclone @ remote:path/to/files *
@@ -136,8 +137,8 @@ rclone @ remote:path/to/files X: --volname \\server\share
You may also specify the network share UNC path as the mountpoint itself. Then rclone
will automatically assign a drive letter, same as with `*` and use that as
mountpoint, and instead use the UNC path specified as the volume name, as if it were
specified with the `--volname` option. This will also implicitly set
mountpoint, and instead use the UNC path specified as the volume name, as if it
were specified with the `--volname` option. This will also implicitly set
the `--network-mode` option. This means the following two examples have same result:
```console
@@ -268,7 +269,7 @@ does not suffer from the same limitations.
Mounting on macOS can be done either via [built-in NFS server](/commands/rclone_serve_nfs/),
[macFUSE](https://osxfuse.github.io/) (also known as osxfuse) or
[FUSE-T](https://www.fuse-t.org/).macFUSE is a traditional FUSE driver utilizing
[FUSE-T](https://www.fuse-t.org/). macFUSE is a traditional FUSE driver utilizing
a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which
"mounts" via an NFSv4 local server.
@@ -311,13 +312,13 @@ current as of FUSE-T version 1.0.14.
As per the [FUSE-T wiki](https://github.com/macos-fuse-t/fuse-t/wiki#caveats):
> File access and modification times cannot be set separately as it seems to be an
> issue with the NFS client which always modifies both. Can be reproduced with
> 'touch -m' and 'touch -a' commands
> File access and modification times cannot be set separately as it seems to be
> an issue with the NFS client which always modifies both. Can be reproduced
> with `touch -m` and `touch -a` commands
This means that viewing files with various tools, notably macOS Finder, will cause
rlcone to update the modification time of the file. This may make rclone upload a
full new copy of the file.
This means that viewing files with various tools, notably macOS Finder, will
cause rlcone to update the modification time of the file. This may make rclone
upload a full new copy of the file.
##### Read Only mounts
@@ -331,9 +332,10 @@ On newer versions of Ubuntu, you may encounter the following error when running
> NOTICE: mount helper error: fusermount3: mount failed: Permission denied
> CRITICAL: Fatal error: failed to mount FUSE fs: fusermount: exit status 1
This may be due to newer [Apparmor](https://wiki.ubuntu.com/AppArmor) restrictions,
which can be disabled with `sudo aa-disable /usr/bin/fusermount3` (you may need to
`sudo apt install apparmor-utils` beforehand).
which can be disabled with `sudo aa-disable /usr/bin/fusermount3` (you may need
to `sudo apt install apparmor-utils` beforehand).
### Limitations
+1 -1
View File
@@ -34,7 +34,7 @@
}
},
"highlight": {
"style": "monokailight"
"noClasses": false
}
}
}
+6 -5
View File
@@ -8,7 +8,7 @@ notoc: true
# Rclone syncs your files to cloud storage
<!-- markdownlint-disable-next-line line-length -->
{{< img width="50%" src="/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" >}}
{{< img width="50%" src="/img/logo_on_light__horizontal_color.svg" dark-src="/img/logo_on_dark__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" class="hide-mobile" >}}
- [About rclone](#about)
- [What can rclone do for you?](#what)
@@ -216,6 +216,7 @@ WebDAV or S3, that work out of the box.)
{{< provider name="SugarSync" home="https://sugarsync.com/" config="/sugarsync/" >}}
{{< provider name="Tencent Cloud Object Storage (COS)" home="https://intl.cloud.tencent.com/product/cos" config="/s3/#tencent-cos" >}}
{{< provider name="Uloz.to" home="https://uloz.to" config="/ulozto/" >}}
{{< provider name="US3" home="https://www.ucloud.cn/site/product/ufile.html" config="/s3/#us3" >}}
{{< provider name="Wasabi" home="https://wasabi.com/" config="/s3/#wasabi" >}}
{{< provider name="WebDAV" home="https://en.wikipedia.org/wiki/WebDAV" config="/webdav/" >}}
{{< provider name="Yandex Disk" home="https://disk.yandex.com/" config="/yandex/" >}}
@@ -248,7 +249,7 @@ These backends adapt or modify other storage providers:
## Links
- {{< icon "fa fa-home fa-fw" >}} [Home page](https://rclone.org/)
- {{< icon "fab fa-github fa-fw" >}} [GitHub project page for source and bug tracker](https://github.com/rclone/rclone)
- {{< icon "fa fa-comments fa-fw" >}} [Rclone Forum](https://forum.rclone.org)
- {{< icon "fas fa-cloud-download-alt fa-fw" >}}[Downloads](/downloads/)
- [Home page](https://rclone.org/)
- [GitHub project page for source and bug tracker](https://github.com/rclone/rclone)
- [Rclone Forum](https://forum.rclone.org)
- [Downloads](/downloads/)
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Remote Aliases"
versionIntroduced: "v1.40"
---
# {{< icon "fa fa-link" >}} Alias
# Alias
The `alias` remote provides a new name for another remote.
+1 -1
View File
@@ -5,7 +5,7 @@ versionIntroduced: "v1.20"
status: Removed
---
# {{< icon "fab fa-amazon" >}} Amazon Drive
# Amazon Drive
Amazon Drive, formerly known as Amazon Cloud Drive, was a cloud storage
service run by Amazon for consumers.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Archive Remote"
versionIntroduced: "v1.72"
---
# {{< icon "fas fa-archive" >}} Archive
# Archive
The Archive backend allows read only access to the content of archive
files on cloud storage without downloading the complete archive. This
+15
View File
@@ -1094,3 +1094,18 @@ put them back in again. -->
- Bjoern Franke <bjo@schafweide.org>
- razorloves <razorloves@gmail.com>
- Duncan F <131309315+duncanaf@users.noreply.github.com>
- Billy Hughes <w.g.h.hughes@gmail.com>
- Bhagyashreek8 <57988406+Bhagyashreek8@users.noreply.github.com>
- Andriy Senyshyn <135705861+sam-mt@users.noreply.github.com>
- Andrew Furman <7343697+furmandev@users.noreply.github.com>
- ZRHan <56144550+ZRHann@users.noreply.github.com>
- Jason <jasonbabo@gmail.com>
- Patrick Farrell <pfarrell@whamcloud.com>
- BizaNator <17164422+BizaNator@users.noreply.github.com>
- lif <1835304752@qq.com>
- jinkeyuu <44105232+jinkeyuu@users.noreply.github.com>
- jinyu.han <jinyu.han@ucloud.cn>
- Claude Opus 4.6 <noreply@anthropic.com>
- Andriy Senyshyn <andriy.senyshyn@kiteworks.com>
- Mike GIllan <mike.gillan@gmail.com>
- Xiangzhe <xiangzhedev@gmail.com>
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Microsoft Azure Blob Storage"
versionIntroduced: "v1.38"
---
# {{< icon "fab fa-windows" >}} Microsoft Azure Blob Storage
# Microsoft Azure Blob Storage
Paths are specified as `remote:container` (or `remote:` for the `lsd`
command.) You may put subdirectories in too, e.g.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Microsoft Azure Files Storage"
versionIntroduced: "v1.65"
---
# {{< icon "fab fa-windows" >}} Microsoft Azure Files Storage
# Microsoft Azure Files Storage
Paths are specified as `remote:` You may put subdirectories in too,
e.g. `remote:path/to/dir`.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Backblaze B2"
versionIntroduced: "v1.26"
---
# {{< icon "fa fa-fire" >}} Backblaze B2
# Backblaze B2
B2 is [Backblaze's cloud storage system](https://www.backblaze.com/cloud-storage).
+12
View File
@@ -820,6 +820,12 @@ running -- and you can therefore be reasonably sure that any *expired* lock
file you may find was left there by an interrupted run, not one that is still
running and just taking awhile.
If a lock file exists but its contents are unreadable (for example, due to an
incomplete write or disk error), it is treated as expired when `--max-lock` is
set to a value greater than `0`. If `--max-lock` is `0` or not set, an
unreadable lock file will produce an error and block future runs until removed
manually.
If `--max-lock` is `0` or not set, the default is that lock files will never
expire, and will block future runs (of these same two bisync paths)
indefinitely.
@@ -1897,6 +1903,12 @@ about *Unison* and synchronization in general.
## Changelog
### `v1.74`
- Added several missing `rc` parameters.
- Optional `rc` parameters are now truly optional.
- `rc` output now provides more structured information.
### `v1.71`
- `bisync` is now officially released from beta.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Box"
versionIntroduced: "v1.38"
---
# {{< icon "fa fa-archive" >}} Box
# Box
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for cache remote"
versionIntroduced: "v1.39"
---
# {{< icon "fa fa-archive" >}} Cache
# Cache
The `cache` remote wraps another existing remote and stores file structure
and its data for long running tasks like `rclone mount`.
+41
View File
@@ -6,6 +6,47 @@ description: "Rclone Changelog"
# Changelog
## v1.73.3 - 2026-03-23
[See commits](https://github.com/rclone/rclone/compare/v1.73.2...v1.73.3)
- Bug Fixes
- build
- Update to google.golang.org/grpc 1.79.3 to fix CVE-2026-33186 (dependabot[bot])
- Update to github.com/buger/jsonparser 1.1.2 to fix GHSA-6g7g-w4f8-9c9x (dependabot[bot])
- doc fixes
- Added text to the label showing version-introduced info (Jan-Philipp Reßler)
- Clarify Filen password change requires updating both password and API key in rclone config (Jason)
- s3: clarify multi tenant support for Cubbit (Marco Ferretti)
- jottacloud: Fix broken link (albertony)
- lib/rest: Fix URLPathEscapeAll breaking WebDAV servers (eg nzbdav) with strict path matching (Andrew Furman)
- list: Fix nil pointer panic in Sorter when temp file creation fails (Nick Craig-Wood)
## v1.73.2 - 2026-03-06
[See commits](https://github.com/rclone/rclone/compare/v1.73.1...v1.73.2)
- Bug Fixes
- build
- Update to go 1.25.8 to fix CVE-2026-27137 CVE-2026-27138 CVE-2026-25679 CVE-2026-27142 (Nick Craig-Wood)
- Update github.com/cloudflare/circl to v1.6.3 to fix CVE-2026-1229 (Nick Craig-Wood)
- Update to golang.org/x/net v0.51.0 to fix CVE-2026-27141 (Nick Craig-Wood)
- docs fixes:
- bisync: Add group Sync to the bisync command (Jan-Philipp Reßler)
- Note that --use-server-modtime only works on some backends (Nick Craig-Wood)
- Document unsupported S3 object keys with double slashes (Adam Kasztenny)
- Fix headers hierarchy for mount.md (Dark Dragon)
- Fix new drive flag typo in changelog (razorloves)
- Archive
- Extract: fix extraction with "./" prefix from tar entry paths (Varun Chawla)
- Drime
- Fix chunk-uploaded files ignoring workspace ID (a1pcm)
- Internxt
- Fix Entry doesn't belong in directory errors on windows (jzunigax2)
- WebDAV
- Escape reserved characters in URL path segments (Varun Chawla)
- Add missing headers for CORS (Romāns Potašovs)
## v1.73.1 - 2026-02-17
[See commits](https://github.com/rclone/rclone/compare/v1.73.0...v1.73.1)
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Split-chunking overlay remote"
versionIntroduced: "v1.50"
---
# {{< icon "fa fa-cut" >}} Chunker
# Chunker
The `chunker` overlay transparently splits large files into smaller chunks
during upload to wrapped remote and transparently assembles them back
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Cloudinary backend"
versionIntroduced: "v1.69"
---
# {{< icon "fa fa-cloud" >}} Cloudinary
# Cloudinary
This is a backend for the [Cloudinary](https://cloudinary.com/) platform
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Combine several remotes into one"
versionIntroduced: "v1.59"
---
# {{< icon "fa fa-folder-plus" >}} Combine
# Combine
The `combine` backend joins remotes together into a single directory
tree.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Compression Remote"
versionIntroduced: "v1.54"
---
# {{< icon "fas fa-compress" >}} Compress
# Compress
## Warning
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Encryption overlay remote"
versionIntroduced: "v1.33"
---
# {{< icon "fa fa-lock" >}} Crypt
# Crypt
Rclone `crypt` remotes encrypt and decrypt other remotes.
+18 -17
View File
@@ -753,21 +753,21 @@ object also.
Here is a table of standard system metadata which, if appropriate, a
backend may implement.
| key | description | example |
|---------------------|-------------|---------|
| mode | File type and mode: octal, unix style | 0100664 |
| uid | User ID of owner: decimal number | 500 |
| gid | Group ID of owner: decimal number | 500 |
| rdev | Device ID (if special file) => hexadecimal | 0 |
| atime | Time of last access: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| mtime | Time of last modification: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| btime | Time of file creation (birth): RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| utime | Time of file upload: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| cache-control | Cache-Control header | no-cache |
| key | description | example |
| --- | ----------- | ------- |
| mode | File type and mode: octal, unix style | 0100664 |
| uid | User ID of owner: decimal number | 500 |
| gid | Group ID of owner: decimal number | 500 |
| rdev | Device ID (if special file) => hexadecimal | 0 |
| atime | Time of last access: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| mtime | Time of last modification: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| btime | Time of file creation (birth): RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| utime | Time of file upload: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
| cache-control | Cache-Control header | no-cache |
| content-disposition | Content-Disposition header | inline |
| content-encoding | Content-Encoding header | gzip |
| content-language | Content-Language header | en-US |
| content-type | Content-Type header | text/plain |
| content-encoding | Content-Encoding header | gzip |
| content-language | Content-Language header | en-US |
| content-type | Content-Type header | text/plain |
The metadata keys `mtime` and `content-type` will take precedence if
supplied in the metadata over reading the `Content-Type` or
@@ -1193,7 +1193,8 @@ on any OS, and the value is defined as following:
- On Windows: `%HOME%` if defined, else `%USERPROFILE%`, or else `%HOMEDRIVE%\%HOMEPATH%`.
- On Unix: `$HOME` if defined, else by looking up current user in OS-specific user
database (e.g. passwd file), or else use the result from shell command `cd && pwd`.
database (e.g. passwd file), or else use the result from shell command
`cd && pwd`.
If you run `rclone config file` you will see where the default location is for
you. Running `rclone config touch` will ensure a configuration file exists,
@@ -2059,7 +2060,7 @@ This is the default.
`CAUTIOUS` will try to prevent rclone from reaching the limit. Only applicable
for `--max-transfer`.
## -M, --metadata
### -M, --metadata
Setting this flag enables rclone to copy the metadata from the source
to the destination. For local backends this is ownership, permissions,
@@ -3444,7 +3445,7 @@ many items, the input is treated as a [CSV encoded](https://godoc.org/encoding/c
string. For example
| Environment variable | Equivalent options |
|----------------------|--------------------|
| -------------------- | ------------------ |
| `RCLONE_EXCLUDE="*.jpg"` | `--exclude "*.jpg"` |
| `RCLONE_EXCLUDE="*.jpg,*.png"` | `--exclude "*.jpg"` `--exclude "*.png"` |
| `RCLONE_EXCLUDE='"*.jpg","*.png"'` | `--exclude "*.jpg"` `--exclude "*.png"` |
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for DOI"
versionIntroduced: "v1.70"
---
# {{< icon "fa fa-building-columns" >}} DOI
# DOI
The DOI remote is a read only remote for reading files from digital object
identifiers (DOI).
+5 -4
View File
@@ -16,7 +16,7 @@ image](https://securebuild.com/images/rclone) through our partner
## Release {{% version %}} OS requirements {#osrequirements}
| OS | Minimum Version |
|:-------:|:-------:|
| :---: | :---: |
| Linux | Kernel 3.2 |
| macOS | 12 (Monterey) |
| Windows | 10, Server 2016 |
@@ -31,7 +31,7 @@ in the Go Wiki.
## Release {{% version %}} {#release}
| Arch-OS | Windows | macOS | Linux | .deb | .rpm | FreeBSD | NetBSD | OpenBSD | Plan9 | Solaris |
|:-------:|:-------:|:-----:|:-----:|:----:|:----:|:-------:|:------:|:-------:|:-----:|:-------:|
| :-----: | :-----: | :---: | :---: | :--: | :--: | :-----: | :----: | :-----: | :---: | :-----: |
| Intel/AMD - 64 Bit | {{< download windows amd64 >}} | {{< download osx amd64 >}} | {{< download linux amd64 >}} | {{< download linux amd64 deb >}} | {{< download linux amd64 rpm >}} | {{< download freebsd amd64 >}} | {{< download netbsd amd64 >}} | {{< download openbsd amd64 >}} | {{< download plan9 amd64 >}} | {{< download solaris amd64 >}} |
| Intel/AMD - 32 Bit | {{< download windows 386 >}} | - | {{< download linux 386 >}} | {{< download linux 386 deb >}} | {{< download linux 386 rpm >}} | {{< download freebsd 386 >}} | {{< download netbsd 386 >}} | {{< download openbsd 386 >}} | {{< download plan9 386 >}} | - |
| ARMv5 - 32 Bit NOHF | - | - | {{< download linux arm >}} | {{< download linux arm deb >}} | {{< download linux arm rpm >}} | {{< download freebsd arm >}} | {{< download netbsd arm >}} | - | - | - |
@@ -120,7 +120,7 @@ If you would like to download the current version (maybe from a
script) from a URL which doesn't change then you can use these links.
| Arch-OS | Windows | macOS | Linux | .deb | .rpm | FreeBSD | NetBSD | OpenBSD | Plan9 | Solaris |
|:-------:|:-------:|:-----:|:-----:|:----:|:----:|:-------:|:------:|:-------:|:-----:|:-------:|
| :-----: | :-----: | :---: | :---: | :--: | :--: | :-----: | :----: | :-----: | :---: | :-----: |
| Intel/AMD - 64 Bit | {{< cdownload windows amd64 >}} | {{< cdownload osx amd64 >}} | {{< cdownload linux amd64 >}} | {{< cdownload linux amd64 deb >}} | {{< cdownload linux amd64 rpm >}} | {{< cdownload freebsd amd64 >}} | {{< cdownload netbsd amd64 >}} | {{< cdownload openbsd amd64 >}} | {{< cdownload plan9 amd64 >}} | {{< cdownload solaris amd64 >}} |
| Intel/AMD - 32 Bit | {{< cdownload windows 386 >}} | - | {{< cdownload linux 386 >}} | {{< cdownload linux 386 deb >}} | {{< cdownload linux 386 rpm >}} | {{< cdownload freebsd 386 >}} | {{< cdownload netbsd 386 >}} | {{< cdownload openbsd 386 >}} | {{< cdownload plan9 386 >}} | - |
| ARMv5 - 32 Bit NOHF | - | - | {{< cdownload linux arm >}} | {{< cdownload linux arm deb >}} | {{< cdownload linux arm rpm >}} | {{< cdownload freebsd arm >}} | {{< cdownload netbsd arm >}} | - | - | - |
@@ -137,12 +137,13 @@ Older downloads can be found at <https://downloads.rclone.org/>
The latest `rclone` version working for:
| OS | Maximum rclone version |
|:-------:|:-------:|
| :---: | :---: |
| Windows 7 | v1.63.1 |
| Windows Server 2008 | v1.63.1 |
| Windows Server 2012 | v1.63.1 |
| Windows XP | v1.42 |
| Windows Vista | v1.42 |
| macOS 10.15 (Catalina) | v1.70.3 |
| macOS 10.14 (Mojave) | v1.63.1 |
| macOS 10.13 (High Sierra) | v1.63.1 |
| macOS 10.12 (Sierra) | v1.56.0 |
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Drime"
versionIntroduced: "v1.73"
---
# {{< icon "fa fa-cloud" >}} Drime
# Drime
[Drime](https://drime.cloud/) is a cloud storage and transfer service focused
on fast, resilient file delivery. It offers both free and paid tiers with
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Google drive"
versionIntroduced: "v0.91"
---
# {{< icon "fab fa-google" >}} Google Drive
# Google Drive
Paths are specified as `drive:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Dropbox"
versionIntroduced: "v1.02"
---
# {{< icon "fab fa-dropbox" >}} Dropbox
# Dropbox
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for 1Fichier"
versionIntroduced: "v1.49"
---
# {{< icon "fa fa-archive" >}} 1Fichier
# 1Fichier
This is a backend for the [1fichier](https://1fichier.com) cloud
storage service. Note that a Premium subscription is required to use
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for the Enterprise File Fabric backend"
versionIntroduced: "v1.54"
---
# {{< icon "fa fa-cloud" >}} Enterprise File Fabric
# Enterprise File Fabric
This backend supports [Storage Made Easy's Enterprise File
Fabric™](https://storagemadeeasy.com/about/) which provides a software
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for FileLu"
versionIntroduced: "v1.70"
---
# {{< icon "fa fa-folder" >}} FileLu
# FileLu
[FileLu](https://filelu.com/) is a reliable cloud storage provider
offering features like secure file uploads, downloads, flexible
+3 -1
View File
@@ -4,12 +4,14 @@ description: "Rclone docs for Filen"
versionIntroduced: "1.73"
---
# {{< icon "fa fa-solid fa-f" >}} Filen
# Filen
## Configuration
The initial setup for Filen requires that you get an API key for your account,
currently this is only possible using the [Filen CLI](https://github.com/FilenCloudDienste/filen-cli).
This means you must first download the CLI, login, and then run the `export-api-key` command.
**Note:** If you change your Filen account password, you will need to update your rclone configuration with the new password and a re-exported API key (via `export-api-key`).
Here is an example of how to make a remote called `FilenRemote`. First run:
rclone config
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Files.com"
versionIntroduced: "v1.68"
---
# {{< icon "fa fa-brands fa-files-pinwheel" >}} Files.com
# Files.com
[Files.com](https://www.files.com/) is a cloud storage service that provides a
secure and easy way to store and share files.
+22 -22
View File
@@ -202,28 +202,28 @@ them into regular expressions.
## Filter pattern examples {#examples}
| Description | Pattern | Matches | Does not match |
| ----------- |-------- | ------- | -------------- |
| Wildcard | `*.jpg` | `/file.jpg` | `/file.png` |
| | | `/dir/file.jpg` | `/dir/file.png` |
| Rooted | `/*.jpg` | `/file.jpg` | `/file.png` |
| | | `/file2.jpg` | `/dir/file.jpg` |
| Alternates | `*.{jpg,png}` | `/file.jpg` | `/file.gif` |
| | | `/dir/file.png` | `/dir/file.gif` |
| Path Wildcard | `dir/**` | `/dir/anyfile` | `file.png` |
| | | `/subdir/dir/subsubdir/anyfile` | `/subdir/file.png` |
| Any Char | `*.t?t` | `/file.txt` | `/file.qxt` |
| | | `/dir/file.tzt` | `/dir/file.png` |
| Range | `*.[a-z]` | `/file.a` | `/file.0` |
| | | `/dir/file.b` | `/dir/file.1` |
| Escape | `*.\?\?\?` | `/file.???` | `/file.abc` |
| | | `/dir/file.???` | `/dir/file.def` |
| Class | `*.\d\d\d` | `/file.012` | `/file.abc` |
| | | `/dir/file.345` | `/dir/file.def` |
| Regexp | `*.{{jpe?g}}` | `/file.jpeg` | `/file.png` |
| | | `/dir/file.jpg` | `/dir/file.jpeeg` |
| Rooted Regexp | `/{{.*\.jpe?g}}` | `/file.jpeg` | `/file.png` |
| | | `/file.jpg` | `/dir/file.jpg` |
| Description | Pattern | Matches | Does not match |
| ----------- | ---------------- | ------------------------------- | ------------------ |
| Wildcard | `*.jpg` | `/file.jpg` | `/file.png` |
| | | `/dir/file.jpg` | `/dir/file.png` |
| Rooted | `/*.jpg` | `/file.jpg` | `/file.png` |
| | | `/file2.jpg` | `/dir/file.jpg` |
| Alternates | `*.{jpg,png}` | `/file.jpg` | `/file.gif` |
| | | `/dir/file.png` | `/dir/file.gif` |
| Path Wildcard | `dir/**` | `/dir/anyfile` | `file.png` |
| | | `/subdir/dir/subsubdir/anyfile` | `/subdir/file.png` |
| Any Char | `*.t?t` | `/file.txt` | `/file.qxt` |
| | | `/dir/file.tzt` | `/dir/file.png` |
| Range | `*.[a-z]` | `/file.a` | `/file.0` |
| | | `/dir/file.b` | `/dir/file.1` |
| Escape | `*.\?\?\?` | `/file.???` | `/file.abc` |
| | | `/dir/file.???` | `/dir/file.def` |
| Class | `*.\d\d\d` | `/file.012` | `/file.abc` |
| | | `/dir/file.345` | `/dir/file.def` |
| Regexp | `*.{{jpe?g}}` | `/file.jpeg` | `/file.png` |
| | | `/dir/file.jpg` | `/dir/file.jpeeg` |
| Rooted Regexp | `/{{.*\.jpe?g}}` | `/file.jpeg` | `/file.png` |
| | | `/file.jpg` | `/dir/file.jpg` |
## How filter rules are applied to files {#how-filter-rules-work}
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for FTP backend"
versionIntroduced: "v1.37"
---
# {{< icon "fa fa-file" >}} FTP
# FTP
FTP is the File Transfer Protocol. Rclone FTP support is provided using the
[github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp)
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Gofile"
versionIntroduced: "v1.68"
---
# {{< icon "fa fa-folder" >}} Gofile
# Gofile
[Gofile](https://gofile.io) is a content storage and distribution
platform. Its aim is to provide as much service as possible for free
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Google Cloud Storage"
versionIntroduced: "v1.02"
---
# {{< icon "fab fa-google" >}} Google Cloud Storage
# Google Cloud Storage
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Google Photos"
versionIntroduced: "v1.49"
---
# {{< icon "fa fa-images" >}} Google Photos
# Google Photos
The rclone backend for [Google Photos](https://www.google.com/photos/about/) is
a specialized backend for transferring photos and videos to and from
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Better checksums for other remotes"
versionIntroduced: "v1.57"
---
# {{< icon "fa fa-check-double" >}} Hasher
# Hasher
Hasher is a special overlay backend to create remotes which handle
checksums for other remotes. It's main functions include:
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Remote for Hadoop Distributed Filesystem"
versionIntroduced: "v1.54"
---
# {{< icon "fa fa-globe" >}} HDFS
# HDFS
[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)
is a distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/)
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for HiDrive"
versionIntroduced: "v1.59"
---
# {{< icon "fa fa-cloud" >}} HiDrive
# HiDrive
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Read only remote for HTTP servers"
versionIntroduced: "v1.37"
---
# {{< icon "fa fa-globe" >}} HTTP
# HTTP
The HTTP remote is a read only remote for reading files of a
webserver. The webserver should provide file listings which rclone
+17 -3
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for iCloud Drive"
versionIntroduced: "v1.69"
---
# {{< icon "fa fa-cloud" >}} iCloud Drive
# iCloud Drive
## Configuration
@@ -12,13 +12,27 @@ The initial setup for an iCloud Drive backend involves getting a trust token/ses
This can be done by simply using the regular iCloud password, and accepting the code
prompt on another iCloud connected device.
**IMPORTANT**: At the moment an app specific password won't be accepted. Only
use your regular password and 2FA.
**IMPORTANT**: App-specific passwords are not accepted. Only use your
regular Apple ID password and 2FA.
`rclone config` walks you through the token creation. The trust token is valid
for 30 days. After which you will have to reauthenticate with `rclone reconnect`
or `rclone config`.
## Authentication
rclone authenticates with Apple using [SRP (Secure Remote Password)](https://en.wikipedia.org/wiki/Secure_Remote_Password_protocol),
the same protocol used by the iCloud web interface. Your password is never
sent to Apple's servers -- instead, a cryptographic proof is exchanged that
verifies you know the password without revealing it.
The authentication flow is:
1. rclone initiates a session with Apple's identity service
2. An SRP key exchange takes place (your password is used locally to derive a key)
3. Apple sends a 2FA prompt to your trusted devices
4. After you enter the 2FA code, rclone receives a trust token for future sessions
Here is an example of how to make a remote called `iclouddrive`. First run:
```console
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for ImageKit backend."
versionIntroduced: "v1.63"
---
# {{< icon "fa fa-cloud" >}} ImageKit
# ImageKit
This is a backend for the [ImageKit.io](https://imagekit.io/) storage service.
+22 -22
View File
@@ -285,8 +285,8 @@ rclone v1.49.1
- go version: go1.12.9
```
There are a few command line options to consider when starting an rclone Docker container
from the rclone image.
There are a few command line options to consider when starting an rclone Docker
container from the rclone image.
- You need to mount the host rclone config dir at `/config/rclone` into the Docker
container. Due to the fact that rclone updates tokens inside its config file,
@@ -300,8 +300,8 @@ from the rclone image.
data files reside on the host with a non-root UID:GID, you need to pass these
on the container start command line.
- If you want to access the RC interface (either via the API or the Web UI), it is
required to set the `--rc-addr` to `:5572` in order to connect to it from outside
- If you want to access the RC interface (either via the API or the Web UI), it
is required to set the `--rc-addr` to `:5572` in order to connect to it from outside
the container. An explanation about why this is necessary can be found in an old
[pythonspeed.com](https://web.archive.org/web/20200808071950/https://pythonspeed.com/articles/docker-connection-refused/)
article.
@@ -309,9 +309,9 @@ from the rclone image.
probably set it to listen to localhost only, with `127.0.0.1:5572` as the
value for `--rc-addr`
- It is possible to use `rclone mount` inside a userspace Docker container, and expose
the resulting fuse mount to the host. The exact `docker run` options to do that
might vary slightly between hosts. See, e.g. the discussion in this
- It is possible to use `rclone mount` inside a userspace Docker container, and
expose the resulting fuse mount to the host. The exact `docker run` options to
do that might vary slightly between hosts. See, e.g. the discussion in this
[thread](https://github.com/moby/moby/issues/9448).
You also need to mount the host `/etc/passwd` and `/etc/group` for fuse to work
@@ -542,8 +542,8 @@ To override them set the corresponding options (as command-line arguments, or as
After installing and configuring rclone, as described above, you are ready to use
rclone as an interactive command line utility. If your goal is to perform *periodic*
operations, such as a regular [sync](https://rclone.org/commands/rclone_sync/), you
will probably want to configure your rclone command in your operating system's
operations, such as a regular [sync](https://rclone.org/commands/rclone_sync/),
you will probably want to configure your rclone command in your operating system's
scheduler. If you need to expose *service*-like features, such as
[remote control](https://rclone.org/rc/), [GUI](https://rclone.org/gui/),
[serve](https://rclone.org/commands/rclone_serve/) or [mount](https://rclone.org/commands/rclone_mount/),
@@ -583,9 +583,9 @@ c:\rclone\rclone.exe sync c:\files remote:/files --no-console --log-file c:\rclo
As mentioned in the [mount](https://rclone.org/commands/rclone_mount/) documentation,
mounted drives created as Administrator are not visible to other accounts, not even
the account that was elevated as Administrator. By running the mount command as the
built-in `SYSTEM` user account, it will create drives accessible for everyone on
the system. Both scheduled task and Windows service can be used to achieve this.
the account that was elevated as Administrator. By running the mount command as
the built-in `SYSTEM` user account, it will create drives accessible for everyone
on the system. Both scheduled task and Windows service can be used to achieve this.
NOTE: Remember that when rclone runs as the `SYSTEM` user, the user profile
that it sees will not be yours. This means that if you normally run rclone with
@@ -615,8 +615,8 @@ will often give you better results.
#### Start from Task Scheduler
Task Scheduler is an administrative tool built into Windows, and it can be used to
configure rclone to be started automatically in a highly configurable way, e.g.
Task Scheduler is an administrative tool built into Windows, and it can be used
to configure rclone to be started automatically in a highly configurable way, e.g.
periodically on a schedule, on user log on, or at system startup. It can run
be configured to run as the current user, or for a mount command that needs to
be available to all users it can run as the `SYSTEM` user.
@@ -656,18 +656,18 @@ To Windows service running any rclone command, the excellent third-party utility
[NSSM](http://nssm.cc), the "Non-Sucking Service Manager", can be used.
It includes some advanced features such as adjusting process priority, defining
process environment variables, redirect to file anything written to stdout, and
customized response to different exit codes, with a GUI to configure everything from
(although it can also be used from command line ).
customized response to different exit codes, with a GUI to configure everything
from (although it can also be used from command line ).
There are also several other alternatives. To mention one more,
[WinSW](https://github.com/winsw/winsw), "Windows Service Wrapper", is worth checking
out. It requires .NET Framework, but it is preinstalled on newer versions of Windows,
and it also provides alternative standalone distributions which includes necessary
runtime (.NET 5). WinSW is a command-line only utility, where you have to manually
create an XML file with service configuration. This may be a drawback for some, but
it can also be an advantage as it is easy to back up and reuse the configuration
settings, without having go through manual steps in a GUI. One thing to note is that
by default it does not restart the service on error, one have to explicit enable
create an XML file with service configuration. This may be a drawback for some,
but it can also be an advantage as it is easy to back up and reuse the configuration
settings, without having go through manual steps in a GUI. One thing to note is
that by default it does not restart the service on error, one have to explicit enable
this in the configuration file (via the "onfailure" parameter).
### Autostart on Linux
@@ -676,8 +676,8 @@ this in the configuration file (via the "onfailure" parameter).
To always run rclone in background, relevant for mount commands etc,
you can use systemd to set up rclone as a system or user service. Running as a
system service ensures that it is run at startup even if the user it is running as
has no active session. Running rclone as a user service ensures that it only
system service ensures that it is run at startup even if the user it is running
as has no active session. Running rclone as a user service ensures that it only
starts after the configured user has logged into the system.
#### Run periodically from cron
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Internet Archive"
versionIntroduced: "v1.59"
---
# {{< icon "fa fa-archive" >}} Internet Archive
# Internet Archive
The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Internxt Drive"
versionIntroduced: "v1.73"
---
# {{< icon "fas fa-cloud" >}} Internxt Drive
# Internxt Drive
[Internxt Drive](https://internxt.com) is a zero-knowledge encrypted cloud storage service.
+2 -2
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Jottacloud"
versionIntroduced: "v1.43"
---
# {{< icon "fa fa-cloud" >}} Jottacloud
# Jottacloud
Jottacloud is a cloud storage service provider from a Norwegian company, using
its own datacenters in Norway.
@@ -12,7 +12,7 @@ its own datacenters in Norway.
In addition to the official service at [jottacloud.com](https://www.jottacloud.com/),
it also provides white-label solutions to different companies. The following
are currently supported by this backend, using a different authentication setup
as described [below](#whitelabel-authentication):
as described [below](#traditional):
- Elkjøp (with subsidiaries):
- Elkjøp Cloud (cloud.elkjop.no)
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Koofr"
versionIntroduced: "v1.47"
---
# {{< icon "fa fa-suitcase" >}} Koofr
# Koofr
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Linkbox"
versionIntroduced: "v1.65"
---
# {{< icon "fa fa-infinity" >}} Linkbox
# Linkbox
Linkbox is [a private cloud drive](https://linkbox.to/).
+2 -2
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for the local filesystem"
versionIntroduced: "v0.91"
---
# {{< icon "fas fa-hdd" >}} Local Filesystem
# Local Filesystem
Local paths are specified as normal filesystem paths, e.g. `/path/to/wherever`, so
@@ -62,7 +62,7 @@ handling file names.
| / | 0x2F | / |
When running on Windows the following characters are replaced. This
list is based on the [Windows file naming conventions](https://docs.microsoft.com/de-de/windows/desktop/FileIO/naming-a-file#naming-conventions).
list is based on the [Windows file naming conventions](https://docs.microsoft.com/windows/desktop/FileIO/naming-a-file#naming-conventions).
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Mail.ru Cloud"
versionIntroduced: "v1.50"
---
# {{< icon "fas fa-at" >}} Mail.ru Cloud
# Mail.ru Cloud
[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a
Russian internet company [Mail.Ru Group](https://mail.ru). The official
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Mega"
versionIntroduced: "v1.41"
---
# {{< icon "fa fa-archive" >}} Mega
# Mega
[Mega](https://mega.nz/) is a cloud storage and file hosting service
known for its security feature where all files are encrypted locally
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Memory backend"
versionIntroduced: "v1.51"
---
# {{< icon "fas fa-memory" >}} Memory
# Memory
The memory backend is an in RAM backend. It does not persist its
data - use the local backend for that.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Akamai NetStorage"
versionIntroduced: "v1.58"
---
# {{< icon "fas fa-database" >}} Akamai NetStorage
# Akamai NetStorage
Paths are specified as `remote:`
You may put subdirectories in too, e.g. `remote:/path/to/dir`.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Microsoft OneDrive"
versionIntroduced: "v1.24"
---
# {{< icon "fab fa-windows" >}} Microsoft OneDrive
# Microsoft OneDrive
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for OpenDrive"
versionIntroduced: "v1.42"
---
# {{< icon "fa fa-file" >}} OpenDrive
# OpenDrive
Paths are specified as `remote:path`
+1 -1
View File
@@ -5,7 +5,7 @@ type: page
versionIntroduced: "v1.60"
---
# {{< icon "fa fa-cloud" >}} Oracle Object Storage
# Oracle Object Storage
Object Storage provided by the Oracle Cloud Infrastructure (OCI).
Read more at <oracle.com>:
@@ -3,7 +3,7 @@ title: "Oracle Object Storage Mount"
description: "Oracle Object Storage mounting tutorial"
---
# {{< icon "fa fa-cloud" >}} Mount Buckets and Expose via NFS Tutorial
# Mount Buckets and Expose via NFS Tutorial
This runbook shows how to [mount](/commands/rclone_mount/) *Oracle Object Storage*
buckets as local file system in OCI compute Instance using rclone tool.
+12 -12
View File
@@ -39,7 +39,7 @@ size by default, though can be configured to check the file hash
change the timestamp of an existing file without having to re-upload it.
| Key | Explanation |
|-----|-------------|
| --- | ----------- |
| `-` | ModTimes not supported - times likely the upload time |
| `R` | ModTimes supported on files but can't be changed without re-upload |
| `R/W` | Read and Write ModTimes fully supported on files |
@@ -186,8 +186,8 @@ will be escaped with the `‛` character to avoid ambiguous file names.
Each cloud storage backend can use a different set of characters,
which will be specified in the documentation for each backend.
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
| Character | Value | Replacement |
| --------- | :---: | :---------- :|
| NUL | 0x00 | ␀ |
| SOH | 0x01 | ␁ |
| STX | 0x02 | ␂ |
@@ -227,9 +227,9 @@ The default encoding will also encode these file names as they are
problematic with many cloud storage systems.
| File name | Replacement |
| --------- |:-----------:|
| --------- | :--------- :|
| . | . |
| .. | .. |
| .. | .. |
#### Invalid UTF-8 bytes {#invalid-utf8}
@@ -269,8 +269,8 @@ list of all possible values by passing an invalid value to this
flag, e.g. `--local-encoding "help"`. The command `rclone help flags encoding`
will show you the defaults for the backends.
| Encoding | Characters | Encoded as |
| --------- | ---------- | ---------- |
| Encoding | Characters | Encoded as |
| -------- | ---------- | ---------- |
| Asterisk | `*` | `*` |
| BackQuote | `` ` `` | ``` |
| BackSlash | `\` | `\` |
@@ -395,12 +395,12 @@ that backend) and/or user metadata (general purpose metadata).
The levels of metadata support are
| Key | Explanation |
|-----|-------------|
| `R` | Read only System Metadata on files only|
| `RW` | Read and write System Metadata on files only|
| `RWU` | Read and write System Metadata and read and write User Metadata on files only|
| --- | ----------- |
| `R` | Read only System Metadata on files only |
| `RW` | Read and write System Metadata on files only |
| `RWU` | Read and write System Metadata and read and write User Metadata on files only |
| `DR` | Read only System Metadata on files and directories |
| `DRW` | Read and write System Metadata on files and directories|
| `DRW` | Read and write System Metadata on files and directories |
| `DRWU` | Read and write System Metadata and read and write User Metadata on files and directories |
See [the metadata docs](/docs/#metadata) for more info.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for pCloud"
versionIntroduced: "v1.39"
---
# {{< icon "fa fa-cloud" >}} pCloud
# pCloud
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for PikPak"
versionIntroduced: "v1.62"
---
# {{< icon "fa fa-cloud" >}} PikPak
# PikPak
PikPak is [a private cloud drive](https://mypikpak.com/).
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Pixeldrain"
versionIntroduced: "v1.68"
---
# {{< icon "fa fa-circle" >}} Pixeldrain
# Pixeldrain
This is the backend for Pixeldrain's premium filesystem feature. This is not the
same as pixeldrain's free file sharing product. The filesystem requires either a
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for premiumize.me"
versionIntroduced: "v1.49"
---
# {{< icon "fa fa-user" >}} premiumize.me
# premiumize.me
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Proton Drive"
versionIntroduced: "v1.64.0"
---
# {{< icon "fa fa-folder" >}} Proton Drive
# Proton Drive
[Proton Drive](https://proton.me/drive) is an end-to-end encrypted Swiss vault
for your files that protects your data.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for put.io"
versionIntroduced: "v1.49"
---
# {{< icon "fas fa-parking" >}} put.io
# put.io
Paths are specified as `remote:path`
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for QingStor Object Storage"
versionIntroduced: "v1.38"
---
# {{< icon "fas fa-hdd" >}} QingStor
# QingStor
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Quatrix"
versionIntroduced: "v1.63.2"
---
# {{< icon "fas fa-shield-alt" >}} Quatrix
# Quatrix
Quatrix by Maytech is [Quatrix Secure Compliant File Sharing | Maytech](https://www.maytech.net/products/quatrix-business).
+145 -7
View File
@@ -4,7 +4,7 @@ description: "Rclone docs for Amazon S3"
versionIntroduced: "v0.91"
---
# {{< icon "fab fa-amazon" >}} Amazon S3 Storage Providers
# Amazon S3 Storage Providers
The S3 backend can be used with a number of different providers:
@@ -55,6 +55,7 @@ The S3 backend can be used with a number of different providers:
{{< provider name="Storj" home="https://storj.io/" config="/s3/#storj" >}}
{{< provider name="Synology C2 Object Storage" home="https://c2.synology.com/en-global/object-storage/overview" config="/s3/#synology-c2" >}}
{{< provider name="Tencent Cloud Object Storage (COS)" home="https://intl.cloud.tencent.com/product/cos" config="/s3/#tencent-cos" >}}
{{< provider name="US3" home="https://www.ucloud.cn/site/product/ufile.html" config="/s3/#us3" >}}
{{< provider name="Wasabi" home="https://wasabi.com/" config="/s3/#wasabi" >}}
{{< provider name="Zadara Object Storage" home="https://www.zadara.com" config="/s3/#zadara" >}}
{{< provider name="Zata" home="https://zata.ai/" config="/s3/#Zata" end="true" >}}
@@ -913,10 +914,9 @@ According to AWS's [documentation on S3 Object Lock](https://docs.aws.amazon.com
> If you configure a default retention period on a bucket, requests to upload
objects in such a bucket must include the Content-MD5 header.
As mentioned in the [Modification times and hashes](#modification-times-and-hashes)
section, small files that are not uploaded as multipart, use a different tag, causing
the upload to fail. A simple solution is to set the `--s3-upload-cutoff 0` and force
all the files to be uploaded as multipart.
Rclone automatically computes and includes the Content-MD5 header when
uploading objects with Object Lock parameters set. No special
configuration is needed.
#### Setting Object Lock retention
@@ -5132,7 +5132,9 @@ can follow this [guide](https://docs.cubbit.io/getting-started/quickstart#api-ke
to retrieve these keys. They will be needed when prompted by `rclone config`.
Default region will correspond to `eu-west-1` and the endpoint has to be specified
as `s3.cubbit.eu`.
as `s3.cubbit.eu`. If you set a custom tenant endpoint, you must use
`s3.{tenant_name}.cubbit.eu`, because the generic `s3.cubbit.eu` endpoint will not
work in this case.
Going through the whole process of creating a new remote by running `rclone config`,
each prompt should be answered as shown below:
@@ -5145,7 +5147,7 @@ env_auth> false
access_key_id> YOUR_ACCESS_KEY
secret_access_key> YOUR_SECRET_KEY
region> eu-west-1 (or leave empty)
endpoint> s3.cubbit.eu
endpoint> s3.cubbit.eu (or s3.{tenant_name}.cubbit.eu)
acl>
```
@@ -9117,6 +9119,142 @@ To configure access to Tencent COS, follow the steps below:
==== ====
cos s3
```
### US3 {#us3}
[US3](https://www.ucloud.cn/site/product/ufile.html) Object Storage is a service that provides cloud storage for unstructured files for internet applications.
To configure access to US3, follow the steps below:
1. First run:
```console
rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
```
2. Give the name of the configuration. For example, name it 'US3'.
```text
name> US3
```
3. Select `s3` storage.
```text
Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
\ "s3"
[snip]
Storage> s3
```
4. Select `US3` provider.
```text
Choose a number from below, or type in your own value.
XX / Amazon Web Services (AWS) S3
\ (AWS)
[snip]
XX / UCloud Object Storage (US3)
\ (US3)
[snip]
provider> US3
```
5. Enter your SecretId and SecretKey of US3.
```text
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
Choose a number from below, or type in your own value
1 / Enter AWS credentials in the next step
\ "false"
2 / Get AWS credentials from the environment (env vars or IAM)
\ "true"
env_auth> 1
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> AKIDxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxx
```
6. Select endpoint.
```text
Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / North China (Beijing 2).
\ "s3-cn-bj.ufileos.com"
2 / North China (Ulanqab).
\ "s3-cn-wlcb.ufileos.com"
3 / Shanghai.
\ "s3-cn-sh2.ufileos.com"
4 / South China (Guangzhou).
\ "s3-cn-gd.ufileos.com"
[snip]
endpoint> 1
```
7. Choose acl and storage class.
```text
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
2 | The AllUsers group gets READ access.
\ (public-read)
acl> 1
The storage class to use when storing new objects in US3.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Default
\ ""
[snip]
storage_class> 1
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
- type: s3
- provider: US3
- access_key_id: xx
- secret_access_key: xx
- endpoint: s3-cn-bj.ufileos.com
- acl: private
Keep this "US3" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
US3 s3
```
### Wasabi
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Seafile"
versionIntroduced: "v1.52"
---
# {{< icon "fa fa-server" >}} Seafile
# Seafile
This is a backend for the [Seafile](https://www.seafile.com/) storage service:
+1 -1
View File
@@ -4,7 +4,7 @@ description: "SFTP"
versionIntroduced: "v1.36"
---
# {{< icon "fa fa-server" >}} SFTP
# SFTP
SFTP is the [Secure (or SSH) File Transfer
Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).
+1 -1
View File
@@ -4,7 +4,7 @@ description: "Shade Backend Docs"
versionIntroduced: "v1.73"
---
# {{< icon "fa fa-moon" >}} Shade
# Shade
This is a backend for the [Shade](https://shade.inc/) platform
Loaded 100 of 165 files, more files were not shown because too many files have changed in this diff. Show more