lucasw89/rclone
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Version v1.76.0

Browse Source
This commit is contained in:
Nick Craig-Wood committed 2026-11-06 14:37:03 +00:00
1 parent 53b2d7aaf2
commit e982ff3666
51 files changed
+28428 -6443
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

MANUAL.html
Generated
+7688 -2672
View File
File diff suppressed because it is too large. Load diff
MANUAL.md
Generated
+5269 -1283
View File
File diff suppressed because it is too large. Load diff
MANUAL.txt
Generated
+5730 -1334
View File
File diff suppressed because it is too large. Load diff
cmd/bisync/rc.md
+4
View File
@@ -5,6 +5,8 @@ This takes the following parameters:
- path1 (required) - (string) a remote directory string e.g. `drive:path1`
- path2 (required) - (string) a remote directory string e.g. `drive:path2`
- dryRun - (bool) dry-run mode
- maxDelete - (int) Safety check on maximum percentage of deleted files
allowed. If exceeded, the bisync run will abort. (default: 50%)
- backupDir1 - (string) --backup-dir for Path1. Must be a non-overlapping path on
the same remote.
- backupDir2 - (string) --backup-dir for Path2. Must be a non-overlapping path on
@@ -34,6 +36,8 @@ unavailable. (warning: may be slow and use lots of data!)
using with --verbose
- ignoreListingChecksum - (bool) Do not use checksums for listings (add --ignore-
checksum to additionally skip post-copy checksum checks)
- maxDeleteRenamesAware - (bool) Exclude tracked renames from the --max-delete
safety check (requires --track-renames)
- maxLock - (Duration) Consider lock files older than this to be expired
(default: 0 (never expire)) (minimum: 2m)
- noCleanup - (bool) Retain working files (useful for troubleshooting and
docs/content/azureblob.md
+67
View File
@@ -919,6 +919,52 @@ Properties:
- Type: int
- Default: 5000
#### --azureblob-use-arrow-list
Use the Apache Arrow listing format.
If set, directory listings are fetched using the ListBlobs Apache
Arrow response format instead of XML. Arrow responses are smaller and
much cheaper to parse, making listings of large containers several
times faster. Combine with "list_parallelism" for the biggest gains.
"Blob Listing with Apache Arrow" is in public preview at Microsoft and
is only supported on flat namespace accounts. On accounts with a
hierarchical namespace (ADLS Gen2), or where the feature is otherwise
unavailable, the server returns XML and the listing transparently
falls back to the normal XML path (logged at debug level).
Properties:
- Config: use_arrow_list
- Env Var: RCLONE_AZUREBLOB_USE_ARROW_LIST
- Type: bool
- Default: false
#### --azureblob-list-parallelism
Number of parallel shards to list a directory with.
If set greater than 1, the blob name keyspace of each directory is
split into ranges which are listed concurrently using the Arrow
startFrom/endBefore range parameters. This can dramatically speed up
listing containers with millions of objects, for both recursive
(ListR) and single directory listings. Speed keeps improving up to a
parallelism of around 30.
This has no effect unless "use_arrow_list" is also set, as Arrow is the
only listing path that supports server-side name ranges. If the
account does not support range listing (e.g. it has a hierarchical
namespace) the listing falls back to sequential. The default of 0 (or
1) lists sequentially.
Properties:
- Config: list_parallelism
- Env Var: RCLONE_AZUREBLOB_LIST_PARALLELISM
- Type: int
- Default: 0
#### --azureblob-access-tier
Access tier of blob: hot, cool, cold or archive.
@@ -1022,6 +1068,9 @@ creates an empty object ending with "/", to persist the folder.
This object also has the metadata "hdi_isfolder = true" to conform to
the Microsoft standard.
This is not needed on storage accounts with a hierarchical namespace
(see the hns option) as these always support empty directories.
Properties:
@@ -1031,6 +1080,24 @@ Properties:
- Type: bool
- Default: false
#### --azureblob-hns
Set if the storage account has a hierarchical namespace.
Storage accounts with hierarchical namespace enabled (Azure Data Lake
Storage Gen2) have real directories which can be empty.
If this is unset then rclone will read it from the storage account
and save it in the config file.
Properties:
- Config: hns
- Env Var: RCLONE_AZUREBLOB_HNS
- Type: Tristate
- Default: unset
#### --azureblob-no-check-container
If set, don't attempt to check the container exists or create it.
docs/content/azurefiles.md
+2 -2
View File
@@ -736,7 +736,7 @@ Properties:
Upload chunk size.
Note that this is stored in memory and there may be up to
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
"--transfers" * "--azurefiles-upload-concurrency" chunks stored at once
in memory.
Properties:
@@ -758,7 +758,7 @@ links and these uploads do not fully utilize your bandwidth, then
increasing this may help to speed up the transfers.
Note that chunks are stored in memory and there may be up to
"--transfers" * "--azurefile-upload-concurrency" chunks stored at once
"--transfers" * "--azurefiles-upload-concurrency" chunks stored at once
in memory.
Properties:
docs/content/b2.md
+7 -2
View File
@@ -569,6 +569,11 @@ If the custom endpoint rewrites the requests for authentication,
e.g., in Cloudflare Workers, this header needs to be handled properly.
Leave blank if you want to use the endpoint provided by Backblaze.
Rclone will not follow a redirect from this endpoint which downgrades
HTTPS to HTTP, and will not send the "Authorization" header or the
SSE-C key headers once a redirect has left the host and port of this
endpoint.
The URL provided here SHOULD have the protocol and SHOULD NOT have
a trailing slash or specify the /file/bucket subpath as rclone will
request files with "{download_url}/file/{bucket_name}/{path}".
@@ -665,7 +670,7 @@ Properties:
To use SSE-C, you may provide the secret encryption key encoded in a UTF-8 compatible string to encrypt/decrypt your data
Alternatively you can provide --sse-customer-key-base64.
Alternatively you can provide --b2-sse-customer-key-base64.
Properties:
@@ -681,7 +686,7 @@ Properties:
To use SSE-C, you may provide the secret encryption key encoded in Base64 format to encrypt/decrypt your data
Alternatively you can provide --sse-customer-key.
Alternatively you can provide --b2-sse-customer-key.
Properties:
docs/content/bisync.md
+15 -6
View File
@@ -1074,14 +1074,20 @@ The following backends have known issues that need more investigation:
- `TestHuaweiDrive` (`huaweidrive`)
- [`TestBisyncRemoteLocal/ext_paths`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
- [`TestBisyncRemoteLocal/extended_filenames`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
- [`TestBisyncRemoteLocal/normalization`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
- [`TestBisyncLocalRemote/ext_paths`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
- [`TestBisyncLocalRemote/extended_filenames`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
- [4 more](https://pub.rclone.org/integration-tests/current/)
- [`TestBisyncLocalRemote/normalization`](https://pub.rclone.org/integration-tests/current/huaweidrive-cmd.bisync-TestHuaweiDrive-1.txt)
- [3 more](https://pub.rclone.org/integration-tests/current/)
- `TestPcloud` (`pcloud`)
- [`TestBisyncRemoteRemote/check_access`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- [`TestBisyncRemoteRemote/rmdirs`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- Updated: 2026-07-31-010017
- [`TestBisyncRemoteRemote/extended_filenames`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- `TestYandex` (`yandex`)
- [`TestBisyncRemoteLocal/all_changed`](https://pub.rclone.org/integration-tests/current/yandex-cmd.bisync-TestYandex-1.txt)
- [`TestBisyncRemoteLocal/backupdir`](https://pub.rclone.org/integration-tests/current/yandex-cmd.bisync-TestYandex-1.txt)
- [`TestBisyncRemoteLocal/basic`](https://pub.rclone.org/integration-tests/current/yandex-cmd.bisync-TestYandex-1.txt)
- [`TestBisyncRemoteLocal/changes`](https://pub.rclone.org/integration-tests/current/yandex-cmd.bisync-TestYandex-1.txt)
- [`TestBisyncRemoteLocal/check_access`](https://pub.rclone.org/integration-tests/current/yandex-cmd.bisync-TestYandex-1.txt)
- [94 more](https://pub.rclone.org/integration-tests/current/)
- Updated: 2026-11-06-010024
<!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs --->
The following backends either have not been tested recently or have known issues
@@ -1099,6 +1105,8 @@ that are deemed unfixable for the time being:
- `TestFilesCom` (`filescom`)
- `TestImageKit` (`imagekit`)
- `TestJottacloud` (`jottacloud`)
- `TestKdrive` (`kdrive`)
- `TestBisyncRemoteRemote/normalization`
- `TestLinkbox` (`linkbox`)
- `TestMailru` (`mailru`)
- `TestMega` (`mega`)
@@ -1111,7 +1119,8 @@ that are deemed unfixable for the time being:
- `TestQuatrix` (`quatrix`)
- `TestS3GCS` (`s3`)
- `TestBisyncRemoteRemote/extended_filenames`
- `TestS3Rclone` (`s3`)
- `TestBisyncRemoteLocal/extended_filenames`
- `TestBisyncLocalRemote/extended_filenames`
- `TestSFTPRsyncNet` (`sftp`)
- `TestStorj` (`storj`)
- `TestWebdavNextcloud` (`webdav`)
docs/content/changelog.md
+300
View File
@@ -6,6 +6,306 @@ description: "Rclone Changelog"
# Changelog
## v1.76.0 - 2026-11-06
[See commits](https://github.com/rclone/rclone/compare/v1.75.0...v1.76.0)
- New backends
- [Dosya](/dosya/) (Kaya)
- [Infomaniak kDrive](/kdrive/) (Guillaume MALLERET)
- New S3 Providers
- [Autonomys Auto Drive](/s3/#autonomys) (Jim Counter)
- [Clever Cloud Cellar](/s3/#clever-cloud-cellar) (David Legrand)
- [Infomaniak Swiss Backup](/s3/#infomaniak) (Louis)
- [Neon Object Storage](/s3/#neon) (Rishi Raj Jain)
- [RelAix.Storage](/s3/#relaix) (mygrexit)
- New commands
- [index](/commands/rclone_index/): Write static directory listings into a remote (Nick Craig-Wood)
- Security
- build
- Disable Multipath TCP listeners by default to mitigate CVE-2026-98432 (Nick Craig-Wood)
- Fix multiple CVEs including CVE-2026-111943 by upgrading to go1.27.3 (Nick Craig-Wood)
- Update `golang.org/x/net` to v0.62.0 to fix multiple CVEs (Nick Craig-Wood)
- Update `github.com/andybalholm/brotli` to v1.2.7 to fix CVE-2026-111497 (Nick Craig-Wood)
- fshttp: Refuse HTTPS to HTTP redirects unless `--allow-https-downgrade` GHSA-m97j-7x65-wh93 CVE-PENDING (Nick Craig-Wood)
- lib/http: Stop `--htpasswd` and `--user` revealing which usernames exist through timing GHSA-hwj7-wp9g-wrv9 CVE-PENDING (Nick Craig-Wood)
- local
- Fix user metadata escaping the root through a planted symlink GHSA-r5wv-vqqh-2r9c CVE-PENDING (Nick Craig-Wood)
- Fix MkdirMetadata finding directories outside the root through a planted symlink (Nick Craig-Wood)
- Fix Remove and Rmdir deleting outside the root through a planted symlink (Nick Craig-Wood)
- Fix DirMove creating the destination parent through a planted symlink (Nick Craig-Wood)
- netstorage: Fix API auth headers leaking to other hosts on redirect GHSA-m97j-7x65-wh93 CVE-PENDING (Nick Craig-Wood)
- rc
- Limit request size when the rc server has no authentication GHSA-f8p4-8g4j-q3m2 CVE-PENDING (Nick Craig-Wood)
- Reject unauthorised and unknown calls before reading the request GHSA-f8p4-8g4j-q3m2 CVE-PENDING (Nick Craig-Wood)
- seafile: Fix HTTPS to HTTP redirects leaking credentials GHSA-m97j-7x65-wh93 CVE-PENDING (Nick Craig-Wood)
- serve: Make auth proxy revocation take effect within 5 minutes GHSA-vmgx-g9c5-669p CVE-PENDING (Nick Craig-Wood)
- serve sftp
- Fix auth proxy letting a public key be used as a password GHSA-66qv-66mg-9h74 CVE-PENDING (Nick Craig-Wood)
- Fix auth proxy running for public keys the client can't use GHSA-66qv-66mg-9h74 CVE-PENDING (Nick Craig-Wood)
- swift
- Fix auth token leaking to other hosts or over plaintext on redirect GHSA-m97j-7x65-wh93 CVE-PENDING (Nick Craig-Wood)
- Fix password being resent to another host on auth redirect GHSA-m97j-7x65-wh93 CVE-PENDING (Nick Craig-Wood)
- ulozto: Fix auth tokens leaking on redirects from download links GHSA-3xpx-hjg2-vrjh CVE-PENDING (Nick Craig-Wood)
- New Features
- accounting
- Stop `--max-transfer` overshooting the limit (Nick Craig-Wood)
- Add "Updated dirs" count of directory modtime/metadata updates to the stats (Nick Craig-Wood)
- batcher: Log when committing uploads has finished (phatlc)
- bisync
- Make `--max-delete` aware of `--track-renames` (Morax)
- Fall back to `--download` when a checksum is missing (nielash)
- Reject `--max-delete` above 100 (nielash)
- build
- Update to go1.27 and make go1.26 the minimum required version (Nick Craig-Wood)
- Update all dependencies (Nick Craig-Wood)
- cmd: Complete enum flag values from their choices (Demiao Huang)
- config: Only run `--password-command` once when using `--daemon` (phatlc)
- doc fixes (ashwinibailangadi, ferrumclaudepilgrim, frodleTian, FTCHD, gomitrah, Nick Craig-Wood, qodelicious, Satvik Mishra, ZainnQureshii)
- docker: Make container user UID and GID configurable via build ARGs (Vaibhav Mashal)
- filter
- Speed up large `--include-from` lists of directories (kaii9)
- Make `[!...]` in globs a negated character class (kevin9327)
- Make `[^...]` in globs exclude `/` and document it (kevin9327)
- fs
- Reject sizes too large for SizeSuffix instead of silently disabling the limit (KBS)
- Stop writing config for on the fly and environment variable remotes into the config file (Nick Craig-Wood)
- Reject durations like 300y that overflow instead of wrapping (kevin9327)
- Accept multi-unit relative times like 1d12h in Time options (kevin)
- Reject NaN sizes and CountSuffix values that overflow (kevin9327)
- Reject `--bwlimit` times without a colon or with signed digits (kevin9327)
- Reject NaN durations instead of wrapping (Soujas K)
- Add MIME type for JPEG XL (.jxl) files (Seth Westphal)
- gui
- Take the GUI out of beta (Nick Craig-Wood)
- Update embedded release to 1.2.1 (Nick Craig-Wood)
- hash: Add CRC-32C (Castagnoli) support (Dirk Petersen)
- index: Add partial runs which only re-index changed directories (Nick Craig-Wood)
- lib/http
- Return 416 for ranges `--rc-serve` and serve restic can't satisfy (Rohit Behera)
- Replace `github.com/abbot/go-http-auth` with built in htpasswd support (Nick Craig-Wood)
- Accept the Basic auth scheme in any case (kevin)
- lib/proxy: Send the decoded user and password in Proxy-Authorization (kevin)
- lib/transform: Make titlecase capitalise words instead of upper-casing them (kevin)
- ncdu: Show progress and the item count when deleting a selection (Lev Devaev)
- operations
- Check checksums in rcat with known size (phatlc)
- Speed up checksum by hashing files in parallel (Teddy Tennant)
- Fall back to a single stream when a multi-thread source ignores ranges (Morax)
- Show directory operations in the `--progress` display (Nick Craig-Wood)
- Add `operations/getfile` remote control API endpoint (Hakan İSMAİL)
- Make `--immutable` work with copyto, moveto and single file copy (phatlc)
- Respect `--backup-dir` in deletefile (Cao Yuhang)
- rc
- Deprecate `--rc-web-gui` flag (Nick Craig-Wood)
- Add sync report parameters to `sync/sync`, `sync/copy` and `sync/move` (Nick Craig-Wood)
- rest: Reject negative sizes in HTTP response headers (Yousef Issa)
- serve http
- Sort directory listings in the browser (Nick Craig-Wood)
- Redesign the directory listing page (Nick Craig-Wood)
- serve s3
- Report the object's modtime as Last-Modified after a PUT (kevin9327)
- Support changing object tags (Nick Craig-Wood)
- Quote the ETags of objects in listings (Nick Craig-Wood)
- Support listing with delimiters other than "/" (Nick Craig-Wood)
- Support directory markers and refuse file/directory conflicts (Nick Craig-Wood)
- Make the ETag of multipart uploads agree with HEAD and support partNumber (Nick Craig-Wood)
- Make conditional writes atomic against other writes through the server (Nick Craig-Wood)
- Make the ETag returned on upload agree with HEAD for any `--etag-hash` (Nick Craig-Wood)
- List empty directories as directory markers (Nick Craig-Wood)
- Allow copying directory markers (Nick Craig-Wood)
- Remove the directories made for a failed upload (Nick Craig-Wood)
- Keep the MD5 ETag of objects on remotes without MD5 (Nick Craig-Wood)
- Don't read directories needlessly when listing with a delimiter (Nick Craig-Wood)
- Don't compute the MD5 of multipart uploads unless the ETag is MD5 (Nick Craig-Wood)
- List the directory markers put for directories holding other keys (Nick Craig-Wood)
- Let a failed CompleteMultipartUpload be retried (Nick Craig-Wood)
- List objects with the storage class they were put with (Nick Craig-Wood)
- Add `--directory-markers` to list every directory as a marker (Nick Craig-Wood)
- Remove a directory with its last key after its marker is deleted (Nick Craig-Wood)
- Make `--no-cleanup` a deprecated alias of `--directory-markers` (Nick Craig-Wood)
- Document which S3 features are supported and the design goals (Nick Craig-Wood)
- serve webdav
- Sort directory listings in the browser (Nick Craig-Wood)
- Redesign the directory listing page (Nick Craig-Wood)
- size: Include source in JSON output (Jiacheng Xu)
- sync: Add pattern-based transfer ordering (Shurong Cao)
- touch: Parse documented timestamp formats (Nicholas Velten)
- Bug Fixes
- accounting: Fix totalBytes counting server-side copies and moves twice (ilya lavrenov)
- bisync
- Fix conflict winner missing from listings when Path2 wins (nielash)
- Fix `--conflict-resolve` comparing a zero size or modtime (nielash)
- Fix endless retry loop for skipped files (nielash)
- Fix listings ignoring a modtime less than 1s older (nielash)
- Fix rc ignoring global `--dry-run` and `--max-delete` settings (nielash)
- Fix rc ignoring legacy backupdir1/2 and invalid values (nielash)
- Fix internal error when a conflict rename reuses deleted names (TastyHeadphones)
- fs: Fix durations like 1d12h being rejected (Rohit Behera)
- lib/encoder: Fix decoding a trailing character after an escaped quote (kevin9327)
- list: Fix ext sort error sometimes being ignored at the end of a listing (Nick Craig-Wood)
- ncdu: Update to tcell v3 which fixes duplicated keys on Windows (Nick Craig-Wood)
- operations
- Fix making a directory with modtime asking twice with `--interactive` (Nick Craig-Wood)
- Fix case- or Unicode-only renames deleting the file (nielash)
- Fix moving a file onto its own name deleting it (nielash)
- Fix cat `--offset` before the start of a file printing its end (b84330808)
- Fix oversized getfile tail requests (唐僧洗发用飘柔)
- rc: Fix LogLevel in `_config` being ignored (Nick Craig-Wood)
- rest: Fix retries reading the request body after the request has finished (Nick Craig-Wood)
- serve dlna: Fix crash on Browse with a negative StartingIndex or RequestedCount (phatlc)
- serve restic: Fix temporary errors confusing restic into leaving locks behind (Azhar)
- serve s3
- Update to gofakes3 v0.0.8 which fixes 500 errors with go1.27 (Nick Craig-Wood)
- Upgrade gofakes3 to v0.0.11 to fix IsTruncated and Docker Distribution Registry (DEVN)
- Fix buckets being undeletable after an aborted multipart upload (Nick Craig-Wood)
- Fix CopyObject metadata including the source's and an mtime (Nick Craig-Wood)
- serve sftp: Fix rmdir deleting a file instead of returning an error (phatlc)
- sync
- Fix directory modtimes being set twice (Nick Craig-Wood)
- Fix `--delete-empty-src-dirs` disabled by a prior error in a reused stats group (ghwany)
- walk: Fix duplicate directories with `--fast-list` when directories are listed after their contents (Nick Craig-Wood)
- Mount
- Fix VFS instance leaks on mount failures and unmounts (Hakan İSMAİL)
- VFS
- Preserve shrinking truncate on uncached files (47th)
- Fix crash renaming a directory while the dir cache expires (Shreyansh Jain)
- Fix virtual entries leak in removed directory (Biagio Peccerillo)
- Fix renamed items listed twice after a case or Unicode-only match (nielash)
- Fix uploads never starting while a dirty file keeps being read (Neil Cawse)
- Local
- Speed up default checksummed copies by writing in larger blocks (Dominik Sander)
- Make out of space errors fatal during multi-thread transfers (ferrumclaudepilgrim)
- Stop `--copy-links` following symlink loops (phatlc)
- Limit the size of symlink targets read from .rclonelink files (Nick Craig-Wood)
- Fix server-side copy on macOS when the destination exists (Ayush Sharma)
- Set only the ownership a non-root user may set (Dirk Petersen)
- Crypt
- Fix nil pointer panic in ObjectInfo.Hash (fixfon)
- Archive
- Read symlinks stored in zip and squashfs archives (Lev Devaev)
- Fix reading an archive in the root of a remote (Oshino Penguin)
- Azure Blob
- Enable on Solaris (acoeur)
- Fix directories on storage accounts with hierarchical namespace (Nick Craig-Wood)
- Fix NewObject on a directory returning "is not a regular file" (Nick Craig-Wood)
- Fix listing a file as a directory on hierarchical namespace accounts (Nick Craig-Wood)
- Fix sync at the root failing on soft deleted containers (Ludovic Bondon)
- Box
- Reuse multipart upload part buffers via the global pool (Nick Craig-Wood)
- Fix ~ and environment variables not being expanded in `box_config_file` (Nick Craig-Wood)
- Chunker
- Clean stale chunks after server-side overwrite (hahahahahayesyeseys)
- Compress
- Buffer small uploads in the global memory pool (Nick Craig-Wood)
- Stop buffering the compressibility check output in memory (Nick Craig-Wood)
- Drime
- Fix endless listing of directories with more than 20,000 entries (Maksym Tymoshyk)
- Drive
- Fetch metadata permissions concurrently (Loi Nguyen)
- Reuse resumable-upload chunk buffers via the global memory pool (SillyZir)
- Stop a fast-list error from hanging the sync (Sash)
- Dropbox
- Reuse upload chunk buffers via the global memory pool (Nick Craig-Wood)
- Filelu
- Reuse multipart upload buffers via the global pool and retry failed parts (Nick Craig-Wood)
- Encode multipart init paths (wjc)
- FTP
- Discard connections that fail with non-FTP errors (Daniel Bae)
- Fix ranged reads failing when the server answers 425 to an early close (Robert Kubiś)
- Hasher
- Keep cached hash when only the modtime changes (Aniket)
- Hidrive
- Stop copying every upload chunk a second time (Nick Craig-Wood)
- Buffer the start of each upload from the global memory pool (Nick Craig-Wood)
- Reuse chunked upload buffers via the global memory pool (Nick Craig-Wood)
- Buffer small file updates from the global memory pool (Nick Craig-Wood)
- Fix truncated file left behind when an upload hits `--max-transfer` (Nick Craig-Wood)
- HTTP
- Add metadata defaults for `--http-no-head` (Giridhar)
- Huaweidrive
- Build small uploads in pooled memory instead of a growing buffer (Nick Craig-Wood)
- Reuse resumable-upload chunk buffers via the global pool (Nick Craig-Wood)
- Fix creating, copying and renaming files with non-ASCII names (Nick Craig-Wood)
- Iclouddrive
- Mark the backend as case insensitive (berce)
- Imagekit
- Stop multipart writers after failed uploads (Florian Josef Altendorfer)
- Jottacloud
- Buffer MD5 pre-reads in the global memory pool (Nick Craig-Wood)
- Fix server-side copy onto a trashed file failing as corrupted (Nick Craig-Wood)
- Fix server-side copy onto a trashed file keeping the old name case (Nick Craig-Wood)
- Fix authentication with whitelabel services elkjop and elko (albertony)
- Fix traditional authentication failing with invalid_client (albertony)
- Linkbox
- Buffer the hashed 10 MiB file prefix in the global memory pool (Nick Craig-Wood)
- Mailru
- Buffer speedup hashing in the global memory pool and fix upload retries (Nick Craig-Wood)
- Onedrive
- OneDrive for Business and SharePoint users must upgrade to v1.76+ before April 1, 2027 (Nick Craig-Wood)
- Authenticate SharePoint downloads, uploads and copies after tempauth retirement (Nick Craig-Wood)
- Return an error if a ranged download doesn't return the requested range (Nick Craig-Wood)
- Reuse upload chunk buffers via the global memory pool (Nick Craig-Wood)
- Allow `--onedrive-upload-cutoff` up to the documented limit of 250 MiB (Can Arslan)
- Add configurable SharePoint tenant API version (Mattias Michaux)
- Make `--onedrive-delta` quicker when not listing from the drive root (Nick Craig-Wood)
- Keep the delta token when a ChangeNotify poll fails (xiaozou-wine)
- Show the size of directories in listings (GingerAdonis)
- Poll async copy status with the configured HTTP client (Soujas K)
- Opendrive
- Reuse upload chunk buffers via the global memory pool (Nick Craig-Wood)
- Oracleobjectstorage
- Add server-side object moves (tomaszni)
- Encode carriage returns and line feeds (tomaszni)
- Purge objects in batches (tomaszni)
- Retry 409 ConcurrentObjectUpdate errors (berce)
- Pcloud
- Stop multipart writers after failed uploads (Florian Josef Altendorfer)
- Pikpak
- Share upload chunk buffers with the global memory pool (Nick Craig-Wood)
- Use pooled memory for the gcid hash buffer and spool unknown-size uploads to disk (Nick Craig-Wood)
- Fix downloads that return an HTTP error page as the file (Cestercian)
- Stop multipart writers after failed uploads (Florian Josef Altendorfer)
- Fix uploads failing when the server is slow to complete them (Nick Craig-Wood)
- Putio
- Reuse upload chunk buffers via the global memory pool (Nick Craig-Wood)
- S3
- Avoid buffering Object Lock uploads when the source MD5 is known (Nick Craig-Wood)
- Add backend link command with signed response header overrides (eliotee)
- Add Fastly EU Central 1 region (Leon Brocard)
- Add the MD5 of multipart chunks to the debug log (interested.tortoise)
- Report the real storage class when an object needs restoring (Dirk Petersen)
- Speed up multi-thread uploads by checksumming chunks in 1 MiB reads (Roman Butor)
- Add IT-MIL region to Scaleway (Alessio Margelli)
- Add missing Huawei OBS regions (Enbiya Göral)
- Fix rmdir of the root directory with `--s3-directory-markers` (Nick Craig-Wood)
- Fix presigned URLs for GCS, Linode and Ceph (Leon Brocard)
- Seafile
- Fix operations on files whose names are not in Unicode NFC form (Fred)
- Stop multipart writers after failed uploads (Florian Josef Altendorfer)
- Fix double slash in the URL when getting a 2FA token (Nick Craig-Wood)
- SFTP
- Implement multi-thread uploads (Splainte)
- Fix data race when using `--sftp-ssh` (Dirk Petersen)
- Fix hash check for names starting with "-" with relative `--sftp-path-override` (Dirk Einecke)
- Fix sync deleting files with control characters in their names (nielash)
- Fix hashing files with a newline in their name on rsync.net (Nick Craig-Wood)
- Shade
- Stop copying each upload chunk into a second heap buffer (Nick Craig-Wood)
- Smb
- Add workstation field to smb backend (Kalin Stoyanov)
- Swift
- Add Infomaniak Swiss Backup provider (Louis)
- WebDAV
- Reject ranged responses which ignore the range (Morax)
- Retry Nextcloud chunk uploads individually (Stochastic)
- Fix Nextcloud public share access by sending X-Requested-With (drakeo338)
- Fix memory leak uploading to Infinite Scale (jxj)
- Yandex
- Add `app_folder` option to support `cloud_api:disk.app_folder` OAuth scope (Sanjay Kanth A)
- Zoho
- Stop multipart writers after failed uploads (Florian Josef Altendorfer)
## v1.75.2 - 2026-10-09
[See commits](https://github.com/rclone/rclone/compare/v1.75.1...v1.75.2)
docs/content/commands/rclone.md
+32 -9
View File
@@ -24,6 +24,7 @@ rclone [flags]
```
--alias-description string Description of the remote
--alias-remote string Remote or path to alias
--allow-https-downgrade Follow redirects from HTTPS to HTTP (insecure)
--archive-description string Description of the remote
--archive-remote string Remote to wrap to read archives from
--ask-password Allow prompt for password for encrypted configuration (default true)
@@ -50,8 +51,10 @@ rclone [flags]
--azureblob-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
--azureblob-endpoint string Endpoint for the service
--azureblob-env-auth Read credentials from runtime (environment variables, CLI or MSI)
--azureblob-hns Tristate Set if the storage account has a hierarchical namespace (default unset)
--azureblob-key string Storage Account Shared Key
--azureblob-list-chunk int Size of blob list (default 5000)
--azureblob-list-parallelism int Number of parallel shards to list a directory with
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any
@@ -64,6 +67,7 @@ rclone [flags]
--azureblob-tenant string ID of the service principal's tenant. Also called its directory ID
--azureblob-upload-concurrency int Concurrency for multipart uploads (default 16)
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256 MiB) (deprecated)
--azureblob-use-arrow-list Use the Apache Arrow listing format
--azureblob-use-az Use Azure CLI tool az for authentication
--azureblob-use-copy-blob Whether to use the Copy Blob API when copying to the same storage account (default true)
--azureblob-use-emulator Uses local storage emulator if provided as 'true'
@@ -224,6 +228,12 @@ rclone [flags]
--doi-doi string The DOI or the doi.org URL
--doi-doi-resolver-api-url string The URL of the DOI resolver API to use
--doi-provider string DOI provider
--dosya-api-key string Your API Key, get it from https://dosya.dev/profile under "API keys"
--dosya-api-url string API base URL (default "https://api.dosya.dev")
--dosya-description string Description of the remote
--dosya-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
--dosya-upload-concurrency int Concurrency for multipart uploads (default 4)
--dosya-workspace-id string Your workspace ID
--drime-access-token string API Access token
--drime-chunk-size SizeSuffix Chunk size to use for uploading (default 5Mi)
--drime-description string Description of the remote
@@ -470,6 +480,8 @@ rclone [flags]
--http-headers CommaSepList Set HTTP headers for all transactions
--http-no-escape Do not escape URL metacharacters in path names
--http-no-head Don't use HEAD requests
--http-no-head-size SizeSuffix Set the reported file size for --http-no-head (default off)
--http-no-head-time Time Set a default modification time for --http-no-head (default off)
--http-no-slash Set this if the site doesn't end directories with /
--http-proxy string HTTP proxy URL
--http-url string URL of HTTP host to connect to
@@ -543,6 +555,17 @@ rclone [flags]
--jottacloud-token-url string Token server url
--jottacloud-trashed-only Only show files that are in the trash
--jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's (default 10Mi)
--kdrive-auth-url string Auth server URL
--kdrive-chunk-size SizeSuffix Files above this size are uploaded in chunks of this size (default 20Mi)
--kdrive-client-id string OAuth Client Id
--kdrive-description string Description of the remote
--kdrive-drive-id string ID of the kDrive to use
--kdrive-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
--kdrive-endpoint string The API endpoint to use
--kdrive-root-folder-id string The folder to use as root
--kdrive-token string OAuth Access Token as a JSON blob
--kdrive-token-url string Token server url
--kdrive-upload-concurrency int Concurrency for chunked uploads (default 4)
--koofr-description string Description of the remote
--koofr-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--koofr-endpoint string The Koofr API endpoint to use
@@ -695,6 +718,7 @@ rclone [flags]
--onedrive-root-folder-id string ID of the root folder
--onedrive-server-side-across-configs Deprecated: use --server-side-across-configs instead
--onedrive-tenant string ID of the service principal's tenant. Also called its directory ID
--onedrive-tenant-api-version string The SharePoint API version to use with tenant_url (default "v2.0")
--onedrive-tenant-url string The tenant URL for non-admin OneDrive access
--onedrive-token string OAuth Access Token as a JSON blob
--onedrive-token-url string Token server url
@@ -709,7 +733,7 @@ rclone [flags]
--oos-decompress If set this will decompress gzip encoded objects
--oos-description string Description of the remote
--oos-disable-checksum Don't store MD5 checksum with object metadata
--oos-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--oos-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--oos-endpoint string Endpoint for Object storage API
--oos-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts for manual recovery
--oos-max-upload-parts int Maximum number of parts in a multipart upload (default 10000)
@@ -840,11 +864,6 @@ rclone [flags]
--rc-template string User-specified template
--rc-user string User name for authentication
--rc-user-from-header string User name from a defined HTTP header
--rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
--rc-web-gui Launch WebGUI on localhost
--rc-web-gui-force-update Force update to latest version of web gui
--rc-web-gui-no-open-browser Don't open the browser automatically
--rc-web-gui-update Check and update to latest version of web gui
--refresh-times Refresh the modtime of remote files
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep Duration Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
@@ -945,7 +964,7 @@ rclone [flags]
--sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-concurrent-writes If set don't use concurrent writes
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available
--sftp-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,Dot)
--sftp-encoding Encoding The encoding for the backend (default Slash,Dot)
--sftp-hashes CommaSepList Comma separated list of supported checksum types
--sftp-host string SSH host to connect to
--sftp-host-key-algorithms SpaceSepList Space separated list of host key algorithms, ordered by preference
@@ -960,6 +979,7 @@ rclone [flags]
--sftp-known-hosts-file string Optional path to known_hosts file
--sftp-macs SpaceSepList Space separated list of MACs (message authentication code) algorithms, ordered by preference
--sftp-md5sum-command string The command used to read MD5 hashes
--sftp-multithread-upload Set this to use multi-thread (multi-connection) uploads
--sftp-pass string SSH password, leave blank to use ssh-agent (obscured)
--sftp-path-override string Override path used by SSH shell commands
--sftp-pin-host-key Pin the server host key on first connection (Trust On First Use)
@@ -1023,7 +1043,8 @@ rclone [flags]
--smb-port int SMB port number (default 445)
--smb-spn string Service principal name
--smb-use-kerberos Use Kerberos authentication
--smb-user string SMB username (default "$USER")
--smb-user string SMB username
--smb-workstation string Workstation name sent in NTLM authentication
--stats Duration Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable) (default 1m0s)
--stats-file-name-length int Max file name length in stats (0 for no limit) (default 45)
--stats-log-level LogLevel Log level to show --stats output DEBUG|INFO|NOTICE|ERROR (default INFO)
@@ -1107,7 +1128,7 @@ rclone [flags]
--use-json-log Use json log format
--use-mmap Use mmap allocator (see docs)
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string (default "rclone/v1.75.0")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.76.0")
-v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number
--webdav-auth-redirect Preserve authentication on redirect
@@ -1125,6 +1146,7 @@ rclone [flags]
--webdav-url string URL of http host to connect to
--webdav-user string User name
--webdav-vendor string Name of the WebDAV site/service/software you are using
--yandex-app-folder Use the application folder as the root
--yandex-auth-url string Auth server URL
--yandex-client-credentials Use client credentials OAuth flow
--yandex-client-id string OAuth Client Id
@@ -1183,6 +1205,7 @@ rclone [flags]
* [rclone gitannex](/commands/rclone_gitannex/) - Speaks with git-annex over stdin/stdout.
* [rclone gui](/commands/rclone_gui/) - Open the web based GUI.
* [rclone hashsum](/commands/rclone_hashsum/) - Produces a hashsum file for all the objects in the path.
* [rclone index](/commands/rclone_index/) - Write static directory listings into a remote.
* [rclone link](/commands/rclone_link/) - Generate public link to file/folder.
* [rclone listremotes](/commands/rclone_listremotes/) - List all the remotes in the config file and defined in environment variables.
* [rclone ls](/commands/rclone_ls/) - List the objects in the path with size and path.
docs/content/commands/rclone_bisync.md
+1
View File
@@ -52,6 +52,7 @@ rclone bisync remote1:path1 remote2:path2 [flags]
--force Bypass --max-delete safety check and run the sync. Consider using with --verbose
-h, --help help for bisync
--ignore-listing-checksum Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks)
--max-delete-renames-aware Exclude tracked renames from the --max-delete safety check (requires --track-renames)
--max-lock Duration Consider lock files older than this to be expired (default: 0 (never expire)) (minimum: 2m) (default 0s)
--no-cleanup Retain working files (useful for troubleshooting and testing).
--no-slow-hash Ignore listing checksums only on backends where they are slow
docs/content/commands/rclone_completion.md
+2
View File
@@ -14,6 +14,8 @@ Output completion script for a given shell.
Generates a shell completion script for rclone.
Run with `--help` to list the supported shells.
Enum-valued flags such as `--log-level` and `--vfs-cache-mode`
also complete their supported values.
## Options
docs/content/commands/rclone_convmv.md
+2 -2
View File
@@ -231,12 +231,12 @@ rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,command=e
```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
// Output: stories/The Quick Brown Fox!-20260731
// Output: stories/The Quick Brown Fox!-20261106
```
```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
// Output: stories/The Quick Brown Fox!-2026-07-31 0340PM
// Output: stories/The Quick Brown Fox!-2026-11-06 0220PM
```
```console
docs/content/commands/rclone_deletefile.md
+24 -2
View File
@@ -10,9 +10,11 @@ Remove a single file from remote.
## Synopsis
Remove a single file from remote. Unlike `delete` it cannot be used to
Remove a single file from remote. Unlike `delete` it cannot be used to
remove a directory and it doesn't obey include/exclude filters - if the
specified file exists, it will always be removed.
specified file exists, it will always be removed from its original location.
The `--backup-dir` and `--suffix` flags are honored when set.
```
rclone deletefile remote:path [flags]
@@ -27,6 +29,26 @@ rclone deletefile remote:path [flags]
Options shared with other commands are described next.
See the [global flags page](/flags/) for global options not listed here.
### Sync Options
Flags used for sync commands
```text
--backup-dir string Make backups into hierarchy based in DIR
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--fix-case Force rename of case insensitive dest to match source
--ignore-errors Delete even if there are I/O errors
--list-cutoff int To save memory, sort directory listings on disk above this threshold (default 1000000)
--max-delete int When synchronizing, limit the number of deletes (default -1)
--max-delete-size SizeSuffix When synchronizing, limit the total size of deletes (default off)
--suffix string Suffix to add to changed files
--suffix-keep-extension Preserve the extension when using --suffix
--track-renames When synchronizing, track file renames and do a server-side move if possible
--track-renames-strategy string Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash")
```
### Important Options
Important flags useful for most commands
docs/content/commands/rclone_gui.md
-5
View File
@@ -101,11 +101,6 @@ Flags to control the Remote Control API
--rc-template string User-specified template
--rc-user string User name for authentication
--rc-user-from-header string User name from a defined HTTP header
--rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
--rc-web-gui Launch WebGUI on localhost
--rc-web-gui-force-update Force update to latest version of web gui
--rc-web-gui-no-open-browser Don't open the browser automatically
--rc-web-gui-update Check and update to latest version of web gui
```
## See Also
docs/content/commands/rclone_hashsum.md
+1
View File
@@ -41,6 +41,7 @@ Supported hashes are:
- blake3
- xxh3
- xxh128
- crc32c
```
Then
docs/content/commands/rclone_index.md
+331
View File
@@ -0,0 +1,331 @@
---
title: "rclone index"
description: "Write static directory listings into a remote."
versionIntroduced: v1.76
# autogenerated - DO NOT EDIT, instead edit the source code in cmd/index/ and as part of making a release run "make commanddocs"
---
# rclone index
Write static directory listings into a remote.
## Synopsis
Write a directory listing, `index.html` by default, into every
directory of `remote:path` so that it can be browsed when it is served
as a static website, for example from an S3 website endpoint, a bucket
behind a CDN, or a plain web server.
This works like `rclone sync`: each run makes the minimum changes needed
to bring the listings into line with the files. Listings which haven't
changed aren't rewritten, so a run with no changes makes no uploads. The
listings never show the listing files themselves.
**Note** that `rclone index` treats every file with an output name
(`index.html` by default) in the directories it indexes as its own, and
will overwrite it or delete it without checking where it came from. To
protect a hand-written index, exclude its directory with
`--index-exclude`.
## Outputs
`--output NAME=FORMAT` says what to write in each directory and can be
repeated. NAME is the file name, and FORMAT is one of the built-in
formats below or the path of a Go template file.
| Format | MIME type | Contents |
|:-------|:----------|:---------|
| `html` | `text/html; charset=utf-8` | The HTML listing from the `rclone serve http` template |
| `json` | `application/json` | JSON in the format of `rclone lsjson` for the directory |
| `caddy` | `application/json` | JSON in the format of Caddy's `file_server browse` |
The default is `--output index.html=html`. The built-in formats are
templates, so they can be printed with `--print-template FORMAT`, copied
and changed. Outputs whose name ends in `.html` or `.htm` are rendered
with Go's `html/template` and everything else with `text/template`, which
has a `json` function for safe encoding. The template data is that of
`rclone serve http` with `.Static` set (see
[rclone serve http](/commands/rclone_serve_http/#template)). Outputs are
uploaded with the MIME type of the built-in format, or that of the
output name's extension for template outputs, since static hosts serve
the stored type.
## Filters
The normal filters (`--include`, `--exclude`, `--filter-from`, `--max-age`, ...)
control what appears in the listings. An excluded file isn't listed, and
an excluded directory isn't listed or indexed.
The `--index-include`, `--index-exclude`, `--index-filter` flags and their
`-from` variants control which directories get listings. They take the same
patterns as the normal filters, applied to directory paths, and
`--index-include` implies excluding all other directories. A directory
excluded here still appears in its parent's listing, but nothing is written
or deleted inside it, so it can have its own hand-made index.
`--index-max-depth` limits how deep listings are written.
## Modification times
Listings show the modification time of each file. When using the s3,
oracleobjectstorage and swift backends rclone stores the modification time as
metadata which isn't returned by the bucket listing, so **every object needs
a HEAD request**, which can mean thousands of transactions per run. Avoid
this with one of:
- `--use-server-modtime` uses the time the object was uploaded instead,
which is what the web server sends as `Last-Modified`. Recommended for s3
unless the original modification times matter.
- `--no-modtime` shows no times at all, which is the cheapest option.
`--dir-time` sets the time shown for directories:
- `newest` (default): the newest modification time of anything below the
directory. Note that this means one new file changes the listing of
every directory above it.
- `dir`: the directory's own modification time. On most file systems
writing a listing into a directory changes its time, so this can take
more than one run to settle.
- `none`: no time.
The listing files are given the newest modification time below their
directory, so the `Last-Modified` header of a listing is meaningful.
## Links
Directory links are `dir/` by default, which works on hosts that serve
index documents: S3 website endpoints, GCS and Azure static websites,
nginx and Apache, and R2 custom domains with a URL rewrite rule. For
hosts that don't (plain S3 REST URLs, B2 friendly URLs) use
`--link-index` to make directory links `dir/index.html` instead.
## Partial runs
After a sync which changed a known set of files there is no need to
walk the whole remote. Tell `rclone index` what changed and it re-indexes
only the directories that could be affected:
rclone index r2:bucket --changed v1.76.0/ --changed version.txt
rclone index r2:bucket --changed-from changes.txt
rclone sync ./public r2:bucket --exclude index.html --combined - | rclone index r2:bucket --changed-combined -
- `--changed PATH` names a changed file or directory relative to
`remote:path` and can be repeated. A directory, given with a trailing
slash or found to be one, is re-indexed completely.
- `--changed-from FILE` reads one path per line exactly as written.
- `--changed-combined FILE` reads the `--combined` report of
`rclone sync`: lines starting with `+`, `-`, `*` and `!` are changes and
`=` lines are ignored.
- Both files accept `-` for standard input and all three flags can be
combined.
A changed file re-indexes its directory and every directory above it,
since their times and their listings can change too, and on bucket
based storage a directory can cease to exist. Each of those is one
directory listing. With `--dir-time newest` the directories above also
need the times of their unchanged subdirectories, which are read from
the modification times of those subdirectories' listings at the cost
of one small request each. With `--use-server-modtime` that is the time
the listing was written rather than the time of the newest file, so
partial runs can drift by the indexing delay until the next full run.
`--dir-time dir` and `none` need no such lookups.
Directories which are neither named nor above a named path are left
alone, as are directories deeper than `--max-depth`. An empty change
list does nothing, and naming the root does a full run. There is no
automatic cutover to a full run since that depends on the size of the
whole remote: a partial run costs one listing per affected directory,
a full run one request per thousand objects on S3-like storage or one
per directory elsewhere. The number
of directories a partial run will list is logged with `-v`, and
`--changed-max-dirs` makes it fall back to a full run above that
number. A good pattern is a partial run after each upload and a
scheduled full run to catch anything missed.
## Site icon
The listings carry no icon or branding of their own. To give a site an
icon, upload a `favicon.ico` to the root of the remote, which browsers
fetch from there for every page. Exclude it from the listings with
`--exclude /favicon.ico` so it stays on the site but isn't shown.
## Deleting listings
On backends which can't have empty directories (S3, B2 and other
bucket-based storage) a directory only exists while it has files in it.
Once the last file in a directory has been removed its listing is
deleted, since the directory only existed because of it. Backends which
can have empty directories keep a listing for each empty directory.
## Using with sync
A sync from a local build deletes files on the destination which aren't
in the source, which includes the listings. Either index the local build
before syncing:
rclone index ./public
rclone sync ./public r2:bucket
or exclude the listings from the sync so it neither uploads nor deletes
them, then index the remote:
rclone sync ./public r2:bucket --exclude index.html
rclone index r2:bucket
## Examples
An S3 website bucket, using the upload times so no object needs a HEAD
request:
rclone index s3:my-site --use-server-modtime
HTML and rclone JSON listings, setting `Cache-Control` on them:
rclone index s3:bucket --output index.html=html --output index.json=json --header-upload "Cache-Control: max-age=300"
Only list release files, and don't write listings under `/private`:
rclone index r2:bucket --include "*.{zip,deb,rpm}" --index-exclude "/private/**"
A B2 bucket served from its friendly URLs:
rclone index b2:bucket --link-index
See what would change:
rclone index r2:bucket --dry-run -v
Listings which haven't changed aren't normally rewritten, so changing
the headers or metadata set on them needs `--index-rewrite`, which
writes every listing a run would render whether it changed or not:
rclone index r2:bucket --index-rewrite --header-upload "Cache-Control: max-age=3600"
```
rclone index remote:path [flags]
```
## Options
```
--changed stringArray Only re-index directories affected by this changed path (directories end in /)
--changed-combined stringArray Read changed paths from a sync --combined report (use - to read from stdin)
--changed-from stringArray Read changed paths from file, one per line (use - to read from stdin)
--changed-max-dirs int Do a full index if a partial one would list more than this many directories (0 for no limit)
--dir-time newest|dir|none How to work out the time shown for a directory (default newest)
-h, --help help for index
--index-exclude stringArray Don't write listings in directories matching pattern
--index-exclude-from stringArray Read directory exclude patterns from file (use - to read from stdin)
--index-filter stringArray Add a rule for which directories get listings
--index-filter-from stringArray Read directory rules from a file (use - to read from stdin)
--index-include stringArray Only write listings in directories matching pattern
--index-include-from stringArray Read directory include patterns from file (use - to read from stdin)
--index-max-depth int Only write listings this many directories deep (-1 for no limit) (default -1)
--index-rewrite Write every listing even if it is unchanged
--link-index Link to dir/NAME rather than dir/ for hosts without index documents
--no-modtime Don't show modification times in listings
--output stringArray Listing to write in each directory as NAME=FORMAT (default [index.html=html])
--print-template string Print the built-in template for FORMAT and exit
```
Options shared with other commands are described next.
See the [global flags page](/flags/) for global options not listed here.
### Copy Options
Flags for anything which can copy a file
```text
--check-first Do all the checks before starting transfers
-c, --checksum Check for changes with size & checksum (if available, or fallback to size only)
--compare-dest stringArray Include additional server-side paths during comparison
--copy-dest stringArray Implies --compare-dest but also copies files from paths into destination
--cutoff-mode HARD|SOFT|CAUTIOUS Mode to stop transfers when reaching the max transfer limit HARD|SOFT|CAUTIOUS (default HARD)
--ignore-case-sync Ignore case when synchronizing
--ignore-checksum Skip post copy check of checksums
--ignore-existing Skip all files that exist on destination
--ignore-size Ignore size when skipping use modtime or checksum
-I, --ignore-times Don't skip items that match size and time - transfer all unconditionally
--immutable Do not modify files, fail if existing files have been modified
--inplace Download directly to destination file instead of atomic download to temp/rename
-l, --links Translate symlinks to/from regular files with a '.rclonelink' extension
--max-backlog int Maximum number of objects in sync or check backlog (default 10000)
--max-duration Duration Maximum duration rclone will transfer data for (default 0s)
--max-transfer SizeSuffix Maximum size of data to transfer (default off)
-M, --metadata If set, preserve metadata when copying objects
--modify-window Duration Max time diff to be considered the same (default 1ns)
--multi-thread-chunk-size SizeSuffix Chunk size for multi-thread downloads / uploads, if not set by filesystem (default 64Mi)
--multi-thread-cutoff SizeSuffix Use multi-thread downloads for files above this size (default 256Mi)
--multi-thread-streams int Number of streams to use for multi-thread downloads (default 4)
--multi-thread-write-buffer-size SizeSuffix In memory buffer size for writing when in multi-thread mode (default 128Ki)
--name-transform stringArray Transform paths during the copy process
--no-check-dest Don't check the destination, copy regardless
--no-traverse Don't traverse destination file system on copy
--no-update-dir-modtime Don't update directory modification times
--no-update-modtime Don't update destination modtime if files identical
--order-by string Instructions on how to order the transfers, e.g. 'size,descending'
--partial-suffix string Add partial-suffix to temporary file name when --inplace is not used (default ".partial")
--refresh-times Refresh the modtime of remote files
--server-side-across-configs Allow server-side operations (e.g. copy) to work across different configs
--size-only Skip based on size only, not modtime or checksum
--streaming-upload-cutoff SizeSuffix Cutoff for switching to chunked upload if file size is unknown, upload starts after reaching cutoff or when file ends (default 100Ki)
-u, --update Skip files that are newer on the destination
```
### Important Options
Important flags useful for most commands
```text
-n, --dry-run Do a trial run with no permanent changes
-i, --interactive Enable interactive mode
-v, --verbose count Print lots more stuff (repeat for more)
```
### Filter Options
Flags for filtering directory listings
```text
--delete-excluded Delete files on dest excluded from sync
--exclude stringArray Exclude files matching pattern
--exclude-from stringArray Read file exclude patterns from file (use - to read from stdin)
--exclude-if-present stringArray Exclude directories if filename is present
--files-from stringArray Read list of source-file names from file (use - to read from stdin)
--files-from-raw stringArray Read list of source-file names from file without any processing of lines (use - to read from stdin)
--files-from0 stringArray Read list of source-file names from file using NUL as separator (use - to read from stdin)
-f, --filter stringArray Add a file filtering rule
--filter-from stringArray Read file filtering patterns from a file (use - to read from stdin)
--hash-filter string Partition filenames by hash k/n or randomly @/n
--ignore-case Ignore case in filters (case insensitive)
--include stringArray Include files matching pattern
--include-from stringArray Read file include patterns from file (use - to read from stdin)
--max-age Duration Only transfer files younger than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--max-depth int If set limits the recursion depth to this (default -1)
--max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
--metadata-exclude stringArray Exclude metadatas matching pattern
--metadata-exclude-from stringArray Read metadata exclude patterns from file (use - to read from stdin)
--metadata-filter stringArray Add a metadata filtering rule
--metadata-filter-from stringArray Read metadata filtering patterns from a file (use - to read from stdin)
--metadata-include stringArray Include metadatas matching pattern
--metadata-include-from stringArray Read metadata include patterns from file (use - to read from stdin)
--min-age Duration Only transfer files older than this in s or suffix ms|s|m|h|d|w|M|y (default off)
--min-size SizeSuffix Only transfer files bigger than this in KiB or suffix B|K|M|G|T|P (default off)
```
### Listing Options
Flags for listing directories
```text
--default-time Time Time to show if modtime is unknown for files and directories (default 2000-01-01T00:00:00Z)
--fast-list Use recursive list if available; uses more memory but fewer transactions
```
## See Also
<!-- markdownlint-capture -->
<!-- markdownlint-disable ul-style line-length -->
* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
<!-- markdownlint-restore -->
docs/content/commands/rclone_ls.md
+4
View File
@@ -41,6 +41,10 @@ Note that `ls` and `lsl` recurse by default - use `--max-depth 1` to stop the re
The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default -
use `-R` to make them recurse.
On the bucket-based remotes (e.g. s3, gcs, b2) a directory is a key prefix
rather than an object, so a listing without `-R` shows each prefix as one
directory entry and not the objects stored under it.
List commands prefer a recursive method that uses more memory but fewer
transactions by default. Use `--disable ListR` to suppress the behavior.
See [`--fast-list`](/docs/#fast-list) for more details.
docs/content/commands/rclone_lsd.md
+5 -1
View File
@@ -13,7 +13,7 @@ Lists the directories in the source path to standard output. Does not
recurse by default. Use the `-R` flag to recurse.
This command lists the total size of the directory (if known, -1 if
not), the modification time (if known, the current time if not), the
not), the modification time (if known, the `--default-time` if not), the
number of objects in the directory (if known, -1 if not) and the name
of the directory, E.g.
@@ -53,6 +53,10 @@ Note that `ls` and `lsl` recurse by default - use `--max-depth 1` to stop the re
The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default -
use `-R` to make them recurse.
On the bucket-based remotes (e.g. s3, gcs, b2) a directory is a key prefix
rather than an object, so a listing without `-R` shows each prefix as one
directory entry and not the objects stored under it.
List commands prefer a recursive method that uses more memory but fewer
transactions by default. Use `--disable ListR` to suppress the behavior.
See [`--fast-list`](/docs/#fast-list) for more details.
docs/content/commands/rclone_lsf.md
+4
View File
@@ -164,6 +164,10 @@ Note that `ls` and `lsl` recurse by default - use `--max-depth 1` to stop the re
The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default -
use `-R` to make them recurse.
On the bucket-based remotes (e.g. s3, gcs, b2) a directory is a key prefix
rather than an object, so a listing without `-R` shows each prefix as one
directory entry and not the objects stored under it.
List commands prefer a recursive method that uses more memory but fewer
transactions by default. Use `--disable ListR` to suppress the behavior.
See [`--fast-list`](/docs/#fast-list) for more details.
docs/content/commands/rclone_lsjson.md
+4
View File
@@ -111,6 +111,10 @@ Note that `ls` and `lsl` recurse by default - use `--max-depth 1` to stop the re
The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default -
use `-R` to make them recurse.
On the bucket-based remotes (e.g. s3, gcs, b2) a directory is a key prefix
rather than an object, so a listing without `-R` shows each prefix as one
directory entry and not the objects stored under it.
List commands prefer a recursive method that uses more memory but fewer
transactions by default. Use `--disable ListR` to suppress the behavior.
See [`--fast-list`](/docs/#fast-list) for more details.
docs/content/commands/rclone_lsl.md
+4
View File
@@ -42,6 +42,10 @@ Note that `ls` and `lsl` recurse by default - use `--max-depth 1` to stop the re
The other list commands `lsd`,`lsf`,`lsjson` do not recurse by default -
use `-R` to make them recurse.
On the bucket-based remotes (e.g. s3, gcs, b2) a directory is a key prefix
rather than an object, so a listing without `-R` shows each prefix as one
directory entry and not the objects stored under it.
List commands prefer a recursive method that uses more memory but fewer
transactions by default. Use `--disable ListR` to suppress the behavior.
See [`--fast-list`](/docs/#fast-list) for more details.
docs/content/commands/rclone_mount.md
+9 -3
View File
@@ -292,6 +292,11 @@ for all `mount` and `serve` commands on macOS. For details, see [vfs-case-sensit
### NFS mount
For macOS (and other platforms where this path is supported), prefer the dedicated
[rclone nfsmount](/commands/rclone_nfsmount/) command. It starts the NFS server and
performs the mount for you. `rclone mount` itself still uses FUSE (macFUSE/FUSE-T)
and does not switch to NFS via a flag.
This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/)
command and mounts it to the specified mountpoint. If you run this in background
mode using |--daemon|, you will need to send SIGTERM signal to the rclone process
@@ -629,7 +634,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -684,7 +689,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -838,7 +844,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
docs/content/commands/rclone_nfsmount.md
+9 -3
View File
@@ -293,6 +293,11 @@ for all `mount` and `serve` commands on macOS. For details, see [vfs-case-sensit
### NFS mount
For macOS (and other platforms where this path is supported), prefer the dedicated
[rclone nfsmount](/commands/rclone_nfsmount/) command. It starts the NFS server and
performs the mount for you. `rclone mount` itself still uses FUSE (macFUSE/FUSE-T)
and does not switch to NFS via a flag.
This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/)
command and mounts it to the specified mountpoint. If you run this in background
mode using |--daemon|, you will need to send SIGTERM signal to the rclone process
@@ -630,7 +635,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -685,7 +690,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -839,7 +845,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
docs/content/commands/rclone_rcd.md
+27 -5
View File
@@ -47,6 +47,13 @@ accept in the HTTP header.
will overriding existing values. The flag may be repeated to add multiple
headers. Use the format `Header-Name: value`.
`--rc-allow-origin` enables cross-origin (CORS) requests from browsers
by setting the `Access-Control-Allow-Origin` header on every response to the
origin given, e.g. `--rc-allow-origin https://app.example.com`. Browser
CORS preflight OPTIONS requests are answered by rclone without needing
authentication as browsers never send credentials with them. Setting this
to `*` allows any origin which is a security risk on an authenticated server.
`--rc-baseurl` controls the URL prefix that rclone serves from. By default
rclone will serve from the root. If you used `--rc-baseurl "/rclone"` then
rclone would serve from a URL starting with "/rclone/". This is
@@ -116,6 +123,13 @@ to be used within the template to server pages:
| .Sort | | The current sort used. This is changeable via '?sort=' parameter. Possible values: namedirfirst, name, size, time (default namedirfirst). |
| .Order | | The current ordering used. This is changeable via '?order=' parameter. Possible values: asc, desc (default asc). |
| .Query | | Currently unused. |
| .Path | | The path of the directory from the root with leading and trailing slashes, e.g. '/dir/'. |
| .IsRoot | | Boolean for if this is the root directory. |
| .UpLink | | The link to the parent directory. |
| .Static | | Boolean for if the listing is being written as a static page by rclone index. |
| .NumDirs | | The number of directories in the listing. |
| .NumFiles | | The number of files in the listing. |
| .TotalSize | | The total size in bytes of the files in the listing. |
| .Breadcrumb | | Allows for creating a relative navigation. |
| | .Link | The link of the Text relative to the root. |
| | .Text | The Name of the directory. |
@@ -125,6 +139,7 @@ to be used within the template to server pages:
| | .IsDir | Boolean for if an entry is a directory or not. |
| | .Size | Size in bytes of the entry. |
| | .ModTime | The UTC timestamp of an entry. |
| | .MimeType | The MIME type of an entry guessed from its name, or 'inode/directory'. |
The server also makes the following functions available so that they can be used
within the template. These functions help extend the options for dynamic
@@ -165,8 +180,20 @@ htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser
```
Use the same hash format and cost for every entry in the file. If the
entries differ, the time taken to reject a wrong password can reveal
which usernames exist.
The password file can be updated while rclone is running.
When authentication is enabled every request must be authenticated,
including HTTP OPTIONS requests, as the answer to an OPTIONS request can
reveal whether a path exists. Health checks or monitoring which use
unauthenticated OPTIONS requests will need to send credentials. The
exception is browser CORS preflight requests when `--rc-allow-origin`
is set, which are answered without authentication as browsers never send
credentials with them.
Use `--rc-realm` to set the authentication realm.
Use `--rc-salt` to change the password hashing salt from the default.
@@ -215,11 +242,6 @@ Flags to control the Remote Control API
--rc-template string User-specified template
--rc-user string User name for authentication
--rc-user-from-header string User name from a defined HTTP header
--rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
--rc-web-gui Launch WebGUI on localhost
--rc-web-gui-force-update Force update to latest version of web gui
--rc-web-gui-no-open-browser Don't open the browser automatically
--rc-web-gui-update Check and update to latest version of web gui
```
## See Also
docs/content/commands/rclone_serve_dlna.md
+4 -3
View File
@@ -100,7 +100,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -155,7 +155,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -309,7 +310,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
docs/content/commands/rclone_serve_docker.md
+4 -3
View File
@@ -162,7 +162,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -217,7 +217,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -371,7 +372,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
docs/content/commands/rclone_serve_ftp.md
+57 -11
View File
@@ -93,7 +93,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -148,7 +148,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -302,7 +303,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
@@ -545,9 +546,10 @@ This config generated must have this extra parameter
- `_root` - root to use for the backend
And it may have this parameter
And it may have these parameters
- `_obscure` - comma separated strings for parameters to obscure
- `_secret_access_key` - the secret for S3 access key auth (see below)
If password authentication was used by the client, input to the proxy
process (on STDIN) would look similar to this:
@@ -555,7 +557,8 @@ process (on STDIN) would look similar to this:
```json
{
"user": "me",
"pass": "mypassword"
"pass": "mypassword",
"client_ip": "192.168.1.1"
}
```
@@ -565,10 +568,44 @@ proxy process (on STDIN) would look similar to this:
```json
{
"user": "me",
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
"client_ip": "192.168.1.1"
}
```
If the client authenticated with an S3 access key (`rclone serve s3`),
the client never sends its secret, only a signature made with it, so
the input contains just the access key ID as the `user` with no `pass`
or `public_key`:
```json
{
"user": "AKIAIOSFODNN7EXAMPLE",
"client_ip": "192.168.1.1"
}
```
In this case the program must look up the secret access key for that
access key ID and return it in the `_secret_access_key` field of the
output. Rclone then uses that secret to verify the signature on the
request, refusing the request if it does not match. This means the
proxy program is the source of truth for both the credentials and the
backend they map to. If the program does not return
`_secret_access_key` or returns it empty the request is refused.
The program's answer for an access key ID is cached (see below) but
is checked with the program again after 5 minutes even if the access
key ID is in constant use, so revoking an access key ID in the
program takes effect within 5 minutes. A rotated secret takes effect
on the first request signed with it.
The `client_ip` key holds the IP address the client connected from,
without a port number. It can be used to restrict logins to certain
networks, or to log authentication attempts centrally. It is omitted if
the client has no IP address, for example when connecting over a unix
socket. Note that if rclone is behind a reverse proxy this will be the
address of the reverse proxy and not the original client.
And as an example return this on STDOUT
```json
@@ -594,11 +631,20 @@ to make proxy to many different sftp backends, you could make the
in the output and the user to `user`. For security you'd probably want
to restrict the `host` to a limited list.
An internal cache of backends is keyed on the `user` and a hash of the
`pass` or `public_key`. This means that if a user's password or
public-key changes, or the proxy returns different config parameters
(eg a rotated `api_key`), a fresh backend will be created on the next
request rather than the cached one being reused.
An internal cache of backends is keyed on the `user`, the kind of
credential presented, a hash of the `pass` or `public_key`, and the
`client_ip`. This means that if a user's password or public-key
changes, the client connects from a new IP address, or the proxy
returns different config parameters (eg a rotated `api_key`), a fresh
backend will be created on the next request rather than the cached one
being reused. A backend created for a public key login is never
reused for a password login, so a public key cannot be sent as a
password to skip the program.
A cached answer is checked with the program again after 5 minutes even
if the user is in constant use, so revoking a user in the program
takes effect within 5 minutes. For `serve sftp` this applies to new
logins - an existing connection is not closed.
This can be used to build general purpose proxies to any kind of
backend that rclone supports.
docs/content/commands/rclone_serve_http.md
+84 -11
View File
@@ -49,6 +49,13 @@ accept in the HTTP header.
will overriding existing values. The flag may be repeated to add multiple
headers. Use the format `Header-Name: value`.
`--allow-origin` enables cross-origin (CORS) requests from browsers
by setting the `Access-Control-Allow-Origin` header on every response to the
origin given, e.g. `--allow-origin https://app.example.com`. Browser
CORS preflight OPTIONS requests are answered by rclone without needing
authentication as browsers never send credentials with them. Setting this
to `*` allows any origin which is a security risk on an authenticated server.
`--baseurl` controls the URL prefix that rclone serves from. By default
rclone will serve from the root. If you used `--baseurl "/rclone"` then
rclone would serve from a URL starting with "/rclone/". This is
@@ -118,6 +125,13 @@ to be used within the template to server pages:
| .Sort | | The current sort used. This is changeable via '?sort=' parameter. Possible values: namedirfirst, name, size, time (default namedirfirst). |
| .Order | | The current ordering used. This is changeable via '?order=' parameter. Possible values: asc, desc (default asc). |
| .Query | | Currently unused. |
| .Path | | The path of the directory from the root with leading and trailing slashes, e.g. '/dir/'. |
| .IsRoot | | Boolean for if this is the root directory. |
| .UpLink | | The link to the parent directory. |
| .Static | | Boolean for if the listing is being written as a static page by rclone index. |
| .NumDirs | | The number of directories in the listing. |
| .NumFiles | | The number of files in the listing. |
| .TotalSize | | The total size in bytes of the files in the listing. |
| .Breadcrumb | | Allows for creating a relative navigation. |
| | .Link | The link of the Text relative to the root. |
| | .Text | The Name of the directory. |
@@ -127,6 +141,7 @@ to be used within the template to server pages:
| | .IsDir | Boolean for if an entry is a directory or not. |
| | .Size | Size in bytes of the entry. |
| | .ModTime | The UTC timestamp of an entry. |
| | .MimeType | The MIME type of an entry guessed from its name, or 'inode/directory'. |
The server also makes the following functions available so that they can be used
within the template. These functions help extend the options for dynamic
@@ -167,8 +182,20 @@ htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser
```
Use the same hash format and cost for every entry in the file. If the
entries differ, the time taken to reject a wrong password can reveal
which usernames exist.
The password file can be updated while rclone is running.
When authentication is enabled every request must be authenticated,
including HTTP OPTIONS requests, as the answer to an OPTIONS request can
reveal whether a path exists. Health checks or monitoring which use
unauthenticated OPTIONS requests will need to send credentials. The
exception is browser CORS preflight requests when `--allow-origin`
is set, which are answered without authentication as browsers never send
credentials with them.
Use `--realm` to set the authentication realm.
Use `--salt` to change the password hashing salt from the default.
@@ -236,7 +263,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -291,7 +318,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -445,7 +473,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
@@ -688,9 +716,10 @@ This config generated must have this extra parameter
- `_root` - root to use for the backend
And it may have this parameter
And it may have these parameters
- `_obscure` - comma separated strings for parameters to obscure
- `_secret_access_key` - the secret for S3 access key auth (see below)
If password authentication was used by the client, input to the proxy
process (on STDIN) would look similar to this:
@@ -698,7 +727,8 @@ process (on STDIN) would look similar to this:
```json
{
"user": "me",
"pass": "mypassword"
"pass": "mypassword",
"client_ip": "192.168.1.1"
}
```
@@ -708,10 +738,44 @@ proxy process (on STDIN) would look similar to this:
```json
{
"user": "me",
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
"client_ip": "192.168.1.1"
}
```
If the client authenticated with an S3 access key (`rclone serve s3`),
the client never sends its secret, only a signature made with it, so
the input contains just the access key ID as the `user` with no `pass`
or `public_key`:
```json
{
"user": "AKIAIOSFODNN7EXAMPLE",
"client_ip": "192.168.1.1"
}
```
In this case the program must look up the secret access key for that
access key ID and return it in the `_secret_access_key` field of the
output. Rclone then uses that secret to verify the signature on the
request, refusing the request if it does not match. This means the
proxy program is the source of truth for both the credentials and the
backend they map to. If the program does not return
`_secret_access_key` or returns it empty the request is refused.
The program's answer for an access key ID is cached (see below) but
is checked with the program again after 5 minutes even if the access
key ID is in constant use, so revoking an access key ID in the
program takes effect within 5 minutes. A rotated secret takes effect
on the first request signed with it.
The `client_ip` key holds the IP address the client connected from,
without a port number. It can be used to restrict logins to certain
networks, or to log authentication attempts centrally. It is omitted if
the client has no IP address, for example when connecting over a unix
socket. Note that if rclone is behind a reverse proxy this will be the
address of the reverse proxy and not the original client.
And as an example return this on STDOUT
```json
@@ -737,11 +801,20 @@ to make proxy to many different sftp backends, you could make the
in the output and the user to `user`. For security you'd probably want
to restrict the `host` to a limited list.
An internal cache of backends is keyed on the `user` and a hash of the
`pass` or `public_key`. This means that if a user's password or
public-key changes, or the proxy returns different config parameters
(eg a rotated `api_key`), a fresh backend will be created on the next
request rather than the cached one being reused.
An internal cache of backends is keyed on the `user`, the kind of
credential presented, a hash of the `pass` or `public_key`, and the
`client_ip`. This means that if a user's password or public-key
changes, the client connects from a new IP address, or the proxy
returns different config parameters (eg a rotated `api_key`), a fresh
backend will be created on the next request rather than the cached one
being reused. A backend created for a public key login is never
reused for a password login, so a public key cannot be sent as a
password to skip the program.
A cached answer is checked with the program again after 5 minutes even
if the user is in constant use, so revoking a user in the program
takes effect within 5 minutes. For `serve sftp` this applies to new
logins - an existing connection is not closed.
This can be used to build general purpose proxies to any kind of
backend that rclone supports.
docs/content/commands/rclone_serve_nfs.md
+4 -3
View File
@@ -167,7 +167,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -222,7 +222,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -376,7 +377,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
docs/content/commands/rclone_serve_restic.md
+19
View File
@@ -126,6 +126,13 @@ accept in the HTTP header.
will overriding existing values. The flag may be repeated to add multiple
headers. Use the format `Header-Name: value`.
`--allow-origin` enables cross-origin (CORS) requests from browsers
by setting the `Access-Control-Allow-Origin` header on every response to the
origin given, e.g. `--allow-origin https://app.example.com`. Browser
CORS preflight OPTIONS requests are answered by rclone without needing
authentication as browsers never send credentials with them. Setting this
to `*` allows any origin which is a security risk on an authenticated server.
`--baseurl` controls the URL prefix that rclone serves from. By default
rclone will serve from the root. If you used `--baseurl "/rclone"` then
rclone would serve from a URL starting with "/rclone/". This is
@@ -210,8 +217,20 @@ htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser
```
Use the same hash format and cost for every entry in the file. If the
entries differ, the time taken to reject a wrong password can reveal
which usernames exist.
The password file can be updated while rclone is running.
When authentication is enabled every request must be authenticated,
including HTTP OPTIONS requests, as the answer to an OPTIONS request can
reveal whether a path exists. Health checks or monitoring which use
unauthenticated OPTIONS requests will need to send credentials. The
exception is browser CORS preflight requests when `--allow-origin`
is set, which are answered without authentication as browsers never send
credentials with them.
Use `--realm` to set the authentication realm.
Use `--salt` to change the password hashing salt from the default.
docs/content/commands/rclone_serve_s3.md
+605 -106
View File
@@ -26,6 +26,21 @@ docs](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)).
`--auth-key` is not provided then `serve s3` will allow anonymous
access.
Alternatively `--auth-proxy` can be used to look up the secret for each
access key ID and choose the backend it maps to (see [Auth
Proxy](#auth-proxy) below). When an auth proxy is in use `--auth-key`
is ignored and every request must be signed with the secret the proxy
returns for its access key ID. Each access key ID is a separate user,
whose multipart uploads and object metadata are private to it, even if
the proxy maps several access key IDs to the same backend.
As on AWS, every `x-amz-*` header in an authenticated request must be
in its signed headers list; a request with an unsigned `x-amz-*`
header is refused with `AccessDenied`. S3 clients sign these headers
as a matter of course, but a header added after signing (for example
by a proxy in front of the server, or `x-amz-meta-*` headers sent
with a presigned URL which only signs the host) is refused.
Like all rclone flags `--auth-key` can be set via environment
variables, in this case `RCLONE_AUTH_KEY`. Since this flag can be
repeated, the input to `RCLONE_AUTH_KEY` is CSV encoded. Because the
@@ -60,6 +75,14 @@ part of the hostname (such as mybucket.local)
Use `--etag-hash` if you want to change the hash uses for the `ETag`.
Note that using anything other than `MD5` (the default) is likely to
cause problems for S3 clients which rely on the Etag being the MD5.
Use `--etag-hash auto` to use the best hash the backend supports - with
`--auth-proxy` that of each user's backend.
The `ETag` of an object made by a multipart upload is its hash, like
any other object, rather than the `-N` suffixed `ETag` AWS S3 gives
these. If the remote doesn't have the MD5 of objects, `HeadObject`
and listings only return the MD5 of objects uploaded through `serve
s3` while the object's metadata is kept (see below).
## Quickstart
@@ -101,29 +124,78 @@ access_key_id = ACCESS_KEY_ID
secret_access_key = SECRET_ACCESS_KEY
```
## Object uploads (PUT)
A `PutObject` upload only ever changes the object at its key atomically, on
success, a failed or interrupted PUT neither removes nor overwrites the
object already stored at the key, and never leaves a partial object visible
at it.
Remotes that upload atomically (e.g. object stores such as `s3`) are streamed
straight to the destination. On remotes where a partial upload would
otherwise be visible (e.g. `local`), and whenever `--vfs-cache-mode` is
`writes` or above, the upload is written to a temporary object that is
renamed into place on success; these remotes need to support a server-side
move or copy for this (nearly all do - without move or copy the upload is
written directly and a failed PUT may leave a partial object at the key). If
`serve s3` is killed part-way through an upload the temporary object (named
with a leading `.rclone_temp_put_`) may be left behind; it is hidden from
S3 listings but must be removed manually.
## Multipart uploads
By default `serve s3` **streams** each multipart upload, in part-number
order, into a single `PutStream` upload to the underlying remote, so the
whole file is never buffered in memory - memory use stays bounded by the
parts in flight. The remote then performs its own internal upload (for
example its own multipart upload, still with bounded memory). This works
for any remote that supports `PutStream`, which is nearly all of them,
including through `crypt`.
The upload is atomic so the destination object only ever changes on a
Multipart uploads are written, in part-number order, to a temporary
object which is renamed into place, server-side, on completion, so the
upload is atomic. The object at the key only ever changes on a
successful completion. A failed or aborted upload never affects any
object already stored under that name. Remotes that upload atomically
already (object stores such as `s3`) are streamed straight to the
destination. On remotes where a partial upload would otherwise be visible
(such as `local`), the parts are streamed to a temporary object that is
moved into place, server-side, on completion; these remotes therefore
also need to support a server-side move or copy.
object already stored under that name and a partly-uploaded object
never becomes visible under it.
With the default `--vfs-cache-mode off` `serve s3` **streams** each
multipart upload, in part-number order, into a single streaming upload
to the underlying remote, so the whole file is never buffered in
memory: only parts which arrive out of order are buffered, within the
limits described below. The remote then performs its own internal upload (for example its own multipart
upload, still with bounded memory). Remotes that don't support
streaming uploads (those that must know the file size before the
upload starts, such as `onedrive`, `pcloud`, `jottacloud`, `mailru`,
`opendrive`, `putio`, `protondrive` and `zoho`) have the parts spooled
to a temporary file on **local disk** instead, and uploaded with the
size then known on completion, so they need local disk space for the
largest objects in flight rather than memory.
With `--vfs-cache-mode writes` (or `full`) the parts are written to a
temporary file in the VFS cache and uploaded by the VFS write-back -
see [Multipart uploads and the VFS
cache](#multipart-uploads-and-the-vfs-cache) below.
The rename into place needs the remote to support a server-side move
or copy, which nearly all do. It is a cheap rename on most remotes,
but on object stores without a real rename (such as `s3` itself) the
move is performed as a server-side copy and delete of the whole
object, which can take time and API calls for large objects.
Concurrent multipart uploads of the same key (which S3 permits) are
safe. Each writes its own temporary object and the last to complete
wins.
On the few remotes that support neither server side move nor copy, the
parts are written straight to the destination object instead and never
buffered in memory. This is at some cost in atomicity - the incomplete
object is visible under its final name while the upload is in flight,
as it also is for a plain object PUT on such remotes, and concurrent
multipart uploads of the same key write to the same object and can
interleave. A failed or aborted upload still leaves any pre-existing
object untouched provided the remote uploads atomically and the VFS
cache is off; on a remote where partial uploads are visible it may
leave partial data at the key (like a plain PUT there), and with
`--vfs-cache-mode writes` (or `full`) a write to the cache cannot be
abandoned, so an aborted upload's partial data is written back to the
remote as if it had completed.
**Features**
- The whole object is never buffered in memory; memory use is bounded by
the parts in flight, not the upload size.
- The whole object is never buffered in memory; only parts arriving out
of order are, within fixed limits, whatever the upload size.
- Parts can be any size. Clients that don't produce uniform-sized parts
work fine - for example PostgreSQL backup tools such as **pgBarman**
and **pgBackRest**, which flush an upload buffer once it grows past
@@ -133,69 +205,275 @@ also need to support a server-side move or copy.
as one continuous stream.
- The destination object only ever changes atomically, on completion: an
aborted or failed upload leaves any pre-existing object of the same
name untouched, and a partly-uploaded object never becomes visible.
- Backend-agnostic - it only needs the remote to support `PutStream`
(plus a server-side move or copy on remotes that don't upload
atomically).
name untouched, and a partly-uploaded object never becomes visible
(except on the few remotes with no server-side move or copy, as
above).
- Multipart uploads go through the VFS like any other upload, so they
show in rclone's transfer stats and obey `--bwlimit`.
- Backend-agnostic - it only needs the remote to support a server-side
move or copy for the rename into place, which nearly all do; a remote
without streaming upload support spools to local disk as above.
**Limitations**
- Parts must arrive in ascending, contiguous part-number order
(1, 2, 3, ...). Parts the client uploads concurrently or out of order
are buffered until their turn. The memory used for this buffering is
capped, per upload, by `--multipart-streaming-buffer-limit` (default
`256M`, `0` for no limit): a part that would take the buffer over the
limit is stalled until the stream drains, so a client that uploads
faster than the remote can accept sees backpressure rather than
unbounded server memory use. Since a stalled part holds its HTTP
request open, clients whose upload concurrency times chunk size
exceeds the limit may need a longer read timeout when the remote is
slow. Non-contiguous part numbers are rejected on completion.
Configure the client to upload in part order, ideally with low
concurrency, for the lowest memory use.
- Parts must arrive in ascending, contiguous part-number order (1, 2, 3,
...). The part the stream needs next is written straight through as it
arrives, so it is never buffered, whatever its size. Parts the client
uploads concurrently or out of order are buffered until their turn.
The memory used for this buffering is capped, per upload, by
`--multipart-streaming-buffer-limit` (default `256M`, `0` for no
limit), and across all the uploads of each user (each access key) by
`--multipart-streaming-buffer-total` (default `1G`, `0` for no
limit), counting each buffered part in the
whole 1 MiB memory pages it occupies: a part that would take the
buffer over either limit is stalled until the stream drains (or, if it
is bigger than the limits, until it is the next part), so a client
that uploads faster than the remote can accept sees backpressure
rather than unbounded server memory use. A stalled part holds its HTTP
request open, and one stalled for more than a minute is failed with a
`SlowDown` error, which S3 clients retry. Clients whose upload
concurrency times part size exceeds the limits will see these stalls
and retries when the remote is slower than the client. Non-contiguous
part numbers are rejected on completion. Configure the client to
upload in part order, ideally with low concurrency, for the lowest
memory use.
- A part uploaded again before completion - typically a client retrying
after a timeout - is accepted: if the earlier copy is still buffered
it is replaced, and if it has already been streamed an identical
re-upload is a no-op. What isn't possible is replacing a part that has
already been streamed with *different* content - that is rejected. A
failure in the stream to the remote itself still aborts the whole
upload and the client must start it again. (The remote's own upload
re-upload is a no-op (a re-upload which arrives while the part is
being streamed waits for it to finish first). If a part fails part way
through being streamed
(for example the client's connection drops) the client's retry of it
carries on where it stopped, provided it has the same content. What
isn't possible is replacing a part that has already been streamed,
in whole or in part, with *different* content - that is rejected. A
part which arrives in full but is then rejected, for example because
it doesn't match the `Content-MD5` the client declared, has already
been streamed, so the whole upload is failed and the client must
start it again. So is a failure in the stream to the remote itself. (The remote's own upload
still retries its internal chunks.)
- A `CompleteMultipartUpload` which fails because the parts it lists
don't match those uploaded, or there is a gap in the part numbers,
leaves the upload open, as on S3, so the client can upload any
missing parts and complete it again. If writing the completed object
to the remote fails, the upload is failed and must be started again.
- Parts are serialised into one stream, so ingest from the client is
effectively single-threaded, although the remote's own upload still
runs concurrently.
- On remotes that don't upload atomically (such as `local`), the
completed object is moved into place with a server-side operation.
This is a cheap rename on most such remotes. On these remotes, if
`serve s3` is killed part-way through an upload the temporary object
(named with a leading `.rclone_multipart_upload_`) may be left behind;
it is hidden from S3 listings but must be removed manually.
effectively single-threaded. When streaming, the remote's own upload
runs concurrently with the parts arriving; with the local disk spool
or the VFS cache the upload to the remote only starts on completion.
- If `serve s3` is killed part-way through an upload the temporary
object (named with a leading `.rclone_temp_multipart_`) may be left
behind; it is hidden from S3 listings but must be removed manually.
### Multipart uploads and the VFS cache
With `--vfs-cache-mode writes` (or `full`) multipart uploads do not
stream to the remote at all. The parts are written, in part-number
order, to a temporary file in the VFS cache. On completion the file is
renamed into place and uploaded by the VFS write-back, exactly like a
plain object PUT. This needs no streaming upload support from the
remote. The rename normally happens in the cache before the upload has
started, but the VFS requires the remote to support a server-side move
or copy to rename files at all (and uses one if the temporary file has
already been written back, e.g. with `--vfs-write-back 0`). On remotes
without either, the parts are written to the cache directly under the
final key instead: the upload still never touches memory, but it loses
its atomicity - the in-flight upload is visible at the key, and an
aborted upload cannot be abandoned once in the cache, so its partial
data is written back to the remote as if it were a completed object.
Remotes that benefit from `--vfs-cache-mode writes`:
- **Remotes over slow or unreliable links.** A failure in a streamed
upload aborts the whole multipart upload and the client must start
again from the first part; a failed write-back upload is retried by
the VFS (see `--vfs-cache-max-age` and friends) without the client
being involved. Ingest from the client also runs at local disk speed
rather than being throttled to the remote's pace.
- **Workloads that read back or overwrite what they just wrote.** The
completed object stays in the cache, so subsequent `GET`/`HEAD`
requests are served locally, and plain PUTs and multipart uploads to
the same key go through the same cache entry so the last write wins
regardless of upload style.
The trade-offs of the VFS cache:
- The whole object lands on local disk, so the cache (`--cache-dir`)
needs space for the largest objects in flight; `--vfs-cache-max-size`
cannot evict files which are still being uploaded.
- The `200 OK` for `CompleteMultipartUpload` means the data is safely
in the **local cache**, not yet on the remote - the same durability
the cache gives plain PUTs. If an acknowledgement must mean the data
has reached the remote (for example WAL archiving), use the default
`--vfs-cache-mode off`.
- The upload to the remote only starts on completion, rather than
overlapping with the parts arriving, so the data reaches the remote
later than with streaming.
- If `serve s3` is killed part-way through an upload, the temporary
file survives in the cache and the VFS cache recovery uploads it to
the remote on restart as a temporary object (named with a leading
`.rclone_temp_multipart_`); as with the streaming path, it is
hidden from S3 listings but must be removed manually.
### Cleaning up temporary objects
If `serve s3` is killed part-way through an upload it can leave a
temporary object behind, named with a leading `.rclone_temp_`. This
whole prefix is reserved: any object whose name (the last
`/`-separated segment of its key) starts with `.rclone_temp_` is
hidden from S3 listings, so don't give real objects such names - an
existing object with such a name disappears from listings (though it
stays accessible directly by its key: only listings hide reserved
names, `GET`, `HEAD` and `DELETE` of the exact key still work). A
temporary object never holds acknowledged data - uploads whose
temporary object survived were never confirmed to the client - so old
ones are safe to delete:
rclone delete --min-age 24h --include ".rclone_temp_*" remote:path
The `--min-age` protects uploads which are still in progress: make sure
it is longer than your longest upload, especially if several `serve s3`
instances share the same remote.
rclone v1.75 named its temporary multipart objects
`.rclone_multipart_upload_*`; leftovers from an older server are also
hidden from listings and can be cleaned up the same way.
### Abandoned uploads
A client which starts a multipart upload and vanishes without either
completing or aborting it would otherwise hold on to its resources
forever.
An incomplete multipart upload which has had no activity for
`--multipart-expiry` (default `24h`) is therefore aborted and cleaned
up, exactly as if the client had called `AbortMultipartUpload`, and a
`NOTICE` is logged.
An upload with a part still being received is never expired, however
slowly the part is arriving, and each completed part restarts the
clock, so the expiry only needs to outlast the client's pauses
*between* parts, not the whole upload.
Late operations on an expired upload fail with `NoSuchUpload`, as they
do on real S3 when a lifecycle rule has aborted the upload. Set
`--multipart-expiry 0` to keep incomplete uploads forever - but note
that abandoned uploads are then never cleaned up and go on counting
towards `--multipart-max-uploads` below, so enough of them will stop
the user starting any new multipart upload until the server is
restarted.
Each multipart upload in progress holds server resources - for example
the buffers the remote's own upload uses once the first part has
arrived - so each user (each access key) can have no more than
`--multipart-max-uploads` (default `1000`, `0` for no limit) in
progress at once. Starting another fails with a `SlowDown` error,
which S3 clients retry, and a one-off `NOTICE` is logged.
### Disabling streaming
If you pass `--disable-multipart-streaming`, or the remote doesn't
support `PutStream` (or doesn't upload atomically and can't move or copy
server-side), multipart uploads are instead **buffered in memory**
by the underlying S3 library: every part is held in memory and the whole
object is written out in one go when the upload completes (the previous
behaviour). This removes the in-order/contiguous-part restriction above,
so parts can be uploaded in any order, but **memory use grows with the
size of the upload**, so it is only suitable for small objects. A one-off
`NOTICE` is logged the first time this happens.
If you pass `--disable-multipart-streaming`, multipart uploads are
instead **buffered in memory** by the underlying S3 library: every
part is held in memory and the whole object is written out in one go
when the upload completes. This removes the in-order/contiguous-part
restriction above, so parts can be uploaded in any order, but **memory
use grows with the size of the upload**, so it is only suitable for
small objects. A one-off `NOTICE` is logged the first time this
happens. This flag is the only thing that makes multipart uploads
buffer in memory - it is never done because of missing remote
capabilities. The limits on multipart uploads above
(`--multipart-streaming-buffer-limit`,
`--multipart-streaming-buffer-total` and `--multipart-max-uploads`) do
not apply to uploads buffered in memory, so don't use this flag if the
server is open to clients you don't trust. Consider
`--vfs-cache-mode writes` instead, which buffers the upload in the VFS
cache on disk and takes precedence over
`--disable-multipart-streaming`.
Alternatively, if the client is an rclone `s3` remote (like the
`[serves3]` example above), you can set `use_multipart_uploads = false`
on it so it uploads each object as a single stream and skips multipart
uploads altogether.
## Memory use
`serve s3` never buffers whole objects in memory (except with
`--disable-multipart-streaming`, above), and limits the memory each
kind of request can use, so that a client, even a hostile one, can't
make it use unbounded memory.
- **Object data.** `PutObject` bodies and multipart parts arriving in
order are streamed straight through to the remote. Parts arriving out
of order are buffered, up to `--multipart-streaming-buffer-limit`
per upload and `--multipart-streaming-buffer-total` per user.
- **Uploads in progress.** Each user can have at most
`--multipart-max-uploads` multipart uploads in progress. An upload
holds little memory until its
first part arrives. After that, with `--vfs-cache-mode off`, it holds
a `--streaming-upload-cutoff` sized buffer plus whatever the remote's
own upload buffers, which for many remotes is up to their chunk size
times their upload concurrency (for example `--s3-chunk-size` times
`--s3-upload-concurrency`). Plain `PutObject` uploads use the same
while they are in progress.
- **Metadata.** The metadata of at most `--metadata-max-objects`
objects of each user is kept, each limited to 2 KB.
- **Request bodies.** The XML bodies of requests such as
`CompleteMultipartUpload` and `DeleteObjects` are limited to 10 MiB,
and `DeleteObjects` to 1000 keys. A browser form (`POST`) upload
holds at most 1 MiB in memory, spooling the rest to a temporary file,
and is limited to 5 GB.
- **Listings.** Each page of a `ListObjects` listing reads only the
directories needed to fill it, and returns at most 1000 keys.
Directory listings are cached by the [VFS directory
cache](#vfs-directory-cache).
The largest of these is usually the remote's upload buffers, which can
be up to `--multipart-max-uploads` times the buffers of one upload for
each user if clients start many uploads and send part of each. The
limits apply to each user separately, so that one user can't stop
others uploading, which means the memory use grows with the number of
users. To limit it, lower
`--multipart-max-uploads`, the remote's chunk size or upload
concurrency, or set `--max-buffer-memory` to limit the memory used by
the buffers of all uploads to remotes which use rclone's buffer pool
(including `s3`, `azureblob` and `b2`). Buffered multipart parts come
from the same pool, so if you set `--max-buffer-memory` make it
comfortably bigger than `--multipart-streaming-buffer-total` times the
number of users uploading at once, otherwise
buffered parts can use up all the memory the remote needs to upload the
parts which would free them.
The number of connections is not limited, so if the server is open to
clients you don't trust, use `--auth-key` or `--auth-proxy` and
consider putting it behind a reverse proxy which limits connections.
## Design goals
These goals explain many of the limitations below.
- **Objects are plain files on the remote.** A bucket is a directory
in the root and a key is a path in it, so the remote stays usable by
other tools, such as `rclone mount` or `rclone sync`. So keys must
be paths (see [Object names](#object-names)).
- **No database.** `serve s3` stores nothing but the objects. What the
remote can't hold, such as user metadata, tags, the MD5 `ETag` of an
object on a remote without MD5 and the directory markers put, is
kept in memory as a best effort, and lost on restart.
- **Works with any rclone remote.** What `serve s3` can do depends on
what the remote supports, such as the hashes used for `ETag`s and
modification times, rather than needing a particular backend.
- **Bounded memory.** Uploads, including multipart uploads, are
streamed to the remote and the memory each request can use is
limited (see [Memory use](#memory-use)). Nothing is written to local
disk unless the VFS cache is used or the remote can't stream uploads
(see [Multipart uploads](#multipart-uploads)).
- **Refuse rather than ignore.** Unsupported features are refused, or
answered as S3 answers when the feature isn't configured, so a
client is never told something was done which wasn't (see
[Compatibility](#compatibility)).
- **A storage gateway, not a full S3.** `serve s3` implements the parts
of the S3 API clients use to store and fetch objects. Versioning,
ACLs, bucket policies, lifecycle rules and the like are out of scope.
- **Compatible with rclone's s3 backend.** This includes its directory
markers, which the tests check.
## Bugs
Multipart server side copies do not work (see
[#7454](https://github.com/rclone/rclone/issues/7454)). These take a
very long time and eventually fail. The default threshold for
multipart server side copies is 5G which is the maximum it can be, so
files above this side will fail to be server side copied.
For a current list of `serve s3` bugs see the [serve
s3](https://github.com/rclone/rclone/labels/serve%20s3) bug category
on GitHub.
@@ -204,57 +482,124 @@ on GitHub.
`serve s3` will treat all directories in the root as buckets and
ignore all files in the root. You can use `CreateBucket` to create
folders under the root, but you can't create empty folders under other
folders not in the root.
folders under the root, and put a directory marker to create an empty
folder below them (see [Object names](#object-names)).
When using `PutObject` or `DeleteObject`, rclone will automatically
create or clean up empty folders. If you don't want to clean up empty
folders automatically, use `--no-cleanup`.
folders automatically, use `--directory-markers` (see [Object
names](#object-names)). The deprecated `--no-cleanup` does the same.
When using `ListObjects`, rclone will use `/` when the delimiter is
empty. This reduces backend requests with no effect on most
operations, but if the delimiter is something other than `/` and
empty, rclone will do a full recursive search of the backend, which
can take some time.
When using `ListObjects` with the delimiter `/`, rclone lists the
directory the prefix is in, showing its subdirectories as common
prefixes, including empty ones. Without a delimiter the listing is
recursive: rclone walks the directory tree below the prefix, reading
only as many directories as it needs to fill each page, and lists
empty directories as their directory markers (see below). Any other
delimiter is supported too, but needs the whole directory tree below
the prefix walking, unless the delimiter is in the directory names.
Versioning is not currently supported.
Versioning is not supported (see [Design goals](#design-goals)).
Metadata will only be saved in memory other than the rclone `mtime`
metadata which will be set as the modification time of the file.
Conditional writes, `PutObject` and `CompleteMultipartUpload` with
`If-None-Match` or `If-Match`, are supported. The conditions are
checked before the object is written, and again as it is moved into
place when it is written to a temporary object first. This happens for
multipart uploads and, for `PutObject`, with remotes which can upload
partial files (such as local disk) or with `--vfs-cache-mode minimal`
or above. The second check is atomic against other writes through
`serve s3`, but neither is atomic against changes made to the remote
some other way. If the remote doesn't support the `--etag-hash`,
objects have no `ETag` so `If-Match` with an `ETag` always fails.
Metadata will only be saved in memory (see [Design
goals](#design-goals)) other than the rclone `mtime` metadata which
will be set as the modification time of the file. The
metadata of at most `--metadata-max-objects` objects (default `100000`,
`0` for no limit) of each user (each access key) is kept; after that
the metadata of the user's least recently used object is forgotten. The metadata of an object is
ignored once the object has been changed some other way, for example
by another `--auth-proxy` user or directly on the backend. Object tags
are kept with the metadata in the same way.
## Object names
`serve s3` stores objects as files in the backend, so object keys are
mapped to file paths rather than treated as the opaque strings AWS S3
allows. Keys must be in canonical path form: keys that contain `..` or
`.` path segments, repeated slashes (`//`), or a leading or trailing
slash are rejected with a `400 Bad Request` (`InvalidArgument`)
instead of being normalised, since normalising them could alias two
distinct keys to the same file or resolve a key outside its bucket.
This matches the behaviour of other S3 servers such as MinIO.
`.` path segments, repeated slashes (`//`), or a leading slash are
rejected with a `400 Bad Request` (`InvalidArgument`) instead of being
normalised, since normalising them could alias two distinct keys to
the same file or resolve a key outside its bucket. This matches the
behaviour of other S3 servers such as MinIO.
## Supported operations
A key ending in `/`, such as `dir/`, is a directory marker, standing
for the directory `dir`. Putting an empty directory marker creates the
directory (one with data is rejected with `400 Bad Request`), and
deleting it removes the directory if it is empty. Directory markers
are found by `HeadObject`, `GetObject` and `CopyObject` for any
directory, but only the markers of empty directories, and of the
directories a marker was put for, are listed as objects, as other
directories may not have been made by putting a marker. The markers
put are remembered in memory with the metadata (see
[Limitations](#limitations)), so after a restart, or once forgotten,
the marker of a directory holding other keys is no longer listed.
While a marker is remembered its directory isn't removed when the
last key below it is deleted.
`serve s3` currently supports the following operations.
With `--directory-markers` every directory is listed as a directory
marker, so the markers are listed after a restart too, as needed by
clients which rely on them, such as rclone's s3 backend with
`--s3-directory-markers`. As every directory has a marker, a directory
isn't removed when the last key below it is deleted. Deleting the
marker of a directory which holds other keys stops it being listed,
and the directory is then removed with the last key below it, but this
is only remembered in memory, like the markers put without
`--directory-markers`.
- Bucket
- `ListBuckets`
- `CreateBucket`
- `DeleteBucket`
- Object
- `HeadObject`
- `ListObjects`
- `GetObject`
- `PutObject`
- `DeleteObject`
- `DeleteObjects`
- `CreateMultipartUpload`
- `CompleteMultipartUpload`
- `AbortMultipartUpload`
- `CopyObject`
- `UploadPart`
As keys are file paths, a key can't be both an object and a directory
holding other objects, for example `a/b` and `a/b/c`. Writing an
object which would need this fails with `400 Bad Request`
(`InvalidRequest`).
Other operations will return error `Unimplemented`.
## Compatibility
`serve s3` implements the parts of the S3 API that S3 clients commonly
use to store and fetch objects. Features it doesn't support are
refused with `501 NotImplemented`, or answered as AWS S3 answers for a
bucket without that feature configured, rather than being silently
ignored.
| Feature | Supported | Notes |
| ------- | --------- | ----- |
| `ListBuckets`, `CreateBucket`, `DeleteBucket`, `HeadBucket`, `GetBucketLocation` | Yes | Buckets are the directories in the root of the remote. |
| `ListObjects`, `ListObjectsV2` | Yes | Prefixes, delimiters, markers, `start-after`, continuation tokens, `fetch-owner` and `encoding-type=url` all work. Delimiters other than `/` need the whole directory tree walking. |
| `HeadObject`, `GetObject` | Yes | Including `Range`, `partNumber`, the `response-*` header overrides and the `If-Match`, `If-None-Match`, `If-Modified-Since` and `If-Unmodified-Since` conditions. |
| `PutObject` | Yes | Including `Content-MD5`, `aws-chunked` uploads and uploads with no `Content-Length`. |
| `CopyObject` | Yes | Including the metadata and tagging directives and the `x-amz-copy-source-if-*` conditions. |
| `DeleteObject`, `DeleteObjects` | Yes | |
| Multipart uploads | Yes | `CreateMultipartUpload`, `UploadPart`, `UploadPartCopy`, `CompleteMultipartUpload`, `AbortMultipartUpload`, `ListMultipartUploads` and `ListParts`. The `ETag` of the object is its hash, not the `-N` suffixed one AWS gives. The 5 MiB minimum part size isn't enforced. |
| Conditional writes | Partial | `If-Match` and `If-None-Match` on `PutObject` and `CompleteMultipartUpload` are checked, but not atomically against changes made to the remote some other way (see [Limitations](#limitations)). |
| Authentication | Partial | AWS Signature Version 4, in headers, presigned URLs and browser form (`POST`) uploads. Signature Version 2 isn't supported. |
| Anonymous access | Partial | All requests are allowed without `--auth-key` or `--auth-proxy`, and none are with them, as buckets and objects can't be made public. |
| Metadata | Partial | `x-amz-meta-*`, the storage class and the standard headers such as `Content-Type` are kept in memory only (see [Limitations](#limitations)), except the modification time. The storage class is only recorded, so doesn't change how the object is stored. |
| Directory markers | Partial | Keys ending in `/` are stored as directories, so can't hold data. A marker put for a directory holding other keys is only listed while it is remembered in memory, unless `--directory-markers` is set (see [Object names](#object-names)). |
| Checksums | Partial | The `x-amz-checksum-*` of a `PutObject` (CRC32, CRC32C, CRC64NVME, SHA1 or SHA256) is checked and returned with the metadata. The `Content-MD5` and `x-amz-checksum-*` of the bodies of other requests, such as `DeleteObjects`, are checked. The checksums of multipart uploads aren't supported. |
| `GetObjectAttributes` | Partial | Everything but the checksums of multipart uploads. |
| Object tagging | Partial | `GetObjectTagging`, `PutObjectTagging` and `DeleteObjectTagging` work, but tags are kept in memory with the metadata. |
| Bucket owner | No | `serve s3` has no AWS account IDs, so `x-amz-expected-bucket-owner` and `x-amz-source-expected-bucket-owner` are ignored. |
| ACLs | Partial | `GetBucketAcl` and `GetObjectAcl` give the owner full control. Only the `private` ACL can be set. |
| CORS | Partial | Preflight (`OPTIONS`) requests are answered, allowing any origin. CORS configuration (`PutBucketCors`) isn't supported. |
| Server side encryption | No | SSE-S3 (`AES256`) is accepted but the data isn't encrypted. SSE-KMS and SSE-C are refused. |
| Versioning | No | Buckets are reported as unversioned and `ListObjectVersions` lists each object as version `null`. Enabling versioning is refused. |
| Object lock, retention and legal hold | No | |
| Bucket configuration | No | Bucket policies, tagging, lifecycle, logging, encryption, website, replication, notifications and the like. Getting them fails as for a bucket without them configured. |
| `SelectObjectContent`, `RestoreObject`, torrents | No | |
The [ceph s3-tests](https://github.com/ceph/s3-tests) can be run
against `serve s3` to check its compatibility. See
[cmd/serve/s3/s3tests](https://github.com/rclone/rclone/tree/master/cmd/serve/s3/s3tests)
for how.
### Authentication
@@ -284,8 +629,20 @@ htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser
```
Use the same hash format and cost for every entry in the file. If the
entries differ, the time taken to reject a wrong password can reveal
which usernames exist.
The password file can be updated while rclone is running.
When authentication is enabled every request must be authenticated,
including HTTP OPTIONS requests, as the answer to an OPTIONS request can
reveal whether a path exists. Health checks or monitoring which use
unauthenticated OPTIONS requests will need to send credentials. The
exception is browser CORS preflight requests when `--allow-origin`
is set, which are answered without authentication as browsers never send
credentials with them.
Use `--realm` to set the authentication realm.
Use `--salt` to change the password hashing salt from the default.
@@ -317,6 +674,13 @@ accept in the HTTP header.
will overriding existing values. The flag may be repeated to add multiple
headers. Use the format `Header-Name: value`.
`--allow-origin` enables cross-origin (CORS) requests from browsers
by setting the `Access-Control-Allow-Origin` header on every response to the
origin given, e.g. `--allow-origin https://app.example.com`. Browser
CORS preflight OPTIONS requests are answered by rclone without needing
authentication as browsers never send credentials with them. Setting this
to `*` allows any origin which is a security risk on an authenticated server.
`--baseurl` controls the URL prefix that rclone serves from. By default
rclone will serve from the root. If you used `--baseurl "/rclone"` then
rclone would serve from a URL starting with "/rclone/". This is
@@ -436,7 +800,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -491,7 +855,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -645,7 +1010,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
@@ -862,6 +1227,135 @@ If the file has no metadata it will be returned as `{}` and if there
is an error reading the metadata the error will be returned as
`{"error":"error string"}`.
## Auth Proxy
If you supply the parameter `--auth-proxy /path/to/program` then
rclone will use that program to generate backends on the fly which
then are used to authenticate incoming requests. This uses a simple
JSON based protocol with input on STDIN and output on STDOUT.
**PLEASE NOTE:** `--auth-proxy` and `--authorized-keys` cannot be used
together, if `--auth-proxy` is set the authorized keys option will be
ignored.
There is an example program
[bin/test_proxy.py](https://github.com/rclone/rclone/blob/master/bin/test_proxy.py)
in the rclone source code.
The program's job is to take a `user` and `pass` on the input and turn
those into the config for a backend on STDOUT in JSON format. This
config will have any default parameters for the backend added, but it
won't use configuration from environment variables or command line
options - it is the job of the proxy program to make a complete
config.
This config generated must have this extra parameter
- `_root` - root to use for the backend
And it may have these parameters
- `_obscure` - comma separated strings for parameters to obscure
- `_secret_access_key` - the secret for S3 access key auth (see below)
If password authentication was used by the client, input to the proxy
process (on STDIN) would look similar to this:
```json
{
"user": "me",
"pass": "mypassword",
"client_ip": "192.168.1.1"
}
```
If public-key authentication was used by the client, input to the
proxy process (on STDIN) would look similar to this:
```json
{
"user": "me",
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
"client_ip": "192.168.1.1"
}
```
If the client authenticated with an S3 access key (`rclone serve s3`),
the client never sends its secret, only a signature made with it, so
the input contains just the access key ID as the `user` with no `pass`
or `public_key`:
```json
{
"user": "AKIAIOSFODNN7EXAMPLE",
"client_ip": "192.168.1.1"
}
```
In this case the program must look up the secret access key for that
access key ID and return it in the `_secret_access_key` field of the
output. Rclone then uses that secret to verify the signature on the
request, refusing the request if it does not match. This means the
proxy program is the source of truth for both the credentials and the
backend they map to. If the program does not return
`_secret_access_key` or returns it empty the request is refused.
The program's answer for an access key ID is cached (see below) but
is checked with the program again after 5 minutes even if the access
key ID is in constant use, so revoking an access key ID in the
program takes effect within 5 minutes. A rotated secret takes effect
on the first request signed with it.
The `client_ip` key holds the IP address the client connected from,
without a port number. It can be used to restrict logins to certain
networks, or to log authentication attempts centrally. It is omitted if
the client has no IP address, for example when connecting over a unix
socket. Note that if rclone is behind a reverse proxy this will be the
address of the reverse proxy and not the original client.
And as an example return this on STDOUT
```json
{
"type": "sftp",
"_root": "",
"_obscure": "pass",
"user": "me",
"pass": "mypassword",
"host": "sftp.example.com"
}
```
This would mean that an SFTP backend would be created on the fly for
the `user` and `pass`/`public_key` returned in the output to the host given. Note
that since `_obscure` is set to `pass`, rclone will obscure the `pass`
parameter before creating the backend (which is required for sftp
backends).
The program can manipulate the supplied `user` in any way, for example
to make proxy to many different sftp backends, you could make the
`user` be `user@example.com` and then set the `host` to `example.com`
in the output and the user to `user`. For security you'd probably want
to restrict the `host` to a limited list.
An internal cache of backends is keyed on the `user`, the kind of
credential presented, a hash of the `pass` or `public_key`, and the
`client_ip`. This means that if a user's password or public-key
changes, the client connects from a new IP address, or the proxy
returns different config parameters (eg a rotated `api_key`), a fresh
backend will be created on the next request rather than the cached one
being reused. A backend created for a public key login is never
reused for a password login, so a public key cannot be sent as a
password to skip the program.
A cached answer is checked with the program again after 5 minutes even
if the user is in constant use, so revoking a user in the program
takes effect within 5 minutes. For `serve sftp` this applies to new
logins - an existing connection is not closed.
This can be used to build general purpose proxies to any kind of
backend that rclone supports.
```
rclone serve s3 remote:path [flags]
```
@@ -878,7 +1372,8 @@ rclone serve s3 remote:path [flags]
--client-ca string Client certificate authority to verify clients with
--dir-cache-time Duration Time to cache directory entries for (default 5m0s)
--dir-perms FileMode Directory permissions (default 777)
--disable-multipart-streaming Buffer multipart uploads in memory instead of streaming them to the backend (see the Multipart uploads docs section)
--directory-markers List every directory as a directory marker object
--disable-multipart-streaming Buffer multipart uploads in memory instead of streaming them to the backend
--etag-hash string Which hash to use for the ETag, or auto or blank for off (default "MD5")
--file-perms FileMode File permissions (default 666)
--force-path-style If true use path style access if false use virtual hosted style (default true)
@@ -888,10 +1383,14 @@ rclone serve s3 remote:path [flags]
--key string TLS PEM Private key
--link-perms FileMode Link permissions (default 666)
--max-header-bytes int Maximum size of request header (default 4096)
--metadata-max-objects int Maximum number of objects of a user whose metadata is kept in memory, 0 for unlimited (default 100000)
--min-tls-version string Minimum TLS version that is acceptable (default "tls1.0")
--multipart-streaming-buffer-limit SizeSuffix Maximum memory buffered per streamed multipart upload for parts arriving out of order, 0 for unlimited (see the Multipart uploads docs section) (default 256Mi)
--multipart-expiry Duration Abort incomplete multipart uploads idle for longer than this, 0 to keep forever (default 1d)
--multipart-max-uploads int Maximum number of streamed multipart uploads a user can have in progress at once, 0 for unlimited (default 1000)
--multipart-streaming-buffer-limit SizeSuffix Maximum memory buffered per streamed multipart upload for parts arriving out of order, 0 for unlimited (default 256Mi)
--multipart-streaming-buffer-total SizeSuffix Maximum memory buffered by all the streamed multipart uploads of a user for parts arriving out of order, 0 for unlimited (default 1Gi)
--no-checksum Don't compare checksums on up/download
--no-cleanup Not to cleanup empty folder after object is deleted
--no-cleanup Deprecated: use --directory-markers instead
--no-modtime Don't read/write the modification time (can speed things up)
--no-seek Don't allow seeking in files
--pass string Password for authentication
docs/content/commands/rclone_serve_sftp.md
+57 -11
View File
@@ -140,7 +140,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -195,7 +195,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -349,7 +350,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
@@ -592,9 +593,10 @@ This config generated must have this extra parameter
- `_root` - root to use for the backend
And it may have this parameter
And it may have these parameters
- `_obscure` - comma separated strings for parameters to obscure
- `_secret_access_key` - the secret for S3 access key auth (see below)
If password authentication was used by the client, input to the proxy
process (on STDIN) would look similar to this:
@@ -602,7 +604,8 @@ process (on STDIN) would look similar to this:
```json
{
"user": "me",
"pass": "mypassword"
"pass": "mypassword",
"client_ip": "192.168.1.1"
}
```
@@ -612,10 +615,44 @@ proxy process (on STDIN) would look similar to this:
```json
{
"user": "me",
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
"client_ip": "192.168.1.1"
}
```
If the client authenticated with an S3 access key (`rclone serve s3`),
the client never sends its secret, only a signature made with it, so
the input contains just the access key ID as the `user` with no `pass`
or `public_key`:
```json
{
"user": "AKIAIOSFODNN7EXAMPLE",
"client_ip": "192.168.1.1"
}
```
In this case the program must look up the secret access key for that
access key ID and return it in the `_secret_access_key` field of the
output. Rclone then uses that secret to verify the signature on the
request, refusing the request if it does not match. This means the
proxy program is the source of truth for both the credentials and the
backend they map to. If the program does not return
`_secret_access_key` or returns it empty the request is refused.
The program's answer for an access key ID is cached (see below) but
is checked with the program again after 5 minutes even if the access
key ID is in constant use, so revoking an access key ID in the
program takes effect within 5 minutes. A rotated secret takes effect
on the first request signed with it.
The `client_ip` key holds the IP address the client connected from,
without a port number. It can be used to restrict logins to certain
networks, or to log authentication attempts centrally. It is omitted if
the client has no IP address, for example when connecting over a unix
socket. Note that if rclone is behind a reverse proxy this will be the
address of the reverse proxy and not the original client.
And as an example return this on STDOUT
```json
@@ -641,11 +678,20 @@ to make proxy to many different sftp backends, you could make the
in the output and the user to `user`. For security you'd probably want
to restrict the `host` to a limited list.
An internal cache of backends is keyed on the `user` and a hash of the
`pass` or `public_key`. This means that if a user's password or
public-key changes, or the proxy returns different config parameters
(eg a rotated `api_key`), a fresh backend will be created on the next
request rather than the cached one being reused.
An internal cache of backends is keyed on the `user`, the kind of
credential presented, a hash of the `pass` or `public_key`, and the
`client_ip`. This means that if a user's password or public-key
changes, the client connects from a new IP address, or the proxy
returns different config parameters (eg a rotated `api_key`), a fresh
backend will be created on the next request rather than the cached one
being reused. A backend created for a public key login is never
reused for a password login, so a public key cannot be sent as a
password to skip the program.
A cached answer is checked with the program again after 5 minutes even
if the user is in constant use, so revoking a user in the program
takes effect within 5 minutes. For `serve sftp` this applies to new
logins - an existing connection is not closed.
This can be used to build general purpose proxies to any kind of
backend that rclone supports.
docs/content/commands/rclone_serve_webdav.md
+84 -11
View File
@@ -126,6 +126,13 @@ accept in the HTTP header.
will overriding existing values. The flag may be repeated to add multiple
headers. Use the format `Header-Name: value`.
`--allow-origin` enables cross-origin (CORS) requests from browsers
by setting the `Access-Control-Allow-Origin` header on every response to the
origin given, e.g. `--allow-origin https://app.example.com`. Browser
CORS preflight OPTIONS requests are answered by rclone without needing
authentication as browsers never send credentials with them. Setting this
to `*` allows any origin which is a security risk on an authenticated server.
`--baseurl` controls the URL prefix that rclone serves from. By default
rclone will serve from the root. If you used `--baseurl "/rclone"` then
rclone would serve from a URL starting with "/rclone/". This is
@@ -195,6 +202,13 @@ to be used within the template to server pages:
| .Sort | | The current sort used. This is changeable via '?sort=' parameter. Possible values: namedirfirst, name, size, time (default namedirfirst). |
| .Order | | The current ordering used. This is changeable via '?order=' parameter. Possible values: asc, desc (default asc). |
| .Query | | Currently unused. |
| .Path | | The path of the directory from the root with leading and trailing slashes, e.g. '/dir/'. |
| .IsRoot | | Boolean for if this is the root directory. |
| .UpLink | | The link to the parent directory. |
| .Static | | Boolean for if the listing is being written as a static page by rclone index. |
| .NumDirs | | The number of directories in the listing. |
| .NumFiles | | The number of files in the listing. |
| .TotalSize | | The total size in bytes of the files in the listing. |
| .Breadcrumb | | Allows for creating a relative navigation. |
| | .Link | The link of the Text relative to the root. |
| | .Text | The Name of the directory. |
@@ -204,6 +218,7 @@ to be used within the template to server pages:
| | .IsDir | Boolean for if an entry is a directory or not. |
| | .Size | Size in bytes of the entry. |
| | .ModTime | The UTC timestamp of an entry. |
| | .MimeType | The MIME type of an entry guessed from its name, or 'inode/directory'. |
The server also makes the following functions available so that they can be used
within the template. These functions help extend the options for dynamic
@@ -244,8 +259,20 @@ htpasswd -B htpasswd user
htpasswd -B htpasswd anotherUser
```
Use the same hash format and cost for every entry in the file. If the
entries differ, the time taken to reject a wrong password can reveal
which usernames exist.
The password file can be updated while rclone is running.
When authentication is enabled every request must be authenticated,
including HTTP OPTIONS requests, as the answer to an OPTIONS request can
reveal whether a path exists. Health checks or monitoring which use
unauthenticated OPTIONS requests will need to send credentials. The
exception is browser CORS preflight requests when `--allow-origin`
is set, which are answered without authentication as browsers never send
credentials with them.
Use `--realm` to set the authentication realm.
Use `--salt` to change the password hashing salt from the default.
@@ -313,7 +340,7 @@ at all times. The buffered data is bound to one open file and won't be
shared.
This flag is a upper limit for the used memory per open file. The
buffer will only use memory for data that is downloaded but not not
buffer will only use memory for data that is downloaded but not
yet read. If the buffer is empty, only a small amount of memory will
be used.
@@ -368,7 +395,8 @@ longest. This cache flushing strategy is efficient and more relevant
files are likely to remain cached.
The `--vfs-cache-max-age` will evict files from the cache
after the set time since last access has passed. The default value of
after the set time since last access has passed; it is based on access time,
not on when the file was first added to the cache. The default value of
1 hour will start evicting files from cache that haven't been accessed
for 1 hour. When a cached file is accessed the 1 hour timer is reset to 0
and will wait for 1 more hour before evicting. Specify the time with
@@ -522,7 +550,7 @@ Rclone reads `--vfs-read-chunk-streams` chunks of size
`--vfs-read-chunk-size` concurrently. The size for each read will stay
constant.
This improves performance performance massively on high latency links
This improves performance massively on high latency links
or very high bandwidth links to high performance object stores.
Some experimentation will be needed to find the optimum values of
@@ -765,9 +793,10 @@ This config generated must have this extra parameter
- `_root` - root to use for the backend
And it may have this parameter
And it may have these parameters
- `_obscure` - comma separated strings for parameters to obscure
- `_secret_access_key` - the secret for S3 access key auth (see below)
If password authentication was used by the client, input to the proxy
process (on STDIN) would look similar to this:
@@ -775,7 +804,8 @@ process (on STDIN) would look similar to this:
```json
{
"user": "me",
"pass": "mypassword"
"pass": "mypassword",
"client_ip": "192.168.1.1"
}
```
@@ -785,10 +815,44 @@ proxy process (on STDIN) would look similar to this:
```json
{
"user": "me",
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf"
"public_key": "AAAAB3NzaC1yc2EAAAADAQABAAABAQDuwESFdAe14hVS6omeyX7edc...JQdf",
"client_ip": "192.168.1.1"
}
```
If the client authenticated with an S3 access key (`rclone serve s3`),
the client never sends its secret, only a signature made with it, so
the input contains just the access key ID as the `user` with no `pass`
or `public_key`:
```json
{
"user": "AKIAIOSFODNN7EXAMPLE",
"client_ip": "192.168.1.1"
}
```
In this case the program must look up the secret access key for that
access key ID and return it in the `_secret_access_key` field of the
output. Rclone then uses that secret to verify the signature on the
request, refusing the request if it does not match. This means the
proxy program is the source of truth for both the credentials and the
backend they map to. If the program does not return
`_secret_access_key` or returns it empty the request is refused.
The program's answer for an access key ID is cached (see below) but
is checked with the program again after 5 minutes even if the access
key ID is in constant use, so revoking an access key ID in the
program takes effect within 5 minutes. A rotated secret takes effect
on the first request signed with it.
The `client_ip` key holds the IP address the client connected from,
without a port number. It can be used to restrict logins to certain
networks, or to log authentication attempts centrally. It is omitted if
the client has no IP address, for example when connecting over a unix
socket. Note that if rclone is behind a reverse proxy this will be the
address of the reverse proxy and not the original client.
And as an example return this on STDOUT
```json
@@ -814,11 +878,20 @@ to make proxy to many different sftp backends, you could make the
in the output and the user to `user`. For security you'd probably want
to restrict the `host` to a limited list.
An internal cache of backends is keyed on the `user` and a hash of the
`pass` or `public_key`. This means that if a user's password or
public-key changes, or the proxy returns different config parameters
(eg a rotated `api_key`), a fresh backend will be created on the next
request rather than the cached one being reused.
An internal cache of backends is keyed on the `user`, the kind of
credential presented, a hash of the `pass` or `public_key`, and the
`client_ip`. This means that if a user's password or public-key
changes, the client connects from a new IP address, or the proxy
returns different config parameters (eg a rotated `api_key`), a fresh
backend will be created on the next request rather than the cached one
being reused. A backend created for a public key login is never
reused for a password login, so a public key cannot be sent as a
password to skip the program.
A cached answer is checked with the program again after 5 minutes even
if the user is in constant use, so revoking a user in the program
takes effect within 5 minutes. For `serve sftp` this applies to new
logins - an existing connection is not closed.
This can be used to build general purpose proxies to any kind of
backend that rclone supports.
docs/content/commands/rclone_test_speed.md
+1 -1
View File
@@ -16,7 +16,7 @@ This command runs a series of uploads and downloads to the remote, measuring
and printing the speed of each test using varying file sizes and numbers of
files.
Test time can be innaccurate with small file caps and large files. As it
Test time can be inaccurate with small file caps and large files. As it
uses the results of an initial test to determine how many files to use in
each subsequent test.
docs/content/drive.md
+2 -2
View File
@@ -1131,7 +1131,7 @@ recommended usage is using the flag form --drive-size-as-quota when
doing rclone ls/lsl/lsf/lsjson/etc only.
If you do use this flag for syncing (not recommended) then you will
need to use --ignore size also.
need to use --ignore-size also.
Properties:
@@ -1838,7 +1838,7 @@ The result is a JSON array of matches, for example:
"webViewLink": "https://drive.google.com/file/d/0AxBe_CDEF4zkGHI4d0FjYko2QkD/view?usp=drivesdk\u0026resourcekey=0-ABCDEFGHIXJQpIGqBJq3MC"
}
]
```
```console
### rescue
docs/content/flags.md
+31 -9
View File
@@ -98,6 +98,7 @@ Flags used for check commands.
Flags for general networking and HTTP stuff.
```
--allow-https-downgrade Follow redirects from HTTPS to HTTP (insecure)
--bind string Local address to bind to for outgoing connections, IPv4, IPv6 or name
--bwlimit BwTimetable Bandwidth limit in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
--bwlimit-file BwTimetable Bandwidth limit per file in KiB/s, or use suffix B|K|M|G|T|P or a full timetable
@@ -121,7 +122,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.75.0")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.76.0")
```
@@ -302,11 +303,6 @@ Flags to control the Remote Control API.
--rc-template string User-specified template
--rc-user string User name for authentication
--rc-user-from-header string User name from a defined HTTP header
--rc-web-fetch-url string URL to fetch the releases for webgui (default "https://api.github.com/repos/rclone/rclone-webui-react/releases/latest")
--rc-web-gui Launch WebGUI on localhost
--rc-web-gui-force-update Force update to latest version of web gui
--rc-web-gui-no-open-browser Don't open the browser automatically
--rc-web-gui-update Check and update to latest version of web gui
```
@@ -368,8 +364,10 @@ Backend-only flags (these can be set in the config file also).
--azureblob-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,RightPeriod,InvalidUtf8)
--azureblob-endpoint string Endpoint for the service
--azureblob-env-auth Read credentials from runtime (environment variables, CLI or MSI)
--azureblob-hns Tristate Set if the storage account has a hierarchical namespace (default unset)
--azureblob-key string Storage Account Shared Key
--azureblob-list-chunk int Size of blob list (default 5000)
--azureblob-list-parallelism int Number of parallel shards to list a directory with
--azureblob-msi-client-id string Object ID of the user-assigned MSI to use, if any
--azureblob-msi-mi-res-id string Azure resource ID of the user-assigned MSI to use, if any
--azureblob-msi-object-id string Object ID of the user-assigned MSI to use, if any
@@ -382,6 +380,7 @@ Backend-only flags (these can be set in the config file also).
--azureblob-tenant string ID of the service principal's tenant. Also called its directory ID
--azureblob-upload-concurrency int Concurrency for multipart uploads (default 16)
--azureblob-upload-cutoff string Cutoff for switching to chunked upload (<= 256 MiB) (deprecated)
--azureblob-use-arrow-list Use the Apache Arrow listing format
--azureblob-use-az Use Azure CLI tool az for authentication
--azureblob-use-copy-blob Whether to use the Copy Blob API when copying to the same storage account (default true)
--azureblob-use-emulator Uses local storage emulator if provided as 'true'
@@ -514,6 +513,12 @@ Backend-only flags (these can be set in the config file also).
--doi-doi string The DOI or the doi.org URL
--doi-doi-resolver-api-url string The URL of the DOI resolver API to use
--doi-provider string DOI provider
--dosya-api-key string Your API Key, get it from https://dosya.dev/profile under "API keys"
--dosya-api-url string API base URL (default "https://api.dosya.dev")
--dosya-description string Description of the remote
--dosya-encoding Encoding The encoding for the backend (default Slash,LtGt,DoubleQuote,SingleQuote,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
--dosya-upload-concurrency int Concurrency for multipart uploads (default 4)
--dosya-workspace-id string Your workspace ID
--drime-access-token string API Access token
--drime-chunk-size SizeSuffix Chunk size to use for uploading (default 5Mi)
--drime-description string Description of the remote
@@ -736,6 +741,8 @@ Backend-only flags (these can be set in the config file also).
--http-headers CommaSepList Set HTTP headers for all transactions
--http-no-escape Do not escape URL metacharacters in path names
--http-no-head Don't use HEAD requests
--http-no-head-size SizeSuffix Set the reported file size for --http-no-head (default off)
--http-no-head-time Time Set a default modification time for --http-no-head (default off)
--http-no-slash Set this if the site doesn't end directories with /
--http-url string URL of HTTP host to connect to
--huaweidrive-auth-url string Auth server URL
@@ -795,6 +802,17 @@ Backend-only flags (these can be set in the config file also).
--jottacloud-token-url string Token server url
--jottacloud-trashed-only Only show files that are in the trash
--jottacloud-upload-resume-limit SizeSuffix Files bigger than this can be resumed if the upload fail's (default 10Mi)
--kdrive-auth-url string Auth server URL
--kdrive-chunk-size SizeSuffix Files above this size are uploaded in chunks of this size (default 20Mi)
--kdrive-client-id string OAuth Client Id
--kdrive-description string Description of the remote
--kdrive-drive-id string ID of the kDrive to use
--kdrive-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot)
--kdrive-endpoint string The API endpoint to use
--kdrive-root-folder-id string The folder to use as root
--kdrive-token string OAuth Access Token as a JSON blob
--kdrive-token-url string Token server url
--kdrive-upload-concurrency int Concurrency for chunked uploads (default 4)
--koofr-description string Description of the remote
--koofr-encoding Encoding The encoding for the backend (default Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot)
--koofr-endpoint string The Koofr API endpoint to use
@@ -880,6 +898,7 @@ Backend-only flags (these can be set in the config file also).
--onedrive-root-folder-id string ID of the root folder
--onedrive-server-side-across-configs Deprecated: use --server-side-across-configs instead
--onedrive-tenant string ID of the service principal's tenant. Also called its directory ID
--onedrive-tenant-api-version string The SharePoint API version to use with tenant_url (default "v2.0")
--onedrive-tenant-url string The tenant URL for non-admin OneDrive access
--onedrive-token string OAuth Access Token as a JSON blob
--onedrive-token-url string Token server url
@@ -894,7 +913,7 @@ Backend-only flags (these can be set in the config file also).
--oos-decompress If set this will decompress gzip encoded objects
--oos-description string Description of the remote
--oos-disable-checksum Don't store MD5 checksum with object metadata
--oos-encoding Encoding The encoding for the backend (default Slash,InvalidUtf8,Dot)
--oos-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--oos-endpoint string Endpoint for Object storage API
--oos-leave-parts-on-error If true avoid calling abort upload on a failure, leaving all successfully uploaded parts for manual recovery
--oos-max-upload-parts int Maximum number of parts in a multipart upload (default 10000)
@@ -1089,7 +1108,7 @@ Backend-only flags (these can be set in the config file also).
--sftp-disable-concurrent-reads If set don't use concurrent reads
--sftp-disable-concurrent-writes If set don't use concurrent writes
--sftp-disable-hashcheck Disable the execution of SSH commands to determine if remote file hashing is available
--sftp-encoding Encoding The encoding for the backend (default Slash,Del,Ctl,Dot)
--sftp-encoding Encoding The encoding for the backend (default Slash,Dot)
--sftp-hashes CommaSepList Comma separated list of supported checksum types
--sftp-host string SSH host to connect to
--sftp-host-key-algorithms SpaceSepList Space separated list of host key algorithms, ordered by preference
@@ -1104,6 +1123,7 @@ Backend-only flags (these can be set in the config file also).
--sftp-known-hosts-file string Optional path to known_hosts file
--sftp-macs SpaceSepList Space separated list of MACs (message authentication code) algorithms, ordered by preference
--sftp-md5sum-command string The command used to read MD5 hashes
--sftp-multithread-upload Set this to use multi-thread (multi-connection) uploads
--sftp-pass string SSH password, leave blank to use ssh-agent (obscured)
--sftp-path-override string Override path used by SSH shell commands
--sftp-pin-host-key Pin the server host key on first connection (Trust On First Use)
@@ -1166,7 +1186,8 @@ Backend-only flags (these can be set in the config file also).
--smb-port int SMB port number (default 445)
--smb-spn string Service principal name
--smb-use-kerberos Use Kerberos authentication
--smb-user string SMB username (default "$USER")
--smb-user string SMB username
--smb-workstation string Workstation name sent in NTLM authentication
--storj-access-grant string Access grant
--storj-api-key string API key
--storj-description string Description of the remote
@@ -1241,6 +1262,7 @@ Backend-only flags (these can be set in the config file also).
--webdav-url string URL of http host to connect to
--webdav-user string User name
--webdav-vendor string Name of the WebDAV site/service/software you are using
--yandex-app-folder Use the application folder as the root
--yandex-auth-url string Auth server URL
--yandex-client-credentials Use client credentials OAuth flow
--yandex-client-id string OAuth Client Id
docs/content/http.md
+43
View File
@@ -212,6 +212,13 @@ For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'
You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
The headers are only sent to the host in the configured URL. If the
server redirects to another host (including a subdomain or a different
port) the headers are not sent to it, or to any further hop in that
redirect chain. When headers are set, a redirect from https to http is
refused as it would send them in cleartext, even with
--allow-https-downgrade.
Properties:
- Config: headers
@@ -265,6 +272,42 @@ Properties:
- Type: bool
- Default: false
#### --http-no-head-size
Set the reported file size for --http-no-head.
When HEAD requests are disabled, rclone normally reports an unknown file size.
Set this option to a non-negative value to report that size for every file,
including 0. Leave it unset (the default, shown as "off") to keep reporting
an unknown size.
The configured size is authoritative. If it does not match the actual file
size, transfers may fail their size check and retry unless --ignore-size is
used. It can also make multi-thread transfers request incorrect ranges and
make mounted files unreadable past the reported size.
Properties:
- Config: no_head_size
- Env Var: RCLONE_HTTP_NO_HEAD_SIZE
- Type: SizeSuffix
- Default: off
#### --http-no-head-time
Set a default modification time for --http-no-head.
When HEAD requests are disabled, rclone normally reports an unset modification
time. Set this option to report the specified time for every file instead.
Changing the configured time later makes every file appear modified to a sync.
Properties:
- Config: no_head_time
- Env Var: RCLONE_HTTP_NO_HEAD_TIME
- Type: Time
- Default: off
#### --http-description
Description of the remote.
docs/content/local.md
+15 -6
View File
@@ -583,6 +583,13 @@ deduplicated by default (i.e. having two identical files does not consume more
storage than having just one.) However, for use cases where data redundancy is
preferable, --local-no-clone can be used to disable cloning and force "deep" copies.
Cloning also preserves macOS-specific metadata such as extended attributes,
which a normal copy does not, so this metadata is lost when cloning is disabled.
If the destination file already exists, cloning replaces it with a new file,
even with --inplace, so hard links to it are broken. Use --local-no-clone to
update existing files in place.
Currently, cloning is only supported when using APFS on macOS (support for other
platforms may be added in the future.)
@@ -672,10 +679,10 @@ Properties:
Make out-of-space errors fatal during transfers.
When enabled, an ENOSPC error during a write returns a fatal error so
that rclone aborts rather than retrying the operation. Useful for
backup scripts that should halt loudly on a full disk rather than spin
retrying.
When enabled, an out of space error while writing, creating a directory,
or creating a file returns a fatal error so that rclone aborts rather
than retrying the operation. Useful for backup scripts that should halt
loudly on a full disk rather than spin retrying.
Properties:
@@ -773,8 +780,10 @@ Metadata is supported on files and directories.
When restoring metadata with `--metadata` rclone applies the
"mode", "uid" and "gid" from the source. These come from the source
remote which may not be trusted, so restoring metadata as root from an
untrusted source can change file ownership and is not recommended. The
setuid, setgid and sticky bits are not restored by default - see the
untrusted source can change file ownership and is not recommended. A
user other than root keeps its own uid and sets the gid only to a group
it is a member of, as rsync does. The setuid, setgid and sticky bits are
not restored by default - see the
`--local-metadata-restore-special-bits` flag.
Here are the possible system metadata items for the local backend.
docs/content/onedrive.md
+35 -27
View File
@@ -528,7 +528,7 @@ Properties:
The tenant URL for non-admin OneDrive access.
Set this to your SharePoint tenant URL to use the SharePoint v2.0 API
Set this to your SharePoint tenant URL to use the SharePoint API
endpoint instead of the standard Microsoft Graph API. This allows
accessing business OneDrive without admin consent.
@@ -545,6 +545,22 @@ Properties:
- Type: string
- Required: false
#### --onedrive-tenant-api-version
The SharePoint API version to use with tenant_url.
Set this to the SharePoint API version matching the browser-extracted
access token. For example, use v2.1 with a driveAccessTokenV21 token.
This only applies when tenant_url is set.
Properties:
- Config: tenant_api_version
- Env Var: RCLONE_ONEDRIVE_TENANT_API_VERSION
- Type: string
- Default: "v2.0"
#### --onedrive-chunk-size
Chunk size to upload files with - must be multiple of 320k (327,680 bytes).
@@ -671,7 +687,7 @@ Properties:
Remove all versions on modifying operations.
Onedrive for business creates versions when rclone uploads new files
Onedrive creates versions when rclone uploads new files
overwriting an existing one and when it sets the modification time.
These versions take up space out of the quota.
@@ -679,9 +695,6 @@ These versions take up space out of the quota.
This flag checks for versions after file upload and setting
modification time and removes all but the last version.
**NB** Onedrive personal can't currently delete versions so don't use
this flag there.
Properties:
@@ -697,8 +710,7 @@ Permanently delete files on removal.
Normally files will get sent to the recycle bin on deletion. Setting
this flag causes them to be permanently deleted. Use with care.
OneDrive personal accounts do not support the permanentDelete API,
it only applies to OneDrive for Business and SharePoint document libraries.
This works with OneDrive for Business, SharePoint document libraries, and OneDrive personal accounts, including free accounts.
Properties:
@@ -744,12 +756,17 @@ Properties:
- Creates a read-write link to the item.
- "embed"
- Creates an embeddable link to the item.
- Only available in OneDrive personal.
#### --onedrive-link-password
Set the password for links created by the link command.
At the time of writing this only works with OneDrive personal paid accounts.
At the time of writing this works with OneDrive for Business and
OneDrive personal paid accounts.
OneDrive personal free accounts can't set a password or an expiry time
(with --expire) on links.
Properties:
@@ -848,21 +865,12 @@ Setting this flag speeds up these things greatly:
rclone size onedrive:
rclone rc vfs/refresh recursive=true
**However** the delta listing API **only** works at the root of the
drive. If you use it not at the root then it recurses from the root
and discards all the data that is not under the directory you asked
for. So it will be correct but may not be very efficient.
This is why this flag is not set as the default.
As a rule of thumb if nearly all of your data is under rclone's root
directory (the `root/directory` in `onedrive:root/directory`) then
using this flag will be a big performance win. If your data is
mostly not under the root then using this flag will be a big
performance loss.
It is recommended if you are mounting your onedrive at the root
(or near the root when using crypt) and using rclone `rc vfs/refresh`.
Rclone asks for the delta listing of the directory being listed. If
the drive only supports delta listings at the root of the drive (as
some older OneDrive for Business and SharePoint drives do) then
rclone lists from the root and discards all the data that is not
under the directory you asked for. So it will be correct but may not
be very efficient.
Properties:
@@ -1060,7 +1068,7 @@ Here are the possible system metadata items for the onedrive backend.
| Name | Help | Type | Example | Read Only |
|------|------|------|---------|-----------|
| btime | Time of file birth (creation) with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | N |
| btime | Time of file birth (creation) with S accuracy. | RFC 3339 | 2006-01-02T15:04:05Z | N |
| content-type | The MIME type of the file. | string | text/plain | **Y** |
| created-by-display-name | Display name of the user that created the item. | string | John Doe | **Y** |
| created-by-id | ID of the user that created the item. | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
@@ -1069,14 +1077,14 @@ Here are the possible system metadata items for the onedrive backend.
| last-modified-by-display-name | Display name of the user that last modified the item. | string | John Doe | **Y** |
| last-modified-by-id | ID of the user that last modified the item. | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| malware-detected | Whether OneDrive has detected that the item contains malware. | boolean | true | **Y** |
| mtime | Time of last modification with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | N |
| mtime | Time of last modification with S accuracy. | RFC 3339 | 2006-01-02T15:04:05Z | N |
| package-type | If present, indicates that this item is a package instead of a folder or file. Packages are treated like files in some contexts and folders in others. | string | oneNote | **Y** |
| permissions | Permissions in a JSON dump of OneDrive format. Enable with --onedrive-metadata-permissions. Properties: id, grantedTo, grantedToIdentities, invitation, inheritedFrom, link, roles, shareId | JSON | {} | N |
| shared-by-id | ID of the user that shared the item (if shared). | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| shared-owner-id | ID of the owner of the shared item (if shared). | string | 48d31887-5fad-4d73-a9f5-3c356e68a038 | **Y** |
| shared-scope | If shared, indicates the scope of how the item is shared: anonymous, organization, or users. | string | users | **Y** |
| shared-time | Time when the item was shared, with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | **Y** |
| utime | Time of upload with S accuracy (mS for OneDrive Personal). | RFC 3339 | 2006-01-02T15:04:05Z | **Y** |
| shared-time | Time when the item was shared, with S accuracy. | RFC 3339 | 2006-01-02T15:04:05Z | **Y** |
| utime | Time of upload with S accuracy. | RFC 3339 | 2006-01-02T15:04:05Z | **Y** |
See the [metadata](/docs/#metadata) docs for more info.
docs/content/oracleobjectstorage/_index.md
+1 -1
View File
@@ -610,7 +610,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_OOS_ENCODING
- Type: Encoding
- Default: Slash,InvalidUtf8,Dot
- Default: Slash,CrLf,InvalidUtf8,Dot
#### --oos-leave-parts-on-error
docs/content/pikpak.md
+2
View File
@@ -196,6 +196,8 @@ Properties:
Files bigger than this will be cached on disk to calculate hash if required.
Files of unknown size are always cached on disk.
Properties:
- Config: hash_memory_limit
docs/content/rc.md
+104 -83
View File
@@ -1225,6 +1225,7 @@ Returns the following values:
"totalTransfers": total number of transfers in the group,
"transferTime" : total time spent on running jobs,
"transfers": number of transferred files,
"updatedDirs": number of directories updated (modtime or metadata set),
"transferring": an array of currently active file transfers:
[
{
@@ -1490,8 +1491,6 @@ Results:
- runningIds - array of integer job ids that are running
- finishedIds - array of integer job ids that are finished
**Authentication is not required for this call.**
### job/status: Reads the status of the job ID {#job-status}
Parameters:
@@ -1512,8 +1511,6 @@ Results:
- output - output of the job as would have been returned if called synchronously
- progress - output of the progress related to the underlying job
**Authentication is not required for this call.**
### job/stop: Stop the running job {#job-stop}
Parameters:
@@ -1843,6 +1840,25 @@ This command does not have a command line equivalent so use this instead:
rclone rc --loopback operations/fsinfo fs=remote:
### operations/getfile: Get a single file into a JSON response. {#operations-getfile}
This takes the following parameters:
- fs - a remote name string e.g. "drive:path/to/file" or "drive:path/to/dir"
- remote - (optional) a path within that remote e.g. "file.txt"
- offset - (optional) start reading at offset N (or from end if negative) (default 0)
- count - (optional) only read N bytes (default -1)
- head - (optional) only read the first N bytes (default 0)
- tail - (optional) only read the last N bytes (default 0)
- maxSize - (optional) maximum size limit in bytes (can only lower the built-in maximum) (default 2097152 / 2MiB)
- base64 - (optional) if true, returns the file contents base64-encoded (default false)
Returns:
- result - contents of the file as string or base64-encoded string
For streaming file content over HTTP with Range support, use the --rc-serve flag instead.
### operations/hashsum: Produces a hashsum file for all the objects in the path. {#operations-hashsum}
Produces a hash file for all the objects in the path using the hash
@@ -1917,6 +1933,26 @@ Example:
See the [hashsum](/commands/rclone_hashsum/) command for more information on the above.
### operations/index: Write static directory listings into a remote {#operations-index}
This takes the following parameters:
- fs - a remote name string e.g. "drive:"
- opt - a dictionary of options to control the indexing (optional)
- output - list of listings to write in each directory as "NAME=FORMAT" (default ["index.html=html"])
- rules - which directories get listings, e.g. {"ExcludeRule": ["/private/**"]}
- maxDepth - only make listings this many directories deep (default -1 for no limit)
- linkIndex - link to "dir/NAME" rather than "dir/"
- dirTime - how to work out directory times: "newest", "dir" or "none"
- noModTime - don't show modification times
- rewrite - write every listing even if it is unchanged
- changed - list of changed paths for a partial run, directories ending in "/"
- changedFrom - list of files of changed paths, one per line
- changedCombined - list of files in the sync --combined report format
- changedMaxDirs - do a full run if a partial one would list more directories than this
See the [index](/commands/rclone_index/) command for more information on the above.
### operations/list: List the given remote and path in JSON format {#operations-list}
This takes the following parameters:
@@ -2136,85 +2172,6 @@ And this sets NOTICE level logs (normal without -v)
rclone rc options/set --json '{"main": {"LogLevel": "NOTICE"}}'
### pluginsctl/addPlugin: Add a plugin using url {#pluginsctl-addPlugin}
Used for adding a plugin to the webgui.
This takes the following parameters:
- url - http url of the github repo where the plugin is hosted (http://github.com/rclone/rclone-webui-react).
Example:
rclone rc pluginsctl/addPlugin
### pluginsctl/getPluginsForType: Get plugins with type criteria {#pluginsctl-getPluginsForType}
This shows all possible plugins by a mime type.
This takes the following parameters:
- type - supported mime type by a loaded plugin e.g. (video/mp4, audio/mp3).
- pluginType - filter plugins based on their type e.g. (DASHBOARD, FILE_HANDLER, TERMINAL).
Returns:
- loadedPlugins - list of current production plugins.
- testPlugins - list of temporarily loaded development plugins, usually running on a different server.
Example:
rclone rc pluginsctl/getPluginsForType type=video/mp4
### pluginsctl/listPlugins: Get the list of currently loaded plugins {#pluginsctl-listPlugins}
This allows you to get the currently enabled plugins and their details.
This takes no parameters and returns:
- loadedPlugins - list of current production plugins.
- testPlugins - list of temporarily loaded development plugins, usually running on a different server.
E.g.
rclone rc pluginsctl/listPlugins
### pluginsctl/listTestPlugins: Show currently loaded test plugins {#pluginsctl-listTestPlugins}
Allows listing of test plugins with the rclone.test set to true in package.json of the plugin.
This takes no parameters and returns:
- loadedTestPlugins - list of currently available test plugins.
E.g.
rclone rc pluginsctl/listTestPlugins
### pluginsctl/removePlugin: Remove a loaded plugin {#pluginsctl-removePlugin}
This allows you to remove a plugin using it's name.
This takes parameters:
- name - name of the plugin in the format `author`/`plugin_name`.
E.g.
rclone rc pluginsctl/removePlugin name=rclone/video-plugin
### pluginsctl/removeTestPlugin: Remove a test plugin {#pluginsctl-removeTestPlugin}
This allows you to remove a plugin using it's name.
This takes the following parameters:
- name - name of the plugin in the format `author`/`plugin_name`.
Example:
rclone rc pluginsctl/removeTestPlugin name=rclone/rclone-webui-react
### rc/error: This returns an error {#rc-error}
This returns an error with the input as part of its error string.
@@ -2389,6 +2346,8 @@ This takes the following parameters:
- path1 (required) - (string) a remote directory string e.g. `drive:path1`
- path2 (required) - (string) a remote directory string e.g. `drive:path2`
- dryRun - (bool) dry-run mode
- maxDelete - (int) Safety check on maximum percentage of deleted files
allowed. If exceeded, the bisync run will abort. (default: 50%)
- backupDir1 - (string) --backup-dir for Path1. Must be a non-overlapping path on
the same remote.
- backupDir2 - (string) --backup-dir for Path2. Must be a non-overlapping path on
@@ -2418,6 +2377,8 @@ unavailable. (warning: may be slow and use lots of data!)
using with --verbose
- ignoreListingChecksum - (bool) Do not use checksums for listings (add --ignore-
checksum to additionally skip post-copy checksum checks)
- maxDeleteRenamesAware - (bool) Exclude tracked renames from the --max-delete
safety check (requires --track-renames)
- maxLock - (Duration) Consider lock files older than this to be expired
(default: 0 (never expire)) (minimum: 2m)
- noCleanup - (bool) Retain working files (useful for troubleshooting and
@@ -2451,7 +2412,27 @@ This takes the following parameters:
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
- createEmptySrcDirs - create empty src directories on destination if set
- combined - make a combined report of changes (default false)
- missingOnSrc - report all files missing from the source (default false)
- missingOnDst - report all files missing from the destination (default false)
- match - report all matching files (default false)
- differ - report all non-matching files (default false)
- error - report all files with errors (hashing or reading) (default false)
- destAfter - report all files that exist on the destination post-copy (default false)
Returns:
- combined - array of strings of combined report of changes
- missingOnSrc - array of strings of all files missing from the source
- missingOnDst - array of strings of all files missing from the destination
- match - array of strings of all matching files
- differ - array of strings of all non-matching files
- error - array of strings of all files with errors (hashing or reading)
- destAfter - array of strings of all files that exist on the destination post-copy
Each report is only returned if its parameter is set to true. If the
operation fails the reports are only available if it was run with
`_async`, as part of the job output.
See the [copy](/commands/rclone_copy/) command for more information on the above.
@@ -2463,7 +2444,27 @@ This takes the following parameters:
- dstFs - a remote name string e.g. "drive:dst" for the destination
- createEmptySrcDirs - create empty src directories on destination if set
- deleteEmptySrcDirs - delete empty src directories if set
- combined - make a combined report of changes (default false)
- missingOnSrc - report all files missing from the source (default false)
- missingOnDst - report all files missing from the destination (default false)
- match - report all matching files (default false)
- differ - report all non-matching files (default false)
- error - report all files with errors (hashing or reading) (default false)
- destAfter - report all files that exist on the destination post-move (default false)
Returns:
- combined - array of strings of combined report of changes
- missingOnSrc - array of strings of all files missing from the source
- missingOnDst - array of strings of all files missing from the destination
- match - array of strings of all matching files
- differ - array of strings of all non-matching files
- error - array of strings of all files with errors (hashing or reading)
- destAfter - array of strings of all files that exist on the destination post-move
Each report is only returned if its parameter is set to true. If the
operation fails the reports are only available if it was run with
`_async`, as part of the job output.
See the [move](/commands/rclone_move/) command for more information on the above.
@@ -2474,7 +2475,27 @@ This takes the following parameters:
- srcFs - a remote name string e.g. "drive:src" for the source
- dstFs - a remote name string e.g. "drive:dst" for the destination
- createEmptySrcDirs - create empty src directories on destination if set
- combined - make a combined report of changes (default false)
- missingOnSrc - report all files missing from the source (default false)
- missingOnDst - report all files missing from the destination (default false)
- match - report all matching files (default false)
- differ - report all non-matching files (default false)
- error - report all files with errors (hashing or reading) (default false)
- destAfter - report all files that exist on the destination post-sync (default false)
Returns:
- combined - array of strings of combined report of changes
- missingOnSrc - array of strings of all files missing from the source
- missingOnDst - array of strings of all files missing from the destination
- match - array of strings of all matching files
- differ - array of strings of all non-matching files
- error - array of strings of all files with errors (hashing or reading)
- destAfter - array of strings of all files that exist on the destination post-sync
Each report is only returned if its parameter is set to true. If the
operation fails the reports are only available if it was run with
`_async`, as part of the job output.
See the [sync](/commands/rclone_sync/) command for more information on the above.
docs/content/s3.md
+285 -74
View File
@@ -975,7 +975,7 @@ mode from COMPLIANCE to GOVERNANCE while preserving the original retention date:
<!-- autogenerated options start - DO NOT EDIT - instead edit fs.RegInfo in backend/s3/s3.go and run make backenddocs to verify --> <!-- markdownlint-disable-line line-length -->
### Standard options
Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, BizflyCloud, Ceph, ChinaMobile, Cloudflare, Cubbit, DigitalOcean, Dreamhost, Exaba, Fastly, FileLu, FlashBlade, GCS, HCP, Hetzner, HuaweiOBS, IBMCOS, IDrive, ImpossibleCloud, Intercolo, IONOS, Leviia, Liara, Linode, LyveCloud, Magalu, Mega, Minio, Netease, Outscale, OVHcloud, Petabox, Qiniu, Rabata, RackCorp, Rclone, Scaleway, Scality, SeaweedFS, Selectel, Servercore, SpectraLogic, Storj, Synology, TencentCOS, US3, Wasabi, Zadara, Zata, ZeroServices, Other).
Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Autonomys, BizflyCloud, Cellar, Ceph, ChinaMobile, Cloudflare, Cubbit, DigitalOcean, Dreamhost, Exaba, Fastly, FileLu, FlashBlade, GCS, HCP, Hetzner, HuaweiOBS, IBMCOS, IDrive, ImpossibleCloud, Infomaniak, Intercolo, IONOS, Leviia, Liara, Linode, LyveCloud, Magalu, Mega, Minio, Neon, Netease, Outscale, OVHcloud, Petabox, Qiniu, Rabata, RackCorp, Rclone, RelAix, Scaleway, Scality, SeaweedFS, Selectel, Servercore, SpectraLogic, Storj, Synology, TencentCOS, US3, Wasabi, Zadara, Zata, ZeroServices, Other).
#### --s3-provider
@@ -994,8 +994,12 @@ Properties:
- Alibaba Cloud Object Storage System (OSS) formerly Aliyun
- "ArvanCloud"
- Arvan Cloud Object Storage (AOS)
- "Autonomys"
- Autonomys Auto Drive
- "BizflyCloud"
- Bizfly Cloud Simple Storage
- "Cellar"
- Clever Cloud Cellar Object Storage
- "Ceph"
- Ceph Object Storage
- "ChinaMobile"
@@ -1030,6 +1034,8 @@ Properties:
- IDrive e2
- "ImpossibleCloud"
- Impossible Cloud Object Storage
- "Infomaniak"
- Infomaniak Swiss Backup
- "Intercolo"
- Intercolo Object Storage
- "IONOS"
@@ -1048,6 +1054,8 @@ Properties:
- MEGA S4 Object Storage
- "Minio"
- Minio Object Storage
- "Neon"
- Neon Object Storage
- "Netease"
- Netease Object Storage (NOS)
- "Outscale"
@@ -1064,6 +1072,8 @@ Properties:
- RackCorp Object Storage
- "Rclone"
- Rclone S3 Server
- "RelAix"
- RelAix.Storage
- "Scaleway"
- Scaleway Object Storage
- "Scality"
@@ -1149,7 +1159,7 @@ Properties:
- Config: region
- Env Var: RCLONE_S3_REGION
- Provider: AWS,BizflyCloud,Ceph,Cloudflare,Cubbit,DigitalOcean,Dreamhost,Exaba,Fastly,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,ImpossibleCloud,Intercolo,IONOS,Leviia,LyveCloud,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Scality,SeaweedFS,Selectel,Servercore,Synology,Wasabi,Zadara,Zata,ZeroServices,Other
- Provider: AWS,BizflyCloud,Ceph,Cloudflare,Cubbit,DigitalOcean,Dreamhost,Exaba,Fastly,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,LyveCloud,Minio,Neon,Netease,Outscale,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Scality,SeaweedFS,Selectel,Servercore,Synology,Wasabi,Zadara,Zata,ZeroServices,Other
- Type: string
- Required: false
- Examples:
@@ -1284,6 +1294,9 @@ Properties:
- "eu-central"
- EU Central
- Provider: Fastly
- "eu-central-1"
- EU Central 1
- Provider: Fastly
- "eu-south-1"
- EU South 1
- Provider: Fastly
@@ -1335,50 +1348,104 @@ Properties:
- "nbg1"
- Nuremberg
- Provider: Hetzner
- "af-north-1"
- AF-Cairo
- Provider: HuaweiOBS
- "af-south-1"
- AF-Johannesburg
- Provider: HuaweiOBS
- "ap-southeast-1"
- CN-Hong Kong
- Provider: HuaweiOBS
- "ap-southeast-2"
- AP-Bangkok
- Provider: HuaweiOBS
- "ap-southeast-3"
- AP-Singapore
- Provider: HuaweiOBS
- "cn-east-3"
- CN East-Shanghai1
- "ap-southeast-4"
- AP-Jakarta
- Provider: HuaweiOBS
- "ap-southeast-5"
- AP-Manila
- Provider: HuaweiOBS
- "cn-east-2"
- CN East-Shanghai2
- Provider: HuaweiOBS
- "cn-east-3"
- CN East-Shanghai1
- Provider: HuaweiOBS
- "cn-east-4"
- CN East-Qingdao
- Provider: HuaweiOBS
- "cn-east-5"
- CN East-Suzhou
- Provider: HuaweiOBS
- "cn-north-1"
- CN North-Beijing1
- Provider: HuaweiOBS
- "cn-north-2"
- CN North-Beijing2
- Provider: HuaweiOBS
- "cn-north-4"
- CN North-Beijing4
- Provider: HuaweiOBS
- "cn-north-9"
- CN North-Ulanqab
- Provider: HuaweiOBS
- "cn-north-11"
- CN North-Hebei
- Provider: HuaweiOBS
- "cn-north-12"
- CN North-Hohhot
- Provider: HuaweiOBS
- "cn-south-1"
- CN South-Guangzhou
- Provider: HuaweiOBS
- "ap-southeast-1"
- CN-Hong Kong
- "cn-south-2"
- CN South-Shenzhen
- Provider: HuaweiOBS
- "sa-argentina-1"
- LA-Buenos Aires1
- "cn-south-4"
- CN South-Guangzhou-InvitationOnly
- Provider: HuaweiOBS
- "sa-peru-1"
- LA-Lima1
- "cn-southwest-2"
- CN Southwest-Guiyang1
- Provider: HuaweiOBS
- "cn-southwest-3"
- CN Southwest-Chongqing
- Provider: HuaweiOBS
- "eu-west-101"
- EU-Dublin
- Provider: HuaweiOBS
- "la-north-2"
- LA-Mexico City2
- Provider: HuaweiOBS
- "la-south-2"
- LA-Santiago
- Provider: HuaweiOBS
- "me-east-1"
- ME-Riyadh
- Provider: HuaweiOBS
- "na-mexico-1"
- LA-Mexico City1
- Provider: HuaweiOBS
- "sa-chile-1"
- LA-Santiago2
- "ru-northwest-2"
- RU-Moscow2
- Provider: HuaweiOBS
- "sa-argentina-1"
- LA-Buenos Aires1
- Provider: HuaweiOBS
- "sa-brazil-1"
- LA-Sao Paulo1
- Provider: HuaweiOBS
- "ru-northwest-2"
- RU-Moscow2
- "sa-chile-1"
- LA-Santiago2
- Provider: HuaweiOBS
- "sa-peru-1"
- LA-Lima1
- Provider: HuaweiOBS
- "tr-west-1"
- TR-Istanbul
- Provider: HuaweiOBS
- "eu-central-2"
- Frankfurt, Germany
@@ -1401,6 +1468,10 @@ Properties:
- "us-east-1"
- New York, USA
- Provider: ImpossibleCloud
- "us-east-1"
- Default region for compatibility.
- All data is stored in Switzerland.
- Provider: Infomaniak
- "de-fra"
- Frankfurt, Germany
- Provider: Intercolo
@@ -1422,6 +1493,18 @@ Properties:
- "us-central-1"
- Lenexa, USA
- Provider: IONOS
- "us-east-1"
- US East (N. Virginia)
- Provider: Neon,Petabox,Rabata
- "us-east-2"
- US East (Ohio)
- Provider: Neon
- "eu-central-1"
- Europe (Frankfurt)
- Provider: Neon,Petabox
- "ap-southeast-1"
- Asia Pacific (Singapore)
- Provider: Neon,Petabox
- "eu-west-2"
- Paris, France
- Provider: Outscale
@@ -1479,15 +1562,6 @@ Properties:
- "rbx-archive"
- Roubaix, France (Cold Archive)
- Provider: OVHcloud
- "us-east-1"
- US East (N. Virginia)
- Provider: Petabox,Rabata
- "eu-central-1"
- Europe (Frankfurt)
- Provider: Petabox
- "ap-southeast-1"
- Asia Pacific (Singapore)
- Provider: Petabox
- "me-south-1"
- Middle East (Bahrain)
- Provider: Petabox
@@ -1595,6 +1669,9 @@ Properties:
- "pl-waw"
- Warsaw, Poland
- Provider: Scaleway
- "it-mil"
- Milan, Italy
- Provider: Scaleway
- "ru-1"
- St. Petersburg
- Provider: Selectel,Servercore
@@ -1661,7 +1738,7 @@ Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cloudflare,Cubbit,DigitalOcean,Dreamhost,Exaba,Fastly,FileLu,FlashBlade,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Mega,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Rclone,Scaleway,Scality,SeaweedFS,Selectel,Servercore,SpectraLogic,Storj,Synology,TencentCOS,US3,Wasabi,Zadara,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,Autonomys,BizflyCloud,Cellar,Ceph,ChinaMobile,Cloudflare,Cubbit,DigitalOcean,Dreamhost,Exaba,Fastly,FileLu,FlashBlade,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Mega,Minio,Neon,Netease,Outscale,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Rclone,RelAix,Scaleway,Scality,SeaweedFS,Selectel,Servercore,SpectraLogic,Storj,Synology,TencentCOS,US3,Wasabi,Zadara,Zata,ZeroServices,Other
- Type: string
- Required: false
- Examples:
@@ -1747,12 +1824,18 @@ Properties:
- "s3.ir-tbz-sh1.arvanstorage.ir"
- Tabriz Iran (Shahriar)
- Provider: ArvanCloud
- "s3.auto-drive.autonomys.xyz"
- Auto Drive mainnet S3 endpoint
- Provider: Autonomys
- "hn.ss.bfcplatform.vn"
- Hanoi endpoint
- Provider: BizflyCloud
- "hcm.ss.bfcplatform.vn"
- Ho Chi Minh endpoint
- Provider: BizflyCloud
- "cellar-c2.services.clever-cloud.com"
- Paris, France
- Provider: Cellar
- "eos-wuxi-1.cmecloud.cn"
- The default endpoint - a good choice if you are unsure.
- East China (Suzhou)
@@ -1889,6 +1972,9 @@ Properties:
- "eu-central.object.fastlystorage.app"
- EU Central
- Provider: Fastly
- "eu-central-1.object.fastlystorage.app"
- EU Central 1
- Provider: Fastly
- "eu-south-1.object.fastlystorage.app"
- EU South 1
- Provider: Fastly
@@ -1943,50 +2029,104 @@ Properties:
- "nbg1.your-objectstorage.com"
- Nuremberg
- Provider: Hetzner
- "obs.af-north-1.myhuaweicloud.com"
- AF-Cairo
- Provider: HuaweiOBS
- "obs.af-south-1.myhuaweicloud.com"
- AF-Johannesburg
- Provider: HuaweiOBS
- "obs.ap-southeast-1.myhuaweicloud.com"
- CN-Hong Kong
- Provider: HuaweiOBS
- "obs.ap-southeast-2.myhuaweicloud.com"
- AP-Bangkok
- Provider: HuaweiOBS
- "obs.ap-southeast-3.myhuaweicloud.com"
- AP-Singapore
- Provider: HuaweiOBS
- "obs.cn-east-3.myhuaweicloud.com"
- CN East-Shanghai1
- "obs.ap-southeast-4.myhuaweicloud.com"
- AP-Jakarta
- Provider: HuaweiOBS
- "obs.ap-southeast-5.myhuaweicloud.com"
- AP-Manila
- Provider: HuaweiOBS
- "obs.cn-east-2.myhuaweicloud.com"
- CN East-Shanghai2
- Provider: HuaweiOBS
- "obs.cn-east-3.myhuaweicloud.com"
- CN East-Shanghai1
- Provider: HuaweiOBS
- "obs.cn-east-4.myhuaweicloud.com"
- CN East-Qingdao
- Provider: HuaweiOBS
- "obs.cn-east-5.myhuaweicloud.com"
- CN East-Suzhou
- Provider: HuaweiOBS
- "obs.cn-north-1.myhuaweicloud.com"
- CN North-Beijing1
- Provider: HuaweiOBS
- "obs.cn-north-2.myhuaweicloud.com"
- CN North-Beijing2
- Provider: HuaweiOBS
- "obs.cn-north-4.myhuaweicloud.com"
- CN North-Beijing4
- Provider: HuaweiOBS
- "obs.cn-north-9.myhuaweicloud.com"
- CN North-Ulanqab
- Provider: HuaweiOBS
- "obs.cn-north-11.myhuaweicloud.com"
- CN North-Hebei
- Provider: HuaweiOBS
- "obs.cn-north-12.myhuaweicloud.com"
- CN North-Hohhot
- Provider: HuaweiOBS
- "obs.cn-south-1.myhuaweicloud.com"
- CN South-Guangzhou
- Provider: HuaweiOBS
- "obs.ap-southeast-1.myhuaweicloud.com"
- CN-Hong Kong
- "obs.cn-south-2.myhuaweicloud.com"
- CN South-Shenzhen
- Provider: HuaweiOBS
- "obs.sa-argentina-1.myhuaweicloud.com"
- LA-Buenos Aires1
- "obs.cn-south-4.myhuaweicloud.com"
- CN South-Guangzhou-InvitationOnly
- Provider: HuaweiOBS
- "obs.sa-peru-1.myhuaweicloud.com"
- LA-Lima1
- "obs.cn-southwest-2.myhuaweicloud.com"
- CN Southwest-Guiyang1
- Provider: HuaweiOBS
- "obs.cn-southwest-3.myhuaweicloud.com"
- CN Southwest-Chongqing
- Provider: HuaweiOBS
- "obs.eu-west-101.myhuaweicloud.eu"
- EU-Dublin
- Provider: HuaweiOBS
- "obs.la-north-2.myhuaweicloud.com"
- LA-Mexico City2
- Provider: HuaweiOBS
- "obs.la-south-2.myhuaweicloud.com"
- LA-Santiago
- Provider: HuaweiOBS
- "obs.me-east-1.myhuaweicloud.com"
- ME-Riyadh
- Provider: HuaweiOBS
- "obs.na-mexico-1.myhuaweicloud.com"
- LA-Mexico City1
- Provider: HuaweiOBS
- "obs.sa-chile-1.myhuaweicloud.com"
- LA-Santiago2
- "obs.ru-northwest-2.myhuaweicloud.com"
- RU-Moscow2
- Provider: HuaweiOBS
- "obs.sa-argentina-1.myhuaweicloud.com"
- LA-Buenos Aires1
- Provider: HuaweiOBS
- "obs.sa-brazil-1.myhuaweicloud.com"
- LA-Sao Paulo1
- Provider: HuaweiOBS
- "obs.ru-northwest-2.myhuaweicloud.com"
- RU-Moscow2
- "obs.sa-chile-1.myhuaweicloud.com"
- LA-Santiago2
- Provider: HuaweiOBS
- "obs.sa-peru-1.myhuaweicloud.com"
- LA-Lima1
- Provider: HuaweiOBS
- "obs.tr-west-1.myhuaweicloud.com"
- TR-Istanbul
- Provider: HuaweiOBS
- "s3.us.cloud-object-storage.appdomain.cloud"
- US Cross Region Endpoint
@@ -2195,6 +2335,18 @@ Properties:
- "us-east-1.storage.impossibleapi.net"
- New York, USA
- Provider: ImpossibleCloud
- "s3.swiss-backup01.infomaniak.com"
- Swiss Backup Location 01
- Provider: Infomaniak
- "s3.swiss-backup02.infomaniak.com"
- Swiss Backup Location 02
- Provider: Infomaniak
- "s3.swiss-backup03.infomaniak.com"
- Swiss Backup Location 03
- Provider: Infomaniak
- "s3.swiss-backup04.infomaniak.com"
- Swiss Backup Location 04
- Provider: Infomaniak
- "de-fra.i3storage.com"
- Frankfurt, Germany
- Provider: Intercolo
@@ -2299,38 +2451,47 @@ Properties:
- "br-ne1.magaluobjects.com"
- Fortaleza, CE (BR), br-ne1
- Provider: Magalu
- "s3.eu-amsterdam.megas4.com"
- Mega S4 Amsterdam
- "s3.eu-luxembourg-1.megas4.com"
- Mega S4 Luxembourg 1
- Provider: Mega
- "s3.eu-luxembourg.megas4.com"
- Mega S4 Luxembourg
- "s3.eu-luxembourg-2.megas4.com"
- Mega S4 Luxembourg 2
- Provider: Mega
- "s3.eu-paris.megas4.com"
- Mega S4 Paris
- "s3.eu-amsterdam-1.megas4.com"
- Mega S4 Amsterdam 1
- Provider: Mega
- "s3.eu-barcelona.megas4.com"
- Mega S4 Barcelona
- "s3.eu-amsterdam-2.megas4.com"
- Mega S4 Amsterdam 2
- Provider: Mega
- "s3.ca-montreal.megas4.com"
- Mega S4 Montreal
- "s3.eu-paris-1.megas4.com"
- Mega S4 Paris 1
- Provider: Mega
- "s3.ca-vancouver.megas4.com"
- Mega S4 Vancouver
- "s3.eu-paris-2.megas4.com"
- Mega S4 Paris 2
- Provider: Mega
- "s3.ap-tokyo.megas4.com"
- Mega S4 Tokyo
- "s3.eu-barcelona-1.megas4.com"
- Mega S4 Barcelona 1
- Provider: Mega
- "s3.eu-central-1.s4.mega.io"
- Mega S4 eu-central-1 (Amsterdam, legacy)
- "s3.eu-barcelona-2.megas4.com"
- Mega S4 Barcelona 2
- Provider: Mega
- "s3.eu-central-2.s4.mega.io"
- Mega S4 eu-central-2 (Bettembourg, legacy)
- "s3.ca-montreal-1.megas4.com"
- Mega S4 Montreal 1
- Provider: Mega
- "s3.ca-central-1.s4.mega.io"
- Mega S4 ca-central-1 (Montreal, legacy)
- "s3.ca-montreal-2.megas4.com"
- Mega S4 Montreal 2
- Provider: Mega
- "s3.ca-west-1.s4.mega.io"
- Mega S4 ca-west-1 (Vancouver, legacy)
- "s3.ca-vancouver-1.megas4.com"
- Mega S4 Vancouver 1
- Provider: Mega
- "s3.ca-vancouver-2.megas4.com"
- Mega S4 Vancouver 2
- Provider: Mega
- "s3.ap-tokyo-1.megas4.com"
- Mega S4 Tokyo 1
- Provider: Mega
- "s3.ap-tokyo-2.megas4.com"
- Mega S4 Tokyo 2
- Provider: Mega
- "oos.eu-west-2.outscale.com"
- Outscale EU West 2 (Paris)
@@ -2497,6 +2658,12 @@ Properties:
- "nz.s3.rackcorp.com"
- Auckland (New Zealand) Endpoint
- Provider: RackCorp
- "s3-gw1.relaix.net"
- Aachen-1, Germany
- Provider: RelAix
- "s3-gw2.relaix.net"
- Aachen-2, Germany
- Provider: RelAix
- "s3.nl-ams.scw.cloud"
- Amsterdam Endpoint
- Provider: Scaleway
@@ -2506,6 +2673,9 @@ Properties:
- "s3.pl-waw.scw.cloud"
- Warsaw Endpoint
- Provider: Scaleway
- "s3.it-mil.scw.cloud"
- Milan Endpoint
- Provider: Scaleway
- "localhost:8333"
- SeaweedFS S3 localhost
- Provider: SeaweedFS
@@ -2742,7 +2912,7 @@ Properties:
- Config: location_constraint
- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
- Provider: AWS,ArvanCloud,Ceph,ChinaMobile,DigitalOcean,Dreamhost,Exaba,GCS,HCP,Hetzner,IBMCOS,ImpossibleCloud,LyveCloud,Minio,Netease,Qiniu,Rabata,RackCorp,Scality,SeaweedFS,Synology,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,ArvanCloud,Ceph,ChinaMobile,DigitalOcean,Dreamhost,Exaba,GCS,HCP,Hetzner,IBMCOS,ImpossibleCloud,Infomaniak,LyveCloud,Minio,Netease,Qiniu,Rabata,RackCorp,Scality,SeaweedFS,Synology,Wasabi,Zata,ZeroServices,Other
- Type: string
- Required: false
- Examples:
@@ -3123,36 +3293,36 @@ Properties:
- Config: acl
- Env Var: RCLONE_S3_ACL
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,TencentCOS,US3,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,TencentCOS,US3,Wasabi,Zata,ZeroServices,Other
- Type: string
- Required: false
- Examples:
- "private"
- Owner gets FULL_CONTROL.
- No one else has access rights (default).
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,US3,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,US3,Wasabi,Zata,ZeroServices,Other
- "public-read"
- Owner gets FULL_CONTROL.
- The AllUsers group gets READ access.
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,TencentCOS,US3,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,TencentCOS,US3,Wasabi,Zata,ZeroServices,Other
- "public-read-write"
- Owner gets FULL_CONTROL.
- The AllUsers group gets READ and WRITE access.
- Granting this on a bucket is generally not recommended.
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- "authenticated-read"
- Owner gets FULL_CONTROL.
- The AuthenticatedUsers group gets READ access.
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- "bucket-owner-read"
- Object owner gets FULL_CONTROL.
- Bucket owner gets READ access.
- If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- "bucket-owner-full-control"
- Both the object owner and the bucket owner get FULL_CONTROL over the object.
- If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,Scality,SeaweedFS,TencentCOS,Wasabi,Zata,ZeroServices,Other
- "private"
- Owner gets FULL_CONTROL.
- No one else has access rights (default).
@@ -3346,7 +3516,7 @@ Properties:
### Advanced options
Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, BizflyCloud, Ceph, ChinaMobile, Cloudflare, Cubbit, DigitalOcean, Dreamhost, Exaba, Fastly, FileLu, FlashBlade, GCS, HCP, Hetzner, HuaweiOBS, IBMCOS, IDrive, ImpossibleCloud, Intercolo, IONOS, Leviia, Liara, Linode, LyveCloud, Magalu, Mega, Minio, Netease, Outscale, OVHcloud, Petabox, Qiniu, Rabata, RackCorp, Rclone, Scaleway, Scality, SeaweedFS, Selectel, Servercore, SpectraLogic, Storj, Synology, TencentCOS, US3, Wasabi, Zadara, Zata, ZeroServices, Other).
Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Autonomys, BizflyCloud, Cellar, Ceph, ChinaMobile, Cloudflare, Cubbit, DigitalOcean, Dreamhost, Exaba, Fastly, FileLu, FlashBlade, GCS, HCP, Hetzner, HuaweiOBS, IBMCOS, IDrive, ImpossibleCloud, Infomaniak, Intercolo, IONOS, Leviia, Liara, Linode, LyveCloud, Magalu, Mega, Minio, Neon, Netease, Outscale, OVHcloud, Petabox, Qiniu, Rabata, RackCorp, Rclone, RelAix, Scaleway, Scality, SeaweedFS, Selectel, Servercore, SpectraLogic, Storj, Synology, TencentCOS, US3, Wasabi, Zadara, Zata, ZeroServices, Other).
#### --s3-bucket-acl
@@ -3365,7 +3535,7 @@ Properties:
- Config: bucket_acl
- Env Var: RCLONE_S3_BUCKET_ACL
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,IDrive,ImpossibleCloud,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Mega,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,Scaleway,SeaweedFS,Servercore,TencentCOS,US3,Wasabi,Zata,ZeroServices,Other
- Provider: AWS,Alibaba,ArvanCloud,BizflyCloud,Cellar,Ceph,ChinaMobile,Cubbit,DigitalOcean,Dreamhost,Exaba,FileLu,GCS,HCP,Hetzner,HuaweiOBS,IBMCOS,IDrive,ImpossibleCloud,Infomaniak,Intercolo,IONOS,Leviia,Liara,Linode,LyveCloud,Magalu,Mega,Minio,Netease,Outscale,OVHcloud,Petabox,Qiniu,RackCorp,RelAix,Scaleway,SeaweedFS,Servercore,TencentCOS,US3,Wasabi,Zata,ZeroServices,Other
- Type: string
- Required: false
- Examples:
@@ -3403,7 +3573,7 @@ Properties:
- Config: sse_customer_algorithm
- Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
- Provider: AWS,Ceph,ChinaMobile,Minio
- Provider: AWS,Cellar,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -3416,13 +3586,13 @@ Properties:
To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
Alternatively you can provide --sse-customer-key-base64.
Alternatively you can provide --s3-sse-customer-key-base64.
Properties:
- Config: sse_customer_key
- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
- Provider: AWS,Ceph,ChinaMobile,Minio
- Provider: AWS,Cellar,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -3433,13 +3603,13 @@ Properties:
If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
Alternatively you can provide --sse-customer-key.
Alternatively you can provide --s3-sse-customer-key.
Properties:
- Config: sse_customer_key_base64
- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64
- Provider: AWS,Ceph,ChinaMobile,Minio
- Provider: AWS,Cellar,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -3457,7 +3627,7 @@ Properties:
- Config: sse_customer_key_md5
- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
- Provider: AWS,Ceph,ChinaMobile,Minio
- Provider: AWS,Cellar,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -4547,6 +4717,47 @@ info on how to pass options and arguments.
These can be run on a running backend using the rc command
[backend/command](/rc/#backend-command).
### link
Generate a signed link with response header overrides.
```console
rclone backend link remote: [options] [<arguments>+]
```
This command generates a signed download link for one file, with optional
overrides for the HTTP response headers. Pass the file path as a separate
argument, relative to the remote path.
Usage examples:
```console
rclone backend link s3:bucket path/to/file -o expire=1h
rclone backend link s3:bucket path/to/file -o response-expires="Thu, 01 Jan 1970 00:00:00 GMT"
rclone backend link s3:bucket path/to/file -o response-content-disposition='attachment; filename="download.txt"'
```
The link expires after 7 days by default. Use `-o expire=1h` to change
its lifetime. Durations must be at least 1 second; values above 7 days are
limited to 7 days, as with `rclone link`.
The `response-expires` option sets the HTTP Expires response header,
not the lifetime of the signed link. It must be an HTTP date, such as
`Thu, 01 Jan 1970 00:00:00 GMT`.
The overrides are included in the signature and must not be changed in the
returned URL. They do not modify the object's stored metadata.
Options:
- "expire": How long the link will be valid (default 7d, maximum 7d).
- "response-cache-control": Set the Cache-Control response header.
- "response-content-disposition": Set the Content-Disposition response header.
- "response-content-encoding": Set the Content-Encoding response header.
- "response-content-language": Set the Content-Language response header.
- "response-content-type": Set the Content-Type response header.
- "response-expires": Set the Expires response header to an HTTP date.
### restore
Restore objects from GLACIER or INTELLIGENT-TIERING archive tier.
docs/content/sftp.md
+29 -1
View File
@@ -920,7 +920,7 @@ Properties:
- Config: encoding
- Env Var: RCLONE_SFTP_ENCODING
- Type: Encoding
- Default: Slash,Del,Ctl,Dot
- Default: Slash,Dot
#### --sftp-set-modtime
@@ -1169,6 +1169,34 @@ Properties:
- Type: bool
- Default: false
#### --sftp-multithread-upload
Set this to use multi-thread (multi-connection) uploads.
Normally rclone uploads a single large file over one SFTP connection.
With this set rclone uploads large files over several SFTP connections
at once (one per chunk), which greatly improves throughput on high
latency links, the same way multi-thread downloads already work. Each
connection writes a different, non-overlapping byte range of the file,
so this requires a server that allows several handles to the same file
with writes at arbitrary offsets, which OpenSSH does.
This is off by default because many SFTP servers only accept sequential
writes (for example some object storage backed SFTP gateways) and would
fail large uploads with "truncate failed" or "invalid offset" style
errors.
This is ignored (uploads stay single-threaded) when
--sftp-disable-concurrent-writes or --sftp-connections is in use.
Properties:
- Config: multithread_upload
- Env Var: RCLONE_SFTP_MULTITHREAD_UPLOAD
- Type: bool
- Default: false
#### --sftp-idle-timeout
Max time before closing idle connections.
docs/content/smb.md
+20 -1
View File
@@ -137,12 +137,14 @@ Properties:
SMB username.
Leave blank to use the current user name.
Properties:
- Config: user
- Env Var: RCLONE_SMB_USER
- Type: string
- Default: "$USER"
- Required: false
#### --smb-port
@@ -219,6 +221,23 @@ Properties:
Here are the Advanced options specific to smb (SMB / CIFS).
#### --smb-workstation
Workstation name sent in NTLM authentication.
The NetBIOS name sent as the client workstation in the NTLM AUTHENTICATE
message. Some servers (e.g. domain-joined Windows servers with per-account
workstation restrictions or NTLM allow-lists) require this to match a
specific name. Leave blank to send no workstation name (the default).
Properties:
- Config: workstation
- Env Var: RCLONE_SMB_WORKSTATION
- Type: string
- Required: false
#### --smb-idle-timeout
Max time before closing idle connections.
docs/content/swift.md
+8
View File
@@ -328,6 +328,14 @@ Properties:
- OVH
- "https://authenticate.ain.net"
- Blomp Cloud Storage
- "https://swiss-backup01.infomaniak.com/identity/v3"
- Infomaniak Swiss Backup Location 01
- "https://swiss-backup02.infomaniak.com/identity/v3"
- Infomaniak Swiss Backup Location 02
- "https://swiss-backup03.infomaniak.com/identity/v3"
- Infomaniak Swiss Backup Location 03
- "https://swiss-backup04.infomaniak.com/identity/v3"
- Infomaniak Swiss Backup Location 04
#### --swift-user-id
docs/content/webdav.md
+7
View File
@@ -249,6 +249,13 @@ For example, to set a Cookie use 'Cookie,name=value', or '"Cookie","name=value"'
You can set multiple headers, e.g. '"Cookie","name=value","Authorization","xxx"'.
The headers are only sent to the host in the configured URL. If the
server redirects to another host (including a subdomain or a different
port) the headers are not sent to it, or to any further hop in that
redirect chain. The exception is the directory listing and file
download requests when auth_redirect is set, as that keeps all
credentials across redirects.
Properties:
docs/content/yandex.md
+20
View File
@@ -278,6 +278,26 @@ Properties:
- Type: Duration
- Default: 0s
#### --yandex-app-folder
Use the application folder as the root.
If you registered your own OAuth application with Yandex with only
the "Application Folder" permission (cloud_api:disk.app_folder)
instead of full disk access, then rclone needs to address paths with
the "app:/" prefix instead of the usual "disk:/" prefix, otherwise
all requests fail with a 403 Forbidden error.
Set this to true if your OAuth token only has access to the
application folder.
Properties:
- Config: app_folder
- Env Var: RCLONE_YANDEX_APP_FOLDER
- Type: bool
- Default: false
#### --yandex-description
Description of the remote.
lib/transform/transform.md
+2 -2
View File
@@ -218,12 +218,12 @@ rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,command=e
```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
// Output: stories/The Quick Brown Fox!-20260731
// Output: stories/The Quick Brown Fox!-20261106
```
```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
// Output: stories/The Quick Brown Fox!-2026-07-31 0455PM
// Output: stories/The Quick Brown Fox!-2026-11-06 0235PM
```
```console
rclone.1
Generated
+7364 -748
View File
File diff suppressed because it is too large. Load diff