Compare commits

...
Author SHA1 Message Date
Anagh Kumar Baranwal f26e41d1c5 fix: mount parsing for linux under the WSL layer which may contain mounts with spaces
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2023-07-18 17:39:25 +05:30
Benjamin 8a6bf35481 Add Leviia Object Storage on index.md 2023-07-18 09:52:05 +01:00
Benjamin f7d27f4bf2 Add Object storage to Leviia on README.md 2023-07-18 09:52:05 +01:00
kapitainsky 378a2d21ee --max-transfer - add new exit code (10)
It adds dedicated exit code (10) for --max-duration flag.

Rclone will exit with exit code 10 if the duration limit is reached.

It behaves in similar fashion as --max-transfer and exit code 8.

discussed on the forum:

https://forum.rclone.org/t/max-duration-option-is-triggering-exit-with-error/39917/6
2023-07-18 09:51:31 +01:00
Nick Craig-Wood 3404eb0444 Changelog updates from Version v1.63.1 2023-07-17 15:15:16 +01:00
Nick Craig-Wood 13e5701f2a build: add new sponsors page to docs 2023-07-17 14:28:40 +01:00
Nick Craig-Wood 432d5d1e20 operations: fix overlapping check on case insensitive file systems
Before this change, the overlapping check could erroneously give this
error on case insensitive file systems:

    Failed to sync: destination and parameter to --backup-dir mustn't overlap

The code was fixed and re-worked to be simpler and more reliable.

See: https://forum.rclone.org/t/backup-dir-cannot-be-in-root-even-when-excluded/39844/
2023-07-17 14:00:04 +01:00
Nick Craig-Wood cc05159518 Add Benjamin to contributors 2023-07-17 14:00:04 +01:00
Benjamin 119ccb2b95 s3: add Leviia S3 Object Storage as provider 2023-07-16 18:08:47 +01:00
Anagh Kumar Baranwal 0ef0e908ca build: update to go1.21rc3 and make go1.19 the minimum required version
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2023-07-16 10:09:25 +01:00
Nick Craig-Wood 0063d14dbb Add darix to contributors 2023-07-14 10:27:20 +01:00
albertony 0d34efb10f box: fix reconnect failing with HTTP 400 Bad Request
The error is:

  Error: failed to configure token with jwt authentication: jwtutil: failed making auth request: 400 Bad Request

With the following additional debug information:

  jwtutil: Response Body: {"error":"invalid_grant","error_description":"Please check the 'aud' claim. Should be a string"}

Problem is that in jwt-go the RegisteredClaims type has Audience field (aud claim) that
is a list, while box apparantly expects it to be a singular string. In jwt-go v4 we
currently use there is an alternative type StandardClaims which matches what box wants.
Unfortunately StandardClaims is marked as deprecated, and is removed in the
newer v5 version, so we this is a short term fix only.

Fixes #7114
2023-07-14 10:24:33 +01:00
darix 415f4b2b93 webdav: nextcloud chunking: add more guidance for the user to check the config 2023-07-10 14:37:09 +01:00
Nick Craig-Wood 07cf5f1d25 operations: fix .rclonelink files not being converted back to symlinks
Before this change the new partial downloads code was causing symlinks
to be copied as regular files.

This was because the partial isn't named .rclonelink so the local
backend saves it as a normal file and renaming it to .rclonelink
doesn't cause it to become a symlink.

This fixes the problem by not copying .rclonelink files using the
partials mechanism but reverting to the previous --inplace behaviour.

This could potentially be fixed better in the future by changing the
local backend Move to change files to and from symlinks depending on
their name. However this was deemed too complicated for a point
release.

This also adds a test in the local backend. This test should ideally
be in operations but it isn't easy to put it there as operations knows
nothing of symlinks.

Fixes #7101
See: https://forum.rclone.org/t/reggression-in-v1-63-0-links-drops-the-rclonelink-extension/39483
2023-07-10 14:30:59 +01:00
Nick Craig-Wood 7d31956169 local: fix partial directory read for corrupted filesystem
Before this change if a directory entry could be listed but not
lstat-ed then rclone would give an error and abort the directory
listing with the error

    failed to read directory entry: failed to read directory "XXX": lstat XXX

This change makes sure that the directory listing carries on even
after this kind of error.

The sync will be failed but it will carry on.

This problem was caused by a programming error setting the err
variable in an outer scope when it should have been using a local err
variable.

See: https://forum.rclone.org/t/sync-aborts-if-even-one-single-unreadable-folder-is-encountered/39653
2023-07-09 17:58:03 +01:00
Nick Craig-Wood 473d443874 smb: fix "Statfs failed: bucket or container name is needed" when mounting
Before this change, if you mounted the root of the smb then it would
give an error on rclone about and periodically in the mount logs:

    Statfs failed: bucket or container name is needed in remote

This fix makes the smb backend return empty usage in this case which
will stop the errors and show the default 1P of free space.

See: https://forum.rclone.org/t/error-statfs-failed-bucket-or-container-name-is-needed-in-remote/39631
2023-07-08 12:24:46 +01:00
Nick Craig-Wood e294b76121 Add Vladislav Vorobev to contributors 2023-07-08 12:24:46 +01:00
Vladislav Vorobev 8f3c583870 docs: no need to disable 2FA for Mail.ru Cloud anymore
This sentence was written at the time when backend used access token, nowadays, users need to generate and use application password instead, see #6398.
2023-07-08 10:27:40 +02:00
Nick Craig-Wood d0d41fe847 rclone config redacted: implement support mechanism for showing redacted config
This introduces a new fs.Option flag, Sensitive and uses this along
with IsPassword to redact the info in the config file for support
purposes.

It adds this flag into backends where appropriate. It was necessary to
add oauthutil.SharedOptions to some backends as they were missing
them.

Fixes #5209
2023-07-07 16:25:14 +01:00
Nick Craig-Wood 297f15a3e3 docs: update the number of providers supported 2023-07-07 16:25:14 +01:00
Nick Craig-Wood d5f0affd4b Add Mahad to contributors 2023-07-07 16:25:14 +01:00
Nick Craig-Wood 0598aafbfd Add BakaWang to contributors 2023-07-07 16:25:14 +01:00
Mahad 528e22f139 docs: drive: Fix step 4 in "Making your own client_id" 2023-07-06 21:24:17 +01:00
BakaWang f1a8420814 s3: add synology to s3 provider list 2023-07-06 10:54:07 +01:00
Nick Craig-Wood e250f1afcd docs: remove old donate page 2023-07-06 10:13:42 +01:00
Nick Craig-Wood ebf24c9872 docs: update contact page on website 2023-07-05 16:57:07 +01:00
PaulandNick Craig-Wood b4c7b240d8 webdav: nextcloud: fix must use /dav/files/USER endpoint not /webdav error
Fix https://github.com/rclone/rclone/issues/7103

Before this change the RegExp validating the endpoint URL was a bit
too strict allowing only /dav/files/USER due to chunking limitations.

This patch adds back support for /dav/files/USER/dir/subdir etc.

Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
2023-07-05 16:56:01 +01:00
77 changed files with 1050 additions and 505 deletions

No files matched your search

+20 -15
View File
@@ -27,12 +27,12 @@ jobs:
strategy:
fail-fast: false
matrix:
job_name: ['linux', 'linux_386', 'mac_amd64', 'mac_arm64', 'windows', 'other_os', 'go1.18', 'go1.19']
job_name: ['linux', 'linux_386', 'mac_amd64', 'mac_arm64', 'windows', 'other_os', 'go1.19', 'go1.20']
include:
- job_name: linux
os: ubuntu-latest
go: '1.20'
go: '1.21.0-rc.3'
gotags: cmount
build_flags: '-include "^linux/"'
check: true
@@ -43,14 +43,14 @@ jobs:
- job_name: linux_386
os: ubuntu-latest
go: '1.20'
go: '1.21.0-rc.3'
goarch: 386
gotags: cmount
quicktest: true
- job_name: mac_amd64
os: macos-11
go: '1.20'
go: '1.21.0-rc.3'
gotags: 'cmount'
build_flags: '-include "^darwin/amd64" -cgo'
quicktest: true
@@ -59,14 +59,14 @@ jobs:
- job_name: mac_arm64
os: macos-11
go: '1.20'
go: '1.21.0-rc.3'
gotags: 'cmount'
build_flags: '-include "^darwin/arm64" -cgo -macos-arch arm64 -cgo-cflags=-I/usr/local/include -cgo-ldflags=-L/usr/local/lib'
deploy: true
- job_name: windows
os: windows-latest
go: '1.20'
go: '1.21.0-rc.3'
gotags: cmount
cgo: '0'
build_flags: '-include "^windows/"'
@@ -76,23 +76,23 @@ jobs:
- job_name: other_os
os: ubuntu-latest
go: '1.20'
go: '1.21.0-rc.3'
build_flags: '-exclude "^(windows/|darwin/|linux/)"'
compile_all: true
deploy: true
- job_name: go1.18
os: ubuntu-latest
go: '1.18'
quicktest: true
racequicktest: true
- job_name: go1.19
os: ubuntu-latest
go: '1.19'
quicktest: true
racequicktest: true
- job_name: go1.20
os: ubuntu-latest
go: '1.20'
quicktest: true
racequicktest: true
name: ${{ matrix.job_name }}
runs-on: ${{ matrix.os }}
@@ -130,6 +130,11 @@ jobs:
- name: Install Libraries on macOS
shell: bash
run: |
# https://github.com/Homebrew/brew/issues/15621#issuecomment-1619266788
# https://github.com/orgs/Homebrew/discussions/4612#discussioncomment-6319008
unset HOMEBREW_NO_INSTALL_FROM_API
brew untap --force homebrew/core
brew untap --force homebrew/cask
brew update
brew install --cask macfuse
if: matrix.os == 'macos-11'
@@ -239,7 +244,7 @@ jobs:
- name: Install Go
uses: actions/setup-go@v4
with:
go-version: '1.20'
go-version: '1.21.0-rc.3'
check-latest: true
- name: Install govulncheck
@@ -264,7 +269,7 @@ jobs:
- name: Set up Go
uses: actions/setup-go@v4
with:
go-version: '1.20'
go-version: '1.21.0-rc.3'
- name: Go module cache
uses: actions/cache@v3
+2
View File
@@ -51,6 +51,7 @@ Rclone *("rsync for cloud storage")* is a command-line program to sync files and
* IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
* IONOS Cloud [:page_facing_up:](https://rclone.org/s3/#ionos)
* Koofr [:page_facing_up:](https://rclone.org/koofr/)
* Leviia Object Storage [:page_facing_up:](https://rclone.org/s3/#leviia)
* Liara Object Storage [:page_facing_up:](https://rclone.org/s3/#liara-object-storage)
* Mail.ru Cloud [:page_facing_up:](https://rclone.org/mailru/)
* Memset Memstore [:page_facing_up:](https://rclone.org/swift/)
@@ -84,6 +85,7 @@ Rclone *("rsync for cloud storage")* is a command-line program to sync files and
* StackPath [:page_facing_up:](https://rclone.org/s3/#stackpath)
* Storj [:page_facing_up:](https://rclone.org/storj/)
* SugarSync [:page_facing_up:](https://rclone.org/sugarsync/)
* Synology C2 Object Storage [:page_facing_up:](https://rclone.org/s3/#synology-c2)
* Tencent Cloud Object Storage (COS) [:page_facing_up:](https://rclone.org/s3/#tencent-cos)
* Wasabi [:page_facing_up:](https://rclone.org/s3/#wasabi)
* WebDAV [:page_facing_up:](https://rclone.org/webdav/)
+22 -12
View File
@@ -1,5 +1,5 @@
//go:build !plan9 && !solaris && !js && go1.18
// +build !plan9,!solaris,!js,go1.18
//go:build !plan9 && !solaris && !js
// +build !plan9,!solaris,!js
// Package azureblob provides an interface to the Microsoft Azure blob object storage system
package azureblob
@@ -95,6 +95,7 @@ Leave blank to use SAS URL or Emulator, otherwise it needs to be set.
If this is blank and if env_auth is set it will be read from the
environment variable ` + "`AZURE_STORAGE_ACCOUNT_NAME`" + ` if possible.
`,
Sensitive: true,
}, {
Name: "env_auth",
Help: `Read credentials from runtime (environment variables, CLI or MSI).
@@ -106,11 +107,13 @@ See the [authentication docs](/azureblob#authentication) for full info.`,
Help: `Storage Account Shared Key.
Leave blank to use SAS URL or Emulator.`,
Sensitive: true,
}, {
Name: "sas_url",
Help: `SAS URL for container level access only.
Leave blank if using account/key or Emulator.`,
Sensitive: true,
}, {
Name: "tenant",
Help: `ID of the service principal's tenant. Also called its directory ID.
@@ -120,6 +123,7 @@ Set this if using
- Service principal with certificate
- User with username and password
`,
Sensitive: true,
}, {
Name: "client_id",
Help: `The ID of the client in use.
@@ -129,6 +133,7 @@ Set this if using
- Service principal with certificate
- User with username and password
`,
Sensitive: true,
}, {
Name: "client_secret",
Help: `One of the service principal's client secrets
@@ -136,6 +141,7 @@ Set this if using
Set this if using
- Service principal with client secret
`,
Sensitive: true,
}, {
Name: "client_certificate_path",
Help: `Path to a PEM or PKCS12 certificate file including the private key.
@@ -173,7 +179,8 @@ Optionally set this if using
Set this if using
- User with username and password
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "password",
Help: `The user's password
@@ -216,17 +223,20 @@ msi_client_id, or msi_mi_res_id parameters.`,
Default: false,
Advanced: true,
}, {
Name: "msi_object_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_mi_res_id specified.",
Advanced: true,
Name: "msi_object_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_mi_res_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "msi_client_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_object_id or msi_mi_res_id specified.",
Advanced: true,
Name: "msi_client_id",
Help: "Object ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_object_id or msi_mi_res_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "msi_mi_res_id",
Help: "Azure resource ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_object_id specified.",
Advanced: true,
Name: "msi_mi_res_id",
Help: "Azure resource ID of the user-assigned MSI to use, if any.\n\nLeave blank if msi_client_id or msi_object_id specified.",
Advanced: true,
Sensitive: true,
}, {
Name: "use_emulator",
Help: "Uses local storage emulator if provided as 'true'.\n\nLeave blank if using real azure storage endpoint.",
+2 -2
View File
@@ -1,5 +1,5 @@
//go:build !plan9 && !solaris && !js && go1.18
// +build !plan9,!solaris,!js,go1.18
//go:build !plan9 && !solaris && !js
// +build !plan9,!solaris,!js
package azureblob
+2 -2
View File
@@ -1,7 +1,7 @@
// Test AzureBlob filesystem interface
//go:build !plan9 && !solaris && !js && go1.18
// +build !plan9,!solaris,!js,go1.18
//go:build !plan9 && !solaris && !js
// +build !plan9,!solaris,!js
package azureblob
+2 -2
View File
@@ -1,7 +1,7 @@
// Build for azureblob for unsupported platforms to stop go complaining
// about "no buildable Go source files "
//go:build plan9 || solaris || js || !go1.18
// +build plan9 solaris js !go1.18
//go:build plan9 || solaris || js
// +build plan9 solaris js
package azureblob
+8 -6
View File
@@ -75,13 +75,15 @@ func init() {
Description: "Backblaze B2",
NewFs: NewFs,
Options: []fs.Option{{
Name: "account",
Help: "Account ID or Application Key ID.",
Required: true,
Name: "account",
Help: "Account ID or Application Key ID.",
Required: true,
Sensitive: true,
}, {
Name: "key",
Help: "Application Key.",
Required: true,
Name: "key",
Help: "Application Key.",
Required: true,
Sensitive: true,
}, {
Name: "endpoint",
Help: "Endpoint for the service.\n\nLeave blank normally.",
+15 -11
View File
@@ -77,7 +77,7 @@ var (
)
type boxCustomClaims struct {
jwt.RegisteredClaims
jwt.StandardClaims
BoxSubType string `json:"box_sub_type,omitempty"`
}
@@ -107,16 +107,18 @@ func init() {
return nil, nil
},
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "root_folder_id",
Help: "Fill in for rclone to use a non root folder as its starting point.",
Default: "0",
Advanced: true,
Name: "root_folder_id",
Help: "Fill in for rclone to use a non root folder as its starting point.",
Default: "0",
Advanced: true,
Sensitive: true,
}, {
Name: "box_config_file",
Help: "Box App config.json location\n\nLeave blank normally." + env.ShellExpandHelp,
}, {
Name: "access_token",
Help: "Box App Primary Access Token\n\nLeave blank normally.",
Name: "access_token",
Help: "Box App Primary Access Token\n\nLeave blank normally.",
Sensitive: true,
}, {
Name: "box_sub_type",
Default: "user",
@@ -206,12 +208,14 @@ func getClaims(boxConfig *api.ConfigJSON, boxSubType string) (claims *boxCustomC
}
claims = &boxCustomClaims{
RegisteredClaims: jwt.RegisteredClaims{
ID: val,
//lint:ignore SA1019 since we need to use jwt.StandardClaims even if deprecated in jwt-go v4 until a more permanent solution is ready in time before jwt-go v5 where it is removed entirely
//nolint:staticcheck // Don't include staticcheck when running golangci-lint to avoid SA1019
StandardClaims: jwt.StandardClaims{
Id: val,
Issuer: boxConfig.BoxAppSettings.ClientID,
Subject: boxConfig.EnterpriseID,
Audience: jwt.ClaimStrings{tokenURL},
ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Second * 45)),
Audience: tokenURL,
ExpiresAt: time.Now().Add(time.Second * 45).Unix(),
},
BoxSubType: boxSubType,
}
+8 -6
View File
@@ -76,17 +76,19 @@ func init() {
Name: "plex_url",
Help: "The URL of the Plex server.",
}, {
Name: "plex_username",
Help: "The username of the Plex user.",
Name: "plex_username",
Help: "The username of the Plex user.",
Sensitive: true,
}, {
Name: "plex_password",
Help: "The password of the Plex user.",
IsPassword: true,
}, {
Name: "plex_token",
Help: "The plex token for authentication - auto set normally.",
Hide: fs.OptionHideBoth,
Advanced: true,
Name: "plex_token",
Help: "The plex token for authentication - auto set normally.",
Hide: fs.OptionHideBoth,
Advanced: true,
Sensitive: true,
}, {
Name: "plex_insecure",
Help: "Skip all certificate verification when connecting to the Plex server.",
+2 -2
View File
@@ -160,11 +160,11 @@ func TestInternalUploadQueueMoreFiles(t *testing.T) {
minSize := 5242880
maxSize := 10485760
totalFiles := 10
rand.Seed(time.Now().Unix())
randInstance := rand.New(rand.NewSource(time.Now().Unix()))
lastFile := ""
for i := 0; i < totalFiles; i++ {
size := int64(rand.Intn(maxSize-minSize) + minSize)
size := int64(randInstance.Intn(maxSize-minSize) + minSize)
testReader := runInstance.randomReader(t, size)
remote := "test/" + strconv.Itoa(i) + ".bin"
runInstance.writeRemoteReader(t, rootFs, remote, testReader)
+19 -14
View File
@@ -277,20 +277,23 @@ Leave blank normally.
Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "service_account_file",
Help: "Service Account Credentials JSON file path.\n\nLeave blank normally.\nNeeded only if you want use SA instead of interactive login." + env.ShellExpandHelp,
}, {
Name: "service_account_credentials",
Help: "Service Account Credentials JSON blob.\n\nLeave blank normally.\nNeeded only if you want use SA instead of interactive login.",
Hide: fs.OptionHideConfigurator,
Advanced: true,
Name: "service_account_credentials",
Help: "Service Account Credentials JSON blob.\n\nLeave blank normally.\nNeeded only if you want use SA instead of interactive login.",
Hide: fs.OptionHideConfigurator,
Advanced: true,
Sensitive: true,
}, {
Name: "team_drive",
Help: "ID of the Shared Drive (Team Drive).",
Hide: fs.OptionHideConfigurator,
Advanced: true,
Name: "team_drive",
Help: "ID of the Shared Drive (Team Drive).",
Hide: fs.OptionHideConfigurator,
Advanced: true,
Sensitive: true,
}, {
Name: "auth_owner_only",
Default: false,
@@ -416,10 +419,11 @@ date is used.`,
Help: "Size of listing chunk 100-1000, 0 to disable.",
Advanced: true,
}, {
Name: "impersonate",
Default: "",
Help: `Impersonate this user when using a service account.`,
Advanced: true,
Name: "impersonate",
Default: "",
Help: `Impersonate this user when using a service account.`,
Advanced: true,
Sensitive: true,
}, {
Name: "alternate_export",
Default: false,
@@ -592,7 +596,8 @@ Note also that opening the folder once in the web interface (with the
user you've authenticated rclone with) seems to be enough so that the
resource key is no needed.
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
+3 -2
View File
@@ -182,8 +182,9 @@ client_secret) to use this option as currently rclone's default set of
permissions doesn't include "members.read". This can be added once
v1.55 or later is in use everywhere.
`,
Default: "",
Advanced: true,
Default: "",
Advanced: true,
Sensitive: true,
}, {
Name: "shared_files",
Help: `Instructs rclone to work on individual shared files.
+3 -2
View File
@@ -38,8 +38,9 @@ func init() {
Description: "1Fichier",
NewFs: NewFs,
Options: []fs.Option{{
Help: "Your API Key, get it from https://1fichier.com/console/params.pl.",
Name: "api_key",
Help: "Your API Key, get it from https://1fichier.com/console/params.pl.",
Name: "api_key",
Sensitive: true,
}, {
Help: "If you want to download a shared folder, add this parameter.",
Name: "shared_folder",
+4 -1
View File
@@ -84,6 +84,7 @@ Leave blank normally.
Fill in to make rclone start with directory of a given ID.
`,
Sensitive: true,
}, {
Name: "permanent_token",
Help: `Permanent Authentication Token.
@@ -97,6 +98,7 @@ These tokens are normally valid for several years.
For more info see: https://docs.storagemadeeasy.com/organisationcloud/api-tokens
`,
Sensitive: true,
}, {
Name: "token",
Help: `Session Token.
@@ -106,7 +108,8 @@ usually valid for 1 hour.
Don't set this value - rclone will set it automatically.
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "token_expiry",
Help: `Token expiry time.
+8 -6
View File
@@ -48,13 +48,15 @@ func init() {
Description: "FTP",
NewFs: NewFs,
Options: []fs.Option{{
Name: "host",
Help: "FTP host to connect to.\n\nE.g. \"ftp.example.com\".",
Required: true,
Name: "host",
Help: "FTP host to connect to.\n\nE.g. \"ftp.example.com\".",
Required: true,
Sensitive: true,
}, {
Name: "user",
Help: "FTP username.",
Default: currentUser,
Name: "user",
Help: "FTP username.",
Default: currentUser,
Sensitive: true,
}, {
Name: "port",
Help: "FTP port number.",
@@ -91,18 +91,21 @@ func init() {
})
},
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "project_number",
Help: "Project number.\n\nOptional - needed only for list/create/delete buckets - see your developer console.",
Name: "project_number",
Help: "Project number.\n\nOptional - needed only for list/create/delete buckets - see your developer console.",
Sensitive: true,
}, {
Name: "user_project",
Help: "User project.\n\nOptional - needed only for requester pays.",
Name: "user_project",
Help: "User project.\n\nOptional - needed only for requester pays.",
Sensitive: true,
}, {
Name: "service_account_file",
Help: "Service Account Credentials JSON file path.\n\nLeave blank normally.\nNeeded only if you want use SA instead of interactive login." + env.ShellExpandHelp,
}, {
Name: "service_account_credentials",
Help: "Service Account Credentials JSON blob.\n\nLeave blank normally.\nNeeded only if you want use SA instead of interactive login.",
Hide: fs.OptionHideBoth,
Name: "service_account_credentials",
Help: "Service Account Credentials JSON blob.\n\nLeave blank normally.\nNeeded only if you want use SA instead of interactive login.",
Hide: fs.OptionHideBoth,
Sensitive: true,
}, {
Name: "anonymous",
Help: "Access public buckets and objects without credentials.\n\nSet to 'true' if you just want to download files and don't configure credentials.",
+7 -4
View File
@@ -19,9 +19,10 @@ func init() {
Description: "Hadoop distributed file system",
NewFs: NewFs,
Options: []fs.Option{{
Name: "namenode",
Help: "Hadoop name node and port.\n\nE.g. \"namenode:8020\" to connect to host namenode at port 8020.",
Required: true,
Name: "namenode",
Help: "Hadoop name node and port.\n\nE.g. \"namenode:8020\" to connect to host namenode at port 8020.",
Required: true,
Sensitive: true,
}, {
Name: "username",
Help: "Hadoop user name.",
@@ -29,6 +30,7 @@ func init() {
Value: "root",
Help: "Connect to hdfs as root.",
}},
Sensitive: true,
}, {
Name: "service_principal_name",
Help: `Kerberos service principal name for the namenode.
@@ -36,7 +38,8 @@ func init() {
Enables KERBEROS authentication. Specifies the Service Principal Name
(SERVICE/FQDN) for the namenode. E.g. \"hdfs/namenode.hadoop.docker\"
for namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "data_transfer_protection",
Help: `Kerberos data transfer protection: authentication|integrity|privacy.
+6 -4
View File
@@ -133,11 +133,13 @@ Owner is able to add custom keys. Metadata feature grabs all the keys including
},
Options: []fs.Option{{
Name: "access_key_id",
Help: "IAS3 Access Key.\n\nLeave blank for anonymous access.\nYou can find one here: https://archive.org/account/s3.php",
Name: "access_key_id",
Help: "IAS3 Access Key.\n\nLeave blank for anonymous access.\nYou can find one here: https://archive.org/account/s3.php",
Sensitive: true,
}, {
Name: "secret_access_key",
Help: "IAS3 Secret Key (password).\n\nLeave blank for anonymous access.",
Name: "secret_access_key",
Help: "IAS3 Secret Key (password).\n\nLeave blank for anonymous access.",
Sensitive: true,
}, {
// their official client (https://github.com/jjjake/internetarchive) hardcodes following the two
Name: "endpoint",
+2 -2
View File
@@ -88,7 +88,7 @@ func init() {
Description: "Jottacloud",
NewFs: NewFs,
Config: Config,
Options: []fs.Option{{
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "md5_memory_limit",
Help: "Files bigger than this will be cached on disk to calculate the MD5 if required.",
Default: fs.SizeSuffix(10 * 1024 * 1024),
@@ -123,7 +123,7 @@ func init() {
Default: (encoder.Display |
encoder.EncodeWin | // :?"*<>|
encoder.EncodeInvalidUtf8),
}},
}}...),
})
}
+4 -3
View File
@@ -61,9 +61,10 @@ func init() {
Default: true,
Advanced: true,
}, {
Name: "user",
Help: "Your user name.",
Required: true,
Name: "user",
Help: "Your user name.",
Required: true,
Sensitive: true,
}, {
Name: "password",
Help: "Your password for rclone (generate one at https://app.koofr.net/app/admin/preferences/password).",
+1 -1
View File
@@ -516,7 +516,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
continue
}
}
err = fmt.Errorf("failed to read directory %q: %w", namepath, fierr)
fierr = fmt.Errorf("failed to get info about directory entry %q: %w", namepath, fierr)
fs.Errorf(dir, "%v", fierr)
_ = accounting.Stats(ctx).Error(fserrors.NoRetryError(fierr)) // fail the sync
continue
+41
View File
@@ -19,6 +19,7 @@ import (
"github.com/rclone/rclone/fs/filter"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/lib/file"
"github.com/rclone/rclone/lib/readers"
@@ -514,3 +515,43 @@ func TestFilterSymlinkCopyLinks(t *testing.T) {
func TestFilterSymlinkLinks(t *testing.T) {
testFilterSymlink(t, false)
}
func TestCopySymlink(t *testing.T) {
ctx := context.Background()
r := fstest.NewRun(t)
defer r.Finalise()
when := time.Now()
f := r.Flocal.(*Fs)
// Create a file and a symlink to it
r.WriteFile("src/file.txt", "hello world", when)
require.NoError(t, os.Symlink("file.txt", filepath.Join(r.LocalName, "src", "link.txt")))
defer func() {
// Reset -L/-l mode
f.opt.FollowSymlinks = false
f.opt.TranslateSymlinks = false
f.lstat = os.Lstat
}()
// Set fs into "-l/--links" mode
f.opt.FollowSymlinks = false
f.opt.TranslateSymlinks = true
f.lstat = os.Lstat
// Create dst
require.NoError(t, f.Mkdir(ctx, "dst"))
// Do copy from src into dst
src, err := f.NewObject(ctx, "src/link.txt.rclonelink")
require.NoError(t, err)
require.NotNil(t, src)
dst, err := operations.Copy(ctx, f, nil, "dst/link.txt.rclonelink", src)
require.NoError(t, err)
require.NotNil(t, dst)
// Test that we made a symlink and it has the right contents
dstPath := filepath.Join(r.LocalName, "dst", "link.txt")
linkContents, err := os.Readlink(dstPath)
require.NoError(t, err)
assert.Equal(t, "file.txt", linkContents)
}
+6 -5
View File
@@ -85,10 +85,11 @@ func init() {
Name: "mailru",
Description: "Mail.ru Cloud",
NewFs: NewFs,
Options: []fs.Option{{
Name: "user",
Help: "User name (usually email).",
Required: true,
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "user",
Help: "User name (usually email).",
Required: true,
Sensitive: true,
}, {
Name: "pass",
Help: `Password.
@@ -213,7 +214,7 @@ Supported quirks: atomicmkdir binlist unknowndirs`,
encoder.EncodeWin | // :?"*<>|
encoder.EncodeBackSlash |
encoder.EncodeInvalidUtf8),
}},
}}...),
})
}
+4 -3
View File
@@ -58,9 +58,10 @@ func init() {
Description: "Mega",
NewFs: NewFs,
Options: []fs.Option{{
Name: "user",
Help: "User name.",
Required: true,
Name: "user",
Help: "User name.",
Required: true,
Sensitive: true,
}, {
Name: "pass",
Help: "Password.",
+6 -4
View File
@@ -65,11 +65,13 @@ HTTP is provided primarily for debugging purposes.`,
Help: `Domain+path of NetStorage host to connect to.
Format should be ` + "`<domain>/<internal folders>`",
Required: true,
Required: true,
Sensitive: true,
}, {
Name: "account",
Help: "Set the NetStorage account name",
Required: true,
Name: "account",
Help: "Set the NetStorage account name",
Required: true,
Sensitive: true,
}, {
Name: "secret",
Help: `Set the NetStorage account secret/G2O key for authentication.
+9 -6
View File
@@ -131,10 +131,11 @@ Note that the chunks will be buffered into memory.`,
Default: defaultChunkSize,
Advanced: true,
}, {
Name: "drive_id",
Help: "The ID of the drive to use.",
Default: "",
Advanced: true,
Name: "drive_id",
Help: "The ID of the drive to use.",
Default: "",
Advanced: true,
Sensitive: true,
}, {
Name: "drive_type",
Help: "The type of the drive (" + driveTypePersonal + " | " + driveTypeBusiness + " | " + driveTypeSharepoint + ").",
@@ -148,7 +149,8 @@ This isn't normally needed, but in special circumstances you might
know the folder ID that you wish to access but not be able to get
there through a path traversal.
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "access_scopes",
Help: `Set scopes to be requested by rclone.
@@ -260,7 +262,8 @@ this flag there.
At the time of writing this only works with OneDrive personal paid accounts.
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "hash_type",
Default: "auto",
+4 -3
View File
@@ -42,9 +42,10 @@ func init() {
Description: "OpenDrive",
NewFs: NewFs,
Options: []fs.Option{{
Name: "username",
Help: "Username.",
Required: true,
Name: "username",
Help: "Username.",
Required: true,
Sensitive: true,
}, {
Name: "password",
Help: "Password.",
+9 -7
View File
@@ -92,14 +92,16 @@ func newOptions() []fs.Option {
Help: noAuthHelpText,
}},
}, {
Name: "namespace",
Help: "Object storage namespace",
Required: true,
Name: "namespace",
Help: "Object storage namespace",
Required: true,
Sensitive: true,
}, {
Name: "compartment",
Help: "Object storage compartment OCID",
Provider: "!no_auth",
Required: true,
Name: "compartment",
Help: "Object storage compartment OCID",
Provider: "!no_auth",
Required: true,
Sensitive: true,
}, {
Name: "region",
Help: "Object storage Region",
+7 -5
View File
@@ -110,10 +110,11 @@ func init() {
encoder.EncodeBackSlash |
encoder.EncodeInvalidUtf8),
}, {
Name: "root_folder_id",
Help: "Fill in for rclone to use a non root folder as its starting point.",
Default: "d0",
Advanced: true,
Name: "root_folder_id",
Help: "Fill in for rclone to use a non root folder as its starting point.",
Default: "d0",
Advanced: true,
Sensitive: true,
}, {
Name: "hostname",
Help: `Hostname to connect to.
@@ -138,7 +139,8 @@ with rclone authorize.
This is only required when you want to use the cleanup command. Due to a bug
in the pcloud API the required API does not support OAuth authentication so
we have to rely on user password authentication for it.`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "password",
Help: "Your pcloud password.",
+6 -4
View File
@@ -158,9 +158,10 @@ func init() {
return nil, fmt.Errorf("unknown state %q", config.State)
},
Options: append(pikpakOAuthOptions(), []fs.Option{{
Name: "user",
Help: "Pikpak username.",
Required: true,
Name: "user",
Help: "Pikpak username.",
Required: true,
Sensitive: true,
}, {
Name: "pass",
Help: "Pikpak password.",
@@ -173,7 +174,8 @@ Leave blank normally.
Fill in for rclone to use a non root folder as its starting point.
`,
Advanced: true,
Advanced: true,
Sensitive: true,
}, {
Name: "use_trash",
Default: true,
+5 -4
View File
@@ -82,14 +82,15 @@ func init() {
OAuth2Config: oauthConfig,
})
},
Options: []fs.Option{{
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "api_key",
Help: `API Key.
This is not normally used - use oauth instead.
`,
Hide: fs.OptionHideBoth,
Default: "",
Hide: fs.OptionHideBoth,
Default: "",
Sensitive: true,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
@@ -99,7 +100,7 @@ This is not normally used - use oauth instead.
encoder.EncodeBackSlash |
encoder.EncodeDoubleQuote |
encoder.EncodeInvalidUtf8),
}},
}}...),
})
}
+2 -2
View File
@@ -67,7 +67,7 @@ func init() {
NoOffline: true,
})
},
Options: []fs.Option{{
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
Advanced: true,
@@ -77,7 +77,7 @@ func init() {
Default: (encoder.Display |
encoder.EncodeBackSlash |
encoder.EncodeInvalidUtf8),
}},
}}...),
})
}
+6 -4
View File
@@ -49,11 +49,13 @@ func init() {
Help: "Get QingStor credentials from the environment (env vars or IAM).",
}},
}, {
Name: "access_key_id",
Help: "QingStor Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
Name: "access_key_id",
Help: "QingStor Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
Sensitive: true,
}, {
Name: "secret_access_key",
Help: "QingStor Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
Name: "secret_access_key",
Help: "QingStor Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
Sensitive: true,
}, {
Name: "endpoint",
Help: "Enter an endpoint URL to connection QingStor API.\n\nLeave blank will use the default value \"https://qingstor.com:443\".",
+78 -12
View File
@@ -66,7 +66,7 @@ import (
func init() {
fs.Register(&fs.RegInfo{
Name: "s3",
Description: "Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi",
Description: "Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Leviia, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi",
NewFs: NewFs,
CommandHelp: commandHelp,
Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
@@ -127,6 +127,9 @@ func init() {
}, {
Value: "LyveCloud",
Help: "Seagate Lyve Cloud",
}, {
Value: "Leviia",
Help: "Leviia Object Storage",
}, {
Value: "Liara",
Help: "Liara Object Storage",
@@ -154,6 +157,9 @@ func init() {
}, {
Value: "Storj",
Help: "Storj (S3 Compatible Gateway)",
}, {
Value: "Synology",
Help: "Synology C2 Object Storage",
}, {
Value: "TencentCOS",
Help: "Tencent Cloud Object Storage (COS)",
@@ -179,11 +185,13 @@ func init() {
Help: "Get AWS credentials from the environment (env vars or IAM).",
}},
}, {
Name: "access_key_id",
Help: "AWS Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
Name: "access_key_id",
Help: "AWS Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
Sensitive: true,
}, {
Name: "secret_access_key",
Help: "AWS Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
Name: "secret_access_key",
Help: "AWS Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
Sensitive: true,
}, {
// References:
// 1. https://docs.aws.amazon.com/general/latest/gr/rande.html
@@ -463,10 +471,30 @@ func init() {
Value: "sa-east-1",
Help: "South America (São Paulo)",
}},
}, {
Name: "region",
Help: "Region where your data stored.\n",
Provider: "Synology",
Examples: []fs.OptionExample{{
Value: "eu-001",
Help: "Europe Region 1",
}, {
Value: "eu-002",
Help: "Europe Region 2",
}, {
Value: "us-001",
Help: "US Region 1",
}, {
Value: "us-002",
Help: "US Region 2",
}, {
Value: "tw-001",
Help: "Asia (Taiwan)",
}},
}, {
Name: "region",
Help: "Region to connect to.\n\nLeave blank if you are using an S3 clone and you don't have a region.",
Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,TencentCOS,HuaweiOBS,IDrive",
Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,Synology,TencentCOS,HuaweiOBS,IDrive",
Examples: []fs.OptionExample{{
Value: "",
Help: "Use this if unsure.\nWill use v4 signatures and an empty region.",
@@ -815,6 +843,15 @@ func init() {
Value: "s3.sa-east-1.petabox.io",
Help: "South America (São Paulo)",
}},
}, {
// Leviia endpoints: https://www.leviia.com/object-storage/
Name: "endpoint",
Help: "Endpoint for Leviia Object Storage API.",
Provider: "Leviia",
Examples: []fs.OptionExample{{
Value: "s3.leviia.com",
Help: "The default endpoint\nLeviia",
}},
}, {
// Liara endpoints: https://liara.ir/landing/object-storage
Name: "endpoint",
@@ -1000,6 +1037,26 @@ func init() {
Value: "gateway.storjshare.io",
Help: "Global Hosted Gateway",
}},
}, {
Name: "endpoint",
Help: "Endpoint for Synology C2 Object Storage API.",
Provider: "Synology",
Examples: []fs.OptionExample{{
Value: "eu-001.s3.synologyc2.net",
Help: "EU Endpoint 1",
}, {
Value: "eu-002.s3.synologyc2.net",
Help: "EU Endpoint 2",
}, {
Value: "us-001.s3.synologyc2.net",
Help: "US Endpoint 1",
}, {
Value: "us-002.s3.synologyc2.net",
Help: "US Endpoint 2",
}, {
Value: "tw-001.s3.synologyc2.net",
Help: "TW Endpoint 1",
}},
}, {
// cos endpoints: https://intl.cloud.tencent.com/document/product/436/6224
Name: "endpoint",
@@ -1156,7 +1213,7 @@ func init() {
}, {
Name: "endpoint",
Help: "Endpoint for S3 API.\n\nRequired when using an S3 clone.",
Provider: "!AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,RackCorp,Qiniu,Petabox",
Provider: "!AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,Synology,RackCorp,Qiniu,Petabox",
Examples: []fs.OptionExample{{
Value: "objects-us-east-1.dream.io",
Help: "Dream Objects endpoint",
@@ -1644,7 +1701,7 @@ func init() {
}, {
Name: "location_constraint",
Help: "Location constraint - must be set to match the Region.\n\nLeave blank if not sure. Used when creating buckets only.",
Provider: "!AWS,Alibaba,ArvanCloud,HuaweiOBS,ChinaMobile,Cloudflare,IBMCOS,IDrive,IONOS,Liara,Qiniu,RackCorp,Scaleway,StackPath,Storj,TencentCOS,Petabox",
Provider: "!AWS,Alibaba,ArvanCloud,HuaweiOBS,ChinaMobile,Cloudflare,IBMCOS,IDrive,IONOS,Leviia,Liara,Qiniu,RackCorp,Scaleway,StackPath,Storj,TencentCOS,Petabox",
}, {
Name: "acl",
Help: `Canned ACL used when creating buckets and storing or copying objects.
@@ -1659,7 +1716,7 @@ doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
`,
Provider: "!Storj,Cloudflare",
Provider: "!Storj,Synology,Cloudflare",
Examples: []fs.OptionExample{{
Value: "default",
Help: "Owner gets Full_CONTROL.\nNo one else has access rights (default).",
@@ -1775,6 +1832,7 @@ header is added and the default (private) will be used.
Value: "arn:aws:kms:us-east-1:*",
Help: "arn:aws:kms:*",
}},
Sensitive: true,
}, {
Name: "sse_customer_key",
Help: `To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
@@ -1786,6 +1844,7 @@ Alternatively you can provide --sse-customer-key-base64.`,
Value: "",
Help: "None",
}},
Sensitive: true,
}, {
Name: "sse_customer_key_base64",
Help: `If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
@@ -1797,6 +1856,7 @@ Alternatively you can provide --sse-customer-key.`,
Value: "",
Help: "None",
}},
Sensitive: true,
}, {
Name: "sse_customer_key_md5",
Help: `If using SSE-C you may provide the secret encryption key MD5 checksum (optional).
@@ -1809,6 +1869,7 @@ If you leave it blank, this is calculated automatically from the sse_customer_ke
Value: "",
Help: "None",
}},
Sensitive: true,
}, {
Name: "storage_class",
Help: "The storage class to use when storing new objects in S3.",
@@ -2050,9 +2111,10 @@ If empty it will default to the environment variable "AWS_PROFILE" or
`,
Advanced: true,
}, {
Name: "session_token",
Help: "An AWS session token.",
Advanced: true,
Name: "session_token",
Help: "An AWS session token.",
Advanced: true,
Sensitive: true,
}, {
Name: "upload_concurrency",
Help: `Concurrency for multipart uploads.
@@ -2969,11 +3031,15 @@ func setQuirks(opt *Options) {
if opt.ChunkSize < 64*fs.Mebi {
opt.ChunkSize = 64 * fs.Mebi
}
case "Synology":
useMultipartEtag = false
case "TencentCOS":
listObjectsV2 = false // untested
useMultipartEtag = false // untested
case "Wasabi":
// No quirks
case "Leviia":
// No quirks
case "Qiniu":
useMultipartEtag = false
urlEncodeListings = false
+11 -6
View File
@@ -67,15 +67,18 @@ func init() {
Value: "https://cloud.seafile.com/",
Help: "Connect to cloud.seafile.com.",
}},
Sensitive: true,
}, {
Name: configUser,
Help: "User name (usually email address).",
Required: true,
Name: configUser,
Help: "User name (usually email address).",
Required: true,
Sensitive: true,
}, {
// Password is not required, it will be left blank for 2FA
Name: configPassword,
Help: "Password.",
IsPassword: true,
Sensitive: true,
}, {
Name: config2FA,
Help: "Two-factor authentication ('true' if the account has 2FA enabled).",
@@ -87,6 +90,7 @@ func init() {
Name: configLibraryKey,
Help: "Library password (for encrypted libraries only).\n\nLeave blank if you pass it through the command line.",
IsPassword: true,
Sensitive: true,
}, {
Name: configCreateLibrary,
Help: "Should rclone create a library if it doesn't exist.",
@@ -94,9 +98,10 @@ func init() {
Default: false,
}, {
// Keep the authentication token after entering the 2FA code
Name: configAuthToken,
Help: "Authentication token.",
Hide: fs.OptionHideBoth,
Name: configAuthToken,
Help: "Authentication token.",
Hide: fs.OptionHideBoth,
Sensitive: true,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
+12 -8
View File
@@ -59,13 +59,15 @@ func init() {
Description: "SSH/SFTP",
NewFs: NewFs,
Options: []fs.Option{{
Name: "host",
Help: "SSH host to connect to.\n\nE.g. \"example.com\".",
Required: true,
Name: "host",
Help: "SSH host to connect to.\n\nE.g. \"example.com\".",
Required: true,
Sensitive: true,
}, {
Name: "user",
Help: "SSH username.",
Default: currentUser,
Name: "user",
Help: "SSH username.",
Default: currentUser,
Sensitive: true,
}, {
Name: "port",
Help: "SSH port number.",
@@ -75,8 +77,9 @@ func init() {
Help: "SSH password, leave blank to use ssh-agent.",
IsPassword: true,
}, {
Name: "key_pem",
Help: "Raw PEM-encoded private key.\n\nIf specified, will override key_file parameter.",
Name: "key_pem",
Help: "Raw PEM-encoded private key.\n\nIf specified, will override key_file parameter.",
Sensitive: true,
}, {
Name: "key_file",
Help: "Path to PEM-encoded private key file.\n\nLeave blank or set key-use-agent to use ssh-agent." + env.ShellExpandHelp,
@@ -87,6 +90,7 @@ func init() {
Only PEM encrypted key files (old OpenSSH format) are supported. Encrypted keys
in the new OpenSSH format can't be used.`,
IsPassword: true,
Sensitive: true,
}, {
Name: "pubkey_file",
Help: `Optional path to public key file.
+3 -2
View File
@@ -155,7 +155,7 @@ func init() {
CheckAuth: checkAuth,
})
},
Options: []fs.Option{{
Options: append(oauthutil.SharedOptions, []fs.Option{{
Name: "upload_cutoff",
Help: "Cutoff for switching to multipart upload.",
Default: defaultUploadCutoff,
@@ -182,6 +182,7 @@ standard values here or any folder ID (long hex number ID).`,
Value: "top",
Help: "Access the home, favorites, and shared folders as well as the connectors.",
}},
Sensitive: true,
}, {
Name: "chunk_size",
Default: defaultChunkSize,
@@ -216,7 +217,7 @@ be set manually to something like: https://XXX.sharefile.com
encoder.EncodeLeftSpace |
encoder.EncodeLeftPeriod |
encoder.EncodeInvalidUtf8),
}},
}}...),
})
}
+2 -1
View File
@@ -45,7 +45,8 @@ func init() {
Note that siad must run with --disable-api-security to open API port for other hosts (not recommended).
Keep default if Sia daemon runs on localhost.`,
Default: "http://127.0.0.1:9980",
Default: "http://127.0.0.1:9980",
Sensitive: true,
}, {
Name: "api_password",
Help: `Sia Daemon API Password.
+15 -10
View File
@@ -41,13 +41,15 @@ func init() {
NewFs: NewFs,
Options: []fs.Option{{
Name: "host",
Help: "SMB server hostname to connect to.\n\nE.g. \"example.com\".",
Required: true,
Name: "host",
Help: "SMB server hostname to connect to.\n\nE.g. \"example.com\".",
Required: true,
Sensitive: true,
}, {
Name: "user",
Help: "SMB username.",
Default: currentUser,
Name: "user",
Help: "SMB username.",
Default: currentUser,
Sensitive: true,
}, {
Name: "port",
Help: "SMB port number.",
@@ -57,9 +59,10 @@ func init() {
Help: "SMB password.",
IsPassword: true,
}, {
Name: "domain",
Help: "Domain name for NTLM authentication.",
Default: "WORKGROUP",
Name: "domain",
Help: "Domain name for NTLM authentication.",
Default: "WORKGROUP",
Sensitive: true,
}, {
Name: "spn",
Help: `Service principal name.
@@ -71,6 +74,7 @@ authentication, and it often needs to be set for clusters. For example:
Leave blank if not sure.
`,
Sensitive: true,
}, {
Name: "idle_timeout",
Default: fs.Duration(60 * time.Second),
@@ -447,7 +451,8 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
func (f *Fs) About(ctx context.Context) (_ *fs.Usage, err error) {
share, dir := f.split("/")
if share == "" {
return nil, fs.ErrorListBucketRequired
// Just return empty info rather than an error if called on the root
return &fs.Usage{}, nil
}
dir = f.toSambaPath(dir)
+12 -9
View File
@@ -98,9 +98,10 @@ func init() {
},
}},
{
Name: "access_grant",
Help: "Access grant.",
Provider: "existing",
Name: "access_grant",
Help: "Access grant.",
Provider: "existing",
Sensitive: true,
},
{
Name: "satellite_address",
@@ -120,14 +121,16 @@ func init() {
},
},
{
Name: "api_key",
Help: "API key.",
Provider: newProvider,
Name: "api_key",
Help: "API key.",
Provider: newProvider,
Sensitive: true,
},
{
Name: "passphrase",
Help: "Encryption passphrase.\n\nTo access existing objects enter passphrase used for uploading.",
Provider: newProvider,
Name: "passphrase",
Help: "Encryption passphrase.\n\nTo access existing objects enter passphrase used for uploading.",
Provider: newProvider,
Sensitive: true,
},
},
})
+29 -21
View File
@@ -132,42 +132,50 @@ func init() {
}
return nil, fmt.Errorf("unknown state %q", config.State)
}, Options: []fs.Option{{
Name: "app_id",
Help: "Sugarsync App ID.\n\nLeave blank to use rclone's.",
Name: "app_id",
Help: "Sugarsync App ID.\n\nLeave blank to use rclone's.",
Sensitive: true,
}, {
Name: "access_key_id",
Help: "Sugarsync Access Key ID.\n\nLeave blank to use rclone's.",
Name: "access_key_id",
Help: "Sugarsync Access Key ID.\n\nLeave blank to use rclone's.",
Sensitive: true,
}, {
Name: "private_access_key",
Help: "Sugarsync Private Access Key.\n\nLeave blank to use rclone's.",
Name: "private_access_key",
Help: "Sugarsync Private Access Key.\n\nLeave blank to use rclone's.",
Sensitive: true,
}, {
Name: "hard_delete",
Help: "Permanently delete files if true\notherwise put them in the deleted files.",
Default: false,
}, {
Name: "refresh_token",
Help: "Sugarsync refresh token.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Name: "refresh_token",
Help: "Sugarsync refresh token.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Sensitive: true,
}, {
Name: "authorization",
Help: "Sugarsync authorization.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Name: "authorization",
Help: "Sugarsync authorization.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Sensitive: true,
}, {
Name: "authorization_expiry",
Help: "Sugarsync authorization expiry.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
}, {
Name: "user",
Help: "Sugarsync user.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Name: "user",
Help: "Sugarsync user.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Sensitive: true,
}, {
Name: "root_id",
Help: "Sugarsync root id.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Name: "root_id",
Help: "Sugarsync root id.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Sensitive: true,
}, {
Name: "deleted_id",
Help: "Sugarsync deleted folder id.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Name: "deleted_id",
Help: "Sugarsync deleted folder id.\n\nLeave blank normally, will be auto configured by rclone.",
Advanced: true,
Sensitive: true,
}, {
Name: config.ConfigEncoding,
Help: config.ConfigEncodingHelp,
+33 -22
View File
@@ -116,11 +116,13 @@ func init() {
},
},
}, {
Name: "user",
Help: "User name to log in (OS_USERNAME).",
Name: "user",
Help: "User name to log in (OS_USERNAME).",
Sensitive: true,
}, {
Name: "key",
Help: "API key or password (OS_PASSWORD).",
Name: "key",
Help: "API key or password (OS_PASSWORD).",
Sensitive: true,
}, {
Name: "auth",
Help: "Authentication URL for server (OS_AUTH_URL).",
@@ -147,20 +149,25 @@ func init() {
Help: "Blomp Cloud Storage",
}},
}, {
Name: "user_id",
Help: "User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).",
Name: "user_id",
Help: "User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).",
Sensitive: true,
}, {
Name: "domain",
Help: "User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)",
Name: "domain",
Help: "User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)",
Sensitive: true,
}, {
Name: "tenant",
Help: "Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME).",
Name: "tenant",
Help: "Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME).",
Sensitive: true,
}, {
Name: "tenant_id",
Help: "Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID).",
Name: "tenant_id",
Help: "Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID).",
Sensitive: true,
}, {
Name: "tenant_domain",
Help: "Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME).",
Name: "tenant_domain",
Help: "Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME).",
Sensitive: true,
}, {
Name: "region",
Help: "Region name - optional (OS_REGION_NAME).",
@@ -168,17 +175,21 @@ func init() {
Name: "storage_url",
Help: "Storage URL - optional (OS_STORAGE_URL).",
}, {
Name: "auth_token",
Help: "Auth Token from alternate authentication - optional (OS_AUTH_TOKEN).",
Name: "auth_token",
Help: "Auth Token from alternate authentication - optional (OS_AUTH_TOKEN).",
Sensitive: true,
}, {
Name: "application_credential_id",
Help: "Application Credential ID (OS_APPLICATION_CREDENTIAL_ID).",
Name: "application_credential_id",
Help: "Application Credential ID (OS_APPLICATION_CREDENTIAL_ID).",
Sensitive: true,
}, {
Name: "application_credential_name",
Help: "Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME).",
Name: "application_credential_name",
Help: "Application Credential Name (OS_APPLICATION_CREDENTIAL_NAME).",
Sensitive: true,
}, {
Name: "application_credential_secret",
Help: "Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET).",
Name: "application_credential_secret",
Help: "Application Credential Secret (OS_APPLICATION_CREDENTIAL_SECRET).",
Sensitive: true,
}, {
Name: "auth_version",
Help: "AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION).",
+3 -2
View File
@@ -43,8 +43,9 @@ func init() {
Description: "Uptobox",
NewFs: NewFs,
Options: []fs.Option{{
Help: "Your access token.\n\nGet it from https://uptobox.com/my_account.",
Name: "access_token",
Help: "Your access token.\n\nGet it from https://uptobox.com/my_account.",
Name: "access_token",
Sensitive: true,
}, {
Help: "Set to make uploaded files private",
Name: "private",
+8 -9
View File
@@ -14,7 +14,6 @@ import (
"io"
"net/http"
"path"
"strings"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/readers"
@@ -41,10 +40,6 @@ func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error)
return
}
func (f *Fs) getChunksUploadURL() string {
return strings.Replace(f.endpointURL, "/dav/files/", "/dav/uploads/", 1)
}
func (o *Object) getChunksUploadDir() (string, error) {
hasher := md5.New()
_, err := hasher.Write([]byte(o.filePath()))
@@ -55,12 +50,16 @@ func (o *Object) getChunksUploadDir() (string, error) {
return uploadDir, nil
}
func (f *Fs) verifyChunkConfig() error {
if f.opt.ChunkSize != 0 && !validateNextCloudChunkedURL.MatchString(f.endpointURL) {
return errors.New("chunked upload with nextcloud must use /dav/files/USER endpoint not /webdav")
func (f *Fs) getChunksUploadURL() (string, error) {
submatch := nextCloudURLRegex.FindStringSubmatch(f.endpointURL)
if submatch == nil {
return "", errors.New("the remote url looks incorrect. Note that nextcloud chunked uploads require you to use the /dav/files/USER endpoint instead of /webdav. Please check 'rclone config show remotename' to verify that the url field ends in /dav/files/USERNAME")
}
return nil
baseURL, user := submatch[1], submatch[2]
chunksUploadURL := fmt.Sprintf("%s/dav/uploads/%s/", baseURL, user)
return chunksUploadURL, nil
}
func (o *Object) shouldUseChunkedUpload(src fs.ObjectInfo) bool {
+19 -9
View File
@@ -96,15 +96,17 @@ func init() {
Help: "Other site/service or software",
}},
}, {
Name: "user",
Help: "User name.\n\nIn case NTLM authentication is used, the username should be in the format 'Domain\\User'.",
Name: "user",
Help: "User name.\n\nIn case NTLM authentication is used, the username should be in the format 'Domain\\User'.",
Sensitive: true,
}, {
Name: "pass",
Help: "Password.",
IsPassword: true,
}, {
Name: "bearer_token",
Help: "Bearer token instead of user/pass (e.g. a Macaroon).",
Name: "bearer_token",
Help: "Bearer token instead of user/pass (e.g. a Macaroon).",
Sensitive: true,
}, {
Name: "bearer_token_command",
Help: "Command to run to get a bearer token.",
@@ -569,7 +571,8 @@ func (f *Fs) fetchAndSetBearerToken() error {
return nil
}
var validateNextCloudChunkedURL = regexp.MustCompile(`^.*/dav/files/[^/]+/?$`)
// The WebDAV url can optionally be suffixed with a path. This suffix needs to be ignored for determining the temporary upload directory of chunks.
var nextCloudURLRegex = regexp.MustCompile(`^(.*)/dav/files/([^/]+)`)
// setQuirks adjusts the Fs for the vendor passed in
func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
@@ -592,11 +595,18 @@ func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
f.propsetMtime = true
f.hasOCSHA1 = true
f.canChunk = true
if err := f.verifyChunkConfig(); err != nil {
return err
if f.opt.ChunkSize == 0 {
fs.Logf(nil, "Chunked uploads are disabled because nextcloud_chunk_size is set to 0")
} else {
chunksUploadURL, err := f.getChunksUploadURL()
if err != nil {
return err
}
f.chunksUploadURL = chunksUploadURL
fs.Logf(nil, "Chunks temporary upload directory: %s", f.chunksUploadURL)
}
f.chunksUploadURL = f.getChunksUploadURL()
fs.Logf(nil, "Chunks temporary upload directory: %s", f.chunksUploadURL)
case "sharepoint":
// To mount sharepoint, two Cookies are required
// They have to be set instead of BasicAuth
+1 -1
View File
@@ -114,7 +114,7 @@ commands_order = [
ignore_docs = [
"downloads.md",
"privacy.md",
"donate.md",
"sponsor.md",
]
def read_doc(doc):
+3
View File
@@ -35,6 +35,7 @@ import (
fslog "github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/fs/rc/rcflags"
"github.com/rclone/rclone/fs/rc/rcserver"
fssync "github.com/rclone/rclone/fs/sync"
"github.com/rclone/rclone/lib/atexit"
"github.com/rclone/rclone/lib/buildinfo"
"github.com/rclone/rclone/lib/exitcode"
@@ -501,6 +502,8 @@ func resolveExitCode(err error) {
os.Exit(exitcode.UncategorizedError)
case errors.Is(err, accounting.ErrorMaxTransferLimitReached):
os.Exit(exitcode.TransferExceeded)
case errors.Is(err, fssync.ErrorMaxDurationReached):
os.Exit(exitcode.DurationExceeded)
case fserrors.ShouldRetry(err):
os.Exit(exitcode.RetryError)
case fserrors.IsNoRetryError(err), fserrors.IsNoLowLevelRetryError(err):
+30
View File
@@ -26,6 +26,7 @@ func init() {
configCommand.AddCommand(configTouchCommand)
configCommand.AddCommand(configPathsCommand)
configCommand.AddCommand(configShowCommand)
configCommand.AddCommand(configRedactedCommand)
configCommand.AddCommand(configDumpCommand)
configCommand.AddCommand(configProvidersCommand)
configCommand.AddCommand(configCreateCommand)
@@ -118,6 +119,35 @@ var configShowCommand = &cobra.Command{
},
}
var configRedactedCommand = &cobra.Command{
Use: "redacted [<remote>]",
Short: `Print redacted (decrypted) config file, or the redacted config for a single remote.`,
Long: `This prints a redacted copy of the config file, either the
whole config file or for a given remote.
The config file will be redacted by replacing all passwords and other
sensitive info with XXX.
This makes the config file suitable for posting online for support.
It should be double checked before posting as the redaction may not be perfect.
`,
Annotations: map[string]string{
"versionIntroduced": "v1.64",
},
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(0, 1, command, args)
if len(args) == 0 {
config.ShowRedactedConfig()
} else {
name := strings.TrimRight(args[0], ":")
config.ShowRedactedRemote(name)
}
fmt.Println("### Double check the config for sensitive info before posting publicly")
},
}
var configDumpCommand = &cobra.Command{
Use: "dump",
Short: `Dump the config file as JSON.`,
+20 -17
View File
@@ -4,22 +4,20 @@
package mountlib
import (
"errors"
"fmt"
"path/filepath"
"strings"
"time"
"github.com/artyom/mtab"
"github.com/moby/sys/mountinfo"
)
const (
mtabPath = "/proc/mounts"
pollInterval = 100 * time.Millisecond
)
// CheckMountEmpty checks if folder is not already a mountpoint.
// On Linux we use the OS-specific /proc/mount API so the check won't access the path.
// On Linux we use the OS-specific /proc/self/mountinfo API so the check won't access the path.
// Directories marked as "mounted" by autofs are considered not mounted.
func CheckMountEmpty(mountpoint string) error {
const msg = "directory already mounted, use --allow-non-empty to mount anyway: %s"
@@ -29,43 +27,48 @@ func CheckMountEmpty(mountpoint string) error {
return fmt.Errorf("cannot get absolute path: %s: %w", mountpoint, err)
}
entries, err := mtab.Entries(mtabPath)
infos, err := mountinfo.GetMounts(mountinfo.SingleEntryFilter(mountpointAbs))
if err != nil {
return fmt.Errorf("cannot read %s: %w", mtabPath, err)
return fmt.Errorf("cannot get mounts: %w", err)
}
foundAutofs := false
for _, entry := range entries {
if entry.Dir == mountpointAbs {
if entry.Type != "autofs" {
return fmt.Errorf(msg, mountpointAbs)
}
foundAutofs = true
for _, info := range infos {
if info.FSType != "autofs" {
return fmt.Errorf(msg, mountpointAbs)
}
foundAutofs = true
}
// It isn't safe to list an autofs in the middle of mounting
if foundAutofs {
return nil
}
return checkMountEmpty(mountpoint)
}
// CheckMountReady checks whether mountpoint is mounted by rclone.
// Only mounts with type "rclone" or "fuse.rclone" count.
func CheckMountReady(mountpoint string) error {
const msg = "mount not ready: %s"
mountpointAbs, err := filepath.Abs(mountpoint)
if err != nil {
return fmt.Errorf("cannot get absolute path: %s: %w", mountpoint, err)
}
entries, err := mtab.Entries(mtabPath)
infos, err := mountinfo.GetMounts(mountinfo.SingleEntryFilter(mountpointAbs))
if err != nil {
return fmt.Errorf("cannot read %s: %w", mtabPath, err)
return fmt.Errorf("cannot get mounts: %w", err)
}
for _, entry := range entries {
if entry.Dir == mountpointAbs && strings.Contains(entry.Type, "rclone") {
for _, info := range infos {
if strings.Contains(info.FSType, "rclone") {
return nil
}
}
return errors.New("mount not ready")
return fmt.Errorf(msg, mountpointAbs)
}
// WaitMountReady waits until mountpoint is mounted by rclone.
+3 -1
View File
@@ -21,7 +21,7 @@ notoc: true
Rclone is a command-line program to manage files on cloud storage. It
is a feature-rich alternative to cloud vendors' web storage
interfaces. [Over 40 cloud storage products](#providers) support
interfaces. [Over 70 cloud storage products](#providers) support
rclone including S3 object stores, business & consumer file storage
services, as well as standard transfer protocols.
@@ -136,6 +136,7 @@ WebDAV or S3, that work out of the box.)
{{< provider name="IDrive e2" home="https://www.idrive.com/e2/?refer=rclone" config="/s3/#idrive-e2" >}}
{{< provider name="IONOS Cloud" home="https://cloud.ionos.com/storage/object-storage" config="/s3/#ionos" >}}
{{< provider name="Koofr" home="https://koofr.eu/" config="/koofr/" >}}
{{< provider name="Leviia Object Storage" home="https://www.leviia.com/object-storage" config="/s3/#leviia" >}}
{{< provider name="Liara Object Storage" home="https://liara.ir/landing/object-storage" config="/s3/#liara-object-storage" >}}
{{< provider name="Mail.ru Cloud" home="https://cloud.mail.ru/" config="/mailru/" >}}
{{< provider name="Memset Memstore" home="https://www.memset.com/cloud/storage/" config="/swift/" >}}
@@ -170,6 +171,7 @@ WebDAV or S3, that work out of the box.)
{{< provider name="SMB / CIFS" home="https://en.wikipedia.org/wiki/Server_Message_Block" config="/smb/" >}}
{{< provider name="StackPath" home="https://www.stackpath.com/products/object-storage/" config="/s3/#stackpath" >}}
{{< provider name="Storj" home="https://storj.io/" config="/storj/" >}}
{{< provider name="Synology" home="https://c2.synology.com/en-global/object-storage/overview" config="/s3/#synology-c2" >}}
{{< provider name="SugarSync" home="https://sugarsync.com/" config="/sugarsync/" >}}
{{< provider name="Tencent Cloud Object Storage (COS)" home="https://intl.cloud.tencent.com/product/cos" config="/s3/#tencent-cos" >}}
{{< provider name="Uptobox" home="https://uptobox.com" config="/uptobox/" >}}
+5
View File
@@ -741,3 +741,8 @@ put them back in again.` >}}
* Sawada Tsunayoshi <34431649+TsunayoshiSawada@users.noreply.github.com>
* Dean Attali <daattali@gmail.com>
* Fjodor42 <molgaard@gmail.com>
* BakaWang <wa11579@hotmail.com>
* Mahad <56235065+Mahad-lab@users.noreply.github.com>
* Vladislav Vorobev <x.miere@gmail.com>
* darix <darix@users.noreply.github.com>
* Benjamin <36415086+bbenjamin-sys@users.noreply.github.com>
+21
View File
@@ -5,6 +5,27 @@ description: "Rclone Changelog"
# Changelog
## v1.63.1 - 2023-07-17
[See commits](https://github.com/rclone/rclone/compare/v1.63.0...v1.63.1)
* Bug Fixes
* build: Fix macos builds for versions < 12 (Anagh Kumar Baranwal)
* dirtree: Fix performance with large directories of directories and `--fast-list` (Nick Craig-Wood)
* operations
* Fix deadlock when using `lsd`/`ls` with `--progress` (Nick Craig-Wood)
* Fix `.rclonelink` files not being converted back to symlinks (Nick Craig-Wood)
* doc fixes (Dean Attali, Mahad, Nick Craig-Wood, Sawada Tsunayoshi, Vladislav Vorobev)
* Local
* Fix partial directory read for corrupted filesystem (Nick Craig-Wood)
* Box
* Fix reconnect failing with HTTP 400 Bad Request (albertony)
* Smb
* Fix "Statfs failed: bucket or container name is needed" when mounting (Nick Craig-Wood)
* WebDAV
* Nextcloud: fix must use /dav/files/USER endpoint not /webdav error (Paul)
* Nextcloud chunking: add more guidance for the user to check the config (darix)
## v1.63.0 - 2023-06-30
[See commits](https://github.com/rclone/rclone/compare/v1.62.0...v1.63.0)
+23 -12
View File
@@ -3,31 +3,42 @@ title: "Contact"
description: "Contact the rclone project"
---
# Contact the rclone project #
# Contact the rclone project
## Forum ##
## Forum
Forum for questions and general discussion:
* https://forum.rclone.org
- https://forum.rclone.org
## GitHub repository ##
## Business support
For business support or sponsorship enquiries please see:
- https://rclone.com/
- sponsorship@rclone.com
## GitHub repository
The project's repository is located at:
* https://github.com/rclone/rclone
- https://github.com/rclone/rclone
There you can file bug reports or contribute with pull requests.
## Twitter ##
## Twitter
You can also follow me on twitter for rclone announcements:
You can also follow Nick on twitter for rclone announcements:
* [@njcw](https://twitter.com/njcw)
- [@njcw](https://twitter.com/njcw)
## Email ##
## Email
Or if all else fails or you want to ask something private or
confidential email [Nick Craig-Wood](mailto:nick@craig-wood.com).
Please don't email me requests for help - those are better directed to
the forum. Thanks!
confidential
- info@rclone.com
Please don't email requests for help to this address - those are
better directed to the forum unless you'd like to sign up for business
support.
+20 -18
View File
@@ -1454,14 +1454,14 @@ what will happen.
### --max-duration=TIME ###
Rclone will stop scheduling new transfers when it has run for the
Rclone will stop transferring when it has run for the
duration specified.
Defaults to off.
When the limit is reached any existing transfers will complete.
When the limit is reached all transfers will stop immediately.
Use `--cutoff-mode` to modify this behaviour.
Rclone won't exit with an error if the transfer limit is reached.
Rclone will exit with exit code 10 if the duration limit is reached.
### --max-transfer=SIZE ###
@@ -1469,9 +1469,24 @@ Rclone will stop transferring when it has reached the size specified.
Defaults to off.
When the limit is reached all transfers will stop immediately.
Use `--cutoff-mode` to modify this behaviour.
Rclone will exit with exit code 8 if the transfer limit is reached.
### --cutoff-mode=hard|soft|cautious ###
This modifies the behavior of `--max-transfer` and `--max-duration`
Defaults to `--cutoff-mode=hard`.
Specifying `--cutoff-mode=hard` will stop transferring immediately
when Rclone reaches the limit.
Specifying `--cutoff-mode=soft` will stop starting new transfers
when Rclone reaches the limit.
Specifying `--cutoff-mode=cautious` will try to prevent Rclone
from reaching the limit. Only applicable for `--max-transfer`
## -M, --metadata
Setting this flag enables rclone to copy the metadata from the source
@@ -1484,20 +1499,6 @@ Add metadata `key` = `value` when uploading. This can be repeated as
many times as required. See the [#metadata](metadata section) for more
info.
### --cutoff-mode=hard|soft|cautious ###
This modifies the behavior of `--max-transfer`
Defaults to `--cutoff-mode=hard`.
Specifying `--cutoff-mode=hard` will stop transferring immediately
when Rclone reaches the limit.
Specifying `--cutoff-mode=soft` will stop starting new transfers
when Rclone reaches the limit.
Specifying `--cutoff-mode=cautious` will try to prevent Rclone
from reaching the limit.
### --modify-window=TIME ###
When checking whether a file has been modified, this is the maximum
@@ -2561,6 +2562,7 @@ it will log a high priority message if the retry was successful.
* `7` - Fatal error (one that more retries won't fix, like account suspended) (Fatal errors)
* `8` - Transfer exceeded - limit set by --max-transfer reached
* `9` - Operation successful, but no files transferred
* `10` - Duration exceeded - limit set by --max-duration reached
Environment Variables
---------------------
-68
View File
@@ -1,68 +0,0 @@
---
title: "Donations"
description: "Donations to the rclone project."
type: page
---
# {{< icon "fa fa-heart heart" >}} Donations to the rclone project
Rclone is a free open-source project with thousands of contributions
from volunteers all round the world and I would like to thank all of
you for donating your time to the project.
However, maintaining rclone is a lot of work - easily the equivalent
of a **full time job** - for me. Nothing stands still in the world of
cloud storage. Rclone needs constant attention adapting to changes by
cloud providers, adding new providers, adding new features, keeping
the integration tests working, fixing bugs and many more things!
I love doing the work and I'd like to spend more time doing it - your
support helps make that possible.
Thank you :-)
{{< nick >}}
PS I'm available for rclone and object storage related consultancy -
[email me](mailto:nick@craig-wood.com) for more info.
{{< monthly_donations >}}
## Personal users
If you are a personal user and you would like to support the project
with sponsorship as a way of saying thank you that would be most
appreciated. {{< icon "fa fa-heart heart" >}}
## Business users
If your business distributes rclone as part of its products (which the
generous MIT licence allows) or uses it internally then it would make
business sense to sponsor the rclone project to ensure that the
project you rely on stays healthy and well maintained.
If you run one of the cloud storage providers that rclone supports and
rclone is driving revenue your way then you know it makes sense to
sponsor the project. {{< icon "far fa-smile" >}}
Note that if you choose the "GitHub Sponsors" option they will provide
proper tax invoices appropriate for your country.
## Monthly donations
Monthly donations help keep rclone development sustainable in the long
run so this is the preferred option. A small amount every month is
much better than a one off donation as it allows planning for the
future.
{{< monthly_donations >}}
## One off donations
If you don't want to contribute monthly then of course we'd love a one
off donation.
{{< one_off_donations >}}
If you require a receipt or wish to contribute in a different way then
please [drop me an email](mailto:nick@craig-wood.com).
+1 -1
View File
@@ -1522,7 +1522,7 @@ be the same account as the Google Drive you want to access)
"Google Drive API".
4. Click "Credentials" in the left-side panel (not "Create
credentials", which opens the wizard), then "Create credentials"
credentials", which opens the wizard).
5. If you already configured an "Oauth Consent Screen", then skip
to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button
+1 -1
View File
@@ -86,7 +86,7 @@ These flags are available for every command.
--max-delete int When synchronizing, limit the number of deletes (default -1)
--max-delete-size SizeSuffix When synchronizing, limit the total size of deletes (default off)
--max-depth int If set limits the recursion depth to this (default -1)
--max-duration Duration Maximum duration rclone will transfer data for (default 0s)
--max-duration Duration Maximum duration rclone will transfer data for (default off)
--max-size SizeSuffix Only transfer files smaller than this in KiB or suffix B|K|M|G|T|P (default off)
--max-stats-groups int Maximum number of stats groups to keep in memory, on max oldest is discarded (default 1000)
--max-transfer SizeSuffix Maximum size of data to transfer (default off)
-2
View File
@@ -8,8 +8,6 @@ versionIntroduced: "v1.50"
[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a Russian internet company [Mail.Ru Group](https://mail.ru). The official desktop client is [Disk-O:](https://disk-o.cloud/en), available on Windows and Mac OS.
Currently it is recommended to disable 2FA on Mail.ru accounts intended for rclone until it gets eventually implemented.
## Features highlights
- Paths may be as deep as required, e.g. `remote:directory/subdirectory`
+292 -9
View File
@@ -22,6 +22,7 @@ The S3 backend can be used with a number of different providers:
{{< provider name="IBM COS S3" home="http://www.ibm.com/cloud/object-storage" config="/s3/#ibm-cos-s3" >}}
{{< provider name="IDrive e2" home="https://www.idrive.com/e2/?refer=rclone" config="/s3/#idrive-e2" >}}
{{< provider name="IONOS Cloud" home="https://cloud.ionos.com/storage/object-storage" config="/s3/#ionos" >}}
{{< provider name="Leviia Object Storage" home="https://www.leviia.com/object-storage/" config="/s3/#leviia" >}}
{{< provider name="Liara Object Storage" home="https://liara.ir/landing/object-storage" config="/s3/#liara-cloud" >}}
{{< provider name="Minio" home="https://www.minio.io/" config="/s3/#minio" >}}
{{< provider name="Petabox" home="https://petabox.io/" config="/s3/#petabox" >}}
@@ -32,6 +33,7 @@ The S3 backend can be used with a number of different providers:
{{< provider name="SeaweedFS" home="https://github.com/chrislusf/seaweedfs/" config="/s3/#seaweedfs" >}}
{{< provider name="StackPath" home="https://www.stackpath.com/products/object-storage/" config="/s3/#stackpath" >}}
{{< provider name="Storj" home="https://storj.io/" config="/s3/#storj" >}}
{{< provider name="Synology C2 Object Storage" home="https://c2.synology.com/en-global/object-storage/overview" config="/s3/#synology-c2" >}}
{{< provider name="Tencent Cloud Object Storage (COS)" home="https://intl.cloud.tencent.com/product/cos" config="/s3/#tencent-cos" >}}
{{< provider name="Wasabi" home="https://wasabi.com/" config="/s3/#wasabi" end="true" >}}
{{< /provider_list >}}
@@ -649,7 +651,7 @@ A simple solution is to set the `--s3-upload-cutoff 0` and force all the files t
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/s3/s3.go then run make backenddocs" >}}
### Standard options
Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi).
Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi).
#### --s3-provider
@@ -708,6 +710,8 @@ Properties:
- StackPath Object Storage
- "Storj"
- Storj (S3 Compatible Gateway)
- "Synology"
- Synology C2 Object Storage
- "TencentCOS"
- Tencent Cloud Object Storage (COS)
- "Wasabi"
@@ -1069,7 +1073,7 @@ Properties:
- Config: region
- Env Var: RCLONE_S3_REGION
- Provider: !AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,TencentCOS,HuaweiOBS,IDrive
- Provider: !AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,IONOS,Petabox,Liara,Qiniu,RackCorp,Scaleway,Storj,Synology,TencentCOS,HuaweiOBS,IDrive
- Type: string
- Required: false
- Examples:
@@ -1565,6 +1569,29 @@ Properties:
#### --s3-endpoint
Endpoint for Synology C2 Object Storage API.
Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: Synology
- Type: string
- Required: false
- Examples:
- "eu-001.s3.synologyc2.net"
- Europe Region 1
- "eu-002.s3.synologyc2.net"
- Europe Region 2
- "us-001.s3.synologyc2.net"
- US Region 1
- "us-002.s3.synologyc2.net"
- US Region 2
- "tw-001.s3.synologyc2.net"
- Asia Region (Taiwan)
#### --s3-endpoint
Endpoint for Tencent COS API.
Properties:
@@ -1702,7 +1729,7 @@ Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: !AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,RackCorp,Qiniu,Petabox
- Provider: !AWS,ArvanCloud,IBMCOS,IDrive,IONOS,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,GCS,Liara,Scaleway,StackPath,Storj,Synology,RackCorp,Qiniu,Petabox
- Type: string
- Required: false
- Examples:
@@ -2368,7 +2395,7 @@ Properties:
### Advanced options
Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi).
Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, China Mobile, Cloudflare, GCS, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi).
#### --s3-bucket-acl
@@ -3421,7 +3448,7 @@ Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
...
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
\ (s3)
...
Storage> s3
@@ -3605,7 +3632,7 @@ Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
\ (s3)
[snip]
Storage> 5
@@ -3900,7 +3927,7 @@ Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
\ (s3)
[snip]
Storage> s3
@@ -4006,7 +4033,7 @@ Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
\ (s3)
[snip]
Storage> s3
@@ -4252,7 +4279,7 @@ Choose a number from below, or type in your own value
\ (alias)
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS, Qiniu and Wasabi
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
\ (s3)
[snip]
Storage> s3
@@ -5132,7 +5159,129 @@ e) Edit this remote
d) Delete this remote
y/e/d> y
```
### Leviia Cloud Object Storage {#leviia}
[Leviia Object Storage](https://www.leviia.com/object-storage/), backup and secure your data in a 100% French cloud, independent of GAFAM..
To configure access to Leviia, follow the steps below:
1. Run `rclone config` and select `n` for a new remote.
```
rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
```
2. Give the name of the configuration. For example, name it 'leviia'.
```
name> leviia
```
3. Select `s3` storage.
```
Choose a number from below, or type in your own value
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Liara, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
\ (s3)
[snip]
Storage> s3
```
4. Select `Leviia` provider.
```
Choose a number from below, or type in your own value
1 / Amazon Web Services (AWS) S3
\ "AWS"
[snip]
15 / Leviia Object Storage
\ (Leviia)
[snip]
provider> Leviia
```
5. Enter your SecretId and SecretKey of Leviia.
```
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
1 / Enter AWS credentials in the next step
\ "false"
2 / Get AWS credentials from the environment (env vars or IAM)
\ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> ZnIx.xxxxxxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxx
```
6. Select endpoint for Leviia.
```
/ The default endpoint
1 | Leviia.
\ (s3.leviia.com)
[snip]
endpoint> 1
7. Choose acl.
```
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
2 | The AllUsers group gets READ access.
\ (public-read)
[snip]
acl> 1
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[leviia]
- type: s3
- provider: Leviia
- access_key_id: ZnIx.xxxxxxx
- secret_access_key: xxxxxxxx
- endpoint: s3.leviia.com
- acl: private
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
leviia s3
```
### Liara {#liara-cloud}
Here is an example of making a [Liara Object Storage](https://liara.ir/landing/object-storage)
@@ -5749,3 +5898,137 @@ use policy `mfs` (most free space) as a member of an rclone union
remote.
See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
### Synology C2 Object Storage {#synology-c2}
[Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.
The S3 compatible gateway is configured using `rclone config` with a
type of `s3` and with a provider name of `Synology`. Here is an example
run of the configurator.
First run:
```
rclone config
```
This will guide you through an interactive setup process.
```
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
Enter name for new remote.1
name> syno
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, GCS, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, IONOS Cloud, Liara, Lyve Cloud, Minio, Netease, Petabox, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS, Qiniu and Wasabi
\ "s3"
Storage> s3
Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
24 / Synology C2 Object Storage
\ (Synology)
provider> Synology
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
1 / Enter AWS credentials in the next step
\ "false"
2 / Get AWS credentials from the environment (env vars or IAM)
\ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> accesskeyid
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> secretaccesskey
Region where your data stored.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Europe Region 1
\ (eu-001)
2 / Europe Region 2
\ (eu-002)
3 / US Region 1
\ (us-001)
4 / US Region 2
\ (us-002)
5 / Asia (Taiwan)
\ (tw-001)
region > 1
Option endpoint.
Endpoint for Synology C2 Object Storage API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / EU Endpoint 1
\ (eu-001.s3.synologyc2.net)
2 / US Endpoint 1
\ (us-001.s3.synologyc2.net)
3 / TW Endpoint 1
\ (tw-001.s3.synologyc2.net)
endpoint> 1
Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> y
Option no_check_bucket.
If set, don't attempt to check the bucket exists or create it.
This can be useful when trying to minimise the number of transactions
rclone does if you know the bucket exists already.
It can also be needed if the user you are using does not have bucket
creation permissions. Before v1.52.0 this would have passed silently
due to a bug.
Enter a boolean value (true or false). Press Enter for the default (true).
no_check_bucket> true
Configuration complete.
Options:
- type: s3
- provider: Synology
- region: eu-001
- endpoint: eu-001.s3.synologyc2.net
- no_check_bucket: true
Keep this "syno" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
+53 -9
View File
@@ -302,19 +302,41 @@ func mustFindByName(name string) *fs.RegInfo {
return fs.MustFind(fsType)
}
// findByName finds the RegInfo for the remote name passed in or
// returns an error
func findByName(name string) (*fs.RegInfo, error) {
fsType := FileGet(name, "type")
if fsType == "" {
return nil, fmt.Errorf("couldn't find type of fs for %q", name)
}
return fs.Find(fsType)
}
// printRemoteOptions prints the options of the remote
func printRemoteOptions(name string, prefix string, sep string) {
fs := mustFindByName(name)
func printRemoteOptions(name string, prefix string, sep string, redacted bool) {
fsInfo, err := findByName(name)
if err != nil {
fmt.Printf("# %v\n", err)
fsInfo = nil
}
for _, key := range LoadedData().GetKeyList(name) {
isPassword := false
for _, option := range fs.Options {
if option.Name == key && option.IsPassword {
isPassword = true
break
isSensitive := false
if fsInfo != nil {
for _, option := range fsInfo.Options {
if option.Name == key {
if option.IsPassword {
isPassword = true
} else if option.Sensitive {
isSensitive = true
}
}
}
}
value := FileGet(name, key)
if isPassword && value != "" {
if redacted && (isSensitive || isPassword) && value != "" {
fmt.Printf("%s%s%sXXX\n", prefix, key, sep)
} else if isPassword && value != "" {
fmt.Printf("%s%s%s*** ENCRYPTED ***\n", prefix, key, sep)
} else {
fmt.Printf("%s%s%s%s\n", prefix, key, sep, value)
@@ -324,13 +346,19 @@ func printRemoteOptions(name string, prefix string, sep string) {
// listRemoteOptions lists the options of the remote
func listRemoteOptions(name string) {
printRemoteOptions(name, "- ", ": ")
printRemoteOptions(name, "- ", ": ", false)
}
// ShowRemote shows the contents of the remote in config file format
func ShowRemote(name string) {
fmt.Printf("[%s]\n", name)
printRemoteOptions(name, "", " = ")
printRemoteOptions(name, "", " = ", false)
}
// ShowRedactedRemote shows the contents of the remote in config file format
func ShowRedactedRemote(name string) {
fmt.Printf("[%s]\n", name)
printRemoteOptions(name, "", " = ", true)
}
// OkRemote prints the contents of the remote and ask if it is OK
@@ -634,6 +662,22 @@ func ShowConfig() {
fmt.Printf("%s", str)
}
// ShowRedactedConfig prints the redacted (unencrypted) config options
func ShowRedactedConfig() {
remotes := LoadedData().GetSectionList()
if len(remotes) == 0 {
fmt.Println("; empty config")
return
}
sort.Strings(remotes)
for i, remote := range remotes {
if i != 0 {
fmt.Println()
}
ShowRedactedRemote(remote)
}
}
// EditConfig edits the config file interactively
func EditConfig(ctx context.Context) (err error) {
for {
+1
View File
@@ -243,6 +243,7 @@ func TestOptionMarshalJSON(t *testing.T) {
"NoPrefix": false,
"Advanced": true,
"Exclusive": false,
"Sensitive": false,
"DefaultStr": "false",
"ValueStr": "true",
"Type": "bool"
+29 -36
View File
@@ -358,7 +358,7 @@ func Copy(ctx context.Context, f fs.Fs, dst fs.Object, remote string, src fs.Obj
inplace = true
remotePartial = remote
)
if !ci.Inplace && f.Features().Move != nil && f.Features().PartialUploads {
if !ci.Inplace && f.Features().Move != nil && f.Features().PartialUploads && !strings.HasSuffix(remote, ".rclonelink") {
// Avoid making the leaf name longer if it's already lengthy to avoid
// trouble with file name length limits.
suffix := "." + random.String(8) + ".partial"
@@ -826,17 +826,19 @@ func Same(fdst, fsrc fs.Info) bool {
return SameConfig(fdst, fsrc) && strings.Trim(fdst.Root(), "/") == strings.Trim(fsrc.Root(), "/")
}
// fixRoot returns the Root with a trailing / if not empty. It is
// aware of case insensitive filesystems.
func fixRoot(f fs.Info) string {
s := strings.Trim(filepath.ToSlash(f.Root()), "/")
// fixRoot returns the Root with a trailing / if not empty.
//
// It returns a case folded version for case insensitive file systems
func fixRoot(f fs.Info) (s string, folded string) {
s = strings.Trim(filepath.ToSlash(f.Root()), "/")
if s != "" {
s += "/"
}
folded = s
if f.Features().CaseInsensitive {
s = strings.ToLower(s)
folded = strings.ToLower(s)
}
return s
return s, folded
}
// OverlappingFilterCheck returns true if fdst and fsrc point to the same
@@ -845,37 +847,28 @@ func OverlappingFilterCheck(ctx context.Context, fdst fs.Fs, fsrc fs.Fs) bool {
if !SameConfig(fdst, fsrc) {
return false
}
fdstRoot := fixRoot(fdst)
fsrcRoot := fixRoot(fsrc)
if strings.HasPrefix(fdstRoot, fsrcRoot) {
fdstRoot, fdstRootFolded := fixRoot(fdst)
fsrcRoot, fsrcRootFolded := fixRoot(fsrc)
if fdstRootFolded == fsrcRootFolded {
return true
} else if strings.HasPrefix(fdstRootFolded, fsrcRootFolded) {
fdstRelative := fdstRoot[len(fsrcRoot):]
return filterCheckR(ctx, fdstRelative, 0, fsrc)
return filterCheck(ctx, fsrc, fdstRelative)
} else if strings.HasPrefix(fsrcRootFolded, fdstRootFolded) {
fsrcRelative := fsrcRoot[len(fdstRoot):]
return filterCheck(ctx, fdst, fsrcRelative)
}
return strings.HasPrefix(fsrcRoot, fdstRoot)
return false
}
// filterCheckR checks if fdst would be included in the sync
func filterCheckR(ctx context.Context, fdstRelative string, pos int, fsrc fs.Fs) bool {
include := true
// filterCheck checks if dir is included in f
func filterCheck(ctx context.Context, f fs.Fs, dir string) bool {
fi := filter.GetConfig(ctx)
includeDirectory := fi.IncludeDirectory(ctx, fsrc)
dirs := strings.SplitAfterN(fdstRelative, "/", pos+2)
newPath := ""
for i := 0; i <= pos; i++ {
newPath += dirs[i]
}
if !strings.HasSuffix(newPath, "/") {
newPath += "/"
}
if strings.HasPrefix(fdstRelative, newPath) {
include, _ = includeDirectory(newPath)
if include {
if newPath == fdstRelative {
return true
}
pos++
include = filterCheckR(ctx, fdstRelative, pos, fsrc)
}
includeDirectory := fi.IncludeDirectory(ctx, f)
include, err := includeDirectory(dir)
if err != nil {
fs.Errorf(f, "Failed to discover whether directory is included: %v", err)
return true
}
return include
}
@@ -886,9 +879,9 @@ func SameDir(fdst, fsrc fs.Info) bool {
if !SameConfig(fdst, fsrc) {
return false
}
fdstRoot := fixRoot(fdst)
fsrcRoot := fixRoot(fsrc)
return fdstRoot == fsrcRoot
_, fdstRootFolded := fixRoot(fdst)
_, fsrcRootFolded := fixRoot(fsrc)
return fdstRootFolded == fsrcRootFolded
}
// Retry runs fn up to maxTries times if it returns a retriable error
+13 -4
View File
@@ -1418,11 +1418,13 @@ func TestOverlappingFilterCheckWithFilter(t *testing.T) {
ctx := context.Background()
fi, err := filter.NewFilter(nil)
require.NoError(t, err)
require.NoError(t, fi.Add(false, "*/exclude/"))
fi.Opt.ExcludeFile = []string{".ignore"}
require.NoError(t, fi.Add(false, "/exclude/"))
require.NoError(t, fi.Add(false, "/Exclude2/"))
require.NoError(t, fi.Add(true, "*"))
ctx = filter.ReplaceConfig(ctx, fi)
src := &testFs{testFsInfo{name: "name", root: "root"}}
src.features.CaseInsensitive = true
slash := string(os.PathSeparator) // native path separator
for _, test := range []struct {
name string
@@ -1430,25 +1432,32 @@ func TestOverlappingFilterCheckWithFilter(t *testing.T) {
expected bool
}{
{"name", "root", true},
{"name", "ROOT", true}, // case insensitive is set
{"name", "/root", true},
{"name", "root/", true},
{"name", "root" + slash, true},
{"name", "root/exclude", false},
{"name", "root/Exclude2", false},
{"name", "root/include", true},
{"name", "root/exclude/", false},
{"name", "root/Exclude2/", false},
{"name", "root/exclude/sub", false},
{"name", "root/Exclude2/sub", false},
{"name", "/root/exclude/", false},
{"name", "root" + slash + "exclude", false},
{"name", "root" + slash + "exclude" + slash, false},
{"name", "root/.ignore", false},
{"name", "root" + slash + ".ignore", false},
{"namey", "root/include", false},
{"namey", "root/include/", false},
{"namey", "root" + slash + "include", false},
{"namey", "root" + slash + "include" + slash, false},
} {
dst := &testFs{testFsInfo{name: test.name, root: test.root}}
dst.features.CaseInsensitive = true
what := fmt.Sprintf("(%q,%q) vs (%q,%q)", src.name, src.root, dst.name, dst.root)
actual := operations.OverlappingFilterCheck(ctx, dst, src)
assert.Equal(t, test.expected, actual, what)
actual = operations.OverlappingFilterCheck(ctx, src, dst)
assert.Equal(t, test.expected, actual, what)
}
}
+1 -4
View File
@@ -389,10 +389,7 @@ func rcSetSoftMemoryLimit(ctx context.Context, in Params) (out Params, err error
if err != nil {
return nil, err
}
oldMemLimit, err := debug.SetMemoryLimit(memLimit)
if err != nil {
return nil, err
}
oldMemLimit := debug.SetMemoryLimit(memLimit)
out = Params{
"existing-mem-limit": oldMemLimit,
}
+1
View File
@@ -154,6 +154,7 @@ type Option struct {
NoPrefix bool // set if the option for this should not use the backend prefix
Advanced bool // set if this is an advanced config option
Exclusive bool // set if the answer can only be one of the examples (empty string allowed unless Required or Default is set)
Sensitive bool // set if this option should be redacted when using rclone config redacted
}
// BaseOption is an alias for Option used internally
+10 -6
View File
@@ -20,6 +20,14 @@ import (
"github.com/rclone/rclone/fs/operations"
)
// ErrorMaxDurationReached defines error when transfer duration is reached
// Used for checking on exit and matching to correct exit code.
var ErrorMaxDurationReached = errors.New("max transfer duration reached as set by --max-duration")
// ErrorMaxDurationReachedFatal is returned from when the max
// duration limit is reached.
var ErrorMaxDurationReachedFatal = fserrors.FatalError(ErrorMaxDurationReached)
type syncCopyMove struct {
// parameters
fdst fs.Fs
@@ -845,10 +853,6 @@ func (s *syncCopyMove) tryRename(src fs.Object) bool {
return true
}
// errorMaxDurationReached defines error when transfer duration is reached
// Used for checking on exit and matching to correct exit code.
var errorMaxDurationReached = fserrors.FatalError(errors.New("max transfer duration reached as set by --max-duration"))
// Syncs fsrc into fdst
//
// If Delete is true then it deletes any files in fdst that aren't in fsrc
@@ -945,8 +949,8 @@ func (s *syncCopyMove) run() error {
// If the duration was exceeded then add a Fatal Error so we don't retry
if !s.maxDurationEndTime.IsZero() && time.Since(s.maxDurationEndTime) > 0 {
fs.Errorf(s.fdst, "%v", errorMaxDurationReached)
s.processError(errorMaxDurationReached)
fs.Errorf(s.fdst, "%v", ErrorMaxDurationReachedFatal)
s.processError(ErrorMaxDurationReachedFatal)
}
// Print nothing to transfer message if there were no transfers and no errors
+15 -7
View File
@@ -996,7 +996,7 @@ func testSyncWithMaxDuration(t *testing.T, cutoffMode fs.CutoffMode) {
accounting.GlobalStats().ResetCounters()
startTime := time.Now()
err := Sync(ctx, r.Fremote, r.Flocal, false)
require.True(t, errors.Is(err, errorMaxDurationReached))
require.True(t, errors.Is(err, ErrorMaxDurationReached))
if cutoffMode == fs.CutoffModeHard {
r.CheckRemoteItems(t, file1)
@@ -1423,7 +1423,7 @@ func TestSyncOverlapWithFilter(t *testing.T) {
require.NoError(t, fi.Add(false, "/rclone-sync-test/"))
require.NoError(t, fi.Add(false, "*/layer2/"))
fi.Opt.ExcludeFile = []string{".ignore"}
ctx = filter.ReplaceConfig(ctx, fi)
filterCtx := filter.ReplaceConfig(ctx, fi)
subRemoteName := r.FremoteName + "/rclone-sync-test"
FremoteSync, err := fs.NewFs(ctx, subRemoteName)
@@ -1453,19 +1453,27 @@ func TestSyncOverlapWithFilter(t *testing.T) {
}
accounting.GlobalStats().ResetCounters()
checkNoErr(Sync(ctx, FremoteSync, r.Fremote, false))
checkNoErr(Sync(filterCtx, FremoteSync, r.Fremote, false))
checkErr(Sync(ctx, FremoteSync, r.Fremote, false))
checkNoErr(Sync(filterCtx, r.Fremote, FremoteSync, false))
checkErr(Sync(ctx, r.Fremote, FremoteSync, false))
checkErr(Sync(filterCtx, r.Fremote, r.Fremote, false))
checkErr(Sync(ctx, r.Fremote, r.Fremote, false))
checkErr(Sync(filterCtx, FremoteSync, FremoteSync, false))
checkErr(Sync(ctx, FremoteSync, FremoteSync, false))
checkNoErr(Sync(ctx, FremoteSync2, r.Fremote, false))
checkNoErr(Sync(filterCtx, FremoteSync2, r.Fremote, false))
checkErr(Sync(ctx, FremoteSync2, r.Fremote, false))
checkNoErr(Sync(filterCtx, r.Fremote, FremoteSync2, false))
checkErr(Sync(ctx, r.Fremote, FremoteSync2, false))
checkErr(Sync(ctx, r.Fremote, r.Fremote, false))
checkErr(Sync(filterCtx, FremoteSync2, FremoteSync2, false))
checkErr(Sync(ctx, FremoteSync2, FremoteSync2, false))
checkNoErr(Sync(ctx, FremoteSync3, r.Fremote, false))
checkNoErr(Sync(filterCtx, FremoteSync3, r.Fremote, false))
checkErr(Sync(ctx, FremoteSync3, r.Fremote, false))
checkNoErr(Sync(filterCtx, r.Fremote, FremoteSync3, false))
checkErr(Sync(ctx, r.Fremote, FremoteSync3, false))
checkErr(Sync(ctx, r.Fremote, r.Fremote, false))
checkErr(Sync(filterCtx, FremoteSync3, FremoteSync3, false))
checkErr(Sync(ctx, FremoteSync3, FremoteSync3, false))
}
+4 -4
View File
@@ -1,8 +1,8 @@
//go:build !go1.18
// +build !go1.18
//go:build !go1.19
// +build !go1.19
package fs
// Upgrade to Go version 1.18 to compile rclone - latest stable go
// Upgrade to Go version 1.19 to compile rclone - latest stable go
// compiler recommended.
func init() { Go_version_1_18_required_for_compilation() }
func init() { Go_version_1_19_required_for_compilation() }
+2 -2
View File
@@ -73,7 +73,7 @@ func main() {
configfile.Install()
// Seed the random number generator
rand.Seed(time.Now().UTC().UnixNano())
randInstance := rand.New(rand.NewSource(time.Now().UTC().UnixNano()))
// Filter selection
if *testRemotes != "" {
@@ -103,7 +103,7 @@ func main() {
// Runs we will do for this test in random order
runs := conf.MakeRuns()
rand.Shuffle(len(runs), runs.Swap)
randInstance.Shuffle(len(runs), runs.Swap)
// Create Report
report := NewReport()
+2 -2
View File
@@ -1,6 +1,6 @@
module github.com/rclone/rclone
go 1.18
go 1.19
require (
bazil.org/fuse v0.0.0-20221209211307-2abb8038c751
@@ -15,7 +15,6 @@ require (
github.com/abbot/go-http-auth v0.4.0
github.com/anacrolix/dms v1.5.0
github.com/anacrolix/log v0.13.1
github.com/artyom/mtab v1.0.0
github.com/atotto/clipboard v0.1.4
github.com/aws/aws-sdk-go v1.44.246
github.com/buengese/sgzip v0.1.1
@@ -39,6 +38,7 @@ require (
github.com/mattn/go-colorable v0.1.13
github.com/mattn/go-runewidth v0.0.14
github.com/mitchellh/go-homedir v1.1.0
github.com/moby/sys/mountinfo v0.6.2
github.com/ncw/go-acd v0.0.0-20201019170801-fe55f33415b1
github.com/ncw/swift/v2 v2.0.1
github.com/oracle/oci-go-sdk/v65 v65.34.0
-2
View File
@@ -82,8 +82,6 @@ github.com/anacrolix/log v0.13.1 h1:BmVwTdxHd5VcNrLylgKwph4P4wf+5VvPgOK4yi91fTY=
github.com/anacrolix/log v0.13.1/go.mod h1:D4+CvN8SnruK6zIFS/xPoRJmtvtnxs+CSfDQ+BFxZ68=
github.com/anacrolix/missinggo v1.1.0/go.mod h1:MBJu3Sk/k3ZfGYcS7z18gwfu72Ey/xopPFJJbTi5yIo=
github.com/anacrolix/tagflag v0.0.0-20180109131632-2146c8d41bf0/go.mod h1:1m2U/K6ZT+JZG0+bdMK6qauP49QT4wE5pmhJXOKKCHw=
github.com/artyom/mtab v1.0.0 h1:r7OSVo5Jeqi8+LotZ0rT2kzfPIBp9KCpEJP8RQqGmSE=
github.com/artyom/mtab v1.0.0/go.mod h1:EHpkp5OmPfS1yZX+/DFTztlJ9di5UzdDLX1/XzWPXw8=
github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
github.com/aws/aws-sdk-go v1.44.246 h1:iLxPX6JU0bxAci9R6/bp8rX0kL871ByCTx0MZlQWv1U=
+6
View File
@@ -10,3 +10,9 @@ import (
func SetGCPercent(percent int) int {
return debug.SetGCPercent(percent)
}
// SetMemoryLimit calls the runtime/debug.SetMemoryLimit function to set the
// soft-memory limit.
func SetMemoryLimit(limit int64) int64 {
return debug.SetMemoryLimit(limit)
}
-14
View File
@@ -1,14 +0,0 @@
//go:build go1.19
// +build go1.19
package debug
import (
"runtime/debug"
)
// SetMemoryLimit calls the runtime/debug.SetMemoryLimit function to set the
// soft-memory limit.
func SetMemoryLimit(limit int64) (int64, error) {
return debug.SetMemoryLimit(limit), nil
}
-14
View File
@@ -1,14 +0,0 @@
//go:build !go1.19
// +build !go1.19
package debug
import (
"fmt"
"runtime"
)
// SetMemoryLimit is a no-op on Go version < 1.19.
func SetMemoryLimit(limit int64) (int64, error) {
return limit, fmt.Errorf("not implemented on Go version below 1.19: %s", runtime.Version())
}
+2
View File
@@ -22,4 +22,6 @@ const (
TransferExceeded
// NoFilesTransferred everything succeeded, but no transfer was made.
NoFilesTransferred
// DurationExceeded is returned when transfer duration exceeded the quota.
DurationExceeded
)
+10 -7
View File
@@ -82,15 +82,18 @@ All done. Please go back to rclone.
// SharedOptions are shared between backends the utilize an OAuth flow
var SharedOptions = []fs.Option{{
Name: config.ConfigClientID,
Help: "OAuth Client Id.\n\nLeave blank normally.",
Name: config.ConfigClientID,
Help: "OAuth Client Id.\n\nLeave blank normally.",
Sensitive: true,
}, {
Name: config.ConfigClientSecret,
Help: "OAuth Client Secret.\n\nLeave blank normally.",
Name: config.ConfigClientSecret,
Help: "OAuth Client Secret.\n\nLeave blank normally.",
Sensitive: true,
}, {
Name: config.ConfigToken,
Help: "OAuth Access Token as a JSON blob.",
Advanced: true,
Name: config.ConfigToken,
Help: "OAuth Access Token as a JSON blob.",
Advanced: true,
Sensitive: true,
}, {
Name: config.ConfigAuthURL,
Help: "Auth server URL.\n\nLeave blank to use the provider defaults.",
+1 -2
View File
@@ -11,7 +11,6 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
"mime/multipart"
"net/http"
"net/url"
@@ -163,7 +162,7 @@ const drainLimit = 10 * 1024 * 1024
// drainAndClose discards up to drainLimit bytes from r and closes
// it. Any errors from the Read or Close are returned.
func drainAndClose(r io.ReadCloser) (err error) {
_, readErr := io.CopyN(ioutil.Discard, r, drainLimit)
_, readErr := io.CopyN(io.Discard, r, drainLimit)
if readErr == io.EOF {
readErr = nil
}