Compare commits

...
Author SHA1 Message Date
Cnly 108ec53c0c onedrive: fix driveID of src being moved not determined correctly 2018-12-26 16:27:58 +08:00
Cnly 64e303321b onedrive: make moving between drives fall back to copying then deleting 2018-12-26 15:57:16 +08:00
Cnly 7d87386d58 fstests: fix TestFsName fails when using remote:with/path 2018-12-26 15:14:25 +08:00
Cnly beb773d20c onedrive: fix broken support for "shared with me" folders - #2536, #2778 2018-12-26 15:10:12 +08:00
Animosity022 5babf2dc5c Update drive.md
Fixed the Google Drive API documentation
2018-12-22 18:37:00 +00:00
Nick Craig-Wood 9012d7c6c1 cmd: fix --progress crash under Windows Jenkins - fixes #2846 2018-12-22 18:05:13 +00:00
Nick Craig-Wood df1faa9a8f webdav: fail soft on time parsing errors
The time format provided by webdav servers seems to vary wildly from
that specified in the RFC - rclone already parses times in 5 different
formats!

If an unparseable time is found, then fail softly logging an ERROR
(just once) but returning the epoch.

This will mean that webdav servers with bad time formats will still be
usable by rclone.
2018-12-20 12:10:15 +00:00
Nick Craig-Wood 3de7ad5223 b2: for a bucket limited application key check the bucket name
Before this fix rclone would just use the authorised bucket regardless
of what bucket you put on the command line.

This uses the new `bucketName` response in the API and checks that the
user is using the correct bucket name to avoid accidents.

Fixes #2839
2018-12-20 12:07:35 +00:00
Garry McNulty 9cb3a68c38 crypt: check for maximum length before decrypting filename
The EME Transform() method will panic if the input data is larger than
2048 bytes.

Fixes #2826
2018-12-19 11:51:44 +00:00
Nick Craig-Wood c1dd76788d httplib: make http serving with auth generate INFO messages on auth fail
2018/12/13 12:13:44 INFO  : /: 127.0.0.1:39696: Basic auth challenge sent
2018/12/13 12:13:54 INFO  : /: 127.0.0.1:40050: Unauthorized request from ncw

Fixes #2834
2018-12-14 13:38:49 +00:00
Nick Craig-Wood 5ee1816a71 filter: parallelise reading of --files-from - fixes #2835
Before this change rclone would read the list of files from the
files-from parameter and check they existed one at a time.  This could
take a very long time for lots of files.

After this change, rclone will check up to --checkers in parallel.
2018-12-13 13:22:30 +00:00
Nick Craig-Wood 63b51c6742 vendor: add golang.org/x/sync as a dependency 2018-12-13 10:45:52 +00:00
Nick Craig-Wood e7684b7ed5 Add William Cocker to contributors 2018-12-06 21:53:53 +00:00
William Cocker dda23baf42 s3 : update doc for Glacier storage class
s3 : update doc for Glacier storage class : related to #923
2018-12-06 21:53:38 +00:00
William Cocker 8575abf599 s3: add GLACIER storage class
Fixes #923
2018-12-06 21:53:05 +00:00
Nick Craig-Wood feea0532cd azureblob: ignore directory markers - fixes #2806
This ignores 0 length blobs if
- they end with /
- they have the metadata hdi_isfolder = true
2018-12-06 21:47:03 +00:00
Nick Craig-Wood d3e8ae1820 Add Mark Otway to contributors 2018-12-06 15:13:03 +00:00
Nick Craig-Wood 91a9a959a2 Add Mathieu Carbou to contributors 2018-12-06 15:12:58 +00:00
Mark Otway 04eae51d11 Fix install for Synology
7z check doesn't work due to misplaced comma, so installation fails on Synology.
2018-12-06 15:12:21 +00:00
Mathieu Carbou 8fb707e16d Fixes #1788: Retry-After support for Dropbox backend 2018-12-05 22:03:30 +00:00
Mathieu Carbou 4138d5aa75 Issue #1788: Pointing to Dropbox's v5.0.0 tag 2018-12-05 22:03:30 +00:00
Nick Craig-Wood fc654a4cec http: fix backend with --files-from and non-existent files
Before this fix the http backend was returning the wrong error code
when files were not found.  This was causing --files-from to error on
missing files instead of skipping them like it should.
2018-12-04 17:40:44 +00:00
Nick Craig-Wood 26b5f55cba Update after goimports change 2018-12-04 10:11:57 +00:00
Nick Craig-Wood 3f572e6bf2 webdav: fix infinite loop on failed directory creation - fixes #2714 2018-12-02 21:03:12 +00:00
Nick Craig-Wood 941ad6bc62 azureblob: use the s3 pacer for 0 delay - fixes #2799 2018-12-02 20:55:16 +00:00
Nick Craig-Wood 5d1d93e163 azureblob: use the rclone HTTP client - fixes #2654
This enables --dump headers and --timeout to work properly.
2018-12-02 20:55:16 +00:00
Nick Craig-Wood 35fba5bfdd Add Garry McNulty to contributors 2018-12-02 20:52:04 +00:00
Garry McNulty 887834da91 b2: cleanup unfinished large files
The `cleanup` command will delete unfinished large file uploads that
were started more than a day ago (to avoid deleting uploads that are
potentially still in progress).

Fixes #2617
2018-12-02 20:51:13 +00:00
Nick Craig-Wood 107293c80e copy,move: restore --no-traverse flag
The --no-traverse flag was not implemented when the new sync routines
(using the march package) was implemented.

This re-implements --no-traverse in march by trying to find a match
for each object with NewObject rather than from a directory listing.
2018-12-02 20:28:13 +00:00
Nick Craig-Wood e3c4ebd59a march: factor calling parameters into a structure 2018-12-02 18:07:26 +00:00
Nick Craig-Wood d99ffde7c0 s3: change --s3-upload-concurrency default to 4 to increase perfomance #2772
Increasing the --s3-upload-concurrency to 4 (from 2) gives an
additional 45% throughput at the cost of 10MB extra memory per transfer.

After testing the upload perfoc
2018-12-02 17:58:34 +00:00
Nick Craig-Wood 198c34ce21 s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772
Before this change rclone would use multipart uploads for any size of
file.  However multipart uploads are less efficient for smaller files
and don't have MD5 checksums so it is advantageous to use single part
uploads if possible.

This implements single part uploads for all files smaller than the
upload_cutoff size.  Streamed files must be uploaded as multipart
files though.
2018-12-02 17:58:34 +00:00
Nick Craig-Wood 0eba88bbfe sftp: check directory is empty before issuing rmdir
Some SFTP servers allow rmdir on full directories which is allowed
according to the RFC so make sure we don't accidentally delete data
here.

See: https://forum.rclone.org/t/rmdir-and-delete-empty-src-dirs-file-does-not-exist/7737
2018-12-02 11:16:30 +00:00
Nick Craig-Wood aeea4430d5 swift: efficiency: slim Object and reduce requests on upload
- Slim down Object to only include necessary data
- Don't HEAD an object after PUT - read the hash from the response
2018-12-02 10:23:55 +00:00
Nick Craig-Wood 4b15c4215c sftp: fix rmdir on Windows based servers (eg CrushFTP)
Before this change we used Remove to remove directories.  This works
fine on Unix based systems but not so well on Windows based ones.
Swap to using RemoveDirectory instead.
2018-11-29 21:34:37 +00:00
Nick Craig-Wood 50452207d9 swift: add --swift-no-chunk to disable segmented uploads in rcat/mount
Fixes #2776
2018-11-29 11:11:30 +00:00
Nick Craig-Wood 01fcad9b9c rc: fix docs for sync/{sync,copy,move} and operations/{copy,move}file 2018-11-29 11:11:30 +00:00
themylogin eb41253764 azureblob: allow building azureblob backend on *BSD
FreeBSD support was added in Azure/azure-storage-blob-go@0562badec5
OpenBSD and NetBSD support was added in Azure/azure-storage-blob-go@1d6dd77d74
2018-11-27 12:20:48 +00:00
Nick Craig-Wood 89625e54cf vendor: update dependencies to latest 2018-11-26 14:10:33 +00:00
Nick Craig-Wood 58f7141c96 drive, googlecloudstorage: disallow on go1.8 due to dependent library changes
golang.org/x/oauth2/google no longer builds on go1.8
2018-11-26 14:10:33 +00:00
Nick Craig-Wood e56c6402a7 serve restic: disallow on go1.8 because of dependent library changes
golang.org/x/net/http2 no longer builds on go1.8
2018-11-26 14:10:33 +00:00
Nick Craig-Wood d0eb8ddc30 serve webdav: disallow on go1.8 due to dependent library changes
golang.org/x/net/webdav no longer builds with go1.8
2018-11-26 14:10:33 +00:00
Nick Craig-Wood a6c28a5faa Start v1.45-DEV development 2018-11-24 15:20:24 +00:00
Nick Craig-Wood d35bd15762 Version v1.45 2018-11-24 13:44:25 +00:00
Nick Craig-Wood 8b8220c4f7 azureblob: wait for up to 60s to create a just deleted container
When a container is deleted, a container with the same name cannot be
created for at least 30 seconds; the container may not be available
for more than 30 seconds if the service is still processing the
request.

We sleep so that we wait at most 60 seconds.  This is mostly useful in
the integration tests where containers get deleted and remade
immediately.
2018-11-24 10:57:37 +00:00
Nick Craig-Wood 5fe3b0ad71 Add Stephen Harris to contributors 2018-11-24 10:57:37 +00:00
Stephen Harris 4c8c87a935 Update PROXY section of the FAQ 2018-11-23 20:14:36 +00:00
Nick Craig-Wood bb10a51b39 test_all: limit to go1.11 so the template used is supported 2018-11-23 17:17:19 +00:00
Nick Craig-Wood df01f7a4eb test_all: fix regexp for retrying nested tests 2018-11-23 17:17:19 +00:00
Nick Craig-Wood e84790ef79 swift: add pacer for retries to make swift more reliable #2740 2018-11-22 22:15:52 +00:00
Nick Craig-Wood 369a8ee17b ncdu: fix deleting files 2018-11-22 21:41:17 +00:00
Nick Craig-Wood 84e21ade6b cmount: fix on Linux - only apply volname for Windows and macOS 2018-11-22 20:41:05 +00:00
Sebastian Bünger 703b0535a4 yandex: update docs 2018-11-22 20:14:50 +00:00
Sebastian Bünger 155264ae12 yandex: complete rewrite
Get rid of the api client and use rest/pacer for all API calls
Add Copy, Move, DirMove, PublicLink, About optional interfaces
Improve general error handling
Remove ListR for now due to inconsitent behaviour
fixes #2586, progress on #2740 and #2178
2018-11-22 20:14:50 +00:00
Nick Craig-Wood 31e2ce03c3 fstests: re-arrange backend integration tests so they can be retried
Before this change backend integration tests depended on each other,
so tests could not be retried.

After this change we nest tests to ensure that tests are provided with
the starting state they expect.

Tell the integration test runner that it can retry backend tests also.

This also includes bin/test_independence.go which runs each test
individually for a backend to prove that they are independent.
2018-11-22 20:12:12 +00:00
Nick Craig-Wood e969505ae4 info: fix control character map output 2018-11-20 14:04:27 +00:00
Nick Craig-Wood 26e2f1a998 Add Alexander to contributors 2018-11-20 10:22:11 +00:00
Alexander 2682d5a9cf - install with busybox if any 2018-11-20 10:22:00 +00:00
Nick Craig-Wood 2191592e80 Add Henry Ptasinski to contributors 2018-11-19 13:33:59 +00:00
Nick Craig-Wood 18f758294e Add Peter Kaminski to contributors 2018-11-19 13:33:59 +00:00
Henry Ptasinski f95c1c61dd s3: add config info for Wasabi's US-West endpoint
Wasabi has two location, US East and US West, with different endpoint URLs.
When configuring S3 to use Wasabi, provide the endpoint information for both
locations.
2018-11-19 13:33:42 +00:00
Nick Craig-Wood 8c8dcdd521 webdav: fix config parsing so --webdav-user and --webdav-pass flags work 2018-11-17 13:14:54 +00:00
Nick Craig-Wood 141c133818 fstest: Wait for longer if neccessary in TestFsChangeNotify 2018-11-16 07:45:24 +00:00
Nick Craig-Wood 0f03e55cd1 fstests: ignore main directory creation in TestFsChangeNotify 2018-11-15 18:39:28 +00:00
Nick Craig-Wood 9e6ba92a11 fstests: attempt to fix TestFsChangeNotify flakiness
This now uses testPut to upload the test files which will retry on
errors properly.
2018-11-15 18:39:28 +00:00
Nick Craig-Wood 762561f88e fstest: factor out retry logic from put code and make testPut return the object too 2018-11-15 18:39:28 +00:00
Nick Craig-Wood 084fe38922 fstests: fixes the integration test errors running crypt over swift.
Skip tests involving errors creating or removing dirs on non root
bucket based fs
2018-11-15 18:39:28 +00:00
Peter Kaminski 63a2a935fc fix typos in original files, per #2727 review request 2018-11-14 22:48:58 +00:00
Peter Kaminski 64fce8438b docs: Fix a couple of minor typos in rclone_mount.md
* "transferring" instead of "transfering"
* "connection" instead of "connnection"
* "mount" instead of "mount mount"
2018-11-14 22:48:58 +00:00
Nick Craig-Wood f92beb4e14 fstest: Fix TestPurge causing errors with subsequent tests on azure
Before this change TestPurge would remove a container and subsequent
tests would fail because the container was still being deleted so
couldn't be created.

This was fixed by introducing an fstest.NewRunIndividual() test runner
for TestPurge which causes the test to be run on a new container.
2018-11-14 17:14:02 +00:00
Nick Craig-Wood f7ce2e8d95 azureblob: fix erroneous Rmdir error "directory not empty"
Before this change Rmdir would check the root rather than the
directory specified for being empty and return "directory not empty"
when it shouldn't have done.
2018-11-14 17:13:39 +00:00
Nick Craig-Wood 3975d82b3b Add brused27 to contributors 2018-11-13 17:00:26 +00:00
brused27 d87aa33ec5 azureblob: Avoid context deadline exceeded error by setting a large TryTimeout value - Fixes #2647 2018-11-13 16:59:53 +00:00
Anagh Kumar Baranwal 1b78f4d1ea Changed the docs scripts to use $HOME & $USER instead of specific values
Signed-off-by: Anagh Kumar Baranwal <anaghk.dos@gmail.com>
2018-11-13 11:00:34 +00:00
Nick Craig-Wood b3704597f3 cmount: make --volname work for Windows - fixes #2679 2018-11-12 16:32:02 +00:00
Nick Craig-Wood 16f797a7d7 filter: add --ignore-case flag - fixes #502
The --ignore-case flag causes the filtering of file names to be case
insensitive.  The flag name comes from GNU tar.
2018-11-12 14:29:37 +00:00
Nick Craig-Wood ee700ec01a lib/readers: add mutex to RepeatableReader - fixes #2572 2018-11-12 12:02:05 +00:00
Nick Craig-Wood 9b3c951ab7 Add Jake Coggiano to contributors 2018-11-12 11:34:28 +00:00
Jake Coggiano 22d17e79e3 dropbox: add dropbox impersonate support - fixes #2577 2018-11-12 11:33:39 +00:00
Jake Coggiano 6d3088a00b vendor: add github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/team/ 2018-11-12 11:33:39 +00:00
Nick Craig-Wood 84202c7471 onedrive: note 50,000 files is limit for one directory #2707 2018-11-11 15:22:19 +00:00
Nick Craig-Wood 96a05516f9 acd,box,onedrive,pcloud: remote log.Fatal from NewFs
And replace with error returns.
2018-11-11 11:00:14 +00:00
Nick Craig-Wood 4f6a942595 cmd: Make --progress update the stats right at the end
Before this when rclone exited the stats would just show the last
printed version, rather than the actual final state.
2018-11-11 09:57:37 +00:00
Nick Craig-Wood c4b0a37b21 rc: improve docs on debugging 2018-11-10 10:18:13 +00:00
Nick Craig-Wood 9322f4baef Add Erik Swanson to contributors 2018-11-08 12:58:41 +00:00
Erik Swanson fa0a1e7261 s3: fix role_arn, credential_source, ...
When the env_auth option is enabled, the AWS SDK's session constructor
now loads configuration from ~/.aws/config and environment variables,
and credentials per the selected (or default) AWS_PROFILE's settings.

This is accomplished by **NOT** including any Credential provider in the
aws.Config passed to the session constructor: If the Config.Credentials
is non-nil, that will always be used and the user's configuration re
role_arn, credential_source, source_profile, etc... from the shared
config will be completely ignored.

(The conditional creation and configuration of the stscreds Credential
provider is complicated enough that it is not worth re-creating that
logic.)
2018-11-08 12:58:23 +00:00
Nick Craig-Wood 4ad08794c9 fserrors: add "server closed idle connection" to retriable errors
This seems to be related to this go issue: https://github.com/golang/go/issues/19943

See: https://forum.rclone.org/t/copy-from-dropbox-to-google-drive-yields-failed-to-copy-failed-to-open-source-object-server-closed-idle-connection-error/7460
2018-11-08 11:12:25 +00:00
Nick Craig-Wood c0f600764b Add Scott Edlund to contributors 2018-11-07 14:27:06 +00:00
Scott Edlund f139e07380 enable softfloat on MIPS arch
MIPS does not have a floating point unit.  Enable softfloat to build binaries run on devices that do not have MIPS_FPU enabled in their kernel.
2018-11-07 14:26:48 +00:00
Nick Craig-Wood c6786eeb2d move: don't create directories with --dry-run - fixes #2676 2018-11-06 13:34:15 +00:00
Nick Craig-Wood 57b85b8155 rc: fix job tests on Windows 2018-11-06 13:03:48 +00:00
389 changed files with 24050 additions and 7636 deletions

No files matched your search

+569 -64
View File
File diff suppressed because it is too large. Load diff
+815 -55
View File
File diff suppressed because it is too large. Load diff
+784 -39
View File
File diff suppressed because it is too large. Load diff
+1 -1
View File
@@ -116,7 +116,7 @@ MANUAL.txt: MANUAL.md
pandoc -s --from markdown --to plain MANUAL.md -o MANUAL.txt
commanddocs: rclone
rclone gendocs docs/content/commands/
XDG_CACHE_HOME="" XDG_CONFIG_HOME="" HOME="\$$HOME" USER="\$$USER" rclone gendocs docs/content/commands/
backenddocs: rclone bin/make_backend_docs.py
./bin/make_backend_docs.py
+2 -2
View File
@@ -21,7 +21,7 @@ import (
"strings"
"time"
"github.com/ncw/go-acd"
acd "github.com/ncw/go-acd"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/configmap"
@@ -264,7 +264,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
oAuthClient, ts, err := oauthutil.NewClientWithBaseClient(name, m, acdConfig, baseClient)
if err != nil {
log.Fatalf("Failed to configure Amazon Drive: %v", err)
return nil, errors.Wrap(err, "failed to configure Amazon Drive")
}
c := acd.NewClient(oAuthClient)
+90 -26
View File
@@ -1,6 +1,6 @@
// Package azureblob provides an interface to the Microsoft Azure blob object storage system
// +build !freebsd,!netbsd,!openbsd,!plan9,!solaris,go1.8
// +build !plan9,!solaris,go1.8
package azureblob
@@ -22,12 +22,14 @@ import (
"sync"
"time"
"github.com/Azure/azure-pipeline-go/pipeline"
"github.com/Azure/azure-storage-blob-go/azblob"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/accounting"
"github.com/ncw/rclone/fs/config/configmap"
"github.com/ncw/rclone/fs/config/configstruct"
"github.com/ncw/rclone/fs/fserrors"
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/lib/pacer"
@@ -50,6 +52,7 @@ const (
defaultUploadCutoff = 256 * fs.MebiByte
maxUploadCutoff = 256 * fs.MebiByte
defaultAccessTier = azblob.AccessTierNone
maxTryTimeout = time.Hour * 24 * 365 //max time of an azure web request response window (whether or not data is flowing)
)
// Register with Fs
@@ -134,6 +137,7 @@ type Fs struct {
root string // the path we are working on if any
opt Options // parsed config options
features *fs.Features // optional features
client *http.Client // http client we are using
svcURL *azblob.ServiceURL // reference to serviceURL
cntURL *azblob.ContainerURL // reference to containerURL
container string // the container we are working on
@@ -271,6 +275,38 @@ func (f *Fs) setUploadCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
return
}
// httpClientFactory creates a Factory object that sends HTTP requests
// to a rclone's http.Client.
//
// copied from azblob.newDefaultHTTPClientFactory
func httpClientFactory(client *http.Client) pipeline.Factory {
return pipeline.FactoryFunc(func(next pipeline.Policy, po *pipeline.PolicyOptions) pipeline.PolicyFunc {
return func(ctx context.Context, request pipeline.Request) (pipeline.Response, error) {
r, err := client.Do(request.WithContext(ctx))
if err != nil {
err = pipeline.NewError(err, "HTTP request failed")
}
return pipeline.NewHTTPResponse(r), err
}
})
}
// newPipeline creates a Pipeline using the specified credentials and options.
//
// this code was copied from azblob.NewPipeline
func (f *Fs) newPipeline(c azblob.Credential, o azblob.PipelineOptions) pipeline.Pipeline {
// Closest to API goes first; closest to the wire goes last
factories := []pipeline.Factory{
azblob.NewTelemetryPolicyFactory(o.Telemetry),
azblob.NewUniqueRequestIDPolicyFactory(),
azblob.NewRetryPolicyFactory(o.Retry),
c,
pipeline.MethodFactoryMarker(), // indicates at what stage in the pipeline the method factory is invoked
azblob.NewRequestLogPolicyFactory(o.RequestLog),
}
return pipeline.NewPipeline(factories, pipeline.Options{HTTPSender: httpClientFactory(f.client), Log: o.Log})
}
// NewFs contstructs an Fs from the path, container:path
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// Parse config into Options struct
@@ -306,6 +342,23 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
string(azblob.AccessTierHot), string(azblob.AccessTierCool), string(azblob.AccessTierArchive))
}
f := &Fs{
name: name,
opt: *opt,
container: container,
root: directory,
pacer: pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant).SetPacer(pacer.S3Pacer),
uploadToken: pacer.NewTokenDispenser(fs.Config.Transfers),
client: fshttp.NewClient(fs.Config),
}
f.features = (&fs.Features{
ReadMimeType: true,
WriteMimeType: true,
BucketBased: true,
SetTier: true,
GetTier: true,
}).Fill(f)
var (
u *url.URL
serviceURL azblob.ServiceURL
@@ -322,7 +375,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
if err != nil {
return nil, errors.Wrap(err, "failed to make azure storage url from account and endpoint")
}
pipeline := azblob.NewPipeline(credential, azblob.PipelineOptions{})
pipeline := f.newPipeline(credential, azblob.PipelineOptions{Retry: azblob.RetryOptions{TryTimeout: maxTryTimeout}})
serviceURL = azblob.NewServiceURL(*u, pipeline)
containerURL = serviceURL.NewContainerURL(container)
case opt.SASURL != "":
@@ -331,7 +384,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
return nil, errors.Wrapf(err, "failed to parse SAS URL")
}
// use anonymous credentials in case of sas url
pipeline := azblob.NewPipeline(azblob.NewAnonymousCredential(), azblob.PipelineOptions{})
pipeline := f.newPipeline(azblob.NewAnonymousCredential(), azblob.PipelineOptions{Retry: azblob.RetryOptions{TryTimeout: maxTryTimeout}})
// Check if we have container level SAS or account level sas
parts := azblob.NewBlobURLParts(*u)
if parts.ContainerName != "" {
@@ -348,24 +401,9 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
default:
return nil, errors.New("Need account+key or connectionString or sasURL")
}
f.svcURL = &serviceURL
f.cntURL = &containerURL
f := &Fs{
name: name,
opt: *opt,
container: container,
root: directory,
svcURL: &serviceURL,
cntURL: &containerURL,
pacer: pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
uploadToken: pacer.NewTokenDispenser(fs.Config.Transfers),
}
f.features = (&fs.Features{
ReadMimeType: true,
WriteMimeType: true,
BucketBased: true,
SetTier: true,
GetTier: true,
}).Fill(f)
if f.root != "" {
f.root += "/"
// Check to see if the (container,directory) is actually an existing file
@@ -471,6 +509,7 @@ func (f *Fs) list(dir string, recurse bool, maxResults uint, fn listFn) error {
MaxResults: int32(maxResults),
}
ctx := context.Background()
directoryMarkers := map[string]struct{}{}
for marker := (azblob.Marker{}); marker.NotDone(); {
var response *azblob.ListBlobsHierarchySegmentResponse
err := f.pacer.Call(func() (bool, error) {
@@ -500,13 +539,29 @@ func (f *Fs) list(dir string, recurse bool, maxResults uint, fn listFn) error {
continue
}
remote := file.Name[len(f.root):]
// Check for directory
isDirectory := strings.HasSuffix(remote, "/")
if isDirectory {
remote = remote[:len(remote)-1]
// is this a directory marker?
if *file.Properties.ContentLength == 0 {
// Note that metadata with hdi_isfolder = true seems to be a
// defacto standard for marking blobs as directories.
endsWithSlash := strings.HasSuffix(remote, "/")
if endsWithSlash || remote == "" || file.Metadata["hdi_isfolder"] == "true" {
if endsWithSlash {
remote = remote[:len(remote)-1]
}
err = fn(remote, file, true)
if err != nil {
return err
}
// Keep track of directory markers. If recursing then
// there will be no Prefixes so no need to keep track
if !recurse {
directoryMarkers[remote] = struct{}{}
}
continue // skip directory marker
}
}
// Send object
err = fn(remote, file, isDirectory)
err = fn(remote, file, false)
if err != nil {
return err
}
@@ -519,6 +574,10 @@ func (f *Fs) list(dir string, recurse bool, maxResults uint, fn listFn) error {
continue
}
remote = remote[len(f.root):]
// Don't send if already sent as a directory marker
if _, found := directoryMarkers[remote]; found {
continue
}
// Send object
err = fn(remote, nil, true)
if err != nil {
@@ -705,6 +764,11 @@ func (f *Fs) Mkdir(dir string) error {
f.containerOK = true
return false, nil
case azblob.ServiceCodeContainerBeingDeleted:
// From https://docs.microsoft.com/en-us/rest/api/storageservices/delete-container
// When a container is deleted, a container with the same name cannot be created
// for at least 30 seconds; the container may not be available for more than 30
// seconds if the service is still processing the request.
time.Sleep(6 * time.Second) // default 10 retries will be 60 seconds
f.containerDeleted = true
return true, err
}
@@ -722,7 +786,7 @@ func (f *Fs) Mkdir(dir string) error {
// isEmpty checks to see if a given directory is empty and returns an error if not
func (f *Fs) isEmpty(dir string) (err error) {
empty := true
err = f.list("", true, 1, func(remote string, object *azblob.BlobItem, isDirectory bool) error {
err = f.list(dir, true, 1, func(remote string, object *azblob.BlobItem, isDirectory bool) error {
empty = false
return nil
})
+1 -1
View File
@@ -1,4 +1,4 @@
// +build !freebsd,!netbsd,!openbsd,!plan9,!solaris,go1.8
// +build !plan9,!solaris,go1.8
package azureblob
+1 -1
View File
@@ -1,6 +1,6 @@
// Test AzureBlob filesystem interface
// +build !freebsd,!netbsd,!openbsd,!plan9,!solaris,go1.8
// +build !plan9,!solaris,go1.8
package azureblob
+1 -1
View File
@@ -1,6 +1,6 @@
// Build for azureblob for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build freebsd netbsd openbsd plan9 solaris !go1.8
// +build plan9 solaris !go1.8
package azureblob
+1
View File
@@ -136,6 +136,7 @@ type AuthorizeAccountResponse struct {
AccountID string `json:"accountId"` // The identifier for the account.
Allowed struct { // An object (see below) containing the capabilities of this auth token, and any restrictions on using it.
BucketID string `json:"bucketId"` // When present, access is restricted to one bucket.
BucketName string `json:"bucketName"` // When present, name of bucket - may be empty
Capabilities []string `json:"capabilities"` // A list of strings, each one naming a capability the key has.
NamePrefix interface{} `json:"namePrefix"` // When present, access is restricted to files whose names start with the prefix
} `json:"allowed"`
+16
View File
@@ -368,6 +368,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
// If this is a key limited to a single bucket, it must exist already
if f.bucket != "" && f.info.Allowed.BucketID != "" {
allowedBucket := f.info.Allowed.BucketName
if allowedBucket == "" {
return nil, errors.New("bucket that application key is restricted to no longer exists")
}
if allowedBucket != f.bucket {
return nil, errors.Errorf("you must use bucket %q with this application key", allowedBucket)
}
f.markBucketOK()
f.setBucketID(f.info.Allowed.BucketID)
}
@@ -980,6 +987,12 @@ func (f *Fs) purge(oldOnly bool) error {
errReturn = err
}
}
var isUnfinishedUploadStale = func(timestamp api.Timestamp) bool {
if time.Since(time.Time(timestamp)).Hours() > 24 {
return true
}
return false
}
// Delete Config.Transfers in parallel
toBeDeleted := make(chan *api.File, fs.Config.Transfers)
@@ -1003,6 +1016,9 @@ func (f *Fs) purge(oldOnly bool) error {
if object.Action == "hide" {
fs.Debugf(remote, "Deleting current version (id %q) as it is a hide marker", object.ID)
toBeDeleted <- object
} else if object.Action == "start" && isUnfinishedUploadStale(object.UploadTimestamp) {
fs.Debugf(remote, "Deleting current version (id %q) as it is a start marker (upload started at %s)", object.ID, time.Time(object.UploadTimestamp).Local())
toBeDeleted <- object
} else {
fs.Debugf(remote, "Not deleting current version (id %q) %q", object.ID, object.Action)
}
+1 -1
View File
@@ -252,7 +252,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
root = parsePath(root)
oAuthClient, ts, err := oauthutil.NewClient(name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to configure Box: %v", err)
return nil, errors.Wrap(err, "failed to configure Box")
}
f := &Fs{
+1 -1
View File
@@ -15,7 +15,7 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/patrickmn/go-cache"
cache "github.com/patrickmn/go-cache"
"golang.org/x/net/websocket"
)
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"time"
"github.com/ncw/rclone/fs"
"github.com/patrickmn/go-cache"
cache "github.com/patrickmn/go-cache"
"github.com/pkg/errors"
)
+4
View File
@@ -41,6 +41,7 @@ var (
ErrorBadDecryptControlChar = errors.New("bad decryption - contains control chars")
ErrorNotAMultipleOfBlocksize = errors.New("not a multiple of blocksize")
ErrorTooShortAfterDecode = errors.New("too short after base32 decode")
ErrorTooLongAfterDecode = errors.New("too long after base32 decode")
ErrorEncryptedFileTooShort = errors.New("file is too short to be encrypted")
ErrorEncryptedFileBadHeader = errors.New("file has truncated block header")
ErrorEncryptedBadMagic = errors.New("not an encrypted file - bad magic string")
@@ -284,6 +285,9 @@ func (c *cipher) decryptSegment(ciphertext string) (string, error) {
// not possible if decodeFilename() working correctly
return "", ErrorTooShortAfterDecode
}
if len(rawCiphertext) > 2048 {
return "", ErrorTooLongAfterDecode
}
paddedPlaintext := eme.Transform(c.block, c.nameTweak[:], rawCiphertext, eme.DirectionDecrypt)
plaintext, err := pkcs7.Unpad(nameCipherBlockSize, paddedPlaintext)
if err != nil {
+5
View File
@@ -194,6 +194,10 @@ func TestEncryptSegment(t *testing.T) {
func TestDecryptSegment(t *testing.T) {
// We've tested the forwards above, now concentrate on the errors
longName := make([]byte, 3328)
for i := range longName {
longName[i] = 'a'
}
c, _ := newCipher(NameEncryptionStandard, "", "", true)
for _, test := range []struct {
in string
@@ -201,6 +205,7 @@ func TestDecryptSegment(t *testing.T) {
}{
{"64=", ErrorBadBase32Encoding},
{"!", base32.CorruptInputError(0)},
{string(longName), ErrorTooLongAfterDecode},
{encodeFileName([]byte("a")), ErrorNotAMultipleOfBlocksize},
{encodeFileName([]byte("123456789abcdef")), ErrorNotAMultipleOfBlocksize},
{encodeFileName([]byte("123456789abcdef0")), pkcs7.ErrorPaddingTooLong},
+3
View File
@@ -1,4 +1,7 @@
// Package drive interfaces with the Google Drive object storage system
// +build go1.9
package drive
// FIXME need to deal with some corner cases
+15 -4
View File
@@ -1,3 +1,5 @@
// +build go1.9
package drive
import (
@@ -243,10 +245,19 @@ func (f *Fs) InternalTestDocumentLink(t *testing.T) {
}
func (f *Fs) InternalTest(t *testing.T) {
t.Run("DocumentImport", f.InternalTestDocumentImport)
t.Run("DocumentUpdate", f.InternalTestDocumentUpdate)
t.Run("DocumentExport", f.InternalTestDocumentExport)
t.Run("DocumentLink", f.InternalTestDocumentLink)
// These tests all depend on each other so run them as nested tests
t.Run("DocumentImport", func(t *testing.T) {
f.InternalTestDocumentImport(t)
t.Run("DocumentUpdate", func(t *testing.T) {
f.InternalTestDocumentUpdate(t)
t.Run("DocumentExport", func(t *testing.T) {
f.InternalTestDocumentExport(t)
t.Run("DocumentLink", func(t *testing.T) {
f.InternalTestDocumentLink(t)
})
})
})
})
}
var _ fstests.InternalTester = (*Fs)(nil)
+3
View File
@@ -1,4 +1,7 @@
// Test Drive filesystem interface
// +build go1.9
package drive
import (
+6
View File
@@ -0,0 +1,6 @@
// Build for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build !go1.9
package drive
+2
View File
@@ -8,6 +8,8 @@
//
// This contains code adapted from google.golang.org/api (C) the GO AUTHORS
// +build go1.9
package drive
import (
+43 -2
View File
@@ -31,9 +31,11 @@ import (
"time"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/auth"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/common"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/files"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/sharing"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/team"
"github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/users"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
@@ -131,13 +133,19 @@ slightly (at most 10%% for 128MB in tests) at the cost of using more
memory. It can be set smaller if you are tight on memory.`, fs.SizeSuffix(maxChunkSize)),
Default: fs.SizeSuffix(defaultChunkSize),
Advanced: true,
}, {
Name: "impersonate",
Help: "Impersonate this user when using a business account.",
Default: "",
Advanced: true,
}},
})
}
// Options defines the configuration for this backend
type Options struct {
ChunkSize fs.SizeSuffix `config:"chunk_size"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
Impersonate string `config:"impersonate"`
}
// Fs represents a remote dropbox server
@@ -149,6 +157,7 @@ type Fs struct {
srv files.Client // the connection to the dropbox server
sharing sharing.Client // as above, but for generating sharing links
users users.Client // as above, but for accessing user information
team team.Client // for the Teams API
slashRoot string // root with "/" prefix, lowercase
slashRootSlash string // root with "/" prefix and postfix, lowercase
pacer *pacer.Pacer // To pace the API calls
@@ -195,7 +204,16 @@ func shouldRetry(err error) (bool, error) {
return false, err
}
baseErrString := errors.Cause(err).Error()
// FIXME there is probably a better way of doing this!
// handle any official Retry-After header from Dropbox's SDK first
switch e := err.(type) {
case auth.RateLimitAPIError:
if e.RateLimitError.RetryAfter > 0 {
fs.Debugf(baseErrString, "Too many requests or write operations. Trying again in %d seconds.", e.RateLimitError.RetryAfter)
time.Sleep(time.Duration(e.RateLimitError.RetryAfter) * time.Second)
}
return true, err
}
// Keep old behaviour for backward compatibility
if strings.Contains(baseErrString, "too_many_write_operations") || strings.Contains(baseErrString, "too_many_requests") {
return true, err
}
@@ -262,6 +280,29 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
Client: oAuthClient, // maybe???
HeaderGenerator: f.headerGenerator,
}
// NOTE: needs to be created pre-impersonation so we can look up the impersonated user
f.team = team.New(config)
if opt.Impersonate != "" {
user := team.UserSelectorArg{
Email: opt.Impersonate,
}
user.Tag = "email"
members := []*team.UserSelectorArg{&user}
args := team.NewMembersGetInfoArgs(members)
memberIds, err := f.team.MembersGetInfo(args)
if err != nil {
return nil, errors.Wrapf(err, "invalid dropbox team member: %q", opt.Impersonate)
}
config.AsMemberID = memberIds[0].MemberInfo.Profile.MemberProfile.TeamMemberId
}
f.srv = files.New(config)
f.sharing = sharing.New(config)
f.users = users.New(config)
@@ -1,4 +1,7 @@
// Package googlecloudstorage provides an interface to Google Cloud Storage
// +build go1.9
package googlecloudstorage
/*
@@ -1,4 +1,7 @@
// Test GoogleCloudStorage filesystem interface
// +build go1.9
package googlecloudstorage_test
import (
@@ -0,0 +1,6 @@
// Build for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build !go1.9
package googlecloudstorage
+4 -1
View File
@@ -193,7 +193,7 @@ func (f *Fs) NewObject(remote string) (fs.Object, error) {
}
err := o.stat()
if err != nil {
return nil, errors.Wrap(err, "Stat failed")
return nil, err
}
return o, nil
}
@@ -416,6 +416,9 @@ func (o *Object) url() string {
func (o *Object) stat() error {
url := o.url()
res, err := o.fs.httpClient.Head(url)
if err == nil && res.StatusCode == http.StatusNotFound {
return fs.ErrorObjectNotFound
}
err = statusError(res, err)
if err != nil {
return errors.Wrap(err, "failed to stat")
+5
View File
@@ -144,6 +144,11 @@ func TestNewObject(t *testing.T) {
dt, ok := fstest.CheckTimeEqualWithPrecision(tObj, tFile, time.Second)
assert.True(t, ok, fmt.Sprintf("%s: Modification time difference too big |%s| > %s (%s vs %s) (precision %s)", o.Remote(), dt, time.Second, tObj, tFile, time.Second))
// check object not found
o, err = f.NewObject("not found.txt")
assert.Nil(t, o)
assert.Equal(t, fs.ErrorObjectNotFound, err)
}
func TestOpen(t *testing.T) {
+1
View File
@@ -285,6 +285,7 @@ type AsyncOperationStatus struct {
// GetID returns a normalized ID of the item
// If DriveID is known it will be prefixed to the ID with # seperator
// Can be parsed using onedrive.parseNormalizedID(normalizedID)
func (i *Item) GetID() string {
if i.IsRemote() && i.RemoteItem.ID != "" {
return i.RemoteItem.ParentReference.DriveID + "#" + i.RemoteItem.ID
+133 -50
View File
@@ -334,20 +334,10 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
return authRety || fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
}
// readMetaDataForPath reads the metadata from the path
func (f *Fs) readMetaDataForPath(path string) (info *api.Item, resp *http.Response, err error) {
var opts rest.Opts
if len(path) == 0 {
opts = rest.Opts{
Method: "GET",
Path: "/root",
}
} else {
opts = rest.Opts{
Method: "GET",
Path: "/root:/" + rest.URLPathEscape(replaceReservedChars(path)),
}
}
// readMetaDataForPathRelativeToID reads the metadata for a path relative to an item that is addressed by its normalized ID.
// if `relPath` == "", it reads the metadata for the item with that ID.
func (f *Fs) readMetaDataForPathRelativeToID(normalizedID string, relPath string) (info *api.Item, resp *http.Response, err error) {
opts := newOptsCall(normalizedID, "GET", ":/"+rest.URLPathEscape(replaceReservedChars(relPath)))
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
return shouldRetry(resp, err)
@@ -356,6 +346,72 @@ func (f *Fs) readMetaDataForPath(path string) (info *api.Item, resp *http.Respon
return info, resp, err
}
// readMetaDataForPath reads the metadata from the path (relative to the absolute root)
func (f *Fs) readMetaDataForPath(path string) (info *api.Item, resp *http.Response, err error) {
firstSlashIndex := strings.IndexRune(path, '/')
if f.driveType != driveTypePersonal || firstSlashIndex == -1 {
var opts rest.Opts
if len(path) == 0 {
opts = rest.Opts{
Method: "GET",
Path: "/root",
}
} else {
opts = rest.Opts{
Method: "GET",
Path: "/root:/" + rest.URLPathEscape(replaceReservedChars(path)),
}
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
return shouldRetry(resp, err)
})
return info, resp, err
}
// The following branch handles the case when we're using OneDrive Personal and the path is in a folder.
// For OneDrive Personal, we need to consider the "shared with me" folders.
// An item in such a folder can only be addressed by its ID relative to the sharer's driveID or
// by its path relative to the folder's ID relative to the sharer's driveID.
// Note: A "shared with me" folder can only be placed in the sharee's absolute root.
// So we read metadata relative to a suitable folder's normalized ID.
var dirCacheFoundRoot bool
var rootNormalizedID string
if f.dirCache != nil {
var ok bool
if rootNormalizedID, ok = f.dirCache.Get(""); ok {
dirCacheFoundRoot = true
}
}
relPath, insideRoot := getRelativePathInsideBase(f.root, path)
var firstDir, baseNormalizedID string
if !insideRoot || !dirCacheFoundRoot {
// We do not have the normalized ID in dirCache for our query to base on. Query it manually.
firstDir, relPath = path[:firstSlashIndex], path[firstSlashIndex+1:]
info, resp, err := f.readMetaDataForPath(firstDir)
if err != nil {
return info, resp, err
}
baseNormalizedID = info.GetID()
} else {
if f.root != "" {
// Read metadata based on root
baseNormalizedID = rootNormalizedID
} else {
// Read metadata based on firstDir
firstDir, relPath = path[:firstSlashIndex], path[firstSlashIndex+1:]
baseNormalizedID, err = f.dirCache.FindDir(firstDir, false)
if err != nil {
return nil, nil, err
}
}
}
return f.readMetaDataForPathRelativeToID(baseNormalizedID, relPath)
}
// errorHandler parses a non 2xx error response into an error
func errorHandler(resp *http.Response) error {
// Decode error response
@@ -404,13 +460,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
if opt.DriveID == "" || opt.DriveType == "" {
log.Fatalf("Unable to get drive_id and drive_type. If you are upgrading from older versions of rclone, please run `rclone config` and re-configure this backend.")
return nil, errors.New("unable to get drive_id and drive_type - if you are upgrading from older versions of rclone, please run `rclone config` and re-configure this backend")
}
root = parsePath(root)
oAuthClient, ts, err := oauthutil.NewClient(name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to configure OneDrive: %v", err)
return nil, errors.Wrap(err, "failed to configure OneDrive")
}
f := &Fs{
@@ -514,18 +570,11 @@ func (f *Fs) NewObject(remote string) (fs.Object, error) {
// FindLeaf finds a directory of name leaf in the folder with ID pathID
func (f *Fs) FindLeaf(pathID, leaf string) (pathIDOut string, found bool, err error) {
// fs.Debugf(f, "FindLeaf(%q, %q)", pathID, leaf)
parent, ok := f.dirCache.GetInv(pathID)
_, ok := f.dirCache.GetInv(pathID)
if !ok {
return "", false, errors.New("couldn't find parent ID")
}
path := leaf
if parent != "" {
path = parent + "/" + path
}
if f.dirCache.FoundRoot() {
path = f.rootSlash() + path
}
info, resp, err := f.readMetaDataForPath(path)
info, resp, err := f.readMetaDataForPathRelativeToID(pathID, leaf)
if err != nil {
if resp != nil && resp.StatusCode == http.StatusNotFound {
return "", false, nil
@@ -867,13 +916,13 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
opts.ExtraHeaders = map[string]string{"Prefer": "respond-async"}
opts.NoResponse = true
id, _, _ := parseDirID(directoryID)
id, dstDriveID, _ := parseNormalizedID(directoryID)
replacedLeaf := replaceReservedChars(leaf)
copyReq := api.CopyItemRequest{
Name: &replacedLeaf,
ParentReference: api.ItemReference{
DriveID: f.driveID,
DriveID: dstDriveID,
ID: id,
},
}
@@ -940,15 +989,23 @@ func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
return nil, err
}
id, dstDriveID, _ := parseNormalizedID(directoryID)
_, srcObjDriveID, _ := parseNormalizedID(srcObj.id)
if dstDriveID != srcObjDriveID {
// https://docs.microsoft.com/en-us/graph/api/driveitem-move?view=graph-rest-1.0
// "Items cannot be moved between Drives using this request."
return nil, fs.ErrorCantMove
}
// Move the object
opts := newOptsCall(srcObj.id, "PATCH", "")
id, _, _ := parseDirID(directoryID)
move := api.MoveItemRequest{
Name: replaceReservedChars(leaf),
ParentReference: &api.ItemReference{
ID: id,
DriveID: dstDriveID,
ID: id,
},
// We set the mod time too as it gets reset otherwise
FileSystemInfo: &api.FileSystemInfoFacet{
@@ -1024,7 +1081,20 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
if err != nil {
return err
}
parsedDstDirID, _, _ := parseDirID(dstDirectoryID)
parsedDstDirID, dstDriveID, _ := parseNormalizedID(dstDirectoryID)
// Find ID of src
srcID, err := srcFs.dirCache.FindDir(srcRemote, false)
if err != nil {
return err
}
_, srcDriveID, _ := parseNormalizedID(srcID)
if dstDriveID != srcDriveID {
// https://docs.microsoft.com/en-us/graph/api/driveitem-move?view=graph-rest-1.0
// "Items cannot be moved between Drives using this request."
return fs.ErrorCantDirMove
}
// Check destination does not exist
if dstRemote != "" {
@@ -1038,14 +1108,8 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
}
}
// Find ID of src
srcID, err := srcFs.dirCache.FindDir(srcRemote, false)
if err != nil {
return err
}
// Get timestamps of src so they can be preserved
srcInfo, _, err := srcFs.readMetaDataForPath(srcPath)
srcInfo, _, err := srcFs.readMetaDataForPathRelativeToID(srcID, "")
if err != nil {
return err
}
@@ -1055,7 +1119,8 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
move := api.MoveItemRequest{
Name: replaceReservedChars(leaf),
ParentReference: &api.ItemReference{
ID: parsedDstDirID,
DriveID: dstDriveID,
ID: parsedDstDirID,
},
// We set the mod time too as it gets reset otherwise
FileSystemInfo: &api.FileSystemInfoFacet{
@@ -1122,7 +1187,7 @@ func (f *Fs) PublicLink(remote string) (link string, err error) {
if err != nil {
return "", err
}
opts := newOptsCall(info.ID, "POST", "/createLink")
opts := newOptsCall(info.GetID(), "POST", "/createLink")
share := api.CreateShareLinkRequest{
Type: "view",
@@ -1270,13 +1335,13 @@ func (o *Object) ModTime() time.Time {
// setModTime sets the modification time of the local fs object
func (o *Object) setModTime(modTime time.Time) (*api.Item, error) {
var opts rest.Opts
_, directoryID, _ := o.fs.dirCache.FindPath(o.remote, false)
_, drive, rootURL := parseDirID(directoryID)
leaf, directoryID, _ := o.fs.dirCache.FindPath(o.remote, false)
trueDirID, drive, rootURL := parseNormalizedID(directoryID)
if drive != "" {
opts = rest.Opts{
Method: "PATCH",
RootURL: rootURL,
Path: "/" + drive + "/root:/" + rest.URLPathEscape(o.srvPath()),
Path: "/" + drive + "/items/" + trueDirID + ":/" + rest.URLPathEscape(leaf),
}
} else {
opts = rest.Opts{
@@ -1344,7 +1409,7 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
// createUploadSession creates an upload session for the object
func (o *Object) createUploadSession(modTime time.Time) (response *api.CreateUploadResponse, err error) {
leaf, directoryID, _ := o.fs.dirCache.FindPath(o.remote, false)
id, drive, rootURL := parseDirID(directoryID)
id, drive, rootURL := parseNormalizedID(directoryID)
var opts rest.Opts
if drive != "" {
opts = rest.Opts{
@@ -1477,13 +1542,13 @@ func (o *Object) uploadSinglepart(in io.Reader, size int64, modTime time.Time) (
fs.Debugf(o, "Starting singlepart upload")
var resp *http.Response
var opts rest.Opts
_, directoryID, _ := o.fs.dirCache.FindPath(o.remote, false)
_, drive, rootURL := parseDirID(directoryID)
leaf, directoryID, _ := o.fs.dirCache.FindPath(o.remote, false)
trueDirID, drive, rootURL := parseNormalizedID(directoryID)
if drive != "" {
opts = rest.Opts{
Method: "PUT",
RootURL: rootURL,
Path: "/" + drive + "/root:/" + rest.URLPathEscape(o.srvPath()) + ":/content",
Path: "/" + drive + "/items/" + trueDirID + ":/" + rest.URLPathEscape(leaf) + ":/content",
ContentLength: &size,
Body: in,
}
@@ -1566,8 +1631,8 @@ func (o *Object) ID() string {
return o.id
}
func newOptsCall(id string, method string, route string) (opts rest.Opts) {
id, drive, rootURL := parseDirID(id)
func newOptsCall(normalizedID string, method string, route string) (opts rest.Opts) {
id, drive, rootURL := parseNormalizedID(normalizedID)
if drive != "" {
return rest.Opts{
@@ -1582,7 +1647,10 @@ func newOptsCall(id string, method string, route string) (opts rest.Opts) {
}
}
func parseDirID(ID string) (string, string, string) {
// parseNormalizedID parses a normalized ID (may be in the form `driveID#itemID` or just `itemID`)
// and returns itemID, driveID, rootURL.
// Such a normalized ID can come from (*Item).GetID()
func parseNormalizedID(ID string) (string, string, string) {
if strings.Index(ID, "#") >= 0 {
s := strings.Split(ID, "#")
return s[1], s[0], graphURL + "/drives"
@@ -1590,6 +1658,21 @@ func parseDirID(ID string) (string, string, string) {
return ID, "", ""
}
// getRelativePathInsideBase checks if `target` is inside `base`. If so, it
// returns a relative path for `target` based on `base` and a boolean `true`.
// Otherwise returns "", false.
func getRelativePathInsideBase(base, target string) (string, bool) {
if base == "" {
return target, true
}
baseSlash := base + "/"
if strings.HasPrefix(target+"/", baseSlash) {
return target[len(baseSlash):], true
}
return "", false
}
// Check the interfaces are satisfied
var (
_ fs.Fs = (*Fs)(nil)
+1 -1
View File
@@ -246,7 +246,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
root = parsePath(root)
oAuthClient, ts, err := oauthutil.NewClient(name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to configure Pcloud: %v", err)
return nil, errors.Wrap(err, "failed to configure Pcloud")
}
f := &Fs{
+1 -1
View File
@@ -69,7 +69,7 @@ func init() {
}},
}, {
Name: "connection_retries",
Help: "Number of connnection retries.",
Help: "Number of connection retries.",
Default: 3,
Advanced: true,
}},
+187 -59
View File
@@ -291,7 +291,11 @@ func init() {
Provider: "DigitalOcean",
}, {
Value: "s3.wasabisys.com",
Help: "Wasabi Object Storage",
Help: "Wasabi US East endpoint",
Provider: "Wasabi",
}, {
Value: "s3.us-west-1.wasabisys.com",
Help: "Wasabi US West endpoint",
Provider: "Wasabi",
}},
}, {
@@ -539,13 +543,24 @@ doesn't copy the ACL from the source but rather writes a fresh one.`,
}, {
Value: "ONEZONE_IA",
Help: "One Zone Infrequent Access storage class",
}, {
Value: "GLACIER",
Help: "Glacier storage class",
}},
}, {
Name: "upload_cutoff",
Help: `Cutoff for switching to chunked upload
Any files larger than this will be uploaded in chunks of chunk_size.
The minimum is 0 and the maximum is 5GB.`,
Default: defaultUploadCutoff,
Advanced: true,
}, {
Name: "chunk_size",
Help: `Chunk size to use for uploading.
Any files larger than this will be uploaded in chunks of this
size. The default is 5MB. The minimum is 5MB.
When uploading files larger than upload_cutoff they will be uploaded
as multipart uploads using this chunk size.
Note that "--s3-upload-concurrency" chunks of this size are buffered
in memory per transfer.
@@ -573,7 +588,7 @@ concurrently.
If you are uploading small numbers of large file over high speed link
and these uploads do not fully utilize your bandwidth, then increasing
this may help to speed up the transfers.`,
Default: 2,
Default: 4,
Advanced: true,
}, {
Name: "force_path_style",
@@ -603,14 +618,16 @@ Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.`,
// Constants
const (
metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime
metaMD5Hash = "Md5chksum" // the meta key to store md5hash in
listChunkSize = 1000 // number of items to read at once
maxRetries = 10 // number of retries to make of operations
maxSizeForCopy = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY
maxFileSize = 5 * 1024 * 1024 * 1024 * 1024 // largest possible upload file size
minChunkSize = fs.SizeSuffix(s3manager.MinUploadPartSize)
minSleep = 10 * time.Millisecond // In case of error, start at 10ms sleep.
metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime
metaMD5Hash = "Md5chksum" // the meta key to store md5hash in
listChunkSize = 1000 // number of items to read at once
maxRetries = 10 // number of retries to make of operations
maxSizeForCopy = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY
maxFileSize = 5 * 1024 * 1024 * 1024 * 1024 // largest possible upload file size
minChunkSize = fs.SizeSuffix(s3manager.MinUploadPartSize)
defaultUploadCutoff = fs.SizeSuffix(200 * 1024 * 1024)
maxUploadCutoff = fs.SizeSuffix(5 * 1024 * 1024 * 1024)
minSleep = 10 * time.Millisecond // In case of error, start at 10ms sleep.
)
// Options defines the configuration for this backend
@@ -626,6 +643,7 @@ type Options struct {
ServerSideEncryption string `config:"server_side_encryption"`
SSEKMSKeyID string `config:"sse_kms_key_id"`
StorageClass string `config:"storage_class"`
UploadCutoff fs.SizeSuffix `config:"upload_cutoff"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
DisableChecksum bool `config:"disable_checksum"`
SessionToken string `config:"session_token"`
@@ -647,6 +665,7 @@ type Fs struct {
bucketOK bool // true if we have created the bucket
bucketDeleted bool // true if we have deleted the bucket
pacer *pacer.Pacer // To pace the API calls
srv *http.Client // a plain http client
}
// Object describes a s3 object
@@ -804,8 +823,21 @@ func s3Connection(opt *Options) (*s3.S3, *session.Session, error) {
WithHTTPClient(fshttp.NewClient(fs.Config)).
WithS3ForcePathStyle(opt.ForcePathStyle)
// awsConfig.WithLogLevel(aws.LogDebugWithSigning)
ses := session.New()
c := s3.New(ses, awsConfig)
awsSessionOpts := session.Options{
Config: *awsConfig,
}
if opt.EnvAuth && opt.AccessKeyID == "" && opt.SecretAccessKey == "" {
// Enable loading config options from ~/.aws/config (selected by AWS_PROFILE env)
awsSessionOpts.SharedConfigState = session.SharedConfigEnable
// The session constructor (aws/session/mergeConfigSrcs) will only use the user's preferred credential source
// (from the shared config file) if the passed-in Options.Config.Credentials is nil.
awsSessionOpts.Config.Credentials = nil
}
ses, err := session.NewSessionWithOptions(awsSessionOpts)
if err != nil {
return nil, nil, err
}
c := s3.New(ses)
if opt.V2Auth || opt.Region == "other-v2-signature" {
fs.Debugf(nil, "Using v2 auth")
signer := func(req *request.Request) {
@@ -837,6 +869,21 @@ func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error)
return
}
func checkUploadCutoff(cs fs.SizeSuffix) error {
if cs > maxUploadCutoff {
return errors.Errorf("%s is greater than %s", cs, maxUploadCutoff)
}
return nil
}
func (f *Fs) setUploadCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
err = checkUploadCutoff(cs)
if err == nil {
old, f.opt.UploadCutoff = f.opt.UploadCutoff, cs
}
return
}
// NewFs constructs an Fs from the path, bucket:path
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// Parse config into Options struct
@@ -849,6 +896,10 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
if err != nil {
return nil, errors.Wrap(err, "s3: chunk size")
}
err = checkUploadCutoff(opt.UploadCutoff)
if err != nil {
return nil, errors.Wrap(err, "s3: upload cutoff")
}
bucket, directory, err := s3ParsePath(root)
if err != nil {
return nil, err
@@ -865,6 +916,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
bucket: bucket,
ses: ses,
pacer: pacer.New().SetMinSleep(minSleep).SetPacer(pacer.S3Pacer),
srv: fshttp.NewClient(fs.Config),
}
f.features = (&fs.Features{
ReadMimeType: true,
@@ -1539,38 +1591,46 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
modTime := src.ModTime()
size := src.Size()
uploader := s3manager.NewUploader(o.fs.ses, func(u *s3manager.Uploader) {
u.Concurrency = o.fs.opt.UploadConcurrency
u.LeavePartsOnError = false
u.S3 = o.fs.c
u.PartSize = int64(o.fs.opt.ChunkSize)
multipart := size < 0 || size >= int64(o.fs.opt.UploadCutoff)
var uploader *s3manager.Uploader
if multipart {
uploader = s3manager.NewUploader(o.fs.ses, func(u *s3manager.Uploader) {
u.Concurrency = o.fs.opt.UploadConcurrency
u.LeavePartsOnError = false
u.S3 = o.fs.c
u.PartSize = int64(o.fs.opt.ChunkSize)
if size == -1 {
// Make parts as small as possible while still being able to upload to the
// S3 file size limit. Rounded up to nearest MB.
u.PartSize = (((maxFileSize / s3manager.MaxUploadParts) >> 20) + 1) << 20
return
}
// Adjust PartSize until the number of parts is small enough.
if size/u.PartSize >= s3manager.MaxUploadParts {
// Calculate partition size rounded up to the nearest MB
u.PartSize = (((size / s3manager.MaxUploadParts) >> 20) + 1) << 20
}
})
if size == -1 {
// Make parts as small as possible while still being able to upload to the
// S3 file size limit. Rounded up to nearest MB.
u.PartSize = (((maxFileSize / s3manager.MaxUploadParts) >> 20) + 1) << 20
return
}
// Adjust PartSize until the number of parts is small enough.
if size/u.PartSize >= s3manager.MaxUploadParts {
// Calculate partition size rounded up to the nearest MB
u.PartSize = (((size / s3manager.MaxUploadParts) >> 20) + 1) << 20
}
})
}
// Set the mtime in the meta data
metadata := map[string]*string{
metaMtime: aws.String(swift.TimeToFloatString(modTime)),
}
if !o.fs.opt.DisableChecksum && size > uploader.PartSize {
// read the md5sum if available for non multpart and if
// disable checksum isn't present.
var md5sum string
if !multipart || !o.fs.opt.DisableChecksum {
hash, err := src.Hash(hash.MD5)
if err == nil && matchMd5.MatchString(hash) {
hashBytes, err := hex.DecodeString(hash)
if err == nil {
metadata[metaMD5Hash] = aws.String(base64.StdEncoding.EncodeToString(hashBytes))
md5sum = base64.StdEncoding.EncodeToString(hashBytes)
if multipart {
metadata[metaMD5Hash] = &md5sum
}
}
}
}
@@ -1579,30 +1639,98 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
mimeType := fs.MimeType(src)
key := o.fs.root + o.remote
req := s3manager.UploadInput{
Bucket: &o.fs.bucket,
ACL: &o.fs.opt.ACL,
Key: &key,
Body: in,
ContentType: &mimeType,
Metadata: metadata,
//ContentLength: &size,
}
if o.fs.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
}
if o.fs.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
}
if o.fs.opt.StorageClass != "" {
req.StorageClass = &o.fs.opt.StorageClass
}
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
_, err = uploader.Upload(&req)
return shouldRetry(err)
})
if err != nil {
return err
if multipart {
req := s3manager.UploadInput{
Bucket: &o.fs.bucket,
ACL: &o.fs.opt.ACL,
Key: &key,
Body: in,
ContentType: &mimeType,
Metadata: metadata,
//ContentLength: &size,
}
if o.fs.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
}
if o.fs.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
}
if o.fs.opt.StorageClass != "" {
req.StorageClass = &o.fs.opt.StorageClass
}
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
_, err = uploader.Upload(&req)
return shouldRetry(err)
})
if err != nil {
return err
}
} else {
req := s3.PutObjectInput{
Bucket: &o.fs.bucket,
ACL: &o.fs.opt.ACL,
Key: &key,
ContentType: &mimeType,
Metadata: metadata,
}
if md5sum != "" {
req.ContentMD5 = &md5sum
}
if o.fs.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
}
if o.fs.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
}
if o.fs.opt.StorageClass != "" {
req.StorageClass = &o.fs.opt.StorageClass
}
// Create the request
putObj, _ := o.fs.c.PutObjectRequest(&req)
// Sign it so we can upload using a presigned request.
//
// Note the SDK doesn't currently support streaming to
// PutObject so we'll use this work-around.
url, headers, err := putObj.PresignRequest(15 * time.Minute)
if err != nil {
return errors.Wrap(err, "s3 upload: sign request")
}
// Set request to nil if empty so as not to make chunked encoding
if size == 0 {
in = nil
}
// create the vanilla http request
httpReq, err := http.NewRequest("PUT", url, in)
if err != nil {
return errors.Wrap(err, "s3 upload: new request")
}
// set the headers we signed and the length
httpReq.Header = headers
httpReq.ContentLength = size
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err := o.fs.srv.Do(httpReq)
if err != nil {
return shouldRetry(err)
}
body, err := rest.ReadBody(resp)
if err != nil {
return shouldRetry(err)
}
if resp.StatusCode >= 200 && resp.StatusCode < 299 {
return false, nil
}
err = errors.Errorf("s3 upload: %s: %s", resp.Status, body)
return fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
})
if err != nil {
return err
}
}
// Read the metadata from the newly created object
+4
View File
@@ -23,4 +23,8 @@ func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
return f.setUploadChunkSize(cs)
}
func (f *Fs) SetUploadCutoff(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
return f.setUploadCutoff(cs)
}
var _ fstests.SetUploadChunkSizer = (*Fs)(nil)
+12 -2
View File
@@ -28,7 +28,7 @@ import (
"github.com/ncw/rclone/lib/readers"
"github.com/pkg/errors"
"github.com/pkg/sftp"
"github.com/xanzy/ssh-agent"
sshagent "github.com/xanzy/ssh-agent"
"golang.org/x/crypto/ssh"
"golang.org/x/time/rate"
)
@@ -594,12 +594,22 @@ func (f *Fs) Mkdir(dir string) error {
// Rmdir removes the root directory of the Fs object
func (f *Fs) Rmdir(dir string) error {
// Check to see if directory is empty as some servers will
// delete recursively with RemoveDirectory
entries, err := f.List(dir)
if err != nil {
return errors.Wrap(err, "Rmdir")
}
if len(entries) != 0 {
return fs.ErrorDirectoryNotEmpty
}
// Remove the directory
root := path.Join(f.root, dir)
c, err := f.getSftpConnection()
if err != nil {
return errors.Wrap(err, "Rmdir")
}
err = c.sftpClient.Remove(root)
err = c.sftpClient.RemoveDirectory(root)
f.putSftpConnection(&c, err)
return err
}
+182 -36
View File
@@ -21,6 +21,7 @@ import (
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/operations"
"github.com/ncw/rclone/fs/walk"
"github.com/ncw/rclone/lib/pacer"
"github.com/ncw/swift"
"github.com/pkg/errors"
)
@@ -30,6 +31,7 @@ const (
directoryMarkerContentType = "application/directory" // content type of directory marker objects
listChunks = 1000 // chunk size to read directory listings
defaultChunkSize = 5 * fs.GibiByte
minSleep = 10 * time.Millisecond // In case of error, start at 10ms sleep.
)
// SharedOptions are shared between swift and hubic
@@ -41,6 +43,20 @@ Above this size files will be chunked into a _segments container. The
default for this is 5GB which is its maximum value.`,
Default: defaultChunkSize,
Advanced: true,
}, {
Name: "no_chunk",
Help: `Don't chunk files during streaming upload.
When doing streaming uploads (eg using rcat or mount) setting this
flag will cause the swift backend to not upload chunked files.
This will limit the maximum upload size to 5GB. However non chunked
files are easier to deal with and have an MD5SUM.
Rclone will still chunk files bigger than chunk_size when doing normal
copy operations.`,
Default: false,
Advanced: true,
}}
// Register with Fs
@@ -173,6 +189,7 @@ type Options struct {
StoragePolicy string `config:"storage_policy"`
EndpointType string `config:"endpoint_type"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
NoChunk bool `config:"no_chunk"`
}
// Fs represents a remote swift server
@@ -187,16 +204,20 @@ type Fs struct {
containerOK bool // true if we have created the container
segmentsContainer string // container to store the segments (if any) in
noCheckContainer bool // don't check the container before creating it
pacer *pacer.Pacer // To pace the API calls
}
// Object describes a swift object
//
// Will definitely have info but maybe not meta
type Object struct {
fs *Fs // what this object is part of
remote string // The remote path
info swift.Object // Info from the swift object if known
headers swift.Headers // The object headers if known
fs *Fs // what this object is part of
remote string // The remote path
size int64
lastModified time.Time
contentType string
md5 string
headers swift.Headers // The object headers if known
}
// ------------------------------------------------------------
@@ -227,6 +248,32 @@ func (f *Fs) Features() *fs.Features {
return f.features
}
// retryErrorCodes is a slice of error codes that we will retry
var retryErrorCodes = []int{
401, // Unauthorized (eg "Token has expired")
408, // Request Timeout
409, // Conflict - various states that could be resolved on a retry
429, // Rate exceeded.
500, // Get occasional 500 Internal Server Error
503, // Service Unavailable/Slow Down - "Reduce your request rate"
504, // Gateway Time-out
}
// shouldRetry returns a boolean as to whether this err deserves to be
// retried. It returns the err as a convenience
func shouldRetry(err error) (bool, error) {
// If this is an swift.Error object extract the HTTP error code
if swiftError, ok := err.(*swift.Error); ok {
for _, e := range retryErrorCodes {
if swiftError.StatusCode == e {
return true, err
}
}
}
// Check for generic failure conditions
return fserrors.ShouldRetry(err), err
}
// Pattern to match a swift path
var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
@@ -337,6 +384,7 @@ func NewFsWithConnection(opt *Options, name, root string, c *swift.Connection, n
segmentsContainer: container + "_segments",
root: directory,
noCheckContainer: noCheckContainer,
pacer: pacer.New().SetMinSleep(minSleep).SetPacer(pacer.S3Pacer),
}
f.features = (&fs.Features{
ReadMimeType: true,
@@ -346,7 +394,11 @@ func NewFsWithConnection(opt *Options, name, root string, c *swift.Connection, n
if f.root != "" {
f.root += "/"
// Check to see if the object exists - ignoring directory markers
info, _, err := f.c.Object(container, directory)
var info swift.Object
err = f.pacer.Call(func() (bool, error) {
info, _, err = f.c.Object(container, directory)
return shouldRetry(err)
})
if err == nil && info.ContentType != directoryMarkerContentType {
f.root = path.Dir(directory)
if f.root == "." {
@@ -398,7 +450,10 @@ func (f *Fs) newObjectWithInfo(remote string, info *swift.Object) (fs.Object, er
}
if info != nil {
// Set info but not headers
o.info = *info
err := o.decodeMetaData(info)
if err != nil {
return nil, err
}
} else {
err := o.readMetaData() // reads info and headers, returning an error
if err != nil {
@@ -436,7 +491,12 @@ func (f *Fs) listContainerRoot(container, root string, dir string, recurse bool,
}
rootLength := len(root)
return f.c.ObjectsWalk(container, &opts, func(opts *swift.ObjectsOpts) (interface{}, error) {
objects, err := f.c.Objects(container, opts)
var objects []swift.Object
var err error
err = f.pacer.Call(func() (bool, error) {
objects, err = f.c.Objects(container, opts)
return shouldRetry(err)
})
if err == nil {
for i := range objects {
object := &objects[i]
@@ -525,7 +585,11 @@ func (f *Fs) listContainers(dir string) (entries fs.DirEntries, err error) {
if dir != "" {
return nil, fs.ErrorListBucketRequired
}
containers, err := f.c.ContainersAll(nil)
var containers []swift.Container
err = f.pacer.Call(func() (bool, error) {
containers, err = f.c.ContainersAll(nil)
return shouldRetry(err)
})
if err != nil {
return nil, errors.Wrap(err, "container listing failed")
}
@@ -586,7 +650,12 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
// About gets quota information
func (f *Fs) About() (*fs.Usage, error) {
containers, err := f.c.ContainersAll(nil)
var containers []swift.Container
var err error
err = f.pacer.Call(func() (bool, error) {
containers, err = f.c.ContainersAll(nil)
return shouldRetry(err)
})
if err != nil {
return nil, errors.Wrap(err, "container listing failed")
}
@@ -636,14 +705,20 @@ func (f *Fs) Mkdir(dir string) error {
// Check to see if container exists first
var err error = swift.ContainerNotFound
if !f.noCheckContainer {
_, _, err = f.c.Container(f.container)
err = f.pacer.Call(func() (bool, error) {
_, _, err = f.c.Container(f.container)
return shouldRetry(err)
})
}
if err == swift.ContainerNotFound {
headers := swift.Headers{}
if f.opt.StoragePolicy != "" {
headers["X-Storage-Policy"] = f.opt.StoragePolicy
}
err = f.c.ContainerCreate(f.container, headers)
err = f.pacer.Call(func() (bool, error) {
err = f.c.ContainerCreate(f.container, headers)
return shouldRetry(err)
})
}
if err == nil {
f.containerOK = true
@@ -660,7 +735,11 @@ func (f *Fs) Rmdir(dir string) error {
if f.root != "" || dir != "" {
return nil
}
err := f.c.ContainerDelete(f.container)
var err error
err = f.pacer.Call(func() (bool, error) {
err = f.c.ContainerDelete(f.container)
return shouldRetry(err)
})
if err == nil {
f.containerOK = false
}
@@ -719,7 +798,10 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
return nil, fs.ErrorCantCopy
}
srcFs := srcObj.fs
_, err = f.c.ObjectCopy(srcFs.container, srcFs.root+srcObj.remote, f.container, f.root+remote, nil)
err = f.pacer.Call(func() (bool, error) {
_, err = f.c.ObjectCopy(srcFs.container, srcFs.root+srcObj.remote, f.container, f.root+remote, nil)
return shouldRetry(err)
})
if err != nil {
return nil, err
}
@@ -768,7 +850,7 @@ func (o *Object) Hash(t hash.Type) (string, error) {
fs.Debugf(o, "Returning empty Md5sum for swift large object")
return "", nil
}
return strings.ToLower(o.info.Hash), nil
return strings.ToLower(o.md5), nil
}
// hasHeader checks for the header passed in returning false if the
@@ -797,7 +879,22 @@ func (o *Object) isStaticLargeObject() (bool, error) {
// Size returns the size of an object in bytes
func (o *Object) Size() int64 {
return o.info.Bytes
return o.size
}
// decodeMetaData sets the metadata in the object from a swift.Object
//
// Sets
// o.lastModified
// o.size
// o.md5
// o.contentType
func (o *Object) decodeMetaData(info *swift.Object) (err error) {
o.lastModified = info.LastModified
o.size = info.Bytes
o.md5 = info.Hash
o.contentType = info.ContentType
return nil
}
// readMetaData gets the metadata if it hasn't already been fetched
@@ -809,15 +906,23 @@ func (o *Object) readMetaData() (err error) {
if o.headers != nil {
return nil
}
info, h, err := o.fs.c.Object(o.fs.container, o.fs.root+o.remote)
var info swift.Object
var h swift.Headers
err = o.fs.pacer.Call(func() (bool, error) {
info, h, err = o.fs.c.Object(o.fs.container, o.fs.root+o.remote)
return shouldRetry(err)
})
if err != nil {
if err == swift.ObjectNotFound {
return fs.ErrorObjectNotFound
}
return err
}
o.info = info
o.headers = h
err = o.decodeMetaData(&info)
if err != nil {
return err
}
return nil
}
@@ -828,17 +933,17 @@ func (o *Object) readMetaData() (err error) {
// LastModified returned in the http headers
func (o *Object) ModTime() time.Time {
if fs.Config.UseServerModTime {
return o.info.LastModified
return o.lastModified
}
err := o.readMetaData()
if err != nil {
fs.Debugf(o, "Failed to read metadata: %s", err)
return o.info.LastModified
return o.lastModified
}
modTime, err := o.headers.ObjectMetadata().GetModTime()
if err != nil {
// fs.Logf(o, "Failed to read mtime from object: %v", err)
return o.info.LastModified
return o.lastModified
}
return modTime
}
@@ -861,7 +966,10 @@ func (o *Object) SetModTime(modTime time.Time) error {
newHeaders[k] = v
}
}
return o.fs.c.ObjectUpdate(o.fs.container, o.fs.root+o.remote, newHeaders)
return o.fs.pacer.Call(func() (bool, error) {
err = o.fs.c.ObjectUpdate(o.fs.container, o.fs.root+o.remote, newHeaders)
return shouldRetry(err)
})
}
// Storable returns if this object is storable
@@ -869,14 +977,17 @@ func (o *Object) SetModTime(modTime time.Time) error {
// It compares the Content-Type to directoryMarkerContentType - that
// makes it a directory marker which is not storable.
func (o *Object) Storable() bool {
return o.info.ContentType != directoryMarkerContentType
return o.contentType != directoryMarkerContentType
}
// Open an object for read
func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
headers := fs.OpenOptionHeaders(options)
_, isRanging := headers["Range"]
in, _, err = o.fs.c.ObjectOpen(o.fs.container, o.fs.root+o.remote, !isRanging, headers)
err = o.fs.pacer.Call(func() (bool, error) {
in, _, err = o.fs.c.ObjectOpen(o.fs.container, o.fs.root+o.remote, !isRanging, headers)
return shouldRetry(err)
})
return
}
@@ -903,13 +1014,20 @@ func (o *Object) removeSegments(except string) error {
}
segmentPath := segmentsRoot + remote
fs.Debugf(o, "Removing segment file %q in container %q", segmentPath, o.fs.segmentsContainer)
return o.fs.c.ObjectDelete(o.fs.segmentsContainer, segmentPath)
var err error
return o.fs.pacer.Call(func() (bool, error) {
err = o.fs.c.ObjectDelete(o.fs.segmentsContainer, segmentPath)
return shouldRetry(err)
})
})
if err != nil {
return err
}
// remove the segments container if empty, ignore errors
err = o.fs.c.ContainerDelete(o.fs.segmentsContainer)
err = o.fs.pacer.Call(func() (bool, error) {
err = o.fs.c.ContainerDelete(o.fs.segmentsContainer)
return shouldRetry(err)
})
if err == nil {
fs.Debugf(o, "Removed empty container %q", o.fs.segmentsContainer)
}
@@ -938,13 +1056,19 @@ func urlEncode(str string) string {
func (o *Object) updateChunks(in0 io.Reader, headers swift.Headers, size int64, contentType string) (string, error) {
// Create the segmentsContainer if it doesn't exist
var err error
_, _, err = o.fs.c.Container(o.fs.segmentsContainer)
err = o.fs.pacer.Call(func() (bool, error) {
_, _, err = o.fs.c.Container(o.fs.segmentsContainer)
return shouldRetry(err)
})
if err == swift.ContainerNotFound {
headers := swift.Headers{}
if o.fs.opt.StoragePolicy != "" {
headers["X-Storage-Policy"] = o.fs.opt.StoragePolicy
}
err = o.fs.c.ContainerCreate(o.fs.segmentsContainer, headers)
err = o.fs.pacer.Call(func() (bool, error) {
err = o.fs.c.ContainerCreate(o.fs.segmentsContainer, headers)
return shouldRetry(err)
})
}
if err != nil {
return "", err
@@ -973,7 +1097,10 @@ func (o *Object) updateChunks(in0 io.Reader, headers swift.Headers, size int64,
segmentReader := io.LimitReader(in, n)
segmentPath := fmt.Sprintf("%s/%08d", segmentsPath, i)
fs.Debugf(o, "Uploading segment file %q into %q", segmentPath, o.fs.segmentsContainer)
_, err := o.fs.c.ObjectPut(o.fs.segmentsContainer, segmentPath, segmentReader, true, "", "", headers)
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
_, err = o.fs.c.ObjectPut(o.fs.segmentsContainer, segmentPath, segmentReader, true, "", "", headers)
return shouldRetry(err)
})
if err != nil {
return "", err
}
@@ -984,7 +1111,10 @@ func (o *Object) updateChunks(in0 io.Reader, headers swift.Headers, size int64,
headers["Content-Length"] = "0" // set Content-Length as we know it
emptyReader := bytes.NewReader(nil)
manifestName := o.fs.root + o.remote
_, err = o.fs.c.ObjectPut(o.fs.container, manifestName, emptyReader, true, "", contentType, headers)
err = o.fs.pacer.Call(func() (bool, error) {
_, err = o.fs.c.ObjectPut(o.fs.container, manifestName, emptyReader, true, "", contentType, headers)
return shouldRetry(err)
})
return uniquePrefix + "/", err
}
@@ -1014,17 +1144,31 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
contentType := fs.MimeType(src)
headers := m.ObjectHeaders()
uniquePrefix := ""
if size > int64(o.fs.opt.ChunkSize) || size == -1 {
if size > int64(o.fs.opt.ChunkSize) || (size == -1 && !o.fs.opt.NoChunk) {
uniquePrefix, err = o.updateChunks(in, headers, size, contentType)
if err != nil {
return err
}
o.headers = nil // wipe old metadata
} else {
headers["Content-Length"] = strconv.FormatInt(size, 10) // set Content-Length as we know it
_, err := o.fs.c.ObjectPut(o.fs.container, o.fs.root+o.remote, in, true, "", contentType, headers)
if size >= 0 {
headers["Content-Length"] = strconv.FormatInt(size, 10) // set Content-Length if we know it
}
var rxHeaders swift.Headers
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
rxHeaders, err = o.fs.c.ObjectPut(o.fs.container, o.fs.root+o.remote, in, true, "", contentType, headers)
return shouldRetry(err)
})
if err != nil {
return err
}
// set Metadata since ObjectPut checked the hash and length so we know the
// object has been safely uploaded
o.lastModified = modTime
o.size = size
o.md5 = rxHeaders["ETag"]
o.contentType = contentType
o.headers = headers
}
// If file was a dynamic large object then remove old/all segments
@@ -1035,8 +1179,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
}
}
// Read the metadata from the newly created object
o.headers = nil // wipe old metadata
// Read the metadata from the newly created object if necessary
return o.readMetaData()
}
@@ -1047,7 +1190,10 @@ func (o *Object) Remove() error {
return err
}
// Remove file/manifest first
err = o.fs.c.ObjectDelete(o.fs.container, o.fs.root+o.remote)
err = o.fs.pacer.Call(func() (bool, error) {
err = o.fs.c.ObjectDelete(o.fs.container, o.fs.root+o.remote)
return shouldRetry(err)
})
if err != nil {
return err
}
@@ -1063,7 +1209,7 @@ func (o *Object) Remove() error {
// MimeType of an Object if known, "" otherwise
func (o *Object) MimeType() string {
return o.info.ContentType
return o.contentType
}
// Check the interfaces are satisfied
+14
View File
@@ -6,7 +6,10 @@ import (
"regexp"
"strconv"
"strings"
"sync"
"time"
"github.com/ncw/rclone/fs"
)
const (
@@ -148,6 +151,8 @@ var timeFormats = []string{
time.RFC3339, // Wed, 31 Oct 2018 13:57:11 CET (as used by komfortcloud.de)
}
var oneTimeError sync.Once
// UnmarshalXML turns XML into a Time
func (t *Time) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
var v string
@@ -171,5 +176,14 @@ func (t *Time) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
break
}
}
if err != nil {
oneTimeError.Do(func() {
fs.Errorf(nil, "Failed to parse time %q - using the epoch", v)
})
// Return the epoch instead
*t = Time(time.Unix(0, 0))
// ignore error
err = nil
}
return err
}
+20 -18
View File
@@ -31,7 +31,6 @@ import (
"github.com/ncw/rclone/backend/webdav/api"
"github.com/ncw/rclone/backend/webdav/odrvcookie"
"github.com/ncw/rclone/fs"
"github.com/ncw/rclone/fs/config"
"github.com/ncw/rclone/fs/config/configmap"
"github.com/ncw/rclone/fs/config/configstruct"
"github.com/ncw/rclone/fs/config/obscure"
@@ -96,10 +95,11 @@ func init() {
// Options defines the configuration for this backend
type Options struct {
URL string `config:"url"`
Vendor string `config:"vendor"`
User string `config:"user"`
Pass string `config:"pass"`
URL string `config:"url"`
Vendor string `config:"vendor"`
User string `config:"user"`
Pass string `config:"pass"`
BearerToken string `config:"bearer_token"`
}
// Fs represents a remote webdav
@@ -283,9 +283,6 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
rootIsDir := strings.HasSuffix(root, "/")
root = strings.Trim(root, "/")
user := config.FileGet(name, "user")
pass := config.FileGet(name, "pass")
bearerToken := config.FileGet(name, "bearer_token")
if !strings.HasSuffix(opt.URL, "/") {
opt.URL += "/"
}
@@ -320,10 +317,10 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
f.features = (&fs.Features{
CanHaveEmptyDirectories: true,
}).Fill(f)
if user != "" || pass != "" {
if opt.User != "" || opt.Pass != "" {
f.srv.SetUserPass(opt.User, opt.Pass)
} else if bearerToken != "" {
f.srv.SetHeader("Authorization", "BEARER "+bearerToken)
} else if opt.BearerToken != "" {
f.srv.SetHeader("Authorization", "BEARER "+opt.BearerToken)
}
f.srv.SetErrorHandler(errorHandler)
err = f.setQuirks(opt.Vendor)
@@ -604,10 +601,9 @@ func (f *Fs) mkParentDir(dirPath string) error {
return f.mkdir(parent)
}
// mkdir makes the directory and parents using native paths
func (f *Fs) mkdir(dirPath string) error {
// defer log.Trace(dirPath, "")("")
// We assume the root is already ceated
// low level mkdir, only makes the directory, doesn't attempt to create parents
func (f *Fs) _mkdir(dirPath string) error {
// We assume the root is already created
if dirPath == "" {
return nil
}
@@ -620,20 +616,26 @@ func (f *Fs) mkdir(dirPath string) error {
Path: dirPath,
NoResponse: true,
}
err := f.pacer.Call(func() (bool, error) {
return f.pacer.Call(func() (bool, error) {
resp, err := f.srv.Call(&opts)
return shouldRetry(resp, err)
})
}
// mkdir makes the directory and parents using native paths
func (f *Fs) mkdir(dirPath string) error {
// defer log.Trace(dirPath, "")("")
err := f._mkdir(dirPath)
if apiErr, ok := err.(*api.Error); ok {
// already exists
if apiErr.StatusCode == http.StatusMethodNotAllowed || apiErr.StatusCode == http.StatusNotAcceptable {
return nil
}
// parent does not exists
// parent does not exist
if apiErr.StatusCode == http.StatusConflict {
err = f.mkParentDir(dirPath)
if err == nil {
err = f.mkdir(dirPath)
err = f._mkdir(dirPath)
}
}
}
-34
View File
@@ -1,34 +0,0 @@
package src
//from yadisk
import (
"io"
"net/http"
)
//RootAddr is the base URL for Yandex Disk API.
const RootAddr = "https://cloud-api.yandex.com" //also https://cloud-api.yandex.net and https://cloud-api.yandex.ru
func (c *Client) setRequestScope(req *http.Request) {
req.Header.Add("Accept", "application/json")
req.Header.Add("Content-Type", "application/json")
req.Header.Add("Authorization", "OAuth "+c.token)
}
func (c *Client) scopedRequest(method, urlPath string, body io.Reader) (*http.Request, error) {
fullURL := RootAddr
if urlPath[:1] != "/" {
fullURL += "/" + urlPath
} else {
fullURL += urlPath
}
req, err := http.NewRequest(method, fullURL, body)
if err != nil {
return req, err
}
c.setRequestScope(req)
return req, nil
}
-133
View File
@@ -1,133 +0,0 @@
package src
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"net/http"
"net/url"
"strings"
"github.com/pkg/errors"
)
//Client struct
type Client struct {
token string
basePath string
HTTPClient *http.Client
}
//NewClient creates new client
func NewClient(token string, client ...*http.Client) *Client {
return newClientInternal(
token,
"https://cloud-api.yandex.com/v1/disk", //also "https://cloud-api.yandex.net/v1/disk" "https://cloud-api.yandex.ru/v1/disk"
client...)
}
func newClientInternal(token string, basePath string, client ...*http.Client) *Client {
c := &Client{
token: token,
basePath: basePath,
}
if len(client) != 0 {
c.HTTPClient = client[0]
} else {
c.HTTPClient = http.DefaultClient
}
return c
}
//ErrorHandler type
type ErrorHandler func(*http.Response) error
var defaultErrorHandler ErrorHandler = func(resp *http.Response) error {
if resp.StatusCode/100 == 5 {
return errors.New("server error")
}
if resp.StatusCode/100 == 4 {
var response DiskClientError
contents, _ := ioutil.ReadAll(resp.Body)
err := json.Unmarshal(contents, &response)
if err != nil {
return err
}
return response
}
if resp.StatusCode/100 == 3 {
return errors.New("redirect error")
}
return nil
}
func (HTTPRequest *HTTPRequest) run(client *Client) ([]byte, error) {
var err error
values := make(url.Values)
for k, v := range HTTPRequest.Parameters {
values.Set(k, fmt.Sprintf("%v", v))
}
var req *http.Request
if HTTPRequest.Method == "POST" {
// TODO json serialize
req, err = http.NewRequest(
"POST",
client.basePath+HTTPRequest.Path,
strings.NewReader(values.Encode()))
if err != nil {
return nil, err
}
// TODO
// req.Header.Set("Content-Type", "application/json")
} else {
req, err = http.NewRequest(
HTTPRequest.Method,
client.basePath+HTTPRequest.Path+"?"+values.Encode(),
nil)
if err != nil {
return nil, err
}
}
for headerName := range HTTPRequest.Headers {
var headerValues = HTTPRequest.Headers[headerName]
for _, headerValue := range headerValues {
req.Header.Set(headerName, headerValue)
}
}
return runRequest(client, req)
}
func runRequest(client *Client, req *http.Request) ([]byte, error) {
return runRequestWithErrorHandler(client, req, defaultErrorHandler)
}
func runRequestWithErrorHandler(client *Client, req *http.Request, errorHandler ErrorHandler) (out []byte, err error) {
resp, err := client.HTTPClient.Do(req)
if err != nil {
return nil, err
}
defer CheckClose(resp.Body, &err)
return checkResponseForErrorsWithErrorHandler(resp, errorHandler)
}
func checkResponseForErrorsWithErrorHandler(resp *http.Response, errorHandler ErrorHandler) ([]byte, error) {
if resp.StatusCode/100 > 2 {
return nil, errorHandler(resp)
}
return ioutil.ReadAll(resp.Body)
}
// CheckClose is a utility function used to check the return from
// Close in a defer statement.
func CheckClose(c io.Closer, err *error) {
cerr := c.Close()
if *err == nil {
*err = cerr
}
}
-51
View File
@@ -1,51 +0,0 @@
package src
import (
"bytes"
"encoding/json"
"io"
"net/url"
)
//CustomPropertyResponse struct we send and is returned by the API for CustomProperty request.
type CustomPropertyResponse struct {
CustomProperties map[string]interface{} `json:"custom_properties"`
}
//SetCustomProperty will set specified data from Yandex Disk
func (c *Client) SetCustomProperty(remotePath string, property string, value string) error {
rcm := map[string]interface{}{
property: value,
}
cpr := CustomPropertyResponse{rcm}
data, _ := json.Marshal(cpr)
body := bytes.NewReader(data)
err := c.SetCustomPropertyRequest(remotePath, body)
if err != nil {
return err
}
return err
}
//SetCustomPropertyRequest will make an CustomProperty request and return a URL to CustomProperty data to.
func (c *Client) SetCustomPropertyRequest(remotePath string, body io.Reader) (err error) {
values := url.Values{}
values.Add("path", remotePath)
req, err := c.scopedRequest("PATCH", "/v1/disk/resources?"+values.Encode(), body)
if err != nil {
return err
}
resp, err := c.HTTPClient.Do(req)
if err != nil {
return err
}
if err := CheckAPIError(resp); err != nil {
return err
}
defer CheckClose(resp.Body, &err)
//If needed we can read response and check if custom_property is set.
return nil
}
-23
View File
@@ -1,23 +0,0 @@
package src
import (
"net/url"
"strconv"
)
// Delete will remove specified file/folder from Yandex Disk
func (c *Client) Delete(remotePath string, permanently bool) error {
values := url.Values{}
values.Add("permanently", strconv.FormatBool(permanently))
values.Add("path", remotePath)
urlPath := "/v1/disk/resources?" + values.Encode()
fullURL := RootAddr
if urlPath[:1] != "/" {
fullURL += "/" + urlPath
} else {
fullURL += urlPath
}
return c.PerformDelete(fullURL)
}
-48
View File
@@ -1,48 +0,0 @@
package src
import "encoding/json"
//DiskInfoRequest type
type DiskInfoRequest struct {
client *Client
HTTPRequest *HTTPRequest
}
func (req *DiskInfoRequest) request() *HTTPRequest {
return req.HTTPRequest
}
//DiskInfoResponse struct is returned by the API for DiskInfo request.
type DiskInfoResponse struct {
TrashSize uint64 `json:"TrashSize"`
TotalSpace uint64 `json:"TotalSpace"`
UsedSpace uint64 `json:"UsedSpace"`
SystemFolders map[string]string `json:"SystemFolders"`
}
//NewDiskInfoRequest create new DiskInfo Request
func (c *Client) NewDiskInfoRequest() *DiskInfoRequest {
return &DiskInfoRequest{
client: c,
HTTPRequest: createGetRequest(c, "/", nil),
}
}
//Exec run DiskInfo Request
func (req *DiskInfoRequest) Exec() (*DiskInfoResponse, error) {
data, err := req.request().run(req.client)
if err != nil {
return nil, err
}
var info DiskInfoResponse
err = json.Unmarshal(data, &info)
if err != nil {
return nil, err
}
if info.SystemFolders == nil {
info.SystemFolders = make(map[string]string)
}
return &info, nil
}
-66
View File
@@ -1,66 +0,0 @@
package src
import (
"encoding/json"
"io"
"net/url"
)
// DownloadResponse struct is returned by the API for Download request.
type DownloadResponse struct {
HRef string `json:"href"`
Method string `json:"method"`
Templated bool `json:"templated"`
}
// Download will get specified data from Yandex.Disk supplying the extra headers
func (c *Client) Download(remotePath string, headers map[string]string) (io.ReadCloser, error) { //io.Writer
ur, err := c.DownloadRequest(remotePath)
if err != nil {
return nil, err
}
return c.PerformDownload(ur.HRef, headers)
}
// DownloadRequest will make an download request and return a URL to download data to.
func (c *Client) DownloadRequest(remotePath string) (ur *DownloadResponse, err error) {
values := url.Values{}
values.Add("path", remotePath)
req, err := c.scopedRequest("GET", "/v1/disk/resources/download?"+values.Encode(), nil)
if err != nil {
return nil, err
}
resp, err := c.HTTPClient.Do(req)
if err != nil {
return nil, err
}
if err := CheckAPIError(resp); err != nil {
return nil, err
}
defer CheckClose(resp.Body, &err)
ur, err = ParseDownloadResponse(resp.Body)
if err != nil {
return nil, err
}
return ur, nil
}
// ParseDownloadResponse tries to read and parse DownloadResponse struct.
func ParseDownloadResponse(data io.Reader) (*DownloadResponse, error) {
dec := json.NewDecoder(data)
var ur DownloadResponse
if err := dec.Decode(&ur); err == io.EOF {
// ok
} else if err != nil {
return nil, err
}
// TODO: check if there is any trash data after JSON and crash if there is.
return &ur, nil
}
-9
View File
@@ -1,9 +0,0 @@
package src
// EmptyTrash will permanently delete all trashed files/folders from Yandex Disk
func (c *Client) EmptyTrash() error {
fullURL := RootAddr
fullURL += "/v1/disk/trash/resources"
return c.PerformDelete(fullURL)
}
-84
View File
@@ -1,84 +0,0 @@
package src
//from yadisk
import (
"encoding/json"
"fmt"
"io"
"net/http"
)
// ErrorResponse represents erroneous API response.
// Implements go's built in `error`.
type ErrorResponse struct {
ErrorName string `json:"error"`
Description string `json:"description"`
Message string `json:"message"`
StatusCode int `json:""`
}
func (e *ErrorResponse) Error() string {
return fmt.Sprintf("[%d - %s] %s (%s)", e.StatusCode, e.ErrorName, e.Description, e.Message)
}
// ProccessErrorResponse tries to represent data passed as
// an ErrorResponse object.
func ProccessErrorResponse(data io.Reader) (*ErrorResponse, error) {
dec := json.NewDecoder(data)
var errorResponse ErrorResponse
if err := dec.Decode(&errorResponse); err == io.EOF {
// ok
} else if err != nil {
return nil, err
}
// TODO: check if there is any trash data after JSON and crash if there is.
return &errorResponse, nil
}
// CheckAPIError is a convenient function to turn erroneous
// API response into go error. It closes the Body on error.
func CheckAPIError(resp *http.Response) (err error) {
if resp.StatusCode >= 200 && resp.StatusCode < 400 {
return nil
}
defer CheckClose(resp.Body, &err)
errorResponse, err := ProccessErrorResponse(resp.Body)
if err != nil {
return err
}
errorResponse.StatusCode = resp.StatusCode
return errorResponse
}
// ProccessErrorString tries to represent data passed as
// an ErrorResponse object.
func ProccessErrorString(data string) (*ErrorResponse, error) {
var errorResponse ErrorResponse
if err := json.Unmarshal([]byte(data), &errorResponse); err == nil {
// ok
} else if err != nil {
return nil, err
}
// TODO: check if there is any trash data after JSON and crash if there is.
return &errorResponse, nil
}
// ParseAPIError Parse json error response from API
func (c *Client) ParseAPIError(jsonErr string) (string, error) { //ErrorName
errorResponse, err := ProccessErrorString(jsonErr)
if err != nil {
return err.Error(), err
}
return errorResponse.ErrorName, nil
}
-14
View File
@@ -1,14 +0,0 @@
package src
import "encoding/json"
//DiskClientError struct
type DiskClientError struct {
Description string `json:"Description"`
Code string `json:"Error"`
}
func (e DiskClientError) Error() string {
b, _ := json.Marshal(e)
return string(b)
}
@@ -1,8 +0,0 @@
package src
// FilesResourceListResponse struct is returned by the API for requests.
type FilesResourceListResponse struct {
Items []ResourceInfoResponse `json:"items"`
Limit *uint64 `json:"limit"`
Offset *uint64 `json:"offset"`
}
@@ -1,78 +0,0 @@
package src
import (
"encoding/json"
"strings"
)
// FlatFileListRequest struct client for FlatFileList Request
type FlatFileListRequest struct {
client *Client
HTTPRequest *HTTPRequest
}
// FlatFileListRequestOptions struct - options for request
type FlatFileListRequestOptions struct {
MediaType []MediaType
Limit *uint32
Offset *uint32
Fields []string
PreviewSize *PreviewSize
PreviewCrop *bool
}
// Request get request
func (req *FlatFileListRequest) Request() *HTTPRequest {
return req.HTTPRequest
}
// NewFlatFileListRequest create new FlatFileList Request
func (c *Client) NewFlatFileListRequest(options ...FlatFileListRequestOptions) *FlatFileListRequest {
var parameters = make(map[string]interface{})
if len(options) > 0 {
opt := options[0]
if opt.Limit != nil {
parameters["limit"] = *opt.Limit
}
if opt.Offset != nil {
parameters["offset"] = *opt.Offset
}
if opt.Fields != nil {
parameters["fields"] = strings.Join(opt.Fields, ",")
}
if opt.PreviewSize != nil {
parameters["preview_size"] = opt.PreviewSize.String()
}
if opt.PreviewCrop != nil {
parameters["preview_crop"] = *opt.PreviewCrop
}
if opt.MediaType != nil {
var strMediaTypes = make([]string, len(opt.MediaType))
for i, t := range opt.MediaType {
strMediaTypes[i] = t.String()
}
parameters["media_type"] = strings.Join(strMediaTypes, ",")
}
}
return &FlatFileListRequest{
client: c,
HTTPRequest: createGetRequest(c, "/resources/files", parameters),
}
}
// Exec run FlatFileList Request
func (req *FlatFileListRequest) Exec() (*FilesResourceListResponse, error) {
data, err := req.Request().run(req.client)
if err != nil {
return nil, err
}
var info FilesResourceListResponse
err = json.Unmarshal(data, &info)
if err != nil {
return nil, err
}
if cap(info.Items) == 0 {
info.Items = []ResourceInfoResponse{}
}
return &info, nil
}
-24
View File
@@ -1,24 +0,0 @@
package src
// HTTPRequest struct
type HTTPRequest struct {
Method string
Path string
Parameters map[string]interface{}
Headers map[string][]string
}
func createGetRequest(client *Client, path string, params map[string]interface{}) *HTTPRequest {
return createRequest(client, "GET", path, params)
}
func createRequest(client *Client, method string, path string, parameters map[string]interface{}) *HTTPRequest {
var headers = make(map[string][]string)
headers["Authorization"] = []string{"OAuth " + client.token}
return &HTTPRequest{
Method: method,
Path: path,
Parameters: parameters,
Headers: headers,
}
}
@@ -1,7 +0,0 @@
package src
// LastUploadedResourceListResponse struct
type LastUploadedResourceListResponse struct {
Items []ResourceInfoResponse `json:"items"`
Limit *uint64 `json:"limit"`
}
@@ -1,74 +0,0 @@
package src
import (
"encoding/json"
"strings"
)
// LastUploadedResourceListRequest struct
type LastUploadedResourceListRequest struct {
client *Client
HTTPRequest *HTTPRequest
}
// LastUploadedResourceListRequestOptions struct
type LastUploadedResourceListRequestOptions struct {
MediaType []MediaType
Limit *uint32
Fields []string
PreviewSize *PreviewSize
PreviewCrop *bool
}
// Request return request
func (req *LastUploadedResourceListRequest) Request() *HTTPRequest {
return req.HTTPRequest
}
// NewLastUploadedResourceListRequest create new LastUploadedResourceList Request
func (c *Client) NewLastUploadedResourceListRequest(options ...LastUploadedResourceListRequestOptions) *LastUploadedResourceListRequest {
var parameters = make(map[string]interface{})
if len(options) > 0 {
opt := options[0]
if opt.Limit != nil {
parameters["limit"] = opt.Limit
}
if opt.Fields != nil {
parameters["fields"] = strings.Join(opt.Fields, ",")
}
if opt.PreviewSize != nil {
parameters["preview_size"] = opt.PreviewSize.String()
}
if opt.PreviewCrop != nil {
parameters["preview_crop"] = opt.PreviewCrop
}
if opt.MediaType != nil {
var strMediaTypes = make([]string, len(opt.MediaType))
for i, t := range opt.MediaType {
strMediaTypes[i] = t.String()
}
parameters["media_type"] = strings.Join(strMediaTypes, ",")
}
}
return &LastUploadedResourceListRequest{
client: c,
HTTPRequest: createGetRequest(c, "/resources/last-uploaded", parameters),
}
}
// Exec run LastUploadedResourceList Request
func (req *LastUploadedResourceListRequest) Exec() (*LastUploadedResourceListResponse, error) {
data, err := req.Request().run(req.client)
if err != nil {
return nil, err
}
var info LastUploadedResourceListResponse
err = json.Unmarshal(data, &info)
if err != nil {
return nil, err
}
if cap(info.Items) == 0 {
info.Items = []ResourceInfoResponse{}
}
return &info, nil
}
-144
View File
@@ -1,144 +0,0 @@
package src
// MediaType struct - media types
type MediaType struct {
mediaType string
}
// Audio - media type
func (m *MediaType) Audio() *MediaType {
return &MediaType{
mediaType: "audio",
}
}
// Backup - media type
func (m *MediaType) Backup() *MediaType {
return &MediaType{
mediaType: "backup",
}
}
// Book - media type
func (m *MediaType) Book() *MediaType {
return &MediaType{
mediaType: "book",
}
}
// Compressed - media type
func (m *MediaType) Compressed() *MediaType {
return &MediaType{
mediaType: "compressed",
}
}
// Data - media type
func (m *MediaType) Data() *MediaType {
return &MediaType{
mediaType: "data",
}
}
// Development - media type
func (m *MediaType) Development() *MediaType {
return &MediaType{
mediaType: "development",
}
}
// Diskimage - media type
func (m *MediaType) Diskimage() *MediaType {
return &MediaType{
mediaType: "diskimage",
}
}
// Document - media type
func (m *MediaType) Document() *MediaType {
return &MediaType{
mediaType: "document",
}
}
// Encoded - media type
func (m *MediaType) Encoded() *MediaType {
return &MediaType{
mediaType: "encoded",
}
}
// Executable - media type
func (m *MediaType) Executable() *MediaType {
return &MediaType{
mediaType: "executable",
}
}
// Flash - media type
func (m *MediaType) Flash() *MediaType {
return &MediaType{
mediaType: "flash",
}
}
// Font - media type
func (m *MediaType) Font() *MediaType {
return &MediaType{
mediaType: "font",
}
}
// Image - media type
func (m *MediaType) Image() *MediaType {
return &MediaType{
mediaType: "image",
}
}
// Settings - media type
func (m *MediaType) Settings() *MediaType {
return &MediaType{
mediaType: "settings",
}
}
// Spreadsheet - media type
func (m *MediaType) Spreadsheet() *MediaType {
return &MediaType{
mediaType: "spreadsheet",
}
}
// Text - media type
func (m *MediaType) Text() *MediaType {
return &MediaType{
mediaType: "text",
}
}
// Unknown - media type
func (m *MediaType) Unknown() *MediaType {
return &MediaType{
mediaType: "unknown",
}
}
// Video - media type
func (m *MediaType) Video() *MediaType {
return &MediaType{
mediaType: "video",
}
}
// Web - media type
func (m *MediaType) Web() *MediaType {
return &MediaType{
mediaType: "web",
}
}
// String - media type
func (m *MediaType) String() string {
return m.mediaType
}
-21
View File
@@ -1,21 +0,0 @@
package src
import (
"net/url"
)
// Mkdir will make specified folder on Yandex Disk
func (c *Client) Mkdir(remotePath string) (int, string, error) {
values := url.Values{}
values.Add("path", remotePath) // only one current folder will be created. Not all the folders in the path.
urlPath := "/v1/disk/resources?" + values.Encode()
fullURL := RootAddr
if urlPath[:1] != "/" {
fullURL += "/" + urlPath
} else {
fullURL += urlPath
}
return c.PerformMkdir(fullURL)
}
-35
View File
@@ -1,35 +0,0 @@
package src
import (
"io/ioutil"
"net/http"
"github.com/pkg/errors"
)
// PerformDelete does the actual delete via DELETE request.
func (c *Client) PerformDelete(url string) error {
req, err := http.NewRequest("DELETE", url, nil)
if err != nil {
return err
}
//set access token and headers
c.setRequestScope(req)
resp, err := c.HTTPClient.Do(req)
if err != nil {
return err
}
//204 - resource deleted.
//202 - folder not empty, content will be deleted soon (async delete).
if resp.StatusCode != 204 && resp.StatusCode != 202 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return errors.Errorf("delete error [%d]: %s", resp.StatusCode, string(body))
}
return nil
}
-40
View File
@@ -1,40 +0,0 @@
package src
import (
"io"
"io/ioutil"
"net/http"
"github.com/pkg/errors"
)
// PerformDownload does the actual download via unscoped GET request.
func (c *Client) PerformDownload(url string, headers map[string]string) (out io.ReadCloser, err error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
// Set any extra headers
for k, v := range headers {
req.Header.Set(k, v)
}
//c.setRequestScope(req)
resp, err := c.HTTPClient.Do(req)
if err != nil {
return nil, err
}
_, isRanging := req.Header["Range"]
if !(resp.StatusCode == http.StatusOK || (isRanging && resp.StatusCode == http.StatusPartialContent)) {
defer CheckClose(resp.Body, &err)
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return nil, errors.Errorf("download error [%d]: %s", resp.StatusCode, string(body))
}
return resp.Body, err
}
-34
View File
@@ -1,34 +0,0 @@
package src
import (
"io/ioutil"
"net/http"
"github.com/pkg/errors"
)
// PerformMkdir does the actual mkdir via PUT request.
func (c *Client) PerformMkdir(url string) (int, string, error) {
req, err := http.NewRequest("PUT", url, nil)
if err != nil {
return 0, "", err
}
//set access token and headers
c.setRequestScope(req)
resp, err := c.HTTPClient.Do(req)
if err != nil {
return 0, "", err
}
if resp.StatusCode != 201 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return 0, "", err
}
//third parameter is the json error response body
return resp.StatusCode, string(body), errors.Errorf("create folder error [%d]: %s", resp.StatusCode, string(body))
}
return resp.StatusCode, "", nil
}
-38
View File
@@ -1,38 +0,0 @@
package src
//from yadisk
import (
"io"
"io/ioutil"
"net/http"
"github.com/pkg/errors"
)
// PerformUpload does the actual upload via unscoped PUT request.
func (c *Client) PerformUpload(url string, data io.Reader, contentType string) (err error) {
req, err := http.NewRequest("PUT", url, data)
if err != nil {
return err
}
req.Header.Set("Content-Type", contentType)
//c.setRequestScope(req)
resp, err := c.HTTPClient.Do(req)
if err != nil {
return err
}
defer CheckClose(resp.Body, &err)
if resp.StatusCode != 201 {
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
return errors.Errorf("upload error [%d]: %s", resp.StatusCode, string(body))
}
return nil
}
-75
View File
@@ -1,75 +0,0 @@
package src
import "fmt"
// PreviewSize struct
type PreviewSize struct {
size string
}
// PredefinedSizeS - set preview size
func (s *PreviewSize) PredefinedSizeS() *PreviewSize {
return &PreviewSize{
size: "S",
}
}
// PredefinedSizeM - set preview size
func (s *PreviewSize) PredefinedSizeM() *PreviewSize {
return &PreviewSize{
size: "M",
}
}
// PredefinedSizeL - set preview size
func (s *PreviewSize) PredefinedSizeL() *PreviewSize {
return &PreviewSize{
size: "L",
}
}
// PredefinedSizeXL - set preview size
func (s *PreviewSize) PredefinedSizeXL() *PreviewSize {
return &PreviewSize{
size: "XL",
}
}
// PredefinedSizeXXL - set preview size
func (s *PreviewSize) PredefinedSizeXXL() *PreviewSize {
return &PreviewSize{
size: "XXL",
}
}
// PredefinedSizeXXXL - set preview size
func (s *PreviewSize) PredefinedSizeXXXL() *PreviewSize {
return &PreviewSize{
size: "XXXL",
}
}
// ExactWidth - set preview size
func (s *PreviewSize) ExactWidth(width uint32) *PreviewSize {
return &PreviewSize{
size: fmt.Sprintf("%dx", width),
}
}
// ExactHeight - set preview size
func (s *PreviewSize) ExactHeight(height uint32) *PreviewSize {
return &PreviewSize{
size: fmt.Sprintf("x%d", height),
}
}
// ExactSize - set preview size
func (s *PreviewSize) ExactSize(width uint32, height uint32) *PreviewSize {
return &PreviewSize{
size: fmt.Sprintf("%dx%d", width, height),
}
}
func (s *PreviewSize) String() string {
return s.size
}
-19
View File
@@ -1,19 +0,0 @@
package src
//ResourceInfoResponse struct is returned by the API for metedata requests.
type ResourceInfoResponse struct {
PublicKey string `json:"public_key"`
Name string `json:"name"`
Created string `json:"created"`
CustomProperties map[string]interface{} `json:"custom_properties"`
Preview string `json:"preview"`
PublicURL string `json:"public_url"`
OriginPath string `json:"origin_path"`
Modified string `json:"modified"`
Path string `json:"path"`
Md5 string `json:"md5"`
ResourceType string `json:"type"`
MimeType string `json:"mime_type"`
Size uint64 `json:"size"`
Embedded *ResourceListResponse `json:"_embedded"`
}
@@ -1,45 +0,0 @@
package src
import "encoding/json"
// ResourceInfoRequest struct
type ResourceInfoRequest struct {
client *Client
HTTPRequest *HTTPRequest
}
// Request of ResourceInfoRequest
func (req *ResourceInfoRequest) Request() *HTTPRequest {
return req.HTTPRequest
}
// NewResourceInfoRequest create new ResourceInfo Request
func (c *Client) NewResourceInfoRequest(path string, options ...ResourceInfoRequestOptions) *ResourceInfoRequest {
return &ResourceInfoRequest{
client: c,
HTTPRequest: createResourceInfoRequest(c, "/resources", path, options...),
}
}
// Exec run ResourceInfo Request
func (req *ResourceInfoRequest) Exec() (*ResourceInfoResponse, error) {
data, err := req.Request().run(req.client)
if err != nil {
return nil, err
}
var info ResourceInfoResponse
err = json.Unmarshal(data, &info)
if err != nil {
return nil, err
}
if info.CustomProperties == nil {
info.CustomProperties = make(map[string]interface{})
}
if info.Embedded != nil {
if cap(info.Embedded.Items) == 0 {
info.Embedded.Items = []ResourceInfoResponse{}
}
}
return &info, nil
}
@@ -1,33 +0,0 @@
package src
import "strings"
func createResourceInfoRequest(c *Client,
apiPath string,
path string,
options ...ResourceInfoRequestOptions) *HTTPRequest {
var parameters = make(map[string]interface{})
parameters["path"] = path
if len(options) > 0 {
opt := options[0]
if opt.SortMode != nil {
parameters["sort"] = opt.SortMode.String()
}
if opt.Limit != nil {
parameters["limit"] = *opt.Limit
}
if opt.Offset != nil {
parameters["offset"] = *opt.Offset
}
if opt.Fields != nil {
parameters["fields"] = strings.Join(opt.Fields, ",")
}
if opt.PreviewSize != nil {
parameters["preview_size"] = opt.PreviewSize.String()
}
if opt.PreviewCrop != nil {
parameters["preview_crop"] = *opt.PreviewCrop
}
}
return createGetRequest(c, apiPath, parameters)
}
@@ -1,11 +0,0 @@
package src
// ResourceInfoRequestOptions struct
type ResourceInfoRequestOptions struct {
SortMode *SortMode
Limit *uint32
Offset *uint32
Fields []string
PreviewSize *PreviewSize
PreviewCrop *bool
}
-12
View File
@@ -1,12 +0,0 @@
package src
// ResourceListResponse struct
type ResourceListResponse struct {
Sort *SortMode `json:"sort"`
PublicKey string `json:"public_key"`
Items []ResourceInfoResponse `json:"items"`
Path string `json:"path"`
Limit *uint64 `json:"limit"`
Offset *uint64 `json:"offset"`
Total *uint64 `json:"total"`
}
-79
View File
@@ -1,79 +0,0 @@
package src
import "strings"
// SortMode struct - sort mode
type SortMode struct {
mode string
}
// Default - sort mode
func (m *SortMode) Default() *SortMode {
return &SortMode{
mode: "",
}
}
// ByName - sort mode
func (m *SortMode) ByName() *SortMode {
return &SortMode{
mode: "name",
}
}
// ByPath - sort mode
func (m *SortMode) ByPath() *SortMode {
return &SortMode{
mode: "path",
}
}
// ByCreated - sort mode
func (m *SortMode) ByCreated() *SortMode {
return &SortMode{
mode: "created",
}
}
// ByModified - sort mode
func (m *SortMode) ByModified() *SortMode {
return &SortMode{
mode: "modified",
}
}
// BySize - sort mode
func (m *SortMode) BySize() *SortMode {
return &SortMode{
mode: "size",
}
}
// Reverse - sort mode
func (m *SortMode) Reverse() *SortMode {
if strings.HasPrefix(m.mode, "-") {
return &SortMode{
mode: m.mode[1:],
}
}
return &SortMode{
mode: "-" + m.mode,
}
}
func (m *SortMode) String() string {
return m.mode
}
// UnmarshalJSON sort mode
func (m *SortMode) UnmarshalJSON(value []byte) error {
if value == nil || len(value) == 0 {
m.mode = ""
return nil
}
m.mode = string(value)
if strings.HasPrefix(m.mode, "\"") && strings.HasSuffix(m.mode, "\"") {
m.mode = m.mode[1 : len(m.mode)-1]
}
return nil
}
@@ -1,45 +0,0 @@
package src
import "encoding/json"
// TrashResourceInfoRequest struct
type TrashResourceInfoRequest struct {
client *Client
HTTPRequest *HTTPRequest
}
// Request of TrashResourceInfoRequest struct
func (req *TrashResourceInfoRequest) Request() *HTTPRequest {
return req.HTTPRequest
}
// NewTrashResourceInfoRequest create new TrashResourceInfo Request
func (c *Client) NewTrashResourceInfoRequest(path string, options ...ResourceInfoRequestOptions) *TrashResourceInfoRequest {
return &TrashResourceInfoRequest{
client: c,
HTTPRequest: createResourceInfoRequest(c, "/trash/resources", path, options...),
}
}
// Exec run TrashResourceInfo Request
func (req *TrashResourceInfoRequest) Exec() (*ResourceInfoResponse, error) {
data, err := req.Request().run(req.client)
if err != nil {
return nil, err
}
var info ResourceInfoResponse
err = json.Unmarshal(data, &info)
if err != nil {
return nil, err
}
if info.CustomProperties == nil {
info.CustomProperties = make(map[string]interface{})
}
if info.Embedded != nil {
if cap(info.Embedded.Items) == 0 {
info.Embedded.Items = []ResourceInfoResponse{}
}
}
return &info, nil
}
+157
View File
@@ -0,0 +1,157 @@
package api
import (
"fmt"
"strings"
)
// DiskInfo contains disk metadata
type DiskInfo struct {
TotalSpace int64 `json:"total_space"`
UsedSpace int64 `json:"used_space"`
TrashSize int64 `json:"trash_size"`
}
// ResourceInfoRequestOptions struct
type ResourceInfoRequestOptions struct {
SortMode *SortMode
Limit uint64
Offset uint64
Fields []string
}
//ResourceInfoResponse struct is returned by the API for metedata requests.
type ResourceInfoResponse struct {
PublicKey string `json:"public_key"`
Name string `json:"name"`
Created string `json:"created"`
CustomProperties map[string]interface{} `json:"custom_properties"`
Preview string `json:"preview"`
PublicURL string `json:"public_url"`
OriginPath string `json:"origin_path"`
Modified string `json:"modified"`
Path string `json:"path"`
Md5 string `json:"md5"`
ResourceType string `json:"type"`
MimeType string `json:"mime_type"`
Size int64 `json:"size"`
Embedded *ResourceListResponse `json:"_embedded"`
}
// ResourceListResponse struct
type ResourceListResponse struct {
Sort *SortMode `json:"sort"`
PublicKey string `json:"public_key"`
Items []ResourceInfoResponse `json:"items"`
Path string `json:"path"`
Limit *uint64 `json:"limit"`
Offset *uint64 `json:"offset"`
Total *uint64 `json:"total"`
}
// AsyncInfo struct is returned by the API for various async operations.
type AsyncInfo struct {
HRef string `json:"href"`
Method string `json:"method"`
Templated bool `json:"templated"`
}
// AsyncStatus is returned when requesting the status of an async operations. Possble values in-progress, success, failure
type AsyncStatus struct {
Status string `json:"status"`
}
//CustomPropertyResponse struct we send and is returned by the API for CustomProperty request.
type CustomPropertyResponse struct {
CustomProperties map[string]interface{} `json:"custom_properties"`
}
// SortMode struct - sort mode
type SortMode struct {
mode string
}
// Default - sort mode
func (m *SortMode) Default() *SortMode {
return &SortMode{
mode: "",
}
}
// ByName - sort mode
func (m *SortMode) ByName() *SortMode {
return &SortMode{
mode: "name",
}
}
// ByPath - sort mode
func (m *SortMode) ByPath() *SortMode {
return &SortMode{
mode: "path",
}
}
// ByCreated - sort mode
func (m *SortMode) ByCreated() *SortMode {
return &SortMode{
mode: "created",
}
}
// ByModified - sort mode
func (m *SortMode) ByModified() *SortMode {
return &SortMode{
mode: "modified",
}
}
// BySize - sort mode
func (m *SortMode) BySize() *SortMode {
return &SortMode{
mode: "size",
}
}
// Reverse - sort mode
func (m *SortMode) Reverse() *SortMode {
if strings.HasPrefix(m.mode, "-") {
return &SortMode{
mode: m.mode[1:],
}
}
return &SortMode{
mode: "-" + m.mode,
}
}
func (m *SortMode) String() string {
return m.mode
}
// UnmarshalJSON sort mode
func (m *SortMode) UnmarshalJSON(value []byte) error {
if value == nil || len(value) == 0 {
m.mode = ""
return nil
}
m.mode = string(value)
if strings.HasPrefix(m.mode, "\"") && strings.HasSuffix(m.mode, "\"") {
m.mode = m.mode[1 : len(m.mode)-1]
}
return nil
}
// ErrorResponse represents erroneous API response.
// Implements go's built in `error`.
type ErrorResponse struct {
ErrorName string `json:"error"`
Description string `json:"description"`
Message string `json:"message"`
StatusCode int `json:""`
}
func (e *ErrorResponse) Error() string {
return fmt.Sprintf("[%d - %s] %s (%s)", e.StatusCode, e.ErrorName, e.Description, e.Message)
}
-71
View File
@@ -1,71 +0,0 @@
package src
//from yadisk
import (
"encoding/json"
"io"
"net/url"
"strconv"
)
// UploadResponse struct is returned by the API for upload request.
type UploadResponse struct {
HRef string `json:"href"`
Method string `json:"method"`
Templated bool `json:"templated"`
}
// Upload will put specified data to Yandex.Disk.
func (c *Client) Upload(data io.Reader, remotePath string, overwrite bool, contentType string) error {
ur, err := c.UploadRequest(remotePath, overwrite)
if err != nil {
return err
}
return c.PerformUpload(ur.HRef, data, contentType)
}
// UploadRequest will make an upload request and return a URL to upload data to.
func (c *Client) UploadRequest(remotePath string, overwrite bool) (ur *UploadResponse, err error) {
values := url.Values{}
values.Add("path", remotePath)
values.Add("overwrite", strconv.FormatBool(overwrite))
req, err := c.scopedRequest("GET", "/v1/disk/resources/upload?"+values.Encode(), nil)
if err != nil {
return nil, err
}
resp, err := c.HTTPClient.Do(req)
if err != nil {
return nil, err
}
if err := CheckAPIError(resp); err != nil {
return nil, err
}
defer CheckClose(resp.Body, &err)
ur, err = ParseUploadResponse(resp.Body)
if err != nil {
return nil, err
}
return ur, nil
}
// ParseUploadResponse tries to read and parse UploadResponse struct.
func ParseUploadResponse(data io.Reader) (*UploadResponse, error) {
dec := json.NewDecoder(data)
var ur UploadResponse
if err := dec.Decode(&ur); err == io.EOF {
// ok
} else if err != nil {
return nil, err
}
// TODO: check if there is any trash data after JSON and crash if there is.
return &ur, nil
}
+824 -374
View File
File diff suppressed because it is too large. Load diff
+3 -1
View File
@@ -63,7 +63,9 @@ var osarches = []string{
// Special environment flags for a given arch
var archFlags = map[string][]string{
"386": {"GO386=387"},
"386": {"GO386=387"},
"mips": {"GOMIPS=softfloat"},
"mipsle": {"GOMIPS=softfloat"},
}
// runEnv - run a shell command with env
+59
View File
@@ -0,0 +1,59 @@
// +build ignore
// Test that the tests in the suite passed in are independent
package main
import (
"flag"
"log"
"os"
"os/exec"
"regexp"
)
var matchLine = regexp.MustCompile(`(?m)^=== RUN\s*(TestIntegration/\S*)\s*$`)
// run the test pass in and grep out the test names
func findTests(packageToTest string) (tests []string) {
cmd := exec.Command("go", "test", "-v", packageToTest)
out, err := cmd.CombinedOutput()
if err != nil {
_, _ = os.Stderr.Write(out)
log.Fatal(err)
}
results := matchLine.FindAllSubmatch(out, -1)
if results == nil {
log.Fatal("No tests found")
}
for _, line := range results {
tests = append(tests, string(line[1]))
}
return tests
}
// run the test passed in with the -run passed in
func runTest(packageToTest string, testName string) {
cmd := exec.Command("go", "test", "-v", packageToTest, "-run", "^"+testName+"$")
out, err := cmd.CombinedOutput()
if err != nil {
log.Printf("%s FAILED ------------------", testName)
_, _ = os.Stderr.Write(out)
log.Printf("%s FAILED ------------------", testName)
} else {
log.Printf("%s OK", testName)
}
}
func main() {
flag.Parse()
args := flag.Args()
if len(args) != 1 {
log.Fatalf("Syntax: %s <test_to_run>", os.Args[0])
}
packageToTest := args[0]
testNames := findTests(packageToTest)
// fmt.Printf("%s\n", testNames)
for _, testName := range testNames {
runTest(packageToTest, testName)
}
}
+5 -1
View File
@@ -53,7 +53,6 @@ func mountOptions(device string, mountpoint string) (options []string) {
// OSX options
if runtime.GOOS == "darwin" {
options = append(options, "-o", "volname="+mountlib.VolumeName)
if mountlib.NoAppleDouble {
options = append(options, "-o", "noappledouble")
}
@@ -70,6 +69,11 @@ func mountOptions(device string, mountpoint string) (options []string) {
options = append(options, "--FileSystemName=rclone")
}
if runtime.GOOS == "darwin" || runtime.GOOS == "windows" {
if mountlib.VolumeName != "" {
options = append(options, "-o", "volname="+mountlib.VolumeName)
}
}
if mountlib.AllowNonEmpty {
options = append(options, "-o", "nonempty")
}
+11
View File
@@ -51,6 +51,17 @@ written a trailing / - meaning "copy the contents of this directory".
This applies to all commands and whether you are talking about the
source or destination.
See the [--no-traverse](/docs/#no-traverse) option for controlling
whether rclone lists the destination directory or not. Supplying this
option when copying a small number of files into a large destination
can speed transfers up greatly.
For example, if you have many files in /path/to/src but only a few of
them change every day, you can to copy all the files which have
changed recently very efficiently like this:
rclone copy --max-age 24h --no-traverse /path/to/src remote:
**Note**: Use the ` + "`-P`" + `/` + "`--progress`" + ` flag to view real-time transfer statistics
`,
Run: func(command *cobra.Command, args []string) {
+1
View File
@@ -138,6 +138,7 @@ func (r *results) checkChar(c rune) {
escape := false
if err != nil {
fs.Infof(r.f, "Couldn't write file 0x%02X", c)
escape = true
} else {
fs.Infof(r.f, "OK writing file 0x%02X", c)
}
+1 -1
View File
@@ -7,7 +7,7 @@ package mountlib
import (
"log"
"github.com/sevlyar/go-daemon"
daemon "github.com/sevlyar/go-daemon"
)
func startBackgroundMode() bool {
+1 -1
View File
@@ -147,7 +147,7 @@ systems are a long way from 100% reliable. The rclone sync/copy
commands cope with this with lots of retries. However rclone ` + commandName + `
can't use retries in the same way without making local copies of the
uploads. Look at the [file caching](#file-caching)
for solutions to make ` + commandName + ` mount more reliable.
for solutions to make ` + commandName + ` more reliable.
### Attribute caching
+5
View File
@@ -37,6 +37,11 @@ into ` + "`dest:path`" + ` then delete the original (if no errors on copy) in
If you want to delete empty source directories after move, use the --delete-empty-src-dirs flag.
See the [--no-traverse](/docs/#no-traverse) option for controlling
whether rclone lists the destination directory or not. Supplying this
option when moving a small number of files into a large destination
can speed transfers up greatly.
**Important**: Since this can cause data loss, test first with the
--dry-run flag.
+2 -9
View File
@@ -424,20 +424,13 @@ func (u *UI) removeEntry(pos int) {
func (u *UI) delete() {
dirPos := u.sortPerm[u.dirPosMap[u.path].entry]
entry := u.entries[dirPos]
file := false
d, _ := u.d.GetDir(dirPos)
if d == nil {
file = true
}
u.boxMenu = []string{"cancel", "confirm"}
if file {
if obj, isFile := entry.(fs.Object); isFile {
u.boxMenuHandler = func(f fs.Fs, p string, o int) (string, error) {
if o != 1 {
return "Aborted!", nil
}
err := f.Rmdir(entry.String())
err := operations.DeleteFile(obj)
if err != nil {
return "", err
}
+6
View File
@@ -27,6 +27,11 @@ const (
//
// It returns a func which should be called to stop the stats.
func startProgress() func() {
err := initTerminal()
if err != nil {
fs.Errorf(nil, "Failed to start progress: %v", err)
return func() {}
}
stopStats := make(chan struct{})
oldLogPrint := fs.LogPrint
if !log.Redirected() {
@@ -51,6 +56,7 @@ func startProgress() func() {
printProgress("")
case <-stopStats:
ticker.Stop()
printProgress("")
fs.LogPrint = oldLogPrint
fmt.Println("")
return
+4
View File
@@ -4,6 +4,10 @@ package cmd
import "os"
func initTerminal() error {
return nil
}
func writeToTerminal(b []byte) {
_, _ = os.Stdout.Write(b)
}
+16 -7
View File
@@ -5,22 +5,31 @@ package cmd
import (
"fmt"
"os"
"sync"
"syscall"
ansiterm "github.com/Azure/go-ansiterm"
"github.com/Azure/go-ansiterm/winterm"
"github.com/pkg/errors"
)
var (
initAnsiParser sync.Once
ansiParser *ansiterm.AnsiParser
ansiParser *ansiterm.AnsiParser
)
func initTerminal() error {
winEventHandler := winterm.CreateWinEventHandler(os.Stdout.Fd(), os.Stdout)
if winEventHandler == nil {
err := syscall.GetLastError()
if err == nil {
err = errors.New("initialization failed")
}
return errors.Wrap(err, "windows terminal")
}
ansiParser = ansiterm.CreateParser("Ground", winEventHandler)
return nil
}
func writeToTerminal(b []byte) {
initAnsiParser.Do(func() {
winEventHandler := winterm.CreateWinEventHandler(os.Stdout.Fd(), os.Stdout)
ansiParser = ansiterm.CreateParser("Ground", winEventHandler)
})
// Remove all non-ASCII characters until this is fixed
// https://github.com/Azure/go-ansiterm/issues/26
r := []rune(string(b))
+24 -1
View File
@@ -4,11 +4,13 @@ package httplib
import (
"crypto/tls"
"crypto/x509"
"encoding/base64"
"fmt"
"io/ioutil"
"log"
"net"
"net/http"
"strings"
"time"
auth "github.com/abbot/go-http-auth"
@@ -143,7 +145,28 @@ func NewServer(handler http.Handler, opt *Options) *Server {
secretProvider = s.singleUserProvider
}
authenticator := auth.NewBasicAuthenticator(s.Opt.Realm, secretProvider)
handler = auth.JustCheck(authenticator, handler.ServeHTTP)
oldHandler := handler
handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if username := authenticator.CheckAuth(r); username == "" {
authHeader := r.Header.Get(authenticator.Headers.V().Authorization)
if authHeader != "" {
s := strings.SplitN(authHeader, " ", 2)
var userName = "UNKNOWN"
if len(s) == 2 && s[0] == "Basic" {
b, err := base64.StdEncoding.DecodeString(s[1])
if err == nil {
userName = strings.SplitN(string(b), ":", 2)[0]
}
}
fs.Infof(r.URL.Path, "%s: Unauthorized request from %s", r.RemoteAddr, userName)
} else {
fs.Infof(r.URL.Path, "%s: Basic auth challenge sent", r.RemoteAddr)
}
authenticator.RequireAuth(w, r)
} else {
oldHandler.ServeHTTP(w, r)
}
})
s.usingAuth = true
}
+3
View File
@@ -1,4 +1,7 @@
// Package restic serves a remote suitable for use with restic
// +build go1.9
package restic
import (
@@ -1,3 +1,5 @@
// +build go1.9
package restic
import (
+3
View File
@@ -1,5 +1,8 @@
// Serve restic tests set up a server and run the integration tests
// for restic against it.
// +build go1.9
package restic
import (
+11
View File
@@ -0,0 +1,11 @@
// Build for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build !go1.9
package restic
import "github.com/spf13/cobra"
// Command definition is nil to show not implemented
var Command *cobra.Command = nil
+2
View File
@@ -1,3 +1,5 @@
// +build go1.9
package restic
import (
+1 -1
View File
@@ -1,4 +1,4 @@
//+build !go1.10
//+build go1.9,!go1.10
// Fallback deadline setting for pre go1.10
+6 -2
View File
@@ -13,8 +13,12 @@ import (
func init() {
Command.AddCommand(http.Command)
Command.AddCommand(webdav.Command)
Command.AddCommand(restic.Command)
if webdav.Command != nil {
Command.AddCommand(webdav.Command)
}
if restic.Command != nil {
Command.AddCommand(restic.Command)
}
if ftp.Command != nil {
Command.AddCommand(ftp.Command)
}
+2
View File
@@ -1,3 +1,5 @@
//+build go1.9
package webdav
import (
+1 -1
View File
@@ -3,7 +3,7 @@
//
// We skip tests on platforms with troublesome character mappings
//+build !windows,!darwin
//+build !windows,!darwin,go1.9
package webdav
+11
View File
@@ -0,0 +1,11 @@
// Build for webdav for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build !go1.9
package webdav
import "github.com/spf13/cobra"
// Command definition is nil to show not implemented
var Command *cobra.Command = nil
+12 -1
View File
@@ -66,7 +66,7 @@ Contributors
* Marvin Watson <marvwatson@users.noreply.github.com>
* Danny Tsai <danny8376@gmail.com>
* Yoni Jah <yonjah+git@gmail.com> <yonjah+github@gmail.com>
* Stephen Harris <github@spuddy.org>
* Stephen Harris <github@spuddy.org> <sweharris@users.noreply.github.com>
* Ihor Dvoretskyi <ihor.dvoretskyi@gmail.com>
* Jon Craton <jncraton@gmail.com>
* Hraban Luyat <hraban@0brg.net>
@@ -210,3 +210,14 @@ Contributors
* buergi <patbuergi@gmx.de>
* Florian Gamboeck <mail@floga.de>
* Ralf Hemberger <10364191+rhemberger@users.noreply.github.com>
* Scott Edlund <sedlund@users.noreply.github.com>
* Erik Swanson <erik@retailnext.net>
* Jake Coggiano <jake@stripe.com>
* brused27 <brused27@noemailaddress>
* Peter Kaminski <kaminski@istori.com>
* Henry Ptasinski <henry@logout.com>
* Alexander <kharkovalexander@gmail.com>
* Garry McNulty <garrmcnu@gmail.com>
* Mathieu Carbou <mathieu.carbou@gmail.com>
* Mark Otway <mark@otway.com>
* William Cocker <37018962+WilliamCocker@users.noreply.github.com>
+2 -2
View File
@@ -181,8 +181,8 @@ versions of files, leaving the current ones intact. You can also
supply a path and only old versions under that path will be deleted,
eg `rclone cleanup remote:bucket/path/to/stuff`.
Note that `cleanup` does not remove partially uploaded files
from the bucket.
Note that `cleanup` will remove partially uploaded files from the bucket
if they are more than a day old.
When you `purge` a bucket, the current and the old versions will be
deleted then the bucket will be deleted.
+70 -1
View File
@@ -1,11 +1,80 @@
---
title: "Documentation"
description: "Rclone Changelog"
date: "2018-10-15"
date: "2018-11-24"
---
# Changelog
## v1.45 - 2018-11-24
* New backends
* The Yandex backend was re-written - see below for details (Sebastian Bünger)
* New commands
* rcd: New command just to serve the remote control API (Nick Craig-Wood)
* New Features
* The remote control API (rc) was greatly expanded to allow full control over rclone (Nick Craig-Wood)
* sensitive operations require authorization or the `--rc-no-auth` flag
* config/* operations to configure rclone
* options/* for reading/setting command line flags
* operations/* for all low level operations, eg copy file, list directory
* sync/* for sync, copy and move
* `--rc-files` flag to serve files on the rc http server
* this is for building web native GUIs for rclone
* Optionally serving objects on the rc http server
* Ensure rclone fails to start up if the `--rc` port is in use already
* See [the rc docs](https://rclone.org/rc/) for more info
* sync/copy/move
* Make `--files-from` only read the objects specified and don't scan directories (Nick Craig-Wood)
* This is a huge speed improvement for destinations with lots of files
* filter: Add `--ignore-case` flag (Nick Craig-Wood)
* ncdu: Add remove function ('d' key) (Henning Surmeier)
* rc command
* Add `--json` flag for structured JSON input (Nick Craig-Wood)
* Add `--user` and `--pass` flags and interpret `--rc-user`, `--rc-pass`, `--rc-addr` (Nick Craig-Wood)
* build
* Require go1.8 or later for compilation (Nick Craig-Wood)
* Enable softfloat on MIPS arch (Scott Edlund)
* Integration test framework revamped with a better report and better retries (Nick Craig-Wood)
* Bug Fixes
* cmd: Make --progress update the stats correctly at the end (Nick Craig-Wood)
* config: Create config directory on save if it is missing (Nick Craig-Wood)
* dedupe: Check for existing filename before renaming a dupe file (ssaqua)
* move: Don't create directories with --dry-run (Nick Craig-Wood)
* operations: Fix Purge and Rmdirs when dir is not the root (Nick Craig-Wood)
* serve http/webdav/restic: Ensure rclone exits if the port is in use (Nick Craig-Wood)
* Mount
* Make `--volname` work for Windows and macOS (Nick Craig-Wood)
* Azure Blob
* Avoid context deadline exceeded error by setting a large TryTimeout value (brused27)
* Fix erroneous Rmdir error "directory not empty" (Nick Craig-Wood)
* Wait for up to 60s to create a just deleted container (Nick Craig-Wood)
* Dropbox
* Add dropbox impersonate support (Jake Coggiano)
* Jottacloud
* Fix bug in `--fast-list` handing of empty folders (albertony)
* Opendrive
* Fix transfer of files with `+` and `&` in (Nick Craig-Wood)
* Fix retries of upload chunks (Nick Craig-Wood)
* S3
* Set ACL for server side copies to that provided by the user (Nick Craig-Wood)
* Fix role_arn, credential_source, ... (Erik Swanson)
* Add config info for Wasabi's US-West endpoint (Henry Ptasinski)
* SFTP
* Ensure file hash checking is really disabled (Jon Fautley)
* Swift
* Add pacer for retries to make swift more reliable (Nick Craig-Wood)
* WebDAV
* Add Content-Type to PUT requests (Nick Craig-Wood)
* Fix config parsing so `--webdav-user` and `--webdav-pass` flags work (Nick Craig-Wood)
* Add RFC3339 date format (Ralf Hemberger)
* Yandex
* The yandex backend was re-written (Sebastian Bünger)
* This implements low level retries (Sebastian Bünger)
* Copy, Move, DirMove, PublicLink and About optional interfaces (Sebastian Bünger)
* Improved general error handling (Sebastian Bünger)
* Removed ListR for now due to inconsistent behaviour (Sebastian Bünger)
## v1.44 - 2018-10-15
* New commands
+18 -11
View File
@@ -1,5 +1,5 @@
---
date: 2018-10-15T11:00:47+01:00
date: 2018-11-24T13:43:29Z
title: "rclone"
slug: rclone
url: /commands/rclone/
@@ -61,13 +61,13 @@ rclone [flags]
--bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s)
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
--cache-chunk-path string Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend")
--cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M)
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G)
--cache-db-path string Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend")
--cache-db-purge Clear all the cached data for this remote on start.
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
--cache-dir string Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")
--cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone")
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
--cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server
--cache-plex-password string The password of the Plex user
@@ -92,8 +92,8 @@ rclone [flags]
--crypt-password2 string Password or pass phrase for salt. Optional but recommended.
--crypt-remote string Remote to encrypt/decrypt.
--crypt-show-mapping For all files listed show how the names encrypt.
--delete-after When synchronizing, delete files on destination after transfering (default)
--delete-before When synchronizing, delete files on destination before transfering
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
@@ -125,6 +125,7 @@ rclone [flags]
--dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M)
--dropbox-client-id string Dropbox App Client Id
--dropbox-client-secret string Dropbox App Client Secret
--dropbox-impersonate string Impersonate this user when using a business account.
-n, --dry-run Do a trial run with no permanent changes
--dump string List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
@@ -139,7 +140,7 @@ rclone [flags]
--ftp-host string FTP host to connect to
--ftp-pass string FTP password
--ftp-port string FTP port, leave blank to use default (21)
--ftp-user string FTP username, leave blank for current username, ncw
--ftp-user string FTP username, leave blank for current username, $USER
--gcs-bucket-acl string Access Control List for new buckets.
--gcs-client-id string Google Application Client Id
--gcs-client-secret string Google Application Client Secret
@@ -153,6 +154,7 @@ rclone [flags]
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G)
--hubic-client-id string Hubic Client Id
--hubic-client-secret string Hubic Client Secret
--ignore-case Ignore case in filters (case insensitive)
--ignore-checksum Skip post copy check of checksums.
--ignore-errors delete even if there are I/O errors
--ignore-existing Skip all files that exist on destination
@@ -205,7 +207,7 @@ rclone [flags]
--pcloud-client-secret string Pcloud App Client Secret
-P, --progress Show progress during transfer.
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-connection-retries int Number of connnection retries. (default 3)
--qingstor-connection-retries int Number of connection retries. (default 3)
--qingstor-endpoint string Enter a endpoint URL to connection QingStor API.
--qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -215,18 +217,21 @@ rclone [flags]
--rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca string Client certificate authority to verify clients with
--rc-files string Path to local files to serve on the HTTP server.
--rc-htpasswd string htpasswd file - if not provided no authentication is done
--rc-key string SSL PEM Private key
--rc-max-header-bytes int Maximum size of request header (default 4096)
--rc-no-auth Don't require auth for certain methods.
--rc-pass string Password for authentication.
--rc-realm string realm for authentication (default "rclone")
--rc-serve Enable the serving of remote objects.
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
--rc-user string User name for authentication.
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)
--s3-access-key-id string AWS Access Key ID.
--s3-acl string Canned ACL used when creating buckets and/or storing objects in S3.
--s3-acl string Canned ACL used when creating buckets and storing or copying objects.
--s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-endpoint string Endpoint for S3 API.
@@ -287,7 +292,7 @@ rclone [flags]
--union-remotes string List of space separated remotes.
-u, --update Skip files that are newer on the destination.
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.44")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.45")
-v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number
--webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon)
@@ -297,6 +302,7 @@ rclone [flags]
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-client-id string Yandex Client Id
--yandex-client-secret string Yandex Client Secret
--yandex-unlink Remove existing public link to file/folder with link command rather than creating.
```
### SEE ALSO
@@ -337,6 +343,7 @@ rclone [flags]
* [rclone purge](/commands/rclone_purge/) - Remove the path and all of its contents.
* [rclone rc](/commands/rclone_rc/) - Run a command against a running rclone.
* [rclone rcat](/commands/rclone_rcat/) - Copies standard input to file on remote.
* [rclone rcd](/commands/rclone_rcd/) - Run rclone listening to remote control commands only.
* [rclone rmdir](/commands/rclone_rmdir/) - Remove the path if empty.
* [rclone rmdirs](/commands/rclone_rmdirs/) - Remove empty directories under the path.
* [rclone serve](/commands/rclone_serve/) - Serve a remote over a protocol.
@@ -348,4 +355,4 @@ rclone [flags]
* [rclone tree](/commands/rclone_tree/) - List the contents of the remote in a tree like fashion.
* [rclone version](/commands/rclone_version/) - Show the version number.
###### Auto generated by spf13/cobra on 15-Oct-2018
###### Auto generated by spf13/cobra on 24-Nov-2018
+17 -11
View File
@@ -1,5 +1,5 @@
---
date: 2018-10-15T11:00:47+01:00
date: 2018-11-24T13:43:29Z
title: "rclone about"
slug: rclone_about
url: /commands/rclone_about/
@@ -104,13 +104,13 @@ rclone about remote: [flags]
--bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s)
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
--cache-chunk-path string Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend")
--cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M)
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G)
--cache-db-path string Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend")
--cache-db-purge Clear all the cached data for this remote on start.
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
--cache-dir string Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")
--cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone")
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
--cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server
--cache-plex-password string The password of the Plex user
@@ -135,8 +135,8 @@ rclone about remote: [flags]
--crypt-password2 string Password or pass phrase for salt. Optional but recommended.
--crypt-remote string Remote to encrypt/decrypt.
--crypt-show-mapping For all files listed show how the names encrypt.
--delete-after When synchronizing, delete files on destination after transfering (default)
--delete-before When synchronizing, delete files on destination before transfering
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
@@ -168,6 +168,7 @@ rclone about remote: [flags]
--dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M)
--dropbox-client-id string Dropbox App Client Id
--dropbox-client-secret string Dropbox App Client Secret
--dropbox-impersonate string Impersonate this user when using a business account.
-n, --dry-run Do a trial run with no permanent changes
--dump string List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
@@ -182,7 +183,7 @@ rclone about remote: [flags]
--ftp-host string FTP host to connect to
--ftp-pass string FTP password
--ftp-port string FTP port, leave blank to use default (21)
--ftp-user string FTP username, leave blank for current username, ncw
--ftp-user string FTP username, leave blank for current username, $USER
--gcs-bucket-acl string Access Control List for new buckets.
--gcs-client-id string Google Application Client Id
--gcs-client-secret string Google Application Client Secret
@@ -195,6 +196,7 @@ rclone about remote: [flags]
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G)
--hubic-client-id string Hubic Client Id
--hubic-client-secret string Hubic Client Secret
--ignore-case Ignore case in filters (case insensitive)
--ignore-checksum Skip post copy check of checksums.
--ignore-errors delete even if there are I/O errors
--ignore-existing Skip all files that exist on destination
@@ -247,7 +249,7 @@ rclone about remote: [flags]
--pcloud-client-secret string Pcloud App Client Secret
-P, --progress Show progress during transfer.
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-connection-retries int Number of connnection retries. (default 3)
--qingstor-connection-retries int Number of connection retries. (default 3)
--qingstor-endpoint string Enter a endpoint URL to connection QingStor API.
--qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -257,18 +259,21 @@ rclone about remote: [flags]
--rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca string Client certificate authority to verify clients with
--rc-files string Path to local files to serve on the HTTP server.
--rc-htpasswd string htpasswd file - if not provided no authentication is done
--rc-key string SSL PEM Private key
--rc-max-header-bytes int Maximum size of request header (default 4096)
--rc-no-auth Don't require auth for certain methods.
--rc-pass string Password for authentication.
--rc-realm string realm for authentication (default "rclone")
--rc-serve Enable the serving of remote objects.
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
--rc-user string User name for authentication.
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)
--s3-access-key-id string AWS Access Key ID.
--s3-acl string Canned ACL used when creating buckets and/or storing objects in S3.
--s3-acl string Canned ACL used when creating buckets and storing or copying objects.
--s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-endpoint string Endpoint for S3 API.
@@ -329,7 +334,7 @@ rclone about remote: [flags]
--union-remotes string List of space separated remotes.
-u, --update Skip files that are newer on the destination.
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.44")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.45")
-v, --verbose count Print lots more stuff (repeat for more)
--webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon)
--webdav-pass string Password.
@@ -338,10 +343,11 @@ rclone about remote: [flags]
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-client-id string Yandex Client Id
--yandex-client-secret string Yandex Client Secret
--yandex-unlink Remove existing public link to file/folder with link command rather than creating.
```
### SEE ALSO
* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
###### Auto generated by spf13/cobra on 15-Oct-2018
###### Auto generated by spf13/cobra on 24-Nov-2018
+17 -11
View File
@@ -1,5 +1,5 @@
---
date: 2018-10-15T11:00:47+01:00
date: 2018-11-24T13:43:29Z
title: "rclone authorize"
slug: rclone_authorize
url: /commands/rclone_authorize/
@@ -63,13 +63,13 @@ rclone authorize [flags]
--bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s)
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
--cache-chunk-path string Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend")
--cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M)
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G)
--cache-db-path string Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend")
--cache-db-purge Clear all the cached data for this remote on start.
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
--cache-dir string Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")
--cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone")
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
--cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server
--cache-plex-password string The password of the Plex user
@@ -94,8 +94,8 @@ rclone authorize [flags]
--crypt-password2 string Password or pass phrase for salt. Optional but recommended.
--crypt-remote string Remote to encrypt/decrypt.
--crypt-show-mapping For all files listed show how the names encrypt.
--delete-after When synchronizing, delete files on destination after transfering (default)
--delete-before When synchronizing, delete files on destination before transfering
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
@@ -127,6 +127,7 @@ rclone authorize [flags]
--dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M)
--dropbox-client-id string Dropbox App Client Id
--dropbox-client-secret string Dropbox App Client Secret
--dropbox-impersonate string Impersonate this user when using a business account.
-n, --dry-run Do a trial run with no permanent changes
--dump string List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
@@ -141,7 +142,7 @@ rclone authorize [flags]
--ftp-host string FTP host to connect to
--ftp-pass string FTP password
--ftp-port string FTP port, leave blank to use default (21)
--ftp-user string FTP username, leave blank for current username, ncw
--ftp-user string FTP username, leave blank for current username, $USER
--gcs-bucket-acl string Access Control List for new buckets.
--gcs-client-id string Google Application Client Id
--gcs-client-secret string Google Application Client Secret
@@ -154,6 +155,7 @@ rclone authorize [flags]
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G)
--hubic-client-id string Hubic Client Id
--hubic-client-secret string Hubic Client Secret
--ignore-case Ignore case in filters (case insensitive)
--ignore-checksum Skip post copy check of checksums.
--ignore-errors delete even if there are I/O errors
--ignore-existing Skip all files that exist on destination
@@ -206,7 +208,7 @@ rclone authorize [flags]
--pcloud-client-secret string Pcloud App Client Secret
-P, --progress Show progress during transfer.
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-connection-retries int Number of connnection retries. (default 3)
--qingstor-connection-retries int Number of connection retries. (default 3)
--qingstor-endpoint string Enter a endpoint URL to connection QingStor API.
--qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -216,18 +218,21 @@ rclone authorize [flags]
--rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca string Client certificate authority to verify clients with
--rc-files string Path to local files to serve on the HTTP server.
--rc-htpasswd string htpasswd file - if not provided no authentication is done
--rc-key string SSL PEM Private key
--rc-max-header-bytes int Maximum size of request header (default 4096)
--rc-no-auth Don't require auth for certain methods.
--rc-pass string Password for authentication.
--rc-realm string realm for authentication (default "rclone")
--rc-serve Enable the serving of remote objects.
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
--rc-user string User name for authentication.
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)
--s3-access-key-id string AWS Access Key ID.
--s3-acl string Canned ACL used when creating buckets and/or storing objects in S3.
--s3-acl string Canned ACL used when creating buckets and storing or copying objects.
--s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-endpoint string Endpoint for S3 API.
@@ -288,7 +293,7 @@ rclone authorize [flags]
--union-remotes string List of space separated remotes.
-u, --update Skip files that are newer on the destination.
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.44")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.45")
-v, --verbose count Print lots more stuff (repeat for more)
--webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon)
--webdav-pass string Password.
@@ -297,10 +302,11 @@ rclone authorize [flags]
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-client-id string Yandex Client Id
--yandex-client-secret string Yandex Client Secret
--yandex-unlink Remove existing public link to file/folder with link command rather than creating.
```
### SEE ALSO
* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
###### Auto generated by spf13/cobra on 15-Oct-2018
###### Auto generated by spf13/cobra on 24-Nov-2018
+17 -11
View File
@@ -1,5 +1,5 @@
---
date: 2018-10-15T11:00:47+01:00
date: 2018-11-24T13:43:29Z
title: "rclone cachestats"
slug: rclone_cachestats
url: /commands/rclone_cachestats/
@@ -62,13 +62,13 @@ rclone cachestats source: [flags]
--bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s)
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
--cache-chunk-path string Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend")
--cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M)
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G)
--cache-db-path string Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend")
--cache-db-purge Clear all the cached data for this remote on start.
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
--cache-dir string Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")
--cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone")
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
--cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server
--cache-plex-password string The password of the Plex user
@@ -93,8 +93,8 @@ rclone cachestats source: [flags]
--crypt-password2 string Password or pass phrase for salt. Optional but recommended.
--crypt-remote string Remote to encrypt/decrypt.
--crypt-show-mapping For all files listed show how the names encrypt.
--delete-after When synchronizing, delete files on destination after transfering (default)
--delete-before When synchronizing, delete files on destination before transfering
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
@@ -126,6 +126,7 @@ rclone cachestats source: [flags]
--dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M)
--dropbox-client-id string Dropbox App Client Id
--dropbox-client-secret string Dropbox App Client Secret
--dropbox-impersonate string Impersonate this user when using a business account.
-n, --dry-run Do a trial run with no permanent changes
--dump string List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
@@ -140,7 +141,7 @@ rclone cachestats source: [flags]
--ftp-host string FTP host to connect to
--ftp-pass string FTP password
--ftp-port string FTP port, leave blank to use default (21)
--ftp-user string FTP username, leave blank for current username, ncw
--ftp-user string FTP username, leave blank for current username, $USER
--gcs-bucket-acl string Access Control List for new buckets.
--gcs-client-id string Google Application Client Id
--gcs-client-secret string Google Application Client Secret
@@ -153,6 +154,7 @@ rclone cachestats source: [flags]
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G)
--hubic-client-id string Hubic Client Id
--hubic-client-secret string Hubic Client Secret
--ignore-case Ignore case in filters (case insensitive)
--ignore-checksum Skip post copy check of checksums.
--ignore-errors delete even if there are I/O errors
--ignore-existing Skip all files that exist on destination
@@ -205,7 +207,7 @@ rclone cachestats source: [flags]
--pcloud-client-secret string Pcloud App Client Secret
-P, --progress Show progress during transfer.
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-connection-retries int Number of connnection retries. (default 3)
--qingstor-connection-retries int Number of connection retries. (default 3)
--qingstor-endpoint string Enter a endpoint URL to connection QingStor API.
--qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -215,18 +217,21 @@ rclone cachestats source: [flags]
--rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca string Client certificate authority to verify clients with
--rc-files string Path to local files to serve on the HTTP server.
--rc-htpasswd string htpasswd file - if not provided no authentication is done
--rc-key string SSL PEM Private key
--rc-max-header-bytes int Maximum size of request header (default 4096)
--rc-no-auth Don't require auth for certain methods.
--rc-pass string Password for authentication.
--rc-realm string realm for authentication (default "rclone")
--rc-serve Enable the serving of remote objects.
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
--rc-user string User name for authentication.
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)
--s3-access-key-id string AWS Access Key ID.
--s3-acl string Canned ACL used when creating buckets and/or storing objects in S3.
--s3-acl string Canned ACL used when creating buckets and storing or copying objects.
--s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-endpoint string Endpoint for S3 API.
@@ -287,7 +292,7 @@ rclone cachestats source: [flags]
--union-remotes string List of space separated remotes.
-u, --update Skip files that are newer on the destination.
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.44")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.45")
-v, --verbose count Print lots more stuff (repeat for more)
--webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon)
--webdav-pass string Password.
@@ -296,10 +301,11 @@ rclone cachestats source: [flags]
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-client-id string Yandex Client Id
--yandex-client-secret string Yandex Client Secret
--yandex-unlink Remove existing public link to file/folder with link command rather than creating.
```
### SEE ALSO
* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
###### Auto generated by spf13/cobra on 15-Oct-2018
###### Auto generated by spf13/cobra on 24-Nov-2018
+17 -11
View File
@@ -1,5 +1,5 @@
---
date: 2018-10-15T11:00:47+01:00
date: 2018-11-24T13:43:29Z
title: "rclone cat"
slug: rclone_cat
url: /commands/rclone_cat/
@@ -84,13 +84,13 @@ rclone cat remote:path [flags]
--bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s)
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
--cache-chunk-path string Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend")
--cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M)
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G)
--cache-db-path string Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend")
--cache-db-purge Clear all the cached data for this remote on start.
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
--cache-dir string Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")
--cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone")
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
--cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server
--cache-plex-password string The password of the Plex user
@@ -115,8 +115,8 @@ rclone cat remote:path [flags]
--crypt-password2 string Password or pass phrase for salt. Optional but recommended.
--crypt-remote string Remote to encrypt/decrypt.
--crypt-show-mapping For all files listed show how the names encrypt.
--delete-after When synchronizing, delete files on destination after transfering (default)
--delete-before When synchronizing, delete files on destination before transfering
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
@@ -148,6 +148,7 @@ rclone cat remote:path [flags]
--dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M)
--dropbox-client-id string Dropbox App Client Id
--dropbox-client-secret string Dropbox App Client Secret
--dropbox-impersonate string Impersonate this user when using a business account.
-n, --dry-run Do a trial run with no permanent changes
--dump string List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
@@ -162,7 +163,7 @@ rclone cat remote:path [flags]
--ftp-host string FTP host to connect to
--ftp-pass string FTP password
--ftp-port string FTP port, leave blank to use default (21)
--ftp-user string FTP username, leave blank for current username, ncw
--ftp-user string FTP username, leave blank for current username, $USER
--gcs-bucket-acl string Access Control List for new buckets.
--gcs-client-id string Google Application Client Id
--gcs-client-secret string Google Application Client Secret
@@ -175,6 +176,7 @@ rclone cat remote:path [flags]
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G)
--hubic-client-id string Hubic Client Id
--hubic-client-secret string Hubic Client Secret
--ignore-case Ignore case in filters (case insensitive)
--ignore-checksum Skip post copy check of checksums.
--ignore-errors delete even if there are I/O errors
--ignore-existing Skip all files that exist on destination
@@ -227,7 +229,7 @@ rclone cat remote:path [flags]
--pcloud-client-secret string Pcloud App Client Secret
-P, --progress Show progress during transfer.
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-connection-retries int Number of connnection retries. (default 3)
--qingstor-connection-retries int Number of connection retries. (default 3)
--qingstor-endpoint string Enter a endpoint URL to connection QingStor API.
--qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -237,18 +239,21 @@ rclone cat remote:path [flags]
--rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca string Client certificate authority to verify clients with
--rc-files string Path to local files to serve on the HTTP server.
--rc-htpasswd string htpasswd file - if not provided no authentication is done
--rc-key string SSL PEM Private key
--rc-max-header-bytes int Maximum size of request header (default 4096)
--rc-no-auth Don't require auth for certain methods.
--rc-pass string Password for authentication.
--rc-realm string realm for authentication (default "rclone")
--rc-serve Enable the serving of remote objects.
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
--rc-user string User name for authentication.
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)
--s3-access-key-id string AWS Access Key ID.
--s3-acl string Canned ACL used when creating buckets and/or storing objects in S3.
--s3-acl string Canned ACL used when creating buckets and storing or copying objects.
--s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-endpoint string Endpoint for S3 API.
@@ -309,7 +314,7 @@ rclone cat remote:path [flags]
--union-remotes string List of space separated remotes.
-u, --update Skip files that are newer on the destination.
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.44")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.45")
-v, --verbose count Print lots more stuff (repeat for more)
--webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon)
--webdav-pass string Password.
@@ -318,10 +323,11 @@ rclone cat remote:path [flags]
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-client-id string Yandex Client Id
--yandex-client-secret string Yandex Client Secret
--yandex-unlink Remove existing public link to file/folder with link command rather than creating.
```
### SEE ALSO
* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
###### Auto generated by spf13/cobra on 15-Oct-2018
###### Auto generated by spf13/cobra on 24-Nov-2018
+17 -11
View File
@@ -1,5 +1,5 @@
---
date: 2018-10-15T11:00:47+01:00
date: 2018-11-24T13:43:29Z
title: "rclone check"
slug: rclone_check
url: /commands/rclone_check/
@@ -78,13 +78,13 @@ rclone check source:path dest:path [flags]
--bwlimit BwTimetable Bandwidth limit in kBytes/s, or use suffix b|k|M|G or a full timetable.
--cache-chunk-clean-interval Duration How often should the cache perform cleanups of the chunk storage. (default 1m0s)
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
--cache-chunk-path string Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-chunk-path string Directory to cache chunk files. (default "$HOME/.cache/rclone/cache-backend")
--cache-chunk-size SizeSuffix The size of a chunk (partial file data). (default 5M)
--cache-chunk-total-size SizeSuffix The total size that the chunks can take up on the local disk. (default 10G)
--cache-db-path string Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
--cache-db-path string Directory to store file structure metadata DB. (default "$HOME/.cache/rclone/cache-backend")
--cache-db-purge Clear all the cached data for this remote on start.
--cache-db-wait-time Duration How long to wait for the DB to be available - 0 is unlimited (default 1s)
--cache-dir string Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")
--cache-dir string Directory rclone will use for caching. (default "$HOME/.cache/rclone")
--cache-info-age Duration How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
--cache-plex-insecure string Skip all certificate verifications when connecting to the Plex server
--cache-plex-password string The password of the Plex user
@@ -109,8 +109,8 @@ rclone check source:path dest:path [flags]
--crypt-password2 string Password or pass phrase for salt. Optional but recommended.
--crypt-remote string Remote to encrypt/decrypt.
--crypt-show-mapping For all files listed show how the names encrypt.
--delete-after When synchronizing, delete files on destination after transfering (default)
--delete-before When synchronizing, delete files on destination before transfering
--delete-after When synchronizing, delete files on destination after transferring (default)
--delete-before When synchronizing, delete files on destination before transferring
--delete-during When synchronizing, delete files during transfer
--delete-excluded Delete files on dest excluded from sync
--disable string Disable a comma separated list of features. Use help to see a list.
@@ -142,6 +142,7 @@ rclone check source:path dest:path [flags]
--dropbox-chunk-size SizeSuffix Upload chunk size. (< 150M). (default 48M)
--dropbox-client-id string Dropbox App Client Id
--dropbox-client-secret string Dropbox App Client Secret
--dropbox-impersonate string Impersonate this user when using a business account.
-n, --dry-run Do a trial run with no permanent changes
--dump string List of items to dump from: headers,bodies,requests,responses,auth,filters,goroutines,openfiles
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
@@ -156,7 +157,7 @@ rclone check source:path dest:path [flags]
--ftp-host string FTP host to connect to
--ftp-pass string FTP password
--ftp-port string FTP port, leave blank to use default (21)
--ftp-user string FTP username, leave blank for current username, ncw
--ftp-user string FTP username, leave blank for current username, $USER
--gcs-bucket-acl string Access Control List for new buckets.
--gcs-client-id string Google Application Client Id
--gcs-client-secret string Google Application Client Secret
@@ -169,6 +170,7 @@ rclone check source:path dest:path [flags]
--hubic-chunk-size SizeSuffix Above this size files will be chunked into a _segments container. (default 5G)
--hubic-client-id string Hubic Client Id
--hubic-client-secret string Hubic Client Secret
--ignore-case Ignore case in filters (case insensitive)
--ignore-checksum Skip post copy check of checksums.
--ignore-errors delete even if there are I/O errors
--ignore-existing Skip all files that exist on destination
@@ -221,7 +223,7 @@ rclone check source:path dest:path [flags]
--pcloud-client-secret string Pcloud App Client Secret
-P, --progress Show progress during transfer.
--qingstor-access-key-id string QingStor Access Key ID
--qingstor-connection-retries int Number of connnection retries. (default 3)
--qingstor-connection-retries int Number of connection retries. (default 3)
--qingstor-endpoint string Enter a endpoint URL to connection QingStor API.
--qingstor-env-auth Get QingStor credentials from runtime. Only applies if access_key_id and secret_access_key is blank.
--qingstor-secret-access-key string QingStor Secret Access Key (password)
@@ -231,18 +233,21 @@ rclone check source:path dest:path [flags]
--rc-addr string IPaddress:Port or :Port to bind server to. (default "localhost:5572")
--rc-cert string SSL PEM key (concatenation of certificate and CA certificate)
--rc-client-ca string Client certificate authority to verify clients with
--rc-files string Path to local files to serve on the HTTP server.
--rc-htpasswd string htpasswd file - if not provided no authentication is done
--rc-key string SSL PEM Private key
--rc-max-header-bytes int Maximum size of request header (default 4096)
--rc-no-auth Don't require auth for certain methods.
--rc-pass string Password for authentication.
--rc-realm string realm for authentication (default "rclone")
--rc-serve Enable the serving of remote objects.
--rc-server-read-timeout duration Timeout for server reading data (default 1h0m0s)
--rc-server-write-timeout duration Timeout for server writing data (default 1h0m0s)
--rc-user string User name for authentication.
--retries int Retry operations this many times if they fail (default 3)
--retries-sleep duration Interval between retrying operations if they fail, e.g 500ms, 60s, 5m. (0 to disable)
--s3-access-key-id string AWS Access Key ID.
--s3-acl string Canned ACL used when creating buckets and/or storing objects in S3.
--s3-acl string Canned ACL used when creating buckets and storing or copying objects.
--s3-chunk-size SizeSuffix Chunk size to use for uploading. (default 5M)
--s3-disable-checksum Don't store MD5 checksum with object metadata
--s3-endpoint string Endpoint for S3 API.
@@ -303,7 +308,7 @@ rclone check source:path dest:path [flags]
--union-remotes string List of space separated remotes.
-u, --update Skip files that are newer on the destination.
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.44")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.45")
-v, --verbose count Print lots more stuff (repeat for more)
--webdav-bearer-token string Bearer token instead of user/pass (eg a Macaroon)
--webdav-pass string Password.
@@ -312,10 +317,11 @@ rclone check source:path dest:path [flags]
--webdav-vendor string Name of the Webdav site/service/software you are using
--yandex-client-id string Yandex Client Id
--yandex-client-secret string Yandex Client Secret
--yandex-unlink Remove existing public link to file/folder with link command rather than creating.
```
### SEE ALSO
* [rclone](/commands/rclone/) - Show help for rclone commands, flags and backends.
###### Auto generated by spf13/cobra on 15-Oct-2018
###### Auto generated by spf13/cobra on 24-Nov-2018
Loaded 100 of 389 files, more files were not shown because too many files have changed in this diff. Show more