Compare commits

...
Author SHA1 Message Date
Nick Craig-Wood c48424c0eb mega: skip undecodable FSNodes - #3740 - FIXME vendor patch DO NOT MERGE
Also test whether we have a decryption key for shared nodes
2019-11-19 20:15:55 +00:00
Nick Craig-Wood e14d968f8d Start v1.50.2-DEV development 2019-11-19 16:51:32 +00:00
Nick Craig-Wood e0eeeaafcd accounting: don't show entries in both transferring and checking
See: https://forum.rclone.org/t/showing-progress-checking/12958
2019-11-19 13:22:33 +00:00
Nick Craig-Wood d46f8d0ae5 accounting: fix memory leak on retries operations
Before this change if an operation was retried on operations.Copy and
the operation was large enough to use an async buffer then an async
buffer was leaked on the retry.  This leaked memory, a file handle and
a go routine.

After this change if Account.WithBuffer is called and there is already
a buffer, then a new one won't be allocated.
2019-11-19 12:11:59 +00:00
Nick Craig-Wood 1e6278556c Add Maciej Zimnoch to contributors 2019-11-18 16:28:19 +00:00
Nick Craig-Wood 303f4ee152 Add Ankur Gupta to contributors 2019-11-18 16:28:19 +00:00
Nguyễn Hữu Luân 2fe8285f89 swift: reserve segments of dynamic large object when delete objects in container what was enabled versioning.
add code handle move object when moving the object is contained by the container what was enabled versioning with "X-History-Location".
2019-11-18 16:26:10 +00:00
Maciej Zimnoch f5443ac939 accounting: clear finished transfer in stats-reset
In order to reduce memory usage `stats-reset` also
clears finished transfers.

Fixes #3734
2019-11-18 14:25:32 +00:00
Maciej Zimnoch 7cf056b2c2 accounting: allow MaxCompletedTransfers to be configurable
rclone library users might be intrested in changing default value to
other, or even disabling it. With current version it's impossible which
leads to races when number of uploaded objects exceeds default limit.

Fixes #3732
2019-11-18 14:25:32 +00:00
Ankur Gupta 75a6c49f87 Fix error counter - fixes #3650
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
2019-11-18 14:13:02 +00:00
Nick Craig-Wood 19229b1215 drive: fix --drive-root-folder-id with team/shared drives
Before this change rclone used the team_drive ID as the root if set
even if the root_folder_id was set too.

This change uses the root_folder_id in preference over the team_drive
which restores the functionality.

This problem was introduced by ba7c2ac443

Fixes #3742
2019-11-16 18:38:21 +00:00
Nick Craig-Wood b5bb4c2a21 vfs: fix tests not to upload a 0 length file
Some remotes can't upload 0 length files, so this fixes the
TestCacheRename test so that it writes something to the file.
2019-11-15 09:26:40 +00:00
Nick Craig-Wood 479c803fd9 vendor: update all dependencies 2019-11-14 21:51:34 +00:00
Nick Craig-Wood 3dcf1e61cf cache: follow move of upstream library github.com/coreos/bbolt github.com/etcd-io/bbolt 2019-11-14 21:51:34 +00:00
Nick Craig-Wood 3da1cbfc81 Add Marco Molteni to contributors 2019-11-14 21:51:34 +00:00
Marco Molteni 0c9a8cf776 doc: add Scaleway to the S3 table of contents
Hello, documentation for Scaleway was already there, but the TOC was missing it.
2019-11-14 21:49:43 +00:00
Nick Craig-Wood f3871377c3 Add Sebastian Brandt to contributors 2019-11-14 12:54:42 +00:00
Nick Craig-Wood cc9a7dc073 Add Barry Muldrey to contributors 2019-11-14 12:54:42 +00:00
Nick Craig-Wood b61dd809ee Add new email for Anagh Kumar Baranwal 2019-11-14 12:54:38 +00:00
Sebastian Brandt f158a398f3 sftp: Retry Creation of Connection - fixes #3656
Removes the existing rate limiter because it is implicitly included in
the pacer.
2019-11-14 12:50:01 +00:00
jaKa acefa5c40d koofr: use rclone HTTP client. 2019-11-14 11:36:44 +00:00
Barry Muldrey 2784c3234b fs/config/configflags: fix --compare-dest and --copy-dest help strings
from rsync manual:

--compare-dest=DIR
    This option instructs rsync to use DIR on the destination machine as an
    additional hierarchy to compare destination files against doing transfers
    (if the files are missing in the destination directory). If a file is found
    in DIR that is identical to the sender's file, the file will NOT be
    transferred to the destination directory. This is useful for creating
    a sparse backup of just files that have changed from an earlier backup.

--copy-dest=DIR
    This option behaves like --compare-dest, but rsync will also copy unchanged
     files found in DIR to the destination directory using a local copy.
      This is useful for doing transfers to a new destination while leaving
       existing files intact, and then doing a flash-cutover when all files
        have been successfully transferred.
2019-11-12 13:37:58 +00:00
Nick Craig-Wood c21a4fee58 mount,cmount: make sure we call unmount when exiting 2019-11-11 22:08:52 +00:00
Nick Craig-Wood 358f5a8084 vfs: fix edge cases when reading ModTime from file
This fixes the unreliable test TestMount/CacheMode=full/TestFileModTime
2019-11-11 16:20:28 +00:00
Nick Craig-Wood 9115752679 proxy: reduce the internal bcrypt strength to fix race tests
Before this change the race tests were taking too long.  The bcrypt
function went from about 20ms to 1s under the race detector and this
is called for every transaction on webdav.

This change reduces the bcrypt strength so it takes 1ms non race so
the race tests pass and still has adequate security for in memory only
storage.
2019-11-11 16:20:28 +00:00
Nick Craig-Wood 51efb349ac vfs: revise locking in file and dir to fix race conditions 2019-11-11 16:20:27 +00:00
Nick Craig-Wood e0d9314059 mounttest: fix occasionally failing test TestRenameOpenHandle 2019-11-11 16:20:27 +00:00
Nick Craig-Wood 21c6babdbb mount: enable async reads for a 20% speedup
Now that the vfs can cope with out of order reads we can enable the
async read feature for an increase in througput on the local disk of
about 20%.
2019-11-11 16:20:27 +00:00
Nick Craig-Wood 5beeac7959 vfs: make ReadAt for non cached files work better with non-sequential reads
This makes ReadAt for non cached files wait a short time (up to 5mS)
if it gets an out of order read (which would normally cause a seek and
which take a long time) to see if the gap will be filled with an in
order read.

This makes mount2 based on go-fuse work more efficiently and enables
async reading in normal mount.

A similar change was done for WriteAt in af030f74f5
2019-11-11 16:20:27 +00:00
Nick Craig-Wood be5392f448 vfs: only calculate one hash for reads
This speeds up mounting on the local backend enormously.
2019-11-11 16:20:27 +00:00
Nick Craig-Wood c00dcb7e67 chunkedreader: disable hash calculation for first segment
This will produce a slight speedup for small files.
2019-11-11 16:20:27 +00:00
Nick Craig-Wood 6150ae89d6 vfs: add a newly created file straight into the directory 2019-11-11 15:20:09 +00:00
Nick Craig-Wood 1e423d21e1 drive: fix listing of the root directory with drive.files scope
We attempt to find the ID of the root folder by doing a GET on the
folder ID "root". With scope "drive.files" this fails with a 404
message.

After this change if we get the 404 message, we just carry on using
"root" as the root folder ID and we cache that for future lookups.

This means that changenotify messages will not work correctly in the
root folder but otherwise has minor consequences.

See: https://forum.rclone.org/t/fresh-raspberry-pi-build-google-drive-404-error-failed-to-ls-googleapi-error-404-file-not-found/12791
2019-11-11 09:07:34 +00:00
Brett Dutro 53d55ae760 Add test for cache renaming functionality 2019-11-10 11:58:46 +00:00
Anagh Kumar Baranwal 5928704e1b On rename, rename in cache too if the file exists
Signed-off-by: Anagh Kumar Baranwal <6824881+darthShadow@users.noreply.github.com>
2019-11-10 11:58:46 +00:00
buengese 5ddfa9f7f6 config: SetValueAndSave ignore error if config section does not exist yet 2019-11-09 16:44:08 +00:00
Nick Craig-Wood 9b5308144f s3: Reduce memory usage streaming files by reducing max stream upload size
Before this change rclone would allow the user to stream (eg with
rclone mount, rclone rcat or uploading google photos or docs) 5TB
files.  This meant that rclone allocated 4 * 525 MB buffers per
transfer which is way too much memory by default.

This change makes rclone use the configured chunk size for streamed
uploads.  This is 5MB by default which means that rclone can stream
upload files up to 48GB by default staying below the 10,000 chunks
limit.

This can be increased with --s3-chunk-size if necessary.

If rclone detects that a file is being streamed to s3 it will make a
single NOTICE level log stating the limitation.

This fixes the enormous memory usage.

Fixes #3568
See: https://forum.rclone.org/t/how-much-memory-does-rclone-need/12743
2019-11-09 15:55:19 +00:00
Aleksandar Jankovic 4b20afa94a backend/s3: fix ExpiryWindow value
ExpiryWindow accepts duration but it was set to value 3.
This changes it to 3 * time.Minute since default is 5 min.
2019-11-05 13:55:55 +00:00
Nick Craig-Wood 049ff1f269 config: check a remote exists when creating a new one 2019-11-05 12:39:33 +00:00
Nick Craig-Wood 3f7af64316 config: give config questions default values - fixes #3672 2019-11-05 11:53:44 +00:00
Nick Craig-Wood 0eaf5475ef Start v1.50.1-DEV development 2019-11-02 15:26:01 +00:00
Nick Craig-Wood 7bf056316f local: fix listings of . on Windows - fixes #3676 2019-10-30 16:00:18 +00:00
Xiaoxing Ye 520ddbcceb config: do not open browser on headless if google fs
On google fs (drive, google photos, and google cloud storage), if
headless is selected, do not open browser.

This also supplies a new option "auth-no-open-browser" for authorize
if the user does not want it.

This should fix #3323.
2019-10-30 14:12:42 +00:00
Nick Craig-Wood 1ce1ea34aa hash: fix hash names for DropboxHash and CRC-32
These were unintentionally renamed as part of 1dc8bcd48c

Fixes #3679
2019-10-30 12:20:10 +00:00
Nick Craig-Wood e6378daadf fshttp: don't print token bucket errors on context cancelled
These happen as a natural part of exceeding --max-transfer and we
don't need to worry the user with them.
2019-10-30 12:20:10 +00:00
Nick Craig-Wood 7ff95c6250 Add Xiaoxing Ye to contributors 2019-10-30 12:20:10 +00:00
Xiaoxing Ye 6d58d9a86f vendor: change goftp/server url
Closing #3674
2019-10-29 17:41:56 +00:00
Chaitanya e0356f5aae rcd: Adding group parameter to stats 2019-10-29 16:39:37 +00:00
Xiaoxing Ye 191cfb79d1 onedrive: no trailing slash reading metadata...
No trailing slash when reading metadata of an item given item ID.

This should fix #3664.
2019-10-29 13:33:11 +00:00
Nick Craig-Wood e81eca4055 fshttp: fix error reporting on tpslimit token bucket errors 2019-10-28 22:11:38 +00:00
Nick Craig-Wood ee3215ac76 build: make replacement of new rclone binary atomic on build
This avoids the "text file busy" message when trying to replace the
binary of a running rclone.
2019-10-28 22:11:38 +00:00
Nick Craig-Wood 199ac61bde rc: add methods to turn on blocking and mutex profiling 2019-10-28 22:11:38 +00:00
Nick Craig-Wood a40cc1167d Add zero-24 to contributors 2019-10-28 16:49:33 +00:00
zero-24 c57ea8d867 docs: add instructions to create your own dropbox app ID 2019-10-28 16:49:16 +00:00
Nick Craig-Wood 1868c77e16 rc: fix formatting of docs 2019-10-27 10:43:40 +00:00
Brett Dutro 378a3f4133 mount: replace use of WriteAt with Write for cache mode >= writes and O_APPEND
os.File.WriteAt returns an error if a file was opened with O_APPEND.
This replaces it with os.File.Write if the file was opened with
O_APPEND.
2019-10-26 17:27:52 +01:00
Nick Craig-Wood daff5a824e Start v1.50.0-DEV development 2019-10-26 12:42:06 +01:00
Nick Craig-Wood 6fabf476cf Version v1.50.0 2019-10-26 11:04:54 +01:00
Nick Craig-Wood ab895390f4 s3: fix nil pointer reference if no metadata returned for object
Fixes #3651 Fixes #3652
2019-10-25 13:45:47 +01:00
Nick Craig-Wood a3a5857874 drive: fix change notify polling when using appDataFolder
See: https://forum.rclone.org/t/remote-changes-arent-picked-up/12520
2019-10-24 12:51:01 +01:00
Nick Craig-Wood 0f0079ff71 b2: remove unverified: prefix on sha1 - fixes #3654 2019-10-23 08:41:56 +01:00
Nick Craig-Wood 18c029e0f0 Add dausruddin to contributors 2019-10-21 22:28:44 +01:00
dausruddin 7eee2f904a drive: fix typo 2019-10-21 22:28:28 +01:00
Nick Craig-Wood 3ef0c73826 drive: fix ChangeNotify polling for shared drives
Before this fix we neglected to add the shared drive ID to the request
when asking for an initial change notify token and this caused a lot
more results to be returned than was necessary.
2019-10-21 20:51:11 +01:00
Nick Craig-Wood 59026c4761 mount, cmount: don't pass huge filenames (>4k) to FUSE as it can't cope 2019-10-21 20:51:11 +01:00
Nick Craig-Wood 76f5e273d2 vfs: stop change notify polling clearing so much of the directory cache
Before this change, change notify polls would clear the directory
cache recursively. So uploading a file to the root would clear the
entire directory cache.

After this change we just invalidate the directory cache of the parent
directory of the item and if the item was a directory we invalidate it
too.
2019-10-21 20:51:11 +01:00
Nick Craig-Wood 2bbfcc74e9 drive: fix --drive-shared-with-me from the root with ls and --fast-list
When we changed recursive lists to use --fast-list by default this
broke listing with --drive-shared-with-me from the root.

This turned out to be an unwarranted assumption in the ListR code that
all items would have a parent folder that we had searched for - this
isn't true for shared with me items.

This was fixed when using --drive-shared-with-me to give items that
didn't have any parents a synthetic parent.

Fixes #3639
2019-10-21 12:16:01 +01:00
Nick Craig-Wood ba7c2ac443 drive: make sure that drive root ID is always canonical
Before this change we used the id "root" as an alias for the root drive ID.

However this causes problems when we receive IDs back from drive which
are not in this format and have been expanded to their canonical ID.

This change looks up the ID "root" and stores it in the
"drive_folder_id" parameter in the config file.

This helps with
- Notifying changes at the root
- Files shared with me at the root

See #3639
2019-10-21 12:16:01 +01:00
Nick Craig-Wood 2d9b8cb981 azureblob: disable logging to the Windows event log
See: https://forum.rclone.org/t/event-log-warning/12430
2019-10-21 11:50:31 +01:00
Ivan Andreev 2e50543053 Add Ivan Andreev to maintainers 2019-10-20 00:33:16 +03:00
Nick Craig-Wood 22bf8589cd Add Saksham Khanna to contributors 2019-10-17 15:05:46 +01:00
Nick Craig-Wood 0871c57f1b Add Carlos Ferreyra to contributors 2019-10-17 15:05:46 +01:00
Saksham Khanna 0c265713fd rc: added command core/quit 2019-10-17 15:04:22 +01:00
Carlos Ferreyra 9cb549a227 sftp: include more ciphers with use_insecure_cipher 2019-10-17 14:58:31 +01:00
Nick Craig-Wood 13e46c4b3f accounting: cull the old time ranges when possible to save memory 2019-10-17 11:43:32 +01:00
Nick Craig-Wood d40972bf1a accounting: allow up to 100 completed transfers in the accounting list
This fixes the core/transfers rc so it shows items again.
2019-10-16 22:13:17 +01:00
Nick Craig-Wood b002ff8d54 accounting: fix total duration calculation
This was broken in e337cae0c5 when we deleted the transfers
immediately.

This is fixed by keeping a merged slice of time ranges of completed
transfers and adding those to the current transfers.
2019-10-16 22:13:17 +01:00
Nick Craig-Wood 38652d046d drive: disable HTTP/2 by default to work around INTERNAL_ERROR problems
Before this change when rclone was compiled with go1.13 it used HTTP/2
to contact drive by default.

This causes lockups and INTERNAL_ERRORs from the HTTP/2 code.

This is a workaround disabling the HTTP/2 code on an option.

It can be re-enabled with `--drive-disable-http2=false`

See #3631
2019-10-16 11:26:08 +01:00
Nick Craig-Wood 0b6cdb7677 fshttp: allow Transport to be customized #3631 2019-10-16 11:26:08 +01:00
Nick Craig-Wood 543100070a sync: free objects after they come out of the transfer pipe
This reduces memory when the transfer pipe shrinks

See: https://forum.rclone.org/t/rclone-memory-consumption-increasing-linearly/12244
2019-10-16 10:27:07 +01:00
Nick Craig-Wood e337cae0c5 accounting: fix memory leak noticeable for transfers of large numbers of objects
Before this fix we weren't removing transfers from the transfer stats.
For transfers with 1000s of objects this uses a noticeable amount of
memory.

See: https://forum.rclone.org/t/rclone-memory-consumption-increasing-linearly/12244
2019-10-16 10:27:07 +01:00
Nick Craig-Wood 90a23ae01b Add Bryce Larson to contributors 2019-10-16 10:27:07 +01:00
Bryce Larson dd150efdd7 docs: fix --use-server-modtime spelling in docs 2019-10-15 19:54:42 +01:00
Nick Craig-Wood af05e290cf Fix --files-from without --no-traverse doing a recursive scan
In a28239f005 we made --files-from obey --no-traverse.  In the
process this caused --files-from without --no-traverse to do a
complete recursive scan unecessarily.

This was only noticeable in users of fs/march, so sync/copy/move/etc
not in ls/lsf/etc.

This fix makes sure that we use conventional directory listings in
fs/march unless `--files-from` and `--no-traverse` is set or
`--fast-list` is active.

Fixes #3619
2019-10-15 19:51:01 +01:00
Nick Craig-Wood f9f9d5029b fserror: make http2 "stream error:" a retriable error
It was reported that v1.49.4 which was accidentally compiled with
go1.13 instead of go1.12 produced errors like this:

    Failed to get StartPageToken: Get https://www.googleapis.com/drive/v3/changes/startPageToken?XXX: stream error: stream ID 1789; INTERNAL_ERROR
    IO error: open file failed: Get https://www.googleapis.com/drive/v3/files/XXX?alt=media: stream error: stream ID 1781; INTERNAL_ERROR

These are errors from the http2 library.  It appears that go1.13 when
communicating with google drive defaults to http2 whereas with go1.12
it doesn't.

It is unclear what is causing these errors, but retrying them since
they don't happen very often seems like a valid strategy.

This was fixed in v1.49.5 by compiling with go1.12 - this fix is
designed to work with go1.13

See: https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108/
2019-10-15 19:46:44 +01:00
Nick Craig-Wood 7d3b67f6cc Add AlexandrBoltris to contributors 2019-10-15 19:46:44 +01:00
Cenk Alti 929f275ae5 putio: add ability to resume uploads 2019-10-14 20:01:16 +01:00
AlexandrBoltris c526bdb579 docs: typo fix in faq.md 2019-10-14 17:07:29 +01:00
Nick Craig-Wood 1b2ffbeca0 cmd: fix environment variables not setting command line flags
Before this fix quite a lot of the commands were ignoring environment
variables intended to set flags.
2019-10-14 17:02:09 +01:00
Nick Craig-Wood 19429083ad cmd: fix spelling of Definition 2019-10-14 17:02:09 +01:00
Nick Craig-Wood 6e378d7d32 config: fix setting of non top level flags from environment variables
Before this fix, attempting to set a non top level environment
variable would fail with "Couldn't find flag".

This fixes it by passing in the flags that the env var is being set
from.

Fixes #3615
2019-10-14 17:02:09 +01:00
Nick Craig-Wood 1fe1a19339 vfs: stop empty dirs disappearing when renamed on bucket based remotes
Before this change when we renamed a directory this cleared the
directory cache for the parent directory too.

If the directory was remaining in the same parent this wasn't
necessary and caused the empty directory to fall out of the cache.

Fixes #3597
2019-10-14 14:38:30 +01:00
Chaitanya b63e9befe8 rc docs: fix code section not rendering properly due to missing quotes 2019-10-13 12:26:37 +01:00
704 changed files with 54731 additions and 31989 deletions

No files matched your search

+1
View File
@@ -12,6 +12,7 @@ Current active maintainers of rclone are:
| Alex Chen | @Cnly | onedrive backend |
| Sandeep Ummadi | @sandeepkru | azureblob backend |
| Sebastian Bünger | @buengese | jottacloud & yandex backends |
| Ivan Andreev | @ivandeex | chunker & mailru backends |
**This is a work in progress Draft**
Generated
+7751 -6307
View File
File diff suppressed because it is too large. Load diff
Generated
+1946 -53
View File
File diff suppressed because it is too large. Load diff
Generated
+7547 -6522
View File
File diff suppressed because it is too large. Load diff
+7 -1
View File
@@ -46,7 +46,8 @@ endif
rclone:
go build -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS)
mkdir -p `go env GOPATH`/bin/
cp -av rclone`go env GOEXE` `go env GOPATH`/bin/
cp -av rclone`go env GOEXE` `go env GOPATH`/bin/rclone`go env GOEXE`.new
mv -v `go env GOPATH`/bin/rclone`go env GOEXE`.new `go env GOPATH`/bin/rclone`go env GOEXE`
test_all:
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
@@ -96,6 +97,11 @@ update:
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
# Tidy the module dependencies
tidy:
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
doc: rclone.1 MANUAL.html MANUAL.txt rcdocs commanddocs
rclone.1: MANUAL.md
+2 -1
View File
@@ -15,6 +15,7 @@ This file describes how to make the various kinds of releases
* make test # see integration test server or run locally
* make tag
* edit docs/content/changelog.md
* make tidy
* make doc
* git status - to check for new man pages - git add them
* git commit -a -v -m "Version v1.XX.0"
@@ -88,7 +89,7 @@ Now
* make TAG=${NEW_TAG} upload_github
* NB this overwrites the current beta so we need to do this
* git co master
* make LAST_TAG=${NEW_TAG} startdev
* make VERSION=${NEW_TAG} startdev
* # cherry pick the changes to the changelog and VERSION
* git checkout ${BASE_TAG}-fixes VERSION docs/content/changelog.md
* git commit --amend
+1 -1
View File
@@ -1 +1 @@
v1.49.5
v1.50.2
+3
View File
@@ -312,6 +312,9 @@ func httpClientFactory(client *http.Client) pipeline.Factory {
//
// this code was copied from azblob.NewPipeline
func (f *Fs) newPipeline(c azblob.Credential, o azblob.PipelineOptions) pipeline.Pipeline {
// Don't log stuff to syslog/Windows Event log
pipeline.SetForceLogEnabled(false)
// Closest to API goes first; closest to the wire goes last
factories := []pipeline.Factory{
azblob.NewTelemetryPolicyFactory(o.Telemetry),
+6
View File
@@ -1375,6 +1375,12 @@ func (o *Object) decodeMetaDataRaw(ID, SHA1 string, Size int64, UploadTimestamp
if o.sha1 == "" || o.sha1 == "none" {
o.sha1 = Info[sha1Key]
}
// Remove unverified prefix - see https://www.backblaze.com/b2/docs/uploading.html
// Some tools (eg Cyberduck) use this
const unverified = "unverified:"
if strings.HasPrefix(o.sha1, unverified) {
o.sha1 = o.sha1[len(unverified):]
}
o.size = Size
// Use the UploadTimestamp if can't get file info
o.modTime = time.Time(UploadTimestamp)
+1 -1
View File
@@ -16,7 +16,7 @@ import (
"sync"
"time"
bolt "github.com/coreos/bbolt"
bolt "github.com/etcd-io/bbolt"
"github.com/pkg/errors"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/walk"
+1
View File
@@ -63,6 +63,7 @@ func init() {
Name: "password",
Help: "Password or pass phrase for encryption.",
IsPassword: true,
Required: true,
}, {
Name: "password2",
Help: "Password or pass phrase for salt. Optional but recommended.\nShould be different to the previous password.",
+89 -30
View File
@@ -10,6 +10,7 @@ package drive
import (
"bytes"
"context"
"crypto/tls"
"fmt"
"io"
"io/ioutil"
@@ -213,7 +214,15 @@ func init() {
}},
}, {
Name: "root_folder_id",
Help: "ID of the root folder\nLeave blank normally.\nFill in to access \"Computers\" folders. (see docs).",
Help: `ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.
Note that if this is blank, the first time rclone runs it will fill it
in with the ID of the root folder.
`,
}, {
Name: "service_account_file",
Help: "Service Account Credentials JSON file path \nLeave blank normally.\nNeeded only if you want use SA instead of interactive login.",
@@ -402,9 +411,23 @@ older versions that have been set to keep forever.`,
This can be useful if you wish to do a server side copy between two
different Google drives. Note that this isn't enabled by default
because it isn't easy to tell if it will work beween any two
because it isn't easy to tell if it will work between any two
configurations.`,
Advanced: true,
}, {
Name: "disable_http2",
Default: true,
Help: `Disable drive using http2
There is currently an unsolved issue with the google drive backend and
HTTP/2. HTTP/2 is therefore disabled by default for the drive backend
but can be re-enabled here. When the issue is solved this flag will
be removed.
See: https://github.com/rclone/rclone/issues/3631
`,
Advanced: true,
}},
})
@@ -452,6 +475,7 @@ type Options struct {
PacerMinSleep fs.Duration `config:"pacer_min_sleep"`
PacerBurst int `config:"pacer_burst"`
ServerSideAcrossConfigs bool `config:"server_side_across_configs"`
DisableHTTP2 bool `config:"disable_http2"`
}
// Fs represents a remote drive server
@@ -565,6 +589,23 @@ func containsString(slice []string, s string) bool {
return false
}
// getRootID returns the canonical ID for the "root" ID
func (f *Fs) getRootID() (string, error) {
var info *drive.File
var err error
err = f.pacer.CallNoRetry(func() (bool, error) {
info, err = f.svc.Files.Get("root").
Fields("id").
SupportsAllDrives(true).
Do()
return shouldRetry(err)
})
if err != nil {
return "", errors.Wrap(err, "couldn't find root directory ID")
}
return info.Id, nil
}
// Lists the directory required calling the user function on each item found
//
// If the user fn ever returns true then it early exits with found = true
@@ -789,7 +830,7 @@ func configTeamDrive(ctx context.Context, opt *Options, m configmap.Mapper, name
} else {
fmt.Printf("Change current team drive ID %q?\n", opt.TeamDriveID)
}
if !config.Confirm() {
if !config.Confirm(false) {
return nil
}
client, err := createOAuthClient(opt, name, m)
@@ -840,6 +881,18 @@ func newPacer(opt *Options) *fs.Pacer {
return fs.NewPacer(pacer.NewGoogleDrive(pacer.MinSleep(opt.PacerMinSleep), pacer.Burst(opt.PacerBurst)))
}
// getClient makes an http client according to the options
func getClient(opt *Options) *http.Client {
t := fshttp.NewTransportCustom(fs.Config, func(t *http.Transport) {
if opt.DisableHTTP2 {
t.TLSNextProto = map[string]func(string, *tls.Conn) http.RoundTripper{}
}
})
return &http.Client{
Transport: t,
}
}
func getServiceAccountClient(opt *Options, credentialsData []byte) (*http.Client, error) {
scopes := driveScopes(opt.Scope)
conf, err := google.JWTConfigFromJSON(credentialsData, scopes...)
@@ -849,7 +902,7 @@ func getServiceAccountClient(opt *Options, credentialsData []byte) (*http.Client
if opt.Impersonate != "" {
conf.Subject = opt.Impersonate
}
ctxWithSpecialClient := oauthutil.Context(fshttp.NewClient(fs.Config))
ctxWithSpecialClient := oauthutil.Context(getClient(opt))
return oauth2.NewClient(ctxWithSpecialClient, conf.TokenSource(ctxWithSpecialClient)), nil
}
@@ -871,7 +924,7 @@ func createOAuthClient(opt *Options, name string, m configmap.Mapper) (*http.Cli
return nil, errors.Wrap(err, "failed to create oauth client from service account")
}
} else {
oAuthClient, _, err = oauthutil.NewClient(name, m, driveConfig)
oAuthClient, _, err = oauthutil.NewClientWithBaseClient(name, m, driveConfig, getClient(opt))
if err != nil {
return nil, errors.Wrap(err, "failed to create oauth client")
}
@@ -968,15 +1021,25 @@ func NewFs(name, path string, m configmap.Mapper) (fs.Fs, error) {
}
// set root folder for a team drive or query the user root folder
if f.isTeamDrive {
if opt.RootFolderID != "" {
// override root folder if set or cached in the config
f.rootFolderID = opt.RootFolderID
} else if f.isTeamDrive {
f.rootFolderID = f.opt.TeamDriveID
} else {
f.rootFolderID = "root"
}
// override root folder if set in the config
if opt.RootFolderID != "" {
f.rootFolderID = opt.RootFolderID
// Look up the root ID and cache it in the config
rootID, err := f.getRootID()
if err != nil {
if gerr, ok := errors.Cause(err).(*googleapi.Error); ok && gerr.Code == 404 {
// 404 means that this scope does not have permission to get the
// root so just use "root"
rootID = "root"
} else {
return nil, err
}
}
f.rootFolderID = rootID
m.Set("root_folder_id", rootID)
}
f.dirCache = dircache.New(root, f.rootFolderID, f)
@@ -1458,6 +1521,10 @@ func (f *Fs) listRRunner(ctx context.Context, wg *sync.WaitGroup, in <-chan list
listRSlices{dirs, paths}.Sort()
var iErr error
_, err := f.list(ctx, dirs, "", false, false, false, func(item *drive.File) bool {
// shared with me items have no parents when at the root
if f.opt.SharedWithMe && len(item.Parents) == 0 && len(paths) == 1 && paths[0] == "" {
item.Parents = dirs
}
for _, parent := range item.Parents {
// only handle parents that are in the requested dirs list
i := sort.SearchStrings(dirs, parent)
@@ -1526,20 +1593,6 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
if err != nil {
return err
}
if directoryID == "root" {
var info *drive.File
err = f.pacer.CallNoRetry(func() (bool, error) {
info, err = f.svc.Files.Get("root").
Fields("id").
SupportsAllDrives(true).
Do()
return shouldRetry(err)
})
if err != nil {
return err
}
directoryID = info.Id
}
mu := sync.Mutex{} // protects in and overflow
wg := sync.WaitGroup{}
@@ -2267,9 +2320,11 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
func (f *Fs) changeNotifyStartPageToken() (pageToken string, err error) {
var startPageToken *drive.StartPageToken
err = f.pacer.Call(func() (bool, error) {
startPageToken, err = f.svc.Changes.GetStartPageToken().
SupportsAllDrives(true).
Do()
changes := f.svc.Changes.GetStartPageToken().SupportsAllDrives(true)
if f.isTeamDrive {
changes.DriveId(f.opt.TeamDriveID)
}
startPageToken, err = changes.Do()
return shouldRetry(err)
})
if err != nil {
@@ -2292,7 +2347,11 @@ func (f *Fs) changeNotifyRunner(ctx context.Context, notifyFunc func(string, fs.
changesCall.SupportsAllDrives(true)
changesCall.IncludeItemsFromAllDrives(true)
if f.isTeamDrive {
changesCall.TeamDriveId(f.opt.TeamDriveID)
changesCall.DriveId(f.opt.TeamDriveID)
}
// If using appDataFolder then need to add Spaces
if f.rootFolderID == "appDataFolder" {
changesCall.Spaces("appDataFolder")
}
changeList, err = changesCall.Context(ctx).Do()
return shouldRetry(err)
+1 -1
View File
@@ -113,7 +113,7 @@ var (
// Register with Fs
func init() {
DbHashType = hash.RegisterHash("Dropbox", 64, dbhash.New)
DbHashType = hash.RegisterHash("DropboxHash", 64, dbhash.New)
fs.Register(&fs.RegInfo{
Name: "dropbox",
Description: "Dropbox",
+3 -3
View File
@@ -84,14 +84,14 @@ func init() {
tokenString, ok := m.Get("token")
if ok && tokenString != "" {
fmt.Printf("Already have a token - refresh?\n")
if !config.Confirm() {
if !config.Confirm(false) {
return
}
}
srv := rest.NewClient(fshttp.NewClient(fs.Config))
fmt.Printf("\nDo you want to create a machine specific API key?\n\nRclone has it's own Jottacloud API KEY which works fine as long as one only uses rclone on a single machine. When you want to use rclone with this account on more than one machine it's recommended to create a machine specific API key. These keys can NOT be shared between machines.\n\n")
if config.Confirm() {
if config.Confirm(false) {
deviceRegistration, err := registerDevice(ctx, srv)
if err != nil {
log.Fatalf("Failed to register device: %v", err)
@@ -127,7 +127,7 @@ func init() {
}
fmt.Printf("\nDo you want to use a non standard device/mountpoint e.g. for accessing files uploaded using the official Jottacloud client?\n\n")
if config.Confirm() {
if config.Confirm(false) {
oAuthClient, _, err := oauthutil.NewClient(name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to load oAuthClient: %s", err)
+4 -1
View File
@@ -16,6 +16,7 @@ import (
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/config/obscure"
"github.com/rclone/rclone/fs/encodings"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
httpclient "github.com/koofr/go-httpclient"
@@ -259,7 +260,9 @@ func NewFs(name, root string, m configmap.Mapper) (ff fs.Fs, err error) {
if err != nil {
return nil, err
}
client := koofrclient.NewKoofrClient(opt.Endpoint, false)
httpClient := httpclient.New()
httpClient.Client = fshttp.NewClient(fs.Config)
client := koofrclient.NewKoofrClientWithHTTPClient(opt.Endpoint, httpClient)
basicAuth := fmt.Sprintf("Basic %s",
base64.StdEncoding.EncodeToString([]byte(opt.User+":"+pass)))
client.HTTPClient.Headers.Set("Authorization", basicAuth)
+8 -8
View File
@@ -350,7 +350,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
err = errors.Wrapf(err, "failed to open directory %q", dir)
fs.Errorf(dir, "%v", err)
if isPerm {
accounting.Stats(ctx).Error(fserrors.NoRetryError(err))
_ = accounting.Stats(ctx).Error(fserrors.NoRetryError(err))
err = nil // ignore error but fail sync
}
return nil, err
@@ -386,7 +386,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
if fierr != nil {
err = errors.Wrapf(err, "failed to read directory %q", namepath)
fs.Errorf(dir, "%v", fierr)
accounting.Stats(ctx).Error(fserrors.NoRetryError(fierr)) // fail the sync
_ = accounting.Stats(ctx).Error(fserrors.NoRetryError(fierr)) // fail the sync
continue
}
fis = append(fis, fi)
@@ -409,7 +409,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
// Skip bad symlinks
err = fserrors.NoRetryError(errors.Wrap(err, "symlink"))
fs.Errorf(newRemote, "Listing error: %v", err)
accounting.Stats(ctx).Error(err)
err = accounting.Stats(ctx).Error(err)
continue
}
if err != nil {
@@ -1084,17 +1084,17 @@ func (o *Object) Remove(ctx context.Context) error {
func cleanRootPath(s string, noUNC bool) string {
if runtime.GOOS == "windows" {
s = filepath.ToSlash(s)
vol := filepath.VolumeName(s)
s = vol + enc.FromStandardPath(s[len(vol):])
s = filepath.FromSlash(s)
if !filepath.IsAbs(s) && !strings.HasPrefix(s, "\\") {
s2, err := filepath.Abs(s)
if err == nil {
s = s2
}
}
s = filepath.ToSlash(s)
vol := filepath.VolumeName(s)
s = vol + enc.FromStandardPath(s[len(vol):])
s = filepath.FromSlash(s)
if !noUNC {
// Convert to UNC
s = uncPath(s)
+1 -1
View File
@@ -54,7 +54,7 @@ var testsWindows = [][2]string{
{`\\?\UNC\theserver\dir\file.txt`, `\\?\UNC\theserver\dir\file.txt`},
{`//?/UNC/theserver/dir\file.txt`, `\\?\UNC\theserver\dir\file.txt`},
{`c:/temp`, `c:\temp`},
{`/temp/file.txt`, `\temp\file.txt`},
{`C:/temp/file.txt`, `C:\temp\file.txt`},
{`c:\!\"#¤%&/()=;:*^?+-`, `c:\!\"#¤%&\()=;:*^?+-`},
{`c:\<>"|?*:&\<>"|?*:&\<>"|?*:&`, `c:\<>"|?*:&\<>"|?*:&\<>"|?*:&`},
}
+6 -1
View File
@@ -351,8 +351,13 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
// instead of simply using `drives/driveID/root:/itemPath` because it works for
// "shared with me" folders in OneDrive Personal (See #2536, #2778)
// This path pattern comes from https://github.com/OneDrive/onedrive-api-docs/issues/908#issuecomment-417488480
//
// If `relPath` == '', do not append the slash (See #3664)
func (f *Fs) readMetaDataForPathRelativeToID(ctx context.Context, normalizedID string, relPath string) (info *api.Item, resp *http.Response, err error) {
opts := newOptsCall(normalizedID, "GET", ":/"+withTrailingColon(rest.URLPathEscape(enc.FromStandardPath(relPath))))
if relPath != "" {
relPath = "/" + withTrailingColon(rest.URLPathEscape(enc.FromStandardPath(relPath)))
}
opts := newOptsCall(normalizedID, "GET", ":"+relPath)
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
+43
View File
@@ -0,0 +1,43 @@
package putio
import (
"fmt"
"net/http"
"github.com/putdotio/go-putio/putio"
"github.com/rclone/rclone/fs/fserrors"
)
func checkStatusCode(resp *http.Response, expected int) error {
if resp.StatusCode != expected {
return &statusCodeError{response: resp}
}
return nil
}
type statusCodeError struct {
response *http.Response
}
func (e *statusCodeError) Error() string {
return fmt.Sprintf("unexpected status code (%d) response while doing %s to %s", e.response.StatusCode, e.response.Request.Method, e.response.Request.URL.String())
}
func (e *statusCodeError) Temporary() bool {
return e.response.StatusCode == 429 || e.response.StatusCode >= 500
}
// shouldRetry returns a boolean as to whether this err deserves to be
// retried. It returns the err as a convenience
func shouldRetry(err error) (bool, error) {
if err == nil {
return false, nil
}
if perr, ok := err.(*putio.ErrorResponse); ok {
err = &statusCodeError{response: perr.Response}
}
if fserrors.ShouldRetry(err) {
return true, err
}
return false, err
}
+84 -43
View File
@@ -17,7 +17,6 @@ import (
"github.com/putdotio/go-putio/putio"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/configmap"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/oauthutil"
@@ -58,23 +57,6 @@ func (f *Fs) Features() *fs.Features {
return f.features
}
// shouldRetry returns a boolean as to whether this err deserves to be
// retried. It returns the err as a convenience
func shouldRetry(err error) (bool, error) {
if err == nil {
return false, nil
}
if fserrors.ShouldRetry(err) {
return true, err
}
if perr, ok := err.(*putio.ErrorResponse); ok {
if perr.Response.StatusCode == 429 || perr.Response.StatusCode >= 500 {
return true, err
}
}
return false, err
}
// NewFs constructs an Fs from the path, container:path
func NewFs(name, root string, m configmap.Mapper) (f fs.Fs, err error) {
// defer log.Trace(name, "root=%v", root)("f=%+v, err=%v", &f, &err)
@@ -318,66 +300,125 @@ func (f *Fs) createUpload(ctx context.Context, name string, size int64, parentID
}
func (f *Fs) sendUpload(ctx context.Context, location string, size int64, in io.Reader) (fileID int64, err error) {
// defer log.Trace(f, "location=%v, size=%v", location, size)("fileID=%v, err=%v", fileID, &err)
// defer log.Trace(f, "location=%v, size=%v", location, size)("fileID=%v, err=%v", &fileID, &err)
if size == 0 {
err = f.pacer.Call(func() (bool, error) {
fs.Debugf(f, "Sending zero length chunk")
fileID, err = f.transferChunk(ctx, location, 0, bytes.NewReader([]byte{}), 0)
_, fileID, err = f.transferChunk(ctx, location, 0, bytes.NewReader([]byte{}), 0)
return shouldRetry(err)
})
return
}
var start int64
var clientOffset int64
var offsetMismatch bool
buf := make([]byte, defaultChunkSize)
for start < size {
reqSize := size - start
if reqSize >= int64(defaultChunkSize) {
reqSize = int64(defaultChunkSize)
for clientOffset < size {
chunkSize := size - clientOffset
if chunkSize >= int64(defaultChunkSize) {
chunkSize = int64(defaultChunkSize)
}
chunk := readers.NewRepeatableLimitReaderBuffer(in, buf, reqSize)
chunk := readers.NewRepeatableLimitReaderBuffer(in, buf, chunkSize)
chunkStart := clientOffset
reqSize := chunkSize
transferOffset := clientOffset
fs.Debugf(f, "chunkStart: %d, reqSize: %d", chunkStart, reqSize)
// Transfer the chunk
err = f.pacer.Call(func() (bool, error) {
fs.Debugf(f, "Sending chunk. start: %d length: %d", start, reqSize)
// TODO get file offset and seek to the position
fileID, err = f.transferChunk(ctx, location, start, chunk, reqSize)
if offsetMismatch {
// Get file offset and seek to the position
offset, err := f.getServerOffset(ctx, location)
if err != nil {
return shouldRetry(err)
}
sentBytes := offset - chunkStart
fs.Debugf(f, "sentBytes: %d", sentBytes)
_, err = chunk.Seek(sentBytes, io.SeekStart)
if err != nil {
return shouldRetry(err)
}
transferOffset = offset
reqSize = chunkSize - sentBytes
offsetMismatch = false
}
fs.Debugf(f, "Sending chunk. transferOffset: %d length: %d", transferOffset, reqSize)
var serverOffset int64
serverOffset, fileID, err = f.transferChunk(ctx, location, transferOffset, chunk, reqSize)
if cerr, ok := err.(*statusCodeError); ok && cerr.response.StatusCode == 409 {
offsetMismatch = true
return true, err
}
if serverOffset != (transferOffset + reqSize) {
offsetMismatch = true
return true, errors.New("connection broken")
}
return shouldRetry(err)
})
if err != nil {
return
}
start += reqSize
clientOffset += chunkSize
}
return
}
func (f *Fs) transferChunk(ctx context.Context, location string, start int64, chunk io.ReadSeeker, chunkSize int64) (fileID int64, err error) {
// defer log.Trace(f, "location=%v, start=%v, chunkSize=%v", location, start, chunkSize)("fileID=%v, err=%v", fileID, &err)
_, _ = chunk.Seek(0, io.SeekStart)
func (f *Fs) getServerOffset(ctx context.Context, location string) (offset int64, err error) {
// defer log.Trace(f, "location=%v", location)("offset=%v, err=%v", &offset, &err)
req, err := f.makeUploadHeadRequest(ctx, location)
if err != nil {
return 0, err
}
resp, err := f.oAuthClient.Do(req)
if err != nil {
return 0, err
}
err = checkStatusCode(resp, 200)
if err != nil {
return 0, err
}
return strconv.ParseInt(resp.Header.Get("upload-offset"), 10, 64)
}
func (f *Fs) transferChunk(ctx context.Context, location string, start int64, chunk io.ReadSeeker, chunkSize int64) (serverOffset, fileID int64, err error) {
// defer log.Trace(f, "location=%v, start=%v, chunkSize=%v", location, start, chunkSize)("fileID=%v, err=%v", &fileID, &err)
req, err := f.makeUploadPatchRequest(ctx, location, chunk, start, chunkSize)
if err != nil {
return 0, err
return
}
req = req.WithContext(ctx)
res, err := f.oAuthClient.Do(req)
resp, err := f.oAuthClient.Do(req)
if err != nil {
return 0, err
return
}
defer func() {
_ = res.Body.Close()
_ = resp.Body.Close()
}()
if res.StatusCode != 204 {
return 0, fmt.Errorf("unexpected status code while transferring chunk: %d", res.StatusCode)
err = checkStatusCode(resp, 204)
if err != nil {
return
}
sfid := res.Header.Get("putio-file-id")
serverOffset, err = strconv.ParseInt(resp.Header.Get("upload-offset"), 10, 64)
if err != nil {
return
}
sfid := resp.Header.Get("putio-file-id")
if sfid != "" {
fileID, err = strconv.ParseInt(sfid, 10, 64)
if err != nil {
return 0, err
return
}
}
return fileID, nil
return
}
func (f *Fs) makeUploadHeadRequest(ctx context.Context, location string) (*http.Request, error) {
req, err := http.NewRequest("HEAD", location, nil)
if err != nil {
return nil, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
req.Header.Set("tus-resumable", "1.0.0")
return req, nil
}
func (f *Fs) makeUploadPatchRequest(ctx context.Context, location string, in io.Reader, offset, length int64) (*http.Request, error) {
+1 -1
View File
@@ -269,7 +269,7 @@ func qsServiceConnection(opt *Options) (*qs.Service, error) {
cf.Protocol = protocol
cf.Host = host
cf.Port = port
cf.ConnectionRetries = opt.ConnectionRetries
// unsupported in v3.1: cf.ConnectionRetries = opt.ConnectionRetries
cf.Connection = fshttp.NewClient(fs.Config)
return qs.Init(cf)
+33 -13
View File
@@ -26,6 +26,7 @@ import (
"regexp"
"strconv"
"strings"
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
@@ -693,14 +694,25 @@ The minimum is 0 and the maximum is 5GB.`,
Name: "chunk_size",
Help: `Chunk size to use for uploading.
When uploading files larger than upload_cutoff they will be uploaded
as multipart uploads using this chunk size.
When uploading files larger than upload_cutoff or files with unknown
size (eg from "rclone rcat" or uploaded with "rclone mount" or google
photos or google docs) they will be uploaded as multipart uploads
using this chunk size.
Note that "--s3-upload-concurrency" chunks of this size are buffered
in memory per transfer.
If you are transferring large files over high speed links and you have
enough memory, then increasing this will speed up the transfers.`,
enough memory, then increasing this will speed up the transfers.
Rclone will automatically increase the chunk size when uploading a
large file of known size to stay below the 10,000 chunks limit.
Files of unknown size are uploaded with the configured
chunk_size. Since the default chunk size is 5MB and there can be at
most 10,000 chunks, this means that by default the maximum size of
file you can stream upload is 48GB. If you wish to stream upload
larger files then you will need to increase chunk_size.`,
Default: minChunkSize,
Advanced: true,
}, {
@@ -771,12 +783,11 @@ WARNING: Storing parts of an incomplete multipart upload counts towards space us
// Constants
const (
metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime
metaMD5Hash = "Md5chksum" // the meta key to store md5hash in
listChunkSize = 1000 // number of items to read at once
maxRetries = 10 // number of retries to make of operations
maxSizeForCopy = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY
maxFileSize = 5 * 1024 * 1024 * 1024 * 1024 // largest possible upload file size
metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime
metaMD5Hash = "Md5chksum" // the meta key to store md5hash in
listChunkSize = 1000 // number of items to read at once
maxRetries = 10 // number of retries to make of operations
maxSizeForCopy = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY
minChunkSize = fs.SizeSuffix(s3manager.MinUploadPartSize)
defaultUploadCutoff = fs.SizeSuffix(200 * 1024 * 1024)
maxUploadCutoff = fs.SizeSuffix(5 * 1024 * 1024 * 1024)
@@ -961,7 +972,7 @@ func s3Connection(opt *Options) (*s3.S3, *session.Session, error) {
Client: ec2metadata.New(session.New(), &aws.Config{
HTTPClient: lowTimeoutClient,
}),
ExpiryWindow: 3,
ExpiryWindow: 3 * time.Minute,
},
}
cred := credentials.NewChainCredentials(providers)
@@ -1872,6 +1883,9 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
o.etag = aws.StringValue(resp.ETag)
o.bytes = size
o.meta = resp.Metadata
if o.meta == nil {
o.meta = map[string]*string{}
}
o.storageClass = aws.StringValue(resp.StorageClass)
if resp.LastModified == nil {
fs.Logf(o, "Failed to read last modified from HEAD: %v", err)
@@ -1979,6 +1993,8 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
return resp.Body, nil
}
var warnStreamUpload sync.Once
// Update the Object from in with modTime and size
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
bucket, bucketPath := o.split()
@@ -1998,10 +2014,14 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
u.S3 = o.fs.c
u.PartSize = int64(o.fs.opt.ChunkSize)
// size can be -1 here meaning we don't know the size of the incoming file. We use ChunkSize
// buffers here (default 5MB). With a maximum number of parts (10,000) this will be a file of
// 48GB which seems like a not too unreasonable limit.
if size == -1 {
// Make parts as small as possible while still being able to upload to the
// S3 file size limit. Rounded up to nearest MB.
u.PartSize = (((maxFileSize / s3manager.MaxUploadParts) >> 20) + 1) << 20
warnStreamUpload.Do(func() {
fs.Logf(o.fs, "Streaming uploads using chunk size %v will have maximum file size of %v",
o.fs.opt.ChunkSize, fs.SizeSuffix(u.PartSize*s3manager.MaxUploadParts))
})
return
}
// Adjust PartSize until the number of parts is small enough.
+28 -12
View File
@@ -29,15 +29,17 @@ import (
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/env"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/readers"
sshagent "github.com/xanzy/ssh-agent"
"golang.org/x/crypto/ssh"
"golang.org/x/time/rate"
)
const (
connectionsPerSecond = 10 // don't make more than this many ssh connections/s
hashCommandNotSupported = "none"
minSleep = 100 * time.Millisecond
maxSleep = 2 * time.Second
decayConstant = 2 // bigger for slower decay, exponential
)
var (
@@ -86,8 +88,19 @@ requested from the ssh-agent. This allows to avoid ` + "`Too many authentication
when the ssh-agent contains many keys.`,
Default: false,
}, {
Name: "use_insecure_cipher",
Help: "Enable the use of the aes128-cbc cipher and diffie-hellman-group-exchange-sha256, diffie-hellman-group-exchange-sha1 key exchange. Those algorithms are insecure and may allow plaintext data to be recovered by an attacker.",
Name: "use_insecure_cipher",
Help: `Enable the use of insecure ciphers and key exchange methods.
This enables the use of the the following insecure ciphers and key exchange methods:
- aes128-cbc
- aes192-cbc
- aes256-cbc
- 3des-cbc
- diffie-hellman-group-exchange-sha256
- diffie-hellman-group-exchange-sha1
Those algorithms are insecure and may allow plaintext data to be recovered by an attacker.`,
Default: false,
Examples: []fs.OptionExample{
{
@@ -179,7 +192,7 @@ type Fs struct {
cachedHashes *hash.Set
poolMu sync.Mutex
pool []*conn
connLimit *rate.Limiter // for limiting number of connections per second
pacer *fs.Pacer // pacer for operations
}
// Object is a remote SFTP file that has been stat'd (so it exists, but is not necessarily open for reading)
@@ -259,10 +272,6 @@ func (c *conn) closed() error {
// Open a new connection to the SFTP server.
func (f *Fs) sftpConnection() (c *conn, err error) {
// Rate limit rate of new connections
err = f.connLimit.Wait(context.Background())
if err != nil {
return nil, errors.Wrap(err, "limiter failed in connect")
}
c = &conn{
err: make(chan error, 1),
}
@@ -296,7 +305,14 @@ func (f *Fs) getSftpConnection() (c *conn, err error) {
if c != nil {
return c, nil
}
return f.sftpConnection()
err = f.pacer.Call(func() (bool, error) {
c, err = f.sftpConnection()
if err != nil {
return true, err
}
return false, nil
})
return c, err
}
// Return an SFTP connection to the pool
@@ -363,7 +379,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
if opt.UseInsecureCipher {
sshConfig.Config.SetDefaults()
sshConfig.Config.Ciphers = append(sshConfig.Config.Ciphers, "aes128-cbc")
sshConfig.Config.Ciphers = append(sshConfig.Config.Ciphers, "aes128-cbc", "aes192-cbc", "aes256-cbc", "3des-cbc")
sshConfig.Config.KeyExchanges = append(sshConfig.Config.KeyExchanges, "diffie-hellman-group-exchange-sha1", "diffie-hellman-group-exchange-sha256")
}
@@ -454,7 +470,7 @@ func NewFsWithConnection(ctx context.Context, name string, root string, m config
config: sshConfig,
url: "sftp://" + opt.User + "@" + opt.Host + ":" + opt.Port + "/" + root,
mkdirLock: newStringLock(),
connLimit: rate.NewLimiter(rate.Limit(connectionsPerSecond), 1),
pacer: fs.NewPacer(pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
}
f.features = (&fs.Features{
CanHaveEmptyDirectories: true,
+42 -11
View File
@@ -530,10 +530,10 @@ type listFn func(remote string, object *swift.Object, isDirectory bool) error
//
// Set recurse to read sub directories
func (f *Fs) listContainerRoot(container, directory, prefix string, addContainer bool, recurse bool, fn listFn) error {
if prefix != "" {
if prefix != "" && !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
if directory != "" {
if directory != "" && !strings.HasSuffix(directory, "/") {
directory += "/"
}
// Options for ObjectsWalk
@@ -952,6 +952,18 @@ func (o *Object) isStaticLargeObject() (bool, error) {
return o.hasHeader("X-Static-Large-Object")
}
func (o *Object) isInContainerVersioning() (bool, error) {
_, headers, err := o.fs.c.Container(o.fs.root)
if err != nil {
return false, err
}
xHistoryLocation := headers["X-History-Location"]
if len(xHistoryLocation) > 0 {
return true, nil
}
return false, nil
}
// Size returns the size of an object in bytes
func (o *Object) Size() int64 {
return o.size
@@ -1083,9 +1095,8 @@ func min(x, y int64) int64 {
//
// if except is passed in then segments with that prefix won't be deleted
func (o *Object) removeSegments(except string) error {
container, containerPath := o.split()
segmentsContainer := container + "_segments"
err := o.fs.listContainerRoot(segmentsContainer, containerPath, "", false, true, func(remote string, object *swift.Object, isDirectory bool) error {
segmentsContainer, prefix, err := o.getSegmentsDlo()
err = o.fs.listContainerRoot(segmentsContainer, prefix, "", false, true, func(remote string, object *swift.Object, isDirectory bool) error {
if isDirectory {
return nil
}
@@ -1114,6 +1125,19 @@ func (o *Object) removeSegments(except string) error {
return nil
}
func (o *Object) getSegmentsDlo() (segmentsContainer string, prefix string, err error) {
if err = o.readMetaData(); err != nil {
return
}
dirManifest := o.headers["X-Object-Manifest"]
delimiter := strings.Index(dirManifest, "/")
if len(dirManifest) == 0 || delimiter < 0 {
err = errors.New("Missing or wrong structure of manifest of Dynamic large object")
return
}
return dirManifest[:delimiter], dirManifest[delimiter+1:], nil
}
// urlEncode encodes a string so that it is a valid URL
//
// We don't use any of Go's standard methods as we need `/` not
@@ -1300,12 +1324,9 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
func (o *Object) Remove(ctx context.Context) (err error) {
container, containerPath := o.split()
isDynamicLargeObject, err := o.isDynamicLargeObject()
if err != nil {
return err
}
// Remove file/manifest first
err = o.fs.pacer.Call(func() (bool, error) {
err = o.fs.c.ObjectDelete(container, containerPath)
@@ -1314,12 +1335,22 @@ func (o *Object) Remove(ctx context.Context) error {
if err != nil {
return err
}
isDynamicLargeObject, err := o.isDynamicLargeObject()
if err != nil {
return err
}
// ...then segments if required
if isDynamicLargeObject {
err = o.removeSegments("")
isInContainerVersioning, err := o.isInContainerVersioning()
if err != nil {
return err
}
if !isInContainerVersioning {
err = o.removeSegments("")
if err != nil {
return err
}
}
}
return nil
}
+4 -2
View File
@@ -9,6 +9,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/spf13/cobra"
)
@@ -19,8 +20,9 @@ var (
func init() {
cmd.Root.AddCommand(commandDefinition)
commandDefinition.Flags().BoolVar(&jsonOutput, "json", false, "Format output as JSON")
commandDefinition.Flags().BoolVar(&fullOutput, "full", false, "Full numbers instead of SI units")
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &jsonOutput, "json", "", false, "Format output as JSON")
flags.BoolVarP(cmdFlags, &fullOutput, "full", "", false, "Full numbers instead of SI units")
}
// printValue formats uv to be output
+14 -4
View File
@@ -3,22 +3,32 @@ package authorize
import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/flags"
"github.com/spf13/cobra"
)
var (
noAutoBrowser bool
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &noAutoBrowser, "auth-no-open-browser", "", false, "Do not automatically open auth link in default browser")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "authorize",
Short: `Remote authorization.`,
Long: `
Remote authorization. Used to authorize a remote or headless
rclone from a machine with a browser - use as instructed by
rclone config.`,
rclone config.
Use the --auth-no-open-browser to prevent rclone to open auth
link in default browser automatically.`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(1, 3, command, args)
config.Authorize(args)
config.Authorize(args, noAutoBrowser)
},
}
+9 -7
View File
@@ -8,6 +8,7 @@ import (
"os"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -22,15 +23,16 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().Int64VarP(&head, "head", "", head, "Only print the first N characters.")
commandDefintion.Flags().Int64VarP(&tail, "tail", "", tail, "Only print the last N characters.")
commandDefintion.Flags().Int64VarP(&offset, "offset", "", offset, "Start printing at offset N (or from end if -ve).")
commandDefintion.Flags().Int64VarP(&count, "count", "", count, "Only print N characters.")
commandDefintion.Flags().BoolVarP(&discard, "discard", "", discard, "Discard the output instead of printing.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.Int64VarP(cmdFlags, &head, "head", "", head, "Only print the first N characters.")
flags.Int64VarP(cmdFlags, &tail, "tail", "", tail, "Only print the last N characters.")
flags.Int64VarP(cmdFlags, &offset, "offset", "", offset, "Start printing at offset N (or from end if -ve).")
flags.Int64VarP(cmdFlags, &count, "count", "", count, "Only print N characters.")
flags.BoolVarP(cmdFlags, &discard, "discard", "", discard, "Discard the output instead of printing.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "cat remote:path",
Short: `Concatenates any files and sends them to stdout.`,
Long: `
+6 -4
View File
@@ -4,6 +4,7 @@ import (
"context"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -15,12 +16,13 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&download, "download", "", download, "Check by downloading rather than with hash.")
commandDefintion.Flags().BoolVarP(&oneway, "one-way", "", oneway, "Check one way only, source files must exist on remote")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &download, "download", "", download, "Check by downloading rather than with hash.")
flags.BoolVarP(cmdFlags, &oneway, "one-way", "", oneway, "Check one way only, source files must exist on remote")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "check source:path dest:path",
Short: `Checks the files in the source and destination match.`,
Long: `
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "cleanup remote:path",
Short: `Clean up the remote if possible`,
Long: `
+13 -13
View File
@@ -82,7 +82,7 @@ func ShowVersion() {
func NewFsFile(remote string) (fs.Fs, string) {
_, _, fsPath, err := fs.ParseRemote(remote)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatalf("Failed to create file system for %q: %v", remote, err)
}
f, err := cache.Get(remote)
@@ -92,7 +92,7 @@ func NewFsFile(remote string) (fs.Fs, string) {
case nil:
return f, ""
default:
fs.CountError(err)
err = fs.CountError(err)
log.Fatalf("Failed to create file system for %q: %v", remote, err)
}
return nil, ""
@@ -107,13 +107,13 @@ func newFsFileAddFilter(remote string) (fs.Fs, string) {
if fileName != "" {
if !filter.Active.InActive() {
err := errors.Errorf("Can't limit to single files when using filters: %v", remote)
fs.CountError(err)
err = fs.CountError(err)
log.Fatalf(err.Error())
}
// Limit transfers to this file
err := filter.Active.AddFile(fileName)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatalf("Failed to limit to single file %q: %v", remote, err)
}
}
@@ -135,7 +135,7 @@ func NewFsSrc(args []string) fs.Fs {
func newFsDir(remote string) fs.Fs {
f, err := cache.Get(remote)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatalf("Failed to create file system for %q: %v", remote, err)
}
return f
@@ -189,11 +189,11 @@ func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs
fdst, err := cache.Get(dstRemote)
switch err {
case fs.ErrorIsFile:
fs.CountError(err)
_ = fs.CountError(err)
log.Fatalf("Source doesn't exist or is a directory and destination is a file")
case nil:
default:
fs.CountError(err)
_ = fs.CountError(err)
log.Fatalf("Failed to create file system for destination %q: %v", dstRemote, err)
}
return
@@ -239,7 +239,7 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
SigInfoHandler()
for try := 1; try <= *retries; try++ {
err = f()
fs.CountError(err)
err = fs.CountError(err)
lastErr := accounting.GlobalStats().GetLastError()
if err == nil {
err = lastErr
@@ -386,12 +386,12 @@ func initConfig() {
fs.Infof(nil, "Creating CPU profile %q\n", *cpuProfile)
f, err := os.Create(*cpuProfile)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatal(err)
}
err = pprof.StartCPUProfile(f)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatal(err)
}
atexit.Register(func() {
@@ -405,17 +405,17 @@ func initConfig() {
fs.Infof(nil, "Saving Memory profile %q\n", *memProfile)
f, err := os.Create(*memProfile)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatal(err)
}
err = pprof.WriteHeapProfile(f)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatal(err)
}
err = f.Close()
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
log.Fatal(err)
}
})
+12 -2
View File
@@ -246,7 +246,12 @@ func (fsys *FS) Readdir(dirPath string,
for _, item := range items {
node, ok := item.(vfs.Node)
if ok {
fill(node.Name(), nil, 0)
name := node.Name()
if len(name) > mountlib.MaxLeafSize {
fs.Errorf(dirPath, "Name too long (%d bytes) for FUSE, skipping: %s", len(name), name)
continue
}
fill(name, nil, 0)
}
}
itemsRead = len(items)
@@ -366,7 +371,12 @@ func (fsys *FS) Write(path string, buff []byte, ofst int64, fh uint64) (n int) {
if errc != 0 {
return errc
}
n, err := handle.WriteAt(buff, ofst)
var err error
if fsys.VFS.Opt.CacheMode < vfs.CacheModeWrites || handle.Node().Mode()&os.ModeAppend == 0 {
n, err = handle.WriteAt(buff, ofst)
} else {
n, err = handle.Write(buff)
}
if err != nil {
return translateError(err)
}
+6 -1
View File
@@ -21,6 +21,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd/mountlib"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/lib/atexit"
"github.com/rclone/rclone/vfs"
"github.com/rclone/rclone/vfs/vfsflags"
)
@@ -207,7 +208,7 @@ func mount(f fs.Fs, mountpoint string) (*vfs.VFS, <-chan error, func() error, er
// If noModTime is set then it
func Mount(f fs.Fs, mountpoint string) error {
// Mount it
FS, errChan, _, err := mount(f, mountpoint)
FS, errChan, unmount, err := mount(f, mountpoint)
if err != nil {
return errors.Wrap(err, "failed to mount FUSE fs")
}
@@ -217,6 +218,10 @@ func Mount(f fs.Fs, mountpoint string) error {
sigHup := make(chan os.Signal, 1)
signal.Notify(sigHup, syscall.SIGHUP)
atexit.Register(func() {
_ = unmount()
})
if err := sdnotify.Ready(); err != nil && err != sdnotify.ErrSdNotifyNoSocket {
return errors.Wrap(err, "failed to notify systemd")
}
+2 -1
View File
@@ -11,6 +11,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/rc"
"github.com/spf13/cobra"
)
@@ -271,7 +272,7 @@ var (
)
func init() {
configUserInfoCommand.Flags().BoolVar(&jsonOutput, "json", false, "Format output as JSON")
flags.BoolVarP(configUserInfoCommand.Flags(), &jsonOutput, "json", "", false, "Format output as JSON")
}
var configUserInfoCommand = &cobra.Command{
+5 -3
View File
@@ -4,6 +4,7 @@ import (
"context"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/sync"
"github.com/spf13/cobra"
@@ -14,11 +15,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&createEmptySrcDirs, "create-empty-src-dirs", "", createEmptySrcDirs, "Create empty source dirs on destination after copy")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &createEmptySrcDirs, "create-empty-src-dirs", "", createEmptySrcDirs, "Create empty source dirs on destination after copy")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "copy source:path dest:path",
Short: `Copy files from source to dest, skipping already copied`,
Long: `
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "copyto source:path dest:path",
Short: `Copy files from source to dest, skipping already copied`,
Long: `
+5 -3
View File
@@ -5,6 +5,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -14,11 +15,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&autoFilename, "auto-filename", "a", autoFilename, "Get the file name from the url and use it for destination file path")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &autoFilename, "auto-filename", "a", autoFilename, "Get the file name from the url and use it for destination file path")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "copyurl https://example.com dest:path",
Short: `Copy url content to dest.`,
Long: `
+8 -6
View File
@@ -7,6 +7,7 @@ import (
"github.com/rclone/rclone/backend/crypt"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
@@ -18,11 +19,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&oneway, "one-way", "", oneway, "Check one way only, source files must exist on destination")
cmd.Root.AddCommand(commandDefinition)
cmdFlag := commandDefinition.Flags()
flags.BoolVarP(cmdFlag, &oneway, "one-way", "", oneway, "Check one way only, source files must exist on destination")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "cryptcheck remote:path cryptedremote:path",
Short: `Cryptcheck checks the integrity of a crypted remote.`,
Long: `
@@ -86,7 +88,7 @@ func cryptCheck(ctx context.Context, fdst, fsrc fs.Fs) error {
underlyingDst := cryptDst.UnWrap()
underlyingHash, err := underlyingDst.Hash(ctx, hashType)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
fs.Errorf(dst, "Error reading hash from underlying %v: %v", underlyingDst, err)
return true, false
}
@@ -95,7 +97,7 @@ func cryptCheck(ctx context.Context, fdst, fsrc fs.Fs) error {
}
cryptHash, err := fcrypt.ComputeHash(ctx, cryptDst, src, hashType)
if err != nil {
fs.CountError(err)
err = fs.CountError(err)
fs.Errorf(dst, "Error computing hash: %v", err)
return true, false
}
@@ -104,7 +106,7 @@ func cryptCheck(ctx context.Context, fdst, fsrc fs.Fs) error {
}
if cryptHash != underlyingHash {
err = errors.Errorf("hashes differ (%s:%s) %q vs (%s:%s) %q", fdst.Name(), fdst.Root(), cryptHash, fsrc.Name(), fsrc.Root(), underlyingHash)
fs.CountError(err)
err = fs.CountError(err)
fs.Errorf(src, err.Error())
return true, false
}
+2 -2
View File
@@ -18,8 +18,8 @@ var (
func init() {
cmd.Root.AddCommand(commandDefinition)
flagSet := commandDefinition.Flags()
flags.BoolVarP(flagSet, &Reverse, "reverse", "", Reverse, "Reverse cryptdecode, encrypts filenames")
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &Reverse, "reverse", "", Reverse, "Reverse cryptdecode, encrypts filenames")
}
var commandDefinition = &cobra.Command{
+2 -2
View File
@@ -12,10 +12,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "dbhashsum remote:path",
Short: `Produces a Dropbox hash file for all the objects in the path.`,
Long: `
+5 -3
View File
@@ -5,6 +5,7 @@ import (
"log"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -14,11 +15,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().VarP(&dedupeMode, "dedupe-mode", "", "Dedupe mode interactive|skip|first|newest|oldest|rename.")
cmd.Root.AddCommand(commandDefinition)
cmdFlag := commandDefinition.Flags()
flags.FVarP(cmdFlag, &dedupeMode, "dedupe-mode", "", "Dedupe mode interactive|skip|first|newest|oldest|rename.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "dedupe [mode] remote:path",
Short: `Interactively find duplicate files and delete/rename them.`,
Long: `
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "delete remote:path",
Short: `Remove the contents of path.`,
Long: `
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "deletefile remote:path",
Short: `Remove a single file from remote.`,
Long: `
+2 -2
View File
@@ -17,7 +17,7 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
const gendocFrontmatterTemplate = `---
@@ -28,7 +28,7 @@ url: %s
---
`
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "gendocs output_directory",
Short: `Output markdown docs for rclone to the directory supplied.`,
Long: `
+10 -8
View File
@@ -22,6 +22,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/cmd/info/internal"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/lib/random"
@@ -41,16 +42,17 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().StringVarP(&writeJSON, "write-json", "", "", "Write results to file.")
commandDefintion.Flags().BoolVarP(&checkNormalization, "check-normalization", "", true, "Check UTF-8 Normalization.")
commandDefintion.Flags().BoolVarP(&checkControl, "check-control", "", true, "Check control characters.")
commandDefintion.Flags().DurationVarP(&uploadWait, "upload-wait", "", 0, "Wait after writing a file.")
commandDefintion.Flags().BoolVarP(&checkLength, "check-length", "", true, "Check max filename length.")
commandDefintion.Flags().BoolVarP(&checkStreaming, "check-streaming", "", true, "Check uploads with indeterminate file size.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.StringVarP(cmdFlags, &writeJSON, "write-json", "", "", "Write results to file.")
flags.BoolVarP(cmdFlags, &checkNormalization, "check-normalization", "", true, "Check UTF-8 Normalization.")
flags.BoolVarP(cmdFlags, &checkControl, "check-control", "", true, "Check control characters.")
flags.DurationVarP(cmdFlags, &uploadWait, "upload-wait", "", 0, "Wait after writing a file.")
flags.BoolVarP(cmdFlags, &checkLength, "check-length", "", true, "Check max filename length.")
flags.BoolVarP(cmdFlags, &checkStreaming, "check-streaming", "", true, "Check uploadxs with indeterminate file size.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "info [remote:path]+",
Short: `Discovers file name or other limitations for paths.`,
Long: `rclone info discovers what filenames and upload methods are possible
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "link remote:path",
Short: `Generate public link to file/folder.`,
Long: `
+5 -3
View File
@@ -6,6 +6,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config"
"github.com/rclone/rclone/fs/config/flags"
"github.com/spf13/cobra"
)
@@ -15,11 +16,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&listLong, "long", "", listLong, "Show the type as well as names.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &listLong, "long", "", listLong, "Show the type as well as names.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "listremotes",
Short: `List all the remotes in the config file.`,
Long: `
+2 -2
View File
@@ -11,10 +11,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "ls remote:path",
Short: `List the objects in the path with size and path.`,
Long: `
+5 -3
View File
@@ -7,6 +7,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/cmd/ls/lshelp"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -16,11 +17,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&recurse, "recursive", "R", false, "Recurse into the listing.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &recurse, "recursive", "R", false, "Recurse into the listing.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "lsd remote:path",
Short: `List all directories/containers/buckets in the path.`,
Long: `
+13 -12
View File
@@ -10,6 +10,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/cmd/ls/lshelp"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
@@ -28,20 +29,20 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
flags := commandDefintion.Flags()
flags.StringVarP(&format, "format", "F", "p", "Output format - see help for details")
flags.StringVarP(&separator, "separator", "s", ";", "Separator for the items in the format.")
flags.BoolVarP(&dirSlash, "dir-slash", "d", true, "Append a slash to directory names.")
flags.VarP(&hashType, "hash", "", "Use this hash when `h` is used in the format MD5|SHA-1|DropboxHash")
flags.BoolVarP(&filesOnly, "files-only", "", false, "Only list files.")
flags.BoolVarP(&dirsOnly, "dirs-only", "", false, "Only list directories.")
flags.BoolVarP(&csv, "csv", "", false, "Output in CSV format.")
flags.BoolVarP(&absolute, "absolute", "", false, "Put a leading / in front of path names.")
commandDefintion.Flags().BoolVarP(&recurse, "recursive", "R", false, "Recurse into the listing.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.StringVarP(cmdFlags, &format, "format", "F", "p", "Output format - see help for details")
flags.StringVarP(cmdFlags, &separator, "separator", "s", ";", "Separator for the items in the format.")
flags.BoolVarP(cmdFlags, &dirSlash, "dir-slash", "d", true, "Append a slash to directory names.")
flags.FVarP(cmdFlags, &hashType, "hash", "", "Use this hash when `h` is used in the format MD5|SHA-1|DropboxHash")
flags.BoolVarP(cmdFlags, &filesOnly, "files-only", "", false, "Only list files.")
flags.BoolVarP(cmdFlags, &dirsOnly, "dirs-only", "", false, "Only list directories.")
flags.BoolVarP(cmdFlags, &csv, "csv", "", false, "Output in CSV format.")
flags.BoolVarP(cmdFlags, &absolute, "absolute", "", false, "Put a leading / in front of path names.")
flags.BoolVarP(cmdFlags, &recurse, "recursive", "R", false, "Recurse into the listing.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "lsf remote:path",
Short: `List directories and objects in remote:path formatted for parsing`,
Long: `
+11 -9
View File
@@ -9,6 +9,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/cmd/ls/lshelp"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -18,17 +19,18 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&opt.Recurse, "recursive", "R", false, "Recurse into the listing.")
commandDefintion.Flags().BoolVarP(&opt.ShowHash, "hash", "", false, "Include hashes in the output (may take longer).")
commandDefintion.Flags().BoolVarP(&opt.NoModTime, "no-modtime", "", false, "Don't read the modification time (can speed things up).")
commandDefintion.Flags().BoolVarP(&opt.ShowEncrypted, "encrypted", "M", false, "Show the encrypted names.")
commandDefintion.Flags().BoolVarP(&opt.ShowOrigIDs, "original", "", false, "Show the ID of the underlying Object.")
commandDefintion.Flags().BoolVarP(&opt.FilesOnly, "files-only", "", false, "Show only files in the listing.")
commandDefintion.Flags().BoolVarP(&opt.DirsOnly, "dirs-only", "", false, "Show only directories in the listing.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &opt.Recurse, "recursive", "R", false, "Recurse into the listing.")
flags.BoolVarP(cmdFlags, &opt.ShowHash, "hash", "", false, "Include hashes in the output (may take longer).")
flags.BoolVarP(cmdFlags, &opt.NoModTime, "no-modtime", "", false, "Don't read the modification time (can speed things up).")
flags.BoolVarP(cmdFlags, &opt.ShowEncrypted, "encrypted", "M", false, "Show the encrypted names.")
flags.BoolVarP(cmdFlags, &opt.ShowOrigIDs, "original", "", false, "Show the ID of the underlying Object.")
flags.BoolVarP(cmdFlags, &opt.FilesOnly, "files-only", "", false, "Show only files in the listing.")
flags.BoolVarP(cmdFlags, &opt.DirsOnly, "dirs-only", "", false, "Show only directories in the listing.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "lsjson remote:path",
Short: `List directories and objects in the path in JSON format.`,
Long: `List directories and objects in the path in JSON format.
+2 -2
View File
@@ -11,10 +11,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "lsl remote:path",
Short: `List the objects in path with modification time, size and path.`,
Long: `
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "md5sum remote:path",
Short: `Produces an md5sum file for all the objects in the path.`,
Long: `
+2 -2
View File
@@ -12,10 +12,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "memtest remote:path",
Short: `Load all the objects at remote:path and report memory stats.`,
Hidden: true,
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "mkdir remote:path",
Short: `Make the path if it doesn't already exist.`,
Run: func(command *cobra.Command, args []string) {
+7 -1
View File
@@ -11,6 +11,7 @@ import (
fusefs "bazil.org/fuse/fs"
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd/mountlib"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/vfs"
)
@@ -96,10 +97,15 @@ func (d *Dir) ReadDirAll(ctx context.Context) (dirents []fuse.Dirent, err error)
return nil, translateError(err)
}
for _, node := range items {
name := node.Name()
if len(name) >= mountlib.MaxLeafSize {
fs.Errorf(d, "Name too long (%d bytes) for FUSE, skipping: %s", len(name), name)
continue
}
var dirent = fuse.Dirent{
// Inode FIXME ???
Type: fuse.DT_File,
Name: node.Name(),
Name: name,
}
if node.IsDir() {
dirent.Type = fuse.DT_Dir
+7 -1
View File
@@ -5,6 +5,7 @@ package mount
import (
"context"
"io"
"os"
"bazil.org/fuse"
fusefs "bazil.org/fuse/fs"
@@ -41,7 +42,12 @@ var _ fusefs.HandleWriter = (*FileHandle)(nil)
// Write data to the file handle
func (fh *FileHandle) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) (err error) {
defer log.Trace(fh, "len=%d, offset=%d", len(req.Data), req.Offset)("written=%d, err=%v", &resp.Size, &err)
n, err := fh.Handle.WriteAt(req.Data, req.Offset)
var n int
if fh.Handle.Node().VFS().Opt.CacheMode < vfs.CacheModeWrites || fh.Handle.Node().Mode()&os.ModeAppend == 0 {
n, err = fh.Handle.WriteAt(req.Data, req.Offset)
} else {
n, err = fh.Handle.Write(req.Data)
}
if err != nil {
return translateError(err)
}
+4 -3
View File
@@ -32,12 +32,10 @@ func mountOptions(device string) (options []fuse.MountOption) {
fuse.Subtype("rclone"),
fuse.FSName(device),
fuse.VolumeName(mountlib.VolumeName),
fuse.AsyncRead(),
// Options from benchmarking in the fuse module
//fuse.MaxReadahead(64 * 1024 * 1024),
//fuse.AsyncRead(), - FIXME this causes
// ReadFileHandle.Read error: read /home/files/ISOs/xubuntu-15.10-desktop-amd64.iso: bad file descriptor
// which is probably related to errors people are having
//fuse.WritebackCache(),
}
if mountlib.NoAppleDouble {
@@ -139,6 +137,9 @@ func Mount(f fs.Fs, mountpoint string) error {
sigHup := make(chan os.Signal, 1)
signal.Notify(sigHup, syscall.SIGHUP)
atexit.IgnoreSignals()
atexit.Register(func() {
_ = unmount()
})
if err := sdnotify.Ready(); err != nil && err != sdnotify.ErrSdNotifyNoSocket {
return errors.Wrap(err, "failed to notify systemd")
+25 -20
View File
@@ -38,6 +38,11 @@ var (
DaemonTimeout time.Duration // OSXFUSE only
)
// Global constants
const (
MaxLeafSize = 4095 // don't pass file names longer than this
)
func init() {
// DaemonTimeout defaults to non zero for macOS
if runtime.GOOS == "darwin" {
@@ -94,7 +99,7 @@ func checkMountpointOverlap(root, mountpoint string) error {
// NewMountCommand makes a mount command with the given name and Mount function
func NewMountCommand(commandName string, Mount func(f fs.Fs, mountpoint string) error) *cobra.Command {
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: commandName + " remote:path /path/to/mountpoint",
Short: `Mount the remote as file system on a mountpoint.`,
Long: `
@@ -295,34 +300,34 @@ be copied to the vfs cache before opening with --vfs-cache-mode full.
}
// Register the command
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
// Add flags
flagSet := commandDefintion.Flags()
flags.BoolVarP(flagSet, &DebugFUSE, "debug-fuse", "", DebugFUSE, "Debug the FUSE internals - needs -v.")
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &DebugFUSE, "debug-fuse", "", DebugFUSE, "Debug the FUSE internals - needs -v.")
// mount options
flags.BoolVarP(flagSet, &AllowNonEmpty, "allow-non-empty", "", AllowNonEmpty, "Allow mounting over a non-empty directory.")
flags.BoolVarP(flagSet, &AllowRoot, "allow-root", "", AllowRoot, "Allow access to root user.")
flags.BoolVarP(flagSet, &AllowOther, "allow-other", "", AllowOther, "Allow access to other users.")
flags.BoolVarP(flagSet, &DefaultPermissions, "default-permissions", "", DefaultPermissions, "Makes kernel enforce access control based on the file mode.")
flags.BoolVarP(flagSet, &WritebackCache, "write-back-cache", "", WritebackCache, "Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.")
flags.FVarP(flagSet, &MaxReadAhead, "max-read-ahead", "", "The number of bytes that can be prefetched for sequential reads.")
flags.DurationVarP(flagSet, &AttrTimeout, "attr-timeout", "", AttrTimeout, "Time for which file/directory attributes are cached.")
flags.StringArrayVarP(flagSet, &ExtraOptions, "option", "o", []string{}, "Option for libfuse/WinFsp. Repeat if required.")
flags.StringArrayVarP(flagSet, &ExtraFlags, "fuse-flag", "", []string{}, "Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required.")
flags.BoolVarP(flagSet, &Daemon, "daemon", "", Daemon, "Run mount as a daemon (background mode).")
flags.StringVarP(flagSet, &VolumeName, "volname", "", VolumeName, "Set the volume name (not supported by all OSes).")
flags.DurationVarP(flagSet, &DaemonTimeout, "daemon-timeout", "", DaemonTimeout, "Time limit for rclone to respond to kernel (not supported by all OSes).")
flags.BoolVarP(cmdFlags, &AllowNonEmpty, "allow-non-empty", "", AllowNonEmpty, "Allow mounting over a non-empty directory.")
flags.BoolVarP(cmdFlags, &AllowRoot, "allow-root", "", AllowRoot, "Allow access to root user.")
flags.BoolVarP(cmdFlags, &AllowOther, "allow-other", "", AllowOther, "Allow access to other users.")
flags.BoolVarP(cmdFlags, &DefaultPermissions, "default-permissions", "", DefaultPermissions, "Makes kernel enforce access control based on the file mode.")
flags.BoolVarP(cmdFlags, &WritebackCache, "write-back-cache", "", WritebackCache, "Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.")
flags.FVarP(cmdFlags, &MaxReadAhead, "max-read-ahead", "", "The number of bytes that can be prefetched for sequential reads.")
flags.DurationVarP(cmdFlags, &AttrTimeout, "attr-timeout", "", AttrTimeout, "Time for which file/directory attributes are cached.")
flags.StringArrayVarP(cmdFlags, &ExtraOptions, "option", "o", []string{}, "Option for libfuse/WinFsp. Repeat if required.")
flags.StringArrayVarP(cmdFlags, &ExtraFlags, "fuse-flag", "", []string{}, "Flags or arguments to be passed direct to libfuse/WinFsp. Repeat if required.")
flags.BoolVarP(cmdFlags, &Daemon, "daemon", "", Daemon, "Run mount as a daemon (background mode).")
flags.StringVarP(cmdFlags, &VolumeName, "volname", "", VolumeName, "Set the volume name (not supported by all OSes).")
flags.DurationVarP(cmdFlags, &DaemonTimeout, "daemon-timeout", "", DaemonTimeout, "Time limit for rclone to respond to kernel (not supported by all OSes).")
if runtime.GOOS == "darwin" {
flags.BoolVarP(flagSet, &NoAppleDouble, "noappledouble", "", NoAppleDouble, "Sets the OSXFUSE option noappledouble.")
flags.BoolVarP(flagSet, &NoAppleXattr, "noapplexattr", "", NoAppleXattr, "Sets the OSXFUSE option noapplexattr.")
flags.BoolVarP(cmdFlags, &NoAppleDouble, "noappledouble", "", NoAppleDouble, "Sets the OSXFUSE option noappledouble.")
flags.BoolVarP(cmdFlags, &NoAppleXattr, "noapplexattr", "", NoAppleXattr, "Sets the OSXFUSE option noapplexattr.")
}
// Add in the generic flags
vfsflags.AddFlags(flagSet)
vfsflags.AddFlags(cmdFlags)
return commandDefintion
return commandDefinition
}
// ClipBlocks clips the blocks pointed to to the OS max
+2
View File
@@ -50,6 +50,8 @@ func TestRenameOpenHandle(t *testing.T) {
err = file.Close()
require.NoError(t, err)
run.waitForWriters()
// verify file was renamed properly
run.checkDir(t, "renamebla 9")
+5
View File
@@ -34,6 +34,11 @@ func osCreate(name string) (*os.File, error) {
return os.OpenFile(name, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0666)
}
// os.Create with append
func osAppend(name string) (*os.File, error) {
return os.OpenFile(name, os.O_WRONLY|os.O_APPEND, 0666)
}
// TestFileModTimeWithOpenWriters tests mod time on open files
func TestFileModTimeWithOpenWriters(t *testing.T) {
run.skipIfNoFUSE(t)
+1
View File
@@ -78,6 +78,7 @@ func RunTests(t *testing.T, fn MountFn) {
t.Run("TestWriteFileDoubleClose", TestWriteFileDoubleClose)
t.Run("TestWriteFileFsync", TestWriteFileFsync)
t.Run("TestWriteFileDup", TestWriteFileDup)
t.Run("TestWriteFileAppend", TestWriteFileAppend)
})
log.Printf("Finished test run with cache mode %v (ok=%v)", cacheMode, ok)
if !ok {
+46
View File
@@ -2,6 +2,7 @@ package mounttest
import (
"os"
"runtime"
"testing"
"github.com/stretchr/testify/assert"
@@ -130,3 +131,48 @@ func TestWriteFileDup(t *testing.T) {
run.waitForWriters()
run.rm(t, "to be synced")
}
// TestWriteFileAppend tests that O_APPEND works on cache backends >= writes
func TestWriteFileAppend(t *testing.T) {
run.skipIfNoFUSE(t)
if run.vfs.Opt.CacheMode < vfs.CacheModeWrites {
t.Skip("not supported on vfs-cache-mode < writes")
return
}
// TODO: Windows needs the v1.5 release of WinFsp to handle O_APPEND properly.
// Until it gets released, skip this test on Windows.
if runtime.GOOS == "windows" {
t.Skip("currently unsupported on Windows")
}
filepath := run.path("to be synced")
fh, err := osCreate(filepath)
require.NoError(t, err)
testData := []byte("0123456789")
appendData := []byte("10")
_, err = fh.Write(testData)
require.NoError(t, err)
err = fh.Close()
require.NoError(t, err)
fh, err = osAppend(filepath)
require.NoError(t, err)
_, err = fh.Write(appendData)
require.NoError(t, err)
err = fh.Close()
require.NoError(t, err)
info, err := os.Stat(filepath)
require.NoError(t, err)
require.EqualValues(t, len(testData)+len(appendData), info.Size())
run.waitForWriters()
run.rm(t, "to be synced")
}
+6 -4
View File
@@ -4,6 +4,7 @@ import (
"context"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/sync"
"github.com/spf13/cobra"
@@ -16,12 +17,13 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&deleteEmptySrcDirs, "delete-empty-src-dirs", "", deleteEmptySrcDirs, "Delete empty source dirs after move")
commandDefintion.Flags().BoolVarP(&createEmptySrcDirs, "create-empty-src-dirs", "", createEmptySrcDirs, "Create empty source dirs on destination after move")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &deleteEmptySrcDirs, "delete-empty-src-dirs", "", deleteEmptySrcDirs, "Delete empty source dirs after move")
flags.BoolVarP(cmdFlags, &createEmptySrcDirs, "create-empty-src-dirs", "", createEmptySrcDirs, "Create empty source dirs on destination after move")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "move source:path dest:path",
Short: `Move files from source to dest.`,
Long: `
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "moveto source:path dest:path",
Short: `Move file or directory from source to dest.`,
Long: `
+2 -2
View File
@@ -24,10 +24,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "ncdu remote:path",
Short: `Explore a remote with a text based user interface.`,
Long: `
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "obscure password",
Short: `Obscure password for use in the rclone.conf`,
Run: func(command *cobra.Command, args []string) {
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "purge remote:path",
Short: `Remove the path and all of its contents.`,
Long: `
+10 -8
View File
@@ -13,6 +13,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/rc"
"github.com/spf13/cobra"
@@ -29,16 +30,17 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&noOutput, "no-output", "", noOutput, "If set don't output the JSON result.")
commandDefintion.Flags().StringVarP(&url, "url", "", url, "URL to connect to rclone remote control.")
commandDefintion.Flags().StringVarP(&jsonInput, "json", "", jsonInput, "Input JSON - use instead of key=value args.")
commandDefintion.Flags().StringVarP(&authUser, "user", "", "", "Username to use to rclone remote control.")
commandDefintion.Flags().StringVarP(&authPass, "pass", "", "", "Password to use to connect to rclone remote control.")
commandDefintion.Flags().BoolVarP(&loopback, "loopback", "", false, "If set connect to this rclone instance not via HTTP.")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &noOutput, "no-output", "", noOutput, "If set don't output the JSON result.")
flags.StringVarP(cmdFlags, &url, "url", "", url, "URL to connect to rclone remote control.")
flags.StringVarP(cmdFlags, &jsonInput, "json", "", jsonInput, "Input JSON - use instead of key=value args.")
flags.StringVarP(cmdFlags, &authUser, "user", "", "", "Username to use to rclone remote control.")
flags.StringVarP(cmdFlags, &authPass, "pass", "", "", "Password to use to connect to rclone remote control.")
flags.BoolVarP(cmdFlags, &loopback, "loopback", "", false, "If set connect to this rclone instance not via HTTP.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "rc commands parameter",
Short: `Run a command against a running rclone.`,
Long: `
+2 -2
View File
@@ -12,10 +12,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "rcat remote:path",
Short: `Copies standard input to file on remote.`,
Long: `
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "reveal password",
Short: `Reveal obscured password from rclone.conf`,
Run: func(command *cobra.Command, args []string) {
+2 -2
View File
@@ -9,10 +9,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "rmdir remote:path",
Short: `Remove the path if empty.`,
Long: `
+1 -1
View File
@@ -214,7 +214,7 @@ func withHeader(name string, value string, next http.Handler) http.Handler {
// serveError returns an http.StatusInternalServerError and logs the error
func serveError(what interface{}, w http.ResponseWriter, text string, err error) {
fs.CountError(err)
err = fs.CountError(err)
fs.Errorf(what, "%s: %v", text, err)
http.Error(w, text+".", http.StatusInternalServerError)
}
+4 -4
View File
@@ -15,7 +15,6 @@ import (
"strconv"
"sync"
ftp "github.com/goftp/server"
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/cmd/serve/proxy"
@@ -29,6 +28,7 @@ import (
"github.com/rclone/rclone/vfs/vfsflags"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
ftp "goftp.io/server"
)
// Options contains options for the http Server
@@ -155,7 +155,7 @@ func newServer(f fs.Fs, opt *Options) (*server, error) {
PassivePorts: opt.PassivePorts,
Auth: s, // implemented by CheckPasswd method
Logger: &Logger{},
//TODO implement a maximum of https://godoc.org/github.com/goftp/server#ServerOpts
//TODO implement a maximum of https://godoc.org/goftp.io/server#ServerOpts
}
s.srv = ftp.NewServer(ftpopt)
return s, nil
@@ -210,8 +210,8 @@ func (l *Logger) PrintResponse(sessionID string, code int, message string) {
// CheckPassword is called with the connection.
func findID(callerName []byte) (string, error) {
// Dump the stack in this format
// github.com/rclone/rclone/vendor/github.com/goftp/server.(*Conn).Serve(0xc0000b2680)
// /home/ncw/go/src/github.com/rclone/rclone/vendor/github.com/goftp/server/conn.go:116 +0x11d
// github.com/rclone/rclone/vendor/goftp.io/server.(*Conn).Serve(0xc0000b2680)
// /home/ncw/go/src/github.com/rclone/rclone/vendor/goftp.io/server/conn.go:116 +0x11d
buf := make([]byte, 4096)
n := runtime.Stack(buf, false)
buf = buf[:n]
+1 -1
View File
@@ -11,7 +11,6 @@ import (
"fmt"
"testing"
ftp "github.com/goftp/server"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/cmd/serve/servetest"
"github.com/rclone/rclone/fs"
@@ -19,6 +18,7 @@ import (
"github.com/rclone/rclone/fs/config/obscure"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
ftp "goftp.io/server"
)
const (
+1 -1
View File
@@ -68,7 +68,7 @@ func (d *Directory) AddEntry(remote string, isDir bool) {
// Error logs the error and if a ResponseWriter is given it writes a http.StatusInternalServerError
func Error(what interface{}, w http.ResponseWriter, text string, err error) {
fs.CountError(err)
err = fs.CountError(err)
fs.Errorf(what, "%s: %v", text, err)
if w != nil {
http.Error(w, text+".", http.StatusInternalServerError)
+4 -1
View File
@@ -208,7 +208,10 @@ func (p *Proxy) call(user, pass string, passwordBytes []byte) (value interface{}
if err != nil {
return nil, false, err
}
pwHash, err := bcrypt.GenerateFromPassword(passwordBytes, bcrypt.DefaultCost)
// The bcrypt cost is a compromise between security and speed. The password is looked up on every
// transaction for WebDAV so we store it lightly hashed. An attacker would find it easier to go after
// the unencrypted password in memory most likely.
pwHash, err := bcrypt.GenerateFromPassword(passwordBytes, bcrypt.MinCost)
if err != nil {
return nil, false, err
}
+1 -1
View File
@@ -271,7 +271,7 @@ func (s *server) postObject(w http.ResponseWriter, r *http.Request, remote strin
_, err := operations.RcatSize(r.Context(), s.f, remote, r.Body, r.ContentLength, time.Now())
if err != nil {
accounting.Stats(r.Context()).Error(err)
err = accounting.Stats(r.Context()).Error(err)
fs.Errorf(remote, "Post request rcat error: %v", err)
http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError)
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "settier tier remote:path",
Short: `Changes storage class/tier of objects in remote.`,
Long: `
+2 -2
View File
@@ -10,10 +10,10 @@ import (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
cmd.Root.AddCommand(commandDefinition)
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "sha1sum remote:path",
Short: `Produces an sha1sum file for all the objects in the path.`,
Long: `
+3 -1
View File
@@ -8,6 +8,7 @@ import (
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
@@ -16,7 +17,8 @@ var jsonOutput bool
func init() {
cmd.Root.AddCommand(commandDefinition)
commandDefinition.Flags().BoolVar(&jsonOutput, "json", false, "format output as JSON")
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &jsonOutput, "json", "", false, "format output as JSON")
}
var commandDefinition = &cobra.Command{
+5 -3
View File
@@ -4,6 +4,7 @@ import (
"context"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/sync"
"github.com/spf13/cobra"
@@ -14,11 +15,12 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&createEmptySrcDirs, "create-empty-src-dirs", "", createEmptySrcDirs, "Create empty source dirs on destination after sync")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &createEmptySrcDirs, "create-empty-src-dirs", "", createEmptySrcDirs, "Create empty source dirs on destination after sync")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "sync source:path dest:path",
Short: `Make source and dest identical, modifying destination only.`,
Long: `
+6 -5
View File
@@ -8,6 +8,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/object"
"github.com/spf13/cobra"
)
@@ -21,13 +22,13 @@ const defaultLayout string = "060102"
const layoutDateWithTime = "2006-01-02T15:04:05"
func init() {
cmd.Root.AddCommand(commandDefintion)
flags := commandDefintion.Flags()
flags.BoolVarP(&notCreateNewFile, "no-create", "C", false, "Do not create the file if it does not exist.")
flags.StringVarP(&timeAsArgument, "timestamp", "t", "", "Change the modification times to the specified time instead of the current time of day. The argument is of the form 'YYMMDD' (ex. 17.10.30) or 'YYYY-MM-DDTHH:MM:SS' (ex. 2006-01-02T15:04:05)")
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &notCreateNewFile, "no-create", "C", false, "Do not create the file if it does not exist.")
flags.StringVarP(cmdFlags, &timeAsArgument, "timestamp", "t", "", "Change the modification times to the specified time instead of the current time of day. The argument is of the form 'YYMMDD' (ex. 17.10.30) or 'YYYY-MM-DDTHH:MM:SS' (ex. 2006-01-02T15:04:05)")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "touch remote:path",
Short: `Create new file or change file modification time.`,
Run: func(command *cobra.Command, args []string) {
+32 -31
View File
@@ -14,6 +14,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/dirtree"
"github.com/rclone/rclone/fs/log"
"github.com/rclone/rclone/fs/walk"
@@ -28,43 +29,43 @@ var (
)
func init() {
cmd.Root.AddCommand(commandDefintion)
flags := commandDefintion.Flags()
cmd.Root.AddCommand(commandDefinition)
cmdFlags := commandDefinition.Flags()
// List
flags.BoolVarP(&opts.All, "all", "a", false, "All files are listed (list . files too).")
flags.BoolVarP(&opts.DirsOnly, "dirs-only", "d", false, "List directories only.")
flags.BoolVarP(&opts.FullPath, "full-path", "", false, "Print the full path prefix for each file.")
//flags.BoolVarP(&opts.IgnoreCase, "ignore-case", "", false, "Ignore case when pattern matching.")
flags.BoolVarP(&noReport, "noreport", "", false, "Turn off file/directory count at end of tree listing.")
// flags.BoolVarP(&opts.FollowLink, "follow", "l", false, "Follow symbolic links like directories.")
flags.IntVarP(&opts.DeepLevel, "level", "", 0, "Descend only level directories deep.")
// flags.StringVarP(&opts.Pattern, "pattern", "P", "", "List only those files that match the pattern given.")
// flags.StringVarP(&opts.IPattern, "exclude", "", "", "Do not list files that match the given pattern.")
flags.StringVarP(&outFileName, "output", "o", "", "Output to file instead of stdout.")
flags.BoolVarP(cmdFlags, &opts.All, "all", "a", false, "All files are listed (list . files too).")
flags.BoolVarP(cmdFlags, &opts.DirsOnly, "dirs-only", "d", false, "List directories only.")
flags.BoolVarP(cmdFlags, &opts.FullPath, "full-path", "", false, "Print the full path prefix for each file.")
//flags.BoolVarP(cmdFlags, &opts.IgnoreCase, "ignore-case", "", false, "Ignore case when pattern matching.")
flags.BoolVarP(cmdFlags, &noReport, "noreport", "", false, "Turn off file/directory count at end of tree listing.")
// flags.BoolVarP(cmdFlags, &opts.FollowLink, "follow", "l", false, "Follow symbolic links like directories.")
flags.IntVarP(cmdFlags, &opts.DeepLevel, "level", "", 0, "Descend only level directories deep.")
// flags.StringVarP(cmdFlags, &opts.Pattern, "pattern", "P", "", "List only those files that match the pattern given.")
// flags.StringVarP(cmdFlags, &opts.IPattern, "exclude", "", "", "Do not list files that match the given pattern.")
flags.StringVarP(cmdFlags, &outFileName, "output", "o", "", "Output to file instead of stdout.")
// Files
flags.BoolVarP(&opts.ByteSize, "size", "s", false, "Print the size in bytes of each file.")
flags.BoolVarP(&opts.UnitSize, "human", "", false, "Print the size in a more human readable way.")
flags.BoolVarP(&opts.FileMode, "protections", "p", false, "Print the protections for each file.")
// flags.BoolVarP(&opts.ShowUid, "uid", "", false, "Displays file owner or UID number.")
// flags.BoolVarP(&opts.ShowGid, "gid", "", false, "Displays file group owner or GID number.")
flags.BoolVarP(&opts.Quotes, "quote", "Q", false, "Quote filenames with double quotes.")
flags.BoolVarP(&opts.LastMod, "modtime", "D", false, "Print the date of last modification.")
// flags.BoolVarP(&opts.Inodes, "inodes", "", false, "Print inode number of each file.")
// flags.BoolVarP(&opts.Device, "device", "", false, "Print device ID number to which each file belongs.")
flags.BoolVarP(cmdFlags, &opts.ByteSize, "size", "s", false, "Print the size in bytes of each file.")
flags.BoolVarP(cmdFlags, &opts.UnitSize, "human", "", false, "Print the size in a more human readable way.")
flags.BoolVarP(cmdFlags, &opts.FileMode, "protections", "p", false, "Print the protections for each file.")
// flags.BoolVarP(cmdFlags, &opts.ShowUid, "uid", "", false, "Displays file owner or UID number.")
// flags.BoolVarP(cmdFlags, &opts.ShowGid, "gid", "", false, "Displays file group owner or GID number.")
flags.BoolVarP(cmdFlags, &opts.Quotes, "quote", "Q", false, "Quote filenames with double quotes.")
flags.BoolVarP(cmdFlags, &opts.LastMod, "modtime", "D", false, "Print the date of last modification.")
// flags.BoolVarP(cmdFlags, &opts.Inodes, "inodes", "", false, "Print inode number of each file.")
// flags.BoolVarP(cmdFlags, &opts.Device, "device", "", false, "Print device ID number to which each file belongs.")
// Sort
flags.BoolVarP(&opts.NoSort, "unsorted", "U", false, "Leave files unsorted.")
flags.BoolVarP(&opts.VerSort, "version", "", false, "Sort files alphanumerically by version.")
flags.BoolVarP(&opts.ModSort, "sort-modtime", "t", false, "Sort files by last modification time.")
flags.BoolVarP(&opts.CTimeSort, "sort-ctime", "", false, "Sort files by last status change time.")
flags.BoolVarP(&opts.ReverSort, "sort-reverse", "r", false, "Reverse the order of the sort.")
flags.BoolVarP(&opts.DirSort, "dirsfirst", "", false, "List directories before files (-U disables).")
flags.StringVarP(&sort, "sort", "", "", "Select sort: name,version,size,mtime,ctime.")
flags.BoolVarP(cmdFlags, &opts.NoSort, "unsorted", "U", false, "Leave files unsorted.")
flags.BoolVarP(cmdFlags, &opts.VerSort, "version", "", false, "Sort files alphanumerically by version.")
flags.BoolVarP(cmdFlags, &opts.ModSort, "sort-modtime", "t", false, "Sort files by last modification time.")
flags.BoolVarP(cmdFlags, &opts.CTimeSort, "sort-ctime", "", false, "Sort files by last status change time.")
flags.BoolVarP(cmdFlags, &opts.ReverSort, "sort-reverse", "r", false, "Reverse the order of the sort.")
flags.BoolVarP(cmdFlags, &opts.DirSort, "dirsfirst", "", false, "List directories before files (-U disables).")
flags.StringVarP(cmdFlags, &sort, "sort", "", "", "Select sort: name,version,size,mtime,ctime.")
// Graphics
flags.BoolVarP(&opts.NoIndent, "noindent", "i", false, "Don't print indentation lines.")
flags.BoolVarP(&opts.Colorize, "color", "C", false, "Turn colorization on always.")
flags.BoolVarP(cmdFlags, &opts.NoIndent, "noindent", "i", false, "Don't print indentation lines.")
flags.BoolVarP(cmdFlags, &opts.Colorize, "color", "C", false, "Turn colorization on always.")
}
var commandDefintion = &cobra.Command{
var commandDefinition = &cobra.Command{
Use: "tree remote:path",
Short: `List the contents of the remote in a tree like fashion.`,
Long: `
+3 -2
View File
@@ -10,6 +10,7 @@ import (
"github.com/pkg/errors"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/config/flags"
"github.com/rclone/rclone/fs/version"
"github.com/spf13/cobra"
)
@@ -20,8 +21,8 @@ var (
func init() {
cmd.Root.AddCommand(commandDefinition)
flags := commandDefinition.Flags()
flags.BoolVarP(&check, "check", "", false, "Check for new version.")
cmdFlags := commandDefinition.Flags()
flags.BoolVarP(cmdFlags, &check, "check", "", false, "Check for new version.")
}
var commandDefinition = &cobra.Command{
+13 -1
View File
@@ -192,7 +192,7 @@ Contributors
* Sheldon Rupp <me@shel.io>
* albertony <12441419+albertony@users.noreply.github.com>
* cron410 <cron410@gmail.com>
* Anagh Kumar Baranwal <anaghk.dos@gmail.com>
* Anagh Kumar Baranwal <anaghk.dos@gmail.com> <6824881+darthShadow@users.noreply.github.com>
* Felix Brucker <felix@felixbrucker.com>
* Santiago Rodríguez <scollazo@users.noreply.github.com>
* Craig Miskell <craig.miskell@fluxfederation.com>
@@ -301,3 +301,15 @@ Contributors
* Vighnesh SK <booterror99@gmail.com>
* Arijit Biswas <dibbyo456@gmail.com>
* Michele Caci <michele.caci@gmail.com>
* AlexandrBoltris <ua2fgb@gmail.com>
* Bryce Larson <blarson@saltstack.com>
* Carlos Ferreyra <crypticmind@gmail.com>
* Saksham Khanna <sakshamkhanna@outlook.com>
* dausruddin <5763466+dausruddin@users.noreply.github.com>
* zero-24 <zero-24@users.noreply.github.com>
* Xiaoxing Ye <ye@xiaoxing.us>
* Barry Muldrey <barry@muldrey.net>
* Sebastian Brandt <sebastian.brandt@friday.de>
* Marco Molteni <marco.molteni@mailbox.org>
* Ankur Gupta <ankur0493@gmail.com>
* Maciej Zimnoch <maciej@scylladb.com>
+24
View File
@@ -252,6 +252,30 @@ Leave blank normally.
- Type: string
- Default: ""
#### --box-box-config-file
Box App config.json location
Leave blank normally.
- Config: box_config_file
- Env Var: RCLONE_BOX_BOX_CONFIG_FILE
- Type: string
- Default: ""
#### --box-box-sub-type
- Config: box_sub_type
- Env Var: RCLONE_BOX_BOX_SUB_TYPE
- Type: string
- Default: "user"
- Examples:
- "user"
- Rclone should act on behalf of a user
- "enterprise"
- Rclone should act on behalf of a service account
### Advanced Options
Here are the advanced options specific to box (Box).
+129 -5
View File
@@ -1,11 +1,137 @@
---
title: "Documentation"
description: "Rclone Changelog"
date: "2019-10-05"
date: "2019-11-19"
---
# Changelog
## v1.50.2 - 2019-11-19
* Bug Fixes
* accounting: Fix memory leak on retries operations (Nick Craig-Wood)
* Drive
* Fix listing of the root directory with drive.files scope (Nick Craig-Wood)
* Fix --drive-root-folder-id with team/shared drives (Nick Craig-Wood)
## v1.50.1 - 2019-11-02
* Bug Fixes
* hash: Fix accidentally changed hash names for `DropboxHash` and `CRC-32` (Nick Craig-Wood)
* fshttp: Fix error reporting on tpslimit token bucket errors (Nick Craig-Wood)
* fshttp: Don't print token bucket errors on context cancelled (Nick Craig-Wood)
* Local
* Fix listings of . on Windows (Nick Craig-Wood)
* Onedrive
* Fix DirMove/Move after Onedrive change (Xiaoxing Ye)
## v1.50.0 - 2019-10-26
* New backends
* [Citrix Sharefile](/sharefile) (Nick Craig-Wood)
* [Chunker](/chunker) - an overlay backend to split files into smaller parts (Ivan Andreev)
* [Mail.ru Cloud](/mailru) (Ivan Andreev)
* New Features
* encodings (Fabian Möller & Nick Craig-Wood)
* All backends now use file name encoding to ensure any file name can be written to any backend.
* See the [restricted file name docs](/overview/#restricted-filenames) for more info and the [local backend docs](/local/#filenames).
* Some file names may look different in rclone if you are using any control characters in names or [unicode FULLWIDTH symbols](https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block)).
* build
* Update to use go1.13 for the build (Nick Craig-Wood)
* Drop support for go1.9 (Nick Craig-Wood)
* Build rclone with GitHub actions (Nick Craig-Wood)
* Convert python scripts to python3 (Nick Craig-Wood)
* Swap Azure/go-ansiterm for mattn/go-colorable (Nick Craig-Wood)
* Dockerfile fixes (Matei David)
* Add [plugin support](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#writing-a-plugin) for backends and commands (Richard Patel)
* config
* Use alternating Red/Green in config to make more obvious (Nick Craig-Wood)
* contrib
* Add sample DLNA server Docker Compose manifest. (pataquets)
* Add sample WebDAV server Docker Compose manifest. (pataquets)
* copyurl
* Add `--auto-filename` flag for using file name from URL in destination path (Denis)
* serve dlna:
* Many compatability improvements (Dan Walters)
* Support for external srt subtitles (Dan Walters)
* rc
* Added command core/quit (Saksham Khanna)
* Bug Fixes
* sync
* Make `--update`/`-u` not transfer files that haven't changed (Nick Craig-Wood)
* Free objects after they come out of the transfer pipe to save memory (Nick Craig-Wood)
* Fix `--files-from without --no-traverse` doing a recursive scan (Nick Craig-Wood)
* operations
* Fix accounting for server side copies (Nick Craig-Wood)
* Display 'All duplicates removed' only if dedupe successful (Sezal Agrawal)
* Display 'Deleted X extra copies' only if dedupe successful (Sezal Agrawal)
* accounting
* Only allow up to 100 completed transfers in the accounting list to save memory (Nick Craig-Wood)
* Cull the old time ranges when possible to save memory (Nick Craig-Wood)
* Fix panic due to server-side copy fallback (Ivan Andreev)
* Fix memory leak noticeable for transfers of large numbers of objects (Nick Craig-Wood)
* Fix total duration calculation (Nick Craig-Wood)
* cmd
* Fix environment variables not setting command line flags (Nick Craig-Wood)
* Make autocomplete compatible with bash's posix mode for macOS (Danil Semelenov)
* Make `--progress` work in git bash on Windows (Nick Craig-Wood)
* Fix 'compopt: command not found' on autocomplete on macOS (Danil Semelenov)
* config
* Fix setting of non top level flags from environment variables (Nick Craig-Wood)
* Check config names more carefully and report errors (Nick Craig-Wood)
* Remove error: can't use `--size-only` and `--ignore-size` together. (Nick Craig-Wood)
* filter: Prevent mixing options when `--files-from` is in use (Michele Caci)
* serve sftp: Fix crash on unsupported operations (eg Readlink) (Nick Craig-Wood)
* Mount
* Allow files of unkown size to be read properly (Nick Craig-Wood)
* Skip tests on <= 2 CPUs to avoid lockup (Nick Craig-Wood)
* Fix panic on File.Open (Nick Craig-Wood)
* Fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD (Nick Craig-Wood)
* Don't pass huge filenames (>4k) to FUSE as it can't cope (Nick Craig-Wood)
* VFS
* Add flag `--vfs-case-insensitive` for windows/macOS mounts (Ivan Andreev)
* Make objects of unknown size readable through the VFS (Nick Craig-Wood)
* Move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush() (Brett Dutro)
* Stop empty dirs disappearing when renamed on bucket based remotes (Nick Craig-Wood)
* Stop change notify polling clearing so much of the directory cache (Nick Craig-Wood)
* Azure Blob
* Disable logging to the Windows event log (Nick Craig-Wood)
* B2
* Remove `unverified:` prefix on sha1 to improve interop (eg with CyberDuck) (Nick Craig-Wood)
* Box
* Add options to get access token via JWT auth (David)
* Drive
* Disable HTTP/2 by default to work around INTERNAL_ERROR problems (Nick Craig-Wood)
* Make sure that drive root ID is always canonical (Nick Craig-Wood)
* Fix `--drive-shared-with-me` from the root with lsand `--fast-list` (Nick Craig-Wood)
* Fix ChangeNotify polling for shared drives (Nick Craig-Wood)
* Fix change notify polling when using appDataFolder (Nick Craig-Wood)
* Dropbox
* Make disallowed filenames errors not retry (Nick Craig-Wood)
* Fix nil pointer exception on restricted files (Nick Craig-Wood)
* Fichier
* Fix accessing files > 2GB on 32 bit systems (Nick Craig-Wood)
* FTP
* Allow disabling EPSV mode (Jon Fautley)
* HTTP
* HEAD directory entries in parallel to speedup (Nick Craig-Wood)
* Add `--http-no-head` to stop rclone doing HEAD in listings (Nick Craig-Wood)
* Putio
* Add ability to resume uploads (Cenk Alti)
* S3
* Fix signature v2_auth headers (Anthony Rusdi)
* Fix encoding for control characters (Nick Craig-Wood)
* Only ask for URL encoded directory listings if we need them on Ceph (Nick Craig-Wood)
* Add option for multipart failiure behaviour (Aleksandar Jankovic)
* Support for multipart copy (庄天翼)
* Fix nil pointer reference if no metadata returned for object (Nick Craig-Wood)
* SFTP
* Fix `--sftp-ask-password` trying to contact the ssh agent (Nick Craig-Wood)
* Fix hashes of files with backslashes (Nick Craig-Wood)
* Include more ciphers with `--sftp-use-insecure-cipher` (Carlos Ferreyra)
* WebDAV
* Parse and return Sharepoint error response (Henning Surmeier)
## v1.49.5 - 2019-10-05
* Bug Fixes
@@ -36,17 +162,15 @@ date: "2019-10-05"
* New Features
* build: Add Docker workflow support (Alfonso Montero)
* Bug Fixes
* accounting: Fix locking in Transfer to avoid deadlock with --progress (Nick Craig-Wood)
* accounting: Fix locking in Transfer to avoid deadlock with `--progress` (Nick Craig-Wood)
* docs: Fix template argument for mktemp in install.sh (Cnly)
* operations: Fix -u/--update with google photos / files of unknown size (Nick Craig-Wood)
* operations: Fix `-u`/`--update` with google photos / files of unknown size (Nick Craig-Wood)
* rc: Fix docs for config/create /update /password (Nick Craig-Wood)
* Google Cloud Storage
* Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
## v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
* Bug Fixes
* config: Fix generated passwords being stored as empty password (Nick Craig-Wood)
* rcd: Added missing parameter for web-gui info logs. (Chaitanya)
+2 -2
View File
@@ -293,7 +293,7 @@ in the same directory).
<!--- autogenerated options start - DO NOT EDIT, instead edit fs.RegInfo in backend/chunker/chunker.go then run make backenddocs -->
### Standard Options
Here are the standard options specific to chunker.
Here are the standard options specific to chunker (Transparently chunk/split large files).
#### --chunker-remote
@@ -341,7 +341,7 @@ Choose how chunker handles hash sums. All modes but "none" require metadata.
### Advanced Options
Here are the advanced options specific to chunker.
Here are the advanced options specific to chunker (Transparently chunk/split large files).
#### --chunker-name-format
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone"
slug: rclone
url: /commands/rclone/
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone about"
slug: rclone_about
url: /commands/rclone_about/
+3 -2
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone authorize"
slug: rclone_authorize
url: /commands/rclone_authorize/
@@ -22,7 +22,8 @@ rclone authorize [flags]
### Options
```
-h, --help help for authorize
--auth-no-open-browser Do not automatically open auth link in default browser
-h, --help help for authorize
```
See the [global flags page](/flags/) for global options not listed here.
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone cachestats"
slug: rclone_cachestats
url: /commands/rclone_cachestats/
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone cat"
slug: rclone_cat
url: /commands/rclone_cat/
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone check"
slug: rclone_check
url: /commands/rclone_check/
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone cleanup"
slug: rclone_cleanup
url: /commands/rclone_cleanup/
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone config"
slug: rclone_config
url: /commands/rclone_config/
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone config create"
slug: rclone_config_create
url: /commands/rclone_config_create/
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone config delete"
slug: rclone_config_delete
url: /commands/rclone_config_delete/
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone config disconnect"
slug: rclone_config_disconnect
url: /commands/rclone_config_disconnect/
+1 -1
View File
@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-26T11:04:03+01:00
title: "rclone config dump"
slug: rclone_config_dump
url: /commands/rclone_config_dump/
Loaded 100 of 704 files, more files were not shown because too many files have changed in this diff. Show more