rc: update docs with new methods

If `--rc-user` or `--rc-pass` is set then the URL that is opened with
`--rc-files` will have the authorization in the URL in the
`http://user:pass@localhost/` style.

.appveyor.yml

@@ -4,6 +4,9 @@ os: Windows Server 2012 R2
 
 clone_folder: c:\gopath\src\github.com\ncw\rclone
 
+cache:
+  - '%LocalAppData%\go-build'
+
 environment:
   GOPATH: C:\gopath
   CPATH: C:\Program Files (x86)\WinFsp\inc\fuse
@@ -43,4 +46,4 @@ artifacts:
 - path: build/*-v*.zip
 
 deploy_script:
-- IF "%APPVEYOR_PULL_REQUEST_NUMBER%" == "" make appveyor_upload
+- IF "%APPVEYOR_REPO_NAME%" == "ncw/rclone" IF "%APPVEYOR_PULL_REQUEST_NUMBER%" == "" make appveyor_upload

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,31 @@
+<!--
+
+Welcome :-) We understand you are having a problem with rclone; we want to help you with that!
+
+If you've just got a question or aren't sure if you've found a bug then please use the rclone forum:
+
+    https://forum.rclone.org/
+
+instead of filing an issue for a quick response.
+
+If you are reporting a bug or asking for a new feature then please use one of the templates here:
+
+    https://github.com/ncw/rclone/issues/new
+
+otherwise fill in the form below.
+
+Thank you
+
+The Rclone Developers
+
+-->
+
+
+#### Output of `rclone version`
+
+
+
+#### Describe the issue
+
+
+

.github/ISSUE_TEMPLATE/Bug.md

@@ -0,0 +1,50 @@
+---
+name: Bug report
+about: Report a problem with rclone
+---
+
+<!--
+
+Welcome :-) We understand you are having a problem with rclone; we want to help you with that!
+
+If you've just got a question or aren't sure if you've found a bug then please use the rclone forum:
+
+    https://forum.rclone.org/
+
+instead of filing an issue for a quick response.
+
+If you think you might have found a bug, please can you try to replicate it with the latest beta?
+
+    https://beta.rclone.org/
+    
+If you can still replicate it with the latest beta, then please fill in the info below which makes our lives much easier.  A log with -vv will make our day :-)
+
+Thank you
+
+The Rclone Developers
+
+-->
+
+#### What is the problem you are having with rclone?
+
+
+
+#### What is your rclone version (output from `rclone version`)
+
+
+
+#### Which OS you are using and how many bits (eg Windows 7, 64 bit)
+
+
+
+####  Which cloud storage system are you using? (eg Google Drive)
+
+
+
+#### The command you were trying to run (eg `rclone copy /tmp remote:tmp`)
+
+
+
+#### A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)
+
+

.github/ISSUE_TEMPLATE/Feature.md

@@ -0,0 +1,36 @@
+---
+name: Feature request
+about: Suggest a new feature or enhancement for rclone
+---
+
+<!--
+
+Welcome :-)
+
+So you've got an idea to improve rclone? We love that! You'll be glad to hear we've incorporated hundreds of ideas from contributors already.
+
+Here is a checklist of things to do:
+
+  1. Please search the old issues first for your idea and +1 or comment on an existing issue if possible.
+  2. Discuss on the forum first: https://forum.rclone.org/
+  3. Make a feature request issue (this is the right place!).
+  4. Be prepared to get involved making the feature :-)
+
+Looking forward to your great idea!
+
+The Rclone Developers
+
+-->
+
+
+#### What is your current rclone version (output from `rclone version`)?
+
+
+
+#### What problem are you are trying to solve?
+
+
+
+#### How do you think rclone should be changed to solve that?
+
+

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,29 @@
+<!--
+Thank you very much for contributing code or documentation to rclone! Please
+fill out the following questions to make it easier for us to review your
+changes.
+
+You do not need to check all the boxes below all at once, feel free to take
+your time and add more commits. If you're done and ready for review, please
+check the last box.
+-->
+
+#### What is the purpose of this change?
+
+<!--
+Describe the changes here
+-->
+
+#### Was the change discussed in an issue or in the forum before?
+
+<!--
+Link issues and relevant forum posts here.
+-->
+
+#### Checklist
+
+- [ ] I have read the [contribution guidelines](https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request).
+- [ ] I have added tests for all changes in this PR if appropriate.
+- [ ] I have added documentation for the changes if appropriate.
+- [ ] All commit messages are in [house style](https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#commit-messages).
+- [ ] I'm done, this Pull Request is ready for review :-)

.travis.yml

@@ -4,12 +4,12 @@ dist: trusty
 os:
 - linux
 go:
-- 1.7.6
-- 1.8.7
-- 1.9.3
-- "1.10.1"
-- "1.11rc1"
+- 1.8.x
+- 1.9.x
+- 1.10.x
+- 1.11.x
 - tip
+go_import_path: github.com/ncw/rclone
 before_install:
 - if [[ $TRAVIS_OS_NAME == linux ]]; then sudo modprobe fuse ; sudo chmod 666 /dev/fuse ; sudo chown root:$USER /etc/fuse.conf ; fi
 - if [[ $TRAVIS_OS_NAME == osx ]]; then brew update && brew tap caskroom/cask && brew cask install osxfuse ; fi
@@ -34,18 +34,25 @@ addons:
     - libfuse-dev
     - rpm
     - pkg-config
+cache:
+  directories:
+    - $HOME/.cache/go-build
 matrix:
   allow_failures:
   - go: tip
   include:
   - os: osx
-    go: "1.11rc1"
+    go: 1.11.x
     env: GOTAGS=""
+    cache:
+      directories:
+        - $HOME/Library/Caches/go-build
 deploy:
   provider: script
   script: make travis_beta
   skip_cleanup: true
   on:
+    repo: ncw/rclone
     all_branches: true
-    go: "1.11rc1"
+    go: 1.11.x
     condition: $TRAVIS_PULL_REQUEST == false

CONTRIBUTING.md

@@ -21,19 +21,19 @@ with the [latest beta of rclone](https://beta.rclone.org/):
 ## Submitting a pull request ##
 
 If you find a bug that you'd like to fix, or a new feature that you'd
-like to implement then please submit a pull request via Github.
+like to implement then please submit a pull request via GitHub.
 
 If it is a big feature then make an issue first so it can be discussed.
 
 You'll need a Go environment set up with GOPATH set.  See [the Go
 getting started docs](https://golang.org/doc/install) for more info.
 
-First in your web browser press the fork button on [rclone's Github
+First in your web browser press the fork button on [rclone's GitHub
 page](https://github.com/ncw/rclone).
 
 Now in your terminal
 
-    go get github.com/ncw/rclone
+    go get -u github.com/ncw/rclone
     cd $GOPATH/src/github.com/ncw/rclone
     git remote rename origin upstream
     git remote add origin git@github.com:YOURUSER/rclone.git
@@ -64,22 +64,31 @@ packages which you can install with
 
 Make sure you
 
-  * Add documentation for a new feature (see below for where)
-  * Add unit tests for a new feature
+  * Add [documentation](#writing-documentation) for a new feature.
+  * Follow the [commit message guidelines](#commit-messages).
+  * Add [unit tests](#testing) for a new feature
   * squash commits down to one per feature
-  * rebase to master `git rebase master`
+  * rebase to master with `git rebase master`
 
 When you are done with that
 
-  git push origin my-new-feature
+    git push origin my-new-feature
 
-Go to the Github website and click [Create pull
+Go to the GitHub website and click [Create pull
 request](https://help.github.com/articles/creating-a-pull-request/).
 
 You patch will get reviewed and you might get asked to fix some stuff.
 
 If so, then make the changes in the same branch, squash the commits,
-rebase it to master then push it to Github with `--force`.
+rebase it to master then push it to GitHub with `--force`.
+
+## Enabling CI for your fork ##
+
+The CI config files for rclone have taken care of forks of the project, so you can enable CI for your fork repo easily.
+
+rclone currently uses [Travis CI](https://travis-ci.org/), [AppVeyor](https://ci.appveyor.com/), and
+[Circle CI](https://circleci.com/) to build the project. To enable them for your fork, simply go into their
+websites, find your fork of rclone, and enable building there.
 
 ## Testing ##
 
@@ -114,6 +123,13 @@ but they can be run against any of the remotes.
     cd fs/operations
     go test -v -remote TestDrive:
 
+If you want to use the integration test framework to run these tests
+all together with an HTML report and test retries then from the
+project root:
+
+    go install github.com/ncw/rclone/fstest/test_all
+    test_all -backend drive
+
 If you want to run all the integration tests against all the remotes,
 then change into the project root and run
 
@@ -166,17 +182,21 @@ with modules beneath.
     * pacer - retries with backoff and paces operations
     * readers - a selection of useful io.Readers
     * rest - a thin abstraction over net/http for REST
-  * vendor - 3rd party code managed by the dep tool
+  * vendor - 3rd party code managed by `go mod`
   * vfs - Virtual FileSystem layer for implementing rclone mount and similar
 
 ## Writing Documentation ##
 
 If you are adding a new feature then please update the documentation.
 
-If you add a new flag, then if it is a general flag, document it in
+If you add a new general flag (not for a backend), then document it in
 `docs/content/docs.md` - the flags there are supposed to be in
-alphabetical order.  If it is a remote specific flag, then document it
-in `docs/content/remote.md`.
+alphabetical order.
+
+If you add a new backend option/flag, then it should be documented in
+the source file in the `Help:` field.  The first line of this is used
+for the flag help, the remainder is shown to the user in `rclone
+config` and is added to the docs with `make backenddocs`.
 
 The only documentation you need to edit are the `docs/content/*.md`
 files.  The MANUAL.*, rclone.1, web site etc are all auto generated
@@ -195,14 +215,20 @@ file.
 ## Commit messages ##
 
 Please make the first line of your commit message a summary of the
-change, and prefix it with the directory of the change followed by a
-colon.  The changelog gets made by looking at just these first lines
-so make it good!
+change that a user (not a developer) of rclone would like to read, and
+prefix it with the directory of the change followed by a colon.  The
+changelog gets made by looking at just these first lines so make it
+good!
 
 If you have more to say about the commit, then enter a blank line and
 carry on the description.  Remember to say why the change was needed -
 the commit itself shows what was changed.
 
+Writing more is better than less.  Comparing the behaviour before the
+change to that after the change is very useful.  Imagine you are
+writing to yourself in 12 months time when you've forgotten everything
+about what you just did and you need to get up to speed quickly.
+
 If the change fixes an issue then write `Fixes #1234` in the commit
 message.  This can be on the subject line if it will fit.  If you
 don't want to close the associated issue just put `#1234` and the
@@ -229,37 +255,53 @@ Fixes #1498
 
 ## Adding a dependency ##
 
-rclone uses the [dep](https://github.com/golang/dep) tool to manage
-its dependencies.  All code that rclone needs for building is stored
-in the `vendor` directory for perfectly reproducable builds.
+rclone uses the [go
+modules](https://tip.golang.org/cmd/go/#hdr-Modules__module_versions__and_more)
+support in go1.11 and later to manage its dependencies.
 
-The `vendor` directory is entirely managed by the `dep` tool.
+**NB** you must be using go1.11 or above to add a dependency to
+rclone.  Rclone will still build with older versions of go, but we use
+the `go mod` command for dependencies which is only in go1.11 and
+above.
 
-To add a new dependency, run `dep ensure` and `dep` will pull in the
-new dependency to the `vendor` directory and update the `Gopkg.lock`
-file.
+rclone can be built with modules outside of the GOPATH, but for
+backwards compatibility with older go versions, rclone also maintains
+a `vendor` directory with all the external code rclone needs for
+building.
 
-You can add constraints on that package in the `Gopkg.toml` file (see
-the `dep` documentation), but don't unless you really need to.
+The `vendor` directory is entirely managed by the `go mod` tool, do
+not add things manually.
 
-Please check in the changes generated by `dep` including the `vendor`
-directory and `Godep.toml` and `Godep.lock` in a single commit
-separate from any other code changes.  Watch out for new files in
-`vendor`.
+To add a dependency `github.com/ncw/new_dependency` see the
+instructions below.  These will fetch the dependency, add it to
+`go.mod` and `go.sum` and vendor it for older go versions.
+
+    GO111MODULE=on go get github.com/ncw/new_dependency
+    GO111MODULE=on go mod vendor
+
+You can add constraints on that package when doing `go get` (see the
+go docs linked above), but don't unless you really need to.
+
+Please check in the changes generated by `go mod` including the
+`vendor` directory and `go.mod` and `go.sum` in a single commit
+separate from any other code changes with the title "vendor: add
+github.com/ncw/new_dependency".  Remember to `git add` any new files
+in `vendor`.
 
 ## Updating a dependency ##
 
 If you need to update a dependency then run
 
-    dep ensure -update github.com/pkg/errors
+    GO111MODULE=on go get -u github.com/pkg/errors
+    GO111MODULE=on go mod vendor
 
 Check in in a single commit as above.
 
 ## Updating all the dependencies ##
 
 In order to update all the dependencies then run `make update`.  This
-just runs `dep ensure -update`.  Check in the changes in a single
-commit as above.
+just uses the go modules to update all the modules to their latest
+stable release. Check in the changes in a single commit as above.
 
 This should be done early in the release cycle to pick up new versions
 of packages in time for them to get some testing.
@@ -308,7 +350,7 @@ Unit tests
 
 Integration tests
 
-  * Add your fs to `fstest/test_all/test_all.go`
+  * Add your backend to `fstest/test_all/config.yaml`
   * Make sure integration tests pass with
       * `cd fs/operations`
       * `go test -v -remote TestRemote:`
@@ -323,8 +365,8 @@ See the [testing](#testing) section for more information on integration tests.
 
 Add your fs to the docs - you'll need to pick an icon for it from [fontawesome](http://fontawesome.io/icons/).  Keep lists of remotes in alphabetical order but with the local file system last.
 
-  * `README.md` - main Github page
-  * `docs/content/remote.md` - main docs page
+  * `README.md` - main GitHub page
+  * `docs/content/remote.md` - main docs page (note the backend options are automatically added to this file with `make backenddocs`)
   * `docs/content/overview.md` - overview docs
   * `docs/content/docs.md` - list of remotes in config section
   * `docs/content/about.md` - front page of rclone.org

Gopkg.lock

@@ -1,633 +0,0 @@
-# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
-
-
-[[projects]]
-  branch = "master"
-  digest = "1:df0ad5dd3f57601a1185ce7fdc76fef7101654c602d6a8c5153bce76ff6253c0"
-  name = "bazil.org/fuse"
-  packages = [
-    ".",
-    "fs",
-    "fuseutil",
-  ]
-  pruneopts = ""
-  revision = "65cc252bf6691cb3c7014bcb2c8dc29de91e3a7e"
-
-[[projects]]
-  digest = "1:4e800c0d846ed856a032380c87b22577ef03c146ccd26203b62ac90ef78e94b1"
-  name = "cloud.google.com/go"
-  packages = ["compute/metadata"]
-  pruneopts = ""
-  revision = "0fd7230b2a7505833d5f69b75cbd6c9582401479"
-  version = "v0.23.0"
-
-[[projects]]
-  digest = "1:6f302284bb48712a01cdcd3216e8bbb293d1edb618f55b5fe7f92521cce930c7"
-  name = "github.com/Azure/azure-pipeline-go"
-  packages = ["pipeline"]
-  pruneopts = ""
-  revision = "7571e8eb0876932ab505918ff7ed5107773e5ee2"
-  version = "0.1.7"
-
-[[projects]]
-  branch = "master"
-  digest = "1:fe7593b80dc3ea36fe51ca434844b801e1695e7b680851a137ed40fa2b4d6184"
-  name = "github.com/Azure/azure-storage-blob-go"
-  packages = ["2018-03-28/azblob"]
-  pruneopts = ""
-  revision = "eaae161d9d5e07363f04ddb19d84d57efc66d1a1"
-
-[[projects]]
-  branch = "master"
-  digest = "1:f9282871e22332a39831bc53a3588fc04d763bf3f13312655580c7f4cb012b66"
-  name = "github.com/Unknwon/goconfig"
-  packages = ["."]
-  pruneopts = ""
-  revision = "ef1e4c783f8f0478bd8bff0edb3dd0bade552599"
-
-[[projects]]
-  digest = "1:ac226c42eb54c121e0704c6f7f64c96c7817ad6d6286e5536e8cea72807e1079"
-  name = "github.com/VividCortex/ewma"
-  packages = ["."]
-  pruneopts = ""
-  revision = "b24eb346a94c3ba12c1da1e564dbac1b498a77ce"
-  version = "v1.1.1"
-
-[[projects]]
-  branch = "master"
-  digest = "1:391632fa3a324c4f461f28baaf45cea8b21e13630b00f27059613f855bb544bb"
-  name = "github.com/a8m/tree"
-  packages = ["."]
-  pruneopts = ""
-  revision = "3cf936ce15d6100c49d9c75f79c220ae7e579599"
-
-[[projects]]
-  digest = "1:61e512f75ec00c5d0e33e1f503fbad293f5850b76a8f62f6035097c8c436315d"
-  name = "github.com/abbot/go-http-auth"
-  packages = ["."]
-  pruneopts = ""
-  revision = "0ddd408d5d60ea76e320503cc7dd091992dee608"
-  version = "v0.4.0"
-
-[[projects]]
-  digest = "1:f9dc8648e19ca5c4ccdf32e13301eaaff14a6662826c98926ba401d98bdea315"
-  name = "github.com/aws/aws-sdk-go"
-  packages = [
-    "aws",
-    "aws/awserr",
-    "aws/awsutil",
-    "aws/client",
-    "aws/client/metadata",
-    "aws/corehandlers",
-    "aws/credentials",
-    "aws/credentials/ec2rolecreds",
-    "aws/credentials/endpointcreds",
-    "aws/credentials/stscreds",
-    "aws/csm",
-    "aws/defaults",
-    "aws/ec2metadata",
-    "aws/endpoints",
-    "aws/request",
-    "aws/session",
-    "aws/signer/v4",
-    "internal/sdkio",
-    "internal/sdkrand",
-    "internal/shareddefaults",
-    "private/protocol",
-    "private/protocol/eventstream",
-    "private/protocol/eventstream/eventstreamapi",
-    "private/protocol/query",
-    "private/protocol/query/queryutil",
-    "private/protocol/rest",
-    "private/protocol/restxml",
-    "private/protocol/xml/xmlutil",
-    "service/s3",
-    "service/s3/s3iface",
-    "service/s3/s3manager",
-    "service/sts",
-  ]
-  pruneopts = ""
-  revision = "bfc1a07cf158c30c41a3eefba8aae043d0bb5bff"
-  version = "v1.14.8"
-
-[[projects]]
-  digest = "1:a82690274ae3235a742afe7eebd9affcb97a80340af7d9b538f583cd953cff19"
-  name = "github.com/billziss-gh/cgofuse"
-  packages = ["fuse"]
-  pruneopts = ""
-  revision = "ea66f9809c71af94522d494d3d617545662ea59d"
-  version = "v1.1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:80db6bfe7a7d5e277987232bb650ccdeb365d00293576c5421dab101401741b6"
-  name = "github.com/coreos/bbolt"
-  packages = ["."]
-  pruneopts = ""
-  revision = "af9db2027c98c61ecd8e17caa5bd265792b9b9a2"
-
-[[projects]]
-  digest = "1:982e2547680f9fd2212c6443ab73ea84eef40ee1cdcecb61d997de838445214c"
-  name = "github.com/cpuguy83/go-md2man"
-  packages = ["md2man"]
-  pruneopts = ""
-  revision = "20f5889cbdc3c73dbd2862796665e7c465ade7d1"
-  version = "v1.0.8"
-
-[[projects]]
-  digest = "1:56c130d885a4aacae1dd9c7b71cfe39912c7ebc1ff7d2b46083c8812996dc43b"
-  name = "github.com/davecgh/go-spew"
-  packages = ["spew"]
-  pruneopts = ""
-  revision = "346938d642f2ec3594ed81d874461961cd0faa76"
-  version = "v1.1.0"
-
-[[projects]]
-  digest = "1:7b4b8c901568da024c49be7ff5e20fdecef629b60679c803041093823fb8d081"
-  name = "github.com/djherbis/times"
-  packages = ["."]
-  pruneopts = ""
-  revision = "95292e44976d1217cf3611dc7c8d9466877d3ed5"
-  version = "v1.0.1"
-
-[[projects]]
-  digest = "1:2a99d23b565e06fe1930f444c53c066216f06465c8d1d097b691f23c169858ea"
-  name = "github.com/dropbox/dropbox-sdk-go-unofficial"
-  packages = [
-    "dropbox",
-    "dropbox/async",
-    "dropbox/common",
-    "dropbox/file_properties",
-    "dropbox/files",
-    "dropbox/seen_state",
-    "dropbox/sharing",
-    "dropbox/team_common",
-    "dropbox/team_policies",
-    "dropbox/users",
-    "dropbox/users_common",
-  ]
-  pruneopts = ""
-  revision = "7afa861bfde5a348d765522b303b6fbd9d250155"
-  version = "v4.1.0"
-
-[[projects]]
-  digest = "1:617b3e0f5989d4ff866a1820480990c65dfc9257eb080da749a45e2d76681b02"
-  name = "github.com/go-ini/ini"
-  packages = ["."]
-  pruneopts = ""
-  revision = "06f5f3d67269ccec1fe5fe4134ba6e982984f7f5"
-  version = "v1.37.0"
-
-[[projects]]
-  digest = "1:f958a1c137db276e52f0b50efee41a1a389dcdded59a69711f3e872757dab34b"
-  name = "github.com/golang/protobuf"
-  packages = ["proto"]
-  pruneopts = ""
-  revision = "b4deda0973fb4c70b50d226b1af49f3da59f5265"
-  version = "v1.1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:9abc49f39e3e23e262594bb4fb70abf74c0c99e94f99153f43b143805e850719"
-  name = "github.com/google/go-querystring"
-  packages = ["query"]
-  pruneopts = ""
-  revision = "53e6ce116135b80d037921a7fdd5138cf32d7a8a"
-
-[[projects]]
-  digest = "1:870d441fe217b8e689d7949fef6e43efbc787e50f200cb1e70dbca9204a1d6be"
-  name = "github.com/inconshreveable/mousetrap"
-  packages = ["."]
-  pruneopts = ""
-  revision = "76626ae9c91c4f2a10f34cad8ce83ea42c93bb75"
-  version = "v1.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:64b78d98b8956492576911baf6a1e3499816d4575e485d12792e4abe7d8b6c46"
-  name = "github.com/jlaffaye/ftp"
-  packages = ["."]
-  pruneopts = ""
-  revision = "2403248fa8cc9f7909862627aa7337f13f8e0bf1"
-
-[[projects]]
-  digest = "1:6f49eae0c1e5dab1dafafee34b207aeb7a42303105960944828c2079b92fc88e"
-  name = "github.com/jmespath/go-jmespath"
-  packages = ["."]
-  pruneopts = ""
-  revision = "0b12d6b5"
-
-[[projects]]
-  branch = "master"
-  digest = "1:2c5ad58492804c40bdaf5d92039b0cde8b5becd2b7feeb37d7d1cc36a8aa8dbe"
-  name = "github.com/kardianos/osext"
-  packages = ["."]
-  pruneopts = ""
-  revision = "ae77be60afb1dcacde03767a8c37337fad28ac14"
-
-[[projects]]
-  digest = "1:1bea35e6d6ea2712107d20a9770cf331b5c18e17ba02d28011939d9bc7c67534"
-  name = "github.com/kr/fs"
-  packages = ["."]
-  pruneopts = ""
-  revision = "1455def202f6e05b95cc7bfc7e8ae67ae5141eba"
-  version = "v0.1.0"
-
-[[projects]]
-  digest = "1:81e673df85e765593a863f67cba4544cf40e8919590f04d67664940786c2b61a"
-  name = "github.com/mattn/go-runewidth"
-  packages = ["."]
-  pruneopts = ""
-  revision = "9e777a8366cce605130a531d2cd6363d07ad7317"
-  version = "v0.0.2"
-
-[[projects]]
-  branch = "master"
-  digest = "1:1f8bb0521ae032c1e4ba38adb361a1840b81c8bf52bea5fcf900edf44c17f6cb"
-  name = "github.com/ncw/go-acd"
-  packages = ["."]
-  pruneopts = ""
-  revision = "887eb06ab6a255fbf5744b5812788e884078620a"
-
-[[projects]]
-  digest = "1:7a4827b2062a21ba644241bdec27959a5be2670f8aa4038ba14cfe2ce389e8d2"
-  name = "github.com/ncw/swift"
-  packages = ["."]
-  pruneopts = ""
-  revision = "b2a7479cf26fa841ff90dd932d0221cb5c50782d"
-  version = "v1.0.39"
-
-[[projects]]
-  branch = "master"
-  digest = "1:1a25d906193d34ce43e96fcab1a2de2b90e0242f9b0c176564db1b268bf48ea5"
-  name = "github.com/nsf/termbox-go"
-  packages = ["."]
-  pruneopts = ""
-  revision = "5c94acc5e6eb520f1bcd183974e01171cc4c23b3"
-
-[[projects]]
-  branch = "master"
-  digest = "1:7cdd61c36e251a51489ac7e73d070e3c67d8c9eb4bebf21a4886d6eec54d909c"
-  name = "github.com/okzk/sdnotify"
-  packages = ["."]
-  pruneopts = ""
-  revision = "d9becc38acbd785892af7637319e2c5e101057f7"
-
-[[projects]]
-  digest = "1:4c0404dc03d974acd5fcd8b8d3ce687b13bd169db032b89275e8b9d77b98ce8c"
-  name = "github.com/patrickmn/go-cache"
-  packages = ["."]
-  pruneopts = ""
-  revision = "a3647f8e31d79543b2d0f0ae2fe5c379d72cedc0"
-  version = "v2.1.0"
-
-[[projects]]
-  digest = "1:f6a7088857981d9d011920d6340f67d7b7649909918cf1ee6ba293718acc9e26"
-  name = "github.com/pengsrc/go-shared"
-  packages = [
-    "buffer",
-    "check",
-    "convert",
-    "log",
-    "reopen",
-  ]
-  pruneopts = ""
-  revision = "807ee759d82c84982a89fb3dc875ef884942f1e5"
-  version = "v0.2.0"
-
-[[projects]]
-  digest = "1:7365acd48986e205ccb8652cc746f09c8b7876030d53710ea6ef7d0bd0dcd7ca"
-  name = "github.com/pkg/errors"
-  packages = ["."]
-  pruneopts = ""
-  revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
-  version = "v0.8.0"
-
-[[projects]]
-  digest = "1:a80af40d2e94dd143f966f7f5968e762a282ea739ba4ecb14d231503c200065b"
-  name = "github.com/pkg/sftp"
-  packages = ["."]
-  pruneopts = ""
-  revision = "57673e38ea946592a59c26592b7e6fbda646975b"
-  version = "1.8.0"
-
-[[projects]]
-  digest = "1:256484dbbcd271f9ecebc6795b2df8cad4c458dd0f5fd82a8c2fa0c29f233411"
-  name = "github.com/pmezard/go-difflib"
-  packages = ["difflib"]
-  pruneopts = ""
-  revision = "792786c7400a136282c1664665ae0a8db921c6c2"
-  version = "v1.0.0"
-
-[[projects]]
-  digest = "1:c426863e69173c90404c3d3ad4acde6ff87f9353658a670ac6fdbe08a633750f"
-  name = "github.com/rfjakob/eme"
-  packages = ["."]
-  pruneopts = ""
-  revision = "01668ae55fe0b79a483095689043cce3e80260db"
-  version = "v1.1"
-
-[[projects]]
-  digest = "1:fd0e88ec70bf0efce538ec8b968a824d992cc60a6cf1539698aa366b3527a053"
-  name = "github.com/russross/blackfriday"
-  packages = ["."]
-  pruneopts = ""
-  revision = "55d61fa8aa702f59229e6cff85793c22e580eaf5"
-  version = "v1.5.1"
-
-[[projects]]
-  branch = "master"
-  digest = "1:05dc5c00381eccf0bbc07717248b8757e0e9318877e15e09316fac9b72f1b3ef"
-  name = "github.com/sevlyar/go-daemon"
-  packages = ["."]
-  pruneopts = ""
-  revision = "f9261e73885de99b1647d68bedadf2b9a99ad11f"
-
-[[projects]]
-  branch = "master"
-  digest = "1:50b5be512f924d289f20e8b2aef8951d98b9bd8c44666cf169514906df597a4c"
-  name = "github.com/skratchdot/open-golang"
-  packages = ["open"]
-  pruneopts = ""
-  revision = "75fb7ed4208cf72d323d7d02fd1a5964a7a9073c"
-
-[[projects]]
-  digest = "1:a1403cc8a94b8d7956ee5e9694badef0e7b051af289caad1cf668331e3ffa4f6"
-  name = "github.com/spf13/cobra"
-  packages = [
-    ".",
-    "doc",
-  ]
-  pruneopts = ""
-  revision = "ef82de70bb3f60c65fb8eebacbb2d122ef517385"
-  version = "v0.0.3"
-
-[[projects]]
-  digest = "1:8e243c568f36b09031ec18dff5f7d2769dcf5ca4d624ea511c8e3197dc3d352d"
-  name = "github.com/spf13/pflag"
-  packages = ["."]
-  pruneopts = ""
-  revision = "583c0c0531f06d5278b7d917446061adc344b5cd"
-  version = "v1.0.1"
-
-[[projects]]
-  digest = "1:c587772fb8ad29ad4db67575dad25ba17a51f072ff18a22b4f0257a4d9c24f75"
-  name = "github.com/stretchr/testify"
-  packages = [
-    "assert",
-    "require",
-  ]
-  pruneopts = ""
-  revision = "f35b8ab0b5a2cef36673838d662e249dd9c94686"
-  version = "v1.2.2"
-
-[[projects]]
-  branch = "master"
-  digest = "1:3c0753359567ac0500e17324c4da80398ee773093b4586e46210eea9dc03d155"
-  name = "github.com/t3rm1n4l/go-mega"
-  packages = ["."]
-  pruneopts = ""
-  revision = "854bf31d998b151cf5f94529c815bc4c67322949"
-
-[[projects]]
-  digest = "1:afc0b8068986a01e2d8f449917829753a54f6bd4d1265c2b4ad9cba75560020f"
-  name = "github.com/xanzy/ssh-agent"
-  packages = ["."]
-  pruneopts = ""
-  revision = "640f0ab560aeb89d523bb6ac322b1244d5c3796c"
-  version = "v0.2.0"
-
-[[projects]]
-  digest = "1:970aab2bde4ae92adf92ccae41eace959c66e2653ebb7e86355477e0307d0be8"
-  name = "github.com/yunify/qingstor-sdk-go"
-  packages = [
-    ".",
-    "config",
-    "logger",
-    "request",
-    "request/builder",
-    "request/data",
-    "request/errors",
-    "request/signer",
-    "request/unpacker",
-    "service",
-    "utils",
-  ]
-  pruneopts = ""
-  revision = "4f9ac88c5fec7350e960aabd0de1f1ede0ad2895"
-  version = "v2.2.14"
-
-[[projects]]
-  branch = "master"
-  digest = "1:419d8420cd7231162a9620ca6bc3b2c9ac98270590773d3f25d90950ccc984cc"
-  name = "golang.org/x/crypto"
-  packages = [
-    "bcrypt",
-    "blowfish",
-    "curve25519",
-    "ed25519",
-    "ed25519/internal/edwards25519",
-    "internal/chacha20",
-    "internal/subtle",
-    "nacl/secretbox",
-    "pbkdf2",
-    "poly1305",
-    "salsa20/salsa",
-    "scrypt",
-    "ssh",
-    "ssh/agent",
-    "ssh/terminal",
-  ]
-  pruneopts = ""
-  revision = "027cca12c2d63e3d62b670d901e8a2c95854feec"
-
-[[projects]]
-  branch = "master"
-  digest = "1:5dc6753986b9eeba4abdf05dedc5ba06bb52dad43cc8aad35ffb42bb7adfa68f"
-  name = "golang.org/x/net"
-  packages = [
-    "context",
-    "context/ctxhttp",
-    "html",
-    "html/atom",
-    "http/httpguts",
-    "http2",
-    "http2/hpack",
-    "idna",
-    "publicsuffix",
-    "webdav",
-    "webdav/internal/xml",
-    "websocket",
-  ]
-  pruneopts = ""
-  revision = "db08ff08e8622530d9ed3a0e8ac279f6d4c02196"
-
-[[projects]]
-  branch = "master"
-  digest = "1:823e7b6793b3f80b5d01da97211790dc89601937e4b70825fdcb5637ac60f04f"
-  name = "golang.org/x/oauth2"
-  packages = [
-    ".",
-    "google",
-    "internal",
-    "jws",
-    "jwt",
-  ]
-  pruneopts = ""
-  revision = "1e0a3fa8ba9a5c9eb35c271780101fdaf1b205d7"
-
-[[projects]]
-  branch = "master"
-  digest = "1:7e5298358e5f751305289e82373c7ac6832bdc492055d6da23c72fa1d8053c3f"
-  name = "golang.org/x/sys"
-  packages = [
-    "unix",
-    "windows",
-  ]
-  pruneopts = ""
-  revision = "6c888cc515d3ed83fc103cf1d84468aad274b0a7"
-
-[[projects]]
-  digest = "1:5acd3512b047305d49e8763eef7ba423901e85d5dd2fd1e71778a0ea8de10bd4"
-  name = "golang.org/x/text"
-  packages = [
-    "collate",
-    "collate/build",
-    "internal/colltab",
-    "internal/gen",
-    "internal/tag",
-    "internal/triegen",
-    "internal/ucd",
-    "language",
-    "secure/bidirule",
-    "transform",
-    "unicode/bidi",
-    "unicode/cldr",
-    "unicode/norm",
-    "unicode/rangetable",
-  ]
-  pruneopts = ""
-  revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
-  version = "v0.3.0"
-
-[[projects]]
-  branch = "master"
-  digest = "1:55a681cb66f28755765fa5fa5104cbd8dc85c55c02d206f9f89566451e3fe1aa"
-  name = "golang.org/x/time"
-  packages = ["rate"]
-  pruneopts = ""
-  revision = "fbb02b2291d28baffd63558aa44b4b56f178d650"
-
-[[projects]]
-  branch = "master"
-  digest = "1:7d15746ff4df12481c89fd953a28122fa75368fb1fb1bb1fed918a78647b3c3a"
-  name = "google.golang.org/api"
-  packages = [
-    "drive/v3",
-    "gensupport",
-    "googleapi",
-    "googleapi/internal/uritemplates",
-    "storage/v1",
-  ]
-  pruneopts = ""
-  revision = "2eea9ba0a3d94f6ab46508083e299a00bbbc65f6"
-
-[[projects]]
-  digest = "1:c1771ca6060335f9768dff6558108bc5ef6c58506821ad43377ee23ff059e472"
-  name = "google.golang.org/appengine"
-  packages = [
-    ".",
-    "internal",
-    "internal/app_identity",
-    "internal/base",
-    "internal/datastore",
-    "internal/log",
-    "internal/modules",
-    "internal/remote_api",
-    "internal/urlfetch",
-    "urlfetch",
-  ]
-  pruneopts = ""
-  revision = "b1f26356af11148e710935ed1ac8a7f5702c7612"
-  version = "v1.1.0"
-
-[[projects]]
-  digest = "1:f0620375dd1f6251d9973b5f2596228cc8042e887cd7f827e4220bc1ce8c30e2"
-  name = "gopkg.in/yaml.v2"
-  packages = ["."]
-  pruneopts = ""
-  revision = "5420a8b6744d3b0345ab293f6fcba19c978f1183"
-  version = "v2.2.1"
-
-[solve-meta]
-  analyzer-name = "dep"
-  analyzer-version = 1
-  input-imports = [
-    "bazil.org/fuse",
-    "bazil.org/fuse/fs",
-    "github.com/Azure/azure-storage-blob-go/2018-03-28/azblob",
-    "github.com/Unknwon/goconfig",
-    "github.com/VividCortex/ewma",
-    "github.com/a8m/tree",
-    "github.com/abbot/go-http-auth",
-    "github.com/aws/aws-sdk-go/aws",
-    "github.com/aws/aws-sdk-go/aws/awserr",
-    "github.com/aws/aws-sdk-go/aws/corehandlers",
-    "github.com/aws/aws-sdk-go/aws/credentials",
-    "github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds",
-    "github.com/aws/aws-sdk-go/aws/defaults",
-    "github.com/aws/aws-sdk-go/aws/ec2metadata",
-    "github.com/aws/aws-sdk-go/aws/request",
-    "github.com/aws/aws-sdk-go/aws/session",
-    "github.com/aws/aws-sdk-go/service/s3",
-    "github.com/aws/aws-sdk-go/service/s3/s3manager",
-    "github.com/billziss-gh/cgofuse/fuse",
-    "github.com/coreos/bbolt",
-    "github.com/djherbis/times",
-    "github.com/dropbox/dropbox-sdk-go-unofficial/dropbox",
-    "github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/common",
-    "github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/files",
-    "github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/sharing",
-    "github.com/dropbox/dropbox-sdk-go-unofficial/dropbox/users",
-    "github.com/jlaffaye/ftp",
-    "github.com/ncw/go-acd",
-    "github.com/ncw/swift",
-    "github.com/nsf/termbox-go",
-    "github.com/okzk/sdnotify",
-    "github.com/patrickmn/go-cache",
-    "github.com/pkg/errors",
-    "github.com/pkg/sftp",
-    "github.com/rfjakob/eme",
-    "github.com/sevlyar/go-daemon",
-    "github.com/skratchdot/open-golang/open",
-    "github.com/spf13/cobra",
-    "github.com/spf13/cobra/doc",
-    "github.com/spf13/pflag",
-    "github.com/stretchr/testify/assert",
-    "github.com/stretchr/testify/require",
-    "github.com/t3rm1n4l/go-mega",
-    "github.com/xanzy/ssh-agent",
-    "github.com/yunify/qingstor-sdk-go/config",
-    "github.com/yunify/qingstor-sdk-go/request/errors",
-    "github.com/yunify/qingstor-sdk-go/service",
-    "golang.org/x/crypto/nacl/secretbox",
-    "golang.org/x/crypto/scrypt",
-    "golang.org/x/crypto/ssh",
-    "golang.org/x/crypto/ssh/terminal",
-    "golang.org/x/net/context",
-    "golang.org/x/net/html",
-    "golang.org/x/net/http2",
-    "golang.org/x/net/publicsuffix",
-    "golang.org/x/net/webdav",
-    "golang.org/x/net/websocket",
-    "golang.org/x/oauth2",
-    "golang.org/x/oauth2/google",
-    "golang.org/x/sys/unix",
-    "golang.org/x/text/unicode/norm",
-    "golang.org/x/time/rate",
-    "google.golang.org/api/drive/v3",
-    "google.golang.org/api/googleapi",
-    "google.golang.org/api/storage/v1",
-  ]
-  solver-name = "gps-cdcl"
-  solver-version = 1

Gopkg.toml

@@ -1,15 +0,0 @@
-# pin this to master to pull in the macOS changes
-# can likely remove for 1.43
-[[override]]
-  branch = "master"
-  name = "github.com/sevlyar/go-daemon"
-
-# pin this to master to pull in the fix for linux/mips
-# can likely remove for 1.43
-[[override]]
-  branch = "master"
-  name = "github.com/coreos/bbolt"
-
-[[constraint]]
-  branch = "master"
-  name = "github.com/Azure/azure-storage-blob-go"

ISSUE_TEMPLATE.md

@@ -1,43 +0,0 @@
-<!--
-
-Hi!
-
-We understand you are having a problem with rclone or have an idea for an improvement - we want to help you with that!
-
-If you've just got a question or aren't sure if you've found a bug then please use the rclone forum
-
-    https://forum.rclone.org/
-
-instead of filing an issue.  We'll reply quickly and it won't increase our massive issue backlog.
-
-If you think you might have found a bug, please can you try to replicate it with the latest beta?
-
-    https://beta.rclone.org/
-    
-If you can still replicate it with the latest beta, then please fill in the info below which makes our lives much easier.  A log with -vv will make our day :-)
-
-If you have an idea for an improvement, then please search the old issues first and if you don't find your idea, make a new issue.
-
-Thanks
-
-The Rclone Developers
-
--->
-
-#### What is the problem you are having with rclone?
-
-
-#### What is your rclone version (eg output from `rclone -V`)
-
-
-#### Which OS you are using and how many bits (eg Windows 7, 64 bit)
-
-
-####  Which cloud storage system are you using? (eg Google Drive)
-
-
-#### The command you were trying to run (eg `rclone copy /tmp remote:tmp`)
-
-
-#### A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)
-

MAINTAINERS.md

@@ -7,6 +7,8 @@ Current active maintainers of rclone are
   * Ishuah Kariuki @ishuah
   * Remus Bunduc @remusb - cache subsystem maintainer
   * Fabian Möller @B4dM4n
+  * Alex Chen @Cnly
+  * Sandeep Ummadi @sandeepkru
 
 **This is a work in progress Draft**
 
@@ -56,7 +58,7 @@ Close tickets as soon as you can - make sure they are tagged with a release.  Po
 
 Try to process pull requests promptly!
 
-Merging pull requests on Github itself works quite well now-a-days so you can squash and rebase or rebase pull requests.  rclone doesn't use merge commits.  Use the squash and rebase option if you need to edit the commit message.
+Merging pull requests on GitHub itself works quite well now-a-days so you can squash and rebase or rebase pull requests.  rclone doesn't use merge commits.  Use the squash and rebase option if you need to edit the commit message.
 
 After merging the commit, in your local master branch, do `git pull` then run `bin/update-authors.py` to update the authors file then `git push`.
 

Makefile

@@ -1,5 +1,9 @@
 SHELL = bash
 BRANCH := $(or $(APPVEYOR_REPO_BRANCH),$(TRAVIS_BRANCH),$(shell git rev-parse --abbrev-ref HEAD))
+LAST_TAG := $(shell git describe --tags --abbrev=0)
+ifeq ($(BRANCH),$(LAST_TAG))
+	BRANCH := master
+endif
 TAG_BRANCH := -$(BRANCH)
 BRANCH_PATH := branch/
 ifeq ($(subst HEAD,,$(subst master,,$(BRANCH))),)
@@ -7,8 +11,10 @@ ifeq ($(subst HEAD,,$(subst master,,$(BRANCH))),)
 	BRANCH_PATH :=
 endif
 TAG := $(shell echo $$(git describe --abbrev=8 --tags | sed 's/-\([0-9]\)-/-00\1-/; s/-\([0-9][0-9]\)-/-0\1-/'))$(TAG_BRANCH)
-LAST_TAG := $(shell git describe --tags --abbrev=0)
 NEW_TAG := $(shell echo $(LAST_TAG) | perl -lpe 's/v//; $$_ += 0.01; $$_ = sprintf("v%.2f", $$_)')
+ifneq ($(TAG),$(LAST_TAG))
+	TAG := $(TAG)-beta
+endif
 GO_VERSION := $(shell go version)
 GO_FILES := $(shell go list ./... | grep -v /vendor/ )
 # Run full tests if go >= go1.11
@@ -44,10 +50,9 @@ version:
 
 # Full suite of integration tests
 test:	rclone
-	go install github.com/ncw/rclone/fstest/test_all
-	-go test -v -count 1 $(BUILDTAGS) $(GO_FILES) 2>&1 | tee test.log
-	-test_all github.com/ncw/rclone/fs/operations github.com/ncw/rclone/fs/sync 2>&1 | tee fs/test_all.log
-	@echo "Written logs in test.log and fs/test_all.log"
+	go install --ldflags "-s -X github.com/ncw/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/ncw/rclone/fstest/test_all
+	-test_all 2>&1 | tee test_all.log
+	@echo "Written logs in test_all.log"
 
 # Quick test
 quicktest:
@@ -82,8 +87,7 @@ build_dep:
 ifdef FULL_TESTS
 	go get -u github.com/kisielk/errcheck
 	go get -u golang.org/x/tools/cmd/goimports
-	go get -u github.com/golang/lint/golint
-	go get -u github.com/tools/godep
+	go get -u golang.org/x/lint/golint
 endif
 
 # Get the release dependencies
@@ -93,15 +97,16 @@ release_dep:
 
 # Update dependencies
 update:
-	go get -u github.com/golang/dep/cmd/dep
-	dep ensure -update -v
+	GO111MODULE=on go get -u ./...
+	GO111MODULE=on go mod tidy
+	GO111MODULE=on go mod vendor
 
-doc:	rclone.1 MANUAL.html MANUAL.txt
+doc:	rclone.1 MANUAL.html MANUAL.txt rcdocs commanddocs
 
 rclone.1:	MANUAL.md
 	pandoc -s --from markdown --to man MANUAL.md -o rclone.1
 
-MANUAL.md:	bin/make_manual.py docs/content/*.md commanddocs
+MANUAL.md:	bin/make_manual.py docs/content/*.md commanddocs backenddocs
 	./bin/make_manual.py
 
 MANUAL.html:	MANUAL.md
@@ -113,6 +118,9 @@ MANUAL.txt:	MANUAL.md
 commanddocs: rclone
 	rclone gendocs docs/content/commands/
 
+backenddocs: rclone bin/make_backend_docs.py
+	./bin/make_backend_docs.py
+
 rcdocs: rclone
 	bin/make_rc_docs.sh
 
@@ -146,8 +154,8 @@ check_sign:
 	cd build && gpg --verify SHA256SUMS && gpg --decrypt SHA256SUMS | sha256sum -c
 
 upload:
-	rclone -v copy --exclude '*current*' build/ memstore:downloads-rclone-org/$(TAG)
-	rclone -v copy --include '*current*' --include version.txt build/ memstore:downloads-rclone-org
+	rclone -P copy build/ memstore:downloads-rclone-org/$(TAG)
+	rclone lsf build --files-only --include '*.{zip,deb,rpm}' --include version.txt | xargs -i bash -c 'i={}; j="$$i"; [[ $$i =~ (.*)(-v[0-9\.]+-)(.*) ]] && j=$${BASH_REMATCH[1]}-current-$${BASH_REMATCH[3]}; rclone copyto -v "memstore:downloads-rclone-org/$(TAG)/$$i" "memstore:downloads-rclone-org/$$j"'
 
 upload_github:
 	./bin/upload-github $(TAG)
@@ -156,16 +164,16 @@ cross:	doc
 	go run bin/cross-compile.go -release current $(BUILDTAGS) $(TAG)
 
 beta:
-	go run bin/cross-compile.go $(BUILDTAGS) $(TAG)β
-	rclone -v copy build/ memstore:pub-rclone-org/$(TAG)β
-	@echo Beta release ready at https://pub.rclone.org/$(TAG)%CE%B2/
+	go run bin/cross-compile.go $(BUILDTAGS) $(TAG)
+	rclone -v copy build/ memstore:pub-rclone-org/$(TAG)
+	@echo Beta release ready at https://pub.rclone.org/$(TAG)/
 
 log_since_last_release:
 	git log $(LAST_TAG)..
 
 compile_all:
 ifdef FULL_TESTS
-	go run bin/cross-compile.go -parallel 8 -compile-only $(BUILDTAGS) $(TAG)β
+	go run bin/cross-compile.go -parallel 8 -compile-only $(BUILDTAGS) $(TAG)
 else
 	@echo Skipping compile all as version of go too old
 endif
@@ -187,19 +195,16 @@ ifeq ($(TRAVIS_OS_NAME),linux)
 	go run bin/get-github-release.go -extract nfpm goreleaser/nfpm 'nfpm_.*_Linux_x86_64.tar.gz'
 endif
 	git log $(LAST_TAG).. > /tmp/git-log.txt
-	go run bin/cross-compile.go -release beta-latest -git-log /tmp/git-log.txt $(BUILD_FLAGS) -parallel 8 $(BUILDTAGS) $(TAG)β
+	go run bin/cross-compile.go -release beta-latest -git-log /tmp/git-log.txt $(BUILD_FLAGS) -parallel 8 $(BUILDTAGS) $(TAG)
 	rclone --config bin/travis.rclone.conf -v copy --exclude '*beta-latest*' build/ $(BETA_UPLOAD)
 ifndef BRANCH_PATH
 	rclone --config bin/travis.rclone.conf -v copy --include '*beta-latest*' --include version.txt build/ $(BETA_UPLOAD_ROOT)
 endif
 	@echo Beta release ready at $(BETA_URL)
 
-# Fetch the windows builds from appveyor
-fetch_windows:
-	rclone -v copy --include 'rclone-v*-windows-*.zip' $(BETA_UPLOAD) build/
-	-#cp -av build/rclone-v*-windows-386.zip build/rclone-current-windows-386.zip
-	-#cp -av build/rclone-v*-windows-amd64.zip build/rclone-current-windows-amd64.zip
-	md5sum build/rclone-*-windows-*.zip | sort
+# Fetch the binary builds from travis and appveyor
+fetch_binaries:
+	rclone -P sync $(BETA_UPLOAD) build/
 
 serve:	website
 	cd docs && hugo server -v -w
@@ -210,10 +215,10 @@ tag:	doc
 	echo -e "package fs\n\n// Version of rclone\nvar Version = \"$(NEW_TAG)\"\n" | gofmt > fs/version.go
 	echo -n "$(NEW_TAG)" > docs/layouts/partials/version.html
 	git tag -s -m "Version $(NEW_TAG)" $(NEW_TAG)
+	bin/make_changelog.py $(LAST_TAG) $(NEW_TAG) > docs/content/changelog.md.new
+	mv docs/content/changelog.md.new docs/content/changelog.md
 	@echo "Edit the new changelog in docs/content/changelog.md"
-	@echo "  * $(NEW_TAG) -" `date -I` >> docs/content/changelog.md
-	@git log $(LAST_TAG)..$(NEW_TAG) --oneline >> docs/content/changelog.md
-	@echo "Then commit the changes"
+	@echo "Then commit all the changes"
 	@echo git commit -m \"Version $(NEW_TAG)\" -a -v
 	@echo "And finally run make retag before make cross etc"
 
@@ -226,4 +231,3 @@ startdev:
 
 winzip:
 	zip -9 rclone-$(TAG).zip rclone.exe
-

README.md

@@ -2,10 +2,11 @@
 
 [Website](https://rclone.org) |
 [Documentation](https://rclone.org/docs/) |
+[Download](https://rclone.org/downloads/) | 
 [Contributing](CONTRIBUTING.md) |
 [Changelog](https://rclone.org/changelog/) |
 [Installation](https://rclone.org/install/) |
-[Forum](https://forum.rclone.org/)
+[Forum](https://forum.rclone.org/) | 
 [G+](https://google.com/+RcloneOrg)
 
 [![Build Status](https://travis-ci.org/ncw/rclone.svg?branch=master)](https://travis-ci.org/ncw/rclone)
@@ -13,50 +14,81 @@
 [![CircleCI](https://circleci.com/gh/ncw/rclone/tree/master.svg?style=svg)](https://circleci.com/gh/ncw/rclone/tree/master)
 [![GoDoc](https://godoc.org/github.com/ncw/rclone?status.svg)](https://godoc.org/github.com/ncw/rclone) 
 
-Rclone is a command line program to sync files and directories to and from
+# Rclone
 
-  * Amazon Drive ([See note](https://rclone.org/amazonclouddrive/#status))
-  * Amazon S3 / Dreamhost / Ceph / Minio / Wasabi
-  * Backblaze B2
-  * Box
-  * Dropbox
-  * FTP
-  * Google Cloud Storage
-  * Google Drive
-  * HTTP
-  * Hubic
-  * Jottacloud
-  * Mega
-  * Microsoft Azure Blob Storage
-  * Microsoft OneDrive
-  * OpenDrive
-  * Openstack Swift / Rackspace cloud files / Memset Memstore / OVH / Oracle Cloud Storage
-  * pCloud
-  * QingStor
-  * SFTP
-  * Webdav / Owncloud / Nextcloud
-  * Yandex Disk
-  * The local filesystem
+Rclone *("rsync for cloud storage")* is a command line program to sync files and directories to and from different cloud storage providers.
 
-Features
+## Storage providers
+
+  * Amazon Drive [:page_facing_up:](https://rclone.org/amazonclouddrive/) ([See note](https://rclone.org/amazonclouddrive/#status))
+  * Amazon S3 [:page_facing_up:](https://rclone.org/s3/)
+  * Backblaze B2 [:page_facing_up:](https://rclone.org/b2/)
+  * Box [:page_facing_up:](https://rclone.org/box/)
+  * Ceph [:page_facing_up:](https://rclone.org/s3/#ceph)
+  * DigitalOcean Spaces [:page_facing_up:](https://rclone.org/s3/#digitalocean-spaces)
+  * Dreamhost [:page_facing_up:](https://rclone.org/s3/#dreamhost)
+  * Dropbox [:page_facing_up:](https://rclone.org/dropbox/)
+  * FTP [:page_facing_up:](https://rclone.org/ftp/)
+  * Google Cloud Storage [:page_facing_up:](https://rclone.org/googlecloudstorage/)
+  * Google Drive [:page_facing_up:](https://rclone.org/drive/)
+  * HTTP [:page_facing_up:](https://rclone.org/http/)
+  * Hubic [:page_facing_up:](https://rclone.org/hubic/)
+  * Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
+  * IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
+  * Memset Memstore [:page_facing_up:](https://rclone.org/swift/)
+  * Mega [:page_facing_up:](https://rclone.org/mega/)
+  * Microsoft Azure Blob Storage [:page_facing_up:](https://rclone.org/azureblob/)
+  * Microsoft OneDrive [:page_facing_up:](https://rclone.org/onedrive/)
+  * Minio [:page_facing_up:](https://rclone.org/s3/#minio)
+  * Nextcloud [:page_facing_up:](https://rclone.org/webdav/#nextcloud)
+  * OVH [:page_facing_up:](https://rclone.org/swift/)
+  * OpenDrive [:page_facing_up:](https://rclone.org/opendrive/)
+  * Openstack Swift [:page_facing_up:](https://rclone.org/swift/)
+  * Oracle Cloud Storage [:page_facing_up:](https://rclone.org/swift/)
+  * ownCloud [:page_facing_up:](https://rclone.org/webdav/#owncloud)
+  * pCloud [:page_facing_up:](https://rclone.org/pcloud/)
+  * put.io [:page_facing_up:](https://rclone.org/webdav/#put-io)
+  * QingStor [:page_facing_up:](https://rclone.org/qingstor/)
+  * Rackspace Cloud Files [:page_facing_up:](https://rclone.org/swift/)
+  * SFTP [:page_facing_up:](https://rclone.org/sftp/)
+  * Wasabi [:page_facing_up:](https://rclone.org/s3/#wasabi)
+  * WebDAV [:page_facing_up:](https://rclone.org/webdav/)
+  * Yandex Disk [:page_facing_up:](https://rclone.org/yandex/)
+  * The local filesystem [:page_facing_up:](https://rclone.org/local/)
+  
+Please see [the full list of all storage providers and their features](https://rclone.org/overview/)
+
+## Features
 
   * MD5/SHA1 hashes checked at all times for file integrity
   * Timestamps preserved on files
   * Partial syncs supported on a whole file basis
-  * Copy mode to just copy new/changed files
-  * Sync (one way) mode to make a directory identical
-  * Check mode to check for file hash equality
+  * [Copy](https://rclone.org/commands/rclone_copy/) mode to just copy new/changed files
+  * [Sync](https://rclone.org/commands/rclone_sync/) (one way) mode to make a directory identical
+  * [Check](https://rclone.org/commands/rclone_check/) mode to check for file hash equality
   * Can sync to and from network, eg two different cloud accounts
-  * Optional encryption (Crypt)
-  * Optional FUSE mount
+  * Optional encryption ([Crypt](https://rclone.org/crypt/))
+  * Optional cache ([Cache](https://rclone.org/cache/))
+  * Optional FUSE mount ([rclone mount](https://rclone.org/commands/rclone_mount/))
 
-See the home page for installation, usage, documentation, changelog
-and configuration walkthroughs.
+## Installation & documentation
 
-  * https://rclone.org/
+Please see the [rclone website](https://rclone.org/) for:
+
+  * [Installation](https://rclone.org/install/)
+  * [Documentation & configuration](https://rclone.org/docs/)
+  * [Changelog](https://rclone.org/changelog/)
+  * [FAQ](https://rclone.org/faq/)
+  * [Storage providers](https://rclone.org/overview/)
+  * [Forum](https://forum.rclone.org/)
+  * ...and more
+
+## Downloads
+
+  * https://rclone.org/downloads/
 
 License
 -------
 
 This is free software under the terms of MIT the license (check the
-COPYING file included in this package).
+[COPYING file](/rclone/COPYING) included in this package).

RELEASE.md

@@ -13,14 +13,9 @@ Making a release
   * git status - to check for new man pages - git add them
   * git commit -a -v -m "Version v1.XX"
   * make retag
-  * make release_dep
-  * # Set the GOPATH for a current stable go compiler
-  * make cross
-  * git checkout docs/content/commands # to undo date changes in commands
   * git push --tags origin master
-  * git push --tags origin master:stable # update the stable branch for packager.io
-  * # Wait for the appveyor and travis builds to complete then fetch the windows binaries from appveyor
-  * make fetch_windows
+  * # Wait for the appveyor and travis builds to complete then...
+  * make fetch_binaries
   * make tarball
   * make sign_upload
   * make check_sign
@@ -31,11 +26,45 @@ Making a release
   * # announce with forum post, twitter post, G+ post
 
 Early in the next release cycle update the vendored dependencies
-  * Review any pinned packages in Gopkg.toml and remove if possible
+  * Review any pinned packages in go.mod and remove if possible
   * make update
   * git status
   * git add new files
-  * carry forward any patches to vendor stuff
   * git commit -a -v
 
-Make the version number be just in a file?
+If `make update` fails with errors like this:
+
+```
+# github.com/cpuguy83/go-md2man/md2man
+../../../../pkg/mod/github.com/cpuguy83/go-md2man@v1.0.8/md2man/md2man.go:11:16: undefined: blackfriday.EXTENSION_NO_INTRA_EMPHASIS
+../../../../pkg/mod/github.com/cpuguy83/go-md2man@v1.0.8/md2man/md2man.go:12:16: undefined: blackfriday.EXTENSION_TABLES
+```
+
+Can be fixed with
+
+    * GO111MODULE=on go get -u github.com/russross/blackfriday@v1.5.2
+    * GO111MODULE=on go mod tidy
+    * GO111MODULE=on go mod vendor
+ 
+
+Making a point release.  If rclone needs a point release due to some
+horrendous bug, then
+  * git branch v1.XX v1.XX-fixes
+  * git cherry-pick any fixes
+  * Test (see above)
+  * make NEW_TAG=v1.XX.1 tag
+  * edit docs/content/changelog.md
+  * make TAG=v1.43.1 doc
+  * git commit -a -v -m "Version v1.XX.1"
+  * git tag -d -v1.XX.1
+  * git tag -s -m "Version v1.XX.1" v1.XX.1
+  * git push --tags -u origin v1.XX-fixes
+  * make BRANCH_PATH= TAG=v1.43.1 fetch_binaries
+  * make TAG=v1.43.1 tarball
+  * make TAG=v1.43.1 sign_upload
+  * make TAG=v1.43.1 check_sign
+  * make TAG=v1.43.1 upload
+  * make TAG=v1.43.1 upload_website
+  * make TAG=v1.43.1 upload_github
+  * NB this overwrites the current beta so after the release, rebuild the last travis build
+  * Announce!

backend/alias/alias.go

@@ -2,13 +2,12 @@ package alias
 
 import (
 	"errors"
-	"path"
-	"path/filepath"
 	"strings"
 
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/config/configmap"
 	"github.com/ncw/rclone/fs/config/configstruct"
+	"github.com/ncw/rclone/fs/fspath"
 )
 
 // Register with Fs
@@ -47,13 +46,9 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if strings.HasPrefix(opt.Remote, name+":") {
 		return nil, errors.New("can't point alias remote at itself - check the value of the remote setting")
 	}
-	_, configName, fsPath, err := fs.ParseRemote(opt.Remote)
+	fsInfo, configName, fsPath, config, err := fs.ConfigFs(opt.Remote)
 	if err != nil {
 		return nil, err
 	}
-	root = path.Join(fsPath, filepath.ToSlash(root))
-	if configName == "local" {
-		return fs.NewFs(root)
-	}
-	return fs.NewFs(configName + ":" + root)
+	return fsInfo.NewFs(configName, fspath.JoinRootPath(fsPath, root), config)
 }

backend/all/all.go

@@ -25,6 +25,7 @@ import (
 	_ "github.com/ncw/rclone/backend/s3"
 	_ "github.com/ncw/rclone/backend/sftp"
 	_ "github.com/ncw/rclone/backend/swift"
+	_ "github.com/ncw/rclone/backend/union"
 	_ "github.com/ncw/rclone/backend/webdav"
 	_ "github.com/ncw/rclone/backend/yandex"
 )

backend/amazonclouddrive/amazonclouddrive.go

@@ -97,13 +97,42 @@ func init() {
 			Hide:     fs.OptionHideBoth,
 			Advanced: true,
 		}, {
-			Name:     "upload_wait_per_gb",
-			Help:     "Additional time per GB to wait after a failed complete upload to see if it appears.",
+			Name: "upload_wait_per_gb",
+			Help: `Additional time per GB to wait after a failed complete upload to see if it appears.
+
+Sometimes Amazon Drive gives an error when a file has been fully
+uploaded but the file appears anyway after a little while.  This
+happens sometimes for files over 1GB in size and nearly every time for
+files bigger than 10GB. This parameter controls the time rclone waits
+for the file to appear.
+
+The default value for this parameter is 3 minutes per GB, so by
+default it will wait 3 minutes for every GB uploaded to see if the
+file appears.
+
+You can disable this feature by setting it to 0. This may cause
+conflict errors as rclone retries the failed upload but the file will
+most likely appear correctly eventually.
+
+These values were determined empirically by observing lots of uploads
+of big files for a range of file sizes.
+
+Upload with the "-v" flag to see more info about what rclone is doing
+in this situation.`,
 			Default:  fs.Duration(180 * time.Second),
 			Advanced: true,
 		}, {
-			Name:     "templink_threshold",
-			Help:     "Files >= this size will be downloaded via their tempLink.",
+			Name: "templink_threshold",
+			Help: `Files >= this size will be downloaded via their tempLink.
+
+Files this size or more will be downloaded via their "tempLink". This
+is to work around a problem with Amazon Drive which blocks downloads
+of files bigger than about 10GB.  The default for this is 9GB which
+shouldn't need to be changed.
+
+To download files above this threshold, rclone requests a "tempLink"
+which downloads the file through a temporary URL directly from the
+underlying S3 storage.`,
 			Default:  defaultTempLinkThreshold,
 			Advanced: true,
 		}},
@@ -283,16 +312,16 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		// Assume it is a file
 		newRoot, remote := dircache.SplitPath(root)
-		newF := *f
-		newF.dirCache = dircache.New(newRoot, f.trueRootID, &newF)
-		newF.root = newRoot
+		tempF := *f
+		tempF.dirCache = dircache.New(newRoot, f.trueRootID, &tempF)
+		tempF.root = newRoot
 		// Make new Fs which is the parent
-		err = newF.dirCache.FindRoot(false)
+		err = tempF.dirCache.FindRoot(false)
 		if err != nil {
 			// No root so return old f
 			return f, nil
 		}
-		_, err := newF.newObjectWithInfo(remote, nil)
+		_, err := tempF.newObjectWithInfo(remote, nil)
 		if err != nil {
 			if err == fs.ErrorObjectNotFound {
 				// File doesn't exist so return old f
@@ -300,8 +329,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 			}
 			return nil, err
 		}
+		// XXX: update the old f here instead of returning tempF, since
+		// `features` were already filled with functions having *f as a receiver.
+		// See https://github.com/ncw/rclone/issues/2182
+		f.dirCache = tempF.dirCache
+		f.root = tempF.root
 		// return an error with an fs which points to the parent
-		return &newF, fs.ErrorIsFile
+		return f, fs.ErrorIsFile
 	}
 	return f, nil
 }
@@ -1240,24 +1274,38 @@ func (o *Object) MimeType() string {
 // Automatically restarts itself in case of unexpected behaviour of the remote.
 //
 // Close the returned channel to stop being notified.
-func (f *Fs) ChangeNotify(notifyFunc func(string, fs.EntryType), pollInterval time.Duration) chan bool {
+func (f *Fs) ChangeNotify(notifyFunc func(string, fs.EntryType), pollIntervalChan <-chan time.Duration) {
 	checkpoint := f.opt.Checkpoint
 
-	quit := make(chan bool)
 	go func() {
+		var ticker *time.Ticker
+		var tickerC <-chan time.Time
 		for {
-			checkpoint = f.changeNotifyRunner(notifyFunc, checkpoint)
-			if err := config.SetValueAndSave(f.name, "checkpoint", checkpoint); err != nil {
-				fs.Debugf(f, "Unable to save checkpoint: %v", err)
-			}
 			select {
-			case <-quit:
-				return
-			case <-time.After(pollInterval):
+			case pollInterval, ok := <-pollIntervalChan:
+				if !ok {
+					if ticker != nil {
+						ticker.Stop()
+					}
+					return
+				}
+				if pollInterval == 0 {
+					if ticker != nil {
+						ticker.Stop()
+						ticker, tickerC = nil, nil
+					}
+				} else {
+					ticker = time.NewTicker(pollInterval)
+					tickerC = ticker.C
+				}
+			case <-tickerC:
+				checkpoint = f.changeNotifyRunner(notifyFunc, checkpoint)
+				if err := config.SetValueAndSave(f.name, "checkpoint", checkpoint); err != nil {
+					fs.Debugf(f, "Unable to save checkpoint: %v", err)
+				}
 			}
 		}
 	}()
-	return quit
 }
 
 func (f *Fs) changeNotifyRunner(notifyFunc func(string, fs.EntryType), checkpoint string) string {

backend/azureblob/azureblob.go

@@ -7,6 +7,7 @@ package azureblob
 import (
 	"bytes"
 	"context"
+	"crypto/md5"
 	"encoding/base64"
 	"encoding/binary"
 	"encoding/hex"
@@ -21,7 +22,7 @@ import (
 	"sync"
 	"time"
 
-	"github.com/Azure/azure-storage-blob-go/2018-03-28/azblob"
+	"github.com/Azure/azure-storage-blob-go/azblob"
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/accounting"
 	"github.com/ncw/rclone/fs/config/configmap"
@@ -37,17 +38,17 @@ const (
 	minSleep              = 10 * time.Millisecond
 	maxSleep              = 10 * time.Second
 	decayConstant         = 1    // bigger for slower decay, exponential
-	listChunkSize         = 5000 // number of items to read at once
+	maxListChunkSize      = 5000 // number of items to read at once
 	modTimeKey            = "mtime"
 	timeFormatIn          = time.RFC3339
 	timeFormatOut         = "2006-01-02T15:04:05.000000000Z07:00"
 	maxTotalParts         = 50000 // in multipart upload
 	storageDefaultBaseURL = "blob.core.windows.net"
 	// maxUncommittedSize = 9 << 30 // can't upload bigger than this
-	defaultChunkSize    = 4 * 1024 * 1024
-	maxChunkSize        = 100 * 1024 * 1024
-	defaultUploadCutoff = 256 * 1024 * 1024
-	maxUploadCutoff     = 256 * 1024 * 1024
+	defaultChunkSize    = 4 * fs.MebiByte
+	maxChunkSize        = 100 * fs.MebiByte
+	defaultUploadCutoff = 256 * fs.MebiByte
+	maxUploadCutoff     = 256 * fs.MebiByte
 	defaultAccessTier   = azblob.AccessTierNone
 )
 
@@ -72,18 +73,44 @@ func init() {
 			Advanced: true,
 		}, {
 			Name:     "upload_cutoff",
-			Help:     "Cutoff for switching to chunked upload.",
+			Help:     "Cutoff for switching to chunked upload (<= 256MB).",
 			Default:  fs.SizeSuffix(defaultUploadCutoff),
 			Advanced: true,
 		}, {
-			Name:     "chunk_size",
-			Help:     "Upload chunk size. Must fit in memory.",
+			Name: "chunk_size",
+			Help: `Upload chunk size (<= 100MB).
+
+Note that this is stored in memory and there may be up to
+"--transfers" chunks stored at once in memory.`,
 			Default:  fs.SizeSuffix(defaultChunkSize),
 			Advanced: true,
+		}, {
+			Name: "list_chunk",
+			Help: `Size of blob list.
+
+This sets the number of blobs requested in each listing chunk. Default
+is the maximum, 5000. "List blobs" requests are permitted 2 minutes
+per megabyte to complete. If an operation is taking longer than 2
+minutes per megabyte on average, it will time out (
+[source](https://docs.microsoft.com/en-us/rest/api/storageservices/setting-timeouts-for-blob-service-operations#exceptions-to-default-timeout-interval)
+). This can be used to limit the number of blobs items to return, to
+avoid the time out.`,
+			Default:  maxListChunkSize,
+			Advanced: true,
 		}, {
 			Name: "access_tier",
-			Help: "Access tier of blob, supports hot, cool and archive tiers.\nArchived blobs can be restored by setting access tier to hot or cool." +
-				" Leave blank if you intend to use default access tier, which is set at account level",
+			Help: `Access tier of blob: hot, cool or archive.
+
+Archived blobs can be restored by setting access tier to hot or
+cool. Leave blank if you intend to use default access tier, which is
+set at account level
+
+If there is no "access tier" specified, rclone doesn't apply any tier.
+rclone performs "Set Tier" operation on blobs while uploading, if objects
+are not modified, specifying "access tier" to new one will have no effect.
+If blobs are in "archive tier" at remote, trying to perform data transfer
+operations from remote will not be allowed. User should first restore by
+tiering blob to "Hot" or "Cool".`,
 			Advanced: true,
 		}},
 	})
@@ -91,13 +118,14 @@ func init() {
 
 // Options defines the configuration for this backend
 type Options struct {
-	Account      string        `config:"account"`
-	Key          string        `config:"key"`
-	Endpoint     string        `config:"endpoint"`
-	SASURL       string        `config:"sas_url"`
-	UploadCutoff fs.SizeSuffix `config:"upload_cutoff"`
-	ChunkSize    fs.SizeSuffix `config:"chunk_size"`
-	AccessTier   string        `config:"access_tier"`
+	Account       string        `config:"account"`
+	Key           string        `config:"key"`
+	Endpoint      string        `config:"endpoint"`
+	SASURL        string        `config:"sas_url"`
+	UploadCutoff  fs.SizeSuffix `config:"upload_cutoff"`
+	ChunkSize     fs.SizeSuffix `config:"chunk_size"`
+	ListChunkSize uint          `config:"list_chunk"`
+	AccessTier    string        `config:"access_tier"`
 }
 
 // Fs represents a remote azure server
@@ -157,7 +185,7 @@ func (f *Fs) Features() *fs.Features {
 }
 
 // Pattern to match a azure path
-var matcher = regexp.MustCompile(`^([^/]*)(.*)$`)
+var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
 
 // parseParse parses a azure 'url'
 func parsePath(path string) (container, directory string, err error) {
@@ -171,6 +199,19 @@ func parsePath(path string) (container, directory string, err error) {
 	return
 }
 
+// validateAccessTier checks if azureblob supports user supplied tier
+func validateAccessTier(tier string) bool {
+	switch tier {
+	case string(azblob.AccessTierHot),
+		string(azblob.AccessTierCool),
+		string(azblob.AccessTierArchive):
+		// valid cases
+		return true
+	default:
+		return false
+	}
+}
+
 // retryErrorCodes is a slice of error codes that we will retry
 var retryErrorCodes = []int{
 	401, // Unauthorized (eg "Token has expired")
@@ -196,6 +237,40 @@ func (f *Fs) shouldRetry(err error) (bool, error) {
 	return fserrors.ShouldRetry(err), err
 }
 
+func checkUploadChunkSize(cs fs.SizeSuffix) error {
+	const minChunkSize = fs.Byte
+	if cs < minChunkSize {
+		return errors.Errorf("%s is less than %s", cs, minChunkSize)
+	}
+	if cs > maxChunkSize {
+		return errors.Errorf("%s is greater than %s", cs, maxChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadChunkSize(cs)
+	if err == nil {
+		old, f.opt.ChunkSize = f.opt.ChunkSize, cs
+	}
+	return
+}
+
+func checkUploadCutoff(cs fs.SizeSuffix) error {
+	if cs > maxUploadCutoff {
+		return errors.Errorf("%v must be less than or equal to %v", cs, maxUploadCutoff)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadCutoff(cs)
+	if err == nil {
+		old, f.opt.UploadCutoff = f.opt.UploadCutoff, cs
+	}
+	return
+}
+
 // NewFs contstructs an Fs from the path, container:path
 func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	// Parse config into Options struct
@@ -205,11 +280,16 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 		return nil, err
 	}
 
-	if opt.UploadCutoff > maxUploadCutoff {
-		return nil, errors.Errorf("azure: upload cutoff (%v) must be less than or equal to %v", opt.UploadCutoff, maxUploadCutoff)
+	err = checkUploadCutoff(opt.UploadCutoff)
+	if err != nil {
+		return nil, errors.Wrap(err, "azure: upload cutoff")
 	}
-	if opt.ChunkSize > maxChunkSize {
-		return nil, errors.Errorf("azure: chunk size can't be greater than %v - was %v", maxChunkSize, opt.ChunkSize)
+	err = checkUploadChunkSize(opt.ChunkSize)
+	if err != nil {
+		return nil, errors.Wrap(err, "azure: chunk size")
+	}
+	if opt.ListChunkSize > maxListChunkSize {
+		return nil, errors.Errorf("azure: blob list size can't be greater than %v - was %v", maxListChunkSize, opt.ListChunkSize)
 	}
 	container, directory, err := parsePath(root)
 	if err != nil {
@@ -221,15 +301,9 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 
 	if opt.AccessTier == "" {
 		opt.AccessTier = string(defaultAccessTier)
-	} else {
-		switch opt.AccessTier {
-		case string(azblob.AccessTierHot):
-		case string(azblob.AccessTierCool):
-		case string(azblob.AccessTierArchive):
-			// valid cases
-		default:
-			return nil, errors.Errorf("azure: Supported access tiers are %s, %s and %s", string(azblob.AccessTierHot), string(azblob.AccessTierCool), azblob.AccessTierArchive)
-		}
+	} else if !validateAccessTier(opt.AccessTier) {
+		return nil, errors.Errorf("Azure Blob: Supported access tiers are %s, %s and %s",
+			string(azblob.AccessTierHot), string(azblob.AccessTierCool), string(azblob.AccessTierArchive))
 	}
 
 	var (
@@ -239,7 +313,11 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	)
 	switch {
 	case opt.Account != "" && opt.Key != "":
-		credential := azblob.NewSharedKeyCredential(opt.Account, opt.Key)
+		credential, err := azblob.NewSharedKeyCredential(opt.Account, opt.Key)
+		if err != nil {
+			return nil, errors.Wrapf(err, "Failed to parse credentials")
+		}
+
 		u, err = url.Parse(fmt.Sprintf("https://%s.%s", opt.Account, opt.Endpoint))
 		if err != nil {
 			return nil, errors.Wrap(err, "failed to make azure storage url from account and endpoint")
@@ -285,6 +363,8 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 		ReadMimeType:  true,
 		WriteMimeType: true,
 		BucketBased:   true,
+		SetTier:       true,
+		GetTier:       true,
 	}).Fill(f)
 	if f.root != "" {
 		f.root += "/"
@@ -474,7 +554,7 @@ func (f *Fs) markContainerOK() {
 
 // listDir lists a single directory
 func (f *Fs) listDir(dir string) (entries fs.DirEntries, err error) {
-	err = f.list(dir, false, listChunkSize, func(remote string, object *azblob.BlobItem, isDirectory bool) error {
+	err = f.list(dir, false, f.opt.ListChunkSize, func(remote string, object *azblob.BlobItem, isDirectory bool) error {
 		entry, err := f.itemToDirEntry(remote, object, isDirectory)
 		if err != nil {
 			return err
@@ -545,7 +625,7 @@ func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
 		return fs.ErrorListBucketRequired
 	}
 	list := walk.NewListRHelper(callback)
-	err = f.list(dir, true, listChunkSize, func(remote string, object *azblob.BlobItem, isDirectory bool) error {
+	err = f.list(dir, true, f.opt.ListChunkSize, func(remote string, object *azblob.BlobItem, isDirectory bool) error {
 		entry, err := f.itemToDirEntry(remote, object, isDirectory)
 		if err != nil {
 			return err
@@ -566,11 +646,11 @@ type listContainerFn func(*azblob.ContainerItem) error
 // listContainersToFn lists the containers to the function supplied
 func (f *Fs) listContainersToFn(fn listContainerFn) error {
 	params := azblob.ListContainersSegmentOptions{
-		MaxResults: int32(listChunkSize),
+		MaxResults: int32(f.opt.ListChunkSize),
 	}
 	ctx := context.Background()
 	for marker := (azblob.Marker{}); marker.NotDone(); {
-		var response *azblob.ListContainersResponse
+		var response *azblob.ListContainersSegmentResponse
 		err := f.pacer.Call(func() (bool, error) {
 			var err error
 			response, err = f.svcURL.ListContainersSegment(ctx, marker, params)
@@ -752,7 +832,7 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
 	var startCopy *azblob.BlobStartCopyFromURLResponse
 
 	err = f.pacer.Call(func() (bool, error) {
-		startCopy, err = dstBlobURL.StartCopyFromURL(ctx, *source, nil, options, options)
+		startCopy, err = dstBlobURL.StartCopyFromURL(ctx, *source, nil, azblob.ModifiedAccessConditions{}, options)
 		return f.shouldRetry(err)
 	})
 	if err != nil {
@@ -837,9 +917,8 @@ func (o *Object) setMetadata(metadata azblob.Metadata) {
 //  o.md5
 //  o.meta
 func (o *Object) decodeMetaDataFromPropertiesResponse(info *azblob.BlobGetPropertiesResponse) (err error) {
-	// NOTE - In BlobGetPropertiesResponse, Client library returns MD5 as base64 decoded string
-	// unlike BlobProperties in BlobItem (used in decodeMetadataFromBlob) which returns base64
-	// encoded bytes. Object needs to maintain this as base64 encoded string.
+	// NOTE - Client library always returns MD5 as base64 decoded string, Object needs to maintain
+	// this as base64 encoded string.
 	o.md5 = base64.StdEncoding.EncodeToString(info.ContentMD5())
 	o.mimeType = info.ContentType()
 	o.size = info.ContentLength()
@@ -851,7 +930,9 @@ func (o *Object) decodeMetaDataFromPropertiesResponse(info *azblob.BlobGetProper
 }
 
 func (o *Object) decodeMetaDataFromBlob(info *azblob.BlobItem) (err error) {
-	o.md5 = string(info.Properties.ContentMD5)
+	// NOTE - Client library always returns MD5 as base64 decoded string, Object needs to maintain
+	// this as base64 encoded string.
+	o.md5 = base64.StdEncoding.EncodeToString(info.Properties.ContentMD5)
 	o.mimeType = *info.Properties.ContentType
 	o.size = *info.Properties.ContentLength
 	o.modTime = info.Properties.LastModified
@@ -1068,7 +1149,7 @@ func (o *Object) uploadMultipart(in io.Reader, size int64, blob *azblob.BlobURL,
 	var (
 		rawID   uint64
 		blockID = "" // id in base64 encoded form
-		blocks  = make([]string, totalParts)
+		blocks  []string
 	)
 
 	// increment the blockID
@@ -1125,11 +1206,14 @@ outer:
 			defer o.fs.uploadToken.Put()
 			fs.Debugf(o, "Uploading part %d/%d offset %v/%v part size %v", part+1, totalParts, fs.SizeSuffix(position), fs.SizeSuffix(size), fs.SizeSuffix(chunkSize))
 
+			// Upload the block, with MD5 for check
+			md5sum := md5.Sum(buf)
+			transactionalMD5 := md5sum[:]
 			err = o.fs.pacer.Call(func() (bool, error) {
 				bufferReader := bytes.NewReader(buf)
 				wrappedReader := wrap(bufferReader)
 				rs := readSeeker{wrappedReader, bufferReader}
-				_, err = blockBlobURL.StageBlock(ctx, blockID, rs, ac)
+				_, err = blockBlobURL.StageBlock(ctx, blockID, &rs, ac, transactionalMD5)
 				return o.fs.shouldRetry(err)
 			})
 
@@ -1206,11 +1290,20 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 		Metadata:        o.meta,
 		BlobHTTPHeaders: httpHeaders,
 	}
+	// FIXME Until https://github.com/Azure/azure-storage-blob-go/pull/75
+	// is merged the SDK can't upload a single blob of exactly the chunk
+	// size, so upload with a multpart upload to work around.
+	// See: https://github.com/ncw/rclone/issues/2653
+	multipartUpload := size >= int64(o.fs.opt.UploadCutoff)
+	if size == int64(o.fs.opt.ChunkSize) {
+		multipartUpload = true
+		fs.Debugf(o, "Setting multipart upload for file of chunk size (%d) to work around SDK bug", size)
+	}
 
 	ctx := context.Background()
 	// Don't retry, return a retry error instead
 	err = o.fs.pacer.CallNoRetry(func() (bool, error) {
-		if size >= int64(o.fs.opt.UploadCutoff) {
+		if multipartUpload {
 			// If a large file upload in chunks
 			err = o.uploadMultipart(in, size, &blob, &httpHeaders)
 		} else {
@@ -1236,16 +1329,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	}
 
 	// Now, set blob tier based on configured access tier
-	desiredAccessTier := azblob.AccessTierType(o.fs.opt.AccessTier)
-	err = o.fs.pacer.Call(func() (bool, error) {
-		_, err := blob.SetTier(ctx, desiredAccessTier)
-		return o.fs.shouldRetry(err)
-	})
-
-	if err != nil {
-		return errors.Wrap(err, "Failed to set Blob Tier")
-	}
-	return nil
+	return o.SetTier(o.fs.opt.AccessTier)
 }
 
 // Remove an object
@@ -1270,6 +1354,41 @@ func (o *Object) AccessTier() azblob.AccessTierType {
 	return o.accessTier
 }
 
+// SetTier performs changing object tier
+func (o *Object) SetTier(tier string) error {
+	if !validateAccessTier(tier) {
+		return errors.Errorf("Tier %s not supported by Azure Blob Storage", tier)
+	}
+
+	// Check if current tier already matches with desired tier
+	if o.GetTier() == tier {
+		return nil
+	}
+	desiredAccessTier := azblob.AccessTierType(tier)
+	blob := o.getBlobReference()
+	ctx := context.Background()
+	err := o.fs.pacer.Call(func() (bool, error) {
+		_, err := blob.SetTier(ctx, desiredAccessTier, azblob.LeaseAccessConditions{})
+		return o.fs.shouldRetry(err)
+	})
+
+	if err != nil {
+		return errors.Wrap(err, "Failed to set Blob Tier")
+	}
+
+	// Set access tier on local object also, this typically
+	// gets updated on get blob properties
+	o.accessTier = desiredAccessTier
+	fs.Debugf(o, "Successfully changed object tier to %s", tier)
+
+	return nil
+}
+
+// GetTier returns object tier in azure as string
+func (o *Object) GetTier() string {
+	return string(o.accessTier)
+}
+
 // Check the interfaces are satisfied
 var (
 	_ fs.Fs        = &Fs{}

backend/azureblob/azureblob_internal_test.go

@@ -0,0 +1,18 @@
+// +build !freebsd,!netbsd,!openbsd,!plan9,!solaris,go1.8
+
+package azureblob
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func (f *Fs) InternalTest(t *testing.T) {
+	// Check first feature flags are set on this
+	// remote
+	enabled := f.Features().SetTier
+	assert.True(t, enabled)
+	enabled = f.Features().GetTier
+	assert.True(t, enabled)
+}

backend/azureblob/azureblob_test.go

@@ -2,19 +2,36 @@
 
 // +build !freebsd,!netbsd,!openbsd,!plan9,!solaris,go1.8
 
-package azureblob_test
+package azureblob
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/azureblob"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
 // TestIntegration runs integration tests against the remote
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
-		RemoteName: "TestAzureBlob:",
-		NilObject:  (*azureblob.Object)(nil),
+		RemoteName:  "TestAzureBlob:",
+		NilObject:   (*Object)(nil),
+		TiersToTest: []string{"Hot", "Cool"},
+		ChunkedUpload: fstests.ChunkedUploadConfig{
+			MaxChunkSize: maxChunkSize,
+		},
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+func (f *Fs) SetUploadCutoff(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadCutoff(cs)
+}
+
+var (
+	_ fstests.SetUploadChunkSizer = (*Fs)(nil)
+	_ fstests.SetUploadCutoffer   = (*Fs)(nil)
+)

backend/b2/b2.go

@@ -48,9 +48,9 @@ const (
 	decayConstant       = 1 // bigger for slower decay, exponential
 	maxParts            = 10000
 	maxVersions         = 100 // maximum number of versions we search in --b2-versions mode
-	minChunkSize        = 5E6
-	defaultChunkSize    = 96 * 1024 * 1024
-	defaultUploadCutoff = 200E6
+	minChunkSize        = 5 * fs.MebiByte
+	defaultChunkSize    = 96 * fs.MebiByte
+	defaultUploadCutoff = 200 * fs.MebiByte
 )
 
 // Globals
@@ -77,14 +77,24 @@ func init() {
 			Help:     "Endpoint for the service.\nLeave blank normally.",
 			Advanced: true,
 		}, {
-			Name:     "test_mode",
-			Help:     "A flag string for X-Bz-Test-Mode header for debugging.",
+			Name: "test_mode",
+			Help: `A flag string for X-Bz-Test-Mode header for debugging.
+
+This is for debugging purposes only. Setting it to one of the strings
+below will cause b2 to return specific errors:
+
+  * "fail_some_uploads"
+  * "expire_some_account_authorization_tokens"
+  * "force_cap_exceeded"
+
+These will be set in the "X-Bz-Test-Mode" header which is documented
+in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).`,
 			Default:  "",
 			Hide:     fs.OptionHideConfigurator,
 			Advanced: true,
 		}, {
 			Name:     "versions",
-			Help:     "Include old versions in directory listings.",
+			Help:     "Include old versions in directory listings.\nNote that when using this no file write operations are permitted,\nso you can't upload files or delete them.",
 			Default:  false,
 			Advanced: true,
 		}, {
@@ -92,13 +102,22 @@ func init() {
 			Help:    "Permanently delete files on remote removal, otherwise hide files.",
 			Default: false,
 		}, {
-			Name:     "upload_cutoff",
-			Help:     "Cutoff for switching to chunked upload.",
+			Name: "upload_cutoff",
+			Help: `Cutoff for switching to chunked upload.
+
+Files above this size will be uploaded in chunks of "--b2-chunk-size".
+
+This value should be set no larger than 4.657GiB (== 5GB).`,
 			Default:  fs.SizeSuffix(defaultUploadCutoff),
 			Advanced: true,
 		}, {
-			Name:     "chunk_size",
-			Help:     "Upload chunk size. Must fit in memory.",
+			Name: "chunk_size",
+			Help: `Upload chunk size. Must fit in memory.
+
+When uploading large files, chunk the file into this size.  Note that
+these chunks are buffered in memory and there might a maximum of
+"--transfers" chunks in progress at once.  5,000,000 Bytes is the
+minimim size.`,
 			Default:  fs.SizeSuffix(defaultChunkSize),
 			Advanced: true,
 		}},
@@ -177,7 +196,7 @@ func (f *Fs) Features() *fs.Features {
 }
 
 // Pattern to match a b2 path
-var matcher = regexp.MustCompile(`^([^/]*)(.*)$`)
+var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
 
 // parseParse parses a b2 'url'
 func parsePath(path string) (bucket, directory string, err error) {
@@ -263,6 +282,37 @@ func errorHandler(resp *http.Response) error {
 	return errResponse
 }
 
+func checkUploadChunkSize(cs fs.SizeSuffix) error {
+	if cs < minChunkSize {
+		return errors.Errorf("%s is less than %s", cs, minChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadChunkSize(cs)
+	if err == nil {
+		old, f.opt.ChunkSize = f.opt.ChunkSize, cs
+		f.fillBufferTokens() // reset the buffer tokens
+	}
+	return
+}
+
+func checkUploadCutoff(opt *Options, cs fs.SizeSuffix) error {
+	if cs < opt.ChunkSize {
+		return errors.Errorf("%v is less than chunk size %v", cs, opt.ChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadCutoff(&f.opt, cs)
+	if err == nil {
+		old, f.opt.UploadCutoff = f.opt.UploadCutoff, cs
+	}
+	return
+}
+
 // NewFs contstructs an Fs from the path, bucket:path
 func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	// Parse config into Options struct
@@ -271,11 +321,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		return nil, err
 	}
-	if opt.UploadCutoff < opt.ChunkSize {
-		return nil, errors.Errorf("b2: upload cutoff (%v) must be greater than or equal to chunk size (%v)", opt.UploadCutoff, opt.ChunkSize)
+	err = checkUploadCutoff(opt, opt.UploadCutoff)
+	if err != nil {
+		return nil, errors.Wrap(err, "b2: upload cutoff")
 	}
-	if opt.ChunkSize < minChunkSize {
-		return nil, errors.Errorf("b2: chunk size can't be less than %v - was %v", minChunkSize, opt.ChunkSize)
+	err = checkUploadChunkSize(opt.ChunkSize)
+	if err != nil {
+		return nil, errors.Wrap(err, "b2: chunk size")
 	}
 	bucket, directory, err := parsePath(root)
 	if err != nil {
@@ -291,13 +343,12 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 		opt.Endpoint = defaultEndpoint
 	}
 	f := &Fs{
-		name:         name,
-		opt:          *opt,
-		bucket:       bucket,
-		root:         directory,
-		srv:          rest.NewClient(fshttp.NewClient(fs.Config)).SetErrorHandler(errorHandler),
-		pacer:        pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
-		bufferTokens: make(chan []byte, fs.Config.Transfers),
+		name:   name,
+		opt:    *opt,
+		bucket: bucket,
+		root:   directory,
+		srv:    rest.NewClient(fshttp.NewClient(fs.Config)).SetErrorHandler(errorHandler),
+		pacer:  pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
 	}
 	f.features = (&fs.Features{
 		ReadMimeType:  true,
@@ -310,10 +361,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 		f.srv.SetHeader(testModeHeader, testMode)
 		fs.Debugf(f, "Setting test header \"%s: %s\"", testModeHeader, testMode)
 	}
-	// Fill up the buffer tokens
-	for i := 0; i < fs.Config.Transfers; i++ {
-		f.bufferTokens <- nil
-	}
+	f.fillBufferTokens()
 	err = f.authorizeAccount()
 	if err != nil {
 		return nil, errors.Wrap(err, "failed to authorize account")
@@ -421,6 +469,14 @@ func (f *Fs) clearUploadURL() {
 	f.uploadMu.Unlock()
 }
 
+// Fill up (or reset) the buffer tokens
+func (f *Fs) fillBufferTokens() {
+	f.bufferTokens = make(chan []byte, fs.Config.Transfers)
+	for i := 0; i < fs.Config.Transfers; i++ {
+		f.bufferTokens <- nil
+	}
+}
+
 // getUploadBlock gets a block from the pool of size chunkSize
 func (f *Fs) getUploadBlock() []byte {
 	buf := <-f.bufferTokens

backend/b2/b2_test.go

@@ -1,10 +1,10 @@
 // Test B2 filesystem interface
-package b2_test
+package b2
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/b2"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
@@ -12,6 +12,23 @@ import (
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
 		RemoteName: "TestB2:",
-		NilObject:  (*b2.Object)(nil),
+		NilObject:  (*Object)(nil),
+		ChunkedUpload: fstests.ChunkedUploadConfig{
+			MinChunkSize:       minChunkSize,
+			NeedMultipleChunks: true,
+		},
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+func (f *Fs) SetUploadCutoff(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadCutoff(cs)
+}
+
+var (
+	_ fstests.SetUploadChunkSizer = (*Fs)(nil)
+	_ fstests.SetUploadCutoffer   = (*Fs)(nil)
+)

backend/box/api/types.go

@@ -61,7 +61,7 @@ func (e *Error) Error() string {
 var _ error = (*Error)(nil)
 
 // ItemFields are the fields needed for FileInfo
-var ItemFields = "type,id,sequence_id,etag,sha1,name,size,created_at,modified_at,content_created_at,content_modified_at,item_status"
+var ItemFields = "type,id,sequence_id,etag,sha1,name,size,created_at,modified_at,content_created_at,content_modified_at,item_status,shared_link"
 
 // Types of things in Item
 const (
@@ -86,6 +86,10 @@ type Item struct {
 	ContentCreatedAt  Time    `json:"content_created_at"`
 	ContentModifiedAt Time    `json:"content_modified_at"`
 	ItemStatus        string  `json:"item_status"` // active, trashed if the file has been moved to the trash, and deleted if the file has been permanently deleted
+	SharedLink        struct {
+		URL    string `json:"url,omitempty"`
+		Access string `json:"access,omitempty"`
+	} `json:"shared_link"`
 }
 
 // ModTime returns the modification time of the item
@@ -145,6 +149,14 @@ type CopyFile struct {
 	Parent Parent `json:"parent"`
 }
 
+// CreateSharedLink is the request for Public Link
+type CreateSharedLink struct {
+	SharedLink struct {
+		URL    string `json:"url,omitempty"`
+		Access string `json:"access,omitempty"`
+	} `json:"shared_link"`
+}
+
 // UploadSessionRequest is uses in Create Upload Session
 type UploadSessionRequest struct {
 	FolderID string `json:"folder_id,omitempty"` // don't pass for update

backend/box/box.go

@@ -85,7 +85,7 @@ func init() {
 			Help: "Box App Client Secret\nLeave blank normally.",
 		}, {
 			Name:     "upload_cutoff",
-			Help:     "Cutoff for switching to multipart upload.",
+			Help:     "Cutoff for switching to multipart upload (>= 50MB).",
 			Default:  fs.SizeSuffix(defaultUploadCutoff),
 			Advanced: true,
 		}, {
@@ -126,6 +126,7 @@ type Object struct {
 	size        int64     // size of the object
 	modTime     time.Time // modification time of the object
 	id          string    // ID of the object
+	publicLink  string    // Public Link for the object
 	sha1        string    // SHA-1 of the object content
 }
 
@@ -282,16 +283,16 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		// Assume it is a file
 		newRoot, remote := dircache.SplitPath(root)
-		newF := *f
-		newF.dirCache = dircache.New(newRoot, rootID, &newF)
-		newF.root = newRoot
+		tempF := *f
+		tempF.dirCache = dircache.New(newRoot, rootID, &tempF)
+		tempF.root = newRoot
 		// Make new Fs which is the parent
-		err = newF.dirCache.FindRoot(false)
+		err = tempF.dirCache.FindRoot(false)
 		if err != nil {
 			// No root so return old f
 			return f, nil
 		}
-		_, err := newF.newObjectWithInfo(remote, nil)
+		_, err := tempF.newObjectWithInfo(remote, nil)
 		if err != nil {
 			if err == fs.ErrorObjectNotFound {
 				// File doesn't exist so return old f
@@ -299,8 +300,14 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 			}
 			return nil, err
 		}
+		f.features.Fill(&tempF)
+		// XXX: update the old f here instead of returning tempF, since
+		// `features` were already filled with functions having *f as a receiver.
+		// See https://github.com/ncw/rclone/issues/2182
+		f.dirCache = tempF.dirCache
+		f.root = tempF.root
 		// return an error with an fs which points to the parent
-		return &newF, fs.ErrorIsFile
+		return f, fs.ErrorIsFile
 	}
 	return f, nil
 }
@@ -844,6 +851,46 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
 	return nil
 }
 
+// PublicLink adds a "readable by anyone with link" permission on the given file or folder.
+func (f *Fs) PublicLink(remote string) (string, error) {
+	id, err := f.dirCache.FindDir(remote, false)
+	var opts rest.Opts
+	if err == nil {
+		fs.Debugf(f, "attempting to share directory '%s'", remote)
+
+		opts = rest.Opts{
+			Method:     "PUT",
+			Path:       "/folders/" + id,
+			Parameters: fieldsValue(),
+		}
+	} else {
+		fs.Debugf(f, "attempting to share single file '%s'", remote)
+		o, err := f.NewObject(remote)
+		if err != nil {
+			return "", err
+		}
+
+		if o.(*Object).publicLink != "" {
+			return o.(*Object).publicLink, nil
+		}
+
+		opts = rest.Opts{
+			Method:     "PUT",
+			Path:       "/files/" + o.(*Object).id,
+			Parameters: fieldsValue(),
+		}
+	}
+
+	shareLink := api.CreateSharedLink{}
+	var info api.Item
+	var resp *http.Response
+	err = f.pacer.Call(func() (bool, error) {
+		resp, err = f.srv.CallJSON(&opts, &shareLink, &info)
+		return shouldRetry(resp, err)
+	})
+	return info.SharedLink.URL, err
+}
+
 // DirCacheFlush resets the directory cache - used in testing as an
 // optional interface
 func (f *Fs) DirCacheFlush() {
@@ -908,6 +955,7 @@ func (o *Object) setMetaData(info *api.Item) (err error) {
 	o.sha1 = info.SHA1
 	o.modTime = info.ModTime()
 	o.id = info.ID
+	o.publicLink = info.SharedLink.URL
 	return nil
 }
 
@@ -1087,6 +1135,7 @@ var (
 	_ fs.Mover           = (*Fs)(nil)
 	_ fs.DirMover        = (*Fs)(nil)
 	_ fs.DirCacheFlusher = (*Fs)(nil)
+	_ fs.PublicLinker    = (*Fs)(nil)
 	_ fs.Object          = (*Object)(nil)
 	_ fs.IDer            = (*Object)(nil)
 )

backend/cache/cache.go

@@ -6,10 +6,13 @@ import (
 	"context"
 	"fmt"
 	"io"
+	"math"
 	"os"
 	"os/signal"
 	"path"
 	"path/filepath"
+	"sort"
+	"strconv"
 	"strings"
 	"sync"
 	"syscall"
@@ -21,6 +24,7 @@ import (
 	"github.com/ncw/rclone/fs/config/configmap"
 	"github.com/ncw/rclone/fs/config/configstruct"
 	"github.com/ncw/rclone/fs/config/obscure"
+	"github.com/ncw/rclone/fs/fspath"
 	"github.com/ncw/rclone/fs/hash"
 	"github.com/ncw/rclone/fs/rc"
 	"github.com/ncw/rclone/fs/walk"
@@ -80,8 +84,16 @@ func init() {
 			Hide:     fs.OptionHideBoth,
 			Advanced: true,
 		}, {
-			Name:    "chunk_size",
-			Help:    "The size of a chunk. Lower value good for slow connections but can affect seamless reading.",
+			Name:     "plex_insecure",
+			Help:     "Skip all certificate verifications when connecting to the Plex server",
+			Advanced: true,
+		}, {
+			Name: "chunk_size",
+			Help: `The size of a chunk (partial file data).
+
+Use lower numbers for slower connections. If the chunk size is
+changed, any downloaded chunks will be invalid and cache-chunk-path
+will need to be cleared or unexpected EOF errors will occur.`,
 			Default: DefCacheChunkSize,
 			Examples: []fs.OptionExample{{
 				Value: "1m",
@@ -94,8 +106,10 @@ func init() {
 				Help:  "10 MB",
 			}},
 		}, {
-			Name:    "info_age",
-			Help:    "How much time should object info (file size, file hashes etc) be stored in cache.\nUse a very high value if you don't plan on changing the source FS from outside the cache.\nAccepted units are: \"s\", \"m\", \"h\".",
+			Name: "info_age",
+			Help: `How long to cache file structure information (directory listings, file size, times etc). 
+If all write operations are done through the cache then you can safely make
+this value very large as the cache store will also be updated in real time.`,
 			Default: DefCacheInfoAge,
 			Examples: []fs.OptionExample{{
 				Value: "1h",
@@ -108,8 +122,11 @@ func init() {
 				Help:  "48 hours",
 			}},
 		}, {
-			Name:    "chunk_total_size",
-			Help:    "The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted.",
+			Name: "chunk_total_size",
+			Help: `The total size that the chunks can take up on the local disk.
+
+If the cache exceeds this value then it will start to delete the
+oldest chunks until it goes under this value.`,
 			Default: DefCacheTotalChunkSize,
 			Examples: []fs.OptionExample{{
 				Value: "500M",
@@ -124,63 +141,143 @@ func init() {
 		}, {
 			Name:     "db_path",
 			Default:  filepath.Join(config.CacheDir, "cache-backend"),
-			Help:     "Directory to cache DB",
+			Help:     "Directory to store file structure metadata DB.\nThe remote name is used as the DB file name.",
 			Advanced: true,
 		}, {
-			Name:     "chunk_path",
-			Default:  filepath.Join(config.CacheDir, "cache-backend"),
-			Help:     "Directory to cache chunk files",
+			Name:    "chunk_path",
+			Default: filepath.Join(config.CacheDir, "cache-backend"),
+			Help: `Directory to cache chunk files.
+
+Path to where partial file data (chunks) are stored locally. The remote
+name is appended to the final path.
+
+This config follows the "--cache-db-path". If you specify a custom
+location for "--cache-db-path" and don't specify one for "--cache-chunk-path"
+then "--cache-chunk-path" will use the same path as "--cache-db-path".`,
 			Advanced: true,
 		}, {
 			Name:     "db_purge",
 			Default:  false,
-			Help:     "Purge the cache DB before",
+			Help:     "Clear all the cached data for this remote on start.",
 			Hide:     fs.OptionHideConfigurator,
 			Advanced: true,
 		}, {
-			Name:     "chunk_clean_interval",
-			Default:  DefCacheChunkCleanInterval,
-			Help:     "Interval at which chunk cleanup runs",
+			Name:    "chunk_clean_interval",
+			Default: DefCacheChunkCleanInterval,
+			Help: `How often should the cache perform cleanups of the chunk storage.
+The default value should be ok for most people. If you find that the
+cache goes over "cache-chunk-total-size" too often then try to lower
+this value to force it to perform cleanups more often.`,
 			Advanced: true,
 		}, {
-			Name:     "read_retries",
-			Default:  DefCacheReadRetries,
-			Help:     "How many times to retry a read from a cache storage",
+			Name:    "read_retries",
+			Default: DefCacheReadRetries,
+			Help: `How many times to retry a read from a cache storage.
+
+Since reading from a cache stream is independent from downloading file
+data, readers can get to a point where there's no more data in the
+cache.  Most of the times this can indicate a connectivity issue if
+cache isn't able to provide file data anymore.
+
+For really slow connections, increase this to a point where the stream is
+able to provide data but your experience will be very stuttering.`,
 			Advanced: true,
 		}, {
-			Name:     "workers",
-			Default:  DefCacheTotalWorkers,
-			Help:     "How many workers should run in parallel to download chunks",
+			Name:    "workers",
+			Default: DefCacheTotalWorkers,
+			Help: `How many workers should run in parallel to download chunks.
+
+Higher values will mean more parallel processing (better CPU needed)
+and more concurrent requests on the cloud provider.  This impacts
+several aspects like the cloud provider API limits, more stress on the
+hardware that rclone runs on but it also means that streams will be
+more fluid and data will be available much more faster to readers.
+
+**Note**: If the optional Plex integration is enabled then this
+setting will adapt to the type of reading performed and the value
+specified here will be used as a maximum number of workers to use.`,
 			Advanced: true,
 		}, {
-			Name:     "chunk_no_memory",
-			Default:  DefCacheChunkNoMemory,
-			Help:     "Disable the in-memory cache for storing chunks during streaming",
+			Name:    "chunk_no_memory",
+			Default: DefCacheChunkNoMemory,
+			Help: `Disable the in-memory cache for storing chunks during streaming.
+
+By default, cache will keep file data during streaming in RAM as well
+to provide it to readers as fast as possible.
+
+This transient data is evicted as soon as it is read and the number of
+chunks stored doesn't exceed the number of workers. However, depending
+on other settings like "cache-chunk-size" and "cache-workers" this footprint
+can increase if there are parallel streams too (multiple files being read
+at the same time).
+
+If the hardware permits it, use this feature to provide an overall better
+performance during streaming but it can also be disabled if RAM is not
+available on the local machine.`,
 			Advanced: true,
 		}, {
-			Name:     "rps",
-			Default:  int(DefCacheRps),
-			Help:     "Limits the number of requests per second to the source FS. -1 disables the rate limiter",
+			Name:    "rps",
+			Default: int(DefCacheRps),
+			Help: `Limits the number of requests per second to the source FS (-1 to disable)
+
+This setting places a hard limit on the number of requests per second
+that cache will be doing to the cloud provider remote and try to
+respect that value by setting waits between reads.
+
+If you find that you're getting banned or limited on the cloud
+provider through cache and know that a smaller number of requests per
+second will allow you to work with it then you can use this setting
+for that.
+
+A good balance of all the other settings should make this setting
+useless but it is available to set for more special cases.
+
+**NOTE**: This will limit the number of requests during streams but
+other API calls to the cloud provider like directory listings will
+still pass.`,
 			Advanced: true,
 		}, {
-			Name:     "writes",
-			Default:  DefCacheWrites,
-			Help:     "Will cache file data on writes through the FS",
+			Name:    "writes",
+			Default: DefCacheWrites,
+			Help: `Cache file data on writes through the FS
+
+If you need to read files immediately after you upload them through
+cache you can enable this flag to have their data stored in the
+cache store at the same time during upload.`,
 			Advanced: true,
 		}, {
-			Name:     "tmp_upload_path",
-			Default:  "",
-			Help:     "Directory to keep temporary files until they are uploaded to the cloud storage",
+			Name:    "tmp_upload_path",
+			Default: "",
+			Help: `Directory to keep temporary files until they are uploaded.
+
+This is the path where cache will use as a temporary storage for new
+files that need to be uploaded to the cloud provider.
+
+Specifying a value will enable this feature. Without it, it is
+completely disabled and files will be uploaded directly to the cloud
+provider`,
 			Advanced: true,
 		}, {
-			Name:     "tmp_wait_time",
-			Default:  DefCacheTmpWaitTime,
-			Help:     "How long should files be stored in local cache before being uploaded",
+			Name:    "tmp_wait_time",
+			Default: DefCacheTmpWaitTime,
+			Help: `How long should files be stored in local cache before being uploaded
+
+This is the duration that a file must wait in the temporary location
+_cache-tmp-upload-path_ before it is selected for upload.
+
+Note that only one file is uploaded at a time and it can take longer
+to start the upload if a queue formed for this purpose.`,
 			Advanced: true,
 		}, {
-			Name:     "db_wait_time",
-			Default:  DefCacheDbWaitTime,
-			Help:     "How long to wait for the DB to be available - 0 is unlimited",
+			Name:    "db_wait_time",
+			Default: DefCacheDbWaitTime,
+			Help: `How long to wait for the DB to be available - 0 is unlimited
+
+Only one process can have the DB open at any one time, so rclone waits
+for this duration for the DB to become available before it gives an
+error.
+
+If you set it to 0 then it will wait forever.`,
 			Advanced: true,
 		}},
 	})
@@ -193,6 +290,7 @@ type Options struct {
 	PlexUsername       string        `config:"plex_username"`
 	PlexPassword       string        `config:"plex_password"`
 	PlexToken          string        `config:"plex_token"`
+	PlexInsecure       bool          `config:"plex_insecure"`
 	ChunkSize          fs.SizeSuffix `config:"chunk_size"`
 	InfoAge            fs.Duration   `config:"info_age"`
 	ChunkTotalSize     fs.SizeSuffix `config:"chunk_total_size"`
@@ -248,7 +346,7 @@ func NewFs(name, rootPath string, m configmap.Mapper) (fs.Fs, error) {
 		return nil, err
 	}
 	if opt.ChunkTotalSize < opt.ChunkSize*fs.SizeSuffix(opt.TotalWorkers) {
-		return nil, errors.Errorf("don't set cache-total-chunk-size(%v) less than cache-chunk-size(%v) * cache-workers(%v)",
+		return nil, errors.Errorf("don't set cache-chunk-total-size(%v) less than cache-chunk-size(%v) * cache-workers(%v)",
 			opt.ChunkTotalSize, opt.ChunkSize, opt.TotalWorkers)
 	}
 
@@ -261,10 +359,15 @@ func NewFs(name, rootPath string, m configmap.Mapper) (fs.Fs, error) {
 		return nil, errors.Wrapf(err, "failed to clean root path %q", rootPath)
 	}
 
-	remotePath := path.Join(opt.Remote, rpath)
-	wrappedFs, wrapErr := fs.NewFs(remotePath)
+	wInfo, wName, wPath, wConfig, err := fs.ConfigFs(opt.Remote)
+	if err != nil {
+		return nil, errors.Wrapf(err, "failed to parse remote %q to wrap", opt.Remote)
+	}
+
+	remotePath := fspath.JoinRootPath(wPath, rootPath)
+	wrappedFs, wrapErr := wInfo.NewFs(wName, remotePath, wConfig)
 	if wrapErr != nil && wrapErr != fs.ErrorIsFile {
-		return nil, errors.Wrapf(wrapErr, "failed to make remote %q to wrap", remotePath)
+		return nil, errors.Wrapf(wrapErr, "failed to make remote %s:%s to wrap", wName, remotePath)
 	}
 	var fsErr error
 	fs.Debugf(name, "wrapped %v:%v at root %v", wrappedFs.Name(), wrappedFs.Root(), rpath)
@@ -290,7 +393,7 @@ func NewFs(name, rootPath string, m configmap.Mapper) (fs.Fs, error) {
 	f.plexConnector = &plexConnector{}
 	if opt.PlexURL != "" {
 		if opt.PlexToken != "" {
-			f.plexConnector, err = newPlexConnectorWithToken(f, opt.PlexURL, opt.PlexToken)
+			f.plexConnector, err = newPlexConnectorWithToken(f, opt.PlexURL, opt.PlexToken, opt.PlexInsecure)
 			if err != nil {
 				return nil, errors.Wrapf(err, "failed to connect to the Plex API %v", opt.PlexURL)
 			}
@@ -300,7 +403,7 @@ func NewFs(name, rootPath string, m configmap.Mapper) (fs.Fs, error) {
 				if err != nil {
 					decPass = opt.PlexPassword
 				}
-				f.plexConnector, err = newPlexConnector(f, opt.PlexURL, opt.PlexUsername, decPass, func(token string) {
+				f.plexConnector, err = newPlexConnector(f, opt.PlexURL, opt.PlexUsername, decPass, opt.PlexInsecure, func(token string) {
 					m.Set("plex_token", token)
 				})
 				if err != nil {
@@ -368,7 +471,7 @@ func NewFs(name, rootPath string, m configmap.Mapper) (fs.Fs, error) {
 	fs.Infof(name, "Chunk Clean Interval: %v", f.opt.ChunkCleanInterval)
 	fs.Infof(name, "Workers: %v", f.opt.TotalWorkers)
 	fs.Infof(name, "File Age: %v", f.opt.InfoAge)
-	if !f.opt.StoreWrites {
+	if f.opt.StoreWrites {
 		fs.Infof(name, "Cache Writes: enabled")
 	}
 
@@ -403,7 +506,9 @@ func NewFs(name, rootPath string, m configmap.Mapper) (fs.Fs, error) {
 	}()
 
 	if doChangeNotify := wrappedFs.Features().ChangeNotify; doChangeNotify != nil {
-		doChangeNotify(f.receiveChangeNotify, time.Duration(f.opt.ChunkCleanInterval))
+		pollInterval := make(chan time.Duration, 1)
+		pollInterval <- time.Duration(f.opt.ChunkCleanInterval)
+		doChangeNotify(f.receiveChangeNotify, pollInterval)
 	}
 
 	f.features = (&fs.Features{
@@ -455,6 +560,39 @@ Eg
 		Title: "Get cache stats",
 		Help: `
 Show statistics for the cache remote.
+`,
+	})
+
+	rc.Add(rc.Call{
+		Path:  "cache/fetch",
+		Fn:    f.rcFetch,
+		Title: "Fetch file chunks",
+		Help: `
+Ensure the specified file chunks are cached on disk.
+
+The chunks= parameter specifies the file chunks to check.
+It takes a comma separated list of array slice indices.
+The slice indices are similar to Python slices: start[:end]
+
+start is the 0 based chunk number from the beginning of the file
+to fetch inclusive. end is 0 based chunk number from the beginning
+of the file to fetch exclisive.
+Both values can be negative, in which case they count from the back
+of the file. The value "-5:" represents the last 5 chunks of a file.
+
+Some valid examples are:
+":5,-5:" -> the first and last five chunks
+"0,-2" -> the first and the second last chunk
+"0:10" -> the first ten chunks
+
+Any parameter with a key that starts with "file" can be used to
+specify files to fetch, eg
+
+    rclone rc cache/fetch chunks=0 file=hello file2=home/goodbye
+
+File names will automatically be encrypted when the a crypt remote
+is used on top of the cache.
+
 `,
 	})
 
@@ -472,6 +610,22 @@ func (f *Fs) httpStats(in rc.Params) (out rc.Params, err error) {
 	return out, nil
 }
 
+func (f *Fs) unwrapRemote(remote string) string {
+	remote = cleanPath(remote)
+	if remote != "" {
+		// if it's wrapped by crypt we need to check what format we got
+		if cryptFs, yes := f.isWrappedByCrypt(); yes {
+			_, err := cryptFs.DecryptFileName(remote)
+			// if it failed to decrypt then it is a decrypted format and we need to encrypt it
+			if err != nil {
+				return cryptFs.EncryptFileName(remote)
+			}
+			// else it's an encrypted format and we can use it as it is
+		}
+	}
+	return remote
+}
+
 func (f *Fs) httpExpireRemote(in rc.Params) (out rc.Params, err error) {
 	out = make(rc.Params)
 	remoteInt, ok := in["remote"]
@@ -485,20 +639,9 @@ func (f *Fs) httpExpireRemote(in rc.Params) (out rc.Params, err error) {
 		withData = true
 	}
 
-	if cleanPath(remote) != "" {
-		// if it's wrapped by crypt we need to check what format we got
-		if cryptFs, yes := f.isWrappedByCrypt(); yes {
-			_, err := cryptFs.DecryptFileName(remote)
-			// if it failed to decrypt then it is a decrypted format and we need to encrypt it
-			if err != nil {
-				remote = cryptFs.EncryptFileName(remote)
-			}
-			// else it's an encrypted format and we can use it as it is
-		}
-
-		if !f.cache.HasEntry(path.Join(f.Root(), remote)) {
-			return out, errors.Errorf("%s doesn't exist in cache", remote)
-		}
+	remote = f.unwrapRemote(remote)
+	if !f.cache.HasEntry(path.Join(f.Root(), remote)) {
+		return out, errors.Errorf("%s doesn't exist in cache", remote)
 	}
 
 	co := NewObject(f, remote)
@@ -528,6 +671,141 @@ func (f *Fs) httpExpireRemote(in rc.Params) (out rc.Params, err error) {
 	return out, nil
 }
 
+func (f *Fs) rcFetch(in rc.Params) (rc.Params, error) {
+	type chunkRange struct {
+		start, end int64
+	}
+	parseChunks := func(ranges string) (crs []chunkRange, err error) {
+		for _, part := range strings.Split(ranges, ",") {
+			var start, end int64 = 0, math.MaxInt64
+			switch ints := strings.Split(part, ":"); len(ints) {
+			case 1:
+				start, err = strconv.ParseInt(ints[0], 10, 64)
+				if err != nil {
+					return nil, errors.Errorf("invalid range: %q", part)
+				}
+				end = start + 1
+			case 2:
+				if ints[0] != "" {
+					start, err = strconv.ParseInt(ints[0], 10, 64)
+					if err != nil {
+						return nil, errors.Errorf("invalid range: %q", part)
+					}
+				}
+				if ints[1] != "" {
+					end, err = strconv.ParseInt(ints[1], 10, 64)
+					if err != nil {
+						return nil, errors.Errorf("invalid range: %q", part)
+					}
+				}
+			default:
+				return nil, errors.Errorf("invalid range: %q", part)
+			}
+			crs = append(crs, chunkRange{start: start, end: end})
+		}
+		return
+	}
+	walkChunkRange := func(cr chunkRange, size int64, cb func(chunk int64)) {
+		if size <= 0 {
+			return
+		}
+		chunks := (size-1)/f.ChunkSize() + 1
+
+		start, end := cr.start, cr.end
+		if start < 0 {
+			start += chunks
+		}
+		if end <= 0 {
+			end += chunks
+		}
+		if end <= start {
+			return
+		}
+		switch {
+		case start < 0:
+			start = 0
+		case start >= chunks:
+			return
+		}
+		switch {
+		case end <= start:
+			end = start + 1
+		case end >= chunks:
+			end = chunks
+		}
+		for i := start; i < end; i++ {
+			cb(i)
+		}
+	}
+	walkChunkRanges := func(crs []chunkRange, size int64, cb func(chunk int64)) {
+		for _, cr := range crs {
+			walkChunkRange(cr, size, cb)
+		}
+	}
+
+	v, ok := in["chunks"]
+	if !ok {
+		return nil, errors.New("missing chunks parameter")
+	}
+	s, ok := v.(string)
+	if !ok {
+		return nil, errors.New("invalid chunks parameter")
+	}
+	delete(in, "chunks")
+	crs, err := parseChunks(s)
+	if err != nil {
+		return nil, errors.Wrap(err, "invalid chunks parameter")
+	}
+	var files [][2]string
+	for k, v := range in {
+		if !strings.HasPrefix(k, "file") {
+			return nil, errors.Errorf("invalid parameter %s=%s", k, v)
+		}
+		switch v := v.(type) {
+		case string:
+			files = append(files, [2]string{v, f.unwrapRemote(v)})
+		default:
+			return nil, errors.Errorf("invalid parameter %s=%s", k, v)
+		}
+	}
+	type fileStatus struct {
+		Error         string
+		FetchedChunks int
+	}
+	fetchedChunks := make(map[string]fileStatus, len(files))
+	for _, pair := range files {
+		file, remote := pair[0], pair[1]
+		var status fileStatus
+		o, err := f.NewObject(remote)
+		if err != nil {
+			fetchedChunks[file] = fileStatus{Error: err.Error()}
+			continue
+		}
+		co := o.(*Object)
+		err = co.refreshFromSource(true)
+		if err != nil {
+			fetchedChunks[file] = fileStatus{Error: err.Error()}
+			continue
+		}
+		handle := NewObjectHandle(co, f)
+		handle.UseMemory = false
+		handle.scaleWorkers(1)
+		walkChunkRanges(crs, co.Size(), func(chunk int64) {
+			_, err := handle.getChunk(chunk * f.ChunkSize())
+			if err != nil {
+				if status.Error == "" {
+					status.Error = err.Error()
+				}
+			} else {
+				status.FetchedChunks++
+			}
+		})
+		fetchedChunks[file] = status
+	}
+
+	return rc.Params{"status": fetchedChunks}, nil
+}
+
 // receiveChangeNotify is a wrapper to notifications sent from the wrapped FS about changed files
 func (f *Fs) receiveChangeNotify(forgetPath string, entryType fs.EntryType) {
 	if crypt, yes := f.isWrappedByCrypt(); yes {
@@ -595,12 +873,15 @@ func (f *Fs) notifyChangeUpstream(remote string, entryType fs.EntryType) {
 // ChangeNotify can subsribe multiple callers
 // this is coupled with the wrapped fs ChangeNotify (if it supports it)
 // and also notifies other caches (i.e VFS) to clear out whenever something changes
-func (f *Fs) ChangeNotify(notifyFunc func(string, fs.EntryType), pollInterval time.Duration) chan bool {
+func (f *Fs) ChangeNotify(notifyFunc func(string, fs.EntryType), pollInterval <-chan time.Duration) {
 	f.parentsForgetMu.Lock()
 	defer f.parentsForgetMu.Unlock()
 	fs.Debugf(f, "subscribing to ChangeNotify")
 	f.parentsForgetFn = append(f.parentsForgetFn, notifyFunc)
-	return make(chan bool)
+	go func() {
+		for range pollInterval {
+		}
+	}()
 }
 
 // Name of the remote (as passed into NewFs)
@@ -701,7 +982,6 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
 		fs.Debugf(dir, "list: cached entries: %v", entries)
 		return entries, nil
 	}
-	// FIXME need to clean existing cached listing
 
 	// we first search any temporary files stored locally
 	var cachedEntries fs.DirEntries
@@ -727,27 +1007,42 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
 	}
 
 	// search from the source
-	entries, err = f.Fs.List(dir)
+	sourceEntries, err := f.Fs.List(dir)
 	if err != nil {
 		return nil, err
 	}
-	fs.Debugf(dir, "list: read %v from source", len(entries))
-	fs.Debugf(dir, "list: source entries: %v", entries)
+	fs.Debugf(dir, "list: read %v from source", len(sourceEntries))
+	fs.Debugf(dir, "list: source entries: %v", sourceEntries)
+
+	sort.Sort(sourceEntries)
+	for _, entry := range entries {
+		entryRemote := entry.Remote()
+		i := sort.Search(len(sourceEntries), func(i int) bool { return sourceEntries[i].Remote() >= entryRemote })
+		if i < len(sourceEntries) && sourceEntries[i].Remote() == entryRemote {
+			continue
+		}
+		fp := path.Join(f.Root(), entryRemote)
+		switch entry.(type) {
+		case fs.Object:
+			_ = f.cache.RemoveObject(fp)
+		case fs.Directory:
+			_ = f.cache.RemoveDir(fp)
+		}
+		fs.Debugf(dir, "list: remove entry: %v", entryRemote)
+	}
+	entries = nil
 
 	// and then iterate over the ones from source (temp Objects will override source ones)
 	var batchDirectories []*Directory
-	for _, entry := range entries {
+	sort.Sort(cachedEntries)
+	tmpCnt := len(cachedEntries)
+	for _, entry := range sourceEntries {
 		switch o := entry.(type) {
 		case fs.Object:
 			// skip over temporary objects (might be uploading)
-			found := false
-			for _, t := range cachedEntries {
-				if t.Remote() == o.Remote() {
-					found = true
-					break
-				}
-			}
-			if found {
+			oRemote := o.Remote()
+			i := sort.Search(tmpCnt, func(i int) bool { return cachedEntries[i].Remote() >= oRemote })
+			if i < tmpCnt && cachedEntries[i].Remote() == oRemote {
 				continue
 			}
 			co := ObjectFromOriginal(f, o).persist()

backend/cache/cache_internal_test.go

@@ -4,6 +4,9 @@ package cache_test
 
 import (
 	"bytes"
+	"encoding/base64"
+	goflag "flag"
+	"fmt"
 	"io"
 	"io/ioutil"
 	"log"
@@ -12,21 +15,12 @@ import (
 	"path"
 	"path/filepath"
 	"runtime"
+	"runtime/debug"
 	"strconv"
 	"strings"
 	"testing"
 	"time"
 
-	"github.com/pkg/errors"
-
-	"encoding/base64"
-	goflag "flag"
-	"fmt"
-	"runtime/debug"
-
-	"encoding/json"
-	"net/http"
-
 	"github.com/ncw/rclone/backend/cache"
 	"github.com/ncw/rclone/backend/crypt"
 	_ "github.com/ncw/rclone/backend/drive"
@@ -36,10 +30,11 @@ import (
 	"github.com/ncw/rclone/fs/config/configmap"
 	"github.com/ncw/rclone/fs/object"
 	"github.com/ncw/rclone/fs/rc"
-	"github.com/ncw/rclone/fs/rc/rcflags"
 	"github.com/ncw/rclone/fstest"
 	"github.com/ncw/rclone/vfs"
 	"github.com/ncw/rclone/vfs/vfsflags"
+	"github.com/pkg/errors"
+	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
@@ -695,8 +690,8 @@ func TestInternalChangeSeenAfterDirCacheFlush(t *testing.T) {
 }
 
 func TestInternalChangeSeenAfterRc(t *testing.T) {
-	rcflags.Opt.Enabled = true
-	rc.Start(&rcflags.Opt)
+	cacheExpire := rc.Calls.Get("cache/expire")
+	assert.NotNil(t, cacheExpire)
 
 	id := fmt.Sprintf("ticsarc%v", time.Now().Unix())
 	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, false, true, nil, nil)
@@ -729,13 +724,8 @@ func TestInternalChangeSeenAfterRc(t *testing.T) {
 	require.NoError(t, err)
 	require.NotEqual(t, o.ModTime().String(), co.ModTime().String())
 
-	m := make(map[string]string)
-	res, err := http.Post(fmt.Sprintf("http://localhost:5572/cache/expire?remote=%s", "data.bin"), "application/json; charset=utf-8", strings.NewReader(""))
-	require.NoError(t, err)
-	defer func() {
-		_ = res.Body.Close()
-	}()
-	_ = json.NewDecoder(res.Body).Decode(&m)
+	// Call the rc function
+	m, err := cacheExpire.Fn(rc.Params{"remote": "data.bin"})
 	require.Contains(t, m, "status")
 	require.Contains(t, m, "message")
 	require.Equal(t, "ok", m["status"])
@@ -755,13 +745,8 @@ func TestInternalChangeSeenAfterRc(t *testing.T) {
 	li1, err = runInstance.list(t, rootFs, "")
 	require.Len(t, li1, 1)
 
-	m = make(map[string]string)
-	res2, err := http.Post("http://localhost:5572/cache/expire?remote=/", "application/json; charset=utf-8", strings.NewReader(""))
-	require.NoError(t, err)
-	defer func() {
-		_ = res2.Body.Close()
-	}()
-	_ = json.NewDecoder(res2.Body).Decode(&m)
+	// Call the rc function
+	m, err = cacheExpire.Fn(rc.Params{"remote": "/"})
 	require.Contains(t, m, "status")
 	require.Contains(t, m, "message")
 	require.Equal(t, "ok", m["status"])

backend/cache/cache_upload_test.go

@@ -3,6 +3,7 @@
 package cache_test
 
 import (
+	"fmt"
 	"math/rand"
 	"os"
 	"path"
@@ -10,8 +11,6 @@ import (
 	"testing"
 	"time"
 
-	"fmt"
-
 	"github.com/ncw/rclone/backend/cache"
 	_ "github.com/ncw/rclone/backend/drive"
 	"github.com/ncw/rclone/fs"

backend/cache/cache_upload_test.go.orig

@@ -1,455 +0,0 @@
-// +build !plan9
-
-package cache_test
-
-import (
-	"math/rand"
-	"os"
-	"path"
-	"strconv"
-	"testing"
-	"time"
-
-	"fmt"
-
-	"github.com/ncw/rclone/backend/cache"
-	_ "github.com/ncw/rclone/backend/drive"
-	"github.com/ncw/rclone/fs"
-	"github.com/stretchr/testify/require"
-)
-
-func TestInternalUploadTempDirCreated(t *testing.T) {
-	id := fmt.Sprintf("tiutdc%v", time.Now().Unix())
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, false, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id)})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	_, err := os.Stat(path.Join(runInstance.tmpUploadDir, id))
-	require.NoError(t, err)
-}
-
-func testInternalUploadQueueOneFile(t *testing.T, id string, rootFs fs.Fs, boltDb *cache.Persistent) {
-	// create some rand test data
-	testSize := int64(524288000)
-	testReader := runInstance.randomReader(t, testSize)
-	bu := runInstance.listenForBackgroundUpload(t, rootFs, "one")
-	runInstance.writeRemoteReader(t, rootFs, "one", testReader)
-	// validate that it exists in temp fs
-	ti, err := os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "one")))
-	require.NoError(t, err)
-
-	if runInstance.rootIsCrypt {
-		require.Equal(t, int64(524416032), ti.Size())
-	} else {
-		require.Equal(t, testSize, ti.Size())
-	}
-	de1, err := runInstance.list(t, rootFs, "")
-	require.NoError(t, err)
-	require.Len(t, de1, 1)
-
-	runInstance.completeBackgroundUpload(t, "one", bu)
-	// check if it was removed from temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "one")))
-	require.True(t, os.IsNotExist(err))
-
-	// check if it can be read
-	data2, err := runInstance.readDataFromRemote(t, rootFs, "one", 0, int64(1024), false)
-	require.NoError(t, err)
-	require.Len(t, data2, 1024)
-}
-
-func TestInternalUploadQueueOneFileNoRest(t *testing.T) {
-	id := fmt.Sprintf("tiuqofnr%v", time.Now().Unix())
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "0s"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	testInternalUploadQueueOneFile(t, id, rootFs, boltDb)
-}
-
-func TestInternalUploadQueueOneFileWithRest(t *testing.T) {
-	id := fmt.Sprintf("tiuqofwr%v", time.Now().Unix())
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "1m"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	testInternalUploadQueueOneFile(t, id, rootFs, boltDb)
-}
-
-func TestInternalUploadMoveExistingFile(t *testing.T) {
-	id := fmt.Sprintf("tiumef%v", time.Now().Unix())
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "3s"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	err := rootFs.Mkdir("one")
-	require.NoError(t, err)
-	err = rootFs.Mkdir("one/test")
-	require.NoError(t, err)
-	err = rootFs.Mkdir("second")
-	require.NoError(t, err)
-
-	// create some rand test data
-	testSize := int64(10485760)
-	testReader := runInstance.randomReader(t, testSize)
-	runInstance.writeObjectReader(t, rootFs, "one/test/data.bin", testReader)
-	runInstance.completeAllBackgroundUploads(t, rootFs, "one/test/data.bin")
-
-	de1, err := runInstance.list(t, rootFs, "one/test")
-	require.NoError(t, err)
-	require.Len(t, de1, 1)
-
-	time.Sleep(time.Second * 5)
-	//_ = os.Remove(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "one/test")))
-	//require.NoError(t, err)
-
-	err = runInstance.dirMove(t, rootFs, "one/test", "second/test")
-	require.NoError(t, err)
-
-	// check if it can be read
-	de1, err = runInstance.list(t, rootFs, "second/test")
-	require.NoError(t, err)
-	require.Len(t, de1, 1)
-}
-
-func TestInternalUploadTempPathCleaned(t *testing.T) {
-	id := fmt.Sprintf("tiutpc%v", time.Now().Unix())
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "5s"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	err := rootFs.Mkdir("one")
-	require.NoError(t, err)
-	err = rootFs.Mkdir("one/test")
-	require.NoError(t, err)
-	err = rootFs.Mkdir("second")
-	require.NoError(t, err)
-
-	// create some rand test data
-	testSize := int64(1048576)
-	testReader := runInstance.randomReader(t, testSize)
-	testReader2 := runInstance.randomReader(t, testSize)
-	runInstance.writeObjectReader(t, rootFs, "one/test/data.bin", testReader)
-	runInstance.writeObjectReader(t, rootFs, "second/data.bin", testReader2)
-
-	runInstance.completeAllBackgroundUploads(t, rootFs, "one/test/data.bin")
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "one/test")))
-	require.True(t, os.IsNotExist(err))
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "one")))
-	require.True(t, os.IsNotExist(err))
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "second")))
-	require.False(t, os.IsNotExist(err))
-
-	runInstance.completeAllBackgroundUploads(t, rootFs, "second/data.bin")
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "second/data.bin")))
-	require.True(t, os.IsNotExist(err))
-
-	de1, err := runInstance.list(t, rootFs, "one/test")
-	require.NoError(t, err)
-	require.Len(t, de1, 1)
-
-	// check if it can be read
-	de1, err = runInstance.list(t, rootFs, "second")
-	require.NoError(t, err)
-	require.Len(t, de1, 1)
-}
-
-func TestInternalUploadQueueMoreFiles(t *testing.T) {
-	id := fmt.Sprintf("tiuqmf%v", time.Now().Unix())
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "1s"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	err := rootFs.Mkdir("test")
-	require.NoError(t, err)
-	minSize := 5242880
-	maxSize := 10485760
-	totalFiles := 10
-	rand.Seed(time.Now().Unix())
-
-	lastFile := ""
-	for i := 0; i < totalFiles; i++ {
-		size := int64(rand.Intn(maxSize-minSize) + minSize)
-		testReader := runInstance.randomReader(t, size)
-		remote := "test/" + strconv.Itoa(i) + ".bin"
-		runInstance.writeRemoteReader(t, rootFs, remote, testReader)
-
-		// validate that it exists in temp fs
-		ti, err := os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, remote)))
-		require.NoError(t, err)
-		require.Equal(t, size, runInstance.cleanSize(t, ti.Size()))
-
-		if runInstance.wrappedIsExternal && i < totalFiles-1 {
-			time.Sleep(time.Second * 3)
-		}
-		lastFile = remote
-	}
-
-	// check if cache lists all files, likely temp upload didn't finish yet
-	de1, err := runInstance.list(t, rootFs, "test")
-	require.NoError(t, err)
-	require.Len(t, de1, totalFiles)
-
-	// wait for background uploader to do its thing
-	runInstance.completeAllBackgroundUploads(t, rootFs, lastFile)
-
-	// retry until we have no more temp files and fail if they don't go down to 0
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test")))
-	require.True(t, os.IsNotExist(err))
-
-	// check if cache lists all files
-	de1, err = runInstance.list(t, rootFs, "test")
-	require.NoError(t, err)
-	require.Len(t, de1, totalFiles)
-}
-
-func TestInternalUploadTempFileOperations(t *testing.T) {
-	id := "tiutfo"
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "1h"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	boltDb.PurgeTempUploads()
-
-	// create some rand test data
-	runInstance.mkdir(t, rootFs, "test")
-	runInstance.writeRemoteString(t, rootFs, "test/one", "one content")
-
-	// check if it can be read
-	data1, err := runInstance.readDataFromRemote(t, rootFs, "test/one", 0, int64(len([]byte("one content"))), false)
-	require.NoError(t, err)
-	require.Equal(t, []byte("one content"), data1)
-	// validate that it exists in temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.NoError(t, err)
-
-	// test DirMove - allowed
-	err = runInstance.dirMove(t, rootFs, "test", "second")
-	if err != errNotSupported {
-		require.NoError(t, err)
-		_, err = rootFs.NewObject("test/one")
-		require.Error(t, err)
-		_, err = rootFs.NewObject("second/one")
-		require.NoError(t, err)
-		// validate that it exists in temp fs
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-		require.Error(t, err)
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "second/one")))
-		require.NoError(t, err)
-		_, err = boltDb.SearchPendingUpload(runInstance.encryptRemoteIfNeeded(t, path.Join(id, "test/one")))
-		require.Error(t, err)
-		var started bool
-		started, err = boltDb.SearchPendingUpload(runInstance.encryptRemoteIfNeeded(t, path.Join(id, "second/one")))
-		require.NoError(t, err)
-		require.False(t, started)
-		runInstance.mkdir(t, rootFs, "test")
-		runInstance.writeRemoteString(t, rootFs, "test/one", "one content")
-	}
-
-	// test Rmdir - allowed
-	err = runInstance.rm(t, rootFs, "test")
-	require.Error(t, err)
-	require.Contains(t, err.Error(), "directory not empty")
-	_, err = rootFs.NewObject("test/one")
-	require.NoError(t, err)
-	// validate that it exists in temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.NoError(t, err)
-	started, err := boltDb.SearchPendingUpload(runInstance.encryptRemoteIfNeeded(t, path.Join(id, "test/one")))
-	require.False(t, started)
-	require.NoError(t, err)
-
-	// test Move/Rename -- allowed
-	err = runInstance.move(t, rootFs, path.Join("test", "one"), path.Join("test", "second"))
-	if err != errNotSupported {
-		require.NoError(t, err)
-		// try to read from it
-		_, err = rootFs.NewObject("test/one")
-		require.Error(t, err)
-		_, err = rootFs.NewObject("test/second")
-		require.NoError(t, err)
-		data2, err := runInstance.readDataFromRemote(t, rootFs, "test/second", 0, int64(len([]byte("one content"))), false)
-		require.NoError(t, err)
-		require.Equal(t, []byte("one content"), data2)
-		// validate that it exists in temp fs
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-		require.Error(t, err)
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/second")))
-		require.NoError(t, err)
-		runInstance.writeRemoteString(t, rootFs, "test/one", "one content")
-	}
-
-	// test Copy -- allowed
-	err = runInstance.copy(t, rootFs, path.Join("test", "one"), path.Join("test", "third"))
-	if err != errNotSupported {
-		require.NoError(t, err)
-		_, err = rootFs.NewObject("test/one")
-		require.NoError(t, err)
-		_, err = rootFs.NewObject("test/third")
-		require.NoError(t, err)
-		data2, err := runInstance.readDataFromRemote(t, rootFs, "test/third", 0, int64(len([]byte("one content"))), false)
-		require.NoError(t, err)
-		require.Equal(t, []byte("one content"), data2)
-		// validate that it exists in temp fs
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-		require.NoError(t, err)
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/third")))
-		require.NoError(t, err)
-	}
-
-	// test Remove -- allowed
-	err = runInstance.rm(t, rootFs, "test/one")
-	require.NoError(t, err)
-	_, err = rootFs.NewObject("test/one")
-	require.Error(t, err)
-	// validate that it doesn't exist in temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.Error(t, err)
-	runInstance.writeRemoteString(t, rootFs, "test/one", "one content")
-
-	// test Update -- allowed
-	firstModTime, err := runInstance.modTime(t, rootFs, "test/one")
-	require.NoError(t, err)
-	err = runInstance.updateData(t, rootFs, "test/one", "one content", " updated")
-	require.NoError(t, err)
-	obj2, err := rootFs.NewObject("test/one")
-	require.NoError(t, err)
-	data2 := runInstance.readDataFromObj(t, obj2, 0, int64(len("one content updated")), false)
-	require.Equal(t, "one content updated", string(data2))
-	tmpInfo, err := os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.NoError(t, err)
-	if runInstance.rootIsCrypt {
-		require.Equal(t, int64(67), tmpInfo.Size())
-	} else {
-		require.Equal(t, int64(len(data2)), tmpInfo.Size())
-	}
-
-	// test SetModTime -- allowed
-	secondModTime, err := runInstance.modTime(t, rootFs, "test/one")
-	require.NoError(t, err)
-	require.NotEqual(t, secondModTime, firstModTime)
-	require.NotEqual(t, time.Time{}, firstModTime)
-	require.NotEqual(t, time.Time{}, secondModTime)
-}
-
-func TestInternalUploadUploadingFileOperations(t *testing.T) {
-	id := "tiuufo"
-	rootFs, boltDb := runInstance.newCacheFs(t, remoteName, id, true, true,
-		nil,
-		map[string]string{"cache-tmp-upload-path": path.Join(runInstance.tmpUploadDir, id), "cache-tmp-wait-time": "1h"})
-	defer runInstance.cleanupFs(t, rootFs, boltDb)
-
-	boltDb.PurgeTempUploads()
-
-	// create some rand test data
-	runInstance.mkdir(t, rootFs, "test")
-	runInstance.writeRemoteString(t, rootFs, "test/one", "one content")
-
-	// check if it can be read
-	data1, err := runInstance.readDataFromRemote(t, rootFs, "test/one", 0, int64(len([]byte("one content"))), false)
-	require.NoError(t, err)
-	require.Equal(t, []byte("one content"), data1)
-	// validate that it exists in temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.NoError(t, err)
-
-	err = boltDb.SetPendingUploadToStarted(runInstance.encryptRemoteIfNeeded(t, path.Join(rootFs.Root(), "test/one")))
-	require.NoError(t, err)
-
-	// test DirMove
-	err = runInstance.dirMove(t, rootFs, "test", "second")
-	if err != errNotSupported {
-		require.Error(t, err)
-		_, err = rootFs.NewObject("test/one")
-		require.NoError(t, err)
-		// validate that it exists in temp fs
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-		require.NoError(t, err)
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "second/one")))
-		require.Error(t, err)
-	}
-
-	// test Rmdir
-	err = runInstance.rm(t, rootFs, "test")
-	require.Error(t, err)
-	_, err = rootFs.NewObject("test/one")
-	require.NoError(t, err)
-	// validate that it doesn't exist in temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.NoError(t, err)
-
-	// test Move/Rename
-	err = runInstance.move(t, rootFs, path.Join("test", "one"), path.Join("test", "second"))
-	if err != errNotSupported {
-		require.Error(t, err)
-		// try to read from it
-		_, err = rootFs.NewObject("test/one")
-		require.NoError(t, err)
-		_, err = rootFs.NewObject("test/second")
-		require.Error(t, err)
-		// validate that it exists in temp fs
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-		require.NoError(t, err)
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/second")))
-		require.Error(t, err)
-	}
-
-	// test Copy -- allowed
-	err = runInstance.copy(t, rootFs, path.Join("test", "one"), path.Join("test", "third"))
-	if err != errNotSupported {
-		require.NoError(t, err)
-		_, err = rootFs.NewObject("test/one")
-		require.NoError(t, err)
-		_, err = rootFs.NewObject("test/third")
-		require.NoError(t, err)
-		data2, err := runInstance.readDataFromRemote(t, rootFs, "test/third", 0, int64(len([]byte("one content"))), false)
-		require.NoError(t, err)
-		require.Equal(t, []byte("one content"), data2)
-		// validate that it exists in temp fs
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-		require.NoError(t, err)
-		_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/third")))
-		require.NoError(t, err)
-	}
-
-	// test Remove
-	err = runInstance.rm(t, rootFs, "test/one")
-	require.Error(t, err)
-	_, err = rootFs.NewObject("test/one")
-	require.NoError(t, err)
-	// validate that it doesn't exist in temp fs
-	_, err = os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	require.NoError(t, err)
-	runInstance.writeRemoteString(t, rootFs, "test/one", "one content")
-
-	// test Update - this seems to work. Why? FIXME
-	//firstModTime, err := runInstance.modTime(t, rootFs, "test/one")
-	//require.NoError(t, err)
-	//err = runInstance.updateData(t, rootFs, "test/one", "one content", " updated", func() {
-	//	data2 := runInstance.readDataFromRemote(t, rootFs, "test/one", 0, int64(len("one content updated")), true)
-	//	require.Equal(t, "one content", string(data2))
-	//
-	//	tmpInfo, err := os.Stat(path.Join(runInstance.tmpUploadDir, id, runInstance.encryptRemoteIfNeeded(t, "test/one")))
-	//	require.NoError(t, err)
-	//	if runInstance.rootIsCrypt {
-	//		require.Equal(t, int64(67), tmpInfo.Size())
-	//	} else {
-	//		require.Equal(t, int64(len(data2)), tmpInfo.Size())
-	//	}
-	//})
-	//require.Error(t, err)
-
-	// test SetModTime -- seems to work cause of previous
-	//secondModTime, err := runInstance.modTime(t, rootFs, "test/one")
-	//require.NoError(t, err)
-	//require.Equal(t, secondModTime, firstModTime)
-	//require.NotEqual(t, time.Time{}, firstModTime)
-	//require.NotEqual(t, time.Time{}, secondModTime)
-}

backend/cache/cache_upload_test.go.rej

@@ -1,12 +0,0 @@
---- cache_upload_test.go
-+++ cache_upload_test.go
-@@ -1500,9 +1469,6 @@ func (r *run) cleanupFs(t *testing.T, f fs.Fs, b *cache.Persistent) {
- 	}
- 	r.tempFiles = nil
- 	debug.FreeOSMemory()
--	for k, v := range r.runDefaultFlagMap {
--		_ = flag.Set(k, v)
--	}
- }
- 
- func (r *run) randomBytes(t *testing.T, size int64) []byte {

backend/cache/directory.go

@@ -3,9 +3,8 @@
 package cache
 
 import (
-	"time"
-
 	"path"
+	"time"
 
 	"github.com/ncw/rclone/fs"
 )

backend/cache/handle.go

@@ -5,12 +5,11 @@ package cache
 import (
 	"fmt"
 	"io"
-	"sync"
-	"time"
-
 	"path"
 	"runtime"
 	"strings"
+	"sync"
+	"time"
 
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/operations"
@@ -49,12 +48,13 @@ type Handle struct {
 	offset         int64
 	seenOffsets    map[int64]bool
 	mu             sync.Mutex
+	workersWg      sync.WaitGroup
 	confirmReading chan bool
-
-	UseMemory bool
-	workers   []*worker
-	closed    bool
-	reading   bool
+	workers        int
+	maxWorkerID    int
+	UseMemory      bool
+	closed         bool
+	reading        bool
 }
 
 // NewObjectHandle returns a new Handle for an existing Object
@@ -95,7 +95,7 @@ func (r *Handle) String() string {
 
 // startReadWorkers will start the worker pool
 func (r *Handle) startReadWorkers() {
-	if r.hasAtLeastOneWorker() {
+	if r.workers > 0 {
 		return
 	}
 	totalWorkers := r.cacheFs().opt.TotalWorkers
@@ -117,26 +117,27 @@ func (r *Handle) startReadWorkers() {
 
 // scaleOutWorkers will increase the worker pool count by the provided amount
 func (r *Handle) scaleWorkers(desired int) {
-	current := len(r.workers)
+	current := r.workers
 	if current == desired {
 		return
 	}
 	if current > desired {
 		// scale in gracefully
-		for i := 0; i < current-desired; i++ {
+		for r.workers > desired {
 			r.preloadQueue <- -1
+			r.workers--
 		}
 	} else {
 		// scale out
-		for i := 0; i < desired-current; i++ {
+		for r.workers < desired {
 			w := &worker{
 				r:  r,
-				ch: r.preloadQueue,
-				id: current + i,
+				id: r.maxWorkerID,
 			}
+			r.workersWg.Add(1)
+			r.workers++
+			r.maxWorkerID++
 			go w.run()
-
-			r.workers = append(r.workers, w)
 		}
 	}
 	// ignore first scale out from 0
@@ -148,7 +149,7 @@ func (r *Handle) scaleWorkers(desired int) {
 func (r *Handle) confirmExternalReading() {
 	// if we have a max value of workers
 	// then we skip this step
-	if len(r.workers) > 1 ||
+	if r.workers > 1 ||
 		!r.cacheFs().plexConnector.isConfigured() {
 		return
 	}
@@ -178,7 +179,7 @@ func (r *Handle) queueOffset(offset int64) {
 			}
 		}
 
-		for i := 0; i < len(r.workers); i++ {
+		for i := 0; i < r.workers; i++ {
 			o := r.preloadOffset + int64(r.cacheFs().opt.ChunkSize)*int64(i)
 			if o < 0 || o >= r.cachedObject.Size() {
 				continue
@@ -193,16 +194,6 @@ func (r *Handle) queueOffset(offset int64) {
 	}
 }
 
-func (r *Handle) hasAtLeastOneWorker() bool {
-	oneWorker := false
-	for i := 0; i < len(r.workers); i++ {
-		if r.workers[i].isRunning() {
-			oneWorker = true
-		}
-	}
-	return oneWorker
-}
-
 // getChunk is called by the FS to retrieve a specific chunk of known start and size from where it can find it
 // it can be from transient or persistent cache
 // it will also build the chunk from the cache's specific chunk boundaries and build the final desired chunk in a buffer
@@ -243,7 +234,7 @@ func (r *Handle) getChunk(chunkStart int64) ([]byte, error) {
 	// not found in ram or
 	// the worker didn't managed to download the chunk in time so we abort and close the stream
 	if err != nil || len(data) == 0 || !found {
-		if !r.hasAtLeastOneWorker() {
+		if r.workers == 0 {
 			fs.Errorf(r, "out of workers")
 			return nil, io.ErrUnexpectedEOF
 		}
@@ -304,14 +295,7 @@ func (r *Handle) Close() error {
 	close(r.preloadQueue)
 	r.closed = true
 	// wait for workers to complete their jobs before returning
-	waitCount := 3
-	for i := 0; i < len(r.workers); i++ {
-		waitIdx := 0
-		for r.workers[i].isRunning() && waitIdx < waitCount {
-			time.Sleep(time.Second)
-			waitIdx++
-		}
-	}
+	r.workersWg.Wait()
 	r.memory.db.Flush()
 
 	fs.Debugf(r, "cache reader closed %v", r.offset)
@@ -348,12 +332,9 @@ func (r *Handle) Seek(offset int64, whence int) (int64, error) {
 }
 
 type worker struct {
-	r       *Handle
-	ch      <-chan int64
-	rc      io.ReadCloser
-	id      int
-	running bool
-	mu      sync.Mutex
+	r  *Handle
+	rc io.ReadCloser
+	id int
 }
 
 // String is a representation of this worker
@@ -398,33 +379,19 @@ func (w *worker) reader(offset, end int64, closeOpen bool) (io.ReadCloser, error
 	})
 }
 
-func (w *worker) isRunning() bool {
-	w.mu.Lock()
-	defer w.mu.Unlock()
-	return w.running
-}
-
-func (w *worker) setRunning(f bool) {
-	w.mu.Lock()
-	defer w.mu.Unlock()
-	w.running = f
-}
-
 // run is the main loop for the worker which receives offsets to preload
 func (w *worker) run() {
 	var err error
 	var data []byte
-	defer w.setRunning(false)
 	defer func() {
 		if w.rc != nil {
 			_ = w.rc.Close()
-			w.setRunning(false)
 		}
+		w.r.workersWg.Done()
 	}()
 
 	for {
-		chunkStart, open := <-w.ch
-		w.setRunning(true)
+		chunkStart, open := <-w.r.preloadQueue
 		if chunkStart < 0 || !open {
 			break
 		}

backend/cache/object.go

@@ -208,11 +208,17 @@ func (o *Object) SetModTime(t time.Time) error {
 
 // Open is used to request a specific part of the file using fs.RangeOption
 func (o *Object) Open(options ...fs.OpenOption) (io.ReadCloser, error) {
-	if err := o.refreshFromSource(true); err != nil {
+	var err error
+
+	if o.Object == nil {
+		err = o.refreshFromSource(true)
+	} else {
+		err = o.refresh()
+	}
+	if err != nil {
 		return nil, err
 	}
 
-	var err error
 	cacheReader := NewObjectHandle(o, o.CacheFs)
 	var offset, limit int64 = 0, -1
 	for _, option := range options {

backend/cache/plex.go

@@ -3,17 +3,16 @@
 package cache
 
 import (
+	"bytes"
+	"crypto/tls"
 	"encoding/json"
 	"fmt"
+	"io/ioutil"
 	"net/http"
 	"net/url"
 	"strings"
-	"time"
-
 	"sync"
-
-	"bytes"
-	"io/ioutil"
+	"time"
 
 	"github.com/ncw/rclone/fs"
 	"github.com/patrickmn/go-cache"
@@ -54,6 +53,7 @@ type plexConnector struct {
 	username   string
 	password   string
 	token      string
+	insecure   bool
 	f          *Fs
 	mu         sync.Mutex
 	running    bool
@@ -63,7 +63,7 @@ type plexConnector struct {
 }
 
 // newPlexConnector connects to a Plex server and generates a token
-func newPlexConnector(f *Fs, plexURL, username, password string, saveToken func(string)) (*plexConnector, error) {
+func newPlexConnector(f *Fs, plexURL, username, password string, insecure bool, saveToken func(string)) (*plexConnector, error) {
 	u, err := url.ParseRequestURI(strings.TrimRight(plexURL, "/"))
 	if err != nil {
 		return nil, err
@@ -75,6 +75,7 @@ func newPlexConnector(f *Fs, plexURL, username, password string, saveToken func(
 		username:   username,
 		password:   password,
 		token:      "",
+		insecure:   insecure,
 		stateCache: cache.New(time.Hour, time.Minute),
 		saveToken:  saveToken,
 	}
@@ -83,7 +84,7 @@ func newPlexConnector(f *Fs, plexURL, username, password string, saveToken func(
 }
 
 // newPlexConnector connects to a Plex server and generates a token
-func newPlexConnectorWithToken(f *Fs, plexURL, token string) (*plexConnector, error) {
+func newPlexConnectorWithToken(f *Fs, plexURL, token string, insecure bool) (*plexConnector, error) {
 	u, err := url.ParseRequestURI(strings.TrimRight(plexURL, "/"))
 	if err != nil {
 		return nil, err
@@ -93,6 +94,7 @@ func newPlexConnectorWithToken(f *Fs, plexURL, token string) (*plexConnector, er
 		f:          f,
 		url:        u,
 		token:      token,
+		insecure:   insecure,
 		stateCache: cache.New(time.Hour, time.Minute),
 	}
 	pc.listenWebsocket()
@@ -107,14 +109,26 @@ func (p *plexConnector) closeWebsocket() {
 	p.running = false
 }
 
+func (p *plexConnector) websocketDial() (*websocket.Conn, error) {
+	u := strings.TrimRight(strings.Replace(strings.Replace(
+		p.url.String(), "http://", "ws://", 1), "https://", "wss://", 1), "/")
+	url := fmt.Sprintf(defPlexNotificationURL, u, p.token)
+
+	config, err := websocket.NewConfig(url, "http://localhost")
+	if err != nil {
+		return nil, err
+	}
+	if p.insecure {
+		config.TlsConfig = &tls.Config{InsecureSkipVerify: true}
+	}
+	return websocket.DialConfig(config)
+}
+
 func (p *plexConnector) listenWebsocket() {
 	p.runningMu.Lock()
 	defer p.runningMu.Unlock()
 
-	u := strings.Replace(p.url.String(), "http://", "ws://", 1)
-	u = strings.Replace(u, "https://", "wss://", 1)
-	conn, err := websocket.Dial(fmt.Sprintf(defPlexNotificationURL, strings.TrimRight(u, "/"), p.token),
-		"", "http://localhost")
+	conn, err := p.websocketDial()
 	if err != nil {
 		fs.Errorf("plex", "%v", err)
 		return

backend/cache/storage_persistent.go

@@ -3,20 +3,17 @@
 package cache
 
 import (
-	"time"
-
 	"bytes"
 	"encoding/binary"
 	"encoding/json"
+	"fmt"
+	"io/ioutil"
 	"os"
 	"path"
 	"strconv"
 	"strings"
 	"sync"
-
-	"io/ioutil"
-
-	"fmt"
+	"time"
 
 	bolt "github.com/coreos/bbolt"
 	"github.com/ncw/rclone/fs"

backend/crypt/cipher.go

@@ -17,11 +17,9 @@ import (
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/accounting"
 	"github.com/pkg/errors"
-
+	"github.com/rfjakob/eme"
 	"golang.org/x/crypto/nacl/secretbox"
 	"golang.org/x/crypto/scrypt"
-
-	"github.com/rfjakob/eme"
 )
 
 // Constants

backend/crypt/crypt.go

@@ -4,7 +4,6 @@ package crypt
 import (
 	"fmt"
 	"io"
-	"path"
 	"strings"
 	"time"
 
@@ -13,6 +12,7 @@ import (
 	"github.com/ncw/rclone/fs/config/configmap"
 	"github.com/ncw/rclone/fs/config/configstruct"
 	"github.com/ncw/rclone/fs/config/obscure"
+	"github.com/ncw/rclone/fs/fspath"
 	"github.com/ncw/rclone/fs/hash"
 	"github.com/pkg/errors"
 )
@@ -67,8 +67,16 @@ func init() {
 			Help:       "Password or pass phrase for salt. Optional but recommended.\nShould be different to the previous password.",
 			IsPassword: true,
 		}, {
-			Name:     "show_mapping",
-			Help:     "For all files listed show how the names encrypt.",
+			Name: "show_mapping",
+			Help: `For all files listed show how the names encrypt.
+
+If this flag is set then for each file that the remote is asked to
+list, it will log (at level INFO) a line stating the decrypted file
+name and the encrypted file name.
+
+This is so you can work out which encrypted names are which decrypted
+names just in case you need to do something with the encrypted file
+names, or for debugging purposes.`,
 			Default:  false,
 			Hide:     fs.OptionHideConfigurator,
 			Advanced: true,
@@ -130,16 +138,20 @@ func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error) {
 	if strings.HasPrefix(remote, name+":") {
 		return nil, errors.New("can't point crypt remote at itself - check the value of the remote setting")
 	}
+	wInfo, wName, wPath, wConfig, err := fs.ConfigFs(remote)
+	if err != nil {
+		return nil, errors.Wrapf(err, "failed to parse remote %q to wrap", remote)
+	}
 	// Look for a file first
-	remotePath := path.Join(remote, cipher.EncryptFileName(rpath))
-	wrappedFs, err := fs.NewFs(remotePath)
+	remotePath := fspath.JoinRootPath(wPath, cipher.EncryptFileName(rpath))
+	wrappedFs, err := wInfo.NewFs(wName, remotePath, wConfig)
 	// if that didn't produce a file, look for a directory
 	if err != fs.ErrorIsFile {
-		remotePath = path.Join(remote, cipher.EncryptDirName(rpath))
-		wrappedFs, err = fs.NewFs(remotePath)
+		remotePath = fspath.JoinRootPath(wPath, cipher.EncryptDirName(rpath))
+		wrappedFs, err = wInfo.NewFs(wName, remotePath, wConfig)
 	}
 	if err != fs.ErrorIsFile && err != nil {
-		return nil, errors.Wrapf(err, "failed to make remote %q to wrap", remotePath)
+		return nil, errors.Wrapf(err, "failed to make remote %s:%q to wrap", wName, remotePath)
 	}
 	f := &Fs{
 		Fs:     wrappedFs,
@@ -161,7 +173,7 @@ func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error) {
 
 	doChangeNotify := wrappedFs.Features().ChangeNotify
 	if doChangeNotify != nil {
-		f.features.ChangeNotify = func(notifyFunc func(string, fs.EntryType), pollInterval time.Duration) chan bool {
+		f.features.ChangeNotify = func(notifyFunc func(string, fs.EntryType), pollInterval <-chan time.Duration) {
 			wrappedNotifyFunc := func(path string, entryType fs.EntryType) {
 				decrypted, err := f.DecryptFileName(path)
 				if err != nil {
@@ -170,7 +182,7 @@ func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error) {
 				}
 				notifyFunc(decrypted, entryType)
 			}
-			return doChangeNotify(wrappedNotifyFunc, pollInterval)
+			doChangeNotify(wrappedNotifyFunc, pollInterval)
 		}
 	}
 

backend/crypt/crypt_test.go

@@ -7,13 +7,30 @@ import (
 	"testing"
 
 	"github.com/ncw/rclone/backend/crypt"
+	_ "github.com/ncw/rclone/backend/drive" // for integration tests
 	_ "github.com/ncw/rclone/backend/local"
+	_ "github.com/ncw/rclone/backend/swift" // for integration tests
 	"github.com/ncw/rclone/fs/config/obscure"
+	"github.com/ncw/rclone/fstest"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
+// TestIntegration runs integration tests against the remote
+func TestIntegration(t *testing.T) {
+	if *fstest.RemoteName == "" {
+		t.Skip("Skipping as -remote not set")
+	}
+	fstests.Run(t, &fstests.Opt{
+		RemoteName: *fstest.RemoteName,
+		NilObject:  (*crypt.Object)(nil),
+	})
+}
+
 // TestStandard runs integration tests against the remote
 func TestStandard(t *testing.T) {
+	if *fstest.RemoteName != "" {
+		t.Skip("Skipping as -remote set")
+	}
 	tempdir := filepath.Join(os.TempDir(), "rclone-crypt-test-standard")
 	name := "TestCrypt"
 	fstests.Run(t, &fstests.Opt{
@@ -30,6 +47,9 @@ func TestStandard(t *testing.T) {
 
 // TestOff runs integration tests against the remote
 func TestOff(t *testing.T) {
+	if *fstest.RemoteName != "" {
+		t.Skip("Skipping as -remote set")
+	}
 	tempdir := filepath.Join(os.TempDir(), "rclone-crypt-test-off")
 	name := "TestCrypt2"
 	fstests.Run(t, &fstests.Opt{
@@ -46,6 +66,9 @@ func TestOff(t *testing.T) {
 
 // TestObfuscate runs integration tests against the remote
 func TestObfuscate(t *testing.T) {
+	if *fstest.RemoteName != "" {
+		t.Skip("Skipping as -remote set")
+	}
 	tempdir := filepath.Join(os.TempDir(), "rclone-crypt-test-obfuscate")
 	name := "TestCrypt3"
 	fstests.Run(t, &fstests.Opt{

backend/drive/drive_internal_test.go

@@ -1,62 +1,55 @@
 package drive
 
 import (
+	"bytes"
 	"encoding/json"
+	"io"
+	"io/ioutil"
+	"mime"
+	"path/filepath"
+	"strings"
 	"testing"
 
-	"google.golang.org/api/drive/v3"
-
+	_ "github.com/ncw/rclone/backend/local"
+	"github.com/ncw/rclone/fs"
+	"github.com/ncw/rclone/fs/operations"
+	"github.com/ncw/rclone/fstest/fstests"
 	"github.com/pkg/errors"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"google.golang.org/api/drive/v3"
 )
 
-const exampleExportFormats = `{
-	"application/vnd.google-apps.document": [
-		"application/rtf",
-		"application/vnd.oasis.opendocument.text",
-		"text/html",
-		"application/pdf",
-		"application/epub+zip",
-		"application/zip",
-		"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
-		"text/plain"
-	],
-	"application/vnd.google-apps.spreadsheet": [
-		"application/x-vnd.oasis.opendocument.spreadsheet",
-		"text/tab-separated-values",
-		"application/pdf",
-		"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
-		"text/csv",
-		"application/zip",
-		"application/vnd.oasis.opendocument.spreadsheet"
-	],
-	"application/vnd.google-apps.jam": [
-		"application/pdf"
-	],
-	"application/vnd.google-apps.script": [
-		"application/vnd.google-apps.script+json"
-	],
-	"application/vnd.google-apps.presentation": [
-		"application/vnd.oasis.opendocument.presentation",
-		"application/pdf",
-		"application/vnd.openxmlformats-officedocument.presentationml.presentation",
-		"text/plain"
-	],
-	"application/vnd.google-apps.form": [
-		"application/zip"
-	],
-	"application/vnd.google-apps.drawing": [
-		"image/svg+xml",
-		"image/png",
-		"application/pdf",
-		"image/jpeg"
-	]
-}`
+/*
+var additionalMimeTypes = map[string]string{
+	"application/vnd.ms-excel.sheet.macroenabled.12":                          ".xlsm",
+	"application/vnd.ms-excel.template.macroenabled.12":                       ".xltm",
+	"application/vnd.ms-powerpoint.presentation.macroenabled.12":              ".pptm",
+	"application/vnd.ms-powerpoint.slideshow.macroenabled.12":                 ".ppsm",
+	"application/vnd.ms-powerpoint.template.macroenabled.12":                  ".potm",
+	"application/vnd.ms-powerpoint":                                           ".ppt",
+	"application/vnd.ms-word.document.macroenabled.12":                        ".docm",
+	"application/vnd.ms-word.template.macroenabled.12":                        ".dotm",
+	"application/vnd.openxmlformats-officedocument.presentationml.template":   ".potx",
+	"application/vnd.openxmlformats-officedocument.spreadsheetml.template":    ".xltx",
+	"application/vnd.openxmlformats-officedocument.wordprocessingml.template": ".dotx",
+	"application/vnd.sun.xml.writer":                                          ".sxw",
+	"text/richtext":                                                           ".rtf",
+}
+*/
 
 // Load the example export formats into exportFormats for testing
-func TestInternalLoadExampleExportFormats(t *testing.T) {
-	exportFormatsOnce.Do(func() {})
-	assert.NoError(t, json.Unmarshal([]byte(exampleExportFormats), &_exportFormats))
+func TestInternalLoadExampleFormats(t *testing.T) {
+	fetchFormatsOnce.Do(func() {})
+	buf, err := ioutil.ReadFile(filepath.FromSlash("test/about.json"))
+	var about struct {
+		ExportFormats map[string][]string `json:"exportFormats,omitempty"`
+		ImportFormats map[string][]string `json:"importFormats,omitempty"`
+	}
+	require.NoError(t, err)
+	require.NoError(t, json.Unmarshal(buf, &about))
+	_exportFormats = fixMimeTypeMap(about.ExportFormats)
+	_importFormats = fixMimeTypeMap(about.ImportFormats)
 }
 
 func TestInternalParseExtensions(t *testing.T) {
@@ -65,27 +58,24 @@ func TestInternalParseExtensions(t *testing.T) {
 		want    []string
 		wantErr error
 	}{
-		{"doc", []string{"doc"}, nil},
-		{" docx ,XLSX, 	pptx,svg", []string{"docx", "xlsx", "pptx", "svg"}, nil},
-		{"docx,svg,Docx", []string{"docx", "svg"}, nil},
-		{"docx,potato,docx", []string{"docx"}, errors.New(`couldn't find mime type for extension "potato"`)},
+		{"doc", []string{".doc"}, nil},
+		{" docx ,XLSX, 	pptx,svg", []string{".docx", ".xlsx", ".pptx", ".svg"}, nil},
+		{"docx,svg,Docx", []string{".docx", ".svg"}, nil},
+		{"docx,potato,docx", []string{".docx"}, errors.New(`couldn't find MIME type for extension ".potato"`)},
 	} {
-		f := new(Fs)
-		gotErr := f.parseExtensions(test.in)
+		extensions, _, gotErr := parseExtensions(test.in)
 		if test.wantErr == nil {
 			assert.NoError(t, gotErr)
 		} else {
 			assert.EqualError(t, gotErr, test.wantErr.Error())
 		}
-		assert.Equal(t, test.want, f.extensions)
+		assert.Equal(t, test.want, extensions)
 	}
 
 	// Test it is appending
-	f := new(Fs)
-	assert.Nil(t, f.parseExtensions("docx,svg"))
-	assert.Nil(t, f.parseExtensions("docx,svg,xlsx"))
-	assert.Equal(t, []string{"docx", "svg", "xlsx"}, f.extensions)
-
+	extensions, _, gotErr := parseExtensions("docx,svg", "docx,svg,xlsx")
+	assert.NoError(t, gotErr)
+	assert.Equal(t, []string{".docx", ".svg", ".xlsx"}, extensions)
 }
 
 func TestInternalFindExportFormat(t *testing.T) {
@@ -99,17 +89,17 @@ func TestInternalFindExportFormat(t *testing.T) {
 		wantMimeType  string
 	}{
 		{[]string{}, "", ""},
-		{[]string{"pdf"}, "pdf", "application/pdf"},
-		{[]string{"pdf", "rtf", "xls"}, "pdf", "application/pdf"},
-		{[]string{"xls", "rtf", "pdf"}, "rtf", "application/rtf"},
-		{[]string{"xls", "csv", "svg"}, "", ""},
+		{[]string{".pdf"}, ".pdf", "application/pdf"},
+		{[]string{".pdf", ".rtf", ".xls"}, ".pdf", "application/pdf"},
+		{[]string{".xls", ".rtf", ".pdf"}, ".rtf", "application/rtf"},
+		{[]string{".xls", ".csv", ".svg"}, "", ""},
 	} {
 		f := new(Fs)
-		f.extensions = test.extensions
+		f.exportExtensions = test.extensions
 		gotExtension, gotFilename, gotMimeType, gotIsDocument := f.findExportFormat(item)
 		assert.Equal(t, test.wantExtension, gotExtension)
 		if test.wantExtension != "" {
-			assert.Equal(t, item.Name+"."+gotExtension, gotFilename)
+			assert.Equal(t, item.Name+gotExtension, gotFilename)
 		} else {
 			assert.Equal(t, "", gotFilename)
 		}
@@ -117,3 +107,146 @@ func TestInternalFindExportFormat(t *testing.T) {
 		assert.Equal(t, true, gotIsDocument)
 	}
 }
+
+func TestMimeTypesToExtension(t *testing.T) {
+	for mimeType, extension := range _mimeTypeToExtension {
+		extensions, err := mime.ExtensionsByType(mimeType)
+		assert.NoError(t, err)
+		assert.Contains(t, extensions, extension)
+	}
+}
+
+func TestExtensionToMimeType(t *testing.T) {
+	for mimeType, extension := range _mimeTypeToExtension {
+		gotMimeType := mime.TypeByExtension(extension)
+		mediatype, _, err := mime.ParseMediaType(gotMimeType)
+		assert.NoError(t, err)
+		assert.Equal(t, mimeType, mediatype)
+	}
+}
+
+func TestExtensionsForExportFormats(t *testing.T) {
+	if _exportFormats == nil {
+		t.Error("exportFormats == nil")
+	}
+	for fromMT, toMTs := range _exportFormats {
+		for _, toMT := range toMTs {
+			if !isInternalMimeType(toMT) {
+				extensions, err := mime.ExtensionsByType(toMT)
+				assert.NoError(t, err, "invalid MIME type %q", toMT)
+				assert.NotEmpty(t, extensions, "No extension found for %q (from: %q)", fromMT, toMT)
+			}
+		}
+	}
+}
+
+func TestExtensionsForImportFormats(t *testing.T) {
+	t.Skip()
+	if _importFormats == nil {
+		t.Error("_importFormats == nil")
+	}
+	for fromMT := range _importFormats {
+		if !isInternalMimeType(fromMT) {
+			extensions, err := mime.ExtensionsByType(fromMT)
+			assert.NoError(t, err, "invalid MIME type %q", fromMT)
+			assert.NotEmpty(t, extensions, "No extension found for %q", fromMT)
+		}
+	}
+}
+
+func (f *Fs) InternalTestDocumentImport(t *testing.T) {
+	oldAllow := f.opt.AllowImportNameChange
+	f.opt.AllowImportNameChange = true
+	defer func() {
+		f.opt.AllowImportNameChange = oldAllow
+	}()
+
+	testFilesPath, err := filepath.Abs(filepath.FromSlash("test/files"))
+	require.NoError(t, err)
+
+	testFilesFs, err := fs.NewFs(testFilesPath)
+	require.NoError(t, err)
+
+	_, f.importMimeTypes, err = parseExtensions("odt,ods,doc")
+	require.NoError(t, err)
+
+	err = operations.CopyFile(f, testFilesFs, "example2.doc", "example2.doc")
+	require.NoError(t, err)
+}
+
+func (f *Fs) InternalTestDocumentUpdate(t *testing.T) {
+	testFilesPath, err := filepath.Abs(filepath.FromSlash("test/files"))
+	require.NoError(t, err)
+
+	testFilesFs, err := fs.NewFs(testFilesPath)
+	require.NoError(t, err)
+
+	_, f.importMimeTypes, err = parseExtensions("odt,ods,doc")
+	require.NoError(t, err)
+
+	err = operations.CopyFile(f, testFilesFs, "example2.xlsx", "example1.ods")
+	require.NoError(t, err)
+}
+
+func (f *Fs) InternalTestDocumentExport(t *testing.T) {
+	var buf bytes.Buffer
+	var err error
+
+	f.exportExtensions, _, err = parseExtensions("txt")
+	require.NoError(t, err)
+
+	obj, err := f.NewObject("example2.txt")
+	require.NoError(t, err)
+
+	rc, err := obj.Open()
+	require.NoError(t, err)
+	defer func() { require.NoError(t, rc.Close()) }()
+
+	_, err = io.Copy(&buf, rc)
+	require.NoError(t, err)
+	text := buf.String()
+
+	for _, excerpt := range []string{
+		"Lorem ipsum dolor sit amet, consectetur",
+		"porta at ultrices in, consectetur at augue.",
+	} {
+		require.Contains(t, text, excerpt)
+	}
+}
+
+func (f *Fs) InternalTestDocumentLink(t *testing.T) {
+	var buf bytes.Buffer
+	var err error
+
+	f.exportExtensions, _, err = parseExtensions("link.html")
+	require.NoError(t, err)
+
+	obj, err := f.NewObject("example2.link.html")
+	require.NoError(t, err)
+
+	rc, err := obj.Open()
+	require.NoError(t, err)
+	defer func() { require.NoError(t, rc.Close()) }()
+
+	_, err = io.Copy(&buf, rc)
+	require.NoError(t, err)
+	text := buf.String()
+
+	require.True(t, strings.HasPrefix(text, "<html>"))
+	require.True(t, strings.HasSuffix(text, "</html>\n"))
+	for _, excerpt := range []string{
+		`<meta http-equiv="refresh"`,
+		`Loading <a href="`,
+	} {
+		require.Contains(t, text, excerpt)
+	}
+}
+
+func (f *Fs) InternalTest(t *testing.T) {
+	t.Run("DocumentImport", f.InternalTestDocumentImport)
+	t.Run("DocumentUpdate", f.InternalTestDocumentUpdate)
+	t.Run("DocumentExport", f.InternalTestDocumentExport)
+	t.Run("DocumentLink", f.InternalTestDocumentLink)
+}
+
+var _ fstests.InternalTester = (*Fs)(nil)

backend/drive/drive_test.go

@@ -1,10 +1,10 @@
 // Test Drive filesystem interface
-package drive_test
+package drive
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/drive"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
@@ -12,6 +12,23 @@ import (
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
 		RemoteName: "TestDrive:",
-		NilObject:  (*drive.Object)(nil),
+		NilObject:  (*Object)(nil),
+		ChunkedUpload: fstests.ChunkedUploadConfig{
+			MinChunkSize:  minChunkSize,
+			CeilChunkSize: fstests.NextPowerOfTwo,
+		},
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+func (f *Fs) SetUploadCutoff(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadCutoff(cs)
+}
+
+var (
+	_ fstests.SetUploadChunkSizer = (*Fs)(nil)
+	_ fstests.SetUploadCutoffer   = (*Fs)(nil)
+)

backend/drive/test/about.json

@@ -0,0 +1,178 @@
+{
+ "importFormats": {
+  "text/tab-separated-values": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/x-vnd.oasis.opendocument.presentation": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "image/jpeg": [
+   "application/vnd.google-apps.document"
+  ],
+  "image/bmp": [
+   "application/vnd.google-apps.document"
+  ],
+  "image/gif": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.ms-excel.sheet.macroenabled.12": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/vnd.openxmlformats-officedocument.wordprocessingml.template": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.ms-powerpoint.presentation.macroenabled.12": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "application/vnd.ms-word.template.macroenabled.12": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.openxmlformats-officedocument.wordprocessingml.document": [
+   "application/vnd.google-apps.document"
+  ],
+  "image/pjpeg": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.google-apps.script+text/plain": [
+   "application/vnd.google-apps.script"
+  ],
+  "application/vnd.ms-excel": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/vnd.sun.xml.writer": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.ms-word.document.macroenabled.12": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.ms-powerpoint.slideshow.macroenabled.12": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "text/rtf": [
+   "application/vnd.google-apps.document"
+  ],
+  "text/plain": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.oasis.opendocument.spreadsheet": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/x-vnd.oasis.opendocument.spreadsheet": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "image/png": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/x-vnd.oasis.opendocument.text": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/msword": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/pdf": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/json": [
+   "application/vnd.google-apps.script"
+  ],
+  "application/x-msmetafile": [
+   "application/vnd.google-apps.drawing"
+  ],
+  "application/vnd.openxmlformats-officedocument.spreadsheetml.template": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/vnd.ms-powerpoint": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "application/vnd.ms-excel.template.macroenabled.12": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "image/x-bmp": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/rtf": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.openxmlformats-officedocument.presentationml.template": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "image/x-png": [
+   "application/vnd.google-apps.document"
+  ],
+  "text/html": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.oasis.opendocument.text": [
+   "application/vnd.google-apps.document"
+  ],
+  "application/vnd.openxmlformats-officedocument.presentationml.presentation": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/vnd.google-apps.script+json": [
+   "application/vnd.google-apps.script"
+  ],
+  "application/vnd.openxmlformats-officedocument.presentationml.slideshow": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "application/vnd.ms-powerpoint.template.macroenabled.12": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "text/csv": [
+   "application/vnd.google-apps.spreadsheet"
+  ],
+  "application/vnd.oasis.opendocument.presentation": [
+   "application/vnd.google-apps.presentation"
+  ],
+  "image/jpg": [
+   "application/vnd.google-apps.document"
+  ],
+  "text/richtext": [
+   "application/vnd.google-apps.document"
+  ]
+ },
+ "exportFormats": {
+  "application/vnd.google-apps.document": [
+   "application/rtf",
+   "application/vnd.oasis.opendocument.text",
+   "text/html",
+   "application/pdf",
+   "application/epub+zip",
+   "application/zip",
+   "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+   "text/plain"
+  ],
+  "application/vnd.google-apps.spreadsheet": [
+   "application/x-vnd.oasis.opendocument.spreadsheet",
+   "text/tab-separated-values",
+   "application/pdf",
+   "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
+   "text/csv",
+   "application/zip",
+   "application/vnd.oasis.opendocument.spreadsheet"
+  ],
+  "application/vnd.google-apps.jam": [
+   "application/pdf"
+  ],
+  "application/vnd.google-apps.script": [
+   "application/vnd.google-apps.script+json"
+  ],
+  "application/vnd.google-apps.presentation": [
+   "application/vnd.oasis.opendocument.presentation",
+   "application/pdf",
+   "application/vnd.openxmlformats-officedocument.presentationml.presentation",
+   "text/plain"
+  ],
+  "application/vnd.google-apps.form": [
+   "application/zip"
+  ],
+  "application/vnd.google-apps.drawing": [
+   "image/svg+xml",
+   "image/png",
+   "application/pdf",
+   "image/jpeg"
+  ]
+ }
+}

backend/drive/upload.go

@@ -50,11 +50,12 @@ type resumableUpload struct {
 }
 
 // Upload the io.Reader in of size bytes with contentType and info
-func (f *Fs) Upload(in io.Reader, size int64, contentType string, fileID string, info *drive.File, remote string) (*drive.File, error) {
-	params := make(url.Values)
-	params.Set("alt", "json")
-	params.Set("uploadType", "resumable")
-	params.Set("fields", partialFields)
+func (f *Fs) Upload(in io.Reader, size int64, contentType, fileID, remote string, info *drive.File) (*drive.File, error) {
+	params := url.Values{
+		"alt":        {"json"},
+		"uploadType": {"resumable"},
+		"fields":     {partialFields},
+	}
 	if f.isTeamDrive {
 		params.Set("supportsTeamDrives", "true")
 	}

backend/dropbox/dropbox.go

@@ -79,8 +79,8 @@ const (
 	// Choose 48MB which is 91% of Maximum speed.  rclone by
 	// default does 4 transfers so this should use 4*48MB = 192MB
 	// by default.
-	defaultChunkSize = 48 * 1024 * 1024
-	maxChunkSize     = 150 * 1024 * 1024
+	defaultChunkSize = 48 * fs.MebiByte
+	maxChunkSize     = 150 * fs.MebiByte
 )
 
 var (
@@ -120,8 +120,15 @@ func init() {
 			Name: config.ConfigClientSecret,
 			Help: "Dropbox App Client Secret\nLeave blank normally.",
 		}, {
-			Name:     "chunk_size",
-			Help:     fmt.Sprintf("Upload chunk size. Max %v.", fs.SizeSuffix(maxChunkSize)),
+			Name: "chunk_size",
+			Help: fmt.Sprintf(`Upload chunk size. (< %v).
+
+Any files larger than this will be uploaded in chunks of this size.
+
+Note that chunks are buffered in memory (one at a time) so rclone can
+deal with retries.  Setting this larger will increase the speed
+slightly (at most 10%% for 128MB in tests) at the cost of using more
+memory.  It can be set smaller if you are tight on memory.`, fs.SizeSuffix(maxChunkSize)),
 			Default:  fs.SizeSuffix(defaultChunkSize),
 			Advanced: true,
 		}},
@@ -195,6 +202,25 @@ func shouldRetry(err error) (bool, error) {
 	return fserrors.ShouldRetry(err), err
 }
 
+func checkUploadChunkSize(cs fs.SizeSuffix) error {
+	const minChunkSize = fs.Byte
+	if cs < minChunkSize {
+		return errors.Errorf("%s is less than %s", cs, minChunkSize)
+	}
+	if cs > maxChunkSize {
+		return errors.Errorf("%s is greater than %s", cs, maxChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadChunkSize(cs)
+	if err == nil {
+		old, f.opt.ChunkSize = f.opt.ChunkSize, cs
+	}
+	return
+}
+
 // NewFs contstructs an Fs from the path, container:path
 func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	// Parse config into Options struct
@@ -203,8 +229,9 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		return nil, err
 	}
-	if opt.ChunkSize > maxChunkSize {
-		return nil, errors.Errorf("chunk size too big, must be < %v", maxChunkSize)
+	err = checkUploadChunkSize(opt.ChunkSize)
+	if err != nil {
+		return nil, errors.Wrap(err, "dropbox: chunk size")
 	}
 
 	// Convert the old token if it exists.  The old token was just

backend/dropbox/dropbox_test.go

@@ -1,10 +1,10 @@
 // Test Dropbox filesystem interface
-package dropbox_test
+package dropbox
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/dropbox"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
@@ -12,6 +12,15 @@ import (
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
 		RemoteName: "TestDropbox:",
-		NilObject:  (*dropbox.Object)(nil),
+		NilObject:  (*Object)(nil),
+		ChunkedUpload: fstests.ChunkedUploadConfig{
+			MaxChunkSize: maxChunkSize,
+		},
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+var _ fstests.SetUploadChunkSizer = (*Fs)(nil)

backend/ftp/ftp.go

@@ -704,6 +704,11 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	path := path.Join(o.fs.root, o.remote)
 	// remove the file if upload failed
 	remove := func() {
+		// Give the FTP server a chance to get its internal state in order after the error.
+		// The error may have been local in which case we closed the connection.  The server
+		// may still be dealing with it for a moment. A sleep isn't ideal but I haven't been
+		// able to think of a better method to find out if the server has finished - ncw
+		time.Sleep(1 * time.Second)
 		removeErr := o.Remove()
 		if removeErr != nil {
 			fs.Debugf(o, "Failed to remove: %v", removeErr)
@@ -717,7 +722,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	}
 	err = c.Stor(path, in)
 	if err != nil {
-		_ = c.Quit()
+		_ = c.Quit() // toss this connection to avoid sync errors
 		remove()
 		return errors.Wrap(err, "update stor")
 	}

backend/googlecloudstorage/googlecloudstorage.go

@@ -345,7 +345,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	}
 
 	// try loading service account credentials from env variable, then from a file
-	if opt.ServiceAccountCredentials != "" && opt.ServiceAccountFile != "" {
+	if opt.ServiceAccountCredentials == "" && opt.ServiceAccountFile != "" {
 		loadedCreds, err := ioutil.ReadFile(os.ExpandEnv(opt.ServiceAccountFile))
 		if err != nil {
 			return nil, errors.Wrap(err, "error opening service account credentials file")

backend/hubic/auth.go

@@ -2,7 +2,9 @@ package hubic
 
 import (
 	"net/http"
+	"time"
 
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/swift"
 )
 
@@ -21,12 +23,17 @@ func newAuth(f *Fs) *auth {
 // Request constructs a http.Request for authentication
 //
 // returns nil for not needed
-func (a *auth) Request(*swift.Connection) (*http.Request, error) {
-	err := a.f.getCredentials()
-	if err != nil {
-		return nil, err
+func (a *auth) Request(*swift.Connection) (r *http.Request, err error) {
+	const retries = 10
+	for try := 1; try <= retries; try++ {
+		err = a.f.getCredentials()
+		if err == nil {
+			break
+		}
+		time.Sleep(100 * time.Millisecond)
+		fs.Debugf(a.f, "retrying auth request %d/%d: %v", try, retries, err)
 	}
-	return nil, nil
+	return nil, err
 }
 
 // Response parses the result of an http request

backend/hubic/hubic.go

@@ -60,13 +60,13 @@ func init() {
 				log.Fatalf("Failed to configure token: %v", err)
 			}
 		},
-		Options: []fs.Option{{
+		Options: append([]fs.Option{{
 			Name: config.ConfigClientID,
 			Help: "Hubic Client Id\nLeave blank normally.",
 		}, {
 			Name: config.ConfigClientSecret,
 			Help: "Hubic Client Secret\nLeave blank normally.",
-		}},
+		}}, swift.SharedOptions...),
 	})
 }
 

backend/jottacloud/api/types.go

@@ -233,15 +233,17 @@ GET http://www.jottacloud.com/JFS/<account>/<device>/<mountpoint>/.../<file>
 
 // JottaFile represents a Jottacloud file
 type JottaFile struct {
-	XMLName    xml.Name
-	Name       string `xml:"name,attr"`
-	Deleted    Flag   `xml:"deleted,attr"`
-	State      string `xml:"currentRevision>state"`
-	CreatedAt  Time   `xml:"currentRevision>created"`
-	ModifiedAt Time   `xml:"currentRevision>modified"`
-	Updated    Time   `xml:"currentRevision>updated"`
-	Size       int64  `xml:"currentRevision>size"`
-	MD5        string `xml:"currentRevision>md5"`
+	XMLName         xml.Name
+	Name            string `xml:"name,attr"`
+	Deleted         Flag   `xml:"deleted,attr"`
+	PublicSharePath string `xml:"publicSharePath"`
+	State           string `xml:"currentRevision>state"`
+	CreatedAt       Time   `xml:"currentRevision>created"`
+	ModifiedAt      Time   `xml:"currentRevision>modified"`
+	Updated         Time   `xml:"currentRevision>updated"`
+	Size            int64  `xml:"currentRevision>size"`
+	MimeType        string `xml:"currentRevision>mime"`
+	MD5             string `xml:"currentRevision>md5"`
 }
 
 // Error is a custom Error for wrapping Jottacloud error responses

backend/jottacloud/jottacloud.go

@@ -25,6 +25,7 @@ import (
 	"github.com/ncw/rclone/fs/fserrors"
 	"github.com/ncw/rclone/fs/fshttp"
 	"github.com/ncw/rclone/fs/hash"
+	"github.com/ncw/rclone/fs/walk"
 	"github.com/ncw/rclone/lib/pacer"
 	"github.com/ncw/rclone/lib/rest"
 	"github.com/pkg/errors"
@@ -39,6 +40,7 @@ const (
 	defaultMountpoint = "Sync"
 	rootURL           = "https://www.jottacloud.com/jfs/"
 	apiURL            = "https://api.jottacloud.com"
+	shareURL          = "https://www.jottacloud.com/"
 	cachePrefix       = "rclone-jcmd5-"
 )
 
@@ -71,6 +73,16 @@ func init() {
 			Help:     "Files bigger than this will be cached on disk to calculate the MD5 if required.",
 			Default:  fs.SizeSuffix(10 * 1024 * 1024),
 			Advanced: true,
+		}, {
+			Name:     "hard_delete",
+			Help:     "Delete files permanently rather than putting them into the trash.",
+			Default:  false,
+			Advanced: true,
+		}, {
+			Name:     "unlink",
+			Help:     "Remove existing public link to file/folder with link command rather than creating.\nDefault is false, meaning link command will create or retrieve public link.",
+			Default:  false,
+			Advanced: true,
 		}},
 	})
 }
@@ -81,12 +93,15 @@ type Options struct {
 	Pass               string        `config:"pass"`
 	Mountpoint         string        `config:"mountpoint"`
 	MD5MemoryThreshold fs.SizeSuffix `config:"md5_memory_limit"`
+	HardDelete         bool          `config:"hard_delete"`
+	Unlink             bool          `config:"unlink"`
 }
 
 // Fs represents a remote jottacloud
 type Fs struct {
 	name        string
 	root        string
+	user        string
 	opt         Options
 	features    *fs.Features
 	endpointURL string
@@ -104,6 +119,7 @@ type Object struct {
 	size        int64
 	modTime     time.Time
 	md5         string
+	mimeType    string
 }
 
 // ------------------------------------------------------------
@@ -179,24 +195,32 @@ func (f *Fs) readMetaDataForPath(path string) (info *api.JottaFile, err error) {
 	return &result, nil
 }
 
-// setEndpointUrl reads the account id and generates the API endpoint URL
-func (f *Fs) setEndpointURL(user, mountpoint string) (err error) {
+// getAccountInfo retrieves account information
+func (f *Fs) getAccountInfo() (info *api.AccountInfo, err error) {
 	opts := rest.Opts{
 		Method: "GET",
-		Path:   rest.URLPathEscape(user),
+		Path:   urlPathEscape(f.user),
 	}
 
-	var result api.AccountInfo
 	var resp *http.Response
 	err = f.pacer.Call(func() (bool, error) {
-		resp, err = f.srv.CallXML(&opts, nil, &result)
+		resp, err = f.srv.CallXML(&opts, nil, &info)
 		return shouldRetry(resp, err)
 	})
 	if err != nil {
-		return err
+		return nil, err
 	}
 
-	f.endpointURL = rest.URLPathEscape(path.Join(result.Username, defaultDevice, mountpoint))
+	return info, nil
+}
+
+// setEndpointUrl reads the account id and generates the API endpoint URL
+func (f *Fs) setEndpointURL(mountpoint string) (err error) {
+	info, err := f.getAccountInfo()
+	if err != nil {
+		return errors.Wrap(err, "failed to get endpoint url")
+	}
+	f.endpointURL = urlPathEscape(path.Join(info.Username, defaultDevice, mountpoint))
 	return nil
 }
 
@@ -217,9 +241,19 @@ func errorHandler(resp *http.Response) error {
 	return errResponse
 }
 
+// Jottacloud want's '+' to be URL encoded even though the RFC states it's not reserved
+func urlPathEscape(in string) string {
+	return strings.Replace(rest.URLPathEscape(in), "+", "%2B", -1)
+}
+
+// filePathRaw returns an unescaped file path (f.root, file)
+func (f *Fs) filePathRaw(file string) string {
+	return path.Join(f.endpointURL, replaceReservedChars(path.Join(f.root, file)))
+}
+
 // filePath returns a escaped file path (f.root, file)
 func (f *Fs) filePath(file string) string {
-	return rest.URLPathEscape(path.Join(f.endpointURL, replaceReservedChars(path.Join(f.root, file))))
+	return urlPathEscape(f.filePathRaw(file))
 }
 
 // filePath returns a escaped file path (f.root, remote)
@@ -253,6 +287,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	f := &Fs{
 		name: name,
 		root: root,
+		user: opt.User,
 		opt:  *opt,
 		//endpointURL: rest.URLPathEscape(path.Join(user, defaultDevice, opt.Mountpoint)),
 		srv:   rest.NewClient(fshttp.NewClient(fs.Config)).SetRoot(rootURL),
@@ -261,6 +296,8 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	f.features = (&fs.Features{
 		CaseInsensitive:         true,
 		CanHaveEmptyDirectories: true,
+		ReadMimeType:            true,
+		WriteMimeType:           true,
 	}).Fill(f)
 
 	if user == "" || pass == "" {
@@ -270,7 +307,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	f.srv.SetUserPass(opt.User, opt.Pass)
 	f.srv.SetErrorHandler(errorHandler)
 
-	err = f.setEndpointURL(opt.User, opt.Mountpoint)
+	err = f.setEndpointURL(opt.Mountpoint)
 	if err != nil {
 		return nil, errors.Wrap(err, "couldn't get account info")
 	}
@@ -412,6 +449,102 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
 	return entries, nil
 }
 
+// listFileDirFn is called from listFileDir to handle an object.
+type listFileDirFn func(fs.DirEntry) error
+
+// List the objects and directories into entries, from a
+// special kind of JottaFolder representing a FileDirLis
+func (f *Fs) listFileDir(remoteStartPath string, startFolder *api.JottaFolder, fn listFileDirFn) error {
+	pathPrefix := "/" + f.filePathRaw("") // Non-escaped prefix of API paths to be cut off, to be left with the remote path including the remoteStartPath
+	pathPrefixLength := len(pathPrefix)
+	startPath := path.Join(pathPrefix, remoteStartPath) // Non-escaped API path up to and including remoteStartPath, to decide if it should be created as a new dir object
+	startPathLength := len(startPath)
+	for i := range startFolder.Folders {
+		folder := &startFolder.Folders[i]
+		if folder.Deleted {
+			return nil
+		}
+		folderPath := restoreReservedChars(path.Join(folder.Path, folder.Name))
+		folderPathLength := len(folderPath)
+		var remoteDir string
+		if folderPathLength > pathPrefixLength {
+			remoteDir = folderPath[pathPrefixLength+1:]
+			if folderPathLength > startPathLength {
+				d := fs.NewDir(remoteDir, time.Time(folder.ModifiedAt))
+				err := fn(d)
+				if err != nil {
+					return err
+				}
+			}
+		}
+		for i := range folder.Files {
+			file := &folder.Files[i]
+			if file.Deleted || file.State != "COMPLETED" {
+				continue
+			}
+			remoteFile := path.Join(remoteDir, restoreReservedChars(file.Name))
+			o, err := f.newObjectWithInfo(remoteFile, file)
+			if err != nil {
+				return err
+			}
+			err = fn(o)
+			if err != nil {
+				return err
+			}
+		}
+	}
+	return nil
+}
+
+// ListR lists the objects and directories of the Fs starting
+// from dir recursively into out.
+//
+// dir should be "" to start from the root, and should not
+// have trailing slashes.
+//
+// This should return ErrDirNotFound if the directory isn't
+// found.
+//
+// It should call callback for each tranche of entries read.
+// These need not be returned in any particular order.  If
+// callback returns an error then the listing will stop
+// immediately.
+//
+// Don't implement this unless you have a more efficient way
+// of listing recursively that doing a directory traversal.
+func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
+	opts := rest.Opts{
+		Method:     "GET",
+		Path:       f.filePath(dir),
+		Parameters: url.Values{},
+	}
+	opts.Parameters.Set("mode", "list")
+
+	var resp *http.Response
+	var result api.JottaFolder // Could be JottaFileDirList, but JottaFolder is close enough
+	err = f.pacer.Call(func() (bool, error) {
+		resp, err = f.srv.CallXML(&opts, nil, &result)
+		return shouldRetry(resp, err)
+	})
+	if err != nil {
+		if apiErr, ok := err.(*api.Error); ok {
+			// does not exist
+			if apiErr.StatusCode == http.StatusNotFound {
+				return fs.ErrorDirNotFound
+			}
+		}
+		return errors.Wrap(err, "couldn't list files")
+	}
+	list := walk.NewListRHelper(callback)
+	err = f.listFileDir(dir, &result, func(entry fs.DirEntry) error {
+		return list.Add(entry)
+	})
+	if err != nil {
+		return err
+	}
+	return list.Flush()
+}
+
 // Creates from the parameters passed in a half finished Object which
 // must have setMetaData called on it
 //
@@ -485,7 +618,11 @@ func (f *Fs) purgeCheck(dir string, check bool) (err error) {
 		NoResponse: true,
 	}
 
-	opts.Parameters.Set("dlDir", "true")
+	if f.opt.HardDelete {
+		opts.Parameters.Set("rmDir", "true")
+	} else {
+		opts.Parameters.Set("dlDir", "true")
+	}
 
 	var resp *http.Response
 	err = f.pacer.Call(func() (bool, error) {
@@ -493,7 +630,7 @@ func (f *Fs) purgeCheck(dir string, check bool) (err error) {
 		return shouldRetry(resp, err)
 	})
 	if err != nil {
-		return errors.Wrap(err, "rmdir failed")
+		return errors.Wrap(err, "couldn't purge directory")
 	}
 
 	// TODO: Parse response?
@@ -566,7 +703,7 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
 	info, err := f.copyOrMove("cp", srcObj.filePath(), remote)
 
 	if err != nil {
-		return nil, errors.Wrap(err, "copy failed")
+		return nil, errors.Wrap(err, "couldn't copy file")
 	}
 
 	return f.newObjectWithInfo(remote, info)
@@ -596,7 +733,7 @@ func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
 	info, err := f.copyOrMove("mv", srcObj.filePath(), remote)
 
 	if err != nil {
-		return nil, errors.Wrap(err, "move failed")
+		return nil, errors.Wrap(err, "couldn't move file")
 	}
 
 	return f.newObjectWithInfo(remote, info)
@@ -640,11 +777,74 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
 	_, err = f.copyOrMove("mvDir", path.Join(f.endpointURL, replaceReservedChars(srcPath))+"/", dstRemote)
 
 	if err != nil {
-		return errors.Wrap(err, "moveDir failed")
+		return errors.Wrap(err, "couldn't move directory")
 	}
 	return nil
 }
 
+// PublicLink generates a public link to the remote path (usually readable by anyone)
+func (f *Fs) PublicLink(remote string) (link string, err error) {
+	opts := rest.Opts{
+		Method:     "GET",
+		Path:       f.filePath(remote),
+		Parameters: url.Values{},
+	}
+
+	if f.opt.Unlink {
+		opts.Parameters.Set("mode", "disableShare")
+	} else {
+		opts.Parameters.Set("mode", "enableShare")
+	}
+
+	var resp *http.Response
+	var result api.JottaFile
+	err = f.pacer.Call(func() (bool, error) {
+		resp, err = f.srv.CallXML(&opts, nil, &result)
+		return shouldRetry(resp, err)
+	})
+
+	if apiErr, ok := err.(*api.Error); ok {
+		// does not exist
+		if apiErr.StatusCode == http.StatusNotFound {
+			return "", fs.ErrorObjectNotFound
+		}
+	}
+	if err != nil {
+		if f.opt.Unlink {
+			return "", errors.Wrap(err, "couldn't remove public link")
+		}
+		return "", errors.Wrap(err, "couldn't create public link")
+	}
+	if f.opt.Unlink {
+		if result.PublicSharePath != "" {
+			return "", errors.Errorf("couldn't remove public link - %q", result.PublicSharePath)
+		}
+		return "", nil
+	}
+	if result.PublicSharePath == "" {
+		return "", errors.New("couldn't create public link - no link path received")
+	}
+	link = path.Join(shareURL, result.PublicSharePath)
+	return link, nil
+}
+
+// About gets quota information
+func (f *Fs) About() (*fs.Usage, error) {
+	info, err := f.getAccountInfo()
+	if err != nil {
+		return nil, err
+	}
+
+	usage := &fs.Usage{
+		Used: fs.NewUsageValue(info.Usage),
+	}
+	if info.Capacity > 0 {
+		usage.Total = fs.NewUsageValue(info.Capacity)
+		usage.Free = fs.NewUsageValue(info.Capacity - info.Usage)
+	}
+	return usage, nil
+}
+
 // Hashes returns the supported hash sets.
 func (f *Fs) Hashes() hash.Set {
 	return hash.Set(hash.MD5)
@@ -688,11 +888,17 @@ func (o *Object) Size() int64 {
 	return o.size
 }
 
+// MimeType of an Object if known, "" otherwise
+func (o *Object) MimeType() string {
+	return o.mimeType
+}
+
 // setMetaData sets the metadata from info
 func (o *Object) setMetaData(info *api.JottaFile) (err error) {
 	o.hasMetaData = true
 	o.size = int64(info.Size)
 	o.md5 = info.MD5
+	o.mimeType = info.MimeType
 	o.modTime = time.Time(info.ModifiedAt)
 	return nil
 }
@@ -879,9 +1085,14 @@ func (o *Object) Remove() error {
 		Method:     "POST",
 		Path:       o.filePath(),
 		Parameters: url.Values{},
+		NoResponse: true,
 	}
 
-	opts.Parameters.Set("dl", "true")
+	if o.fs.opt.HardDelete {
+		opts.Parameters.Set("rm", "true")
+	} else {
+		opts.Parameters.Set("dl", "true")
+	}
 
 	return o.fs.pacer.Call(func() (bool, error) {
 		resp, err := o.fs.srv.CallXML(&opts, nil, nil)
@@ -891,10 +1102,14 @@ func (o *Object) Remove() error {
 
 // Check the interfaces are satisfied
 var (
-	_ fs.Fs       = (*Fs)(nil)
-	_ fs.Purger   = (*Fs)(nil)
-	_ fs.Copier   = (*Fs)(nil)
-	_ fs.Mover    = (*Fs)(nil)
-	_ fs.DirMover = (*Fs)(nil)
-	_ fs.Object   = (*Object)(nil)
+	_ fs.Fs           = (*Fs)(nil)
+	_ fs.Purger       = (*Fs)(nil)
+	_ fs.Copier       = (*Fs)(nil)
+	_ fs.Mover        = (*Fs)(nil)
+	_ fs.DirMover     = (*Fs)(nil)
+	_ fs.ListRer      = (*Fs)(nil)
+	_ fs.PublicLinker = (*Fs)(nil)
+	_ fs.Abouter      = (*Fs)(nil)
+	_ fs.Object       = (*Object)(nil)
+	_ fs.MimeTyper    = (*Object)(nil)
 )

backend/jottacloud/jottacloud_internal_test.go

@@ -4,50 +4,25 @@ import (
 	"crypto/md5"
 	"fmt"
 	"io"
-	"io/ioutil"
 	"testing"
 
+	"github.com/ncw/rclone/lib/readers"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
-// A test reader to return a test pattern of size
-type testReader struct {
-	size int64
-	c    byte
-}
-
-// Reader is the interface that wraps the basic Read method.
-func (r *testReader) Read(p []byte) (n int, err error) {
-	for i := range p {
-		if r.size <= 0 {
-			return n, io.EOF
-		}
-		p[i] = r.c
-		r.c = (r.c + 1) % 253
-		r.size--
-		n++
-	}
-	return
-}
-
 func TestReadMD5(t *testing.T) {
-	// smoke test the reader
-	b, err := ioutil.ReadAll(&testReader{size: 10})
-	require.NoError(t, err)
-	assert.Equal(t, []byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}, b)
-
 	// Check readMD5 for different size and threshold
 	for _, size := range []int64{0, 1024, 10 * 1024, 100 * 1024} {
 		t.Run(fmt.Sprintf("%d", size), func(t *testing.T) {
 			hasher := md5.New()
-			n, err := io.Copy(hasher, &testReader{size: size})
+			n, err := io.Copy(hasher, readers.NewPatternReader(size))
 			require.NoError(t, err)
 			assert.Equal(t, n, size)
 			wantMD5 := fmt.Sprintf("%x", hasher.Sum(nil))
 			for _, threshold := range []int64{512, 1024, 10 * 1024, 20 * 1024} {
 				t.Run(fmt.Sprintf("%d", threshold), func(t *testing.T) {
-					in := &testReader{size: size}
+					in := readers.NewPatternReader(size)
 					gotMD5, out, cleanup, err := readMD5(in, size, threshold)
 					defer cleanup()
 					require.NoError(t, err)

backend/jottacloud/replace.go

@@ -27,21 +27,14 @@ import (
 var (
 	charMap = map[rune]rune{
 		'\\': '＼', // FULLWIDTH REVERSE SOLIDUS
-		'+':  '＋', // FULLWIDTH PLUS SIGN
 		'*':  '＊', // FULLWIDTH ASTERISK
 		'<':  '＜', // FULLWIDTH LESS-THAN SIGN
 		'>':  '＞', // FULLWIDTH GREATER-THAN SIGN
 		'?':  '？', // FULLWIDTH QUESTION MARK
-		'!':  '！', // FULLWIDTH EXCLAMATION MARK
-		'&':  '＆', // FULLWIDTH AMPERSAND
 		':':  '：', // FULLWIDTH COLON
 		';':  '；', // FULLWIDTH SEMICOLON
 		'|':  '｜', // FULLWIDTH VERTICAL LINE
-		'#':  '＃', // FULLWIDTH NUMBER SIGN
-		'%':  '％', // FULLWIDTH PERCENT SIGN
 		'"':  '＂', // FULLWIDTH QUOTATION MARK - not on the list but seems to be reserved
-		'\'': '＇', // FULLWIDTH APOSTROPHE
-		'~':  '～', // FULLWIDTH TILDE
 		' ':  '␠', // SYMBOL FOR SPACE
 	}
 	invCharMap           map[rune]rune

backend/jottacloud/replace_test.go

@@ -9,8 +9,8 @@ func TestReplace(t *testing.T) {
 	}{
 		{"", ""},
 		{"abc 123", "abc 123"},
-		{`\+*<>?!&:;|#%"'~`, `＼＋＊＜＞？！＆：；｜＃％＂＇～`},
-		{`\+*<>?!&:;|#%"'~\+*<>?!&:;|#%"'~`, `＼＋＊＜＞？！＆：；｜＃％＂＇～＼＋＊＜＞？！＆：；｜＃％＂＇～`},
+		{`\*<>?:;|"`, `＼＊＜＞？：；｜＂`},
+		{`\*<>?:;|"\*<>?:;|"`, `＼＊＜＞？：；｜＂＼＊＜＞？：；｜＂`},
 		{" leading space", "␠leading space"},
 		{"trailing space ", "trailing space␠"},
 		{" leading space/ leading space/ leading space", "␠leading space/␠leading space/␠leading space"},

backend/local/local.go

@@ -16,8 +16,10 @@ import (
 	"unicode/utf8"
 
 	"github.com/ncw/rclone/fs"
+	"github.com/ncw/rclone/fs/accounting"
 	"github.com/ncw/rclone/fs/config/configmap"
 	"github.com/ncw/rclone/fs/config/configstruct"
+	"github.com/ncw/rclone/fs/fserrors"
 	"github.com/ncw/rclone/fs/hash"
 	"github.com/ncw/rclone/lib/readers"
 	"github.com/pkg/errors"
@@ -47,19 +49,33 @@ func init() {
 			ShortOpt: "L",
 			Advanced: true,
 		}, {
-			Name:     "skip_links",
-			Help:     "Don't warn about skipped symlinks.",
+			Name: "skip_links",
+			Help: `Don't warn about skipped symlinks.
+This flag disables warning messages on skipped symlinks or junction
+points, as you explicitly acknowledge that they should be skipped.`,
 			Default:  false,
 			NoPrefix: true,
 			Advanced: true,
 		}, {
-			Name:     "no_unicode_normalization",
-			Help:     "Don't apply unicode normalization to paths and filenames",
+			Name: "no_unicode_normalization",
+			Help: `Don't apply unicode normalization to paths and filenames (Deprecated)
+
+This flag is deprecated now.  Rclone no longer normalizes unicode file
+names, but it compares them with unicode normalization in the sync
+routine instead.`,
 			Default:  false,
 			Advanced: true,
 		}, {
-			Name:     "no_check_updated",
-			Help:     "Don't check to see if the files change during upload",
+			Name: "no_check_updated",
+			Help: `Don't check to see if the files change during upload
+
+Normally rclone checks the size and modification time of files as they
+are being uploaded and aborts with a message which starts "can't copy
+- source file is being updated" if the file changes during upload.
+
+However on some file systems this modification time check may fail (eg
+[Glusterfs #2206](https://github.com/ncw/rclone/issues/2206)) so this
+check can be disabled with this flag.`,
 			Default:  false,
 			Advanced: true,
 		}, {
@@ -151,7 +167,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	}
 	if err == nil && fi.Mode().IsRegular() {
 		// It is a file, so use the parent as the root
-		f.root, _ = getDirFile(f.root)
+		f.root = filepath.Dir(f.root)
 		// return an error with an fs which points to the parent
 		return f, fs.ErrorIsFile
 	}
@@ -280,6 +296,13 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
 			// Follow symlinks if required
 			if f.opt.FollowSymlinks && (mode&os.ModeSymlink) != 0 {
 				fi, err = os.Stat(newPath)
+				if os.IsNotExist(err) {
+					// Skip bad symlinks
+					err = fserrors.NoRetryError(errors.Wrap(err, "symlink"))
+					fs.Errorf(newRemote, "Listing error: %v", err)
+					accounting.Stats.Error(err)
+					continue
+				}
 				if err != nil {
 					return nil, err
 				}
@@ -565,7 +588,7 @@ func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
 	}
 
 	// Create parent of destination
-	dstParentPath, _ := getDirFile(dstPath)
+	dstParentPath := filepath.Dir(dstPath)
 	err = os.MkdirAll(dstParentPath, 0777)
 	if err != nil {
 		return err
@@ -784,7 +807,7 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
 
 // mkdirAll makes all the directories needed to store the object
 func (o *Object) mkdirAll() error {
-	dir, _ := getDirFile(o.path)
+	dir := filepath.Dir(o.path)
 	return os.MkdirAll(dir, 0777)
 }
 
@@ -808,6 +831,12 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 		return err
 	}
 
+	// Pre-allocate the file for performance reasons
+	err = preAllocate(src.Size(), out)
+	if err != nil {
+		fs.Debugf(o, "Failed to pre-allocate: %v", err)
+	}
+
 	// Calculate the hash of the object we are reading as we go along
 	hash, err := hash.NewMultiHasherTypes(hashes)
 	if err != nil {
@@ -872,17 +901,6 @@ func (o *Object) Remove() error {
 	return remove(o.path)
 }
 
-// Return the directory and file from an OS path. Assumes
-// os.PathSeparator is used.
-func getDirFile(s string) (string, string) {
-	i := strings.LastIndex(s, string(os.PathSeparator))
-	dir, file := s[:i], s[i+1:]
-	if dir == "" {
-		dir = string(os.PathSeparator)
-	}
-	return dir, file
-}
-
 // cleanPathFragment cleans an OS path fragment which is part of a
 // bigger path and not necessarily absolute
 func cleanPathFragment(s string) string {

backend/local/preallocate_other.go

@@ -0,0 +1,10 @@
+//+build !windows,!linux
+
+package local
+
+import "os"
+
+// preAllocate the file for performance reasons
+func preAllocate(size int64, out *os.File) error {
+	return nil
+}

backend/local/preallocate_unix.go

@@ -0,0 +1,22 @@
+//+build linux
+
+package local
+
+import (
+	"os"
+
+	"golang.org/x/sys/unix"
+)
+
+// preAllocate the file for performance reasons
+func preAllocate(size int64, out *os.File) error {
+	if size <= 0 {
+		return nil
+	}
+	err := unix.Fallocate(int(out.Fd()), unix.FALLOC_FL_KEEP_SIZE, 0, size)
+	// FIXME could be doing something here
+	// if err == unix.ENOSPC {
+	// 	log.Printf("No space")
+	// }
+	return err
+}

backend/local/preallocate_windows.go

@@ -0,0 +1,79 @@
+//+build windows
+
+package local
+
+import (
+	"os"
+	"syscall"
+	"unsafe"
+
+	"github.com/pkg/errors"
+	"golang.org/x/sys/windows"
+)
+
+var (
+	ntdll                        = windows.NewLazySystemDLL("ntdll.dll")
+	ntQueryVolumeInformationFile = ntdll.NewProc("NtQueryVolumeInformationFile")
+	ntSetInformationFile         = ntdll.NewProc("NtSetInformationFile")
+)
+
+type fileAllocationInformation struct {
+	AllocationSize uint64
+}
+
+type fileFsSizeInformation struct {
+	TotalAllocationUnits     uint64
+	AvailableAllocationUnits uint64
+	SectorsPerAllocationUnit uint32
+	BytesPerSector           uint32
+}
+
+type ioStatusBlock struct {
+	Status, Information uintptr
+}
+
+// preAllocate the file for performance reasons
+func preAllocate(size int64, out *os.File) error {
+	if size <= 0 {
+		return nil
+	}
+
+	var (
+		iosb       ioStatusBlock
+		fsSizeInfo fileFsSizeInformation
+		allocInfo  fileAllocationInformation
+	)
+
+	// Query info about the block sizes on the file system
+	_, _, e1 := ntQueryVolumeInformationFile.Call(
+		uintptr(out.Fd()),
+		uintptr(unsafe.Pointer(&iosb)),
+		uintptr(unsafe.Pointer(&fsSizeInfo)),
+		uintptr(unsafe.Sizeof(fsSizeInfo)),
+		uintptr(3), // FileFsSizeInformation
+	)
+	if e1 != nil && e1 != syscall.Errno(0) {
+		return errors.Wrap(e1, "preAllocate NtQueryVolumeInformationFile failed")
+	}
+
+	// Calculate the allocation size
+	clusterSize := uint64(fsSizeInfo.BytesPerSector) * uint64(fsSizeInfo.SectorsPerAllocationUnit)
+	if clusterSize <= 0 {
+		return errors.Errorf("preAllocate clusterSize %d <= 0", clusterSize)
+	}
+	allocInfo.AllocationSize = (1 + uint64(size-1)/clusterSize) * clusterSize
+
+	// Ask for the allocation
+	_, _, e1 = ntSetInformationFile.Call(
+		uintptr(out.Fd()),
+		uintptr(unsafe.Pointer(&iosb)),
+		uintptr(unsafe.Pointer(&allocInfo)),
+		uintptr(unsafe.Sizeof(allocInfo)),
+		uintptr(19), // FileAllocationInformation
+	)
+	if e1 != nil && e1 != syscall.Errno(0) {
+		return errors.Wrap(e1, "preAllocate NtSetInformationFile failed")
+	}
+
+	return nil
+}

backend/mega/mega.go

@@ -63,13 +63,20 @@ func init() {
 			Required:   true,
 			IsPassword: true,
 		}, {
-			Name:     "debug",
-			Help:     "Output more debug from Mega.",
+			Name: "debug",
+			Help: `Output more debug from Mega.
+
+If this flag is set (along with -vv) it will print further debugging
+information from the mega backend.`,
 			Default:  false,
 			Advanced: true,
 		}, {
-			Name:     "hard_delete",
-			Help:     "Delete files permanently rather than putting them into the trash.",
+			Name: "hard_delete",
+			Help: `Delete files permanently rather than putting them into the trash.
+
+Normally the mega backend will put all deletions into the trash rather
+than permanently deleting them.  If you specify this then rclone will
+permanently delete objects instead.`,
 			Default:  false,
 			Advanced: true,
 		}},

backend/onedrive/api/types.go

@@ -9,6 +9,9 @@ import (
 
 const (
 	timeFormat = `"` + time.RFC3339 + `"`
+
+	// PackageTypeOneNote is the package type value for OneNote files
+	PackageTypeOneNote = "oneNote"
 )
 
 // Error is returned from one drive when things go wrong
@@ -91,9 +94,10 @@ func (t *Timestamp) UnmarshalJSON(data []byte) error {
 // ItemReference groups data needed to reference a OneDrive item
 // across the service into a single structure.
 type ItemReference struct {
-	DriveID string `json:"driveId"` // Unique identifier for the Drive that contains the item.	Read-only.
-	ID      string `json:"id"`      // Unique identifier for the item.	Read/Write.
-	Path    string `json:"path"`    // Path that used to navigate to the item.	Read/Write.
+	DriveID   string `json:"driveId"`   // Unique identifier for the Drive that contains the item.	Read-only.
+	ID        string `json:"id"`        // Unique identifier for the item.	Read/Write.
+	Path      string `json:"path"`      // Path that used to navigate to the item.	Read/Write.
+	DriveType string `json:"driveType"` // Type of the drive,	Read-Only
 }
 
 // RemoteItemFacet groups data needed to reference a OneDrive remote item
@@ -106,6 +110,7 @@ type RemoteItemFacet struct {
 	LastModifiedDateTime Timestamp            `json:"lastModifiedDateTime"` // Date and time the item was last modified. Read-only.
 	Folder               *FolderFacet         `json:"folder"`               // Folder metadata, if the item is a folder. Read-only.
 	File                 *FileFacet           `json:"file"`                 // File metadata, if the item is a file. Read-only.
+	Package              *PackageFacet        `json:"package"`              // If present, indicates that this item is a package instead of a folder or file. Packages are treated like files in some contexts and folders in others. Read-only.
 	FileSystemInfo       *FileSystemInfoFacet `json:"fileSystemInfo"`       // File system information on client. Read-write.
 	ParentReference      *ItemReference       `json:"parentReference"`      // Parent information, if the item has a parent. Read-write.
 	Size                 int64                `json:"size"`                 // Size of the item in bytes. Read-only.
@@ -146,6 +151,13 @@ type FileSystemInfoFacet struct {
 type DeletedFacet struct {
 }
 
+// PackageFacet indicates that a DriveItem is the top level item
+// in a "package" or a collection of items that should be treated as a collection instead of individual items.
+// `oneNote` is the only currently defined value.
+type PackageFacet struct {
+	Type string `json:"type"`
+}
+
 // Item represents metadata for an item in OneDrive
 type Item struct {
 	ID                   string               `json:"id"`                   // The unique identifier of the item within the Drive. Read-only.
@@ -169,6 +181,7 @@ type Item struct {
 	//	Audio                *AudioFacet          `json:"audio"`                // Audio metadata, if the item is an audio file. Read-only.
 	//	Video                *VideoFacet          `json:"video"`                // Video metadata, if the item is a video. Read-only.
 	//	Location             *LocationFacet       `json:"location"`             // Location metadata, if the item has location data. Read-only.
+	Package *PackageFacet `json:"package"` // If present, indicates that this item is a package instead of a folder or file. Packages are treated like files in some contexts and folders in others. Read-only.
 	Deleted *DeletedFacet `json:"deleted"` // Information about the deleted state of the item. Read-only.
 }
 
@@ -237,6 +250,28 @@ type MoveItemRequest struct {
 	FileSystemInfo  *FileSystemInfoFacet `json:"fileSystemInfo,omitempty"`  // File system information on client. Read-write.
 }
 
+//CreateShareLinkRequest is the request to create a sharing link
+//Always Type:view and Scope:anonymous for public sharing
+type CreateShareLinkRequest struct {
+	Type  string `json:"type"`            //Link type in View, Edit or Embed
+	Scope string `json:"scope,omitempty"` //Optional. Scope in anonymousi, organization
+}
+
+//CreateShareLinkResponse is the response from CreateShareLinkRequest
+type CreateShareLinkResponse struct {
+	ID    string   `json:"id"`
+	Roles []string `json:"roles"`
+	Link  struct {
+		Type        string `json:"type"`
+		Scope       string `json:"scope"`
+		WebURL      string `json:"webUrl"`
+		Application struct {
+			ID          string `json:"id"`
+			DisplayName string `json:"displayName"`
+		} `json:"application"`
+	} `json:"link"`
+}
+
 // AsyncOperationStatus provides information on the status of a asynchronous job progress.
 //
 // The following API calls return AsyncOperationStatus resources:
@@ -244,7 +279,6 @@ type MoveItemRequest struct {
 // Copy Item
 // Upload From URL
 type AsyncOperationStatus struct {
-	Operation          string  `json:"operation"`          // The type of job being run.
 	PercentageComplete float64 `json:"percentageComplete"` // An float value between 0 and 100 that indicates the percentage complete.
 	Status             string  `json:"status"`             // A string value that maps to an enumeration of possible values about the status of the job. "notStarted | inProgress | completed | updating | failed | deletePending | deleteFailed | waiting"
 }
@@ -281,6 +315,24 @@ func (i *Item) GetFolder() *FolderFacet {
 	return i.Folder
 }
 
+// GetPackage returns a normalized Package of the item
+func (i *Item) GetPackage() *PackageFacet {
+	if i.IsRemote() && i.RemoteItem.Package != nil {
+		return i.RemoteItem.Package
+	}
+	return i.Package
+}
+
+// GetPackageType returns the package type of the item if available,
+// otherwise ""
+func (i *Item) GetPackageType() string {
+	pack := i.GetPackage()
+	if pack == nil {
+		return ""
+	}
+	return pack.Type
+}
+
 // GetFile returns a normalized File of the item
 func (i *Item) GetFile() *FileFacet {
 	if i.IsRemote() && i.RemoteItem.File != nil {

backend/onedrive/onedrive.go

@@ -10,7 +10,6 @@ import (
 	"io"
 	"log"
 	"net/http"
-	"net/url"
 	"path"
 	"strings"
 	"time"
@@ -33,48 +32,34 @@ import (
 )
 
 const (
-	rclonePersonalClientID              = "0000000044165769"
-	rclonePersonalEncryptedClientSecret = "ugVWLNhKkVT1-cbTRO-6z1MlzwdW6aMwpKgNaFG-qXjEn_WfDnG9TVyRA5yuoliU"
-	rcloneBusinessClientID              = "52857fec-4bc2-483f-9f1b-5fe28e97532c"
-	rcloneBusinessEncryptedClientSecret = "6t4pC8l6L66SFYVIi8PgECDyjXy_ABo1nsTaE-Lr9LpzC6yT4vNOwHsakwwdEui0O6B0kX8_xbBLj91J"
-	minSleep                            = 10 * time.Millisecond
-	maxSleep                            = 2 * time.Second
-	decayConstant                       = 2                                     // bigger for slower decay, exponential
-	rootURLPersonal                     = "https://api.onedrive.com/v1.0/drive" // root URL for requests
-	discoveryServiceURL                 = "https://api.office.com/discovery/"
-	configResourceURL                   = "resource_url"
+	rcloneClientID              = "b15665d9-eda6-4092-8539-0eec376afd59"
+	rcloneEncryptedClientSecret = "_JUdzh3LnKNqSPcf4Wu5fgMFIQOI8glZu_akYgR8yf6egowNBg-R"
+	minSleep                    = 10 * time.Millisecond
+	maxSleep                    = 2 * time.Second
+	decayConstant               = 2 // bigger for slower decay, exponential
+	graphURL                    = "https://graph.microsoft.com/v1.0"
+	configDriveID               = "drive_id"
+	configDriveType             = "drive_type"
+	driveTypePersonal           = "personal"
+	driveTypeBusiness           = "business"
+	driveTypeSharepoint         = "documentLibrary"
+	defaultChunkSize            = 10 * fs.MebiByte
+	chunkSizeMultiple           = 320 * fs.KibiByte
 )
 
 // Globals
 var (
-	// Description of how to auth for this app for a personal account
-	oauthPersonalConfig = &oauth2.Config{
-		Scopes: []string{
-			"wl.signin",          // Allow single sign-on capabilities
-			"wl.offline_access",  // Allow receiving a refresh token
-			"onedrive.readwrite", // r/w perms to all of a user's OneDrive files
-		},
-		Endpoint: oauth2.Endpoint{
-			AuthURL:  "https://login.live.com/oauth20_authorize.srf",
-			TokenURL: "https://login.live.com/oauth20_token.srf",
-		},
-		ClientID:     rclonePersonalClientID,
-		ClientSecret: obscure.MustReveal(rclonePersonalEncryptedClientSecret),
-		RedirectURL:  oauthutil.RedirectLocalhostURL,
-	}
-
 	// Description of how to auth for this app for a business account
-	oauthBusinessConfig = &oauth2.Config{
+	oauthConfig = &oauth2.Config{
 		Endpoint: oauth2.Endpoint{
-			AuthURL:  "https://login.microsoftonline.com/common/oauth2/authorize",
-			TokenURL: "https://login.microsoftonline.com/common/oauth2/token",
+			AuthURL:  "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
+			TokenURL: "https://login.microsoftonline.com/common/oauth2/v2.0/token",
 		},
-		ClientID:     rcloneBusinessClientID,
-		ClientSecret: obscure.MustReveal(rcloneBusinessEncryptedClientSecret),
+		Scopes:       []string{"Files.Read", "Files.ReadWrite", "Files.Read.All", "Files.ReadWrite.All", "offline_access"},
+		ClientID:     rcloneClientID,
+		ClientSecret: obscure.MustReveal(rcloneEncryptedClientSecret),
 		RedirectURL:  oauthutil.RedirectLocalhostURL,
 	}
-	oauthBusinessResource = oauth2.SetAuthURLParam("resource", discoveryServiceURL)
-	sharedURL             = "https://api.onedrive.com/v1.0/drives" // root URL for remote shared resources
 )
 
 // Register with Fs
@@ -84,147 +69,143 @@ func init() {
 		Description: "Microsoft OneDrive",
 		NewFs:       NewFs,
 		Config: func(name string, m configmap.Mapper) {
-			// choose account type
-			fmt.Printf("Choose OneDrive account type?\n")
-			fmt.Printf(" * Say b for a OneDrive business account\n")
-			fmt.Printf(" * Say p for a personal OneDrive account\n")
-			isPersonal := config.Command([]string{"bBusiness", "pPersonal"}) == 'p'
+			err := oauthutil.Config("onedrive", name, m, oauthConfig)
+			if err != nil {
+				log.Fatalf("Failed to configure token: %v", err)
+				return
+			}
 
-			if isPersonal {
-				// for personal accounts we don't safe a field about the account
-				err := oauthutil.Config("onedrive", name, m, oauthPersonalConfig)
-				if err != nil {
-					log.Fatalf("Failed to configure token: %v", err)
-				}
-			} else {
-				err := oauthutil.ConfigErrorCheck("onedrive", name, m, func(req *http.Request) oauthutil.AuthError {
-					var resp oauthutil.AuthError
+			// Are we running headless?
+			if automatic, _ := m.Get(config.ConfigAutomatic); automatic != "" {
+				// Yes, okay we are done
+				return
+			}
 
-					resp.Name = req.URL.Query().Get("error")
-					resp.Code = strings.Split(req.URL.Query().Get("error_description"), ":")[0] // error_description begins with XXXXXXXXXXXX:
-					resp.Description = strings.Join(strings.Split(req.URL.Query().Get("error_description"), ":")[1:], ":")
-					resp.HelpURL = "https://rclone.org/onedrive/#troubleshooting"
-					return resp
-				}, oauthBusinessConfig, oauthBusinessResource)
-				if err != nil {
-					log.Fatalf("Failed to configure token: %v", err)
-					return
-				}
+			type driveResource struct {
+				DriveID   string `json:"id"`
+				DriveName string `json:"name"`
+				DriveType string `json:"driveType"`
+			}
+			type drivesResponse struct {
+				Drives []driveResource `json:"value"`
+			}
 
-				// Are we running headless?
-				if automatic, _ := m.Get(config.ConfigAutomatic); automatic != "" {
-					// Yes, okay we are done
-					return
-				}
+			type siteResource struct {
+				SiteID   string `json:"id"`
+				SiteName string `json:"displayName"`
+				SiteURL  string `json:"webUrl"`
+			}
+			type siteResponse struct {
+				Sites []siteResource `json:"value"`
+			}
 
-				type serviceResource struct {
-					ServiceAPIVersion  string `json:"serviceApiVersion"`
-					ServiceEndpointURI string `json:"serviceEndpointUri"`
-					ServiceResourceID  string `json:"serviceResourceId"`
-				}
-				type serviceResponse struct {
-					Services []serviceResource `json:"value"`
-				}
+			oAuthClient, _, err := oauthutil.NewClient(name, m, oauthConfig)
+			if err != nil {
+				log.Fatalf("Failed to configure OneDrive: %v", err)
+			}
+			srv := rest.NewClient(oAuthClient)
 
-				oAuthClient, _, err := oauthutil.NewClient(name, m, oauthBusinessConfig)
-				if err != nil {
-					log.Fatalf("Failed to configure OneDrive: %v", err)
-					return
-				}
-				srv := rest.NewClient(oAuthClient)
+			var opts rest.Opts
+			var finalDriveID string
+			var siteID string
+			switch config.Choose("Your choice",
+				[]string{"onedrive", "sharepoint", "driveid", "siteid", "search"},
+				[]string{"OneDrive Personal or Business", "Root Sharepoint site", "Type in driveID", "Type in SiteID", "Search a Sharepoint site"},
+				false) {
 
-				opts := rest.Opts{
-					Method:  "GET",
-					RootURL: discoveryServiceURL,
-					Path:    "/v2.0/me/services",
-				}
-				services := serviceResponse{}
-				resp, err := srv.CallJSON(&opts, nil, &services)
-				if err != nil {
-					fs.Errorf(nil, "Failed to query available services: %v", err)
-					return
-				}
-				if resp.StatusCode != 200 {
-					fs.Errorf(nil, "Failed to query available services: Got HTTP error code %d", resp.StatusCode)
-					return
-				}
-
-				var resourcesURL []string
-				var resourcesID []string
-
-				for _, service := range services.Services {
-					if service.ServiceAPIVersion == "v2.0" {
-						resourcesID = append(resourcesID, service.ServiceResourceID)
-						resourcesURL = append(resourcesURL, service.ServiceEndpointURI)
-					}
-					// we only support 2.0 API
-					fs.Infof(nil, "Skipping API %s endpoint %s", service.ServiceAPIVersion, service.ServiceEndpointURI)
-				}
-
-				var foundService string
-				if len(resourcesID) == 0 {
-					fs.Errorf(nil, "No Service found")
-					return
-				} else if len(resourcesID) == 1 {
-					foundService = resourcesID[0]
-				} else {
-					foundService = config.Choose("Choose resource URL", resourcesID, resourcesURL, false)
-				}
-
-				m.Set(configResourceURL, foundService)
-				oauthBusinessResource = oauth2.SetAuthURLParam("resource", foundService)
-
-				// get the token from the inital config
-				// we need to update the token with a resource
-				// specific token we will query now
-				token, err := oauthutil.GetToken(name, m)
-				if err != nil {
-					fs.Errorf(nil, "Error while getting token: %s", err)
-					return
-				}
-
-				// values for the token query
-				values := url.Values{}
-				values.Set("refresh_token", token.RefreshToken)
-				values.Set("grant_type", "refresh_token")
-				values.Set("resource", foundService)
-				values.Set("client_id", oauthBusinessConfig.ClientID)
-				values.Set("client_secret", oauthBusinessConfig.ClientSecret)
+			case "onedrive":
 				opts = rest.Opts{
-					Method:      "POST",
-					RootURL:     oauthBusinessConfig.Endpoint.TokenURL,
-					ContentType: "application/x-www-form-urlencoded",
-					Body:        strings.NewReader(values.Encode()),
+					Method:  "GET",
+					RootURL: graphURL,
+					Path:    "/me/drives",
+				}
+			case "sharepoint":
+				opts = rest.Opts{
+					Method:  "GET",
+					RootURL: graphURL,
+					Path:    "/sites/root/drives",
+				}
+			case "driveid":
+				fmt.Printf("Paste your Drive ID here> ")
+				finalDriveID = config.ReadLine()
+			case "siteid":
+				fmt.Printf("Paste your Site ID here> ")
+				siteID = config.ReadLine()
+			case "search":
+				fmt.Printf("What to search for> ")
+				searchTerm := config.ReadLine()
+				opts = rest.Opts{
+					Method:  "GET",
+					RootURL: graphURL,
+					Path:    "/sites?search=" + searchTerm,
 				}
 
-				// tokenJSON is the struct representing the HTTP response from OAuth2
-				// providers returning a token in JSON form.
-				// we are only interested in the new tokens, all other fields we don't care
-				type tokenJSON struct {
-					AccessToken  string `json:"access_token"`
-					RefreshToken string `json:"refresh_token"`
-				}
-				jsonToken := tokenJSON{}
-				resp, err = srv.CallJSON(&opts, nil, &jsonToken)
+				sites := siteResponse{}
+				_, err := srv.CallJSON(&opts, nil, &sites)
 				if err != nil {
-					fs.Errorf(nil, "Failed to get resource token: %v", err)
-					return
-				}
-				if resp.StatusCode != 200 {
-					fs.Errorf(nil, "Failed to get resource token: Got HTTP error code %d", resp.StatusCode)
-					return
+					log.Fatalf("Failed to query available sites: %v", err)
 				}
 
-				// update the tokens
-				token.AccessToken = jsonToken.AccessToken
-				token.RefreshToken = jsonToken.RefreshToken
-
-				// finally save them in the config
-				err = oauthutil.PutToken(name, m, token, true)
-				if err != nil {
-					fs.Errorf(nil, "Error while setting token: %s", err)
+				if len(sites.Sites) == 0 {
+					log.Fatalf("Search for '%s' returned no results", searchTerm)
+				} else {
+					fmt.Printf("Found %d sites, please select the one you want to use:\n", len(sites.Sites))
+					for index, site := range sites.Sites {
+						fmt.Printf("%d: %s (%s) id=%s\n", index, site.SiteName, site.SiteURL, site.SiteID)
+					}
+					siteID = sites.Sites[config.ChooseNumber("Chose drive to use:", 0, len(sites.Sites)-1)].SiteID
 				}
 			}
+
+			// if we have a siteID we need to ask for the drives
+			if siteID != "" {
+				opts = rest.Opts{
+					Method:  "GET",
+					RootURL: graphURL,
+					Path:    "/sites/" + siteID + "/drives",
+				}
+			}
+
+			// We don't have the final ID yet?
+			// query Microsoft Graph
+			if finalDriveID == "" {
+				drives := drivesResponse{}
+				_, err := srv.CallJSON(&opts, nil, &drives)
+				if err != nil {
+					log.Fatalf("Failed to query available drives: %v", err)
+				}
+
+				if len(drives.Drives) == 0 {
+					log.Fatalf("No drives found")
+				} else {
+					fmt.Printf("Found %d drives, please select the one you want to use:\n", len(drives.Drives))
+					for index, drive := range drives.Drives {
+						fmt.Printf("%d: %s (%s) id=%s\n", index, drive.DriveName, drive.DriveType, drive.DriveID)
+					}
+					finalDriveID = drives.Drives[config.ChooseNumber("Chose drive to use:", 0, len(drives.Drives)-1)].DriveID
+				}
+			}
+
+			// Test the driveID and get drive type
+			opts = rest.Opts{
+				Method:  "GET",
+				RootURL: graphURL,
+				Path:    "/drives/" + finalDriveID + "/root"}
+			var rootItem api.Item
+			_, err = srv.CallJSON(&opts, nil, &rootItem)
+			if err != nil {
+				log.Fatalf("Failed to query root for drive %s: %v", finalDriveID, err)
+			}
+
+			fmt.Printf("Found drive '%s' of type '%s', URL: %s\nIs that okay?\n", rootItem.Name, rootItem.ParentReference.DriveType, rootItem.WebURL)
+			// This does not work, YET :)
+			if !config.Confirm() {
+				log.Fatalf("Cancelled by user")
+			}
+
+			m.Set(configDriveID, finalDriveID)
+			m.Set(configDriveType, rootItem.ParentReference.DriveType)
+			config.SaveConfig()
 		},
 		Options: []fs.Option{{
 			Name: config.ConfigClientID,
@@ -233,9 +214,33 @@ func init() {
 			Name: config.ConfigClientSecret,
 			Help: "Microsoft App Client Secret\nLeave blank normally.",
 		}, {
-			Name:     "chunk_size",
-			Help:     "Chunk size to upload files with - must be multiple of 320k.",
-			Default:  fs.SizeSuffix(10 * 1024 * 1024),
+			Name: "chunk_size",
+			Help: `Chunk size to upload files with - must be multiple of 320k.
+
+Above this size files will be chunked - must be multiple of 320k. Note
+that the chunks will be buffered into memory.`,
+			Default:  defaultChunkSize,
+			Advanced: true,
+		}, {
+			Name:     "drive_id",
+			Help:     "The ID of the drive to use",
+			Default:  "",
+			Advanced: true,
+		}, {
+			Name:     "drive_type",
+			Help:     "The type of the drive ( personal | business | documentLibrary )",
+			Default:  "",
+			Advanced: true,
+		}, {
+			Name: "expose_onenote_files",
+			Help: `Set to make OneNote files show up in directory listings.
+
+By default rclone will hide OneNote files in directory listings because
+operations like "Open" and "Update" won't work on them.  But this
+behaviour may also prevent you from deleting them.  If you want to
+delete OneNote files or otherwise want them to show up in directory
+listing, set this option.`,
+			Default:  false,
 			Advanced: true,
 		}},
 	})
@@ -243,8 +248,10 @@ func init() {
 
 // Options defines the configuration for this backend
 type Options struct {
-	ChunkSize   fs.SizeSuffix `config:"chunk_size"`
-	ResourceURL string        `config:"resource_url"`
+	ChunkSize          fs.SizeSuffix `config:"chunk_size"`
+	DriveID            string        `config:"drive_id"`
+	DriveType          string        `config:"drive_type"`
+	ExposeOneNoteFiles bool          `config:"expose_onenote_files"`
 }
 
 // Fs represents a remote one drive
@@ -257,22 +264,24 @@ type Fs struct {
 	dirCache     *dircache.DirCache // Map of directory path to directory id
 	pacer        *pacer.Pacer       // pacer for API calls
 	tokenRenewer *oauthutil.Renew   // renew the token on expiry
-	isBusiness   bool               // true if this is an OneDrive Business account
+	driveID      string             // ID to use for querying Microsoft Graph
+	driveType    string             // https://developer.microsoft.com/en-us/graph/docs/api-reference/v1.0/resources/drive
 }
 
 // Object describes a one drive object
 //
 // Will definitely have info but maybe not meta
 type Object struct {
-	fs           *Fs       // what this object is part of
-	remote       string    // The remote path
-	hasMetaData  bool      // whether info below has been set
-	size         int64     // size of the object
-	modTime      time.Time // modification time of the object
-	id           string    // ID of the object
-	sha1         string    // SHA-1 of the object content
-	quickxorhash string    // QuickXorHash of the object content
-	mimeType     string    // Content-Type of object from server (may not be as uploaded)
+	fs            *Fs       // what this object is part of
+	remote        string    // The remote path
+	hasMetaData   bool      // whether info below has been set
+	isOneNoteFile bool      // Whether the object is a OneNote file
+	size          int64     // size of the object
+	modTime       time.Time // modification time of the object
+	id            string    // ID of the object
+	sha1          string    // SHA-1 of the object content
+	quickxorhash  string    // QuickXorHash of the object content
+	mimeType      string    // Content-Type of object from server (may not be as uploaded)
 }
 
 // ------------------------------------------------------------
@@ -327,9 +336,17 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
 
 // readMetaDataForPath reads the metadata from the path
 func (f *Fs) readMetaDataForPath(path string) (info *api.Item, resp *http.Response, err error) {
-	opts := rest.Opts{
-		Method: "GET",
-		Path:   "/root:/" + rest.URLPathEscape(replaceReservedChars(path)),
+	var opts rest.Opts
+	if len(path) == 0 {
+		opts = rest.Opts{
+			Method: "GET",
+			Path:   "/root",
+		}
+	} else {
+		opts = rest.Opts{
+			Method: "GET",
+			Path:   "/root:/" + rest.URLPathEscape(replaceReservedChars(path)),
+		}
 	}
 	err = f.pacer.Call(func() (bool, error) {
 		resp, err = f.srv.CallJSON(&opts, nil, &info)
@@ -353,6 +370,25 @@ func errorHandler(resp *http.Response) error {
 	return errResponse
 }
 
+func checkUploadChunkSize(cs fs.SizeSuffix) error {
+	const minChunkSize = fs.Byte
+	if cs%chunkSizeMultiple != 0 {
+		return errors.Errorf("%s is not a multiple of %s", cs, chunkSizeMultiple)
+	}
+	if cs < minChunkSize {
+		return errors.Errorf("%s is less than %s", cs, minChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadChunkSize(cs)
+	if err == nil {
+		old, f.opt.ChunkSize = f.opt.ChunkSize, cs
+	}
+	return
+}
+
 // NewFs constructs an Fs from the path, container:path
 func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	// Parse config into Options struct
@@ -361,26 +397,16 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		return nil, err
 	}
-	if opt.ChunkSize%(320*1024) != 0 {
-		return nil, errors.Errorf("chunk size %d is not a multiple of 320k", opt.ChunkSize)
-	}
-	// if we have a resource URL it's a business account otherwise a personal one
-	isBusiness := opt.ResourceURL != ""
-	var rootURL string
-	var oauthConfig *oauth2.Config
-	if !isBusiness {
-		// personal account setup
-		oauthConfig = oauthPersonalConfig
-		rootURL = rootURLPersonal
-	} else {
-		// business account setup
-		oauthConfig = oauthBusinessConfig
-		rootURL = opt.ResourceURL + "_api/v2.0/drives/me"
-		sharedURL = opt.ResourceURL + "_api/v2.0/drives"
 
-		// update the URL in the AuthOptions
-		oauthBusinessResource = oauth2.SetAuthURLParam("resource", opt.ResourceURL)
+	err = checkUploadChunkSize(opt.ChunkSize)
+	if err != nil {
+		return nil, errors.Wrap(err, "onedrive: chunk size")
 	}
+
+	if opt.DriveID == "" || opt.DriveType == "" {
+		log.Fatalf("Unable to get drive_id and drive_type. If you are upgrading from older versions of rclone, please run `rclone config` and re-configure this backend.")
+	}
+
 	root = parsePath(root)
 	oAuthClient, ts, err := oauthutil.NewClient(name, m, oauthConfig)
 	if err != nil {
@@ -388,19 +414,17 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	}
 
 	f := &Fs{
-		name:       name,
-		root:       root,
-		opt:        *opt,
-		srv:        rest.NewClient(oAuthClient).SetRoot(rootURL),
-		pacer:      pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
-		isBusiness: isBusiness,
+		name:      name,
+		root:      root,
+		opt:       *opt,
+		driveID:   opt.DriveID,
+		driveType: opt.DriveType,
+		srv:       rest.NewClient(oAuthClient).SetRoot(graphURL + "/drives/" + opt.DriveID),
+		pacer:     pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
 	}
 	f.features = (&fs.Features{
-		CaseInsensitive: true,
-		// OneDrive for business doesn't support mime types properly
-		// so we disable it until resolved
-		// https://github.com/OneDrive/onedrive-api-docs/issues/643
-		ReadMimeType:            !f.isBusiness,
+		CaseInsensitive:         true,
+		ReadMimeType:            true,
 		CanHaveEmptyDirectories: true,
 	}).Fill(f)
 	f.srv.SetErrorHandler(errorHandler)
@@ -424,16 +448,16 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		// Assume it is a file
 		newRoot, remote := dircache.SplitPath(root)
-		newF := *f
-		newF.dirCache = dircache.New(newRoot, rootInfo.ID, &newF)
-		newF.root = newRoot
+		tempF := *f
+		tempF.dirCache = dircache.New(newRoot, rootInfo.ID, &tempF)
+		tempF.root = newRoot
 		// Make new Fs which is the parent
-		err = newF.dirCache.FindRoot(false)
+		err = tempF.dirCache.FindRoot(false)
 		if err != nil {
 			// No root so return old f
 			return f, nil
 		}
-		_, err := newF.newObjectWithInfo(remote, nil)
+		_, err := tempF.newObjectWithInfo(remote, nil)
 		if err != nil {
 			if err == fs.ErrorObjectNotFound {
 				// File doesn't exist so return old f
@@ -441,8 +465,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 			}
 			return nil, err
 		}
+		// XXX: update the old f here instead of returning tempF, since
+		// `features` were already filled with functions having *f as a receiver.
+		// See https://github.com/ncw/rclone/issues/2182
+		f.dirCache = tempF.dirCache
+		f.root = tempF.root
 		// return an error with an fs which points to the parent
-		return &newF, fs.ErrorIsFile
+		return f, fs.ErrorIsFile
 	}
 	return f, nil
 }
@@ -503,6 +532,9 @@ func (f *Fs) FindLeaf(pathID, leaf string) (pathIDOut string, found bool, err er
 		}
 		return "", false, err
 	}
+	if info.GetPackageType() == api.PackageTypeOneNote {
+		return "", false, errors.New("found OneNote file when looking for folder")
+	}
 	if info.GetFolder() == nil {
 		return "", false, errors.New("found file when looking for folder")
 	}
@@ -546,8 +578,7 @@ type listAllFn func(*api.Item) bool
 func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
 	// Top parameter asks for bigger pages of data
 	// https://dev.onedrive.com/odata/optional-query-parameters.htm
-	opts := newOptsCall(dirID, "GET", "/children?top=1000")
-
+	opts := newOptsCall(dirID, "GET", "/children?$top=1000")
 OUTER:
 	for {
 		var result api.ListChildrenResponse
@@ -612,6 +643,11 @@ func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
 	}
 	var iErr error
 	_, err = f.listAll(directoryID, false, false, func(info *api.Item) bool {
+		if !f.opt.ExposeOneNoteFiles && info.GetPackageType() == api.PackageTypeOneNote {
+			fs.Debugf(info.Name, "OneNote file not shown in directory listing")
+			return false
+		}
+
 		remote := path.Join(dir, info.GetName())
 		folder := info.GetFolder()
 		if folder != nil {
@@ -718,15 +754,17 @@ func (f *Fs) purgeCheck(dir string, check bool) error {
 	if err != nil {
 		return err
 	}
-	item, _, err := f.readMetaDataForPath(root)
-	if err != nil {
-		return err
-	}
-	if item.Folder == nil {
-		return errors.New("not a folder")
-	}
-	if check && item.Folder.ChildCount != 0 {
-		return errors.New("folder not empty")
+	if check {
+		// check to see if there are any items
+		found, err := f.listAll(rootID, false, false, func(item *api.Item) bool {
+			return true
+		})
+		if err != nil {
+			return err
+		}
+		if found {
+			return fs.ErrorDirectoryNotEmpty
+		}
 	}
 	err = f.deleteObject(rootID)
 	if err != nil {
@@ -755,16 +793,11 @@ func (f *Fs) Precision() time.Duration {
 func (f *Fs) waitForJob(location string, o *Object) error {
 	deadline := time.Now().Add(fs.Config.Timeout)
 	for time.Now().Before(deadline) {
-		opts := rest.Opts{
-			Method:       "GET",
-			RootURL:      location,
-			IgnoreStatus: true, // Ignore the http status response since it seems to return valid info on 500 errors
-		}
 		var resp *http.Response
 		var err error
 		var body []byte
 		err = f.pacer.Call(func() (bool, error) {
-			resp, err = f.srv.Call(&opts)
+			resp, err = http.Get(location)
 			if err != nil {
 				return fserrors.ShouldRetry(err), err
 			}
@@ -780,19 +813,18 @@ func (f *Fs) waitForJob(location string, o *Object) error {
 		if err != nil {
 			return errors.Wrapf(err, "async status result not JSON: %q", body)
 		}
-		// See if we decoded anything...
-		if !(status.Operation == "" && status.PercentageComplete == 0 && status.Status == "") {
-			if status.Status == "failed" || status.Status == "deleteFailed" {
-				return errors.Errorf("%s: async operation %q returned %q", o.remote, status.Operation, status.Status)
+
+		switch status.Status {
+		case "failed":
+		case "deleteFailed":
+			{
+				return errors.Errorf("%s: async operation returned %q", o.remote, status.Status)
 			}
-		} else if resp.StatusCode == 200 {
-			var info api.Item
-			err = json.Unmarshal(body, &info)
-			if err != nil {
-				return errors.Wrapf(err, "async item result not JSON: %q", body)
-			}
-			return o.setMetaData(&info)
+		case "completed":
+			err = o.readMetaData()
+			return errors.Wrapf(err, "async operation completed but readMetaData failed")
 		}
+
 		time.Sleep(1 * time.Second)
 	}
 	return errors.Errorf("async operation didn't complete after %v", fs.Config.Timeout)
@@ -831,7 +863,7 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
 	}
 
 	// Copy the object
-	opts := newOptsCall(srcObj.id, "POST", "/action.copy")
+	opts := newOptsCall(srcObj.id, "POST", "/copy")
 	opts.ExtraHeaders = map[string]string{"Prefer": "respond-async"}
 	opts.NoResponse = true
 
@@ -841,7 +873,8 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
 	copyReq := api.CopyItemRequest{
 		Name: &replacedLeaf,
 		ParentReference: api.ItemReference{
-			ID: id,
+			DriveID: f.driveID,
+			ID:      id,
 		},
 	}
 	var resp *http.Response
@@ -1077,10 +1110,36 @@ func (f *Fs) About() (usage *fs.Usage, err error) {
 
 // Hashes returns the supported hash sets.
 func (f *Fs) Hashes() hash.Set {
-	if f.isBusiness {
-		return hash.Set(hash.QuickXorHash)
+	if f.driveType == driveTypePersonal {
+		return hash.Set(hash.SHA1)
 	}
-	return hash.Set(hash.SHA1)
+	return hash.Set(hash.QuickXorHash)
+}
+
+// PublicLink returns a link for downloading without accout.
+func (f *Fs) PublicLink(remote string) (link string, err error) {
+	info, _, err := f.readMetaDataForPath(f.srvPath(remote))
+	if err != nil {
+		return "", err
+	}
+	opts := newOptsCall(info.ID, "POST", "/createLink")
+
+	share := api.CreateShareLinkRequest{
+		Type:  "view",
+		Scope: "anonymous",
+	}
+
+	var resp *http.Response
+	var result api.CreateShareLinkResponse
+	err = f.pacer.Call(func() (bool, error) {
+		resp, err = f.srv.CallJSON(&opts, &share, &result)
+		return shouldRetry(resp, err)
+	})
+	if err != nil {
+		fmt.Println(err)
+		return "", err
+	}
+	return result.Link.WebURL, nil
 }
 
 // ------------------------------------------------------------
@@ -1103,23 +1162,28 @@ func (o *Object) Remote() string {
 	return o.remote
 }
 
+// srvPath returns a path for use in server given a remote
+func (f *Fs) srvPath(remote string) string {
+	return replaceReservedChars(f.rootSlash() + remote)
+}
+
 // srvPath returns a path for use in server
 func (o *Object) srvPath() string {
-	return replaceReservedChars(o.fs.rootSlash() + o.remote)
+	return o.fs.srvPath(o.remote)
 }
 
 // Hash returns the SHA-1 of an object returning a lowercase hex string
 func (o *Object) Hash(t hash.Type) (string, error) {
-	if o.fs.isBusiness {
-		if t != hash.QuickXorHash {
-			return "", hash.ErrUnsupported
+	if o.fs.driveType == driveTypePersonal {
+		if t == hash.SHA1 {
+			return o.sha1, nil
+		}
+	} else {
+		if t == hash.QuickXorHash {
+			return o.quickxorhash, nil
 		}
-		return o.quickxorhash, nil
 	}
-	if t != hash.SHA1 {
-		return "", hash.ErrUnsupported
-	}
-	return o.sha1, nil
+	return "", hash.ErrUnsupported
 }
 
 // Size returns the size of an object in bytes
@@ -1140,6 +1204,8 @@ func (o *Object) setMetaData(info *api.Item) (err error) {
 	o.hasMetaData = true
 	o.size = info.GetSize()
 
+	o.isOneNoteFile = info.GetPackageType() == api.PackageTypeOneNote
+
 	// Docs: https://docs.microsoft.com/en-us/onedrive/developer/rest-api/resources/hashes
 	//
 	// We use SHA1 for onedrive personal and QuickXorHash for onedrive for business
@@ -1251,6 +1317,10 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
 	if o.id == "" {
 		return nil, errors.New("can't download - no id")
 	}
+	if o.isOneNoteFile {
+		return nil, errors.New("can't open a OneNote file")
+	}
+
 	fs.FixRangeOption(options, o.size)
 	var resp *http.Response
 	opts := newOptsCall(o.id, "GET", "/content")
@@ -1280,12 +1350,12 @@ func (o *Object) createUploadSession(modTime time.Time) (response *api.CreateUpl
 		opts = rest.Opts{
 			Method:  "POST",
 			RootURL: rootURL,
-			Path:    "/" + drive + "/items/" + id + ":/" + rest.URLPathEscape(leaf) + ":/upload.createSession",
+			Path:    "/" + drive + "/items/" + id + ":/" + rest.URLPathEscape(replaceReservedChars(leaf)) + ":/createUploadSession",
 		}
 	} else {
 		opts = rest.Opts{
 			Method: "POST",
-			Path:   "/root:/" + rest.URLPathEscape(o.srvPath()) + ":/upload.createSession",
+			Path:   "/root:/" + rest.URLPathEscape(o.srvPath()) + ":/createUploadSession",
 		}
 	}
 	createRequest := api.CreateUploadRequest{}
@@ -1294,6 +1364,12 @@ func (o *Object) createUploadSession(modTime time.Time) (response *api.CreateUpl
 	var resp *http.Response
 	err = o.fs.pacer.Call(func() (bool, error) {
 		resp, err = o.fs.srv.CallJSON(&opts, &createRequest, &response)
+		if apiErr, ok := err.(*api.Error); ok {
+			if apiErr.ErrorInfo.Code == "nameAlreadyExists" {
+				// Make the error more user-friendly
+				err = errors.New(err.Error() + " (is it a OneNote file?)")
+			}
+		}
 		return shouldRetry(resp, err)
 	})
 	return response, err
@@ -1347,6 +1423,10 @@ func (o *Object) cancelUploadSession(url string) (err error) {
 
 // uploadMultipart uploads a file using multipart upload
 func (o *Object) uploadMultipart(in io.Reader, size int64, modTime time.Time) (info *api.Item, err error) {
+	if size <= 0 {
+		panic("size passed into uploadMultipart must be > 0")
+	}
+
 	// Create upload session
 	fs.Debugf(o, "Starting multipart upload")
 	session, err := o.createUploadSession(modTime)
@@ -1387,8 +1467,14 @@ func (o *Object) uploadMultipart(in io.Reader, size int64, modTime time.Time) (i
 	return info, nil
 }
 
-// uploadSinglepart uploads a file as a single part
+// Update the content of a remote file within 4MB size in one single request
+// This function will set modtime after uploading, which will create a new version for the remote file
 func (o *Object) uploadSinglepart(in io.Reader, size int64, modTime time.Time) (info *api.Item, err error) {
+	if size < 0 || size > int64(fs.SizeSuffix(4*1024*1024)) {
+		panic("size passed into uploadSinglepart must be >= 0 and <= 4MiB")
+	}
+
+	fs.Debugf(o, "Starting singlepart upload")
 	var resp *http.Response
 	var opts rest.Opts
 	_, directoryID, _ := o.fs.dirCache.FindPath(o.remote, false)
@@ -1409,13 +1495,19 @@ func (o *Object) uploadSinglepart(in io.Reader, size int64, modTime time.Time) (
 			Body:          in,
 		}
 	}
-	// for go1.8 (see release notes) we must nil the Body if we want a
-	// "Content-Length: 0" header which onedrive requires for all files.
+
 	if size == 0 {
 		opts.Body = nil
 	}
-	err = o.fs.pacer.CallNoRetry(func() (bool, error) {
+
+	err = o.fs.pacer.Call(func() (bool, error) {
 		resp, err = o.fs.srv.CallJSON(&opts, nil, &info)
+		if apiErr, ok := err.(*api.Error); ok {
+			if apiErr.ErrorInfo.Code == "nameAlreadyExists" {
+				// Make the error more user-friendly
+				err = errors.New(err.Error() + " (is it a OneNote file?)")
+			}
+		}
 		return shouldRetry(resp, err)
 	})
 	if err != nil {
@@ -1434,6 +1526,10 @@ func (o *Object) uploadSinglepart(in io.Reader, size int64, modTime time.Time) (
 //
 // The new object may have been created if an error is returned
 func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (err error) {
+	if o.hasMetaData && o.isOneNoteFile {
+		return errors.New("can't upload content to a OneNote file")
+	}
+
 	o.fs.tokenRenewer.Start()
 	defer o.fs.tokenRenewer.Stop()
 
@@ -1441,15 +1537,17 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	modTime := src.ModTime()
 
 	var info *api.Item
-	if size <= 0 {
-		// This is for 0 length files, or files with an unknown size
+	if size > 0 {
+		info, err = o.uploadMultipart(in, size, modTime)
+	} else if size == 0 {
 		info, err = o.uploadSinglepart(in, size, modTime)
 	} else {
-		info, err = o.uploadMultipart(in, size, modTime)
+		panic("src file size must be >= 0")
 	}
 	if err != nil {
 		return err
 	}
+
 	return o.setMetaData(info)
 }
 
@@ -1487,7 +1585,7 @@ func newOptsCall(id string, method string, route string) (opts rest.Opts) {
 func parseDirID(ID string) (string, string, string) {
 	if strings.Index(ID, "#") >= 0 {
 		s := strings.Split(ID, "#")
-		return s[1], s[0], sharedURL
+		return s[1], s[0], graphURL + "/drives"
 	}
 	return ID, "", ""
 }
@@ -1501,6 +1599,7 @@ var (
 	_ fs.DirMover        = (*Fs)(nil)
 	_ fs.DirCacheFlusher = (*Fs)(nil)
 	_ fs.Abouter         = (*Fs)(nil)
+	_ fs.PublicLinker    = (*Fs)(nil)
 	_ fs.Object          = (*Object)(nil)
 	_ fs.MimeTyper       = &Object{}
 	_ fs.IDer            = &Object{}

backend/onedrive/onedrive_test.go

@@ -1,10 +1,10 @@
 // Test OneDrive filesystem interface
-package onedrive_test
+package onedrive
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/onedrive"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
@@ -12,6 +12,15 @@ import (
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
 		RemoteName: "TestOneDrive:",
-		NilObject:  (*onedrive.Object)(nil),
+		NilObject:  (*Object)(nil),
+		ChunkedUpload: fstests.ChunkedUploadConfig{
+			CeilChunkSize: fstests.NextMultipleOf(chunkSizeMultiple),
+		},
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+var _ fstests.SetUploadChunkSizer = (*Fs)(nil)

backend/opendrive/opendrive.go

@@ -6,6 +6,7 @@ import (
 	"io"
 	"mime/multipart"
 	"net/http"
+	"net/url"
 	"path"
 	"strconv"
 	"strings"
@@ -20,6 +21,7 @@ import (
 	"github.com/ncw/rclone/fs/hash"
 	"github.com/ncw/rclone/lib/dircache"
 	"github.com/ncw/rclone/lib/pacer"
+	"github.com/ncw/rclone/lib/readers"
 	"github.com/ncw/rclone/lib/rest"
 	"github.com/pkg/errors"
 )
@@ -177,17 +179,17 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		// Assume it is a file
 		newRoot, remote := dircache.SplitPath(root)
-		newF := *f
-		newF.dirCache = dircache.New(newRoot, "0", &newF)
-		newF.root = newRoot
+		tempF := *f
+		tempF.dirCache = dircache.New(newRoot, "0", &tempF)
+		tempF.root = newRoot
 
 		// Make new Fs which is the parent
-		err = newF.dirCache.FindRoot(false)
+		err = tempF.dirCache.FindRoot(false)
 		if err != nil {
 			// No root so return old f
 			return f, nil
 		}
-		_, err := newF.newObjectWithInfo(remote, nil)
+		_, err := tempF.newObjectWithInfo(remote, nil)
 		if err != nil {
 			if err == fs.ErrorObjectNotFound {
 				// File doesn't exist so return old f
@@ -195,8 +197,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 			}
 			return nil, err
 		}
+		// XXX: update the old f here instead of returning tempF, since
+		// `features` were already filled with functions having *f as a receiver.
+		// See https://github.com/ncw/rclone/issues/2182
+		f.dirCache = tempF.dirCache
+		f.root = tempF.root
 		// return an error with an fs which points to the parent
-		return &newF, fs.ErrorIsFile
+		return f, fs.ErrorIsFile
 	}
 	return f, nil
 }
@@ -925,8 +932,9 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	// resp.Body.Close()
 	// fs.Debugf(nil, "PostOpen: %#v", openResponse)
 
-	// 1 MB chunks size
+	// 10 MB chunks size
 	chunkSize := int64(1024 * 1024 * 10)
+	buf := make([]byte, int(chunkSize))
 	chunkOffset := int64(0)
 	remainingBytes := size
 	chunkCounter := 0
@@ -939,14 +947,19 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 		remainingBytes -= currentChunkSize
 		fs.Debugf(o, "Uploading chunk %d, size=%d, remain=%d", chunkCounter, currentChunkSize, remainingBytes)
 
+		chunk := readers.NewRepeatableLimitReaderBuffer(in, buf, currentChunkSize)
 		err = o.fs.pacer.Call(func() (bool, error) {
+			// seek to the start in case this is a retry
+			if _, err = chunk.Seek(0, io.SeekStart); err != nil {
+				return false, err
+			}
 			var formBody bytes.Buffer
 			w := multipart.NewWriter(&formBody)
 			fw, err := w.CreateFormFile("file_data", o.remote)
 			if err != nil {
 				return false, err
 			}
-			if _, err = io.CopyN(fw, in, currentChunkSize); err != nil {
+			if _, err = io.Copy(fw, chunk); err != nil {
 				return false, err
 			}
 			// Add session_id
@@ -1077,7 +1090,7 @@ func (o *Object) readMetaData() (err error) {
 	err = o.fs.pacer.Call(func() (bool, error) {
 		opts := rest.Opts{
 			Method: "GET",
-			Path:   "/folder/itembyname.json/" + o.fs.session.SessionID + "/" + directoryID + "?name=" + rest.URLPathEscape(replaceReservedChars(leaf)),
+			Path:   "/folder/itembyname.json/" + o.fs.session.SessionID + "/" + directoryID + "?name=" + url.QueryEscape(replaceReservedChars(leaf)),
 		}
 		resp, err = o.fs.srv.CallJSON(&opts, nil, &folderList)
 		return o.fs.shouldRetry(resp, err)

backend/pcloud/pcloud.go

@@ -276,16 +276,16 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		// Assume it is a file
 		newRoot, remote := dircache.SplitPath(root)
-		newF := *f
-		newF.dirCache = dircache.New(newRoot, rootID, &newF)
-		newF.root = newRoot
+		tempF := *f
+		tempF.dirCache = dircache.New(newRoot, rootID, &tempF)
+		tempF.root = newRoot
 		// Make new Fs which is the parent
-		err = newF.dirCache.FindRoot(false)
+		err = tempF.dirCache.FindRoot(false)
 		if err != nil {
 			// No root so return old f
 			return f, nil
 		}
-		_, err := newF.newObjectWithInfo(remote, nil)
+		_, err := tempF.newObjectWithInfo(remote, nil)
 		if err != nil {
 			if err == fs.ErrorObjectNotFound {
 				// File doesn't exist so return old f
@@ -293,8 +293,13 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 			}
 			return nil, err
 		}
+		// XXX: update the old f here instead of returning tempF, since
+		// `features` were already filled with functions having *f as a receiver.
+		// See https://github.com/ncw/rclone/issues/2182
+		f.dirCache = tempF.dirCache
+		f.root = tempF.root
 		// return an error with an fs which points to the parent
-		return &newF, fs.ErrorIsFile
+		return f, fs.ErrorIsFile
 	}
 	return f, nil
 }
@@ -1107,6 +1112,12 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 		return shouldRetry(resp, err)
 	})
 	if err != nil {
+		// sometimes pcloud leaves a half complete file on
+		// error, so delete it if it exists
+		delObj, delErr := o.fs.NewObject(o.remote)
+		if delErr == nil && delObj != nil {
+			_ = delObj.Remove()
+		}
 		return err
 	}
 	if len(result.Items) != 1 {

backend/qingstor/qingstor.go

@@ -132,10 +132,12 @@ type Object struct {
 
 // ------------------------------------------------------------
 
+// Pattern to match a qingstor path
+var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
+
 // parseParse parses a qingstor 'url'
 func qsParsePath(path string) (bucket, key string, err error) {
 	// Pattern to match a qingstor path
-	var matcher = regexp.MustCompile(`^([^/]*)(.*)$`)
 	parts := matcher.FindStringSubmatch(path)
 	if parts == nil {
 		err = errors.Errorf("Couldn't parse bucket out of qingstor path %q", path)

backend/s3/s3.go

@@ -39,9 +39,11 @@ import (
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/config/configmap"
 	"github.com/ncw/rclone/fs/config/configstruct"
+	"github.com/ncw/rclone/fs/fserrors"
 	"github.com/ncw/rclone/fs/fshttp"
 	"github.com/ncw/rclone/fs/hash"
 	"github.com/ncw/rclone/fs/walk"
+	"github.com/ncw/rclone/lib/pacer"
 	"github.com/ncw/rclone/lib/rest"
 	"github.com/ncw/swift"
 	"github.com/pkg/errors"
@@ -446,7 +448,12 @@ func init() {
 			Provider: "!AWS,IBMCOS",
 		}, {
 			Name: "acl",
-			Help: "Canned ACL used when creating buckets and/or storing objects in S3.\nFor more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl",
+			Help: `Canned ACL used when creating buckets and storing or copying objects.
+
+For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
+
+Note that this ACL is applied when server side copying objects as S3
+doesn't copy the ACL from the source but rather writes a fresh one.`,
 			Examples: []fs.OptionExample{{
 				Value:    "private",
 				Help:     "Owner gets FULL_CONTROL. No one else has access rights (default).",
@@ -498,10 +505,24 @@ func init() {
 			}, {
 				Value: "AES256",
 				Help:  "AES256",
+			}, {
+				Value: "aws:kms",
+				Help:  "aws:kms",
+			}},
+		}, {
+			Name:     "sse_kms_key_id",
+			Help:     "If using KMS ID you must provide the ARN of Key.",
+			Provider: "AWS",
+			Examples: []fs.OptionExample{{
+				Value: "",
+				Help:  "None",
+			}, {
+				Value: "arn:aws:kms:us-east-1:*",
+				Help:  "arn:aws:kms:*",
 			}},
 		}, {
 			Name:     "storage_class",
-			Help:     "The storage class to use when storing objects in S3.",
+			Help:     "The storage class to use when storing new objects in S3.",
 			Provider: "AWS",
 			Examples: []fs.OptionExample{{
 				Value: "",
@@ -520,9 +541,18 @@ func init() {
 				Help:  "One Zone Infrequent Access storage class",
 			}},
 		}, {
-			Name:     "chunk_size",
-			Help:     "Chunk size to use for uploading",
-			Default:  fs.SizeSuffix(s3manager.MinUploadPartSize),
+			Name: "chunk_size",
+			Help: `Chunk size to use for uploading.
+
+Any files larger than this will be uploaded in chunks of this
+size. The default is 5MB. The minimum is 5MB.
+
+Note that "--s3-upload-concurrency" chunks of this size are buffered
+in memory per transfer.
+
+If you are transferring large files over high speed links and you have
+enough memory, then increasing this will speed up the transfers.`,
+			Default:  minChunkSize,
 			Advanced: true,
 		}, {
 			Name:     "disable_checksum",
@@ -532,18 +562,41 @@ func init() {
 		}, {
 			Name:     "session_token",
 			Help:     "An AWS session token",
-			Hide:     fs.OptionHideBoth,
 			Advanced: true,
 		}, {
-			Name:     "upload_concurrency",
-			Help:     "Concurrency for multipart uploads.",
+			Name: "upload_concurrency",
+			Help: `Concurrency for multipart uploads.
+
+This is the number of chunks of the same file that are uploaded
+concurrently.
+
+If you are uploading small numbers of large file over high speed link
+and these uploads do not fully utilize your bandwidth, then increasing
+this may help to speed up the transfers.`,
 			Default:  2,
 			Advanced: true,
 		}, {
-			Name:     "force_path_style",
-			Help:     "If true use path style access if false use virtual hosted style.\nSome providers (eg Aliyun OSS or Netease COS) require this.",
+			Name: "force_path_style",
+			Help: `If true use path style access if false use virtual hosted style.
+
+If this is true (the default) then rclone will use path style access,
+if false then rclone will use virtual path style. See [the AWS S3
+docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro)
+for more info.
+
+Some providers (eg Aliyun OSS or Netease COS) require this set to false.`,
 			Default:  true,
 			Advanced: true,
+		}, {
+			Name: "v2_auth",
+			Help: `If true use v2 authentication.
+
+If this is false (the default) then rclone will use v4 authentication.
+If it is set then rclone will use v2 authentication.
+
+Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.`,
+			Default:  false,
+			Advanced: true,
 		}},
 	})
 }
@@ -556,6 +609,8 @@ const (
 	maxRetries     = 10                            // number of retries to make of operations
 	maxSizeForCopy = 5 * 1024 * 1024 * 1024        // The maximum size of object we can COPY
 	maxFileSize    = 5 * 1024 * 1024 * 1024 * 1024 // largest possible upload file size
+	minChunkSize   = fs.SizeSuffix(s3manager.MinUploadPartSize)
+	minSleep       = 10 * time.Millisecond // In case of error, start at 10ms sleep.
 )
 
 // Options defines the configuration for this backend
@@ -569,12 +624,14 @@ type Options struct {
 	LocationConstraint   string        `config:"location_constraint"`
 	ACL                  string        `config:"acl"`
 	ServerSideEncryption string        `config:"server_side_encryption"`
+	SSEKMSKeyID          string        `config:"sse_kms_key_id"`
 	StorageClass         string        `config:"storage_class"`
 	ChunkSize            fs.SizeSuffix `config:"chunk_size"`
 	DisableChecksum      bool          `config:"disable_checksum"`
 	SessionToken         string        `config:"session_token"`
 	UploadConcurrency    int           `config:"upload_concurrency"`
 	ForcePathStyle       bool          `config:"force_path_style"`
+	V2Auth               bool          `config:"v2_auth"`
 }
 
 // Fs represents a remote s3 server
@@ -589,6 +646,7 @@ type Fs struct {
 	bucketOKMu    sync.Mutex       // mutex to protect bucket OK
 	bucketOK      bool             // true if we have created the bucket
 	bucketDeleted bool             // true if we have deleted the bucket
+	pacer         *pacer.Pacer     // To pace the API calls
 }
 
 // Object describes a s3 object
@@ -634,8 +692,39 @@ func (f *Fs) Features() *fs.Features {
 	return f.features
 }
 
+// retryErrorCodes is a slice of error codes that we will retry
+// See: https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
+var retryErrorCodes = []int{
+	409, // Conflict - various states that could be resolved on a retry
+	503, // Service Unavailable/Slow Down - "Reduce your request rate"
+}
+
+//S3 is pretty resilient, and the built in retry handling is probably sufficient
+// as it should notice closed connections and timeouts which are the most likely
+// sort of failure modes
+func shouldRetry(err error) (bool, error) {
+
+	// If this is an awserr object, try and extract more useful information to determine if we should retry
+	if awsError, ok := err.(awserr.Error); ok {
+		// Simple case, check the original embedded error in case it's generically retriable
+		if fserrors.ShouldRetry(awsError.OrigErr()) {
+			return true, err
+		}
+		//Failing that, if it's a RequestFailure it's probably got an http status code we can check
+		if reqErr, ok := err.(awserr.RequestFailure); ok {
+			for _, e := range retryErrorCodes {
+				if reqErr.StatusCode() == e {
+					return true, err
+				}
+			}
+		}
+	}
+	//Ok, not an awserr, check for generic failure conditions
+	return fserrors.ShouldRetry(err), err
+}
+
 // Pattern to match a s3 path
-var matcher = regexp.MustCompile(`^([^/]*)(.*)$`)
+var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
 
 // parseParse parses a s3 'url'
 func s3ParsePath(path string) (bucket, directory string, err error) {
@@ -717,7 +806,7 @@ func s3Connection(opt *Options) (*s3.S3, *session.Session, error) {
 	// awsConfig.WithLogLevel(aws.LogDebugWithSigning)
 	ses := session.New()
 	c := s3.New(ses, awsConfig)
-	if opt.Region == "other-v2-signature" {
+	if opt.V2Auth || opt.Region == "other-v2-signature" {
 		fs.Debugf(nil, "Using v2 auth")
 		signer := func(req *request.Request) {
 			// Ignore AnonymousCredentials object
@@ -733,6 +822,21 @@ func s3Connection(opt *Options) (*s3.S3, *session.Session, error) {
 	return c, ses, nil
 }
 
+func checkUploadChunkSize(cs fs.SizeSuffix) error {
+	if cs < minChunkSize {
+		return errors.Errorf("%s is less than %s", cs, minChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadChunkSize(cs)
+	if err == nil {
+		old, f.opt.ChunkSize = f.opt.ChunkSize, cs
+	}
+	return
+}
+
 // NewFs constructs an Fs from the path, bucket:path
 func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	// Parse config into Options struct
@@ -741,8 +845,9 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		return nil, err
 	}
-	if opt.ChunkSize < fs.SizeSuffix(s3manager.MinUploadPartSize) {
-		return nil, errors.Errorf("s3 chunk size (%v) must be >= %v", opt.ChunkSize, fs.SizeSuffix(s3manager.MinUploadPartSize))
+	err = checkUploadChunkSize(opt.ChunkSize)
+	if err != nil {
+		return nil, errors.Wrap(err, "s3: chunk size")
 	}
 	bucket, directory, err := s3ParsePath(root)
 	if err != nil {
@@ -759,6 +864,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 		c:      c,
 		bucket: bucket,
 		ses:    ses,
+		pacer:  pacer.New().SetMinSleep(minSleep).SetPacer(pacer.S3Pacer),
 	}
 	f.features = (&fs.Features{
 		ReadMimeType:  true,
@@ -772,7 +878,10 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 			Bucket: &f.bucket,
 			Key:    &directory,
 		}
-		_, err = f.c.HeadObject(&req)
+		err = f.pacer.Call(func() (bool, error) {
+			_, err = f.c.HeadObject(&req)
+			return shouldRetry(err)
+		})
 		if err == nil {
 			f.root = path.Dir(directory)
 			if f.root == "." {
@@ -849,7 +958,12 @@ func (f *Fs) list(dir string, recurse bool, fn listFn) error {
 			MaxKeys:   &maxKeys,
 			Marker:    marker,
 		}
-		resp, err := f.c.ListObjects(&req)
+		var resp *s3.ListObjectsOutput
+		var err error
+		err = f.pacer.Call(func() (bool, error) {
+			resp, err = f.c.ListObjects(&req)
+			return shouldRetry(err)
+		})
 		if err != nil {
 			if awsErr, ok := err.(awserr.RequestFailure); ok {
 				if awsErr.StatusCode() == http.StatusNotFound {
@@ -974,7 +1088,11 @@ func (f *Fs) listBuckets(dir string) (entries fs.DirEntries, err error) {
 		return nil, fs.ErrorListBucketRequired
 	}
 	req := s3.ListBucketsInput{}
-	resp, err := f.c.ListBuckets(&req)
+	var resp *s3.ListBucketsOutput
+	err = f.pacer.Call(func() (bool, error) {
+		resp, err = f.c.ListBuckets(&req)
+		return shouldRetry(err)
+	})
 	if err != nil {
 		return nil, err
 	}
@@ -1059,7 +1177,10 @@ func (f *Fs) dirExists() (bool, error) {
 	req := s3.HeadBucketInput{
 		Bucket: &f.bucket,
 	}
-	_, err := f.c.HeadBucket(&req)
+	err := f.pacer.Call(func() (bool, error) {
+		_, err := f.c.HeadBucket(&req)
+		return shouldRetry(err)
+	})
 	if err == nil {
 		return true, nil
 	}
@@ -1096,7 +1217,10 @@ func (f *Fs) Mkdir(dir string) error {
 			LocationConstraint: &f.opt.LocationConstraint,
 		}
 	}
-	_, err := f.c.CreateBucket(&req)
+	err := f.pacer.Call(func() (bool, error) {
+		_, err := f.c.CreateBucket(&req)
+		return shouldRetry(err)
+	})
 	if err, ok := err.(awserr.Error); ok {
 		if err.Code() == "BucketAlreadyOwnedByYou" {
 			err = nil
@@ -1121,7 +1245,10 @@ func (f *Fs) Rmdir(dir string) error {
 	req := s3.DeleteBucketInput{
 		Bucket: &f.bucket,
 	}
-	_, err := f.c.DeleteBucket(&req)
+	err := f.pacer.Call(func() (bool, error) {
+		_, err := f.c.DeleteBucket(&req)
+		return shouldRetry(err)
+	})
 	if err == nil {
 		f.bucketOK = false
 		f.bucketDeleted = true
@@ -1164,11 +1291,24 @@ func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
 	source := pathEscape(srcFs.bucket + "/" + srcFs.root + srcObj.remote)
 	req := s3.CopyObjectInput{
 		Bucket:            &f.bucket,
+		ACL:               &f.opt.ACL,
 		Key:               &key,
 		CopySource:        &source,
 		MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
 	}
-	_, err = f.c.CopyObject(&req)
+	if f.opt.ServerSideEncryption != "" {
+		req.ServerSideEncryption = &f.opt.ServerSideEncryption
+	}
+	if f.opt.SSEKMSKeyID != "" {
+		req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
+	}
+	if f.opt.StorageClass != "" {
+		req.StorageClass = &f.opt.StorageClass
+	}
+	err = f.pacer.Call(func() (bool, error) {
+		_, err = f.c.CopyObject(&req)
+		return shouldRetry(err)
+	})
 	if err != nil {
 		return nil, err
 	}
@@ -1245,7 +1385,12 @@ func (o *Object) readMetaData() (err error) {
 		Bucket: &o.fs.bucket,
 		Key:    &key,
 	}
-	resp, err := o.fs.c.HeadObject(&req)
+	var resp *s3.HeadObjectOutput
+	err = o.fs.pacer.Call(func() (bool, error) {
+		var err error
+		resp, err = o.fs.c.HeadObject(&req)
+		return shouldRetry(err)
+	})
 	if err != nil {
 		if awsErr, ok := err.(awserr.RequestFailure); ok {
 			if awsErr.StatusCode() == http.StatusNotFound {
@@ -1329,7 +1474,19 @@ func (o *Object) SetModTime(modTime time.Time) error {
 		Metadata:          o.meta,
 		MetadataDirective: &directive,
 	}
-	_, err = o.fs.c.CopyObject(&req)
+	if o.fs.opt.ServerSideEncryption != "" {
+		req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
+	}
+	if o.fs.opt.SSEKMSKeyID != "" {
+		req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
+	}
+	if o.fs.opt.StorageClass != "" {
+		req.StorageClass = &o.fs.opt.StorageClass
+	}
+	err = o.fs.pacer.Call(func() (bool, error) {
+		_, err := o.fs.c.CopyObject(&req)
+		return shouldRetry(err)
+	})
 	return err
 }
 
@@ -1356,7 +1513,12 @@ func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
 			}
 		}
 	}
-	resp, err := o.fs.c.GetObject(&req)
+	var resp *s3.GetObjectOutput
+	err = o.fs.pacer.Call(func() (bool, error) {
+		var err error
+		resp, err = o.fs.c.GetObject(&req)
+		return shouldRetry(err)
+	})
 	if err, ok := err.(awserr.RequestFailure); ok {
 		if err.Code() == "InvalidObjectState" {
 			return nil, errors.Errorf("Object in GLACIER, restore first: %v", key)
@@ -1429,10 +1591,16 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 	if o.fs.opt.ServerSideEncryption != "" {
 		req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
 	}
+	if o.fs.opt.SSEKMSKeyID != "" {
+		req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
+	}
 	if o.fs.opt.StorageClass != "" {
 		req.StorageClass = &o.fs.opt.StorageClass
 	}
-	_, err = uploader.Upload(&req)
+	err = o.fs.pacer.CallNoRetry(func() (bool, error) {
+		_, err = uploader.Upload(&req)
+		return shouldRetry(err)
+	})
 	if err != nil {
 		return err
 	}
@@ -1450,7 +1618,10 @@ func (o *Object) Remove() error {
 		Bucket: &o.fs.bucket,
 		Key:    &key,
 	}
-	_, err := o.fs.c.DeleteObject(&req)
+	err := o.fs.pacer.Call(func() (bool, error) {
+		_, err := o.fs.c.DeleteObject(&req)
+		return shouldRetry(err)
+	})
 	return err
 }
 

backend/s3/s3_test.go

@@ -1,10 +1,10 @@
 // Test S3 filesystem interface
-package s3_test
+package s3
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/s3"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
@@ -12,6 +12,15 @@ import (
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
 		RemoteName: "TestS3:",
-		NilObject:  (*s3.Object)(nil),
+		NilObject:  (*Object)(nil),
+		ChunkedUpload: fstests.ChunkedUploadConfig{
+			MinChunkSize: minChunkSize,
+		},
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+var _ fstests.SetUploadChunkSizer = (*Fs)(nil)

backend/s3/v2sign.go

@@ -44,16 +44,7 @@ func sign(AccessKey, SecretKey string, req *http.Request) {
 	req.Header.Set("Date", date)
 
 	// Sort out URI
-	uri := req.URL.Opaque
-	if uri != "" {
-		if strings.HasPrefix(uri, "//") {
-			// Strip off //host/uri
-			uri = "/" + strings.Join(strings.Split(uri, "/")[3:], "/")
-			req.URL.Opaque = uri // reset to plain URI otherwise Ceph gets confused
-		}
-	} else {
-		uri = req.URL.Path
-	}
+	uri := req.URL.EscapedPath()
 	if uri == "" {
 		uri = "/"
 	}

backend/sftp/sftp.go

@@ -90,9 +90,20 @@ func init() {
 			Help:     "Allow asking for SFTP password when needed.",
 			Advanced: true,
 		}, {
-			Name:     "path_override",
-			Default:  "",
-			Help:     "Override path used by SSH connection.",
+			Name:    "path_override",
+			Default: "",
+			Help: `Override path used by SSH connection.
+
+This allows checksum calculation when SFTP and SSH paths are
+different. This issue affects among others Synology NAS boxes.
+
+Shared folders can be found in directories representing volumes
+
+    rclone sync /home/local/directory remote:/directory --ssh-path-override /volume2/directory
+
+Home directory can be found in a shared folder called "home"
+
+    rclone sync /home/local/directory remote:/home/directory --ssh-path-override /volume1/homes/USER/directory`,
 			Advanced: true,
 		}, {
 			Name:     "set_modtime",
@@ -758,6 +769,10 @@ func (o *Object) Hash(r hash.Type) (string, error) {
 		return "", hash.ErrUnsupported
 	}
 
+	if o.fs.opt.DisableHashCheck {
+		return "", nil
+	}
+
 	c, err := o.fs.getSftpConnection()
 	if err != nil {
 		return "", errors.Wrap(err, "Hash get SFTP connection")

backend/swift/swift.go

@@ -29,15 +29,27 @@ import (
 const (
 	directoryMarkerContentType = "application/directory" // content type of directory marker objects
 	listChunks                 = 1000                    // chunk size to read directory listings
+	defaultChunkSize           = 5 * fs.GibiByte
 )
 
+// SharedOptions are shared between swift and hubic
+var SharedOptions = []fs.Option{{
+	Name: "chunk_size",
+	Help: `Above this size files will be chunked into a _segments container.
+
+Above this size files will be chunked into a _segments container.  The
+default for this is 5GB which is its maximum value.`,
+	Default:  defaultChunkSize,
+	Advanced: true,
+}}
+
 // Register with Fs
 func init() {
 	fs.Register(&fs.RegInfo{
 		Name:        "swift",
 		Description: "Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)",
 		NewFs:       NewFs,
-		Options: []fs.Option{{
+		Options: append([]fs.Option{{
 			Name:    "env_auth",
 			Help:    "Get swift credentials from environment variables in standard OpenStack form.",
 			Default: false,
@@ -121,8 +133,13 @@ func init() {
 				Value: "admin",
 			}},
 		}, {
-			Name:    "storage_policy",
-			Help:    "The storage policy to use when creating a new container",
+			Name: "storage_policy",
+			Help: `The storage policy to use when creating a new container
+
+This applies the specified storage policy when creating a new
+container. The policy cannot be changed afterwards. The allowed
+configuration values and their meaning depend on your Swift storage
+provider.`,
 			Default: "",
 			Examples: []fs.OptionExample{{
 				Help:  "Default",
@@ -134,12 +151,7 @@ func init() {
 				Help:  "OVH Public Cloud Archive",
 				Value: "pca",
 			}},
-		}, {
-			Name:     "chunk_size",
-			Help:     "Above this size files will be chunked into a _segments container.",
-			Default:  fs.SizeSuffix(5 * 1024 * 1024 * 1024),
-			Advanced: true,
-		}},
+		}}, SharedOptions...),
 	})
 }
 
@@ -216,7 +228,7 @@ func (f *Fs) Features() *fs.Features {
 }
 
 // Pattern to match a swift path
-var matcher = regexp.MustCompile(`^([^/]*)(.*)$`)
+var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
 
 // parseParse parses a swift 'url'
 func parsePath(path string) (container, directory string, err error) {
@@ -277,12 +289,37 @@ func swiftConnection(opt *Options, name string) (*swift.Connection, error) {
 	// provided by wrapping the existing auth, so we can just
 	// override one or the other or both.
 	if StorageUrl != "" || AuthToken != "" {
+		// Re-write StorageURL and AuthToken if they are being
+		// overridden as c.Authenticate above will have
+		// overwritten them.
+		if StorageUrl != "" {
+			c.StorageUrl = StorageUrl
+		}
+		if AuthToken != "" {
+			c.AuthToken = AuthToken
+		}
 		c.Auth = newAuth(c.Auth, StorageUrl, AuthToken)
 	}
 	return c, nil
 }
 
-// NewFsWithConnection contstructs an Fs from the path, container:path
+func checkUploadChunkSize(cs fs.SizeSuffix) error {
+	const minChunkSize = fs.Byte
+	if cs < minChunkSize {
+		return errors.Errorf("%s is less than %s", cs, minChunkSize)
+	}
+	return nil
+}
+
+func (f *Fs) setUploadChunkSize(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	err = checkUploadChunkSize(cs)
+	if err == nil {
+		old, f.opt.ChunkSize = f.opt.ChunkSize, cs
+	}
+	return
+}
+
+// NewFsWithConnection constructs an Fs from the path, container:path
 // and authenticated connection.
 //
 // if noCheckContainer is set then the Fs won't check the container
@@ -332,6 +369,10 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		return nil, err
 	}
+	err = checkUploadChunkSize(opt.ChunkSize)
+	if err != nil {
+		return nil, errors.Wrap(err, "swift: chunk size")
+	}
 
 	c, err := swiftConnection(opt, name)
 	if err != nil {

backend/swift/swift_test.go

@@ -1,10 +1,10 @@
 // Test Swift filesystem interface
-package swift_test
+package swift
 
 import (
 	"testing"
 
-	"github.com/ncw/rclone/backend/swift"
+	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fstest/fstests"
 )
 
@@ -12,6 +12,12 @@ import (
 func TestIntegration(t *testing.T) {
 	fstests.Run(t, &fstests.Opt{
 		RemoteName: "TestSwift:",
-		NilObject:  (*swift.Object)(nil),
+		NilObject:  (*Object)(nil),
 	})
 }
+
+func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
+	return f.setUploadChunkSize(cs)
+}
+
+var _ fstests.SetUploadChunkSizer = (*Fs)(nil)

backend/union/union.go

@@ -0,0 +1,423 @@
+package union
+
+import (
+	"fmt"
+	"io"
+	"path"
+	"path/filepath"
+	"strings"
+	"time"
+
+	"github.com/ncw/rclone/fs"
+	"github.com/ncw/rclone/fs/config/configmap"
+	"github.com/ncw/rclone/fs/config/configstruct"
+	"github.com/ncw/rclone/fs/hash"
+	"github.com/pkg/errors"
+)
+
+// Register with Fs
+func init() {
+	fsi := &fs.RegInfo{
+		Name:        "union",
+		Description: "A stackable unification remote, which can appear to merge the contents of several remotes",
+		NewFs:       NewFs,
+		Options: []fs.Option{{
+			Name:     "remotes",
+			Help:     "List of space separated remotes.\nCan be 'remotea:test/dir remoteb:', '\"remotea:test/space dir\" remoteb:', etc.\nThe last remote is used to write to.",
+			Required: true,
+		}},
+	}
+	fs.Register(fsi)
+}
+
+// Options defines the configuration for this backend
+type Options struct {
+	Remotes fs.SpaceSepList `config:"remotes"`
+}
+
+// Fs represents a union of remotes
+type Fs struct {
+	name     string       // name of this remote
+	features *fs.Features // optional features
+	opt      Options      // options for this Fs
+	root     string       // the path we are working on
+	remotes  []fs.Fs      // slice of remotes
+	wr       fs.Fs        // writable remote
+	hashSet  hash.Set     // intersection of hash types
+}
+
+// Object describes a union Object
+//
+// This is a wrapped object which returns the Union Fs as its parent
+type Object struct {
+	fs.Object
+	fs *Fs // what this object is part of
+}
+
+// Wrap an existing object in the union Object
+func (f *Fs) wrapObject(o fs.Object) *Object {
+	return &Object{
+		Object: o,
+		fs:     f,
+	}
+}
+
+// Fs returns the union Fs as the parent
+func (o *Object) Fs() fs.Info {
+	return o.fs
+}
+
+// Name of the remote (as passed into NewFs)
+func (f *Fs) Name() string {
+	return f.name
+}
+
+// Root of the remote (as passed into NewFs)
+func (f *Fs) Root() string {
+	return f.root
+}
+
+// String converts this Fs to a string
+func (f *Fs) String() string {
+	return fmt.Sprintf("union root '%s'", f.root)
+}
+
+// Features returns the optional features of this Fs
+func (f *Fs) Features() *fs.Features {
+	return f.features
+}
+
+// Rmdir removes the root directory of the Fs object
+func (f *Fs) Rmdir(dir string) error {
+	return f.wr.Rmdir(dir)
+}
+
+// Hashes returns hash.HashNone to indicate remote hashing is unavailable
+func (f *Fs) Hashes() hash.Set {
+	return f.hashSet
+}
+
+// Mkdir makes the root directory of the Fs object
+func (f *Fs) Mkdir(dir string) error {
+	return f.wr.Mkdir(dir)
+}
+
+// Purge all files in the root and the root directory
+//
+// Implement this if you have a way of deleting all the files
+// quicker than just running Remove() on the result of List()
+//
+// Return an error if it doesn't exist
+func (f *Fs) Purge() error {
+	return f.wr.Features().Purge()
+}
+
+// Copy src to this remote using server side copy operations.
+//
+// This is stored with the remote path given
+//
+// It returns the destination Object and a possible error
+//
+// Will only be called if src.Fs().Name() == f.Name()
+//
+// If it isn't possible then return fs.ErrorCantCopy
+func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
+	if src.Fs() != f.wr {
+		fs.Debugf(src, "Can't copy - not same remote type")
+		return nil, fs.ErrorCantCopy
+	}
+	o, err := f.wr.Features().Copy(src, remote)
+	if err != nil {
+		return nil, err
+	}
+	return f.wrapObject(o), nil
+}
+
+// Move src to this remote using server side move operations.
+//
+// This is stored with the remote path given
+//
+// It returns the destination Object and a possible error
+//
+// Will only be called if src.Fs().Name() == f.Name()
+//
+// If it isn't possible then return fs.ErrorCantMove
+func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
+	if src.Fs() != f.wr {
+		fs.Debugf(src, "Can't move - not same remote type")
+		return nil, fs.ErrorCantMove
+	}
+	o, err := f.wr.Features().Move(src, remote)
+	if err != nil {
+		return nil, err
+	}
+	return f.wrapObject(o), err
+}
+
+// DirMove moves src, srcRemote to this remote at dstRemote
+// using server side move operations.
+//
+// Will only be called if src.Fs().Name() == f.Name()
+//
+// If it isn't possible then return fs.ErrorCantDirMove
+//
+// If destination exists then return fs.ErrorDirExists
+func (f *Fs) DirMove(src fs.Fs, srcRemote, dstRemote string) error {
+	srcFs, ok := src.(*Fs)
+	if !ok {
+		fs.Debugf(srcFs, "Can't move directory - not same remote type")
+		return fs.ErrorCantDirMove
+	}
+	return f.wr.Features().DirMove(srcFs.wr, srcRemote, dstRemote)
+}
+
+// ChangeNotify calls the passed function with a path
+// that has had changes. If the implementation
+// uses polling, it should adhere to the given interval.
+// At least one value will be written to the channel,
+// specifying the initial value and updated values might
+// follow. A 0 Duration should pause the polling.
+// The ChangeNotify implemantion must empty the channel
+// regulary. When the channel gets closed, the implemantion
+// should stop polling and release resources.
+func (f *Fs) ChangeNotify(fn func(string, fs.EntryType), ch <-chan time.Duration) {
+	var remoteChans []chan time.Duration
+
+	for _, remote := range f.remotes {
+		if ChangeNotify := remote.Features().ChangeNotify; ChangeNotify != nil {
+			ch := make(chan time.Duration)
+			remoteChans = append(remoteChans, ch)
+			ChangeNotify(fn, ch)
+		}
+	}
+
+	go func() {
+		for i := range ch {
+			for _, c := range remoteChans {
+				c <- i
+			}
+		}
+		for _, c := range remoteChans {
+			close(c)
+		}
+	}()
+}
+
+// DirCacheFlush resets the directory cache - used in testing
+// as an optional interface
+func (f *Fs) DirCacheFlush() {
+	for _, remote := range f.remotes {
+		if DirCacheFlush := remote.Features().DirCacheFlush; DirCacheFlush != nil {
+			DirCacheFlush()
+		}
+	}
+}
+
+// PutStream uploads to the remote path with the modTime given of indeterminate size
+//
+// May create the object even if it returns an error - if so
+// will return the object and the error, otherwise will return
+// nil and the error
+func (f *Fs) PutStream(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
+	o, err := f.wr.Features().PutStream(in, src, options...)
+	if err != nil {
+		return nil, err
+	}
+	return f.wrapObject(o), err
+}
+
+// About gets quota information from the Fs
+func (f *Fs) About() (*fs.Usage, error) {
+	return f.wr.Features().About()
+}
+
+// Put in to the remote path with the modTime given of the given size
+//
+// May create the object even if it returns an error - if so
+// will return the object and the error, otherwise will return
+// nil and the error
+func (f *Fs) Put(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
+	o, err := f.wr.Put(in, src, options...)
+	if err != nil {
+		return nil, err
+	}
+	return f.wrapObject(o), err
+}
+
+// List the objects and directories in dir into entries.  The
+// entries can be returned in any order but should be for a
+// complete directory.
+//
+// dir should be "" to list the root, and should not have
+// trailing slashes.
+//
+// This should return ErrDirNotFound if the directory isn't
+// found.
+func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
+	set := make(map[string]fs.DirEntry)
+	found := false
+	for _, remote := range f.remotes {
+		var remoteEntries, err = remote.List(dir)
+		if err == fs.ErrorDirNotFound {
+			continue
+		}
+		if err != nil {
+			return nil, errors.Wrapf(err, "List failed on %v", remote)
+		}
+		found = true
+		for _, remoteEntry := range remoteEntries {
+			set[remoteEntry.Remote()] = remoteEntry
+		}
+	}
+	if !found {
+		return nil, fs.ErrorDirNotFound
+	}
+	for _, entry := range set {
+		if o, ok := entry.(fs.Object); ok {
+			entry = f.wrapObject(o)
+		}
+		entries = append(entries, entry)
+	}
+	return entries, nil
+}
+
+// NewObject creates a new remote union file object based on the first Object it finds (reverse remote order)
+func (f *Fs) NewObject(path string) (fs.Object, error) {
+	for i := range f.remotes {
+		var remote = f.remotes[len(f.remotes)-i-1]
+		var obj, err = remote.NewObject(path)
+		if err == fs.ErrorObjectNotFound {
+			continue
+		}
+		if err != nil {
+			return nil, errors.Wrapf(err, "NewObject failed on %v", remote)
+		}
+		return f.wrapObject(obj), nil
+	}
+	return nil, fs.ErrorObjectNotFound
+}
+
+// Precision is the greatest Precision of all remotes
+func (f *Fs) Precision() time.Duration {
+	var greatestPrecision time.Duration
+	for _, remote := range f.remotes {
+		if remote.Precision() > greatestPrecision {
+			greatestPrecision = remote.Precision()
+		}
+	}
+	return greatestPrecision
+}
+
+// NewFs constructs an Fs from the path.
+//
+// The returned Fs is the actual Fs, referenced by remote in the config
+func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
+	// Parse config into Options struct
+	opt := new(Options)
+	err := configstruct.Set(m, opt)
+	if err != nil {
+		return nil, err
+	}
+	if len(opt.Remotes) == 0 {
+		return nil, errors.New("union can't point to an empty remote - check the value of the remotes setting")
+	}
+	if len(opt.Remotes) == 1 {
+		return nil, errors.New("union can't point to a single remote - check the value of the remotes setting")
+	}
+	for _, remote := range opt.Remotes {
+		if strings.HasPrefix(remote, name+":") {
+			return nil, errors.New("can't point union remote at itself - check the value of the remote setting")
+		}
+	}
+
+	var remotes []fs.Fs
+	for i := range opt.Remotes {
+		// Last remote first so we return the correct (last) matching fs in case of fs.ErrorIsFile
+		var remote = opt.Remotes[len(opt.Remotes)-i-1]
+		_, configName, fsPath, err := fs.ParseRemote(remote)
+		if err != nil {
+			return nil, err
+		}
+		var rootString = path.Join(fsPath, filepath.ToSlash(root))
+		if configName != "local" {
+			rootString = configName + ":" + rootString
+		}
+		myFs, err := fs.NewFs(rootString)
+		if err != nil {
+			if err == fs.ErrorIsFile {
+				return myFs, err
+			}
+			return nil, err
+		}
+		remotes = append(remotes, myFs)
+	}
+
+	// Reverse the remotes again so they are in the order as before
+	for i, j := 0, len(remotes)-1; i < j; i, j = i+1, j-1 {
+		remotes[i], remotes[j] = remotes[j], remotes[i]
+	}
+
+	f := &Fs{
+		name:    name,
+		root:    root,
+		opt:     *opt,
+		remotes: remotes,
+		wr:      remotes[len(remotes)-1],
+	}
+	var features = (&fs.Features{
+		CaseInsensitive:         true,
+		DuplicateFiles:          false,
+		ReadMimeType:            true,
+		WriteMimeType:           true,
+		CanHaveEmptyDirectories: true,
+		BucketBased:             true,
+		SetTier:                 true,
+		GetTier:                 true,
+	}).Fill(f)
+	features = features.Mask(f.wr) // mask the features just on the writable fs
+
+	// FIXME maybe should be masking the bools here?
+
+	// Clear ChangeNotify and DirCacheFlush if all are nil
+	clearChangeNotify := true
+	clearDirCacheFlush := true
+	for _, remote := range f.remotes {
+		remoteFeatures := remote.Features()
+		if remoteFeatures.ChangeNotify != nil {
+			clearChangeNotify = false
+		}
+		if remoteFeatures.DirCacheFlush != nil {
+			clearDirCacheFlush = false
+		}
+	}
+	if clearChangeNotify {
+		features.ChangeNotify = nil
+	}
+	if clearDirCacheFlush {
+		features.DirCacheFlush = nil
+	}
+
+	f.features = features
+
+	// Get common intersection of hashes
+	hashSet := f.remotes[0].Hashes()
+	for _, remote := range f.remotes[1:] {
+		hashSet = hashSet.Overlap(remote.Hashes())
+	}
+	f.hashSet = hashSet
+
+	return f, nil
+}
+
+// Check the interfaces are satisfied
+var (
+	_ fs.Fs              = (*Fs)(nil)
+	_ fs.Purger          = (*Fs)(nil)
+	_ fs.PutStreamer     = (*Fs)(nil)
+	_ fs.Copier          = (*Fs)(nil)
+	_ fs.Mover           = (*Fs)(nil)
+	_ fs.DirMover        = (*Fs)(nil)
+	_ fs.DirCacheFlusher = (*Fs)(nil)
+	_ fs.ChangeNotifier  = (*Fs)(nil)
+	_ fs.Abouter         = (*Fs)(nil)
+)

backend/union/union_test.go

@@ -0,0 +1,18 @@
+// Test Union filesystem interface
+package union_test
+
+import (
+	"testing"
+
+	_ "github.com/ncw/rclone/backend/local"
+	"github.com/ncw/rclone/fstest/fstests"
+)
+
+// TestIntegration runs integration tests against the remote
+func TestIntegration(t *testing.T) {
+	fstests.Run(t, &fstests.Opt{
+		RemoteName:  "TestUnion:",
+		NilObject:   nil,
+		SkipFsMatch: true,
+	})
+}

backend/webdav/api/types.go

@@ -12,6 +12,9 @@ import (
 const (
 	// Wed, 27 Sep 2017 14:28:34 GMT
 	timeFormat = time.RFC1123
+	// The same as time.RFC1123 with optional leading zeros on the date
+	// see https://github.com/ncw/rclone/issues/2574
+	noZerosRFC1123 = "Mon, _2 Jan 2006 15:04:05 MST"
 )
 
 // Multistatus contains responses returned from an HTTP 207 return code
@@ -138,9 +141,11 @@ func (t *Time) MarshalXML(e *xml.Encoder, start xml.StartElement) error {
 
 // Possible time formats to parse the time with
 var timeFormats = []string{
-	timeFormat,    // Wed, 27 Sep 2017 14:28:34 GMT (as per RFC)
-	time.RFC1123Z, // Fri, 05 Jan 2018 14:14:38 +0000 (as used by mydrive.ch)
-	time.UnixDate, // Wed May 17 15:31:58 UTC 2017 (as used in an internal server)
+	timeFormat,     // Wed, 27 Sep 2017 14:28:34 GMT (as per RFC)
+	time.RFC1123Z,  // Fri, 05 Jan 2018 14:14:38 +0000 (as used by mydrive.ch)
+	time.UnixDate,  // Wed May 17 15:31:58 UTC 2017 (as used in an internal server)
+	noZerosRFC1123, // Fri, 7 Sep 2018 08:49:58 GMT (as used by server in #2574)
+	time.RFC3339,   // Wed, 31 Oct 2018 13:57:11 CET (as used by komfortcloud.de)
 }
 
 // UnmarshalXML turns XML into a Time
@@ -151,6 +156,12 @@ func (t *Time) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error {
 		return err
 	}
 
+	// If time is missing then return the epoch
+	if v == "" {
+		*t = Time(time.Unix(0, 0))
+		return nil
+	}
+
 	// Parse the time format in multiple possible ways
 	var newT time.Time
 	for _, timeFormat := range timeFormats {

backend/webdav/odrvcookie/renew.go

@@ -0,0 +1,32 @@
+package odrvcookie
+
+import (
+	"time"
+
+	"github.com/ncw/rclone/lib/rest"
+)
+
+// CookieRenew holds information for the renew
+type CookieRenew struct {
+	srv     *rest.Client
+	timer   *time.Ticker
+	renewFn func()
+}
+
+// NewRenew returns and starts a CookieRenew
+func NewRenew(interval time.Duration, renewFn func()) *CookieRenew {
+	renew := CookieRenew{
+		timer:   time.NewTicker(interval),
+		renewFn: renewFn,
+	}
+	go renew.Renew()
+	return &renew
+}
+
+// Renew calls the renewFn for every tick
+func (c *CookieRenew) Renew() {
+	for {
+		<-c.timer.C // wait for tick
+		c.renewFn()
+	}
+}

backend/webdav/webdav.go

@@ -372,6 +372,17 @@ func (f *Fs) setQuirks(vendor string) error {
 		if err != nil {
 			return err
 		}
+
+		odrvcookie.NewRenew(12*time.Hour, func() {
+			spCookies, err := spCk.Cookies()
+			if err != nil {
+				fs.Errorf("could not renew cookies: %s", err.Error())
+				return
+			}
+			f.srv.SetCookie(&spCookies.FedAuth, &spCookies.RtFa)
+			fs.Debugf(spCookies, "successfully renewed sharepoint cookies")
+		})
+
 		f.srv.SetCookie(&spCookies.FedAuth, &spCookies.RtFa)
 
 		// sharepoint, unlike the other vendors, only lists files if the depth header is set to 0
@@ -957,6 +968,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 		Body:          in,
 		NoResponse:    true,
 		ContentLength: &size, // FIXME this isn't necessary with owncloud - See https://github.com/nextcloud/nextcloud-snap/issues/365
+		ContentType:   fs.MimeType(src),
 	}
 	if o.fs.useOCMtime {
 		opts.ExtraHeaders = map[string]string{
@@ -968,6 +980,12 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOptio
 		return shouldRetry(resp, err)
 	})
 	if err != nil {
+		// Give the WebDAV server a chance to get its internal state in order after the
+		// error.  The error may have been local in which case we closed the connection.
+		// The server may still be dealing with it for a moment. A sleep isn't ideal but I
+		// haven't been able to think of a better method to find out if the server has
+		// finished - ncw
+		time.Sleep(1 * time.Second)
 		// Remove failed upload
 		_ = o.Remove()
 		return err

backend/yandex/api/client.go

@@ -67,10 +67,8 @@ var defaultErrorHandler ErrorHandler = func(resp *http.Response) error {
 func (HTTPRequest *HTTPRequest) run(client *Client) ([]byte, error) {
 	var err error
 	values := make(url.Values)
-	if HTTPRequest.Parameters != nil {
-		for k, v := range HTTPRequest.Parameters {
-			values.Set(k, fmt.Sprintf("%v", v))
-		}
+	for k, v := range HTTPRequest.Parameters {
+		values.Set(k, fmt.Sprintf("%v", v))
 	}
 
 	var req *http.Request

bin/bisect-go-rclone.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# An example script to run when bisecting go with git bisect -run when
+# looking for an rclone regression
+
+# Run this from the go root
+
+set -e
+
+# Compile the go version
+cd src
+./make.bash || exit 125
+
+# Make sure we are using it
+source ~/bin/use-go1.11
+go version
+
+# Compile rclone
+cd ~/go/src/github.com/ncw/rclone
+make
+
+# run the failing test
+go run -race race.go

bin/bisect-rclone.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+# Example script for git bisect run
+#
+# Copy this file into /tmp say before running as it will be
+# overwritten by the bisect as it is checked in.
+#
+# Change the test below to find out whether rclone is working or not
+#
+# Run from the project root
+#
+# git bisect start
+# git checkout master
+# git bisect bad
+# git checkout v1.41 (or whatever is the first good one)
+# git bisect good
+# git bisect run /tmp/bisect-rclone.sh
+
+set -e
+
+# Compile notifying git on compile failure
+make || exit 125
+rclone version
+
+# Test whatever it is that is going wrong - exit with non zero exit code on failure
+# commented out examples follow
+
+# truncate -s 10M /tmp/10M
+# rclone delete azure:rclone-test1/10M || true
+# rclone --retries 1 copyto -vv /tmp/10M azure:rclone-test1/10M --azureblob-upload-cutoff 1M
+
+# rm -f "/tmp/tests's.docx" || true
+# rclone -vv --retries 1 copy "drive:test/tests's.docx" /tmp

bin/make_backend_docs.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python2
+"""
+Make backend documentation
+"""
+
+import os
+import subprocess
+
+marker = "<!--- autogenerated options"
+start = marker + " start"
+stop = marker + " stop"
+
+def find_backends():
+    """Return a list of all backends"""
+    return [ x for x in os.listdir("backend") if x not in ("all",) ]
+
+def output_docs(backend, out):
+    """Output documentation for backend options to out"""
+    out.flush()
+    subprocess.check_call(["rclone", "help", "backend", backend], stdout=out)
+
+def alter_doc(backend):
+    """Alter the documentation for backend"""
+    doc_file = "docs/content/"+backend+".md"
+    if not os.path.exists(doc_file):
+        raise ValueError("Didn't find doc file %s" % (doc_file,))
+    new_file = doc_file+"~new~"
+    altered = False
+    with open(doc_file, "r") as in_file, open(new_file, "w") as out_file:
+        in_docs = False
+        for line in in_file:
+            if not in_docs:
+                if start in line:
+                    in_docs = True
+                    start_full = start + " - DO NOT EDIT, instead edit fs.RegInfo in backend/%s/%s.go then run make backenddocs -->\n" % (backend, backend)
+                    out_file.write(start_full)
+                    output_docs(backend, out_file)
+                    out_file.write(stop+" -->\n")
+                    altered = True
+            if not in_docs:
+                out_file.write(line)
+            if in_docs:
+                if stop in line:
+                    in_docs = False
+    os.rename(doc_file, doc_file+"~")
+    os.rename(new_file, doc_file)
+    if not altered:
+        raise ValueError("Didn't find '%s' markers for in %s" % (start, doc_file))
+
+if __name__ == "__main__":
+    failed, success = 0, 0
+    for backend in find_backends():
+        try:
+            alter_doc(backend)
+        except Exception, e:
+            print "Failed adding docs for %s backend: %s" % (backend, e)
+            failed += 1
+        else:
+            success += 1
+    print "Added docs for %d backends with %d failures" % (success, failed)

bin/make_changelog.py

@@ -0,0 +1,173 @@
+#!/usr/bin/python
+"""
+Generate a markdown changelog for the rclone project
+"""
+
+import os
+import sys
+import re
+import datetime
+import subprocess
+from collections import defaultdict
+
+IGNORE_RES = [
+    r"^Add .* to contributors$",
+    r"^Start v\d+.\d+-DEV development$",
+    r"^Version v\d.\d+$",
+]
+
+IGNORE_RE = re.compile("(?:" + "|".join(IGNORE_RES) + ")")
+
+CATEGORY = re.compile(r"(^[\w/ ]+(?:, *[\w/ ]+)*):\s*(.*)$")
+
+backends = [ x for x in os.listdir("backend") if x != "all"]
+
+backend_aliases = {
+    "amazon cloud drive" : "amazonclouddrive",
+    "acd" : "amazonclouddrive",
+    "google cloud storage" : "googlecloudstorage",
+    "gcs" : "googlecloudstorage",
+    "azblob" : "azureblob",
+    "mountlib": "mount",
+    "cmount": "mount",
+    "mount/cmount": "mount",
+}
+
+backend_titles = {
+    "amazonclouddrive": "Amazon Cloud Drive",
+    "googlecloudstorage": "Google Cloud Storage",
+    "azureblob": "Azure Blob",
+    "ftp": "FTP",
+    "sftp": "SFTP",
+    "http": "HTTP",
+    "webdav": "WebDAV",
+}
+
+STRIP_FIX_RE = re.compile(r"(\s+-)?\s+((fixes|addresses)\s+)?#\d+", flags=re.I)
+
+STRIP_PATH_RE = re.compile(r"^(backend|fs)/")
+
+IS_FIX_RE = re.compile(r"\b(fix|fixes)\b", flags=re.I)
+
+def make_out(data, indent=""):
+    """Return a out, lines the first being a function for output into the second"""
+    out_lines = []
+    def out(category, title=None):
+        if title == None:
+            title = category
+        lines = data.get(category)
+        if not lines:
+            return
+        del(data[category])
+        if indent != "" and len(lines) == 1:
+            out_lines.append(indent+"* " + title+": " + lines[0])
+            return
+        out_lines.append(indent+"* " + title)
+        for line in lines:
+            out_lines.append(indent+"    * " + line)
+    return out, out_lines
+
+
+def process_log(log):
+    """Process the incoming log into a category dict of lists"""
+    by_category = defaultdict(list)
+    for log_line in reversed(log.split("\n")):
+        log_line = log_line.strip()
+        hash, author, timestamp, message = log_line.split("|", 3)
+        message = message.strip()
+        if IGNORE_RE.search(message):
+            continue
+        match = CATEGORY.search(message)
+        categories = "UNKNOWN"
+        if match:
+            categories = match.group(1).lower()
+            message = match.group(2)
+        message = STRIP_FIX_RE.sub("", message)
+        message = message +" ("+author+")"
+        message = message[0].upper()+message[1:]
+        seen = set()
+        for category in categories.split(","):
+            category = category.strip()
+            category = STRIP_PATH_RE.sub("", category)
+            category = backend_aliases.get(category, category)
+            if category in seen:
+                continue
+            by_category[category].append(message)
+            seen.add(category)
+            #print category, hash, author, timestamp, message
+    return by_category
+
+def main():
+    if len(sys.argv) != 3:
+        print >>sys.stderr, "Syntax: %s vX.XX vX.XY" % sys.argv[0]
+        sys.exit(1)
+    version, next_version = sys.argv[1], sys.argv[2]
+    log = subprocess.check_output(["git", "log", '''--pretty=format:%H|%an|%aI|%s'''] + [version+".."+next_version])
+    by_category = process_log(log)
+
+    # Output backends first so remaining in by_category are core items
+    out, backend_lines = make_out(by_category)
+    out("mount", title="Mount")
+    out("vfs", title="VFS")
+    out("local", title="Local")
+    out("cache", title="Cache")
+    out("crypt", title="Crypt")
+    backend_names = sorted(x for x in by_category.keys() if x in backends)
+    for backend_name in backend_names:
+        if backend_name in backend_titles:
+            backend_title = backend_titles[backend_name]
+        else:
+            backend_title = backend_name.title()
+        out(backend_name, title=backend_title)
+
+    # Split remaining in by_category into new features and fixes
+    new_features = defaultdict(list)
+    bugfixes = defaultdict(list)
+    for name, messages in by_category.iteritems():
+        for message in messages:
+            if IS_FIX_RE.search(message):
+                bugfixes[name].append(message)
+            else:
+                new_features[name].append(message)
+
+    # Output new features
+    out, new_features_lines = make_out(new_features, indent="    ")
+    for name in sorted(new_features.keys()):
+        out(name)
+
+    # Output bugfixes
+    out, bugfix_lines = make_out(bugfixes, indent="    ")
+    for name in sorted(bugfixes.keys()):
+        out(name)
+
+    # Read old changlog and split
+    with open("docs/content/changelog.md") as fd:
+        old_changelog = fd.read()
+    heading = "# Changelog"
+    i = old_changelog.find(heading)
+    if i < 0:
+        raise AssertionError("Couldn't find heading in old changelog")
+    i += len(heading)
+    old_head, old_tail = old_changelog[:i], old_changelog[i:]
+
+    # Update the build date
+    old_head = re.sub(r"\d\d\d\d-\d\d-\d\d", str(datetime.date.today()), old_head)
+
+    # Output combined changelog with new part
+    sys.stdout.write(old_head)
+    sys.stdout.write("""
+
+## %s - %s
+
+* New backends
+* New commands
+* New Features
+%s
+* Bug Fixes
+%s
+%s""" % (next_version, datetime.date.today(), "\n".join(new_features_lines), "\n".join(bugfix_lines), "\n".join(backend_lines)))
+    sys.stdout.write(old_tail)
+                
+
+if __name__ == "__main__":
+    main()

bin/make_manual.py

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 """
 Make single page versions of the documentation for release and
 conversion into man pages etc.
@@ -44,6 +44,7 @@ docs = [
     "swift.md",
     "pcloud.md",
     "sftp.md",
+    "union.md",
     "webdav.md",
     "yandex.md",
 

bin/make_rc_docs.sh

@@ -4,18 +4,20 @@
 set -e
 
 go install
-mkdir -p /tmp/rclone_cache_test
+mkdir -p /tmp/rclone/cache_test
+mkdir -p /tmp/rclone/rc_mount
 export RCLONE_CONFIG_RCDOCS_TYPE=cache
 export RCLONE_CONFIG_RCDOCS_REMOTE=/tmp/rclone/cache_test
-rclone -q --rc mount rcdocs: /mnt/tmp/ &
+rclone -q --rc mount rcdocs: /tmp/rclone/rc_mount &
 sleep 0.5
-rclone rc > /tmp/z.md
-fusermount -z -u /mnt/tmp/
+rclone rc > /tmp/rclone/z.md
+fusermount -u -z /tmp/rclone/rc_mount > /dev/null 2>&1 || umount /tmp/rclone/rc_mount
 
 awk '
     BEGIN       {p=1}
-    /^<!--- autogenerated start/  {print;system("cat /tmp/z.md");p=0}
+    /^<!--- autogenerated start/  {print;system("cat /tmp/rclone/z.md");p=0}
     /^<!--- autogenerated stop/   {p=1}
-    p' docs/content/rc.md > /tmp/rc.md
+    p' docs/content/rc.md > /tmp/rclone/rc.md
 
-mv /tmp/rc.md docs/content/rc.md
+mv /tmp/rclone/rc.md docs/content/rc.md
+rm -rf /tmp/rclone

bin/tidy-beta

@@ -0,0 +1,17 @@
+#!/bin/sh
+# Use this script after a release to tidy the betas
+
+version="$1"
+if [ "$version" = "" ]; then
+    echo "Syntax: $0 <version> [delete]"
+    exit 1
+fi
+dry_run="--dry-run"
+if [ "$2" = "delete" ]; then
+    dry_run=""
+else
+    echo "Running in --dry-run mode"
+    echo "Use '$0 $version delete' to actually delete files"
+fi
+
+rclone ${dry_run} --fast-list -P --checkers 16 --transfers 16 delete --include "**/${version}**" memstore:beta-rclone-org

cmd/all/all.go

@@ -14,6 +14,7 @@ import (
 	_ "github.com/ncw/rclone/cmd/config"
 	_ "github.com/ncw/rclone/cmd/copy"
 	_ "github.com/ncw/rclone/cmd/copyto"
+	_ "github.com/ncw/rclone/cmd/copyurl"
 	_ "github.com/ncw/rclone/cmd/cryptcheck"
 	_ "github.com/ncw/rclone/cmd/cryptdecode"
 	_ "github.com/ncw/rclone/cmd/dbhashsum"
@@ -42,10 +43,12 @@ import (
 	_ "github.com/ncw/rclone/cmd/purge"
 	_ "github.com/ncw/rclone/cmd/rc"
 	_ "github.com/ncw/rclone/cmd/rcat"
+	_ "github.com/ncw/rclone/cmd/rcd"
 	_ "github.com/ncw/rclone/cmd/reveal"
 	_ "github.com/ncw/rclone/cmd/rmdir"
 	_ "github.com/ncw/rclone/cmd/rmdirs"
 	_ "github.com/ncw/rclone/cmd/serve"
+	_ "github.com/ncw/rclone/cmd/settier"
 	_ "github.com/ncw/rclone/cmd/sha1sum"
 	_ "github.com/ncw/rclone/cmd/size"
 	_ "github.com/ncw/rclone/cmd/sync"

cmd/cmd.go

@@ -17,12 +17,9 @@ import (
 	"runtime/pprof"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
-	"github.com/pkg/errors"
-	"github.com/spf13/cobra"
-	"github.com/spf13/pflag"
-
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/accounting"
 	"github.com/ncw/rclone/fs/config/configflags"
@@ -32,9 +29,12 @@ import (
 	"github.com/ncw/rclone/fs/fserrors"
 	"github.com/ncw/rclone/fs/fspath"
 	fslog "github.com/ncw/rclone/fs/log"
-	"github.com/ncw/rclone/fs/rc"
 	"github.com/ncw/rclone/fs/rc/rcflags"
+	"github.com/ncw/rclone/fs/rc/rcserver"
 	"github.com/ncw/rclone/lib/atexit"
+	"github.com/pkg/errors"
+	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
 )
 
 // Globals
@@ -66,81 +66,6 @@ const (
 	exitCodeTransferExceeded
 )
 
-// Root is the main rclone command
-var Root = &cobra.Command{
-	Use:   "rclone",
-	Short: "Sync files and directories to and from local and remote object stores - " + fs.Version,
-	Long: `
-Rclone is a command line program to sync files and directories to and
-from various cloud storage systems and using file transfer services, such as:
-
-  * Amazon Drive
-  * Amazon S3
-  * Backblaze B2
-  * Box
-  * Dropbox
-  * FTP
-  * Google Cloud Storage
-  * Google Drive
-  * HTTP
-  * Hubic
-  * Jottacloud
-  * Mega
-  * Microsoft Azure Blob Storage
-  * Microsoft OneDrive
-  * OpenDrive
-  * Openstack Swift / Rackspace cloud files / Memset Memstore
-  * pCloud
-  * QingStor
-  * SFTP
-  * Webdav / Owncloud / Nextcloud
-  * Yandex Disk
-  * The local filesystem
-
-Features
-
-  * MD5/SHA1 hashes checked at all times for file integrity
-  * Timestamps preserved on files
-  * Partial syncs supported on a whole file basis
-  * Copy mode to just copy new/changed files
-  * Sync (one way) mode to make a directory identical
-  * Check mode to check for file hash equality
-  * Can sync to and from network, eg two different cloud accounts
-
-See the home page for installation, usage, documentation, changelog
-and configuration walkthroughs.
-
-  * https://rclone.org/
-`,
-	PersistentPostRun: func(cmd *cobra.Command, args []string) {
-		fs.Debugf("rclone", "Version %q finishing with parameters %q", fs.Version, os.Args)
-		atexit.Run()
-	},
-}
-
-// runRoot implements the main rclone command with no subcommands
-func runRoot(cmd *cobra.Command, args []string) {
-	if version {
-		ShowVersion()
-		resolveExitCode(nil)
-	} else {
-		_ = Root.Usage()
-		_, _ = fmt.Fprintf(os.Stderr, "Command not found.\n")
-		resolveExitCode(errorCommandNotFound)
-	}
-}
-
-func init() {
-	// Add global flags
-	configflags.AddFlags(pflag.CommandLine)
-	filterflags.AddFlags(pflag.CommandLine)
-	rcflags.AddFlags(pflag.CommandLine)
-
-	Root.Run = runRoot
-	Root.Flags().BoolVarP(&version, "version", "V", false, "Print the version number")
-	cobra.OnInitialize(initConfig)
-}
-
 // ShowVersion prints the version to stdout
 func ShowVersion() {
 	fmt.Printf("rclone %s\n", fs.Version)
@@ -293,11 +218,13 @@ func ShowStats() bool {
 // Run the function with stats and retries if required
 func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 	var err error
-	var stopStats chan struct{}
+	stopStats := func() {}
 	if !showStats && ShowStats() {
 		showStats = true
 	}
-	if showStats {
+	if fs.Config.Progress {
+		stopStats = startProgress()
+	} else if showStats {
 		stopStats = StartStats()
 	}
 	SigInfoHandler()
@@ -309,11 +236,11 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 			}
 			break
 		}
-		if fserrors.IsFatalError(err) {
+		if fserrors.IsFatalError(err) || accounting.Stats.HadFatalError() {
 			fs.Errorf(nil, "Fatal error received - not attempting retries")
 			break
 		}
-		if fserrors.IsNoRetryError(err) {
+		if fserrors.IsNoRetryError(err) || (accounting.Stats.Errored() && !accounting.Stats.HadRetryError()) {
 			fs.Errorf(nil, "Can't retry this error - not attempting retries")
 			break
 		}
@@ -329,9 +256,7 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 			time.Sleep(*retriesInterval)
 		}
 	}
-	if showStats {
-		close(stopStats)
-	}
+	stopStats()
 	if err != nil {
 		log.Printf("Failed to %s: %v", cmd.Name(), err)
 		resolveExitCode(err)
@@ -382,24 +307,31 @@ func CheckArgs(MinArgs, MaxArgs int, cmd *cobra.Command, args []string) {
 
 // StartStats prints the stats every statsInterval
 //
-// It returns a channel which should be closed to stop the stats.
-func StartStats() chan struct{} {
+// It returns a func which should be called to stop the stats.
+func StartStats() func() {
+	if *statsInterval <= 0 {
+		return func() {}
+	}
 	stopStats := make(chan struct{})
-	if *statsInterval > 0 {
-		go func() {
-			ticker := time.NewTicker(*statsInterval)
-			for {
-				select {
-				case <-ticker.C:
-					accounting.Stats.Log()
-				case <-stopStats:
-					ticker.Stop()
-					return
-				}
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		ticker := time.NewTicker(*statsInterval)
+		for {
+			select {
+			case <-ticker.C:
+				accounting.Stats.Log()
+			case <-stopStats:
+				ticker.Stop()
+				return
 			}
-		}()
+		}
+	}()
+	return func() {
+		close(stopStats)
+		wg.Wait()
 	}
-	return stopStats
 }
 
 // initConfig is run by cobra after initialising the flags
@@ -420,8 +352,11 @@ func initConfig() {
 	// Write the args for debug purposes
 	fs.Debugf("rclone", "Version %q starting with parameters %q", fs.Version, os.Args)
 
-	// Start the remote control if configured
-	rc.Start(&rcflags.Opt)
+	// Start the remote control server if configured
+	_, err = rcserver.Start(&rcflags.Opt)
+	if err != nil {
+		log.Fatalf("Failed to start remote control: %v", err)
+	}
 
 	// Setup CPU profiling if desired
 	if *cpuProfile != "" {
@@ -499,8 +434,11 @@ func resolveExitCode(err error) {
 	}
 }
 
+var backendFlags map[string]struct{}
+
 // AddBackendFlags creates flags for all the backend options
 func AddBackendFlags() {
+	backendFlags = map[string]struct{}{}
 	for _, fsInfo := range fs.Registry {
 		done := map[string]struct{}{}
 		for i := range fsInfo.Options {
@@ -511,10 +449,7 @@ func AddBackendFlags() {
 			}
 			done[opt.Name] = struct{}{}
 			// Make a flag from each option
-			name := strings.Replace(opt.Name, "_", "-", -1) // convert snake_case to kebab-case
-			if !opt.NoPrefix {
-				name = fsInfo.Prefix + "-" + name
-			}
+			name := opt.FlagName(fsInfo.Prefix)
 			found := pflag.CommandLine.Lookup(name) != nil
 			if !found {
 				// Take first line of help only
@@ -531,6 +466,7 @@ func AddBackendFlags() {
 				if opt.Hide&fs.OptionHideCommandLine != 0 {
 					flag.Hidden = true
 				}
+				backendFlags[name] = struct{}{}
 			} else {
 				fs.Errorf(nil, "Not adding duplicate flag --%s", name)
 			}
@@ -541,6 +477,7 @@ func AddBackendFlags() {
 
 // Main runs rclone interpreting flags and commands out of os.Args
 func Main() {
+	setupRootCommand(Root)
 	AddBackendFlags()
 	if err := Root.Execute(); err != nil {
 		log.Fatalf("Fatal error: %v", err)

cmd/cmount/fs.go

@@ -8,11 +8,11 @@ import (
 	"io"
 	"os"
 	"path"
-	"runtime"
 	"sync"
 	"time"
 
 	"github.com/billziss-gh/cgofuse/fuse"
+	"github.com/ncw/rclone/cmd/mountlib"
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/fs/log"
 	"github.com/ncw/rclone/vfs"
@@ -263,10 +263,7 @@ func (fsys *FS) Releasedir(path string, fh uint64) (errc int) {
 func (fsys *FS) Statfs(path string, stat *fuse.Statfs_t) (errc int) {
 	defer log.Trace(path, "")("stat=%+v, errc=%d", stat, &errc)
 	const blockSize = 4096
-	fsBlocks := uint64(1 << 50)
-	if runtime.GOOS == "windows" {
-		fsBlocks = (1 << 43) - 1
-	}
+	const fsBlocks = (1 << 50) / blockSize
 	stat.Blocks = fsBlocks  // Total data blocks in file system.
 	stat.Bfree = fsBlocks   // Free blocks in file system.
 	stat.Bavail = fsBlocks  // Free blocks in file system if you're not root.
@@ -285,6 +282,9 @@ func (fsys *FS) Statfs(path string, stat *fuse.Statfs_t) (errc int) {
 	if free >= 0 {
 		stat.Bavail = uint64(free) / blockSize
 	}
+	mountlib.ClipBlocks(&stat.Blocks)
+	mountlib.ClipBlocks(&stat.Bfree)
+	mountlib.ClipBlocks(&stat.Bavail)
 	return 0
 }
 

cmd/config/config.go

@@ -1,8 +1,11 @@
 package config
 
 import (
+	"errors"
+
 	"github.com/ncw/rclone/cmd"
 	"github.com/ncw/rclone/fs/config"
+	"github.com/ncw/rclone/fs/rc"
 	"github.com/spf13/cobra"
 )
 
@@ -93,7 +96,16 @@ you would do:
 `,
 	RunE: func(command *cobra.Command, args []string) error {
 		cmd.CheckArgs(2, 256, command, args)
-		return config.CreateRemote(args[0], args[1], args[2:])
+		in, err := argsToMap(args[2:])
+		if err != nil {
+			return err
+		}
+		err = config.CreateRemote(args[0], args[1], in)
+		if err != nil {
+			return err
+		}
+		config.ShowRemote(args[0])
+		return nil
 	},
 }
 
@@ -110,7 +122,16 @@ For example to update the env_auth field of a remote of name myremote you would
 `,
 	RunE: func(command *cobra.Command, args []string) error {
 		cmd.CheckArgs(3, 256, command, args)
-		return config.UpdateRemote(args[0], args[1:])
+		in, err := argsToMap(args[1:])
+		if err != nil {
+			return err
+		}
+		err = config.UpdateRemote(args[0], in)
+		if err != nil {
+			return err
+		}
+		config.ShowRemote(args[0])
+		return nil
 	},
 }
 
@@ -136,6 +157,29 @@ For example to set password of a remote of name myremote you would do:
 `,
 	RunE: func(command *cobra.Command, args []string) error {
 		cmd.CheckArgs(3, 256, command, args)
-		return config.PasswordRemote(args[0], args[1:])
+		in, err := argsToMap(args[1:])
+		if err != nil {
+			return err
+		}
+		err = config.PasswordRemote(args[0], in)
+		if err != nil {
+			return err
+		}
+		config.ShowRemote(args[0])
+		return nil
 	},
 }
+
+// This takes a list of arguments in key value key value form and
+// converts it into a map
+func argsToMap(args []string) (out rc.Params, err error) {
+	if len(args)%2 != 0 {
+		return nil, errors.New("found key without value")
+	}
+	out = rc.Params{}
+	// Set the config
+	for i := 0; i < len(args); i += 2 {
+		out[args[i]] = args[i+1]
+	}
+	return out, nil
+}

cmd/copy/copy.go

@@ -50,6 +50,8 @@ If you are familiar with ` + "`rsync`" + `, rclone always works as if you had
 written a trailing / - meaning "copy the contents of this directory".
 This applies to all commands and whether you are talking about the
 source or destination.
+
+**Note**: Use the ` + "`-P`" + `/` + "`--progress`" + ` flag to view real-time transfer statistics
 `,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(2, 2, command, args)

cmd/copyto/copyto.go

@@ -40,6 +40,8 @@ This will:
 This doesn't transfer unchanged files, testing by size and
 modification time or MD5SUM.  It doesn't delete files from the
 destination.
+
+**Note**: Use the ` + "`-P`" + `/` + "`--progress`" + ` flag to view real-time transfer statistics
 `,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(2, 2, command, args)

cmd/copyurl/copyurl.go

@@ -0,0 +1,29 @@
+package copyurl
+
+import (
+	"github.com/ncw/rclone/cmd"
+	"github.com/ncw/rclone/fs/operations"
+	"github.com/spf13/cobra"
+)
+
+func init() {
+	cmd.Root.AddCommand(commandDefintion)
+}
+
+var commandDefintion = &cobra.Command{
+	Use:   "copyurl https://example.com dest:path",
+	Short: `Copy url content to dest.`,
+	Long: `
+Download urls content and copy it to destination 
+without saving it in tmp storage.
+`,
+	Run: func(command *cobra.Command, args []string) {
+		cmd.CheckArgs(2, 2, command, args)
+		fsdst, dstFileName := cmd.NewFsDstFile(args[1:])
+
+		cmd.Run(true, true, command, func() error {
+			_, err := operations.CopyURL(fsdst, dstFileName, args[0])
+			return err
+		})
+	},
+}

cmd/delete/delete.go

@@ -14,9 +14,13 @@ var commandDefintion = &cobra.Command{
 	Use:   "delete remote:path",
 	Short: `Remove the contents of path.`,
 	Long: `
-Remove the contents of path.  Unlike ` + "`" + `purge` + "`" + ` it obeys include/exclude
+Remove the files in path.  Unlike ` + "`" + `purge` + "`" + ` it obeys include/exclude
 filters so can be used to selectively delete files.
 
+` + "`" + `rclone delete` + "`" + ` only deletes objects but leaves the directory structure
+alone. If you want to delete a directory and all of its contents use
+` + "`" + `rclone purge` + "`" + `
+
 Eg delete all files bigger than 100MBytes
 
 Check what would be deleted first (use either)

cmd/deletefile/deletefile.go

@@ -3,7 +3,6 @@ package deletefile
 import (
 	"github.com/ncw/rclone/cmd"
 	"github.com/ncw/rclone/fs/operations"
-
 	"github.com/pkg/errors"
 	"github.com/spf13/cobra"
 )

cmd/help.go

@@ -0,0 +1,250 @@
+package cmd
+
+import (
+	"fmt"
+	"log"
+	"os"
+	"regexp"
+	"strings"
+
+	"github.com/ncw/rclone/fs"
+	"github.com/ncw/rclone/fs/config/configflags"
+	"github.com/ncw/rclone/fs/filter/filterflags"
+	"github.com/ncw/rclone/fs/rc/rcflags"
+	"github.com/ncw/rclone/lib/atexit"
+	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
+)
+
+// Root is the main rclone command
+var Root = &cobra.Command{
+	Use:   "rclone",
+	Short: "Show help for rclone commands, flags and backends.",
+	Long: `
+Rclone syncs files to and from cloud storage providers as well as
+mounting them, listing them in lots of different ways.
+
+See the home page (https://rclone.org/) for installation, usage,
+documentation, changelog and configuration walkthroughs.
+
+`,
+	PersistentPostRun: func(cmd *cobra.Command, args []string) {
+		fs.Debugf("rclone", "Version %q finishing with parameters %q", fs.Version, os.Args)
+		atexit.Run()
+	},
+}
+
+// root help command
+var helpCommand = &cobra.Command{
+	Use:   "help",
+	Short: Root.Short,
+	Long:  Root.Long,
+	Run: func(command *cobra.Command, args []string) {
+		Root.SetOutput(os.Stdout)
+		_ = Root.Usage()
+	},
+}
+
+// to filter the flags with
+var flagsRe *regexp.Regexp
+
+// Show the flags
+var helpFlags = &cobra.Command{
+	Use:   "flags [<regexp to match>]",
+	Short: "Show the global flags for rclone",
+	Run: func(command *cobra.Command, args []string) {
+		if len(args) > 0 {
+			re, err := regexp.Compile(args[0])
+			if err != nil {
+				log.Fatalf("Failed to compile flags regexp: %v", err)
+			}
+			flagsRe = re
+		}
+		Root.SetOutput(os.Stdout)
+		_ = command.Usage()
+	},
+}
+
+// Show the backends
+var helpBackends = &cobra.Command{
+	Use:   "backends",
+	Short: "List the backends available",
+	Run: func(command *cobra.Command, args []string) {
+		showBackends()
+	},
+}
+
+// Show a single backend
+var helpBackend = &cobra.Command{
+	Use:   "backend <name>",
+	Short: "List full info about a backend",
+	Run: func(command *cobra.Command, args []string) {
+		if len(args) == 0 {
+			Root.SetOutput(os.Stdout)
+			_ = command.Usage()
+			return
+		}
+		showBackend(args[0])
+	},
+}
+
+// runRoot implements the main rclone command with no subcommands
+func runRoot(cmd *cobra.Command, args []string) {
+	if version {
+		ShowVersion()
+		resolveExitCode(nil)
+	} else {
+		_ = cmd.Usage()
+		if len(args) > 0 {
+			_, _ = fmt.Fprintf(os.Stderr, "Command not found.\n")
+		}
+		resolveExitCode(errorCommandNotFound)
+	}
+}
+
+// setupRootCommand sets default usage, help, and error handling for
+// the root command.
+//
+// Helpful example: http://rtfcode.com/xref/moby-17.03.2-ce/cli/cobra.go
+func setupRootCommand(rootCmd *cobra.Command) {
+	// Add global flags
+	configflags.AddFlags(pflag.CommandLine)
+	filterflags.AddFlags(pflag.CommandLine)
+	rcflags.AddFlags(pflag.CommandLine)
+
+	Root.Run = runRoot
+	Root.Flags().BoolVarP(&version, "version", "V", false, "Print the version number")
+
+	cobra.AddTemplateFunc("showGlobalFlags", func(cmd *cobra.Command) bool {
+		return cmd.CalledAs() == "flags"
+	})
+	cobra.AddTemplateFunc("showCommands", func(cmd *cobra.Command) bool {
+		return cmd.CalledAs() != "flags"
+	})
+	cobra.AddTemplateFunc("showLocalFlags", func(cmd *cobra.Command) bool {
+		// Don't show local flags (which are the global ones on the root) on "rclone" and
+		// "rclone help" (which shows the global help)
+		return cmd.CalledAs() != "rclone" && cmd.CalledAs() != ""
+	})
+	cobra.AddTemplateFunc("backendFlags", func(cmd *cobra.Command, include bool) *pflag.FlagSet {
+		backendFlagSet := pflag.NewFlagSet("Backend Flags", pflag.ExitOnError)
+		cmd.InheritedFlags().VisitAll(func(flag *pflag.Flag) {
+			matched := flagsRe == nil || flagsRe.MatchString(flag.Name)
+			if _, ok := backendFlags[flag.Name]; matched && ok == include {
+				backendFlagSet.AddFlag(flag)
+			}
+		})
+		return backendFlagSet
+	})
+	rootCmd.SetUsageTemplate(usageTemplate)
+	// rootCmd.SetHelpTemplate(helpTemplate)
+	// rootCmd.SetFlagErrorFunc(FlagErrorFunc)
+	rootCmd.SetHelpCommand(helpCommand)
+	// rootCmd.PersistentFlags().BoolP("help", "h", false, "Print usage")
+	// rootCmd.PersistentFlags().MarkShorthandDeprecated("help", "please use --help")
+
+	rootCmd.AddCommand(helpCommand)
+	helpCommand.AddCommand(helpFlags)
+	helpCommand.AddCommand(helpBackends)
+	helpCommand.AddCommand(helpBackend)
+
+	cobra.OnInitialize(initConfig)
+
+}
+
+var usageTemplate = `Usage:{{if .Runnable}}
+  {{.UseLine}}{{end}}{{if .HasAvailableSubCommands}}
+  {{.CommandPath}} [command]{{end}}{{if gt (len .Aliases) 0}}
+
+Aliases:
+  {{.NameAndAliases}}{{end}}{{if .HasExample}}
+
+Examples:
+{{.Example}}{{end}}{{if and (showCommands .) .HasAvailableSubCommands}}
+
+Available Commands:{{range .Commands}}{{if (or .IsAvailableCommand (eq .Name "help"))}}
+  {{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{if and (showLocalFlags .) .HasAvailableLocalFlags}}
+
+Flags:
+{{.LocalFlags.FlagUsages | trimTrailingWhitespaces}}{{end}}{{if and (showGlobalFlags .) .HasAvailableInheritedFlags}}
+
+Global Flags:
+{{(backendFlags . false).FlagUsages | trimTrailingWhitespaces}}
+
+Backend Flags:
+{{(backendFlags . true).FlagUsages | trimTrailingWhitespaces}}{{end}}{{if .HasHelpSubCommands}}
+
+Additional help topics:{{range .Commands}}{{if .IsAdditionalHelpTopicCommand}}
+  {{rpad .CommandPath .CommandPathPadding}} {{.Short}}{{end}}{{end}}{{end}}
+
+Use "rclone [command] --help" for more information about a command.
+Use "rclone help flags" for to see the global flags.
+Use "rclone help backends" for a list of supported services.
+`
+
+// show all the backends
+func showBackends() {
+	fmt.Printf("All rclone backends:\n\n")
+	for _, backend := range fs.Registry {
+		fmt.Printf("  %-12s %s\n", backend.Prefix, backend.Description)
+	}
+	fmt.Printf("\nTo see more info about a particular backend use:\n")
+	fmt.Printf("  rclone help backend <name>\n")
+}
+
+func quoteString(v interface{}) string {
+	switch v.(type) {
+	case string:
+		return fmt.Sprintf("%q", v)
+	}
+	return fmt.Sprint(v)
+}
+
+// show a single backend
+func showBackend(name string) {
+	backend, err := fs.Find(name)
+	if err != nil {
+		log.Fatal(err)
+	}
+	var standardOptions, advancedOptions fs.Options
+	done := map[string]struct{}{}
+	for _, opt := range backend.Options {
+		// Skip if done already (eg with Provider options)
+		if _, doneAlready := done[opt.Name]; doneAlready {
+			continue
+		}
+		if opt.Advanced {
+			advancedOptions = append(advancedOptions, opt)
+		} else {
+			standardOptions = append(standardOptions, opt)
+		}
+	}
+	optionsType := "standard"
+	for _, opts := range []fs.Options{standardOptions, advancedOptions} {
+		if len(opts) == 0 {
+			continue
+		}
+		fmt.Printf("### %s Options\n\n", strings.Title(optionsType))
+		fmt.Printf("Here are the %s options specific to %s (%s).\n\n", optionsType, backend.Name, backend.Description)
+		optionsType = "advanced"
+		for _, opt := range opts {
+			done[opt.Name] = struct{}{}
+			fmt.Printf("#### --%s\n\n", opt.FlagName(backend.Prefix))
+			fmt.Printf("%s\n\n", opt.Help)
+			fmt.Printf("- Config:      %s\n", opt.Name)
+			fmt.Printf("- Env Var:     %s\n", opt.EnvVarName(backend.Prefix))
+			fmt.Printf("- Type:        %s\n", opt.Type())
+			fmt.Printf("- Default:     %s\n", quoteString(opt.GetValue()))
+			if len(opt.Examples) > 0 {
+				fmt.Printf("- Examples:\n")
+				for _, ex := range opt.Examples {
+					fmt.Printf("    - %s\n", quoteString(ex.Value))
+					for _, line := range strings.Split(ex.Help, "\n") {
+						fmt.Printf("        - %s\n", line)
+					}
+				}
+			}
+			fmt.Printf("\n")
+		}
+	}
+}

cmd/link/link.go

@@ -34,7 +34,9 @@ without account.
 			if err != nil {
 				return err
 			}
-			fmt.Println(link)
+			if link != "" {
+				fmt.Println(link)
+			}
 			return nil
 		})
 	},