Compare commits

...
90 Commits
Author SHA1 Message Date
Nick Craig-Wood ad38432cc3 config: experiment with a text UI for config 2019-09-25 10:17:07 +01:00
Nick Craig-Wood 956fc5062b vendor: add github.com/manifoldco/promptui and dependencies 2019-09-25 10:12:15 +01:00
Nick Craig-Wood 28cc2009d4 Add Anthony Rusdi to contributors 2019-09-21 14:39:03 +01:00
Nick Craig-Wood dd4fe9ff60 Add David to contributors 2019-09-21 14:39:03 +01:00
Anthony Rusdi 899f285319 s3: fix signature v2_auth headers
When used with v2_auth = true, PresignRequest doesn't return
signed headers, so remote dest authentication would be fail.
This commit copying back HTTPRequest.Header to headers.

Tested with RiakCS v2.1.0.

Signed-off-by: Anthony Rusdi <33247310+antrusd@users.noreply.github.com>
2019-09-21 14:38:51 +01:00
David 4788545b05 box: add options to get access token via JWT auth 2019-09-20 17:15:16 +01:00
David 1934426789 jwtutil: functionality to get an access token via JWT authentication 2019-09-20 17:15:16 +01:00
David 643192b347 vendor: add pkcs8 helpers for decrypting encrypted private keys 2019-09-20 17:15:16 +01:00
Nick Craig-Wood 1031bcfc5a build: remove azure pipelines build 2019-09-20 16:08:18 +01:00
Nick Craig-Wood ce00c0a0d9 build: build rclone with github actions 2019-09-20 16:08:18 +01:00
Nick Craig-Wood 1164eed2af lib/pacer: make tests more reliable 2019-09-20 16:07:55 +01:00
Nick Craig-Wood 557edecd40 log: add Stack() function for debugging who calls what 2019-09-20 11:53:08 +01:00
Nick Craig-Wood b242b0a078 lib/cache,rc/jobs: make tests more reliable 2019-09-20 11:53:08 +01:00
Nick Craig-Wood 08b86cc94b mount: skip tests on <= 2 CPUs to avoid lockup in #3154 2019-09-20 11:53:08 +01:00
Nick Craig-Wood 56544bb2fd accounting: fix file handle leak on errors - fixes #3547
In 53a1a0e3ef we introduced a problem where if there was an
error on the file being transferred then the file was re-opened and
the old one wasn't closed.

This was partially fixed in bfbddab46b however this didn't
address the case of the old file being closed.

This is now fixed by
- marking the file as open again in UpdateReader
- moving the stopping the accounting machinery to a new method Done
2019-09-19 16:20:07 +01:00
Matei David 70e043e641 Fixed typo in Docker image doc. 2019-09-19 16:17:57 +01:00
Dan Walters c49a71f438 dlna: move root descriptor xml template to the static assets
Reduce binary size.
2019-09-17 12:52:32 +01:00
Dan Walters 5f07bbf8ce dlna: fake out implementation of X_MS_MediaReceiverRegistrar
Using the same responses as minidlna.

Fixes #3502.
2019-09-17 12:52:02 +01:00
Dan Walters 2f10472df3 dlna: count the number of children in the response to BrowseMetadata 2019-09-17 12:28:20 +01:00
Matei David ab89e93968 Add Matei David to contributors 2019-09-17 10:12:32 +01:00
Matei David 070a8bfcd8 Dockerfile fixes
- ref: https://forum.rclone.org/t/run-docker-container-in-userspace/11734/7
- enable userspace operation
- enable Docker userspace mount exposed to the host
- add more Docker image usage documentation
2019-09-17 10:12:32 +01:00
Ivan Andreev 8fe87c8157 mailru: skip extra http request if data fits in hash 2019-09-17 10:04:51 +01:00
Ivan Andreev 8fb44a822d mailru: fix rare nil pointer panic 2019-09-17 10:04:51 +01:00
Nick Craig-Wood 3cff258577 sftp: fix --sftp-ask-password trying to contact the ssh agent
See: https://forum.rclone.org/t/rclone-command-line/11766
2019-09-16 11:16:27 +01:00
Nick Craig-Wood 66347aff2a fstest: calculate hashes for uploaded test files to fix minio integration tests
Before this change we didn't calculate any hashes for test files
created in the Run framework.

This means that files were uploaded to S3 without a `Content-MD5`
header.  This in turn caused minio to disengage `--compat` mode which
in turn caused the `TestSyncAfterChangingModtimeOnlyWithNoUpdateModTime`
test to fail in `fs/sync`.

After this change we supply all hashes supported by the destination Fs
on the upload object.

This means that the `Content-MD5` is set and minio engages `--compat`
mode to fix the problem.  Using `--compat` on the command line also
fixes the problem.

This much better replicates how objects are actually uploaded with
operations.Copy so should improve the integration tests.
2019-09-16 10:59:01 +01:00
Nick Craig-Wood b8b12a4000 Start v1.49.3-DEV development 2019-09-15 18:10:08 +01:00
Dan Walters 8c038326b9 dlna: correct output for ContentDirectoryService#Browse with BrowseMetadata
We were marshalling the "cds object" instead of the "upnp object".

Fixes #3253  (I think)
2019-09-15 16:30:39 +01:00
pataquets fd4b25932c Contrib: Add sample WebDAV server Docker Compose manifest. 2019-09-15 16:06:54 +01:00
pataquets 4374fd1df1 Contrib: Add sample DLNA server Docker Compose manifest. 2019-09-15 16:06:54 +01:00
Nick Craig-Wood b6065561cf test_all: add ignores for tests which will never pass
- s3 backends which don't support SetTier
- mega which makes a duplicate for TestDirRename
2019-09-15 13:16:15 +01:00
Nick Craig-Wood ef7bfd3f03 fs: Make prefix free backend config read prefix free env var also
Before this change you could only configure the local backend flags
which don't have the local prefix (eg `--copy-links`) with
`RCLONE_LOCAL_COPY_LINKS`.

This change makes `RCLONE_COPY_LINKS` valid too which is much more
logical for the users.

Fixes #3534
2019-09-14 18:26:07 +01:00
Nick Craig-Wood ae2edc3b5b help: add short options to backend documentation also 2019-09-14 18:24:05 +01:00
Nick Craig-Wood 0baafb158f mount: allow files of unkown size to be read properly
Before this change, files of unknown size (eg Google Docs) would
appear in file listings with 0 size and would only allow 0 bytes to be
read.

This change sets the direct_io flag in the FUSE return which bypasses
the cache for these files.  This means that they can be read properly.

This is compatible with some, but not all applications.
2019-09-14 13:22:33 +01:00
Nick Craig-Wood ba121eddf0 vfs: make objects of unknown size readable through the VFS
These objects (eg Google Docs) appear with 0 length in the VFS.

Before this change, these only read 0 bytes.

After this change, even though the size appears to be 0, the objects
can be read to the end.  If the objects are read to the end then the
size on the handle will be updated.
2019-09-14 13:09:07 +01:00
Nick Craig-Wood 2e80e035c9 fstest/mockobject: add UnknownSize() method to make Size() return -1 2019-09-14 13:07:01 +01:00
Nick Craig-Wood ea9b6087cf fstest/mockfs: allow fs.Objects to be added to the root 2019-09-14 13:05:36 +01:00
Nick Craig-Wood 6959c997e2 config: remove error: can't use --size-only and --ignore-size together.
As sometimes they are useful together, for example when the remote
changes the sizes of the uploaded files.

See: https://forum.rclone.org/t/files-upload-will-be-auto-compress-how-do-i-sync-a-file-to-remote/10578
2019-09-14 10:13:44 +01:00
Nick Craig-Wood 25786cafd3 s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier
- Read the storage class for each object
- Implement SetTier/GetTier
- Check the storage class on the **object** before using SetModTime

This updates the fix in 1a2fb52 so that SetModTime works when you are
using objects which have been migrated to GLACIER but you aren't using
GLACIER as a storage class.

Fixes #3522
2019-09-14 09:18:55 +01:00
Nick Craig-Wood 23dc313fa5 azureblob: add missing type assertions for GetTier/SetTier 2019-09-14 09:18:55 +01:00
Nick Craig-Wood 1a16849df0 http: fix race introduced in 7982aaf151 2019-09-14 08:48:13 +01:00
Nick Craig-Wood 3b68340eac http: add --http-no-head to stop rclone doing HEAD in listings #3523 2019-09-14 00:17:39 +01:00
Nick Craig-Wood 7982aaf151 http: HEAD directory entries in parallel to speedup #3523 2019-09-14 00:16:44 +01:00
Nick Craig-Wood 7b29ed8ec1 Add Lars Lehtonen to contributors 2019-09-13 23:50:54 +01:00
Lars Lehtonen c93e0ff8ee rest: fix missing error check 2019-09-13 23:50:39 +01:00
Nick Craig-Wood 3b91fb6a2f Add David Baumgold to contributors 2019-09-13 18:39:36 +01:00
David Baumgold 7d8c15c030 docs: GitHub has a capital H 2019-09-13 18:39:23 +01:00
Nick Craig-Wood bfbddab46b fs/accounting: Fix "file already closed" on transfer retries
This was caused by the recent reworking of the accounting interface.
The Transfer object was recycling the Accounting object without
resetting the stream.

See: https://forum.rclone.org/t/error-file-already-closed/11469/
See: https://forum.rclone.org/t/rclone-b2-sync-post-error-method-not-supported/11718/
2019-09-13 18:35:02 +01:00
Nick Craig-Wood e09a4ff019 cmd: Make --progress work in git bash on Windows - fixes #3531
This detects the presence of a VT100 terminal by using the TERM
environment variable and switches to using VT100 codes directly under
windows if it is found.

This makes --progress work correctly with git bash.
2019-09-13 15:24:47 +01:00
Nick Craig-Wood 48e23d8c85 build: fix circleci build to use billziss/xgo-cgofuse container
Use built rclone to do upload
2019-09-12 21:59:50 +01:00
Aleksandar Jankovic 934440a9df accounting: fix total duration calculation
Fixes: #3498
2019-09-12 10:29:40 +01:00
Nick Craig-Wood 29b4f211ab gcs: add context to SDK calls #3257 2019-09-09 23:27:07 +01:00
Nick Craig-Wood bd863f8868 drive: add context to SDK calls #3257 2019-09-09 23:27:07 +01:00
Nick Craig-Wood 66c23723e3 Add context to all http.NewRequest #3257
When we drop support for go1.12 we can use http.NewRequestWithContext
2019-09-09 23:27:07 +01:00
Nick Craig-Wood 58a531a203 rest: add context propagation to rest library #3257
This fixes up the calling and propagates the contexts for the backends
which use lib/rest.
2019-09-09 23:27:07 +01:00
Ivan Andreev ba1daea072 mailru: backend for mail.ru 2019-09-09 21:56:16 +01:00
Ivan Andreev bdcd0b4c64 Add mailru hash (mrhash) 2019-09-09 21:34:15 +01:00
Nick Craig-Wood 94eb9a4014 Start v1.49.2-DEV development 2019-09-08 17:36:21 +01:00
Nick Craig-Wood e028c006fc test_all: write index.json and add branch, commit and Go version to report 2019-09-08 11:35:56 +01:00
Nick Craig-Wood 3f3f038b73 build: make sure we add version info to test_all build 2019-09-08 11:35:36 +01:00
calisro 2298834e83 docs: spell out google photos download limitations 2019-09-07 11:47:53 +01:00
Nick Craig-Wood 07dfb3aa11 bin: convert python scripts to python3 2019-09-06 22:55:28 +01:00
Danil Semelenov 1382dba3c8 cmd: make autocomplete compatible with bash's posix mode #3489 2019-09-06 13:11:08 +01:00
Nick Craig-Wood f1347139fa config: check config names more carefully and report errors - fixes #3506
Before this change it was possible to make a remote with an invalid
name in the config file, either manually or with `rclone config
create` (but not with `rclone config`).

When this remote was used, because it was invalid, rclone would
presume this remote name was a local directory for a very suprising
user experience!

This change checks remote names more carefully and returns errors
- when the user tries to use an invalid remote name on the command line
- when an invalid remote name is used in `rclone config create/update/password`
- when the user tries to enter an invalid remote name in `rclone config`

This does not prevent the user entering a remote name with invalid
characters in the config manually, but such a remote will fail
immediately when it is used on the command line.
2019-09-06 12:07:09 +01:00
Nick Craig-Wood 27a730ef8f docs: Update changelog and RELEASE.md from v1.49.1 release 2019-09-06 12:07:09 +01:00
Ivan Andreev d0c6e5cf5a vfs: skip TestCaseSensitivity on case insensitive backends 2019-09-06 10:44:59 +01:00
Nick Craig-Wood cf9b973fe4 accounting: fix locking in Transfer to avoid deadlock with --progress
Before this change, using -P occasionally deadlocked on the Transfer
mutex when Transfer.Done() was called with a non nil error and the
StatsInfo mutex since they mutually call each other.

This was fixed by making sure that the Transfer mutex is always
released before calling any StatsInfo methods.

This improves on: 6f87267b34

Fixes #3505
2019-09-06 10:00:44 +01:00
Nick Craig-Wood ffa1dac10b build: apply gofmt from go1.13 to change case of number literals 2019-09-05 13:59:06 +01:00
Nick Craig-Wood 7b0966880e Add Ivan Andreev to contributors 2019-09-04 21:32:16 +01:00
Ivan Andreev 1c4e33d4ad vfs: add flag --vfs-case-insensitive for windows/macOS mounts
rclone mount when run on Windows & macOS will now default to `--vfs-case-insensitive`.
This means that
2019-09-04 21:30:48 +01:00
Nick Craig-Wood 530ba66d35 operations: fix -u/--update with google photos / files of unknown size
Before this change if -u/--update was in effect we compared the size
of the files to see if the transfer should go ahead.  This was
comparing -1 with an actual size so the transfer always proceeded.

After this change we use the existing `sizeDiffers` function which
does the correct comparison with -1 for files of unknown length.

See: https://forum.rclone.org/t/sync-with-google-photos-to-local-drive-will-result-in-recoping/11605
2019-09-04 17:31:17 +01:00
Danil Semelenov b3db38ae31 Disable __rclone_custom_func if posix mode is on
A workaround for #3489. Code in `__rclone_custom_func` relies on process substitutions `<(...)` to preserve changes of variables within `while` bodies, which is not supported in the posix mode.
2019-09-04 14:48:10 +01:00
Danil Semelenov c0d1869204 Fix 'compopt: command not found' on autocomplete on macOS
As reported in #3489.
2019-09-04 14:47:26 +01:00
Nick Craig-Wood 89b6d89077 build: drop support for go1.9 2019-09-04 10:23:48 +01:00
Nick Craig-Wood ef7b001626 build: update to use go1.13 for the build 2019-09-04 10:23:48 +01:00
Nick Craig-Wood f97a3e853e Add Alfonso Montero to contributors 2019-09-03 17:26:13 +01:00
Denis b71ac141cc copyurl: add --auto-filename flag for using file name from url in destination path (#3451) 2019-09-03 17:25:19 +01:00
Nick Craig-Wood 5932acfee3 rc: fix docs for config/create /update /password 2019-09-03 08:34:15 +01:00
Alfonso Montero e2ce687f93 README.md: Add Docker pulls badge 2019-09-02 15:14:45 +01:00
Nick Craig-Wood a3fb460c6b docs: add info on how to build and use the docker images 2019-09-02 14:30:11 +01:00
Nick Craig-Wood 8d296d0e1d Start v1.49.1-DEV development 2019-09-02 13:10:47 +01:00
Nick Craig-Wood 20a57aaccb gcs: fix need for elevated permissions on SetModTime - fixes #3493
Before this change we used PATCH on the object to update the metadata.

Apparently this requires the "full_control" scope which Google were
unhappy with in their oauth review.

This changes it to update the metadata by copying the object ontop of
itself (which is the way s3 works).  This can be done with normal
permissions.
2019-09-02 09:26:33 +01:00
Nick Craig-Wood 50a4ed8fc4 operations: fix accounting for server side copies
See: https://forum.rclone.org/t/b2-server-side-copy-doesnt-show-cumulative-progress/11154
2019-09-02 09:26:33 +01:00
Cnly e2b5ed6c7a docs: fix template argument for mktemp in install.sh 2019-09-02 13:01:45 +08:00
Alfonso Montero 16e7da2cb5 Add Docker workflow support #3460
* Use a multi-stage build to reduce final image size.
* Run 'quicktest' make target before building.
* Built binary won't run on Alpine unless statically linked.
2019-08-29 11:04:57 +01:00
yparitcher 52df19ad34 allow usage of -short in the testing framework 2019-08-29 09:53:23 +01:00
Nick Craig-Wood 693112d57e config: Fix generated passwords being stored as empty password - Fixes #3492 2019-08-28 12:11:03 +01:00
Nick Craig-Wood 0edbc9578d googlephotos,onedrive: fix crash on error response - fixes #3491
This fixes a crash on the google photos backend when an error is
returned from the rest.Call function.

This turned out to be a mis-understanding of the rest docs so
- improved rest.Call docs
- fixed mis-understanding in google photos backend
- fixed similar mis-understading in onedrive backend
2019-08-28 12:11:03 +01:00
Chaitanya 7211c2dca7 rcd: Added missing parameter for web-gui info logs. 2019-08-27 17:21:21 +01:00
Nick Craig-Wood af192d2507 vendor: update all dependencies 2019-08-26 18:00:17 +01:00
Nick Craig-Wood d1a39dcc4b Start v1.49.0-DEV development 2019-08-26 17:44:09 +01:00
598 changed files with 87094 additions and 3050 deletions

No files matched your search

+1
View File
@@ -40,6 +40,7 @@ build_script:
test_script:
- make GOTAGS=cmount quicktest
- make GOTAGS=cmount racequicktest
artifacts:
- path: rclone.exe
+11 -18
View File
@@ -14,37 +14,30 @@ jobs:
- run:
name: Cross-compile rclone
command: |
docker pull rclone/xgo-cgofuse
docker pull billziss/xgo-cgofuse
go get -v github.com/karalabe/xgo
xgo \
--image=rclone/xgo-cgofuse \
--targets=darwin/386,darwin/amd64,linux/386,linux/amd64,windows/386,windows/amd64 \
-image=billziss/xgo-cgofuse \
-targets=darwin/386,darwin/amd64,linux/386,linux/amd64,windows/386,windows/amd64 \
-tags cmount \
-dest build \
.
xgo \
--targets=android/*,ios/* \
-image=billziss/xgo-cgofuse \
-targets=android/*,ios/* \
-dest build \
.
- run:
name: Prepare artifacts
command: |
mkdir -p /tmp/rclone.dist
cp -R rclone-* /tmp/rclone.dist
mkdir build
cp -R rclone-* build/
- run:
name: Build rclone
command: |
go version
go build
docker pull golang
docker run --rm -v "$PWD":/usr/src/rclone -w /usr/src/rclone golang go build -mod=vendor -v
- run:
name: Upload artifacts
command: |
if [[ $CIRCLE_PULL_REQUEST != "" ]]; then
make circleci_upload
fi
make circleci_upload
- store_artifacts:
path: /tmp/rclone.dist
path: build
+206
View File
@@ -0,0 +1,206 @@
---
# Github Actions build for rclone
# -*- compile-command: "yamllint -f parsable build.yml" -*-
name: build
# Trigger the workflow on push or pull request
on:
push:
branches:
- '*'
tags:
- '*'
pull_request:
jobs:
build:
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
job_name: ['linux', 'mac', 'windows_amd64', 'windows_386', 'other_os', 'modules_race', 'go1.10', 'go1.11', 'go1.12']
include:
- job_name: linux
os: ubuntu-latest
go: '1.13.x'
modules: off
gotags: cmount
build_flags: '-include "^linux/"'
check: true
quicktest: true
deploy: true
- job_name: mac
os: macOS-latest
go: '1.13.x'
modules: off
gotags: '' # cmount doesn't work on osx travis for some reason
build_flags: '-include "^darwin/amd64" -cgo'
quicktest: true
racequicktest: true
deploy: true
- job_name: windows_amd64
os: windows-latest
go: '1.13.x'
modules: off
gotags: cmount
build_flags: '-include "^windows/amd64" -cgo'
quicktest: true
racequicktest: true
deploy: true
- job_name: windows_386
os: windows-latest
go: '1.13.x'
modules: off
gotags: cmount
goarch: '386'
cgo: '1'
build_flags: '-include "^windows/386" -cgo'
quicktest: true
deploy: true
- job_name: other_os
os: ubuntu-latest
go: '1.13.x'
modules: off
build_flags: '-exclude "^(windows/|darwin/amd64|linux/)"'
compile_all: true
deploy: true
- job_name: modules_race
os: ubuntu-latest
go: '1.13.x'
modules: on
quicktest: true
racequicktest: true
- job_name: go1.10
os: ubuntu-latest
go: '1.10.x'
modules: off
quicktest: true
- job_name: go1.11
os: ubuntu-latest
go: '1.11.x'
modules: off
quicktest: true
- job_name: go1.12
os: ubuntu-latest
go: '1.12.x'
modules: off
quicktest: true
name: ${{ matrix.job_name }}
runs-on: ${{ matrix.os }}
steps:
- name: Checkout
uses: actions/checkout@master
with:
path: ./src/github.com/${{ github.repository }}
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ matrix.go }}
- name: Set environment variables
shell: bash
run: |
echo '::set-env name=GOPATH::${{ runner.workspace }}'
echo '::add-path::${{ runner.workspace }}/bin'
echo '::set-env name=GO111MODULE::${{ matrix.modules }}'
echo '::set-env name=GOTAGS::${{ matrix.gotags }}'
echo '::set-env name=BUILD_FLAGS::${{ matrix.build_flags }}'
if [[ "${{ matrix.goarch }}" != "" ]]; then echo '::set-env name=GOARCH::${{ matrix.goarch }}' ; fi
if [[ "${{ matrix.cgo }}" != "" ]]; then echo '::set-env name=CGO_ENABLED::${{ matrix.cgo }}' ; fi
- name: Install Libraries on Linux
shell: bash
run: |
sudo modprobe fuse
sudo chmod 666 /dev/fuse
sudo chown root:$USER /etc/fuse.conf
sudo apt-get install fuse libfuse-dev rpm pkg-config
if: matrix.os == 'ubuntu-latest'
- name: Install Libraries on macOS
shell: bash
run: |
brew update
brew tap caskroom/cask
brew cask install osxfuse
if: matrix.os == 'macOS-latest'
- name: Install Libraries on Windows
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
choco install -y winfsp zip
Write-Host "::set-env name=CPATH::C:\Program Files\WinFsp\inc\fuse;C:\Program Files (x86)\WinFsp\inc\fuse"
if ($env:GOARCH -eq "386") {
choco install -y mingw --forcex86 --force
Write-Host "::add-path::C:\\ProgramData\\chocolatey\\lib\\mingw\\tools\\install\\mingw32\\bin"
}
# Copy mingw32-make.exe to make.exe so the same command line
# can be used on Windows as on macOS and Linux
$path = (get-command mingw32-make.exe).Path
Copy-Item -Path $path -Destination (Join-Path (Split-Path -Path $path) 'make.exe')
if: matrix.os == 'windows-latest'
- name: Print Go version and environment
shell: bash
run: |
printf "Using go at: $(which go)\n"
printf "Go version: $(go version)\n"
printf "\n\nGo environment:\n\n"
go env
printf "\n\nRclone environment:\n\n"
make vars
printf "\n\nSystem environment:\n\n"
env
- name: Run tests
shell: bash
run: |
make
make quicktest
if: matrix.quicktest
- name: Race test
shell: bash
run: |
make racequicktest
if: matrix.racequicktest
- name: Code quality test
shell: bash
run: |
make build_dep
make check
if: matrix.check
- name: Compile all architectures test
shell: bash
run: |
make
make compile_all
if: matrix.compile_all
- name: Deploy built binaries
shell: bash
run: |
make release_dep
make travis_beta
env:
RCLONE_CONFIG_PASS: ${{ secrets.RCLONE_CONFIG_PASS }}
BETA_SUBDIR: 'github_actions' # FIXME remove when removing travis/appveyor
# working-directory: '$(modulePath)'
if: matrix.deploy && github.head_ref == ''
+7 -7
View File
@@ -50,9 +50,6 @@ matrix:
allow_failures:
- go: tip
include:
- go: 1.9.x
script:
- make quicktest
- go: 1.10.x
script:
- make quicktest
@@ -60,6 +57,9 @@ matrix:
script:
- make quicktest
- go: 1.12.x
script:
- make quicktest
- go: 1.13.x
name: Linux
env:
- GOTAGS=cmount
@@ -69,7 +69,7 @@ matrix:
- make build_dep
- make check
- make quicktest
- go: 1.12.x
- go: 1.13.x
name: Go Modules / Race
env:
- GO111MODULE=on
@@ -77,7 +77,7 @@ matrix:
script:
- make quicktest
- make racequicktest
- go: 1.12.x
- go: 1.13.x
name: Other OS
env:
- DEPLOY=true
@@ -85,7 +85,7 @@ matrix:
script:
- make
- make compile_all
- go: 1.12.x
- go: 1.13.x
name: macOS
os: osx
env:
@@ -101,7 +101,7 @@ matrix:
- make racequicktest
# - os: windows
# name: Windows
# go: 1.12.x
# go: 1.13.x
# env:
# - GOTAGS=cmount
# - CPATH='C:\Program Files (x86)\WinFsp\inc\fuse'
+22
View File
@@ -0,0 +1,22 @@
FROM golang AS builder
COPY . /go/src/github.com/rclone/rclone/
WORKDIR /go/src/github.com/rclone/rclone/
RUN make quicktest
RUN \
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
make
RUN ./rclone version
# Begin final image
FROM alpine:latest
RUN apk --no-cache add ca-certificates fuse
COPY --from=builder /go/src/github.com/rclone/rclone/rclone /usr/local/bin/
ENTRYPOINT [ "rclone" ]
WORKDIR /data
ENV XDG_CONFIG_HOME=/config
+9 -7
View File
@@ -1,5 +1,5 @@
SHELL = bash
BRANCH := $(or $(APPVEYOR_REPO_BRANCH),$(TRAVIS_BRANCH),$(BUILD_SOURCEBRANCHNAME),$(shell git rev-parse --abbrev-ref HEAD))
BRANCH := $(or $(APPVEYOR_REPO_BRANCH),$(TRAVIS_BRANCH),$(BUILD_SOURCEBRANCHNAME),$(lastword $(subst /, ,$(GITHUB_REF))),$(shell git rev-parse --abbrev-ref HEAD))
LAST_TAG := $(shell git describe --tags --abbrev=0)
ifeq ($(BRANCH),$(LAST_TAG))
BRANCH := master
@@ -30,12 +30,15 @@ BUILDTAGS=-tags "$(GOTAGS)"
LINTTAGS=--build-tags "$(GOTAGS)"
endif
.PHONY: rclone vars version
.PHONY: rclone test_all vars version
rclone:
touch fs/version.go
go install -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS)
cp -av `go env GOPATH`/bin/rclone .
go build -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS)
mkdir -p `go env GOPATH`/bin/
cp -av rclone`go env GOEXE` `go env GOPATH`/bin/
test_all:
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
vars:
@echo SHELL="'$(SHELL)'"
@@ -50,8 +53,7 @@ version:
@echo '$(TAG)'
# Full suite of integration tests
test: rclone
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
test: rclone test_all
-test_all 2>&1 | tee test_all.log
@echo "Written logs in test_all.log"
+2
View File
@@ -14,6 +14,7 @@
[![CircleCI](https://circleci.com/gh/rclone/rclone/tree/master.svg?style=svg)](https://circleci.com/gh/rclone/rclone/tree/master)
[![Go Report Card](https://goreportcard.com/badge/github.com/rclone/rclone)](https://goreportcard.com/report/github.com/rclone/rclone)
[![GoDoc](https://godoc.org/github.com/rclone/rclone?status.svg)](https://godoc.org/github.com/rclone/rclone)
[![Docker Pulls](https://img.shields.io/docker/pulls/rclone/rclone)](https://hub.docker.com/r/rclone/rclone)
# Rclone
@@ -40,6 +41,7 @@ Rclone *("rsync for cloud storage")* is a command line program to sync files and
* Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
* IBM COS S3 [:page_facing_up:](https://rclone.org/s3/#ibm-cos-s3)
* Koofr [:page_facing_up:](https://rclone.org/koofr/)
* Mail.ru Cloud [:page_facing_up:](https://rclone.org/mailru/)
* Memset Memstore [:page_facing_up:](https://rclone.org/swift/)
* Mega [:page_facing_up:](https://rclone.org/mega/)
* Microsoft Azure Blob Storage [:page_facing_up:](https://rclone.org/azureblob/)
+59 -20
View File
@@ -1,8 +1,14 @@
Extra required software for making a release
# Release
This file describes how to make the various kinds of releases
## Extra required software for making a release
* [github-release](https://github.com/aktau/github-release) for uploading packages
* pandoc for making the html and man pages
Making a release
## Making a release
* git status - make sure everything is checked in
* Check travis & appveyor builds are green
* make check
@@ -26,8 +32,8 @@ Making a release
* # announce with forum post, twitter post, G+ post
Early in the next release cycle update the vendored dependencies
* Review any pinned packages in go.mod and remove if possible
* GO111MODULE=on go get -u github.com/spf13/cobra@master
* make update
* git status
* git add new files
@@ -48,24 +54,57 @@ Can be fixed with
* GO111MODULE=on go mod vendor
Making a point release. If rclone needs a point release due to some
horrendous bug, then
* git branch v1.XX v1.XX-fixes
## Making a point release
If rclone needs a point release due to some horrendous bug then a
point release is necessary.
First make the release branch. If this is a second point release then
this will be done already.
* BASE_TAG=v1.XX # eg v1.49
* NEW_TAG=${BASE_TAG}.Y # eg v1.49.1
* echo $BASE_TAG $NEW_TAG # v1.49 v1.49.1
* git branch ${BASE_TAG} ${BASE_TAG}-fixes
Now
* git co ${BASE_TAG}-fixes
* git cherry-pick any fixes
* Test (see above)
* make NEW_TAG=v1.XX.1 tag
* make NEW_TAG=${NEW_TAG} tag
* edit docs/content/changelog.md
* make TAG=v1.43.1 doc
* git commit -a -v -m "Version v1.XX.1"
* git tag -d -v1.XX.1
* git tag -s -m "Version v1.XX.1" v1.XX.1
* git push --tags -u origin v1.XX-fixes
* make BRANCH_PATH= TAG=v1.43.1 fetch_binaries
* make TAG=v1.43.1 tarball
* make TAG=v1.43.1 sign_upload
* make TAG=v1.43.1 check_sign
* make TAG=v1.43.1 upload
* make TAG=v1.43.1 upload_website
* make TAG=v1.43.1 upload_github
* NB this overwrites the current beta so after the release, rebuild the last travis build
* make TAG=${NEW_TAG} doc
* git commit -a -v -m "Version ${NEW_TAG}"
* git tag -d ${NEW_TAG}
* git tag -s -m "Version ${NEW_TAG}" ${NEW_TAG}
* git push --tags -u origin ${BASE_TAG}-fixes
* Wait for builds to complete
* make BRANCH_PATH= TAG=${NEW_TAG} fetch_binaries
* make TAG=${NEW_TAG} tarball
* make TAG=${NEW_TAG} sign_upload
* make TAG=${NEW_TAG} check_sign
* make TAG=${NEW_TAG} upload
* make TAG=${NEW_TAG} upload_website
* make TAG=${NEW_TAG} upload_github
* NB this overwrites the current beta so we need to do this
* git co master
* make LAST_TAG=${NEW_TAG} startdev
* # cherry pick the changes to the changelog
* git checkout ${BASE_TAG}-fixes docs/content/changelog.md
* git commit --amend
* git push
* Announce!
## Making a manual build of docker
The rclone docker image should autobuild on docker hub. If it doesn't
or needs to be updated then rebuild like this.
```
docker build -t rclone/rclone:1.49.1 -t rclone/rclone:1.49 -t rclone/rclone:1 -t rclone/rclone:latest .
docker push rclone/rclone:1.49.1
docker push rclone/rclone:1.49
docker push rclone/rclone:1
docker push rclone/rclone:latest
```
-239
View File
@@ -1,239 +0,0 @@
---
# Azure pipelines build for rclone
# Parts stolen shamelessly from all round the Internet, especially Caddy
# -*- compile-command: "yamllint -f parsable azure-pipelines.yml" -*-
trigger:
branches:
include:
- '*'
tags:
include:
- '*'
variables:
GOROOT: $(gorootDir)/go
GOPATH: $(system.defaultWorkingDirectory)/gopath
GOCACHE: $(system.defaultWorkingDirectory)/gocache
GOBIN: $(GOPATH)/bin
modulePath: '$(GOPATH)/src/github.com/$(build.repository.name)'
GO111MODULE: 'off'
GOTAGS: cmount
GO_LATEST: false
CPATH: ''
GO_INSTALL_ARCH: amd64
strategy:
matrix:
linux:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: latest
GOTAGS: cmount
BUILD_FLAGS: '-include "^linux/"'
MAKE_CHECK: true
MAKE_QUICKTEST: true
DEPLOY: true
mac:
imageName: macos-10.13
gorootDir: /usr/local
GO_VERSION: latest
GOTAGS: "" # cmount doesn't work on osx travis for some reason
BUILD_FLAGS: '-include "^darwin/" -cgo'
MAKE_QUICKTEST: true
MAKE_RACEQUICKTEST: true
DEPLOY: true
windows_amd64:
imageName: windows-2019
gorootDir: C:\
GO_VERSION: latest
BUILD_FLAGS: '-include "^windows/amd64" -cgo'
MAKE_QUICKTEST: true
DEPLOY: true
windows_386:
imageName: windows-2019
gorootDir: C:\
GO_VERSION: latest
GO_INSTALL_ARCH: 386
BUILD_FLAGS: '-include "^windows/386" -cgo'
MAKE_QUICKTEST: true
DEPLOY: true
other_os:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: latest
BUILD_FLAGS: '-exclude "^(windows|darwin|linux)/"'
MAKE_COMPILE_ALL: true
DEPLOY: true
modules_race:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: latest
GO111MODULE: on
GOPROXY: https://proxy.golang.org
MAKE_QUICKTEST: true
MAKE_RACEQUICKTEST: true
go1.9:
imageName: ubuntu-16.04
gorootDir: /usr/local
GOCACHE: '' # build caching only came in go1.10
GO_VERSION: go1.9.7
MAKE_QUICKTEST: true
go1.10:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: go1.10.8
MAKE_QUICKTEST: true
go1.11:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: go1.11.12
MAKE_QUICKTEST: true
pool:
vmImage: $(imageName)
steps:
- bash: |
latestGo=$(curl "https://golang.org/VERSION?m=text")
echo "##vso[task.setvariable variable=GO_VERSION]$latestGo"
echo "##vso[task.setvariable variable=GO_LATEST]true"
echo "Latest Go version: $latestGo"
condition: eq( variables['GO_VERSION'], 'latest' )
continueOnError: false
displayName: "Get latest Go version"
- bash: |
sudo rm -f $(which go)
echo '##vso[task.prependpath]$(GOBIN)'
echo '##vso[task.prependpath]$(GOROOT)/bin'
mkdir -p '$(modulePath)'
shopt -s extglob
shopt -s dotglob
mv !(gopath) '$(modulePath)'
continueOnError: false
displayName: Remove old Go, set GOBIN/GOROOT, and move project into GOPATH
- task: CacheBeta@0
inputs:
key: go-build-cache | "$(Agent.JobName)"
path: $(GOCACHE)
continueOnError: true
displayName: Cache go build
condition: ne( variables['GOCACHE'], '' )
# Install Libraries (varies by platform)
- bash: |
sudo modprobe fuse
sudo chmod 666 /dev/fuse
sudo chown root:$USER /etc/fuse.conf
sudo apt-get install fuse libfuse-dev rpm pkg-config
condition: eq( variables['Agent.OS'], 'Linux' )
continueOnError: false
displayName: Install Libraries on Linux
- bash: |
brew update
brew tap caskroom/cask
brew cask install osxfuse
condition: eq( variables['Agent.OS'], 'Darwin' )
continueOnError: false
displayName: Install Libraries on macOS
- powershell: |
$ProgressPreference = 'SilentlyContinue'
choco install -y winfsp zip
Write-Host "##vso[task.setvariable variable=CPATH]C:\Program Files\WinFsp\inc\fuse;C:\Program Files (x86)\WinFsp\inc\fuse"
if ($env:GO_INSTALL_ARCH -eq "386") {
choco install -y mingw --forcex86 --force
Write-Host "##vso[task.prependpath]C:\\ProgramData\\chocolatey\\lib\\mingw\\tools\\install\\mingw32\\bin"
}
# Copy mingw32-make.exe to make.exe so the same command line
# can be used on Windows as on macOS and Linux
$path = (get-command mingw32-make.exe).Path
Copy-Item -Path $path -Destination (Join-Path (Split-Path -Path $path) 'make.exe')
condition: eq( variables['Agent.OS'], 'Windows_NT' )
continueOnError: false
displayName: Install Libraries on Windows
# Install Go (this varies by platform)
- bash: |
wget "https://dl.google.com/go/$(GO_VERSION).linux-$(GO_INSTALL_ARCH).tar.gz"
sudo mkdir $(gorootDir)
sudo chown ${USER}:${USER} $(gorootDir)
tar -C $(gorootDir) -xzf "$(GO_VERSION).linux-$(GO_INSTALL_ARCH).tar.gz"
condition: eq( variables['Agent.OS'], 'Linux' )
continueOnError: false
displayName: Install Go on Linux
- bash: |
wget "https://dl.google.com/go/$(GO_VERSION).darwin-$(GO_INSTALL_ARCH).tar.gz"
sudo tar -C $(gorootDir) -xzf "$(GO_VERSION).darwin-$(GO_INSTALL_ARCH).tar.gz"
condition: eq( variables['Agent.OS'], 'Darwin' )
continueOnError: false
displayName: Install Go on macOS
- powershell: |
$ProgressPreference = 'SilentlyContinue'
Write-Host "Downloading Go $(GO_VERSION) for $(GO_INSTALL_ARCH)"
(New-Object System.Net.WebClient).DownloadFile("https://dl.google.com/go/$(GO_VERSION).windows-$(GO_INSTALL_ARCH).zip", "$(GO_VERSION).windows-$(GO_INSTALL_ARCH).zip")
Write-Host "Extracting Go"
Expand-Archive "$(GO_VERSION).windows-$(GO_INSTALL_ARCH).zip" -DestinationPath "$(gorootDir)"
condition: eq( variables['Agent.OS'], 'Windows_NT' )
continueOnError: false
displayName: Install Go on Windows
# Display environment for debugging
- bash: |
printf "Using go at: $(which go)\n"
printf "Go version: $(go version)\n"
printf "\n\nGo environment:\n\n"
go env
printf "\n\nRclone environment:\n\n"
make vars
printf "\n\nSystem environment:\n\n"
env
workingDirectory: '$(modulePath)'
displayName: Print Go version and environment
# Run Tests
- bash: |
make
make quicktest
workingDirectory: '$(modulePath)'
displayName: Run tests
condition: eq( variables['MAKE_QUICKTEST'], 'true' )
- bash: |
make racequicktest
workingDirectory: '$(modulePath)'
displayName: Race test
condition: eq( variables['MAKE_RACEQUICKTEST'], 'true' )
- bash: |
make build_dep
make check
workingDirectory: '$(modulePath)'
displayName: Code quality test
condition: eq( variables['MAKE_CHECK'], 'true' )
- bash: |
make
make compile_all
workingDirectory: '$(modulePath)'
displayName: Compile all architectures test
condition: eq( variables['MAKE_COMPILE_ALL'], 'true' )
- bash: |
make travis_beta
env:
RCLONE_CONFIG_PASS: $(RCLONE_CONFIG_PASS)
BETA_SUBDIR: 'azure_pipelines' # FIXME remove when removing travis/appveyor
workingDirectory: '$(modulePath)'
displayName: Deploy built binaries
condition: and( eq( variables['DEPLOY'], 'true' ), ne( variables['Build.Reason'], 'PullRequest' ) )
+1
View File
@@ -20,6 +20,7 @@ import (
_ "github.com/rclone/rclone/backend/jottacloud"
_ "github.com/rclone/rclone/backend/koofr"
_ "github.com/rclone/rclone/backend/local"
_ "github.com/rclone/rclone/backend/mailru"
_ "github.com/rclone/rclone/backend/mega"
_ "github.com/rclone/rclone/backend/onedrive"
_ "github.com/rclone/rclone/backend/opendrive"
+3 -1
View File
@@ -1115,7 +1115,7 @@ func (o *Object) parseTimeString(timeString string) (err error) {
fs.Debugf(o, "Failed to parse mod time string %q: %v", timeString, err)
return err
}
o.modTime = time.Unix(unixMilliseconds/1E3, (unixMilliseconds%1E3)*1E6).UTC()
o.modTime = time.Unix(unixMilliseconds/1e3, (unixMilliseconds%1e3)*1e6).UTC()
return nil
}
@@ -1508,4 +1508,6 @@ var (
_ fs.ListRer = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
_ fs.GetTierer = &Object{}
_ fs.SetTierer = &Object{}
)
+2 -2
View File
@@ -50,7 +50,7 @@ type Timestamp time.Time
// MarshalJSON turns a Timestamp into JSON (in UTC)
func (t *Timestamp) MarshalJSON() (out []byte, err error) {
timestamp := (*time.Time)(t).UTC().UnixNano()
return []byte(strconv.FormatInt(timestamp/1E6, 10)), nil
return []byte(strconv.FormatInt(timestamp/1e6, 10)), nil
}
// UnmarshalJSON turns JSON into a Timestamp
@@ -59,7 +59,7 @@ func (t *Timestamp) UnmarshalJSON(data []byte) error {
if err != nil {
return err
}
*t = Timestamp(time.Unix(timestamp/1E3, (timestamp%1E3)*1E6).UTC())
*t = Timestamp(time.Unix(timestamp/1e3, (timestamp%1e3)*1e6).UTC())
return nil
}
+56 -56
View File
@@ -273,11 +273,11 @@ func (f *Fs) shouldRetryNoReauth(resp *http.Response, err error) (bool, error) {
// shouldRetry returns a boolean as to whether this resp and err
// deserve to be retried. It returns the err as a convenience
func (f *Fs) shouldRetry(resp *http.Response, err error) (bool, error) {
func (f *Fs) shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, error) {
if resp != nil && resp.StatusCode == 401 {
fs.Debugf(f, "Unauthorized: %v", err)
// Reauth
authErr := f.authorizeAccount()
authErr := f.authorizeAccount(ctx)
if authErr != nil {
err = authErr
}
@@ -393,7 +393,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
fs.Debugf(f, "Setting test header \"%s: %s\"", testModeHeader, testMode)
}
f.fillBufferTokens()
err = f.authorizeAccount()
err = f.authorizeAccount(ctx)
if err != nil {
return nil, errors.Wrap(err, "failed to authorize account")
}
@@ -431,7 +431,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// authorizeAccount gets the API endpoint and auth token. Can be used
// for reauthentication too.
func (f *Fs) authorizeAccount() error {
func (f *Fs) authorizeAccount(ctx context.Context) error {
f.authMu.Lock()
defer f.authMu.Unlock()
opts := rest.Opts{
@@ -443,7 +443,7 @@ func (f *Fs) authorizeAccount() error {
ExtraHeaders: map[string]string{"Authorization": ""}, // unset the Authorization for this request
}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, nil, &f.info)
resp, err := f.srv.CallJSON(ctx, &opts, nil, &f.info)
return f.shouldRetryNoReauth(resp, err)
})
if err != nil {
@@ -466,10 +466,10 @@ func (f *Fs) hasPermission(permission string) bool {
// getUploadURL returns the upload info with the UploadURL and the AuthorizationToken
//
// This should be returned with returnUploadURL when finished
func (f *Fs) getUploadURL(bucket string) (upload *api.GetUploadURLResponse, err error) {
func (f *Fs) getUploadURL(ctx context.Context, bucket string) (upload *api.GetUploadURLResponse, err error) {
f.uploadMu.Lock()
defer f.uploadMu.Unlock()
bucketID, err := f.getBucketID(bucket)
bucketID, err := f.getBucketID(ctx, bucket)
if err != nil {
return nil, err
}
@@ -489,8 +489,8 @@ func (f *Fs) getUploadURL(bucket string) (upload *api.GetUploadURLResponse, err
BucketID: bucketID,
}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &upload)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &upload)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, errors.Wrap(err, "failed to get upload URL")
@@ -609,7 +609,7 @@ func (f *Fs) list(ctx context.Context, bucket, directory, prefix string, addBuck
if !recurse {
delimiter = "/"
}
bucketID, err := f.getBucketID(bucket)
bucketID, err := f.getBucketID(ctx, bucket)
if err != nil {
return err
}
@@ -636,8 +636,8 @@ func (f *Fs) list(ctx context.Context, bucket, directory, prefix string, addBuck
for {
var response api.ListFileNamesResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
@@ -727,7 +727,7 @@ func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addB
// listBuckets returns all the buckets to out
func (f *Fs) listBuckets(ctx context.Context) (entries fs.DirEntries, err error) {
err = f.listBucketsToFn(func(bucket *api.Bucket) error {
err = f.listBucketsToFn(ctx, func(bucket *api.Bucket) error {
d := fs.NewDir(bucket.Name, time.Time{})
entries = append(entries, d)
return nil
@@ -820,7 +820,7 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
type listBucketFn func(*api.Bucket) error
// listBucketsToFn lists the buckets to the function supplied
func (f *Fs) listBucketsToFn(fn listBucketFn) error {
func (f *Fs) listBucketsToFn(ctx context.Context, fn listBucketFn) error {
var account = api.ListBucketsRequest{
AccountID: f.info.AccountID,
BucketID: f.info.Allowed.BucketID,
@@ -832,8 +832,8 @@ func (f *Fs) listBucketsToFn(fn listBucketFn) error {
Path: "/b2_list_buckets",
}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &account, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &account, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
@@ -862,14 +862,14 @@ func (f *Fs) listBucketsToFn(fn listBucketFn) error {
// getbucketType finds the bucketType for the current bucket name
// can be one of allPublic. allPrivate, or snapshot
func (f *Fs) getbucketType(bucket string) (bucketType string, err error) {
func (f *Fs) getbucketType(ctx context.Context, bucket string) (bucketType string, err error) {
f.bucketTypeMutex.Lock()
bucketType = f._bucketType[bucket]
f.bucketTypeMutex.Unlock()
if bucketType != "" {
return bucketType, nil
}
err = f.listBucketsToFn(func(bucket *api.Bucket) error {
err = f.listBucketsToFn(ctx, func(bucket *api.Bucket) error {
// listBucketsToFn reads bucket Types
return nil
})
@@ -897,14 +897,14 @@ func (f *Fs) clearBucketType(bucket string) {
}
// getBucketID finds the ID for the current bucket name
func (f *Fs) getBucketID(bucket string) (bucketID string, err error) {
func (f *Fs) getBucketID(ctx context.Context, bucket string) (bucketID string, err error) {
f.bucketIDMutex.Lock()
bucketID = f._bucketID[bucket]
f.bucketIDMutex.Unlock()
if bucketID != "" {
return bucketID, nil
}
err = f.listBucketsToFn(func(bucket *api.Bucket) error {
err = f.listBucketsToFn(ctx, func(bucket *api.Bucket) error {
// listBucketsToFn sets IDs
return nil
})
@@ -970,15 +970,15 @@ func (f *Fs) makeBucket(ctx context.Context, bucket string) error {
}
var response api.Bucket
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
if apiErr, ok := err.(*api.Error); ok {
if apiErr.Code == "duplicate_bucket_name" {
// Check this is our bucket - buckets are globally unique and this
// might be someone elses.
_, getBucketErr := f.getBucketID(bucket)
_, getBucketErr := f.getBucketID(ctx, bucket)
if getBucketErr == nil {
// found so it is our bucket
return nil
@@ -1009,7 +1009,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
Method: "POST",
Path: "/b2_delete_bucket",
}
bucketID, err := f.getBucketID(bucket)
bucketID, err := f.getBucketID(ctx, bucket)
if err != nil {
return err
}
@@ -1019,8 +1019,8 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
}
var response api.Bucket
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return errors.Wrap(err, "failed to delete bucket")
@@ -1038,8 +1038,8 @@ func (f *Fs) Precision() time.Duration {
}
// hide hides a file on the remote
func (f *Fs) hide(bucket, bucketPath string) error {
bucketID, err := f.getBucketID(bucket)
func (f *Fs) hide(ctx context.Context, bucket, bucketPath string) error {
bucketID, err := f.getBucketID(ctx, bucket)
if err != nil {
return err
}
@@ -1053,8 +1053,8 @@ func (f *Fs) hide(bucket, bucketPath string) error {
}
var response api.File
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
if apiErr, ok := err.(*api.Error); ok {
@@ -1070,7 +1070,7 @@ func (f *Fs) hide(bucket, bucketPath string) error {
}
// deleteByID deletes a file version given Name and ID
func (f *Fs) deleteByID(ID, Name string) error {
func (f *Fs) deleteByID(ctx context.Context, ID, Name string) error {
opts := rest.Opts{
Method: "POST",
Path: "/b2_delete_file_version",
@@ -1081,8 +1081,8 @@ func (f *Fs) deleteByID(ID, Name string) error {
}
var response api.File
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return errors.Wrapf(err, "failed to delete %q", Name)
@@ -1132,7 +1132,7 @@ func (f *Fs) purge(ctx context.Context, bucket, directory string, oldOnly bool)
continue
}
tr := accounting.Stats(ctx).NewCheckingTransfer(oi)
err = f.deleteByID(object.ID, object.Name)
err = f.deleteByID(ctx, object.ID, object.Name)
checkErr(err)
tr.Done(err)
}
@@ -1205,7 +1205,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
fs.Debugf(src, "Can't copy - not same remote type")
return nil, fs.ErrorCantCopy
}
destBucketID, err := f.getBucketID(dstBucket)
destBucketID, err := f.getBucketID(ctx, dstBucket)
if err != nil {
return nil, err
}
@@ -1221,8 +1221,8 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
}
var response api.FileInfo
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, err
@@ -1245,7 +1245,7 @@ func (f *Fs) Hashes() hash.Set {
// getDownloadAuthorization returns authorization token for downloading
// without account.
func (f *Fs) getDownloadAuthorization(bucket, remote string) (authorization string, err error) {
func (f *Fs) getDownloadAuthorization(ctx context.Context, bucket, remote string) (authorization string, err error) {
validDurationInSeconds := time.Duration(f.opt.DownloadAuthorizationDuration).Nanoseconds() / 1e9
if validDurationInSeconds <= 0 || validDurationInSeconds > 604800 {
return "", errors.New("--b2-download-auth-duration must be between 1 sec and 1 week")
@@ -1253,7 +1253,7 @@ func (f *Fs) getDownloadAuthorization(bucket, remote string) (authorization stri
if !f.hasPermission("shareFiles") {
return "", errors.New("sharing a file link requires the shareFiles permission")
}
bucketID, err := f.getBucketID(bucket)
bucketID, err := f.getBucketID(ctx, bucket)
if err != nil {
return "", err
}
@@ -1268,8 +1268,8 @@ func (f *Fs) getDownloadAuthorization(bucket, remote string) (authorization stri
}
var response api.GetDownloadAuthorizationResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return "", errors.Wrap(err, "failed to get download authorization")
@@ -1301,12 +1301,12 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (link string, err er
}
absPath := "/" + bucketPath
link = RootURL + "/file/" + urlEncode(bucket) + absPath
bucketType, err := f.getbucketType(bucket)
bucketType, err := f.getbucketType(ctx, bucket)
if err != nil {
return "", err
}
if bucketType == "allPrivate" || bucketType == "snapshot" {
AuthorizationToken, err := f.getDownloadAuthorization(bucket, remote)
AuthorizationToken, err := f.getDownloadAuthorization(ctx, bucket, remote)
if err != nil {
return "", err
}
@@ -1453,7 +1453,7 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
// timeString returns modTime as the number of milliseconds
// elapsed since January 1, 1970 UTC as a decimal string.
func timeString(modTime time.Time) string {
return strconv.FormatInt(modTime.UnixNano()/1E6, 10)
return strconv.FormatInt(modTime.UnixNano()/1e6, 10)
}
// parseTimeString converts a decimal string number of milliseconds
@@ -1468,7 +1468,7 @@ func (o *Object) parseTimeString(timeString string) (err error) {
fs.Debugf(o, "Failed to parse mod time string %q: %v", timeString, err)
return nil
}
o.modTime = time.Unix(unixMilliseconds/1E3, (unixMilliseconds%1E3)*1E6).UTC()
o.modTime = time.Unix(unixMilliseconds/1e3, (unixMilliseconds%1e3)*1e6).UTC()
return nil
}
@@ -1505,8 +1505,8 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
}
var response api.FileInfo
err = o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.CallJSON(&opts, &request, &response)
return o.fs.shouldRetry(resp, err)
resp, err := o.fs.srv.CallJSON(ctx, &opts, &request, &response)
return o.fs.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
@@ -1604,8 +1604,8 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
return o.fs.shouldRetry(resp, err)
resp, err = o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, errors.Wrap(err, "failed to open for download")
@@ -1701,7 +1701,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
o.fs.putUploadBlock(buf)
return err
}
return up.Stream(buf)
return up.Stream(ctx, buf)
} else if err == io.EOF || err == io.ErrUnexpectedEOF {
fs.Debugf(o, "File has %d bytes, which makes only one chunk. Using direct upload.", n)
defer o.fs.putUploadBlock(buf)
@@ -1715,7 +1715,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
if err != nil {
return err
}
return up.Upload()
return up.Upload(ctx)
}
modTime := src.ModTime(ctx)
@@ -1729,7 +1729,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
// Get upload URL
upload, err := o.fs.getUploadURL(bucket)
upload, err := o.fs.getUploadURL(ctx, bucket)
if err != nil {
return err
}
@@ -1807,8 +1807,8 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
var response api.FileInfo
// Don't retry, return a retry error instead
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err := o.fs.srv.CallJSON(&opts, nil, &response)
retry, err := o.fs.shouldRetry(resp, err)
resp, err := o.fs.srv.CallJSON(ctx, &opts, nil, &response)
retry, err := o.fs.shouldRetry(ctx, resp, err)
// On retryable error clear UploadURL
if retry {
fs.Debugf(o, "Clearing upload URL because of error: %v", err)
@@ -1829,9 +1829,9 @@ func (o *Object) Remove(ctx context.Context) error {
return errNotWithVersions
}
if o.fs.opt.HardDelete {
return o.fs.deleteByID(o.id, bucketPath)
return o.fs.deleteByID(ctx, o.id, bucketPath)
}
return o.fs.hide(bucket, bucketPath)
return o.fs.hide(ctx, bucket, bucketPath)
}
// MimeType of an Object if known, "" otherwise
+28 -28
View File
@@ -105,7 +105,7 @@ func (f *Fs) newLargeUpload(ctx context.Context, o *Object, in io.Reader, src fs
Path: "/b2_start_large_file",
}
bucket, bucketPath := o.split()
bucketID, err := f.getBucketID(bucket)
bucketID, err := f.getBucketID(ctx, bucket)
if err != nil {
return nil, err
}
@@ -125,8 +125,8 @@ func (f *Fs) newLargeUpload(ctx context.Context, o *Object, in io.Reader, src fs
}
var response api.StartLargeFileResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, &request, &response)
return f.shouldRetry(resp, err)
resp, err := f.srv.CallJSON(ctx, &opts, &request, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, err
@@ -150,7 +150,7 @@ func (f *Fs) newLargeUpload(ctx context.Context, o *Object, in io.Reader, src fs
// getUploadURL returns the upload info with the UploadURL and the AuthorizationToken
//
// This should be returned with returnUploadURL when finished
func (up *largeUpload) getUploadURL() (upload *api.GetUploadPartURLResponse, err error) {
func (up *largeUpload) getUploadURL(ctx context.Context) (upload *api.GetUploadPartURLResponse, err error) {
up.uploadMu.Lock()
defer up.uploadMu.Unlock()
if len(up.uploads) == 0 {
@@ -162,8 +162,8 @@ func (up *largeUpload) getUploadURL() (upload *api.GetUploadPartURLResponse, err
ID: up.id,
}
err := up.f.pacer.Call(func() (bool, error) {
resp, err := up.f.srv.CallJSON(&opts, &request, &upload)
return up.f.shouldRetry(resp, err)
resp, err := up.f.srv.CallJSON(ctx, &opts, &request, &upload)
return up.f.shouldRetry(ctx, resp, err)
})
if err != nil {
return nil, errors.Wrap(err, "failed to get upload URL")
@@ -192,12 +192,12 @@ func (up *largeUpload) clearUploadURL() {
}
// Transfer a chunk
func (up *largeUpload) transferChunk(part int64, body []byte) error {
func (up *largeUpload) transferChunk(ctx context.Context, part int64, body []byte) error {
err := up.f.pacer.Call(func() (bool, error) {
fs.Debugf(up.o, "Sending chunk %d length %d", part, len(body))
// Get upload URL
upload, err := up.getUploadURL()
upload, err := up.getUploadURL(ctx)
if err != nil {
return false, err
}
@@ -241,8 +241,8 @@ func (up *largeUpload) transferChunk(part int64, body []byte) error {
var response api.UploadPartResponse
resp, err := up.f.srv.CallJSON(&opts, nil, &response)
retry, err := up.f.shouldRetry(resp, err)
resp, err := up.f.srv.CallJSON(ctx, &opts, nil, &response)
retry, err := up.f.shouldRetry(ctx, resp, err)
if err != nil {
fs.Debugf(up.o, "Error sending chunk %d (retry=%v): %v: %#v", part, retry, err, err)
}
@@ -264,7 +264,7 @@ func (up *largeUpload) transferChunk(part int64, body []byte) error {
}
// finish closes off the large upload
func (up *largeUpload) finish() error {
func (up *largeUpload) finish(ctx context.Context) error {
fs.Debugf(up.o, "Finishing large file upload with %d parts", up.parts)
opts := rest.Opts{
Method: "POST",
@@ -276,8 +276,8 @@ func (up *largeUpload) finish() error {
}
var response api.FileInfo
err := up.f.pacer.Call(func() (bool, error) {
resp, err := up.f.srv.CallJSON(&opts, &request, &response)
return up.f.shouldRetry(resp, err)
resp, err := up.f.srv.CallJSON(ctx, &opts, &request, &response)
return up.f.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
@@ -286,7 +286,7 @@ func (up *largeUpload) finish() error {
}
// cancel aborts the large upload
func (up *largeUpload) cancel() error {
func (up *largeUpload) cancel(ctx context.Context) error {
opts := rest.Opts{
Method: "POST",
Path: "/b2_cancel_large_file",
@@ -296,18 +296,18 @@ func (up *largeUpload) cancel() error {
}
var response api.CancelLargeFileResponse
err := up.f.pacer.Call(func() (bool, error) {
resp, err := up.f.srv.CallJSON(&opts, &request, &response)
return up.f.shouldRetry(resp, err)
resp, err := up.f.srv.CallJSON(ctx, &opts, &request, &response)
return up.f.shouldRetry(ctx, resp, err)
})
return err
}
func (up *largeUpload) managedTransferChunk(wg *sync.WaitGroup, errs chan error, part int64, buf []byte) {
func (up *largeUpload) managedTransferChunk(ctx context.Context, wg *sync.WaitGroup, errs chan error, part int64, buf []byte) {
wg.Add(1)
go func(part int64, buf []byte) {
defer wg.Done()
defer up.f.putUploadBlock(buf)
err := up.transferChunk(part, buf)
err := up.transferChunk(ctx, part, buf)
if err != nil {
select {
case errs <- err:
@@ -317,7 +317,7 @@ func (up *largeUpload) managedTransferChunk(wg *sync.WaitGroup, errs chan error,
}(part, buf)
}
func (up *largeUpload) finishOrCancelOnError(err error, errs chan error) error {
func (up *largeUpload) finishOrCancelOnError(ctx context.Context, err error, errs chan error) error {
if err == nil {
select {
case err = <-errs:
@@ -326,19 +326,19 @@ func (up *largeUpload) finishOrCancelOnError(err error, errs chan error) error {
}
if err != nil {
fs.Debugf(up.o, "Cancelling large file upload due to error: %v", err)
cancelErr := up.cancel()
cancelErr := up.cancel(ctx)
if cancelErr != nil {
fs.Errorf(up.o, "Failed to cancel large file upload: %v", cancelErr)
}
return err
}
return up.finish()
return up.finish(ctx)
}
// Stream uploads the chunks from the input, starting with a required initial
// chunk. Assumes the file size is unknown and will upload until the input
// reaches EOF.
func (up *largeUpload) Stream(initialUploadBlock []byte) (err error) {
func (up *largeUpload) Stream(ctx context.Context, initialUploadBlock []byte) (err error) {
fs.Debugf(up.o, "Starting streaming of large file (id %q)", up.id)
errs := make(chan error, 1)
hasMoreParts := true
@@ -346,7 +346,7 @@ func (up *largeUpload) Stream(initialUploadBlock []byte) (err error) {
// Transfer initial chunk
up.size = int64(len(initialUploadBlock))
up.managedTransferChunk(&wg, errs, 1, initialUploadBlock)
up.managedTransferChunk(ctx, &wg, errs, 1, initialUploadBlock)
outer:
for part := int64(2); hasMoreParts; part++ {
@@ -388,16 +388,16 @@ outer:
}
// Transfer the chunk
up.managedTransferChunk(&wg, errs, part, buf)
up.managedTransferChunk(ctx, &wg, errs, part, buf)
}
wg.Wait()
up.sha1s = up.sha1s[:up.parts]
return up.finishOrCancelOnError(err, errs)
return up.finishOrCancelOnError(ctx, err, errs)
}
// Upload uploads the chunks from the input
func (up *largeUpload) Upload() error {
func (up *largeUpload) Upload(ctx context.Context) error {
fs.Debugf(up.o, "Starting upload of large file in %d chunks (id %q)", up.parts, up.id)
remaining := up.size
errs := make(chan error, 1)
@@ -428,10 +428,10 @@ outer:
}
// Transfer the chunk
up.managedTransferChunk(&wg, errs, part, buf)
up.managedTransferChunk(ctx, &wg, errs, part, buf)
remaining -= reqSize
}
wg.Wait()
return up.finishOrCancelOnError(err, errs)
return up.finishOrCancelOnError(ctx, err, errs)
}
+20
View File
@@ -202,3 +202,23 @@ type CommitUpload struct {
ContentModifiedAt Time `json:"content_modified_at"`
} `json:"attributes"`
}
// ConfigJSON defines the shape of a box config.json
type ConfigJSON struct {
BoxAppSettings AppSettings `json:"boxAppSettings"`
EnterpriseID string `json:"enterpriseID"`
}
// AppSettings defines the shape of the boxAppSettings within box config.json
type AppSettings struct {
ClientID string `json:"clientID"`
ClientSecret string `json:"clientSecret"`
AppAuth AppAuth `json:"appAuth"`
}
// AppAuth defines the shape of the appAuth within boxAppSettings in config.json
type AppAuth struct {
PublicKeyID string `json:"publicKeyID"`
PrivateKey string `json:"privateKey"`
Passphrase string `json:"passphrase"`
}
+142 -25
View File
@@ -11,8 +11,12 @@ package box
import (
"context"
"crypto/rsa"
"encoding/json"
"encoding/pem"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"net/url"
@@ -21,6 +25,10 @@ import (
"strings"
"time"
"github.com/rclone/rclone/lib/jwtutil"
"github.com/youmark/pkcs8"
"github.com/pkg/errors"
"github.com/rclone/rclone/backend/box/api"
"github.com/rclone/rclone/fs"
@@ -29,12 +37,14 @@ import (
"github.com/rclone/rclone/fs/config/configstruct"
"github.com/rclone/rclone/fs/config/obscure"
"github.com/rclone/rclone/fs/fserrors"
"github.com/rclone/rclone/fs/fshttp"
"github.com/rclone/rclone/fs/hash"
"github.com/rclone/rclone/lib/dircache"
"github.com/rclone/rclone/lib/oauthutil"
"github.com/rclone/rclone/lib/pacer"
"github.com/rclone/rclone/lib/rest"
"golang.org/x/oauth2"
"golang.org/x/oauth2/jws"
)
const (
@@ -49,6 +59,7 @@ const (
listChunks = 1000 // chunk size to read directory listings
minUploadCutoff = 50000000 // upload cutoff can be no lower than this
defaultUploadCutoff = 50 * 1024 * 1024
tokenURL = "https://api.box.com/oauth2/token"
)
// Globals
@@ -73,9 +84,34 @@ func init() {
Description: "Box",
NewFs: NewFs,
Config: func(name string, m configmap.Mapper) {
err := oauthutil.Config("box", name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to configure token: %v", err)
jsonFile, ok := m.Get("box_config_file")
boxSubType, boxSubTypeOk := m.Get("box_sub_type")
var err error
if ok && boxSubTypeOk && jsonFile != "" && boxSubType != "" {
boxConfig, err := getBoxConfig(jsonFile)
if err != nil {
log.Fatalf("Failed to configure token: %v", err)
}
privateKey, err := getDecryptedPrivateKey(boxConfig)
if err != nil {
log.Fatalf("Failed to configure token: %v", err)
}
claims, err := getClaims(boxConfig, boxSubType)
if err != nil {
log.Fatalf("Failed to configure token: %v", err)
}
signingHeaders := getSigningHeaders(boxConfig)
queryParams := getQueryParams(boxConfig)
client := fshttp.NewClient(fs.Config)
err = jwtutil.Config("box", name, claims, signingHeaders, queryParams, privateKey, m, client)
if err != nil {
log.Fatalf("Failed to configure token with jwt authentication: %v", err)
}
} else {
err = oauthutil.Config("box", name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to configure token with oauth authentication: %v", err)
}
}
},
Options: []fs.Option{{
@@ -84,6 +120,19 @@ func init() {
}, {
Name: config.ConfigClientSecret,
Help: "Box App Client Secret\nLeave blank normally.",
}, {
Name: "box_config_file",
Help: "Box App config.json location\nLeave blank normally.",
}, {
Name: "box_sub_type",
Default: "user",
Examples: []fs.OptionExample{{
Value: "user",
Help: "Rclone should act on behalf of a user",
}, {
Value: "enterprise",
Help: "Rclone should act on behalf of a service account",
}},
}, {
Name: "upload_cutoff",
Help: "Cutoff for switching to multipart upload (>= 50MB).",
@@ -98,6 +147,74 @@ func init() {
})
}
func getBoxConfig(configFile string) (boxConfig *api.ConfigJSON, err error) {
file, err := ioutil.ReadFile(configFile)
if err != nil {
return nil, errors.Wrap(err, "box: failed to read Box config")
}
err = json.Unmarshal(file, &boxConfig)
if err != nil {
return nil, errors.Wrap(err, "box: failed to parse Box config")
}
return boxConfig, nil
}
func getClaims(boxConfig *api.ConfigJSON, boxSubType string) (claims *jws.ClaimSet, err error) {
val, err := jwtutil.RandomHex(20)
if err != nil {
return nil, errors.Wrap(err, "box: failed to generate random string for jti")
}
claims = &jws.ClaimSet{
Iss: boxConfig.BoxAppSettings.ClientID,
Sub: boxConfig.EnterpriseID,
Aud: tokenURL,
Iat: time.Now().Unix(),
Exp: time.Now().Add(time.Second * 45).Unix(),
PrivateClaims: map[string]interface{}{
"box_sub_type": boxSubType,
"aud": tokenURL,
"jti": val,
},
}
return claims, nil
}
func getSigningHeaders(boxConfig *api.ConfigJSON) *jws.Header {
signingHeaders := &jws.Header{
Algorithm: "RS256",
Typ: "JWT",
KeyID: boxConfig.BoxAppSettings.AppAuth.PublicKeyID,
}
return signingHeaders
}
func getQueryParams(boxConfig *api.ConfigJSON) map[string]string {
queryParams := map[string]string{
"client_id": boxConfig.BoxAppSettings.ClientID,
"client_secret": boxConfig.BoxAppSettings.ClientSecret,
}
return queryParams
}
func getDecryptedPrivateKey(boxConfig *api.ConfigJSON) (key *rsa.PrivateKey, err error) {
block, rest := pem.Decode([]byte(boxConfig.BoxAppSettings.AppAuth.PrivateKey))
if len(rest) > 0 {
return nil, errors.Wrap(err, "box: extra data included in private key")
}
rsaKey, err := pkcs8.ParsePKCS8PrivateKey(block.Bytes, []byte(boxConfig.BoxAppSettings.AppAuth.Passphrase))
if err != nil {
return nil, errors.Wrap(err, "box: failed to decrypt private key")
}
return rsaKey.(*rsa.PrivateKey), nil
}
// Options defines the configuration for this backend
type Options struct {
UploadCutoff fs.SizeSuffix `config:"upload_cutoff"`
@@ -204,7 +321,7 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string) (info *api.It
return nil, err
}
found, err := f.listAll(directoryID, false, true, func(item *api.Item) bool {
found, err := f.listAll(ctx, directoryID, false, true, func(item *api.Item) bool {
if item.Name == leaf {
info = item
return true
@@ -352,7 +469,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
// FindLeaf finds a directory of name leaf in the folder with ID pathID
func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
// Find the leaf in pathID
found, err = f.listAll(pathID, true, false, func(item *api.Item) bool {
found, err = f.listAll(ctx, pathID, true, false, func(item *api.Item) bool {
if item.Name == leaf {
pathIDOut = item.ID
return true
@@ -386,7 +503,7 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
},
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &mkdir, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &mkdir, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -408,7 +525,7 @@ type listAllFn func(*api.Item) bool
// Lists the directory required calling the user function on each item found
//
// If the user fn ever returns true then it early exits with found = true
func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
func (f *Fs) listAll(ctx context.Context, dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
opts := rest.Opts{
Method: "GET",
Path: "/folders/" + dirID + "/items",
@@ -423,7 +540,7 @@ OUTER:
var result api.FolderItems
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -479,7 +596,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, err
}
var iErr error
_, err = f.listAll(directoryID, false, false, func(info *api.Item) bool {
_, err = f.listAll(ctx, directoryID, false, false, func(info *api.Item) bool {
remote := path.Join(dir, info.Name)
if info.Type == api.ItemTypeFolder {
// cache the directory ID for later lookups
@@ -581,14 +698,14 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
}
// deleteObject removes an object by ID
func (f *Fs) deleteObject(id string) error {
func (f *Fs) deleteObject(ctx context.Context, id string) error {
opts := rest.Opts{
Method: "DELETE",
Path: "/files/" + id,
NoResponse: true,
}
return f.pacer.Call(func() (bool, error) {
resp, err := f.srv.Call(&opts)
resp, err := f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
}
@@ -619,7 +736,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
opts.Parameters.Set("recursive", strconv.FormatBool(!check))
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -692,7 +809,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
var resp *http.Response
var info *api.Item
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &copyFile, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &copyFile, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -715,7 +832,7 @@ func (f *Fs) Purge(ctx context.Context) error {
}
// move a file or folder
func (f *Fs) move(endpoint, id, leaf, directoryID string) (info *api.Item, err error) {
func (f *Fs) move(ctx context.Context, endpoint, id, leaf, directoryID string) (info *api.Item, err error) {
// Move the object
opts := rest.Opts{
Method: "PUT",
@@ -730,7 +847,7 @@ func (f *Fs) move(endpoint, id, leaf, directoryID string) (info *api.Item, err e
}
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &move, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &move, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -762,7 +879,7 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
}
// Do the move
info, err := f.move("/files/", srcObj.id, leaf, directoryID)
info, err := f.move(ctx, "/files/", srcObj.id, leaf, directoryID)
if err != nil {
return nil, err
}
@@ -845,7 +962,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
}
// Do the move
_, err = f.move("/folders/", srcID, leaf, directoryID)
_, err = f.move(ctx, "/folders/", srcID, leaf, directoryID)
if err != nil {
return err
}
@@ -887,7 +1004,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (string, error) {
var info api.Item
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &shareLink, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &shareLink, &info)
return shouldRetry(resp, err)
})
return info.SharedLink.URL, err
@@ -1006,7 +1123,7 @@ func (o *Object) setModTime(ctx context.Context, modTime time.Time) (*api.Item,
}
var info *api.Item
err := o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.CallJSON(&opts, &update, &info)
resp, err := o.fs.srv.CallJSON(ctx, &opts, &update, &info)
return shouldRetry(resp, err)
})
return info, err
@@ -1039,7 +1156,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1051,7 +1168,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
// upload does a single non-multipart upload
//
// This is recommended for less than 50 MB of content
func (o *Object) upload(in io.Reader, leaf, directoryID string, modTime time.Time) (err error) {
func (o *Object) upload(ctx context.Context, in io.Reader, leaf, directoryID string, modTime time.Time) (err error) {
upload := api.UploadFile{
Name: replaceReservedChars(leaf),
ContentModifiedAt: api.Time(modTime),
@@ -1078,7 +1195,7 @@ func (o *Object) upload(in io.Reader, leaf, directoryID string, modTime time.Tim
opts.Path = "/files/content"
}
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, &upload, &result)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &upload, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1111,16 +1228,16 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// Upload with simple or multipart
if size <= int64(o.fs.opt.UploadCutoff) {
err = o.upload(in, leaf, directoryID, modTime)
err = o.upload(ctx, in, leaf, directoryID, modTime)
} else {
err = o.uploadMultipart(in, leaf, directoryID, size, modTime)
err = o.uploadMultipart(ctx, in, leaf, directoryID, size, modTime)
}
return err
}
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
return o.fs.deleteObject(o.id)
return o.fs.deleteObject(ctx, o.id)
}
// ID returns the ID of the Object if known, or "" if not
+14 -13
View File
@@ -4,6 +4,7 @@ package box
import (
"bytes"
"context"
"crypto/sha1"
"encoding/base64"
"encoding/json"
@@ -22,7 +23,7 @@ import (
)
// createUploadSession creates an upload session for the object
func (o *Object) createUploadSession(leaf, directoryID string, size int64) (response *api.UploadSessionResponse, err error) {
func (o *Object) createUploadSession(ctx context.Context, leaf, directoryID string, size int64) (response *api.UploadSessionResponse, err error) {
opts := rest.Opts{
Method: "POST",
Path: "/files/upload_sessions",
@@ -41,7 +42,7 @@ func (o *Object) createUploadSession(leaf, directoryID string, size int64) (resp
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, &request, &response)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &request, &response)
return shouldRetry(resp, err)
})
return
@@ -53,7 +54,7 @@ func sha1Digest(digest []byte) string {
}
// uploadPart uploads a part in an upload session
func (o *Object) uploadPart(SessionID string, offset, totalSize int64, chunk []byte, wrap accounting.WrapFn) (response *api.UploadPartResponse, err error) {
func (o *Object) uploadPart(ctx context.Context, SessionID string, offset, totalSize int64, chunk []byte, wrap accounting.WrapFn) (response *api.UploadPartResponse, err error) {
chunkSize := int64(len(chunk))
sha1sum := sha1.Sum(chunk)
opts := rest.Opts{
@@ -70,7 +71,7 @@ func (o *Object) uploadPart(SessionID string, offset, totalSize int64, chunk []b
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
opts.Body = wrap(bytes.NewReader(chunk))
resp, err = o.fs.srv.CallJSON(&opts, nil, &response)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &response)
return shouldRetry(resp, err)
})
if err != nil {
@@ -80,7 +81,7 @@ func (o *Object) uploadPart(SessionID string, offset, totalSize int64, chunk []b
}
// commitUpload finishes an upload session
func (o *Object) commitUpload(SessionID string, parts []api.Part, modTime time.Time, sha1sum []byte) (result *api.FolderItems, err error) {
func (o *Object) commitUpload(ctx context.Context, SessionID string, parts []api.Part, modTime time.Time, sha1sum []byte) (result *api.FolderItems, err error) {
opts := rest.Opts{
Method: "POST",
Path: "/files/upload_sessions/" + SessionID + "/commit",
@@ -104,7 +105,7 @@ func (o *Object) commitUpload(SessionID string, parts []api.Part, modTime time.T
outer:
for tries = 0; tries < maxTries; tries++ {
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, &request, nil)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &request, nil)
if err != nil {
return shouldRetry(resp, err)
}
@@ -154,7 +155,7 @@ outer:
}
// abortUpload cancels an upload session
func (o *Object) abortUpload(SessionID string) (err error) {
func (o *Object) abortUpload(ctx context.Context, SessionID string) (err error) {
opts := rest.Opts{
Method: "DELETE",
Path: "/files/upload_sessions/" + SessionID,
@@ -163,16 +164,16 @@ func (o *Object) abortUpload(SessionID string) (err error) {
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
return err
}
// uploadMultipart uploads a file using multipart upload
func (o *Object) uploadMultipart(in io.Reader, leaf, directoryID string, size int64, modTime time.Time) (err error) {
func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, leaf, directoryID string, size int64, modTime time.Time) (err error) {
// Create upload session
session, err := o.createUploadSession(leaf, directoryID, size)
session, err := o.createUploadSession(ctx, leaf, directoryID, size)
if err != nil {
return errors.Wrap(err, "multipart upload create session failed")
}
@@ -183,7 +184,7 @@ func (o *Object) uploadMultipart(in io.Reader, leaf, directoryID string, size in
defer func() {
if err != nil {
fs.Debugf(o, "Cancelling multipart upload: %v", err)
cancelErr := o.abortUpload(session.ID)
cancelErr := o.abortUpload(ctx, session.ID)
if cancelErr != nil {
fs.Logf(o, "Failed to cancel multipart upload: %v", err)
}
@@ -235,7 +236,7 @@ outer:
defer wg.Done()
defer o.fs.uploadToken.Put()
fs.Debugf(o, "Uploading part %d/%d offset %v/%v part size %v", part+1, session.TotalParts, fs.SizeSuffix(position), fs.SizeSuffix(size), fs.SizeSuffix(chunkSize))
partResponse, err := o.uploadPart(session.ID, position, size, buf, wrap)
partResponse, err := o.uploadPart(ctx, session.ID, position, size, buf, wrap)
if err != nil {
err = errors.Wrap(err, "multipart upload failed to upload part")
select {
@@ -263,7 +264,7 @@ outer:
}
// Finalise the upload session
result, err := o.commitUpload(session.ID, parts, modTime, hash.Sum(nil))
result, err := o.commitUpload(ctx, session.ID, parts, modTime, hash.Sum(nil))
if err != nil {
return errors.Wrap(err, "multipart upload failed to finalize")
}
+15 -15
View File
@@ -705,16 +705,16 @@ var (
// Test test infrastructure first!
func TestRandomSource(t *testing.T) {
source := newRandomSource(1E8)
sink := newRandomSource(1E8)
source := newRandomSource(1e8)
sink := newRandomSource(1e8)
n, err := io.Copy(sink, source)
assert.NoError(t, err)
assert.Equal(t, int64(1E8), n)
assert.Equal(t, int64(1e8), n)
source = newRandomSource(1E8)
source = newRandomSource(1e8)
buf := make([]byte, 16)
_, _ = source.Read(buf)
sink = newRandomSource(1E8)
sink = newRandomSource(1e8)
_, err = io.Copy(sink, source)
assert.Error(t, err, "Error in stream")
}
@@ -754,23 +754,23 @@ func testEncryptDecrypt(t *testing.T, bufSize int, copySize int64) {
}
func TestEncryptDecrypt1(t *testing.T) {
testEncryptDecrypt(t, 1, 1E7)
testEncryptDecrypt(t, 1, 1e7)
}
func TestEncryptDecrypt32(t *testing.T) {
testEncryptDecrypt(t, 32, 1E8)
testEncryptDecrypt(t, 32, 1e8)
}
func TestEncryptDecrypt4096(t *testing.T) {
testEncryptDecrypt(t, 4096, 1E8)
testEncryptDecrypt(t, 4096, 1e8)
}
func TestEncryptDecrypt65536(t *testing.T) {
testEncryptDecrypt(t, 65536, 1E8)
testEncryptDecrypt(t, 65536, 1e8)
}
func TestEncryptDecrypt65537(t *testing.T) {
testEncryptDecrypt(t, 65537, 1E8)
testEncryptDecrypt(t, 65537, 1e8)
}
var (
@@ -803,7 +803,7 @@ func TestEncryptData(t *testing.T) {
} {
c, err := newCipher(NameEncryptionStandard, "", "", true)
assert.NoError(t, err)
c.cryptoRand = newRandomSource(1E8) // nodge the crypto rand generator
c.cryptoRand = newRandomSource(1e8) // nodge the crypto rand generator
// Check encode works
buf := bytes.NewBuffer(test.in)
@@ -826,7 +826,7 @@ func TestEncryptData(t *testing.T) {
func TestNewEncrypter(t *testing.T) {
c, err := newCipher(NameEncryptionStandard, "", "", true)
assert.NoError(t, err)
c.cryptoRand = newRandomSource(1E8) // nodge the crypto rand generator
c.cryptoRand = newRandomSource(1e8) // nodge the crypto rand generator
z := &zeroes{}
@@ -853,7 +853,7 @@ func TestNewEncrypterErrUnexpectedEOF(t *testing.T) {
fh, err := c.newEncrypter(in, nil)
assert.NoError(t, err)
n, err := io.CopyN(ioutil.Discard, fh, 1E6)
n, err := io.CopyN(ioutil.Discard, fh, 1e6)
assert.Equal(t, io.ErrUnexpectedEOF, err)
assert.Equal(t, int64(32), n)
}
@@ -885,7 +885,7 @@ func (c *closeDetector) Close() error {
func TestNewDecrypter(t *testing.T) {
c, err := newCipher(NameEncryptionStandard, "", "", true)
assert.NoError(t, err)
c.cryptoRand = newRandomSource(1E8) // nodge the crypto rand generator
c.cryptoRand = newRandomSource(1e8) // nodge the crypto rand generator
cd := newCloseDetector(bytes.NewBuffer(file0))
fh, err := c.newDecrypter(cd)
@@ -936,7 +936,7 @@ func TestNewDecrypterErrUnexpectedEOF(t *testing.T) {
fh, err := c.newDecrypter(in)
assert.NoError(t, err)
n, err := io.CopyN(ioutil.Discard, fh, 1E6)
n, err := io.CopyN(ioutil.Discard, fh, 1e6)
assert.Equal(t, io.ErrUnexpectedEOF, err)
assert.Equal(t, int64(16), n)
}
+22 -20
View File
@@ -156,6 +156,7 @@ func init() {
Description: "Google Drive",
NewFs: NewFs,
Config: func(name string, m configmap.Mapper) {
ctx := context.TODO()
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
@@ -177,7 +178,7 @@ func init() {
log.Fatalf("Failed to configure token: %v", err)
}
}
err = configTeamDrive(opt, m, name)
err = configTeamDrive(ctx, opt, m, name)
if err != nil {
log.Fatalf("Failed to configure team drive: %v", err)
}
@@ -663,7 +664,7 @@ OUTER:
for {
var files *drive.FileList
err = f.pacer.Call(func() (bool, error) {
files, err = list.Fields(googleapi.Field(fields)).Do()
files, err = list.Fields(googleapi.Field(fields)).Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -778,7 +779,7 @@ func parseExtensions(extensionsIn ...string) (extensions, mimeTypes []string, er
}
// Figure out if the user wants to use a team drive
func configTeamDrive(opt *Options, m configmap.Mapper, name string) error {
func configTeamDrive(ctx context.Context, opt *Options, m configmap.Mapper, name string) error {
// Stop if we are running non-interactive config
if fs.Config.AutoConfirm {
return nil
@@ -806,7 +807,7 @@ func configTeamDrive(opt *Options, m configmap.Mapper, name string) error {
for {
var teamDrives *drive.TeamDriveList
err = newPacer(opt).Call(func() (bool, error) {
teamDrives, err = listTeamDrives.Do()
teamDrives, err = listTeamDrives.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -1734,7 +1735,7 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
}
} else {
// Upload the file in chunks
info, err = f.Upload(in, size, srcMimeType, "", remote, createInfo)
info, err = f.Upload(ctx, in, size, srcMimeType, "", remote, createInfo)
if err != nil {
return nil, err
}
@@ -1975,7 +1976,7 @@ func (f *Fs) Purge(ctx context.Context) error {
// CleanUp empties the trash
func (f *Fs) CleanUp(ctx context.Context) error {
err := f.pacer.Call(func() (bool, error) {
err := f.svc.Files.EmptyTrash().Do()
err := f.svc.Files.EmptyTrash().Context(ctx).Do()
return shouldRetry(err)
})
@@ -1994,7 +1995,7 @@ func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
var about *drive.About
var err error
err = f.pacer.Call(func() (bool, error) {
about, err = f.svc.About.Get().Fields("storageQuota").Do()
about, err = f.svc.About.Get().Fields("storageQuota").Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -2253,7 +2254,7 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
}
}
fs.Debugf(f, "Checking for changes on remote")
startPageToken, err = f.changeNotifyRunner(notifyFunc, startPageToken)
startPageToken, err = f.changeNotifyRunner(ctx, notifyFunc, startPageToken)
if err != nil {
fs.Infof(f, "Change notify listener failure: %s", err)
}
@@ -2275,7 +2276,7 @@ func (f *Fs) changeNotifyStartPageToken() (pageToken string, err error) {
return startPageToken.StartPageToken, nil
}
func (f *Fs) changeNotifyRunner(notifyFunc func(string, fs.EntryType), startPageToken string) (newStartPageToken string, err error) {
func (f *Fs) changeNotifyRunner(ctx context.Context, notifyFunc func(string, fs.EntryType), startPageToken string) (newStartPageToken string, err error) {
pageToken := startPageToken
for {
var changeList *drive.ChangeList
@@ -2291,7 +2292,7 @@ func (f *Fs) changeNotifyRunner(notifyFunc func(string, fs.EntryType), startPage
if f.isTeamDrive {
changesCall.TeamDriveId(f.opt.TeamDriveID)
}
changeList, err = changesCall.Do()
changeList, err = changesCall.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -2499,7 +2500,7 @@ func (o *baseObject) Storable() bool {
// httpResponse gets an http.Response object for the object
// using the url and method passed in
func (o *baseObject) httpResponse(url, method string, options []fs.OpenOption) (req *http.Request, res *http.Response, err error) {
func (o *baseObject) httpResponse(ctx context.Context, url, method string, options []fs.OpenOption) (req *http.Request, res *http.Response, err error) {
if url == "" {
return nil, nil, errors.New("forbidden to download - check sharing permission")
}
@@ -2507,6 +2508,7 @@ func (o *baseObject) httpResponse(url, method string, options []fs.OpenOption) (
if err != nil {
return req, nil, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
fs.OpenOptionAddHTTPHeaders(req.Header, options)
if o.bytes == 0 {
// Don't supply range requests for 0 length objects as they always fail
@@ -2577,8 +2579,8 @@ func isGoogleError(err error, what string) bool {
}
// open a url for reading
func (o *baseObject) open(url string, options ...fs.OpenOption) (in io.ReadCloser, err error) {
_, res, err := o.httpResponse(url, "GET", options)
func (o *baseObject) open(ctx context.Context, url string, options ...fs.OpenOption) (in io.ReadCloser, err error) {
_, res, err := o.httpResponse(ctx, url, "GET", options)
if err != nil {
if isGoogleError(err, "cannotDownloadAbusiveFile") {
if o.fs.opt.AcknowledgeAbuse {
@@ -2589,7 +2591,7 @@ func (o *baseObject) open(url string, options ...fs.OpenOption) (in io.ReadClose
url += "?"
}
url += "acknowledgeAbuse=true"
_, res, err = o.httpResponse(url, "GET", options)
_, res, err = o.httpResponse(ctx, url, "GET", options)
} else {
err = errors.Wrap(err, "Use the --drive-acknowledge-abuse flag to download this file")
}
@@ -2618,7 +2620,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
o.v2Download = false
}
}
return o.baseObject.open(o.url, options...)
return o.baseObject.open(ctx, o.url, options...)
}
func (o *documentObject) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) {
// Update the size with what we are reading as it can change from
@@ -2643,7 +2645,7 @@ func (o *documentObject) Open(ctx context.Context, options ...fs.OpenOption) (in
if offset != 0 {
return nil, errors.New("partial downloads are not supported while exporting Google Documents")
}
in, err = o.baseObject.open(o.url, options...)
in, err = o.baseObject.open(ctx, o.url, options...)
if in != nil {
in = &openDocumentFile{o: o, in: in}
}
@@ -2678,7 +2680,7 @@ func (o *linkObject) Open(ctx context.Context, options ...fs.OpenOption) (in io.
return ioutil.NopCloser(bytes.NewReader(data)), nil
}
func (o *baseObject) update(updateInfo *drive.File, uploadMimeType string, in io.Reader,
func (o *baseObject) update(ctx context.Context, updateInfo *drive.File, uploadMimeType string, in io.Reader,
src fs.ObjectInfo) (info *drive.File, err error) {
// Make the API request to upload metadata and file data.
size := src.Size()
@@ -2696,7 +2698,7 @@ func (o *baseObject) update(updateInfo *drive.File, uploadMimeType string, in io
return
}
// Upload the file in chunks
return o.fs.Upload(in, size, uploadMimeType, o.id, o.remote, updateInfo)
return o.fs.Upload(ctx, in, size, uploadMimeType, o.id, o.remote, updateInfo)
}
// Update the already existing object
@@ -2710,7 +2712,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
MimeType: srcMimeType,
ModifiedTime: src.ModTime(ctx).Format(timeFormatOut),
}
info, err := o.baseObject.update(updateInfo, srcMimeType, in, src)
info, err := o.baseObject.update(ctx, updateInfo, srcMimeType, in, src)
if err != nil {
return err
}
@@ -2747,7 +2749,7 @@ func (o *documentObject) Update(ctx context.Context, in io.Reader, src fs.Object
}
updateInfo.MimeType = importMimeType
info, err := o.baseObject.update(updateInfo, srcMimeType, in, src)
info, err := o.baseObject.update(ctx, updateInfo, srcMimeType, in, src)
if err != nil {
return err
}
+12 -9
View File
@@ -11,6 +11,7 @@
package drive
import (
"context"
"encoding/json"
"fmt"
"io"
@@ -50,7 +51,7 @@ type resumableUpload struct {
}
// Upload the io.Reader in of size bytes with contentType and info
func (f *Fs) Upload(in io.Reader, size int64, contentType, fileID, remote string, info *drive.File) (*drive.File, error) {
func (f *Fs) Upload(ctx context.Context, in io.Reader, size int64, contentType, fileID, remote string, info *drive.File) (*drive.File, error) {
params := url.Values{
"alt": {"json"},
"uploadType": {"resumable"},
@@ -81,6 +82,7 @@ func (f *Fs) Upload(in io.Reader, size int64, contentType, fileID, remote string
if err != nil {
return false, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
googleapi.Expand(req.URL, map[string]string{
"fileId": fileID,
})
@@ -106,12 +108,13 @@ func (f *Fs) Upload(in io.Reader, size int64, contentType, fileID, remote string
MediaType: contentType,
ContentLength: size,
}
return rx.Upload()
return rx.Upload(ctx)
}
// Make an http.Request for the range passed in
func (rx *resumableUpload) makeRequest(start int64, body io.ReadSeeker, reqSize int64) *http.Request {
func (rx *resumableUpload) makeRequest(ctx context.Context, start int64, body io.ReadSeeker, reqSize int64) *http.Request {
req, _ := http.NewRequest("POST", rx.URI, body)
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
req.ContentLength = reqSize
if reqSize != 0 {
req.Header.Set("Content-Range", fmt.Sprintf("bytes %v-%v/%v", start, start+reqSize-1, rx.ContentLength))
@@ -129,8 +132,8 @@ var rangeRE = regexp.MustCompile(`^0\-(\d+)$`)
// Query drive for the amount transferred so far
//
// If error is nil, then start should be valid
func (rx *resumableUpload) transferStatus() (start int64, err error) {
req := rx.makeRequest(0, nil, 0)
func (rx *resumableUpload) transferStatus(ctx context.Context) (start int64, err error) {
req := rx.makeRequest(ctx, 0, nil, 0)
res, err := rx.f.client.Do(req)
if err != nil {
return 0, err
@@ -157,9 +160,9 @@ func (rx *resumableUpload) transferStatus() (start int64, err error) {
}
// Transfer a chunk - caller must call googleapi.CloseBody(res) if err == nil || res != nil
func (rx *resumableUpload) transferChunk(start int64, chunk io.ReadSeeker, chunkSize int64) (int, error) {
func (rx *resumableUpload) transferChunk(ctx context.Context, start int64, chunk io.ReadSeeker, chunkSize int64) (int, error) {
_, _ = chunk.Seek(0, io.SeekStart)
req := rx.makeRequest(start, chunk, chunkSize)
req := rx.makeRequest(ctx, start, chunk, chunkSize)
res, err := rx.f.client.Do(req)
if err != nil {
return 599, err
@@ -192,7 +195,7 @@ func (rx *resumableUpload) transferChunk(start int64, chunk io.ReadSeeker, chunk
// Upload uploads the chunks from the input
// It retries each chunk using the pacer and --low-level-retries
func (rx *resumableUpload) Upload() (*drive.File, error) {
func (rx *resumableUpload) Upload(ctx context.Context) (*drive.File, error) {
start := int64(0)
var StatusCode int
var err error
@@ -207,7 +210,7 @@ func (rx *resumableUpload) Upload() (*drive.File, error) {
// Transfer the chunk
err = rx.f.pacer.Call(func() (bool, error) {
fs.Debugf(rx.remote, "Sending chunk %d length %d", start, reqSize)
StatusCode, err = rx.transferChunk(start, chunk, reqSize)
StatusCode, err = rx.transferChunk(ctx, start, chunk, reqSize)
again, err := shouldRetry(err)
if StatusCode == statusResumeIncomplete || StatusCode == http.StatusCreated || StatusCode == http.StatusOK {
again = false
+21 -21
View File
@@ -32,7 +32,7 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
var isAlphaNumeric = regexp.MustCompile(`^[a-zA-Z0-9]+$`).MatchString
func (f *Fs) getDownloadToken(url string) (*GetTokenResponse, error) {
func (f *Fs) getDownloadToken(ctx context.Context, url string) (*GetTokenResponse, error) {
request := DownloadRequest{
URL: url,
Single: 1,
@@ -44,7 +44,7 @@ func (f *Fs) getDownloadToken(url string) (*GetTokenResponse, error) {
var token GetTokenResponse
err := f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, &request, &token)
resp, err := f.rest.CallJSON(ctx, &opts, &request, &token)
return shouldRetry(resp, err)
})
if err != nil {
@@ -72,7 +72,7 @@ func (f *Fs) listSharedFiles(ctx context.Context, id string) (entries fs.DirEntr
var sharedFiles SharedFolderResponse
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, nil, &sharedFiles)
resp, err := f.rest.CallJSON(ctx, &opts, nil, &sharedFiles)
return shouldRetry(resp, err)
})
if err != nil {
@@ -88,7 +88,7 @@ func (f *Fs) listSharedFiles(ctx context.Context, id string) (entries fs.DirEntr
return entries, nil
}
func (f *Fs) listFiles(directoryID int) (filesList *FilesList, err error) {
func (f *Fs) listFiles(ctx context.Context, directoryID int) (filesList *FilesList, err error) {
// fs.Debugf(f, "Requesting files for dir `%s`", directoryID)
request := ListFilesRequest{
FolderID: directoryID,
@@ -101,7 +101,7 @@ func (f *Fs) listFiles(directoryID int) (filesList *FilesList, err error) {
filesList = &FilesList{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, &request, filesList)
resp, err := f.rest.CallJSON(ctx, &opts, &request, filesList)
return shouldRetry(resp, err)
})
if err != nil {
@@ -111,7 +111,7 @@ func (f *Fs) listFiles(directoryID int) (filesList *FilesList, err error) {
return filesList, nil
}
func (f *Fs) listFolders(directoryID int) (foldersList *FoldersList, err error) {
func (f *Fs) listFolders(ctx context.Context, directoryID int) (foldersList *FoldersList, err error) {
// fs.Debugf(f, "Requesting folders for id `%s`", directoryID)
request := ListFolderRequest{
@@ -125,7 +125,7 @@ func (f *Fs) listFolders(directoryID int) (foldersList *FoldersList, err error)
foldersList = &FoldersList{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, &request, foldersList)
resp, err := f.rest.CallJSON(ctx, &opts, &request, foldersList)
return shouldRetry(resp, err)
})
if err != nil {
@@ -153,12 +153,12 @@ func (f *Fs) listDir(ctx context.Context, dir string) (entries fs.DirEntries, er
return nil, err
}
files, err := f.listFiles(folderID)
files, err := f.listFiles(ctx, folderID)
if err != nil {
return nil, err
}
folders, err := f.listFolders(folderID)
folders, err := f.listFolders(ctx, folderID)
if err != nil {
return nil, err
}
@@ -205,7 +205,7 @@ func getRemote(dir, fileName string) string {
return dir + "/" + fileName
}
func (f *Fs) makeFolder(leaf string, folderID int) (response *MakeFolderResponse, err error) {
func (f *Fs) makeFolder(ctx context.Context, leaf string, folderID int) (response *MakeFolderResponse, err error) {
name := replaceReservedChars(leaf)
// fs.Debugf(f, "Creating folder `%s` in id `%s`", name, directoryID)
@@ -221,7 +221,7 @@ func (f *Fs) makeFolder(leaf string, folderID int) (response *MakeFolderResponse
response = &MakeFolderResponse{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, &request, response)
resp, err := f.rest.CallJSON(ctx, &opts, &request, response)
return shouldRetry(resp, err)
})
if err != nil {
@@ -233,7 +233,7 @@ func (f *Fs) makeFolder(leaf string, folderID int) (response *MakeFolderResponse
return response, err
}
func (f *Fs) removeFolder(name string, folderID int) (response *GenericOKResponse, err error) {
func (f *Fs) removeFolder(ctx context.Context, name string, folderID int) (response *GenericOKResponse, err error) {
// fs.Debugf(f, "Removing folder with id `%s`", directoryID)
request := &RemoveFolderRequest{
@@ -248,7 +248,7 @@ func (f *Fs) removeFolder(name string, folderID int) (response *GenericOKRespons
response = &GenericOKResponse{}
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.rest.CallJSON(&opts, request, response)
resp, err = f.rest.CallJSON(ctx, &opts, request, response)
return shouldRetry(resp, err)
})
if err != nil {
@@ -263,7 +263,7 @@ func (f *Fs) removeFolder(name string, folderID int) (response *GenericOKRespons
return response, nil
}
func (f *Fs) deleteFile(url string) (response *GenericOKResponse, err error) {
func (f *Fs) deleteFile(ctx context.Context, url string) (response *GenericOKResponse, err error) {
request := &RemoveFileRequest{
Files: []RmFile{
{url},
@@ -277,7 +277,7 @@ func (f *Fs) deleteFile(url string) (response *GenericOKResponse, err error) {
response = &GenericOKResponse{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, request, response)
resp, err := f.rest.CallJSON(ctx, &opts, request, response)
return shouldRetry(resp, err)
})
@@ -290,7 +290,7 @@ func (f *Fs) deleteFile(url string) (response *GenericOKResponse, err error) {
return response, nil
}
func (f *Fs) getUploadNode() (response *GetUploadNodeResponse, err error) {
func (f *Fs) getUploadNode(ctx context.Context) (response *GetUploadNodeResponse, err error) {
// fs.Debugf(f, "Requesting Upload node")
opts := rest.Opts{
@@ -301,7 +301,7 @@ func (f *Fs) getUploadNode() (response *GetUploadNodeResponse, err error) {
response = &GetUploadNodeResponse{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, nil, response)
resp, err := f.rest.CallJSON(ctx, &opts, nil, response)
return shouldRetry(resp, err)
})
if err != nil {
@@ -313,7 +313,7 @@ func (f *Fs) getUploadNode() (response *GetUploadNodeResponse, err error) {
return response, err
}
func (f *Fs) uploadFile(in io.Reader, size int64, fileName, folderID, uploadID, node string) (response *http.Response, err error) {
func (f *Fs) uploadFile(ctx context.Context, in io.Reader, size int64, fileName, folderID, uploadID, node string) (response *http.Response, err error) {
// fs.Debugf(f, "Uploading File `%s`", fileName)
fileName = replaceReservedChars(fileName)
@@ -343,7 +343,7 @@ func (f *Fs) uploadFile(in io.Reader, size int64, fileName, folderID, uploadID,
}
err = f.pacer.CallNoRetry(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, nil, nil)
resp, err := f.rest.CallJSON(ctx, &opts, nil, nil)
return shouldRetry(resp, err)
})
@@ -356,7 +356,7 @@ func (f *Fs) uploadFile(in io.Reader, size int64, fileName, folderID, uploadID,
return response, err
}
func (f *Fs) endUpload(uploadID string, nodeurl string) (response *EndFileUploadResponse, err error) {
func (f *Fs) endUpload(ctx context.Context, uploadID string, nodeurl string) (response *EndFileUploadResponse, err error) {
// fs.Debugf(f, "Ending File Upload `%s`", uploadID)
if len(uploadID) > 10 || !isAlphaNumeric(uploadID) {
@@ -377,7 +377,7 @@ func (f *Fs) endUpload(uploadID string, nodeurl string) (response *EndFileUpload
response = &EndFileUploadResponse{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.rest.CallJSON(&opts, nil, response)
resp, err := f.rest.CallJSON(ctx, &opts, nil, response)
return shouldRetry(resp, err)
})
+8 -8
View File
@@ -74,7 +74,7 @@ func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut strin
if err != nil {
return "", false, err
}
folders, err := f.listFolders(folderID)
folders, err := f.listFolders(ctx, folderID)
if err != nil {
return "", false, err
}
@@ -95,7 +95,7 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
if err != nil {
return "", err
}
resp, err := f.makeFolder(leaf, folderID)
resp, err := f.makeFolder(ctx, leaf, folderID)
if err != nil {
return "", err
}
@@ -251,7 +251,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
if err != nil {
return nil, err
}
files, err := f.listFiles(folderID)
files, err := f.listFiles(ctx, folderID)
if err != nil {
return nil, err
}
@@ -298,13 +298,13 @@ func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options .
// This will create a duplicate if we upload a new file without
// checking to see if there is one already - use Put() for that.
func (f *Fs) putUnchecked(ctx context.Context, in io.Reader, remote string, size int64, options ...fs.OpenOption) (fs.Object, error) {
if size > int64(100E9) {
if size > int64(100e9) {
return nil, errors.New("File too big, cant upload")
} else if size == 0 {
return nil, fs.ErrorCantUploadEmptyFiles
}
nodeResponse, err := f.getUploadNode()
nodeResponse, err := f.getUploadNode(ctx)
if err != nil {
return nil, err
}
@@ -314,12 +314,12 @@ func (f *Fs) putUnchecked(ctx context.Context, in io.Reader, remote string, size
return nil, err
}
_, err = f.uploadFile(in, size, leaf, directoryID, nodeResponse.ID, nodeResponse.URL)
_, err = f.uploadFile(ctx, in, size, leaf, directoryID, nodeResponse.ID, nodeResponse.URL)
if err != nil {
return nil, err
}
fileUploadResponse, err := f.endUpload(nodeResponse.ID, nodeResponse.URL)
fileUploadResponse, err := f.endUpload(ctx, nodeResponse.ID, nodeResponse.URL)
if err != nil {
return nil, err
}
@@ -393,7 +393,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return err
}
_, err = f.removeFolder(dir, folderID)
_, err = f.removeFolder(ctx, dir, folderID)
if err != nil {
return err
}
+3 -3
View File
@@ -75,7 +75,7 @@ func (o *Object) SetModTime(context.Context, time.Time) error {
// Open opens the file for read. Call Close() on the returned io.ReadCloser
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadCloser, error) {
fs.FixRangeOption(options, int64(o.file.Size))
downloadToken, err := o.fs.getDownloadToken(o.file.URL)
downloadToken, err := o.fs.getDownloadToken(ctx, o.file.URL)
if err != nil {
return nil, err
@@ -89,7 +89,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadClo
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.rest.Call(&opts)
resp, err = o.fs.rest.Call(ctx, &opts)
return shouldRetry(resp, err)
})
@@ -131,7 +131,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
func (o *Object) Remove(ctx context.Context) error {
// fs.Debugf(f, "Removing file `%s` with url `%s`", o.file.Filename, o.file.URL)
_, err := o.fs.deleteFile(o.file.URL)
_, err := o.fs.deleteFile(ctx, o.file.URL)
if err != nil {
return err
@@ -374,6 +374,7 @@ func (f *Fs) setRoot(root string) {
// NewFs constructs an Fs from the path, bucket:path
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
ctx := context.TODO()
var oAuthClient *http.Client
// Parse config into Options struct
@@ -438,7 +439,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
if f.rootBucket != "" && f.rootDirectory != "" {
// Check to see if the object exists
err = f.pacer.Call(func() (bool, error) {
_, err = f.svc.Objects.Get(f.rootBucket, f.rootDirectory).Do()
_, err = f.svc.Objects.Get(f.rootBucket, f.rootDirectory).Context(ctx).Do()
return shouldRetry(err)
})
if err == nil {
@@ -457,7 +458,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// Return an Object from a path
//
// If it can't be found it returns the error fs.ErrorObjectNotFound.
func (f *Fs) newObjectWithInfo(remote string, info *storage.Object) (fs.Object, error) {
func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *storage.Object) (fs.Object, error) {
o := &Object{
fs: f,
remote: remote,
@@ -465,7 +466,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *storage.Object) (fs.Object,
if info != nil {
o.setMetaData(info)
} else {
err := o.readMetaData() // reads info and meta, returning an error
err := o.readMetaData(ctx) // reads info and meta, returning an error
if err != nil {
return nil, err
}
@@ -476,7 +477,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *storage.Object) (fs.Object,
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
return f.newObjectWithInfo(remote, nil)
return f.newObjectWithInfo(ctx, remote, nil)
}
// listFn is called from list to handle an object.
@@ -504,7 +505,7 @@ func (f *Fs) list(ctx context.Context, bucket, directory, prefix string, addBuck
for {
var objects *storage.Objects
err = f.pacer.Call(func() (bool, error) {
objects, err = list.Do()
objects, err = list.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -563,12 +564,12 @@ func (f *Fs) list(ctx context.Context, bucket, directory, prefix string, addBuck
}
// Convert a list item into a DirEntry
func (f *Fs) itemToDirEntry(remote string, object *storage.Object, isDirectory bool) (fs.DirEntry, error) {
func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *storage.Object, isDirectory bool) (fs.DirEntry, error) {
if isDirectory {
d := fs.NewDir(remote, time.Time{}).SetSize(int64(object.Size))
return d, nil
}
o, err := f.newObjectWithInfo(remote, object)
o, err := f.newObjectWithInfo(ctx, remote, object)
if err != nil {
return nil, err
}
@@ -579,7 +580,7 @@ func (f *Fs) itemToDirEntry(remote string, object *storage.Object, isDirectory b
func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addBucket bool) (entries fs.DirEntries, err error) {
// List the objects
err = f.list(ctx, bucket, directory, prefix, addBucket, false, func(remote string, object *storage.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
entry, err := f.itemToDirEntry(ctx, remote, object, isDirectory)
if err != nil {
return err
}
@@ -605,7 +606,7 @@ func (f *Fs) listBuckets(ctx context.Context) (entries fs.DirEntries, err error)
for {
var buckets *storage.Buckets
err = f.pacer.Call(func() (bool, error) {
buckets, err = listBuckets.Do()
buckets, err = listBuckets.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -664,7 +665,7 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
list := walk.NewListRHelper(callback)
listR := func(bucket, directory, prefix string, addBucket bool) error {
return f.list(ctx, bucket, directory, prefix, addBucket, true, func(remote string, object *storage.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
entry, err := f.itemToDirEntry(ctx, remote, object, isDirectory)
if err != nil {
return err
}
@@ -731,7 +732,7 @@ func (f *Fs) makeBucket(ctx context.Context, bucket string) (err error) {
// List something from the bucket to see if it exists. Doing it like this enables the use of a
// service account that only has the "Storage Object Admin" role. See #2193 for details.
err = f.pacer.Call(func() (bool, error) {
_, err = f.svc.Objects.List(bucket).MaxResults(1).Do()
_, err = f.svc.Objects.List(bucket).MaxResults(1).Context(ctx).Do()
return shouldRetry(err)
})
if err == nil {
@@ -766,7 +767,7 @@ func (f *Fs) makeBucket(ctx context.Context, bucket string) (err error) {
if !f.opt.BucketPolicyOnly {
insertBucket.PredefinedAcl(f.opt.BucketACL)
}
_, err = insertBucket.Do()
_, err = insertBucket.Context(ctx).Do()
return shouldRetry(err)
})
}, nil)
@@ -783,7 +784,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) (err error) {
}
return f.cache.Remove(bucket, func() error {
return f.pacer.Call(func() (bool, error) {
err = f.svc.Buckets.Delete(bucket).Do()
err = f.svc.Buckets.Delete(bucket).Context(ctx).Do()
return shouldRetry(err)
})
})
@@ -824,7 +825,11 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
var newObject *storage.Object
err = f.pacer.Call(func() (bool, error) {
newObject, err = f.svc.Objects.Copy(srcBucket, srcPath, dstBucket, dstPath, nil).Do()
copyObject := f.svc.Objects.Copy(srcBucket, srcPath, dstBucket, dstPath, nil)
if !f.opt.BucketPolicyOnly {
copyObject.DestinationPredefinedAcl(f.opt.ObjectACL)
}
newObject, err = copyObject.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -907,25 +912,33 @@ func (o *Object) setMetaData(info *storage.Object) {
}
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData() (err error) {
if !o.modTime.IsZero() {
return nil
}
// readObjectInfo reads the definition for an object
func (o *Object) readObjectInfo(ctx context.Context) (object *storage.Object, err error) {
bucket, bucketPath := o.split()
var object *storage.Object
err = o.fs.pacer.Call(func() (bool, error) {
object, err = o.fs.svc.Objects.Get(bucket, bucketPath).Do()
object, err = o.fs.svc.Objects.Get(bucket, bucketPath).Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
if gErr, ok := err.(*googleapi.Error); ok {
if gErr.Code == http.StatusNotFound {
return fs.ErrorObjectNotFound
return nil, fs.ErrorObjectNotFound
}
}
return nil, err
}
return object, nil
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData(ctx context.Context) (err error) {
if !o.modTime.IsZero() {
return nil
}
object, err := o.readObjectInfo(ctx)
if err != nil {
return err
}
o.setMetaData(object)
@@ -937,7 +950,7 @@ func (o *Object) readMetaData() (err error) {
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
err := o.readMetaData()
err := o.readMetaData(ctx)
if err != nil {
// fs.Logf(o, "Failed to read metadata: %v", err)
return time.Now()
@@ -954,16 +967,27 @@ func metadataFromModTime(modTime time.Time) map[string]string {
// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) (err error) {
// This only adds metadata so will perserve other metadata
bucket, bucketPath := o.split()
object := storage.Object{
Bucket: bucket,
Name: bucketPath,
Metadata: metadataFromModTime(modTime),
// read the complete existing object first
object, err := o.readObjectInfo(ctx)
if err != nil {
return err
}
// Add the mtime to the existing metadata
mtime := modTime.Format(timeFormatOut)
if object.Metadata == nil {
object.Metadata = make(map[string]string, 1)
}
object.Metadata[metaMtime] = mtime
// Copy the object to itself to update the metadata
// Using PATCH requires too many permissions
bucket, bucketPath := o.split()
var newObject *storage.Object
err = o.fs.pacer.Call(func() (bool, error) {
newObject, err = o.fs.svc.Objects.Patch(bucket, bucketPath, &object).Do()
copyObject := o.fs.svc.Objects.Copy(bucket, bucketPath, bucket, bucketPath, object)
if !o.fs.opt.BucketPolicyOnly {
copyObject.DestinationPredefinedAcl(o.fs.opt.ObjectACL)
}
newObject, err = copyObject.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -984,6 +1008,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
if err != nil {
return nil, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
fs.FixRangeOption(options, o.bytes)
fs.OpenOptionAddHTTPHeaders(req.Header, options)
var res *http.Response
@@ -1031,7 +1056,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
if !o.fs.opt.BucketPolicyOnly {
insertObject.PredefinedAcl(o.fs.opt.ObjectACL)
}
newObject, err = insertObject.Do()
newObject, err = insertObject.Context(ctx).Do()
return shouldRetry(err)
})
if err != nil {
@@ -1046,7 +1071,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
func (o *Object) Remove(ctx context.Context) (err error) {
bucket, bucketPath := o.split()
err = o.fs.pacer.Call(func() (bool, error) {
err = o.fs.svc.Objects.Delete(bucket, bucketPath).Do()
err = o.fs.svc.Objects.Delete(bucket, bucketPath).Context(ctx).Do()
return shouldRetry(err)
})
return err
+20 -21
View File
@@ -290,7 +290,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
// fetchEndpoint gets the openid endpoint named from the Google config
func (f *Fs) fetchEndpoint(name string) (endpoint string, err error) {
func (f *Fs) fetchEndpoint(ctx context.Context, name string) (endpoint string, err error) {
// Get openID config without auth
opts := rest.Opts{
Method: "GET",
@@ -298,7 +298,7 @@ func (f *Fs) fetchEndpoint(name string) (endpoint string, err error) {
}
var openIDconfig map[string]interface{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.unAuth.CallJSON(&opts, nil, &openIDconfig)
resp, err := f.unAuth.CallJSON(ctx, &opts, nil, &openIDconfig)
return shouldRetry(resp, err)
})
if err != nil {
@@ -316,7 +316,7 @@ func (f *Fs) fetchEndpoint(name string) (endpoint string, err error) {
// UserInfo fetches info about the current user with oauth2
func (f *Fs) UserInfo(ctx context.Context) (userInfo map[string]string, err error) {
endpoint, err := f.fetchEndpoint("userinfo_endpoint")
endpoint, err := f.fetchEndpoint(ctx, "userinfo_endpoint")
if err != nil {
return nil, err
}
@@ -327,7 +327,7 @@ func (f *Fs) UserInfo(ctx context.Context) (userInfo map[string]string, err erro
RootURL: endpoint,
}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, nil, &userInfo)
resp, err := f.srv.CallJSON(ctx, &opts, nil, &userInfo)
return shouldRetry(resp, err)
})
if err != nil {
@@ -338,7 +338,7 @@ func (f *Fs) UserInfo(ctx context.Context) (userInfo map[string]string, err erro
// Disconnect kills the token and refresh token
func (f *Fs) Disconnect(ctx context.Context) (err error) {
endpoint, err := f.fetchEndpoint("revocation_endpoint")
endpoint, err := f.fetchEndpoint(ctx, "revocation_endpoint")
if err != nil {
return err
}
@@ -358,7 +358,7 @@ func (f *Fs) Disconnect(ctx context.Context) (err error) {
}
var res interface{}
err = f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(&opts, nil, &res)
resp, err := f.srv.CallJSON(ctx, &opts, nil, &res)
return shouldRetry(resp, err)
})
if err != nil {
@@ -423,7 +423,7 @@ func findID(name string) string {
// list the albums into an internal cache
// FIXME cache invalidation
func (f *Fs) listAlbums(shared bool) (all *albums, err error) {
func (f *Fs) listAlbums(ctx context.Context, shared bool) (all *albums, err error) {
f.albumsMu.Lock()
defer f.albumsMu.Unlock()
all, ok := f.albums[shared]
@@ -445,7 +445,7 @@ func (f *Fs) listAlbums(shared bool) (all *albums, err error) {
var result api.ListAlbums
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -482,7 +482,7 @@ type listFn func(remote string, object *api.MediaItem, isDirectory bool) error
// dir is the starting directory, "" for root
//
// Set recurse to read sub directories
func (f *Fs) list(filter api.SearchFilter, fn listFn) (err error) {
func (f *Fs) list(ctx context.Context, filter api.SearchFilter, fn listFn) (err error) {
opts := rest.Opts{
Method: "POST",
Path: "/mediaItems:search",
@@ -494,7 +494,7 @@ func (f *Fs) list(filter api.SearchFilter, fn listFn) (err error) {
var result api.MediaItems
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &filter, &result)
resp, err = f.srv.CallJSON(ctx, &opts, &filter, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -543,7 +543,7 @@ func (f *Fs) itemToDirEntry(ctx context.Context, remote string, item *api.MediaI
// listDir lists a single directory
func (f *Fs) listDir(ctx context.Context, prefix string, filter api.SearchFilter) (entries fs.DirEntries, err error) {
// List the objects
err = f.list(filter, func(remote string, item *api.MediaItem, isDirectory bool) error {
err = f.list(ctx, filter, func(remote string, item *api.MediaItem, isDirectory bool) error {
entry, err := f.itemToDirEntry(ctx, prefix+remote, item, isDirectory)
if err != nil {
return err
@@ -638,7 +638,7 @@ func (f *Fs) createAlbum(ctx context.Context, albumTitle string) (album *api.Alb
var result api.Album
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, request, &result)
resp, err = f.srv.CallJSON(ctx, &opts, request, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -654,7 +654,7 @@ func (f *Fs) createAlbum(ctx context.Context, albumTitle string) (album *api.Alb
func (f *Fs) getOrCreateAlbum(ctx context.Context, albumTitle string) (album *api.Album, err error) {
f.createMu.Lock()
defer f.createMu.Unlock()
albums, err := f.listAlbums(false)
albums, err := f.listAlbums(ctx, false)
if err != nil {
return nil, err
}
@@ -708,7 +708,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) (err error) {
return err
}
albumTitle := match[1]
allAlbums, err := f.listAlbums(false)
allAlbums, err := f.listAlbums(ctx, false)
if err != nil {
return err
}
@@ -773,7 +773,7 @@ func (o *Object) Size() int64 {
RootURL: o.downloadURL(),
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -824,7 +824,7 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
var item api.MediaItem
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &item)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &item)
return shouldRetry(resp, err)
})
if err != nil {
@@ -901,7 +901,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -954,9 +954,8 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
var token []byte
var resp *http.Response
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
if err != nil {
_ = resp.Body.Close()
return shouldRetry(resp, err)
}
token, err = rest.ReadBody(resp)
@@ -987,7 +986,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
var result api.BatchCreateResponse
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, request, &result)
resp, err = o.fs.srv.CallJSON(ctx, &opts, request, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1030,7 +1029,7 @@ func (o *Object) Remove(ctx context.Context) (err error) {
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, &request, nil)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &request, nil)
return shouldRetry(resp, err)
})
if err != nil {
+2 -2
View File
@@ -20,7 +20,7 @@ import (
// file pattern parsing
type lister interface {
listDir(ctx context.Context, prefix string, filter api.SearchFilter) (entries fs.DirEntries, err error)
listAlbums(shared bool) (all *albums, err error)
listAlbums(ctx context.Context, shared bool) (all *albums, err error)
listUploads(ctx context.Context, dir string) (entries fs.DirEntries, err error)
dirTime() time.Time
}
@@ -296,7 +296,7 @@ func yearMonthDayFilter(ctx context.Context, f lister, match []string) (sf api.S
// is a prefix of another album, or actual files, or a combination of
// the two.
func albumsToEntries(ctx context.Context, f lister, shared bool, prefix string, albumPath string) (entries fs.DirEntries, err error) {
albums, err := f.listAlbums(shared)
albums, err := f.listAlbums(ctx, shared)
if err != nil {
return nil, err
}
+1 -1
View File
@@ -44,7 +44,7 @@ func (f *testLister) listDir(ctx context.Context, prefix string, filter api.Sear
}
// mock listAlbums for testing
func (f *testLister) listAlbums(shared bool) (all *albums, err error) {
func (f *testLister) listAlbums(ctx context.Context, shared bool) (all *albums, err error) {
return f.albums, nil
}
+72 -20
View File
@@ -13,6 +13,7 @@ import (
"path"
"strconv"
"strings"
"sync"
"time"
"github.com/pkg/errors"
@@ -77,6 +78,26 @@ Note that this may cause rclone to confuse genuine HTML files with
directories.`,
Default: false,
Advanced: true,
}, {
Name: "no_head",
Help: `Don't use HEAD requests to find file sizes in dir listing
If your site is being very slow to load then you can try this option.
Normally rclone does a HEAD request for each potential file in a
directory listing to:
- find its size
- check it really exists
- check to see if it is a directory
If you set this option, rclone will not do the HEAD request. This will mean
- directory listings are much quicker
- rclone won't have the times or sizes of any files
- some files that don't exist may be in the listing
`,
Default: false,
Advanced: true,
}},
}
fs.Register(fsi)
@@ -86,6 +107,7 @@ directories.`,
type Options struct {
Endpoint string `config:"url"`
NoSlash bool `config:"no_slash"`
NoHead bool `config:"no_head"`
Headers fs.CommaSepList `config:"headers"`
}
@@ -124,6 +146,7 @@ func statusError(res *http.Response, err error) error {
// NewFs creates a new Fs object from the name and root. It connects to
// the host specified in the config file.
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
ctx := context.TODO()
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
@@ -162,6 +185,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// check to see if points to a file
req, err := http.NewRequest("HEAD", u.String(), nil)
if err == nil {
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
addHeaders(req, opt)
res, err := noRedir.Do(req)
err = statusError(res, err)
@@ -237,7 +261,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
fs: f,
remote: remote,
}
err := o.stat()
err := o.stat(ctx)
if err != nil {
return nil, err
}
@@ -355,7 +379,7 @@ func (f *Fs) addHeaders(req *http.Request) {
}
// Read the directory passed in
func (f *Fs) readDir(dir string) (names []string, err error) {
func (f *Fs) readDir(ctx context.Context, dir string) (names []string, err error) {
URL := f.url(dir)
u, err := url.Parse(URL)
if err != nil {
@@ -369,6 +393,7 @@ func (f *Fs) readDir(dir string) (names []string, err error) {
if err != nil {
return nil, errors.Wrap(err, "readDir failed")
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
f.addHeaders(req)
res, err := f.httpClient.Do(req)
if err == nil {
@@ -408,34 +433,53 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
if !strings.HasSuffix(dir, "/") && dir != "" {
dir += "/"
}
names, err := f.readDir(dir)
names, err := f.readDir(ctx, dir)
if err != nil {
return nil, errors.Wrapf(err, "error listing %q", dir)
}
var (
entriesMu sync.Mutex // to protect entries
wg sync.WaitGroup
in = make(chan string, fs.Config.Checkers)
)
add := func(entry fs.DirEntry) {
entriesMu.Lock()
entries = append(entries, entry)
entriesMu.Unlock()
}
for i := 0; i < fs.Config.Checkers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for remote := range in {
file := &Object{
fs: f,
remote: remote,
}
switch err := file.stat(ctx); err {
case nil:
add(file)
case fs.ErrorNotAFile:
// ...found a directory not a file
add(fs.NewDir(remote, timeUnset))
default:
fs.Debugf(remote, "skipping because of error: %v", err)
}
}
}()
}
for _, name := range names {
isDir := name[len(name)-1] == '/'
name = strings.TrimRight(name, "/")
remote := path.Join(dir, name)
if isDir {
dir := fs.NewDir(remote, timeUnset)
entries = append(entries, dir)
add(fs.NewDir(remote, timeUnset))
} else {
file := &Object{
fs: f,
remote: remote,
}
switch err = file.stat(); err {
case nil:
entries = append(entries, file)
case fs.ErrorNotAFile:
// ...found a directory not a file
dir := fs.NewDir(remote, timeUnset)
entries = append(entries, dir)
default:
fs.Debugf(remote, "skipping because of error: %v", err)
}
in <- remote
}
}
close(in)
wg.Wait()
return entries, nil
}
@@ -492,12 +536,19 @@ func (o *Object) url() string {
}
// stat updates the info field in the Object
func (o *Object) stat() error {
func (o *Object) stat(ctx context.Context) error {
if o.fs.opt.NoHead {
o.size = -1
o.modTime = timeUnset
o.contentType = fs.MimeType(ctx, o)
return nil
}
url := o.url()
req, err := http.NewRequest("HEAD", url, nil)
if err != nil {
return errors.Wrap(err, "stat failed")
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
o.fs.addHeaders(req)
res, err := o.fs.httpClient.Do(req)
if err == nil && res.StatusCode == http.StatusNotFound {
@@ -546,6 +597,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
if err != nil {
return nil, errors.Wrap(err, "Open failed")
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
// Add optional headers
for k, v := range fs.OpenOptionHeaders(options) {
+2 -1
View File
@@ -1,6 +1,7 @@
package hubic
import (
"context"
"net/http"
"time"
@@ -26,7 +27,7 @@ func newAuth(f *Fs) *auth {
func (a *auth) Request(*swift.Connection) (r *http.Request, err error) {
const retries = 10
for try := 1; try <= retries; try++ {
err = a.f.getCredentials()
err = a.f.getCredentials(context.TODO())
if err == nil {
break
}
+3 -1
View File
@@ -7,6 +7,7 @@ package hubic
// to be revisted after some actual experience.
import (
"context"
"encoding/json"
"fmt"
"io/ioutil"
@@ -115,11 +116,12 @@ func (f *Fs) String() string {
// getCredentials reads the OpenStack Credentials using the Hubic API
//
// The credentials are read into the Fs
func (f *Fs) getCredentials() (err error) {
func (f *Fs) getCredentials(ctx context.Context) (err error) {
req, err := http.NewRequest("GET", "https://api.hubic.com/1.0/account/credentials", nil)
if err != nil {
return err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
resp, err := f.client.Do(req)
if err != nil {
return err
+56 -53
View File
@@ -77,6 +77,7 @@ func init() {
Description: "JottaCloud",
NewFs: NewFs,
Config: func(name string, m configmap.Mapper) {
ctx := context.TODO()
tokenString, ok := m.Get("token")
if ok && tokenString != "" {
fmt.Printf("Already have a token - refresh?\n")
@@ -88,7 +89,7 @@ func init() {
srv := rest.NewClient(fshttp.NewClient(fs.Config))
fmt.Printf("\nDo you want to create a machine specific API key?\n\nRclone has it's own Jottacloud API KEY which works fine as long as one only uses rclone on a single machine. When you want to use rclone with this account on more than one machine it's recommended to create a machine specific API key. These keys can NOT be shared between machines.\n\n")
if config.Confirm() {
deviceRegistration, err := registerDevice(srv)
deviceRegistration, err := registerDevice(ctx, srv)
if err != nil {
log.Fatalf("Failed to register device: %v", err)
}
@@ -113,7 +114,7 @@ func init() {
username := config.ReadLine()
password := config.GetPassword("Your Jottacloud password is only required during setup and will not be stored.")
token, err := doAuth(srv, username, password)
token, err := doAuth(ctx, srv, username, password)
if err != nil {
log.Fatalf("Failed to get oauth token: %s", err)
}
@@ -132,7 +133,7 @@ func init() {
srv = rest.NewClient(oAuthClient).SetRoot(rootURL)
apiSrv := rest.NewClient(oAuthClient).SetRoot(apiURL)
device, mountpoint, err := setupMountpoint(srv, apiSrv)
device, mountpoint, err := setupMountpoint(ctx, srv, apiSrv)
if err != nil {
log.Fatalf("Failed to setup mountpoint: %s", err)
}
@@ -246,7 +247,7 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
}
// registerDevice register a new device for use with the jottacloud API
func registerDevice(srv *rest.Client) (reg *api.DeviceRegistrationResponse, err error) {
func registerDevice(ctx context.Context, srv *rest.Client) (reg *api.DeviceRegistrationResponse, err error) {
// random generator to generate random device names
seededRand := rand.New(rand.NewSource(time.Now().UnixNano()))
randonDeviceNamePartLength := 21
@@ -269,12 +270,12 @@ func registerDevice(srv *rest.Client) (reg *api.DeviceRegistrationResponse, err
}
var deviceRegistration *api.DeviceRegistrationResponse
_, err = srv.CallJSON(&opts, nil, &deviceRegistration)
_, err = srv.CallJSON(ctx, &opts, nil, &deviceRegistration)
return deviceRegistration, err
}
// doAuth runs the actual token request
func doAuth(srv *rest.Client, username, password string) (token oauth2.Token, err error) {
func doAuth(ctx context.Context, srv *rest.Client, username, password string) (token oauth2.Token, err error) {
// prepare out token request with username and password
values := url.Values{}
values.Set("grant_type", "PASSWORD")
@@ -291,7 +292,7 @@ func doAuth(srv *rest.Client, username, password string) (token oauth2.Token, er
// do the first request
var jsonToken api.TokenJSON
resp, err := srv.CallJSON(&opts, nil, &jsonToken)
resp, err := srv.CallJSON(ctx, &opts, nil, &jsonToken)
if err != nil {
// if 2fa is enabled the first request is expected to fail. We will do another request with the 2fa code as an additional http header
if resp != nil {
@@ -303,7 +304,7 @@ func doAuth(srv *rest.Client, username, password string) (token oauth2.Token, er
authCode = strings.Replace(authCode, "-", "", -1) // remove any "-" contained in the code so we have a 6 digit number
opts.ExtraHeaders = make(map[string]string)
opts.ExtraHeaders["X-Jottacloud-Otp"] = authCode
resp, err = srv.CallJSON(&opts, nil, &jsonToken)
resp, err = srv.CallJSON(ctx, &opts, nil, &jsonToken)
}
}
}
@@ -316,13 +317,13 @@ func doAuth(srv *rest.Client, username, password string) (token oauth2.Token, er
}
// setupMountpoint sets up a custom device and mountpoint if desired by the user
func setupMountpoint(srv *rest.Client, apiSrv *rest.Client) (device, mountpoint string, err error) {
cust, err := getCustomerInfo(apiSrv)
func setupMountpoint(ctx context.Context, srv *rest.Client, apiSrv *rest.Client) (device, mountpoint string, err error) {
cust, err := getCustomerInfo(ctx, apiSrv)
if err != nil {
return "", "", err
}
acc, err := getDriveInfo(srv, cust.Username)
acc, err := getDriveInfo(ctx, srv, cust.Username)
if err != nil {
return "", "", err
}
@@ -333,7 +334,7 @@ func setupMountpoint(srv *rest.Client, apiSrv *rest.Client) (device, mountpoint
fmt.Printf("Please select the device to use. Normally this will be Jotta\n")
device = config.Choose("Devices", deviceNames, nil, false)
dev, err := getDeviceInfo(srv, path.Join(cust.Username, device))
dev, err := getDeviceInfo(ctx, srv, path.Join(cust.Username, device))
if err != nil {
return "", "", err
}
@@ -351,13 +352,13 @@ func setupMountpoint(srv *rest.Client, apiSrv *rest.Client) (device, mountpoint
}
// getCustomerInfo queries general information about the account
func getCustomerInfo(srv *rest.Client) (info *api.CustomerInfo, err error) {
func getCustomerInfo(ctx context.Context, srv *rest.Client) (info *api.CustomerInfo, err error) {
opts := rest.Opts{
Method: "GET",
Path: "account/v1/customer",
}
_, err = srv.CallJSON(&opts, nil, &info)
_, err = srv.CallJSON(ctx, &opts, nil, &info)
if err != nil {
return nil, errors.Wrap(err, "couldn't get customer info")
}
@@ -366,13 +367,13 @@ func getCustomerInfo(srv *rest.Client) (info *api.CustomerInfo, err error) {
}
// getDriveInfo queries general information about the account and the available devices and mountpoints.
func getDriveInfo(srv *rest.Client, username string) (info *api.DriveInfo, err error) {
func getDriveInfo(ctx context.Context, srv *rest.Client, username string) (info *api.DriveInfo, err error) {
opts := rest.Opts{
Method: "GET",
Path: username,
}
_, err = srv.CallXML(&opts, nil, &info)
_, err = srv.CallXML(ctx, &opts, nil, &info)
if err != nil {
return nil, errors.Wrap(err, "couldn't get drive info")
}
@@ -381,13 +382,13 @@ func getDriveInfo(srv *rest.Client, username string) (info *api.DriveInfo, err e
}
// getDeviceInfo queries Information about a jottacloud device
func getDeviceInfo(srv *rest.Client, path string) (info *api.JottaDevice, err error) {
func getDeviceInfo(ctx context.Context, srv *rest.Client, path string) (info *api.JottaDevice, err error) {
opts := rest.Opts{
Method: "GET",
Path: urlPathEscape(path),
}
_, err = srv.CallXML(&opts, nil, &info)
_, err = srv.CallXML(ctx, &opts, nil, &info)
if err != nil {
return nil, errors.Wrap(err, "couldn't get device info")
}
@@ -407,7 +408,7 @@ func (f *Fs) setEndpointURL() {
}
// readMetaDataForPath reads the metadata from the path
func (f *Fs) readMetaDataForPath(path string) (info *api.JottaFile, err error) {
func (f *Fs) readMetaDataForPath(ctx context.Context, path string) (info *api.JottaFile, err error) {
opts := rest.Opts{
Method: "GET",
Path: f.filePath(path),
@@ -415,7 +416,7 @@ func (f *Fs) readMetaDataForPath(path string) (info *api.JottaFile, err error) {
var result api.JottaFile
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &result)
resp, err = f.srv.CallXML(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
@@ -492,6 +493,7 @@ func grantTypeFilter(req *http.Request) {
// NewFs constructs an Fs from the path, container:path
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
ctx := context.TODO()
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
@@ -546,11 +548,11 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// Renew the token in the background
f.tokenRenewer = oauthutil.NewRenew(f.String(), ts, func() error {
_, err := f.readMetaDataForPath("")
_, err := f.readMetaDataForPath(ctx, "")
return err
})
cust, err := getCustomerInfo(f.apiSrv)
cust, err := getCustomerInfo(ctx, f.apiSrv)
if err != nil {
return nil, err
}
@@ -582,7 +584,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// Return an Object from a path
//
// If it can't be found it returns the error fs.ErrorObjectNotFound.
func (f *Fs) newObjectWithInfo(remote string, info *api.JottaFile) (fs.Object, error) {
func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *api.JottaFile) (fs.Object, error) {
o := &Object{
fs: f,
remote: remote,
@@ -592,7 +594,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *api.JottaFile) (fs.Object, e
// Set info
err = o.setMetaData(info)
} else {
err = o.readMetaData(false) // reads info and meta, returning an error
err = o.readMetaData(ctx, false) // reads info and meta, returning an error
}
if err != nil {
return nil, err
@@ -603,11 +605,11 @@ func (f *Fs) newObjectWithInfo(remote string, info *api.JottaFile) (fs.Object, e
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
return f.newObjectWithInfo(remote, nil)
return f.newObjectWithInfo(ctx, remote, nil)
}
// CreateDir makes a directory
func (f *Fs) CreateDir(path string) (jf *api.JottaFolder, err error) {
func (f *Fs) CreateDir(ctx context.Context, path string) (jf *api.JottaFolder, err error) {
// fs.Debugf(f, "CreateDir(%q, %q)\n", pathID, leaf)
var resp *http.Response
opts := rest.Opts{
@@ -619,7 +621,7 @@ func (f *Fs) CreateDir(path string) (jf *api.JottaFolder, err error) {
opts.Parameters.Set("mkDir", "true")
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &jf)
resp, err = f.srv.CallXML(ctx, &opts, nil, &jf)
return shouldRetry(resp, err)
})
if err != nil {
@@ -648,7 +650,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
var resp *http.Response
var result api.JottaFolder
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &result)
resp, err = f.srv.CallXML(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
@@ -682,7 +684,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
continue
}
remote := path.Join(dir, restoreReservedChars(item.Name))
o, err := f.newObjectWithInfo(remote, item)
o, err := f.newObjectWithInfo(ctx, remote, item)
if err != nil {
continue
}
@@ -696,7 +698,7 @@ type listFileDirFn func(fs.DirEntry) error
// List the objects and directories into entries, from a
// special kind of JottaFolder representing a FileDirLis
func (f *Fs) listFileDir(remoteStartPath string, startFolder *api.JottaFolder, fn listFileDirFn) error {
func (f *Fs) listFileDir(ctx context.Context, remoteStartPath string, startFolder *api.JottaFolder, fn listFileDirFn) error {
pathPrefix := "/" + f.filePathRaw("") // Non-escaped prefix of API paths to be cut off, to be left with the remote path including the remoteStartPath
pathPrefixLength := len(pathPrefix)
startPath := path.Join(pathPrefix, remoteStartPath) // Non-escaped API path up to and including remoteStartPath, to decide if it should be created as a new dir object
@@ -725,7 +727,7 @@ func (f *Fs) listFileDir(remoteStartPath string, startFolder *api.JottaFolder, f
continue
}
remoteFile := path.Join(remoteDir, restoreReservedChars(file.Name))
o, err := f.newObjectWithInfo(remoteFile, file)
o, err := f.newObjectWithInfo(ctx, remoteFile, file)
if err != nil {
return err
}
@@ -754,7 +756,7 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
var resp *http.Response
var result api.JottaFolder // Could be JottaFileDirList, but JottaFolder is close enough
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &result)
resp, err = f.srv.CallXML(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -767,7 +769,7 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
return errors.Wrap(err, "couldn't list files")
}
list := walk.NewListRHelper(callback)
err = f.listFileDir(dir, &result, func(entry fs.DirEntry) error {
err = f.listFileDir(ctx, dir, &result, func(entry fs.DirEntry) error {
return list.Add(entry)
})
if err != nil {
@@ -821,7 +823,7 @@ func (f *Fs) mkParentDir(ctx context.Context, dirPath string) error {
// Mkdir creates the container if it doesn't exist
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
_, err := f.CreateDir(dir)
_, err := f.CreateDir(ctx, dir)
return err
}
@@ -860,7 +862,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) (err error)
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -888,7 +890,7 @@ func (f *Fs) Purge(ctx context.Context) error {
}
// copyOrMoves copies or moves directories or files depending on the method parameter
func (f *Fs) copyOrMove(method, src, dest string) (info *api.JottaFile, err error) {
func (f *Fs) copyOrMove(ctx context.Context, method, src, dest string) (info *api.JottaFile, err error) {
opts := rest.Opts{
Method: "POST",
Path: src,
@@ -899,7 +901,7 @@ func (f *Fs) copyOrMove(method, src, dest string) (info *api.JottaFile, err erro
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &info)
resp, err = f.srv.CallXML(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -928,13 +930,13 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
if err != nil {
return nil, err
}
info, err := f.copyOrMove("cp", srcObj.filePath(), remote)
info, err := f.copyOrMove(ctx, "cp", srcObj.filePath(), remote)
if err != nil {
return nil, errors.Wrap(err, "couldn't copy file")
}
return f.newObjectWithInfo(remote, info)
return f.newObjectWithInfo(ctx, remote, info)
//return f.newObjectWithInfo(remote, &result)
}
@@ -958,13 +960,13 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
if err != nil {
return nil, err
}
info, err := f.copyOrMove("mv", srcObj.filePath(), remote)
info, err := f.copyOrMove(ctx, "mv", srcObj.filePath(), remote)
if err != nil {
return nil, errors.Wrap(err, "couldn't move file")
}
return f.newObjectWithInfo(remote, info)
return f.newObjectWithInfo(ctx, remote, info)
//return f.newObjectWithInfo(remote, result)
}
@@ -1002,7 +1004,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
return fs.ErrorDirExists
}
_, err = f.copyOrMove("mvDir", path.Join(f.endpointURL, replaceReservedChars(srcPath))+"/", dstRemote)
_, err = f.copyOrMove(ctx, "mvDir", path.Join(f.endpointURL, replaceReservedChars(srcPath))+"/", dstRemote)
if err != nil {
return errors.Wrap(err, "couldn't move directory")
@@ -1027,7 +1029,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (link string, err er
var resp *http.Response
var result api.JottaFile
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &result)
resp, err = f.srv.CallXML(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
@@ -1058,7 +1060,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (link string, err er
// About gets quota information
func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
info, err := getDriveInfo(f.srv, f.user)
info, err := getDriveInfo(ctx, f.srv, f.user)
if err != nil {
return nil, err
}
@@ -1113,7 +1115,8 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
// Size returns the size of an object in bytes
func (o *Object) Size() int64 {
err := o.readMetaData(false)
ctx := context.TODO()
err := o.readMetaData(ctx, false)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return 0
@@ -1137,11 +1140,11 @@ func (o *Object) setMetaData(info *api.JottaFile) (err error) {
}
// readMetaData reads and updates the metadata for an object
func (o *Object) readMetaData(force bool) (err error) {
func (o *Object) readMetaData(ctx context.Context, force bool) (err error) {
if o.hasMetaData && !force {
return nil
}
info, err := o.fs.readMetaDataForPath(o.remote)
info, err := o.fs.readMetaDataForPath(ctx, o.remote)
if err != nil {
return err
}
@@ -1156,7 +1159,7 @@ func (o *Object) readMetaData(force bool) (err error) {
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
err := o.readMetaData(false)
err := o.readMetaData(ctx, false)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return time.Now()
@@ -1188,7 +1191,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
opts.Parameters.Set("mode", "bin")
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1298,7 +1301,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// send it
var response api.AllocateFileResponse
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.apiSrv.CallJSON(&opts, &request, &response)
resp, err = o.fs.apiSrv.CallJSON(ctx, &opts, &request, &response)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1329,7 +1332,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
// send the remaining bytes
resp, err = o.fs.apiSrv.CallJSON(&opts, nil, &result)
resp, err = o.fs.apiSrv.CallJSON(ctx, &opts, nil, &result)
if err != nil {
return err
}
@@ -1341,7 +1344,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
o.modTime = time.Unix(result.Modified/1000, 0)
} else {
// If the file state is COMPLETE we don't need to upload it because the file was already found but we still ned to update our metadata
return o.readMetaData(true)
return o.readMetaData(ctx, true)
}
return nil
@@ -1363,7 +1366,7 @@ func (o *Object) Remove(ctx context.Context) error {
}
return o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.CallXML(&opts, nil, nil)
resp, err := o.fs.srv.CallXML(ctx, &opts, nil, nil)
return shouldRetry(resp, err)
})
}
+107
View File
@@ -0,0 +1,107 @@
package api
// BIN protocol constants
const (
BinContentType = "application/x-www-form-urlencoded"
TreeIDLength = 12
DunnoNodeIDLength = 16
)
// Operations in binary protocol
const (
OperationAddFile = 103 // 0x67
OperationRename = 105 // 0x69
OperationCreateFolder = 106 // 0x6A
OperationFolderList = 117 // 0x75
OperationSharedFoldersList = 121 // 0x79
// TODO investigate opcodes below
Operation154MaybeItemInfo = 154 // 0x9A
Operation102MaybeAbout = 102 // 0x66
Operation104MaybeDelete = 104 // 0x68
)
// CreateDir protocol constants
const (
MkdirResultOK = 0
MkdirResultSourceNotExists = 1
MkdirResultAlreadyExists = 4
MkdirResultExistsDifferentCase = 9
MkdirResultInvalidName = 10
MkdirResultFailed254 = 254
)
// Move result codes
const (
MoveResultOK = 0
MoveResultSourceNotExists = 1
MoveResultFailed002 = 2
MoveResultAlreadyExists = 4
MoveResultFailed005 = 5
MoveResultFailed254 = 254
)
// AddFile result codes
const (
AddResultOK = 0
AddResultError01 = 1
AddResultDunno04 = 4
AddResultWrongPath = 5
AddResultNoFreeSpace = 7
AddResultDunno09 = 9
AddResultInvalidName = 10
AddResultNotModified = 12
AddResultFailedA = 253
AddResultFailedB = 254
)
// List request options
const (
ListOptTotalSpace = 1
ListOptDelete = 2
ListOptFingerprint = 4
ListOptUnknown8 = 8
ListOptUnknown16 = 16
ListOptFolderSize = 32
ListOptUsedSpace = 64
ListOptUnknown128 = 128
ListOptUnknown256 = 256
)
// ListOptDefaults ...
const ListOptDefaults = ListOptUnknown128 | ListOptUnknown256 | ListOptFolderSize | ListOptTotalSpace | ListOptUsedSpace
// List parse flags
const (
ListParseDone = 0
ListParseReadItem = 1
ListParsePin = 2
ListParsePinUpper = 3
ListParseUnknown15 = 15
)
// List operation results
const (
ListResultOK = 0
ListResultNotExists = 1
ListResultDunno02 = 2
ListResultDunno03 = 3
ListResultAlreadyExists04 = 4
ListResultDunno05 = 5
ListResultDunno06 = 6
ListResultDunno07 = 7
ListResultDunno08 = 8
ListResultAlreadyExists09 = 9
ListResultDunno10 = 10
ListResultDunno11 = 11
ListResultDunno12 = 12
ListResultFailedB = 253
ListResultFailedA = 254
)
// Directory item types
const (
ListItemMountPoint = 0
ListItemFile = 1
ListItemFolder = 2
ListItemSharedFolder = 3
)
+225
View File
@@ -0,0 +1,225 @@
package api
// BIN protocol helpers
import (
"bufio"
"bytes"
"encoding/binary"
"io"
"log"
"time"
"github.com/pkg/errors"
"github.com/rclone/rclone/lib/readers"
)
// protocol errors
var (
ErrorPrematureEOF = errors.New("Premature EOF")
ErrorInvalidLength = errors.New("Invalid length")
ErrorZeroTerminate = errors.New("String must end with zero")
)
// BinWriter is a binary protocol writer
type BinWriter struct {
b *bytes.Buffer // growing byte buffer
a []byte // temporary buffer for next varint
}
// NewBinWriter creates a binary protocol helper
func NewBinWriter() *BinWriter {
return &BinWriter{
b: new(bytes.Buffer),
a: make([]byte, binary.MaxVarintLen64),
}
}
// Bytes returns binary data
func (w *BinWriter) Bytes() []byte {
return w.b.Bytes()
}
// Reader returns io.Reader with binary data
func (w *BinWriter) Reader() io.Reader {
return bytes.NewReader(w.b.Bytes())
}
// WritePu16 writes a short as unsigned varint
func (w *BinWriter) WritePu16(val int) {
if val < 0 || val > 65535 {
log.Fatalf("Invalid UInt16 %v", val)
}
w.WritePu64(int64(val))
}
// WritePu32 writes a signed long as unsigned varint
func (w *BinWriter) WritePu32(val int64) {
if val < 0 || val > 4294967295 {
log.Fatalf("Invalid UInt32 %v", val)
}
w.WritePu64(val)
}
// WritePu64 writes an unsigned (actually, signed) long as unsigned varint
func (w *BinWriter) WritePu64(val int64) {
if val < 0 {
log.Fatalf("Invalid UInt64 %v", val)
}
w.b.Write(w.a[:binary.PutUvarint(w.a, uint64(val))])
}
// WriteString writes a zero-terminated string
func (w *BinWriter) WriteString(str string) {
buf := []byte(str)
w.WritePu64(int64(len(buf) + 1))
w.b.Write(buf)
w.b.WriteByte(0)
}
// Write writes a byte buffer
func (w *BinWriter) Write(buf []byte) {
w.b.Write(buf)
}
// WriteWithLength writes a byte buffer prepended with its length as varint
func (w *BinWriter) WriteWithLength(buf []byte) {
w.WritePu64(int64(len(buf)))
w.b.Write(buf)
}
// BinReader is a binary protocol reader helper
type BinReader struct {
b *bufio.Reader
count *readers.CountingReader
err error // keeps the first error encountered
}
// NewBinReader creates a binary protocol reader helper
func NewBinReader(reader io.Reader) *BinReader {
r := &BinReader{}
r.count = readers.NewCountingReader(reader)
r.b = bufio.NewReader(r.count)
return r
}
// Count returns number of bytes read
func (r *BinReader) Count() uint64 {
return r.count.BytesRead()
}
// Error returns first encountered error or nil
func (r *BinReader) Error() error {
return r.err
}
// check() keeps the first error encountered in a stream
func (r *BinReader) check(err error) bool {
if err == nil {
return true
}
if r.err == nil {
// keep the first error
r.err = err
}
if err != io.EOF {
log.Fatalf("Error parsing response: %v", err)
}
return false
}
// ReadByteAsInt reads a single byte as uint32, returns -1 for EOF or errors
func (r *BinReader) ReadByteAsInt() int {
if octet, err := r.b.ReadByte(); r.check(err) {
return int(octet)
}
return -1
}
// ReadByteAsShort reads a single byte as uint16, returns -1 for EOF or errors
func (r *BinReader) ReadByteAsShort() int16 {
if octet, err := r.b.ReadByte(); r.check(err) {
return int16(octet)
}
return -1
}
// ReadIntSpl reads two bytes as little-endian uint16, returns -1 for EOF or errors
func (r *BinReader) ReadIntSpl() int {
var val uint16
if r.check(binary.Read(r.b, binary.LittleEndian, &val)) {
return int(val)
}
return -1
}
// ReadULong returns uint64 equivalent of -1 for EOF or errors
func (r *BinReader) ReadULong() uint64 {
if val, err := binary.ReadUvarint(r.b); r.check(err) {
return val
}
return 0xffffffffffffffff
}
// ReadPu32 returns -1 for EOF or errors
func (r *BinReader) ReadPu32() int64 {
if val, err := binary.ReadUvarint(r.b); r.check(err) {
return int64(val)
}
return -1
}
// ReadNBytes reads given number of bytes, returns invalid data for EOF or errors
func (r *BinReader) ReadNBytes(len int) []byte {
buf := make([]byte, len)
n, err := r.b.Read(buf)
if r.check(err) {
return buf
}
if n != len {
r.check(ErrorPrematureEOF)
}
return buf
}
// ReadBytesByLength reads buffer length and its bytes
func (r *BinReader) ReadBytesByLength() []byte {
len := r.ReadPu32()
if len < 0 {
r.check(ErrorInvalidLength)
return []byte{}
}
return r.ReadNBytes(int(len))
}
// ReadString reads a zero-terminated string with length
func (r *BinReader) ReadString() string {
len := int(r.ReadPu32())
if len < 1 {
r.check(ErrorInvalidLength)
return ""
}
buf := make([]byte, len-1)
n, err := r.b.Read(buf)
if !r.check(err) {
return ""
}
if n != len-1 {
r.check(ErrorPrematureEOF)
return ""
}
zeroByte, err := r.b.ReadByte()
if !r.check(err) {
return ""
}
if zeroByte != 0 {
r.check(ErrorZeroTerminate)
return ""
}
return string(buf)
}
// ReadDate reads a Unix encoded time
func (r *BinReader) ReadDate() time.Time {
return time.Unix(r.ReadPu32(), 0)
}
+248
View File
@@ -0,0 +1,248 @@
package api
import (
"fmt"
)
// M1 protocol constants and structures
const (
APIServerURL = "https://cloud.mail.ru"
PublicLinkURL = "https://cloud.mail.ru/public/"
DispatchServerURL = "https://dispatcher.cloud.mail.ru"
OAuthURL = "https://o2.mail.ru/token"
OAuthClientID = "cloud-win"
)
// ServerErrorResponse represents erroneous API response.
type ServerErrorResponse struct {
Message string `json:"body"`
Time int64 `json:"time"`
Status int `json:"status"`
}
func (e *ServerErrorResponse) Error() string {
return fmt.Sprintf("server error %d (%s)", e.Status, e.Message)
}
// FileErrorResponse represents erroneous API response for a file
type FileErrorResponse struct {
Body struct {
Home struct {
Value string `json:"value"`
Error string `json:"error"`
} `json:"home"`
} `json:"body"`
Status int `json:"status"`
Account string `json:"email,omitempty"`
Time int64 `json:"time,omitempty"`
Message string // non-json, calculated field
}
func (e *FileErrorResponse) Error() string {
return fmt.Sprintf("file error %d (%s)", e.Status, e.Body.Home.Error)
}
// UserInfoResponse contains account metadata
type UserInfoResponse struct {
Body struct {
AccountType string `json:"account_type"`
AccountVerified bool `json:"account_verified"`
Cloud struct {
Beta struct {
Allowed bool `json:"allowed"`
Asked bool `json:"asked"`
} `json:"beta"`
Billing struct {
ActiveCostID string `json:"active_cost_id"`
ActiveRateID string `json:"active_rate_id"`
AutoProlong bool `json:"auto_prolong"`
Basequota int64 `json:"basequota"`
Enabled bool `json:"enabled"`
Expires int `json:"expires"`
Prolong bool `json:"prolong"`
Promocodes struct {
} `json:"promocodes"`
Subscription []interface{} `json:"subscription"`
Version string `json:"version"`
} `json:"billing"`
Bonuses struct {
CameraUpload bool `json:"camera_upload"`
Complete bool `json:"complete"`
Desktop bool `json:"desktop"`
Feedback bool `json:"feedback"`
Links bool `json:"links"`
Mobile bool `json:"mobile"`
Registration bool `json:"registration"`
} `json:"bonuses"`
Enable struct {
Sharing bool `json:"sharing"`
} `json:"enable"`
FileSizeLimit int64 `json:"file_size_limit"`
Space struct {
BytesTotal int64 `json:"bytes_total"`
BytesUsed int `json:"bytes_used"`
Overquota bool `json:"overquota"`
} `json:"space"`
} `json:"cloud"`
Cloudflags struct {
Exists bool `json:"exists"`
} `json:"cloudflags"`
Domain string `json:"domain"`
Login string `json:"login"`
Newbie bool `json:"newbie"`
UI struct {
ExpandLoader bool `json:"expand_loader"`
Kind string `json:"kind"`
Sidebar bool `json:"sidebar"`
Sort struct {
Order string `json:"order"`
Type string `json:"type"`
} `json:"sort"`
Thumbs bool `json:"thumbs"`
} `json:"ui"`
} `json:"body"`
Email string `json:"email"`
Status int `json:"status"`
Time int64 `json:"time"`
}
// ListItem ...
type ListItem struct {
Count struct {
Folders int `json:"folders"`
Files int `json:"files"`
} `json:"count,omitempty"`
Kind string `json:"kind"`
Type string `json:"type"`
Name string `json:"name"`
Home string `json:"home"`
Size int64 `json:"size"`
Mtime int64 `json:"mtime,omitempty"`
Hash string `json:"hash,omitempty"`
VirusScan string `json:"virus_scan,omitempty"`
Tree string `json:"tree,omitempty"`
Grev int `json:"grev,omitempty"`
Rev int `json:"rev,omitempty"`
}
// ItemInfoResponse ...
type ItemInfoResponse struct {
Email string `json:"email"`
Body ListItem `json:"body"`
Time int64 `json:"time"`
Status int `json:"status"`
}
// FolderInfoResponse ...
type FolderInfoResponse struct {
Body struct {
Count struct {
Folders int `json:"folders"`
Files int `json:"files"`
} `json:"count"`
Tree string `json:"tree"`
Name string `json:"name"`
Grev int `json:"grev"`
Size int64 `json:"size"`
Sort struct {
Order string `json:"order"`
Type string `json:"type"`
} `json:"sort"`
Kind string `json:"kind"`
Rev int `json:"rev"`
Type string `json:"type"`
Home string `json:"home"`
List []ListItem `json:"list"`
} `json:"body,omitempty"`
Time int64 `json:"time"`
Status int `json:"status"`
Email string `json:"email"`
}
// ShardInfoResponse ...
type ShardInfoResponse struct {
Email string `json:"email"`
Body struct {
Video []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"video"`
ViewDirect []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"view_direct"`
WeblinkView []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"weblink_view"`
WeblinkVideo []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"weblink_video"`
WeblinkGet []struct {
Count int `json:"count"`
URL string `json:"url"`
} `json:"weblink_get"`
Stock []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"stock"`
WeblinkThumbnails []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"weblink_thumbnails"`
PublicUpload []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"public_upload"`
Auth []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"auth"`
Web []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"web"`
View []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"view"`
Upload []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"upload"`
Get []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"get"`
Thumbnails []struct {
Count string `json:"count"`
URL string `json:"url"`
} `json:"thumbnails"`
} `json:"body"`
Time int64 `json:"time"`
Status int `json:"status"`
}
// CleanupResponse ...
type CleanupResponse struct {
Email string `json:"email"`
Time int64 `json:"time"`
StatusStr string `json:"status"`
}
// GenericResponse ...
type GenericResponse struct {
Email string `json:"email"`
Time int64 `json:"time"`
Status int `json:"status"`
// ignore other fields
}
// GenericBodyResponse ...
type GenericBodyResponse struct {
Email string `json:"email"`
Body string `json:"body"`
Time int64 `json:"time"`
Status int `json:"status"`
}
File diff suppressed because it is too large. Load diff
+18
View File
@@ -0,0 +1,18 @@
// Test Mailru filesystem interface
package mailru_test
import (
"testing"
"github.com/rclone/rclone/backend/mailru"
"github.com/rclone/rclone/fstest/fstests"
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestMailru:",
NilObject: (*mailru.Object)(nil),
SkipBadWindowsCharacters: true,
})
}
+134
View File
@@ -0,0 +1,134 @@
// Package mrhash implements the mailru hash, which is a modified SHA1.
// If file size is less than or equal to the SHA1 block size (20 bytes),
// its hash is simply its data right-padded with zero bytes.
// Hash sum of a larger file is computed as a SHA1 sum of the file data
// bytes concatenated with a decimal representation of the data length.
package mrhash
import (
"crypto/sha1"
"encoding"
"encoding/hex"
"errors"
"hash"
"strconv"
)
const (
// BlockSize of the checksum in bytes.
BlockSize = sha1.BlockSize
// Size of the checksum in bytes.
Size = sha1.Size
startString = "mrCloud"
hashError = "hash function returned error"
)
// Global errors
var (
ErrorInvalidHash = errors.New("invalid hash")
)
type digest struct {
total int // bytes written into hash so far
sha hash.Hash // underlying SHA1
small []byte // small content
}
// New returns a new hash.Hash computing the Mailru checksum.
func New() hash.Hash {
d := &digest{}
d.Reset()
return d
}
// Write writes len(p) bytes from p to the underlying data stream. It returns
// the number of bytes written from p (0 <= n <= len(p)) and any error
// encountered that caused the write to stop early. Write must return a non-nil
// error if it returns n < len(p). Write must not modify the slice data, even
// temporarily.
//
// Implementations must not retain p.
func (d *digest) Write(p []byte) (n int, err error) {
n, err = d.sha.Write(p)
if err != nil {
panic(hashError)
}
d.total += n
if d.total <= Size {
d.small = append(d.small, p...)
}
return n, nil
}
// Sum appends the current hash to b and returns the resulting slice.
// It does not change the underlying hash state.
func (d *digest) Sum(b []byte) []byte {
// If content is small, return it padded to Size
if d.total <= Size {
padded := make([]byte, Size)
copy(padded, d.small)
return append(b, padded...)
}
endString := strconv.Itoa(d.total)
copy, err := cloneSHA1(d.sha)
if err == nil {
_, err = copy.Write([]byte(endString))
}
if err != nil {
panic(hashError)
}
return copy.Sum(b)
}
// cloneSHA1 clones state of SHA1 hash
func cloneSHA1(orig hash.Hash) (clone hash.Hash, err error) {
state, err := orig.(encoding.BinaryMarshaler).MarshalBinary()
if err != nil {
return nil, err
}
clone = sha1.New()
err = clone.(encoding.BinaryUnmarshaler).UnmarshalBinary(state)
return
}
// Reset resets the Hash to its initial state.
func (d *digest) Reset() {
d.sha = sha1.New()
_, _ = d.sha.Write([]byte(startString))
d.total = 0
}
// Size returns the number of bytes Sum will return.
func (d *digest) Size() int {
return Size
}
// BlockSize returns the hash's underlying block size.
// The Write method must be able to accept any amount
// of data, but it may operate more efficiently if all writes
// are a multiple of the block size.
func (d *digest) BlockSize() int {
return BlockSize
}
// Sum returns the Mailru checksum of the data.
func Sum(data []byte) []byte {
var d digest
d.Reset()
_, _ = d.Write(data)
return d.Sum(nil)
}
// DecodeString converts a string to the Mailru hash
func DecodeString(s string) ([]byte, error) {
b, err := hex.DecodeString(s)
if err != nil || len(b) != Size {
return nil, ErrorInvalidHash
}
return b, nil
}
// must implement this interface
var (
_ hash.Hash = (*digest)(nil)
)
+81
View File
@@ -0,0 +1,81 @@
package mrhash_test
import (
"encoding/hex"
"fmt"
"testing"
"github.com/rclone/rclone/backend/mailru/mrhash"
"github.com/stretchr/testify/assert"
)
func testChunk(t *testing.T, chunk int) {
data := make([]byte, chunk)
for i := 0; i < chunk; i++ {
data[i] = 'A'
}
for _, test := range []struct {
n int
want string
}{
{0, "0000000000000000000000000000000000000000"},
{1, "4100000000000000000000000000000000000000"},
{2, "4141000000000000000000000000000000000000"},
{19, "4141414141414141414141414141414141414100"},
{20, "4141414141414141414141414141414141414141"},
{21, "eb1d05e78a18691a5aa196a6c2b60cd40b5faafb"},
{22, "037e6d960601118a0639afbeff30fe716c66ed2d"},
{4096, "45a16aa192502b010280fb5b44274c601a91fd9f"},
{4194303, "fa019d5bd26498cf6abe35e0d61801bf19bf704b"},
{4194304, "5ed0e07aa6ea5c1beb9402b4d807258f27d40773"},
{4194305, "67bd0b9247db92e0e7d7e29a0947a50fedcb5452"},
{8388607, "41a8e2eb044c2e242971b5445d7be2a13fc0dd84"},
{8388608, "267a970917c624c11fe624276ec60233a66dc2c0"},
{8388609, "37b60b308d553d2732aefb62b3ea88f74acfa13f"},
} {
d := mrhash.New()
var toWrite int
for toWrite = test.n; toWrite >= chunk; toWrite -= chunk {
n, err := d.Write(data)
assert.Nil(t, err)
assert.Equal(t, chunk, n)
}
n, err := d.Write(data[:toWrite])
assert.Nil(t, err)
assert.Equal(t, toWrite, n)
got1 := hex.EncodeToString(d.Sum(nil))
assert.Equal(t, test.want, got1, fmt.Sprintf("when testing length %d", n))
got2 := hex.EncodeToString(d.Sum(nil))
assert.Equal(t, test.want, got2, fmt.Sprintf("when testing length %d (2nd sum)", n))
}
}
func TestHashChunk16M(t *testing.T) { testChunk(t, 16*1024*1024) }
func TestHashChunk8M(t *testing.T) { testChunk(t, 8*1024*1024) }
func TestHashChunk4M(t *testing.T) { testChunk(t, 4*1024*1024) }
func TestHashChunk2M(t *testing.T) { testChunk(t, 2*1024*1024) }
func TestHashChunk1M(t *testing.T) { testChunk(t, 1*1024*1024) }
func TestHashChunk64k(t *testing.T) { testChunk(t, 64*1024) }
func TestHashChunk32k(t *testing.T) { testChunk(t, 32*1024) }
func TestHashChunk2048(t *testing.T) { testChunk(t, 2048) }
func TestHashChunk2047(t *testing.T) { testChunk(t, 2047) }
func TestSumCalledTwice(t *testing.T) {
d := mrhash.New()
assert.NotPanics(t, func() { d.Sum(nil) })
d.Reset()
assert.NotPanics(t, func() { d.Sum(nil) })
assert.NotPanics(t, func() { d.Sum(nil) })
_, _ = d.Write([]byte{1})
assert.NotPanics(t, func() { d.Sum(nil) })
}
func TestSize(t *testing.T) {
d := mrhash.New()
assert.Equal(t, 20, d.Size())
}
func TestBlockSize(t *testing.T) {
d := mrhash.New()
assert.Equal(t, 64, d.BlockSize())
}
+47 -44
View File
@@ -72,6 +72,7 @@ func init() {
Description: "Microsoft OneDrive",
NewFs: NewFs,
Config: func(name string, m configmap.Mapper) {
ctx := context.TODO()
err := oauthutil.Config("onedrive", name, m, oauthConfig)
if err != nil {
log.Fatalf("Failed to configure token: %v", err)
@@ -143,7 +144,7 @@ func init() {
}
sites := siteResponse{}
_, err := srv.CallJSON(&opts, nil, &sites)
_, err := srv.CallJSON(ctx, &opts, nil, &sites)
if err != nil {
log.Fatalf("Failed to query available sites: %v", err)
}
@@ -172,7 +173,7 @@ func init() {
// query Microsoft Graph
if finalDriveID == "" {
drives := drivesResponse{}
_, err := srv.CallJSON(&opts, nil, &drives)
_, err := srv.CallJSON(ctx, &opts, nil, &drives)
if err != nil {
log.Fatalf("Failed to query available drives: %v", err)
}
@@ -194,7 +195,7 @@ func init() {
RootURL: graphURL,
Path: "/drives/" + finalDriveID + "/root"}
var rootItem api.Item
_, err = srv.CallJSON(&opts, nil, &rootItem)
_, err = srv.CallJSON(ctx, &opts, nil, &rootItem)
if err != nil {
log.Fatalf("Failed to query root for drive %s: %v", finalDriveID, err)
}
@@ -343,10 +344,10 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
// instead of simply using `drives/driveID/root:/itemPath` because it works for
// "shared with me" folders in OneDrive Personal (See #2536, #2778)
// This path pattern comes from https://github.com/OneDrive/onedrive-api-docs/issues/908#issuecomment-417488480
func (f *Fs) readMetaDataForPathRelativeToID(normalizedID string, relPath string) (info *api.Item, resp *http.Response, err error) {
func (f *Fs) readMetaDataForPathRelativeToID(ctx context.Context, normalizedID string, relPath string) (info *api.Item, resp *http.Response, err error) {
opts := newOptsCall(normalizedID, "GET", ":/"+withTrailingColon(rest.URLPathEscape(replaceReservedChars(relPath))))
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
@@ -371,7 +372,7 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string) (info *api.It
}
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
return info, resp, err
@@ -426,7 +427,7 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string) (info *api.It
}
}
return f.readMetaDataForPathRelativeToID(baseNormalizedID, relPath)
return f.readMetaDataForPathRelativeToID(ctx, baseNormalizedID, relPath)
}
// errorHandler parses a non 2xx error response into an error
@@ -592,7 +593,7 @@ func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut strin
if !ok {
return "", false, errors.New("couldn't find parent ID")
}
info, resp, err := f.readMetaDataForPathRelativeToID(pathID, leaf)
info, resp, err := f.readMetaDataForPathRelativeToID(ctx, pathID, leaf)
if err != nil {
if resp != nil && resp.StatusCode == http.StatusNotFound {
return "", false, nil
@@ -619,7 +620,7 @@ func (f *Fs) CreateDir(ctx context.Context, dirID, leaf string) (newID string, e
ConflictBehavior: "fail",
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &mkdir, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &mkdir, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -642,7 +643,7 @@ type listAllFn func(*api.Item) bool
// Lists the directory required calling the user function on each item found
//
// If the user fn ever returns true then it early exits with found = true
func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
func (f *Fs) listAll(ctx context.Context, dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
// Top parameter asks for bigger pages of data
// https://dev.onedrive.com/odata/optional-query-parameters.htm
opts := newOptsCall(dirID, "GET", "/children?$top=1000")
@@ -651,7 +652,7 @@ OUTER:
var result api.ListChildrenResponse
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -709,7 +710,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, err
}
var iErr error
_, err = f.listAll(directoryID, false, false, func(info *api.Item) bool {
_, err = f.listAll(ctx, directoryID, false, false, func(info *api.Item) bool {
if !f.opt.ExposeOneNoteFiles && info.GetPackageType() == api.PackageTypeOneNote {
fs.Debugf(info.Name, "OneNote file not shown in directory listing")
return false
@@ -793,12 +794,12 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
}
// deleteObject removes an object by ID
func (f *Fs) deleteObject(id string) error {
func (f *Fs) deleteObject(ctx context.Context, id string) error {
opts := newOptsCall(id, "DELETE", "")
opts.NoResponse = true
return f.pacer.Call(func() (bool, error) {
resp, err := f.srv.Call(&opts)
resp, err := f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
}
@@ -821,7 +822,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
}
if check {
// check to see if there are any items
found, err := f.listAll(rootID, false, false, func(item *api.Item) bool {
found, err := f.listAll(ctx, rootID, false, false, func(item *api.Item) bool {
return true
})
if err != nil {
@@ -831,7 +832,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
return fs.ErrorDirectoryNotEmpty
}
}
err = f.deleteObject(rootID)
err = f.deleteObject(ctx, rootID)
if err != nil {
return err
}
@@ -941,7 +942,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
}
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &copyReq, nil)
resp, err = f.srv.CallJSON(ctx, &opts, &copyReq, nil)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1029,7 +1030,7 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
var resp *http.Response
var info api.Item
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &move, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &move, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1122,7 +1123,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
}
// Get timestamps of src so they can be preserved
srcInfo, _, err := srcFs.readMetaDataForPathRelativeToID(srcID, "")
srcInfo, _, err := srcFs.readMetaDataForPathRelativeToID(ctx, srcID, "")
if err != nil {
return err
}
@@ -1144,7 +1145,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
var resp *http.Response
var info api.Item
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &move, &info)
resp, err = f.srv.CallJSON(ctx, &opts, &move, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1170,7 +1171,7 @@ func (f *Fs) About(ctx context.Context) (usage *fs.Usage, err error) {
}
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &drive)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &drive)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1210,7 +1211,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (link string, err er
var resp *http.Response
var result api.CreateShareLinkResponse
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, &share, &result)
resp, err = f.srv.CallJSON(ctx, &opts, &share, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1370,7 +1371,7 @@ func (o *Object) setModTime(ctx context.Context, modTime time.Time) (*api.Item,
}
var info *api.Item
err := o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.CallJSON(&opts, &update, &info)
resp, err := o.fs.srv.CallJSON(ctx, &opts, &update, &info)
return shouldRetry(resp, err)
})
return info, err
@@ -1405,7 +1406,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
opts.Options = options
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1441,7 +1442,7 @@ func (o *Object) createUploadSession(ctx context.Context, modTime time.Time) (re
createRequest.Item.FileSystemInfo.LastModifiedDateTime = api.Timestamp(modTime)
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, &createRequest, &response)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &createRequest, &response)
if apiErr, ok := err.(*api.Error); ok {
if apiErr.ErrorInfo.Code == "nameAlreadyExists" {
// Make the error more user-friendly
@@ -1454,7 +1455,7 @@ func (o *Object) createUploadSession(ctx context.Context, modTime time.Time) (re
}
// uploadFragment uploads a part
func (o *Object) uploadFragment(url string, start int64, totalSize int64, chunk io.ReadSeeker, chunkSize int64) (info *api.Item, err error) {
func (o *Object) uploadFragment(ctx context.Context, url string, start int64, totalSize int64, chunk io.ReadSeeker, chunkSize int64) (info *api.Item, err error) {
opts := rest.Opts{
Method: "PUT",
RootURL: url,
@@ -1464,28 +1465,30 @@ func (o *Object) uploadFragment(url string, start int64, totalSize int64, chunk
}
// var response api.UploadFragmentResponse
var resp *http.Response
var body []byte
err = o.fs.pacer.Call(func() (bool, error) {
_, _ = chunk.Seek(0, io.SeekStart)
resp, err = o.fs.srv.Call(&opts)
if resp != nil {
defer fs.CheckClose(resp.Body, &err)
resp, err = o.fs.srv.Call(ctx, &opts)
if err != nil {
return shouldRetry(resp, err)
}
retry, err := shouldRetry(resp, err)
if !retry && resp != nil {
if resp.StatusCode == 200 || resp.StatusCode == 201 {
// we are done :)
// read the item
info = &api.Item{}
return false, json.NewDecoder(resp.Body).Decode(info)
}
body, err = rest.ReadBody(resp)
if err != nil {
return shouldRetry(resp, err)
}
return retry, err
if resp.StatusCode == 200 || resp.StatusCode == 201 {
// we are done :)
// read the item
info = &api.Item{}
return false, json.Unmarshal(body, info)
}
return false, nil
})
return info, err
}
// cancelUploadSession cancels an upload session
func (o *Object) cancelUploadSession(url string) (err error) {
func (o *Object) cancelUploadSession(ctx context.Context, url string) (err error) {
opts := rest.Opts{
Method: "DELETE",
RootURL: url,
@@ -1493,7 +1496,7 @@ func (o *Object) cancelUploadSession(url string) (err error) {
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
return
@@ -1515,7 +1518,7 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, size int64,
}
fs.Debugf(o, "Cancelling multipart upload")
cancelErr := o.cancelUploadSession(uploadURL)
cancelErr := o.cancelUploadSession(ctx, uploadURL)
if cancelErr != nil {
fs.Logf(o, "Failed to cancel multipart upload: %v", cancelErr)
}
@@ -1551,7 +1554,7 @@ func (o *Object) uploadMultipart(ctx context.Context, in io.Reader, size int64,
}
seg := readers.NewRepeatableReader(io.LimitReader(in, n))
fs.Debugf(o, "Uploading segment %d/%d size %d", position, size, n)
info, err = o.uploadFragment(uploadURL, position, size, seg, n)
info, err = o.uploadFragment(ctx, uploadURL, position, size, seg, n)
if err != nil {
return nil, err
}
@@ -1592,7 +1595,7 @@ func (o *Object) uploadSinglepart(ctx context.Context, in io.Reader, size int64,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &info)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &info)
if apiErr, ok := err.(*api.Error); ok {
if apiErr.ErrorInfo.Code == "nameAlreadyExists" {
// Make the error more user-friendly
@@ -1644,7 +1647,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
return o.fs.deleteObject(o.id)
return o.fs.deleteObject(ctx, o.id)
}
// MimeType of an Object if known, "" otherwise
+22 -22
View File
@@ -161,7 +161,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
Method: "POST",
Path: "/session/login.json",
}
resp, err = f.srv.CallJSON(&opts, &account, &f.session)
resp, err = f.srv.CallJSON(ctx, &opts, &account, &f.session)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -246,7 +246,7 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
}
// deleteObject removes an object by ID
func (f *Fs) deleteObject(id string) error {
func (f *Fs) deleteObject(ctx context.Context, id string) error {
return f.pacer.Call(func() (bool, error) {
removeDirData := removeFolder{SessionID: f.session.SessionID, FolderID: id}
opts := rest.Opts{
@@ -254,7 +254,7 @@ func (f *Fs) deleteObject(id string) error {
NoResponse: true,
Path: "/folder/remove.json",
}
resp, err := f.srv.CallJSON(&opts, &removeDirData, nil)
resp, err := f.srv.CallJSON(ctx, &opts, &removeDirData, nil)
return f.shouldRetry(resp, err)
})
}
@@ -275,14 +275,14 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
if err != nil {
return err
}
item, err := f.readMetaDataForFolderID(rootID)
item, err := f.readMetaDataForFolderID(ctx, rootID)
if err != nil {
return err
}
if check && len(item.Files) != 0 {
return errors.New("folder not empty")
}
err = f.deleteObject(rootID)
err = f.deleteObject(ctx, rootID)
if err != nil {
return err
}
@@ -353,7 +353,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
Method: "POST",
Path: "/file/move_copy.json",
}
resp, err = f.srv.CallJSON(&opts, &copyFileData, &response)
resp, err = f.srv.CallJSON(ctx, &opts, &copyFileData, &response)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -410,7 +410,7 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
Method: "POST",
Path: "/file/move_copy.json",
}
resp, err = f.srv.CallJSON(&opts, &copyFileData, &response)
resp, err = f.srv.CallJSON(ctx, &opts, &copyFileData, &response)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -509,7 +509,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
Method: "POST",
Path: "/folder/move_copy.json",
}
resp, err = f.srv.CallJSON(&opts, &moveFolderData, &response)
resp, err = f.srv.CallJSON(ctx, &opts, &moveFolderData, &response)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -589,14 +589,14 @@ func (f *Fs) createObject(ctx context.Context, remote string, modTime time.Time,
}
// readMetaDataForPath reads the metadata from the path
func (f *Fs) readMetaDataForFolderID(id string) (info *FolderList, err error) {
func (f *Fs) readMetaDataForFolderID(ctx context.Context, id string) (info *FolderList, err error) {
var resp *http.Response
opts := rest.Opts{
Method: "GET",
Path: "/folder/list.json/" + f.session.SessionID + "/" + id,
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -641,7 +641,7 @@ func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options .
Method: "POST",
Path: "/upload/create_file.json",
}
resp, err = o.fs.srv.CallJSON(&opts, &createFileData, &response)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &createFileData, &response)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -694,7 +694,7 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
Method: "POST",
Path: "/folder.json",
}
resp, err = f.srv.CallJSON(&opts, &createDirData, &response)
resp, err = f.srv.CallJSON(ctx, &opts, &createDirData, &response)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -722,7 +722,7 @@ func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut strin
Method: "GET",
Path: "/folder/list.json/" + f.session.SessionID + "/" + pathID,
}
resp, err = f.srv.CallJSON(&opts, nil, &folderList)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &folderList)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -769,7 +769,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
}
folderList := FolderList{}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &folderList)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &folderList)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -853,7 +853,7 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
}
update := modTimeFile{SessionID: o.fs.session.SessionID, FileID: o.id, FileModificationTime: strconv.FormatInt(modTime.Unix(), 10)}
err := o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.CallJSON(&opts, &update, nil)
resp, err := o.fs.srv.CallJSON(ctx, &opts, &update, nil)
return o.fs.shouldRetry(resp, err)
})
@@ -873,7 +873,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
}
var resp *http.Response
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -892,7 +892,7 @@ func (o *Object) Remove(ctx context.Context) error {
NoResponse: true,
Path: "/file.json/" + o.fs.session.SessionID + "/" + o.id,
}
resp, err := o.fs.srv.Call(&opts)
resp, err := o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(resp, err)
})
}
@@ -920,7 +920,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
Method: "POST",
Path: "/upload/open_file_upload.json",
}
resp, err := o.fs.srv.CallJSON(&opts, &openUploadData, &openResponse)
resp, err := o.fs.srv.CallJSON(ctx, &opts, &openUploadData, &openResponse)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -966,7 +966,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
MultipartFileName: o.remote, // ..name of the file for the attached file
}
resp, err = o.fs.srv.CallJSON(&opts, nil, &reply)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &reply)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -989,7 +989,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
Method: "POST",
Path: "/upload/close_file_upload.json",
}
resp, err = o.fs.srv.CallJSON(&opts, &closeUploadData, &closeResponse)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &closeUploadData, &closeResponse)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -1015,7 +1015,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
NoResponse: true,
Path: "/file/access.json",
}
resp, err = o.fs.srv.CallJSON(&opts, &update, nil)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &update, nil)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -1040,7 +1040,7 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
Method: "GET",
Path: "/folder/itembyname.json/" + o.fs.session.SessionID + "/" + directoryID + "?name=" + url.QueryEscape(replaceReservedChars(leaf)),
}
resp, err = o.fs.srv.CallJSON(&opts, nil, &folderList)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &folderList)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
+19 -19
View File
@@ -201,7 +201,7 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string) (info *api.It
return nil, err
}
found, err := f.listAll(directoryID, false, true, func(item *api.Item) bool {
found, err := f.listAll(ctx, directoryID, false, true, func(item *api.Item) bool {
if item.Name == leaf {
info = item
return true
@@ -334,7 +334,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
// FindLeaf finds a directory of name leaf in the folder with ID pathID
func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
// Find the leaf in pathID
found, err = f.listAll(pathID, true, false, func(item *api.Item) bool {
found, err = f.listAll(ctx, pathID, true, false, func(item *api.Item) bool {
if item.Name == leaf {
pathIDOut = item.ID
return true
@@ -357,7 +357,7 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
opts.Parameters.Set("name", replaceReservedChars(leaf))
opts.Parameters.Set("folderid", dirIDtoNumber(pathID))
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -400,7 +400,7 @@ type listAllFn func(*api.Item) bool
// Lists the directory required calling the user function on each item found
//
// If the user fn ever returns true then it early exits with found = true
func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
func (f *Fs) listAll(ctx context.Context, dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
opts := rest.Opts{
Method: "GET",
Path: "/listfolder",
@@ -412,7 +412,7 @@ func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn list
var result api.ItemResult
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -458,7 +458,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, err
}
var iErr error
_, err = f.listAll(directoryID, false, false, func(info *api.Item) bool {
_, err = f.listAll(ctx, directoryID, false, false, func(info *api.Item) bool {
remote := path.Join(dir, info.Name)
if info.IsFolder {
// cache the directory ID for later lookups
@@ -563,7 +563,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
var resp *http.Response
var result api.ItemResult
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -628,7 +628,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
var resp *http.Response
var result api.ItemResult
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -666,7 +666,7 @@ func (f *Fs) CleanUp(ctx context.Context) error {
var resp *http.Response
var result api.Error
return f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Update(err)
return shouldRetry(resp, err)
})
@@ -706,7 +706,7 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
var resp *http.Response
var result api.ItemResult
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -803,7 +803,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
var resp *http.Response
var result api.ItemResult
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -830,7 +830,7 @@ func (f *Fs) About(ctx context.Context) (usage *fs.Usage, err error) {
var resp *http.Response
var q api.UserInfo
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &q)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &q)
err = q.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -881,7 +881,7 @@ func (o *Object) getHashes(ctx context.Context) (err error) {
}
opts.Parameters.Set("fileid", fileIDtoNumber(o.id))
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &result)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -984,7 +984,7 @@ func (o *Object) Storable() bool {
}
// downloadURL fetches the download link
func (o *Object) downloadURL() (URL string, err error) {
func (o *Object) downloadURL(ctx context.Context) (URL string, err error) {
if o.id == "" {
return "", errors.New("can't download - no id")
}
@@ -1000,7 +1000,7 @@ func (o *Object) downloadURL() (URL string, err error) {
}
opts.Parameters.Set("fileid", fileIDtoNumber(o.id))
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &result)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -1016,7 +1016,7 @@ func (o *Object) downloadURL() (URL string, err error) {
// Open an object for read
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) {
url, err := o.downloadURL()
url, err := o.downloadURL(ctx)
if err != nil {
return nil, err
}
@@ -1027,7 +1027,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1104,7 +1104,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &result)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
@@ -1134,7 +1134,7 @@ func (o *Object) Remove(ctx context.Context) error {
var result api.ItemResult
opts.Parameters.Set("fileid", fileIDtoNumber(o.id))
return o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.CallJSON(&opts, nil, &result)
resp, err := o.fs.srv.CallJSON(ctx, &opts, nil, &result)
err = result.Error.Update(err)
return shouldRetry(resp, err)
})
+15 -15
View File
@@ -200,7 +200,7 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string, directoriesOn
}
lcLeaf := strings.ToLower(leaf)
found, err := f.listAll(directoryID, directoriesOnly, filesOnly, func(item *api.Item) bool {
found, err := f.listAll(ctx, directoryID, directoriesOnly, filesOnly, func(item *api.Item) bool {
if strings.ToLower(item.Name) == lcLeaf {
info = item
return true
@@ -361,7 +361,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
// FindLeaf finds a directory of name leaf in the folder with ID pathID
func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
// Find the leaf in pathID
found, err = f.listAll(pathID, true, false, func(item *api.Item) bool {
found, err = f.listAll(ctx, pathID, true, false, func(item *api.Item) bool {
if item.Name == leaf {
pathIDOut = item.ID
return true
@@ -386,7 +386,7 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
},
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -411,7 +411,7 @@ type listAllFn func(*api.Item) bool
// Lists the directory required calling the user function on each item found
//
// If the user fn ever returns true then it early exits with found = true
func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
func (f *Fs) listAll(ctx context.Context, dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
opts := rest.Opts{
Method: "GET",
Path: "/folder/list",
@@ -423,7 +423,7 @@ func (f *Fs) listAll(dirID string, directoriesOnly bool, filesOnly bool, fn list
var result api.FolderListResponse
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -475,7 +475,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
return nil, err
}
var iErr error
_, err = f.listAll(directoryID, false, false, func(info *api.Item) bool {
_, err = f.listAll(ctx, directoryID, false, false, func(info *api.Item) bool {
remote := path.Join(dir, info.Name)
if info.Type == api.ItemTypeFolder {
// cache the directory ID for later lookups
@@ -589,7 +589,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
// need to check if empty as it will delete recursively by default
if check {
found, err := f.listAll(rootID, false, false, func(item *api.Item) bool {
found, err := f.listAll(ctx, rootID, false, false, func(item *api.Item) bool {
return true
})
if err != nil {
@@ -611,7 +611,7 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
var resp *http.Response
var result api.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -690,7 +690,7 @@ func (f *Fs) move(ctx context.Context, isFile bool, id, oldLeaf, newLeaf, oldDir
var resp *http.Response
var result api.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -860,7 +860,7 @@ func (f *Fs) About(ctx context.Context) (usage *fs.Usage, err error) {
Parameters: f.baseParams(),
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
if err != nil {
@@ -992,7 +992,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1036,7 +1036,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
},
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &info)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &info)
if err != nil {
return shouldRetry(resp, err)
}
@@ -1096,7 +1096,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
var result api.Response
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &result)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1138,7 +1138,7 @@ func (f *Fs) renameLeaf(ctx context.Context, isFile bool, id string, newLeaf str
var resp *http.Response
var result api.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1163,7 +1163,7 @@ func (f *Fs) remove(ctx context.Context, id string) (err error) {
var resp *http.Response
var result api.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &result)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &result)
return shouldRetry(resp, err)
})
if err != nil {
+4 -2
View File
@@ -289,6 +289,7 @@ func (f *Fs) createUpload(ctx context.Context, name string, size int64, parentID
if err != nil {
return false, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
req.Header.Set("tus-resumable", "1.0.0")
req.Header.Set("upload-length", strconv.FormatInt(size, 10))
b64name := base64.StdEncoding.EncodeToString([]byte(name))
@@ -354,7 +355,7 @@ func (f *Fs) sendUpload(ctx context.Context, location string, size int64, in io.
func (f *Fs) transferChunk(ctx context.Context, location string, start int64, chunk io.ReadSeeker, chunkSize int64) (fileID int64, err error) {
// defer log.Trace(f, "location=%v, start=%v, chunkSize=%v", location, start, chunkSize)("fileID=%v, err=%v", fileID, &err)
_, _ = chunk.Seek(0, io.SeekStart)
req, err := f.makeUploadPatchRequest(location, chunk, start, chunkSize)
req, err := f.makeUploadPatchRequest(ctx, location, chunk, start, chunkSize)
if err != nil {
return 0, err
}
@@ -379,11 +380,12 @@ func (f *Fs) transferChunk(ctx context.Context, location string, start int64, ch
return fileID, nil
}
func (f *Fs) makeUploadPatchRequest(location string, in io.Reader, offset, length int64) (*http.Request, error) {
func (f *Fs) makeUploadPatchRequest(ctx context.Context, location string, in io.Reader, offset, length int64) (*http.Request, error) {
req, err := http.NewRequest("PATCH", location, in)
if err != nil {
return nil, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
req.Header.Set("tus-resumable", "1.0.0")
req.Header.Set("upload-offset", strconv.FormatInt(offset, 10))
req.Header.Set("content-length", strconv.FormatInt(length, 10))
+5 -1
View File
@@ -223,7 +223,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
var resp *http.Response
headers := fs.OpenOptionHeaders(options)
err = o.fs.pacer.Call(func() (bool, error) {
req, _ := http.NewRequest(http.MethodGet, storageURL, nil)
req, err := http.NewRequest(http.MethodGet, storageURL, nil)
if err != nil {
return shouldRetry(err)
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
req.Header.Set("User-Agent", o.fs.client.UserAgent)
// merge headers with extra headers
+70 -46
View File
@@ -818,6 +818,7 @@ type Object struct {
lastModified time.Time // Last modified
meta map[string]*string // The object metadata if known - may be nil
mimeType string // MimeType of object - may be ""
storageClass string // eg GLACIER
}
// ------------------------------------------------------------
@@ -1089,6 +1090,8 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
WriteMimeType: true,
BucketBased: true,
BucketBasedRootOK: true,
SetTier: true,
GetTier: true,
}).Fill(f)
if f.rootBucket != "" && f.rootDirectory != "" {
// Check to see if the object exists
@@ -1132,6 +1135,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *s3.Obje
}
o.etag = aws.StringValue(info.ETag)
o.bytes = aws.Int64Value(info.Size)
o.storageClass = aws.StringValue(info.StorageClass)
} else {
err := o.readMetaData(ctx) // reads info and meta, returning an error
if err != nil {
@@ -1550,6 +1554,31 @@ func pathEscape(s string) string {
return strings.Replace(rest.URLPathEscape(s), "+", "%2B", -1)
}
// copy does a server side copy
//
// It adds the boiler plate to the req passed in and calls the s3
// method
func (f *Fs) copy(ctx context.Context, req *s3.CopyObjectInput, dstBucket, dstPath, srcBucket, srcPath string) error {
req.Bucket = &dstBucket
req.ACL = &f.opt.ACL
req.Key = &dstPath
source := pathEscape(path.Join(srcBucket, srcPath))
req.CopySource = &source
if f.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &f.opt.ServerSideEncryption
}
if f.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
}
if req.StorageClass == nil && f.opt.StorageClass != "" {
req.StorageClass = &f.opt.StorageClass
}
return f.pacer.Call(func() (bool, error) {
_, err := f.c.CopyObjectWithContext(ctx, req)
return f.shouldRetry(err)
})
}
// Copy src to this remote using server side copy operations.
//
// This is stored with the remote path given
@@ -1571,27 +1600,10 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
return nil, fs.ErrorCantCopy
}
srcBucket, srcPath := srcObj.split()
source := pathEscape(path.Join(srcBucket, srcPath))
req := s3.CopyObjectInput{
Bucket: &dstBucket,
ACL: &f.opt.ACL,
Key: &dstPath,
CopySource: &source,
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
}
if f.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &f.opt.ServerSideEncryption
}
if f.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
}
if f.opt.StorageClass != "" {
req.StorageClass = &f.opt.StorageClass
}
err = f.pacer.Call(func() (bool, error) {
_, err = f.c.CopyObjectWithContext(ctx, &req)
return f.shouldRetry(err)
})
err = f.copy(ctx, &req, dstBucket, dstPath, srcBucket, srcPath)
if err != nil {
return nil, err
}
@@ -1691,6 +1703,7 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
o.etag = aws.StringValue(resp.ETag)
o.bytes = size
o.meta = resp.Metadata
o.storageClass = aws.StringValue(resp.StorageClass)
if resp.LastModified == nil {
fs.Logf(o, "Failed to read last modified from HEAD: %v", err)
o.lastModified = time.Now()
@@ -1741,39 +1754,19 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
return nil
}
// Guess the content type
mimeType := fs.MimeType(ctx, o)
// Can't update metadata here, so return this error to force a recopy
if o.storageClass == "GLACIER" || o.storageClass == "DEEP_ARCHIVE" {
return fs.ErrorCantSetModTime
}
// Copy the object to itself to update the metadata
bucket, bucketPath := o.split()
sourceKey := path.Join(bucket, bucketPath)
directive := s3.MetadataDirectiveReplace // replace metadata with that passed in
req := s3.CopyObjectInput{
Bucket: &bucket,
ACL: &o.fs.opt.ACL,
Key: &bucketPath,
ContentType: &mimeType,
CopySource: aws.String(pathEscape(sourceKey)),
ContentType: aws.String(fs.MimeType(ctx, o)), // Guess the content type
Metadata: o.meta,
MetadataDirective: &directive,
MetadataDirective: aws.String(s3.MetadataDirectiveReplace), // replace metadata with that passed in
}
if o.fs.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
}
if o.fs.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
}
if o.fs.opt.StorageClass == "GLACIER" || o.fs.opt.StorageClass == "DEEP_ARCHIVE" {
return fs.ErrorCantSetModTime
}
if o.fs.opt.StorageClass != "" {
req.StorageClass = &o.fs.opt.StorageClass
}
err = o.fs.pacer.Call(func() (bool, error) {
_, err := o.fs.c.CopyObjectWithContext(ctx, &req)
return o.fs.shouldRetry(err)
})
return err
return o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath)
}
// Storable raturns a boolean indicating if this object is storable
@@ -1932,6 +1925,10 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
return errors.Wrap(err, "s3 upload: sign request")
}
if o.fs.opt.V2Auth && headers == nil {
headers = putObj.HTTPRequest.Header
}
// Set request to nil if empty so as not to make chunked encoding
if size == 0 {
in = nil
@@ -1942,7 +1939,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
if err != nil {
return errors.Wrap(err, "s3 upload: new request")
}
httpReq = httpReq.WithContext(ctx)
httpReq = httpReq.WithContext(ctx) // go1.13 can use NewRequestWithContext
// set the headers we signed and the length
httpReq.Header = headers
@@ -1998,6 +1995,31 @@ func (o *Object) MimeType(ctx context.Context) string {
return o.mimeType
}
// SetTier performs changing storage class
func (o *Object) SetTier(tier string) (err error) {
ctx := context.TODO()
tier = strings.ToUpper(tier)
bucket, bucketPath := o.split()
req := s3.CopyObjectInput{
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
StorageClass: aws.String(tier),
}
err = o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath)
if err != nil {
return err
}
o.storageClass = tier
return err
}
// GetTier returns storage class as string
func (o *Object) GetTier() string {
if o.storageClass == "" {
return "STANDARD"
}
return o.storageClass
}
// Check the interfaces are satisfied
var (
_ fs.Fs = &Fs{}
@@ -2006,4 +2028,6 @@ var (
_ fs.ListRer = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
_ fs.GetTierer = &Object{}
_ fs.SetTierer = &Object{}
)
+3 -2
View File
@@ -11,8 +11,9 @@ import (
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestS3:",
NilObject: (*Object)(nil),
RemoteName: "TestS3:",
NilObject: (*Object)(nil),
TiersToTest: []string{"STANDARD", "STANDARD_IA"},
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: minChunkSize,
},
+9 -4
View File
@@ -103,9 +103,14 @@ when the ssh-agent contains many keys.`,
Default: false,
Help: "Disable the execution of SSH commands to determine if remote file hashing is available.\nLeave blank or set to false to enable hashing (recommended), set to true to disable hashing.",
}, {
Name: "ask_password",
Default: false,
Help: "Allow asking for SFTP password when needed.",
Name: "ask_password",
Default: false,
Help: `Allow asking for SFTP password when needed.
If this is set and no password is supplied then rclone will:
- ask for a password
- not contact the ssh agent
`,
Advanced: true,
}, {
Name: "path_override",
@@ -364,7 +369,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
keyFile := env.ShellExpand(opt.KeyFile)
// Add ssh agent-auth if no password or file specified
if (opt.Pass == "" && keyFile == "") || opt.KeyUseAgent {
if (opt.Pass == "" && keyFile == "" && !opt.AskPassword) || opt.KeyUseAgent {
sshAgentClient, _, err := sshagent.New()
if err != nil {
return nil, errors.Wrap(err, "couldn't connect to ssh-agent")
+5 -3
View File
@@ -3,6 +3,7 @@ package odrvcookie
import (
"bytes"
"context"
"encoding/xml"
"html/template"
"net/http"
@@ -91,8 +92,8 @@ func New(pUser, pPass, pEndpoint string) CookieAuth {
// Cookies creates a CookieResponse. It fetches the auth token and then
// retrieves the Cookies
func (ca *CookieAuth) Cookies() (*CookieResponse, error) {
tokenResp, err := ca.getSPToken()
func (ca *CookieAuth) Cookies(ctx context.Context) (*CookieResponse, error) {
tokenResp, err := ca.getSPToken(ctx)
if err != nil {
return nil, err
}
@@ -140,7 +141,7 @@ func (ca *CookieAuth) getSPCookie(conf *SuccessResponse) (*CookieResponse, error
return &cookieResponse, nil
}
func (ca *CookieAuth) getSPToken() (conf *SuccessResponse, err error) {
func (ca *CookieAuth) getSPToken(ctx context.Context) (conf *SuccessResponse, err error) {
reqData := map[string]interface{}{
"Username": ca.user,
"Password": ca.pass,
@@ -160,6 +161,7 @@ func (ca *CookieAuth) getSPToken() (conf *SuccessResponse, err error) {
if err != nil {
return nil, err
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
client := fshttp.NewClient(fs.Config)
resp, err := client.Do(req)
+41 -40
View File
@@ -205,7 +205,7 @@ func itemIsDir(item *api.Response) bool {
}
// readMetaDataForPath reads the metadata from the path
func (f *Fs) readMetaDataForPath(path string, depth string) (info *api.Prop, err error) {
func (f *Fs) readMetaDataForPath(ctx context.Context, path string, depth string) (info *api.Prop, err error) {
// FIXME how do we read back additional properties?
opts := rest.Opts{
Method: "PROPFIND",
@@ -221,7 +221,7 @@ func (f *Fs) readMetaDataForPath(path string, depth string) (info *api.Prop, err
var result api.Multistatus
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &result)
resp, err = f.srv.CallXML(ctx, &opts, nil, &result)
return f.shouldRetry(resp, err)
})
if apiErr, ok := err.(*api.Error); ok {
@@ -229,7 +229,7 @@ func (f *Fs) readMetaDataForPath(path string, depth string) (info *api.Prop, err
switch apiErr.StatusCode {
case http.StatusNotFound:
if f.retryWithZeroDepth && depth != "0" {
return f.readMetaDataForPath(path, "0")
return f.readMetaDataForPath(ctx, path, "0")
}
return nil, fs.ErrorObjectNotFound
case http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther:
@@ -353,7 +353,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
}
f.srv.SetErrorHandler(errorHandler)
err = f.setQuirks(opt.Vendor)
err = f.setQuirks(ctx, opt.Vendor)
if err != nil {
return nil, err
}
@@ -424,7 +424,7 @@ func (f *Fs) fetchAndSetBearerToken() error {
}
// setQuirks adjusts the Fs for the vendor passed in
func (f *Fs) setQuirks(vendor string) error {
func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
switch vendor {
case "owncloud":
f.canStream = true
@@ -440,13 +440,13 @@ func (f *Fs) setQuirks(vendor string) error {
// They have to be set instead of BasicAuth
f.srv.RemoveHeader("Authorization") // We don't need this Header if using cookies
spCk := odrvcookie.New(f.opt.User, f.opt.Pass, f.endpointURL)
spCookies, err := spCk.Cookies()
spCookies, err := spCk.Cookies(ctx)
if err != nil {
return err
}
odrvcookie.NewRenew(12*time.Hour, func() {
spCookies, err := spCk.Cookies()
spCookies, err := spCk.Cookies(ctx)
if err != nil {
fs.Errorf("could not renew cookies: %s", err.Error())
return
@@ -477,7 +477,7 @@ func (f *Fs) setQuirks(vendor string) error {
// Return an Object from a path
//
// If it can't be found it returns the error fs.ErrorObjectNotFound.
func (f *Fs) newObjectWithInfo(remote string, info *api.Prop) (fs.Object, error) {
func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *api.Prop) (fs.Object, error) {
o := &Object{
fs: f,
remote: remote,
@@ -487,7 +487,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *api.Prop) (fs.Object, error)
// Set info
err = o.setMetaData(info)
} else {
err = o.readMetaData() // reads info and meta, returning an error
err = o.readMetaData(ctx) // reads info and meta, returning an error
}
if err != nil {
return nil, err
@@ -498,7 +498,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *api.Prop) (fs.Object, error)
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
return f.newObjectWithInfo(remote, nil)
return f.newObjectWithInfo(ctx, remote, nil)
}
// Read the normal props, plus the checksums
@@ -528,7 +528,7 @@ type listAllFn func(string, bool, *api.Prop) bool
// Lists the directory required calling the user function on each item found
//
// If the user fn ever returns true then it early exits with found = true
func (f *Fs) listAll(dir string, directoriesOnly bool, filesOnly bool, depth string, fn listAllFn) (found bool, err error) {
func (f *Fs) listAll(ctx context.Context, dir string, directoriesOnly bool, filesOnly bool, depth string, fn listAllFn) (found bool, err error) {
opts := rest.Opts{
Method: "PROPFIND",
Path: f.dirPath(dir), // FIXME Should not start with /
@@ -542,7 +542,7 @@ func (f *Fs) listAll(dir string, directoriesOnly bool, filesOnly bool, depth str
var result api.Multistatus
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &result)
resp, err = f.srv.CallXML(ctx, &opts, nil, &result)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -550,7 +550,7 @@ func (f *Fs) listAll(dir string, directoriesOnly bool, filesOnly bool, depth str
// does not exist
if apiErr.StatusCode == http.StatusNotFound {
if f.retryWithZeroDepth && depth != "0" {
return f.listAll(dir, directoriesOnly, filesOnly, "0", fn)
return f.listAll(ctx, dir, directoriesOnly, filesOnly, "0", fn)
}
return found, fs.ErrorDirNotFound
}
@@ -625,14 +625,14 @@ func (f *Fs) listAll(dir string, directoriesOnly bool, filesOnly bool, depth str
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
var iErr error
_, err = f.listAll(dir, false, false, defaultDepth, func(remote string, isDir bool, info *api.Prop) bool {
_, err = f.listAll(ctx, dir, false, false, defaultDepth, func(remote string, isDir bool, info *api.Prop) bool {
if isDir {
d := fs.NewDir(remote, time.Time(info.Modified))
// .SetID(info.ID)
// FIXME more info from dir? can set size, items?
entries = append(entries, d)
} else {
o, err := f.newObjectWithInfo(remote, info)
o, err := f.newObjectWithInfo(ctx, remote, info)
if err != nil {
iErr = err
return true
@@ -696,7 +696,7 @@ func (f *Fs) mkParentDir(ctx context.Context, dirPath string) error {
}
// low level mkdir, only makes the directory, doesn't attempt to create parents
func (f *Fs) _mkdir(dirPath string) error {
func (f *Fs) _mkdir(ctx context.Context, dirPath string) error {
// We assume the root is already created
if dirPath == "" {
return nil
@@ -711,7 +711,7 @@ func (f *Fs) _mkdir(dirPath string) error {
NoResponse: true,
}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.Call(&opts)
resp, err := f.srv.Call(ctx, &opts)
return f.shouldRetry(resp, err)
})
if apiErr, ok := err.(*api.Error); ok {
@@ -727,13 +727,13 @@ func (f *Fs) _mkdir(dirPath string) error {
// mkdir makes the directory and parents using native paths
func (f *Fs) mkdir(ctx context.Context, dirPath string) error {
// defer log.Trace(dirPath, "")("")
err := f._mkdir(dirPath)
err := f._mkdir(ctx, dirPath)
if apiErr, ok := err.(*api.Error); ok {
// parent does not exist so create it first then try again
if apiErr.StatusCode == http.StatusConflict {
err = f.mkParentDir(ctx, dirPath)
if err == nil {
err = f._mkdir(dirPath)
err = f._mkdir(ctx, dirPath)
}
}
}
@@ -749,17 +749,17 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
// dirNotEmpty returns true if the directory exists and is not Empty
//
// if the directory does not exist then err will be ErrorDirNotFound
func (f *Fs) dirNotEmpty(dir string) (found bool, err error) {
return f.listAll(dir, false, false, defaultDepth, func(remote string, isDir bool, info *api.Prop) bool {
func (f *Fs) dirNotEmpty(ctx context.Context, dir string) (found bool, err error) {
return f.listAll(ctx, dir, false, false, defaultDepth, func(remote string, isDir bool, info *api.Prop) bool {
return true
})
}
// purgeCheck removes the root directory, if check is set then it
// refuses to do so if it has anything in
func (f *Fs) purgeCheck(dir string, check bool) error {
func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
if check {
notEmpty, err := f.dirNotEmpty(dir)
notEmpty, err := f.dirNotEmpty(ctx, dir)
if err != nil {
return err
}
@@ -775,7 +775,7 @@ func (f *Fs) purgeCheck(dir string, check bool) error {
var resp *http.Response
var err error
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, nil)
resp, err = f.srv.CallXML(ctx, &opts, nil, nil)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -789,7 +789,7 @@ func (f *Fs) purgeCheck(dir string, check bool) error {
//
// Returns an error if it isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return f.purgeCheck(dir, true)
return f.purgeCheck(ctx, dir, true)
}
// Precision return the precision of this Fs
@@ -835,10 +835,10 @@ func (f *Fs) copyOrMove(ctx context.Context, src fs.Object, remote string, metho
},
}
if f.useOCMtime {
opts.ExtraHeaders["X-OC-Mtime"] = fmt.Sprintf("%f", float64(src.ModTime(ctx).UnixNano())/1E9)
opts.ExtraHeaders["X-OC-Mtime"] = fmt.Sprintf("%f", float64(src.ModTime(ctx).UnixNano())/1e9)
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -870,7 +870,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
// deleting all the files quicker than just running Remove() on the
// result of List()
func (f *Fs) Purge(ctx context.Context) error {
return f.purgeCheck("", false)
return f.purgeCheck(ctx, "", false)
}
// Move src to this remote using server side move operations.
@@ -904,7 +904,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
dstPath := f.filePath(dstRemote)
// Check if destination exists
_, err := f.dirNotEmpty(dstRemote)
_, err := f.dirNotEmpty(ctx, dstRemote)
if err == nil {
return fs.ErrorDirExists
}
@@ -934,7 +934,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
},
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -975,7 +975,7 @@ func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
var resp *http.Response
var err error
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallXML(&opts, nil, &q)
resp, err = f.srv.CallXML(ctx, &opts, nil, &q)
return f.shouldRetry(resp, err)
})
if err != nil {
@@ -1028,7 +1028,8 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
// Size returns the size of an object in bytes
func (o *Object) Size() int64 {
err := o.readMetaData()
ctx := context.TODO()
err := o.readMetaData(ctx)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return 0
@@ -1052,11 +1053,11 @@ func (o *Object) setMetaData(info *api.Prop) (err error) {
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData() (err error) {
func (o *Object) readMetaData(ctx context.Context) (err error) {
if o.hasMetaData {
return nil
}
info, err := o.fs.readMetaDataForPath(o.remote, defaultDepth)
info, err := o.fs.readMetaDataForPath(ctx, o.remote, defaultDepth)
if err != nil {
return err
}
@@ -1068,7 +1069,7 @@ func (o *Object) readMetaData() (err error) {
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
err := o.readMetaData()
err := o.readMetaData(ctx)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return time.Now()
@@ -1095,7 +1096,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -1128,7 +1129,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
if o.fs.useOCMtime || o.fs.hasChecksums {
opts.ExtraHeaders = map[string]string{}
if o.fs.useOCMtime {
opts.ExtraHeaders["X-OC-Mtime"] = fmt.Sprintf("%f", float64(src.ModTime(ctx).UnixNano())/1E9)
opts.ExtraHeaders["X-OC-Mtime"] = fmt.Sprintf("%f", float64(src.ModTime(ctx).UnixNano())/1e9)
}
if o.fs.hasChecksums {
// Set an upload checksum - prefer SHA1
@@ -1143,7 +1144,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
}
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(resp, err)
})
if err != nil {
@@ -1159,7 +1160,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
}
// read metadata from remote
o.hasMetaData = false
return o.readMetaData()
return o.readMetaData(ctx)
}
// Remove an object
@@ -1170,7 +1171,7 @@ func (o *Object) Remove(ctx context.Context) error {
NoResponse: true,
}
return o.fs.pacer.Call(func() (bool, error) {
resp, err := o.fs.srv.Call(&opts)
resp, err := o.fs.srv.Call(ctx, &opts)
return o.fs.shouldRetry(resp, err)
})
}
+59 -57
View File
@@ -200,7 +200,7 @@ func (f *Fs) dirPath(file string) string {
return path.Join(f.diskRoot, file) + "/"
}
func (f *Fs) readMetaDataForPath(path string, options *api.ResourceInfoRequestOptions) (*api.ResourceInfoResponse, error) {
func (f *Fs) readMetaDataForPath(ctx context.Context, path string, options *api.ResourceInfoRequestOptions) (*api.ResourceInfoResponse, error) {
opts := rest.Opts{
Method: "GET",
Path: "/resources",
@@ -226,7 +226,7 @@ func (f *Fs) readMetaDataForPath(path string, options *api.ResourceInfoRequestOp
var info api.ResourceInfoResponse
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
@@ -239,6 +239,7 @@ func (f *Fs) readMetaDataForPath(path string, options *api.ResourceInfoRequestOp
// NewFs constructs an Fs from the path, container:path
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
ctx := context.TODO()
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
@@ -284,7 +285,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
//request object meta info
// Check to see if the object exists and is a file
//request object meta info
if info, err := f.readMetaDataForPath(f.diskRoot, &api.ResourceInfoRequestOptions{}); err != nil {
if info, err := f.readMetaDataForPath(ctx, f.diskRoot, &api.ResourceInfoRequestOptions{}); err != nil {
} else {
if info.ResourceType == "file" {
@@ -301,7 +302,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
}
// Convert a list item into a DirEntry
func (f *Fs) itemToDirEntry(remote string, object *api.ResourceInfoResponse) (fs.DirEntry, error) {
func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *api.ResourceInfoResponse) (fs.DirEntry, error) {
switch object.ResourceType {
case "dir":
t, err := time.Parse(time.RFC3339Nano, object.Modified)
@@ -311,7 +312,7 @@ func (f *Fs) itemToDirEntry(remote string, object *api.ResourceInfoResponse) (fs
d := fs.NewDir(remote, t).SetSize(object.Size)
return d, nil
case "file":
o, err := f.newObjectWithInfo(remote, object)
o, err := f.newObjectWithInfo(ctx, remote, object)
if err != nil {
return nil, err
}
@@ -343,7 +344,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
Limit: limit,
Offset: offset,
}
info, err := f.readMetaDataForPath(root, opts)
info, err := f.readMetaDataForPath(ctx, root, opts)
if err != nil {
if apiErr, ok := err.(*api.ErrorResponse); ok {
@@ -360,7 +361,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
//list all subdirs
for _, element := range info.Embedded.Items {
remote := path.Join(dir, element.Name)
entry, err := f.itemToDirEntry(remote, &element)
entry, err := f.itemToDirEntry(ctx, remote, &element)
if err != nil {
return nil, err
}
@@ -386,7 +387,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
// Return an Object from a path
//
// If it can't be found it returns the error fs.ErrorObjectNotFound.
func (f *Fs) newObjectWithInfo(remote string, info *api.ResourceInfoResponse) (fs.Object, error) {
func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *api.ResourceInfoResponse) (fs.Object, error) {
o := &Object{
fs: f,
remote: remote,
@@ -395,7 +396,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *api.ResourceInfoResponse) (f
if info != nil {
err = o.setMetaData(info)
} else {
err = o.readMetaData()
err = o.readMetaData(ctx)
if apiErr, ok := err.(*api.ErrorResponse); ok {
// does not exist
if apiErr.ErrorName == "DiskNotFoundError" {
@@ -412,7 +413,7 @@ func (f *Fs) newObjectWithInfo(remote string, info *api.ResourceInfoResponse) (f
// NewObject finds the Object at remote. If it can't be found it
// returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
return f.newObjectWithInfo(remote, nil)
return f.newObjectWithInfo(ctx, remote, nil)
}
// Creates from the parameters passed in a half finished Object which
@@ -446,7 +447,7 @@ func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, opt
}
// CreateDir makes a directory
func (f *Fs) CreateDir(path string) (err error) {
func (f *Fs) CreateDir(ctx context.Context, path string) (err error) {
//fmt.Printf("CreateDir: %s\n", path)
var resp *http.Response
@@ -460,7 +461,7 @@ func (f *Fs) CreateDir(path string) (err error) {
opts.Parameters.Set("path", path)
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -474,7 +475,7 @@ func (f *Fs) CreateDir(path string) (err error) {
// This really needs improvement and especially proper error checking
// but Yandex does not publish a List of possible errors and when they're
// expected to occur.
func (f *Fs) mkDirs(path string) (err error) {
func (f *Fs) mkDirs(ctx context.Context, path string) (err error) {
//trim filename from path
//dirString := strings.TrimSuffix(path, filepath.Base(path))
//trim "disk:" from path
@@ -483,7 +484,7 @@ func (f *Fs) mkDirs(path string) (err error) {
return nil
}
if err = f.CreateDir(dirString); err != nil {
if err = f.CreateDir(ctx, dirString); err != nil {
if apiErr, ok := err.(*api.ErrorResponse); ok {
// allready exists
if apiErr.ErrorName != "DiskPathPointsToExistentDirectoryError" {
@@ -493,7 +494,7 @@ func (f *Fs) mkDirs(path string) (err error) {
for _, element := range dirs {
if element != "" {
mkdirpath += element + "/" //path separator /
if err = f.CreateDir(mkdirpath); err != nil {
if err = f.CreateDir(ctx, mkdirpath); err != nil {
// ignore errors while creating dirs
}
}
@@ -505,7 +506,7 @@ func (f *Fs) mkDirs(path string) (err error) {
return err
}
func (f *Fs) mkParentDirs(resPath string) error {
func (f *Fs) mkParentDirs(ctx context.Context, resPath string) error {
// defer log.Trace(dirPath, "")("")
// chop off trailing / if it exists
if strings.HasSuffix(resPath, "/") {
@@ -515,17 +516,17 @@ func (f *Fs) mkParentDirs(resPath string) error {
if parent == "." {
parent = ""
}
return f.mkDirs(parent)
return f.mkDirs(ctx, parent)
}
// Mkdir creates the container if it doesn't exist
func (f *Fs) Mkdir(ctx context.Context, dir string) error {
path := f.filePath(dir)
return f.mkDirs(path)
return f.mkDirs(ctx, path)
}
// waitForJob waits for the job with status in url to complete
func (f *Fs) waitForJob(location string) (err error) {
func (f *Fs) waitForJob(ctx context.Context, location string) (err error) {
opts := rest.Opts{
RootURL: location,
Method: "GET",
@@ -535,7 +536,7 @@ func (f *Fs) waitForJob(location string) (err error) {
var resp *http.Response
var body []byte
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
if err != nil {
return fserrors.ShouldRetry(err), err
}
@@ -564,7 +565,7 @@ func (f *Fs) waitForJob(location string) (err error) {
return errors.Errorf("async operation didn't complete after %v", fs.Config.Timeout)
}
func (f *Fs) delete(path string, hardDelete bool) (err error) {
func (f *Fs) delete(ctx context.Context, path string, hardDelete bool) (err error) {
opts := rest.Opts{
Method: "DELETE",
Path: "/resources",
@@ -577,7 +578,7 @@ func (f *Fs) delete(path string, hardDelete bool) (err error) {
var resp *http.Response
var body []byte
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
if err != nil {
return fserrors.ShouldRetry(err), err
}
@@ -595,19 +596,19 @@ func (f *Fs) delete(path string, hardDelete bool) (err error) {
if err != nil {
return errors.Wrapf(err, "async info result not JSON: %q", body)
}
return f.waitForJob(info.HRef)
return f.waitForJob(ctx, info.HRef)
}
return nil
}
// purgeCheck remotes the root directory, if check is set then it
// refuses to do so if it has anything in
func (f *Fs) purgeCheck(dir string, check bool) error {
func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
root := f.filePath(dir)
if check {
//to comply with rclone logic we check if the directory is empty before delete.
//send request to get list of objects in this directory.
info, err := f.readMetaDataForPath(root, &api.ResourceInfoRequestOptions{})
info, err := f.readMetaDataForPath(ctx, root, &api.ResourceInfoRequestOptions{})
if err != nil {
return errors.Wrap(err, "rmdir failed")
}
@@ -616,14 +617,14 @@ func (f *Fs) purgeCheck(dir string, check bool) error {
}
}
//delete directory
return f.delete(root, false)
return f.delete(ctx, root, false)
}
// Rmdir deletes the container
//
// Returns an error if it isn't empty
func (f *Fs) Rmdir(ctx context.Context, dir string) error {
return f.purgeCheck(dir, true)
return f.purgeCheck(ctx, dir, true)
}
// Purge deletes all the files and the container
@@ -632,11 +633,11 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
// deleting all the files quicker than just running Remove() on the
// result of List()
func (f *Fs) Purge(ctx context.Context) error {
return f.purgeCheck("", false)
return f.purgeCheck(ctx, "", false)
}
// copyOrMoves copies or moves directories or files depending on the method parameter
func (f *Fs) copyOrMove(method, src, dst string, overwrite bool) (err error) {
func (f *Fs) copyOrMove(ctx context.Context, method, src, dst string, overwrite bool) (err error) {
opts := rest.Opts{
Method: "POST",
Path: "/resources/" + method,
@@ -650,7 +651,7 @@ func (f *Fs) copyOrMove(method, src, dst string, overwrite bool) (err error) {
var resp *http.Response
var body []byte
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
if err != nil {
return fserrors.ShouldRetry(err), err
}
@@ -668,7 +669,7 @@ func (f *Fs) copyOrMove(method, src, dst string, overwrite bool) (err error) {
if err != nil {
return errors.Wrapf(err, "async info result not JSON: %q", body)
}
return f.waitForJob(info.HRef)
return f.waitForJob(ctx, info.HRef)
}
return nil
}
@@ -690,11 +691,11 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
}
dstPath := f.filePath(remote)
err := f.mkParentDirs(dstPath)
err := f.mkParentDirs(ctx, dstPath)
if err != nil {
return nil, err
}
err = f.copyOrMove("copy", srcObj.filePath(), dstPath, false)
err = f.copyOrMove(ctx, "copy", srcObj.filePath(), dstPath, false)
if err != nil {
return nil, errors.Wrap(err, "couldn't copy file")
@@ -720,11 +721,11 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
}
dstPath := f.filePath(remote)
err := f.mkParentDirs(dstPath)
err := f.mkParentDirs(ctx, dstPath)
if err != nil {
return nil, err
}
err = f.copyOrMove("move", srcObj.filePath(), dstPath, false)
err = f.copyOrMove(ctx, "move", srcObj.filePath(), dstPath, false)
if err != nil {
return nil, errors.Wrap(err, "couldn't move file")
@@ -758,12 +759,12 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
return errors.New("can't move root directory")
}
err := f.mkParentDirs(dstPath)
err := f.mkParentDirs(ctx, dstPath)
if err != nil {
return err
}
_, err = f.readMetaDataForPath(dstPath, &api.ResourceInfoRequestOptions{})
_, err = f.readMetaDataForPath(ctx, dstPath, &api.ResourceInfoRequestOptions{})
if apiErr, ok := err.(*api.ErrorResponse); ok {
// does not exist
if apiErr.ErrorName == "DiskNotFoundError" {
@@ -775,7 +776,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
return fs.ErrorDirExists
}
err = f.copyOrMove("move", srcPath, dstPath, false)
err = f.copyOrMove(ctx, "move", srcPath, dstPath, false)
if err != nil {
return errors.Wrap(err, "couldn't move directory")
@@ -802,7 +803,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (link string, err er
var resp *http.Response
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
@@ -819,7 +820,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string) (link string, err er
return "", errors.Wrap(err, "couldn't create public link")
}
info, err := f.readMetaDataForPath(f.filePath(remote), &api.ResourceInfoRequestOptions{})
info, err := f.readMetaDataForPath(ctx, f.filePath(remote), &api.ResourceInfoRequestOptions{})
if err != nil {
return "", err
}
@@ -840,7 +841,7 @@ func (f *Fs) CleanUp(ctx context.Context) (err error) {
}
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.Call(&opts)
resp, err = f.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
return err
@@ -857,7 +858,7 @@ func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
var info api.DiskInfo
var err error
err = f.pacer.Call(func() (bool, error) {
resp, err = f.srv.CallJSON(&opts, nil, &info)
resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
return shouldRetry(resp, err)
})
@@ -924,11 +925,11 @@ func (o *Object) setMetaData(info *api.ResourceInfoResponse) (err error) {
}
// readMetaData reads ands sets the new metadata for a storage.Object
func (o *Object) readMetaData() (err error) {
func (o *Object) readMetaData(ctx context.Context) (err error) {
if o.hasMetaData {
return nil
}
info, err := o.fs.readMetaDataForPath(o.filePath(), &api.ResourceInfoRequestOptions{})
info, err := o.fs.readMetaDataForPath(ctx, o.filePath(), &api.ResourceInfoRequestOptions{})
if err != nil {
return err
}
@@ -943,7 +944,7 @@ func (o *Object) readMetaData() (err error) {
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
func (o *Object) ModTime(ctx context.Context) time.Time {
err := o.readMetaData()
err := o.readMetaData(ctx)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return time.Now()
@@ -953,7 +954,8 @@ func (o *Object) ModTime(ctx context.Context) time.Time {
// Size returns the size of an object in bytes
func (o *Object) Size() int64 {
err := o.readMetaData()
ctx := context.TODO()
err := o.readMetaData(ctx)
if err != nil {
fs.Logf(o, "Failed to read metadata: %v", err)
return 0
@@ -974,7 +976,7 @@ func (o *Object) Storable() bool {
return true
}
func (o *Object) setCustomProperty(property string, value string) (err error) {
func (o *Object) setCustomProperty(ctx context.Context, property string, value string) (err error) {
var resp *http.Response
opts := rest.Opts{
Method: "PATCH",
@@ -990,7 +992,7 @@ func (o *Object) setCustomProperty(property string, value string) (err error) {
cpr := api.CustomPropertyResponse{CustomProperties: rcm}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, &cpr, nil)
resp, err = o.fs.srv.CallJSON(ctx, &opts, &cpr, nil)
return shouldRetry(resp, err)
})
return err
@@ -1001,7 +1003,7 @@ func (o *Object) setCustomProperty(property string, value string) (err error) {
// Commits the datastore
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
// set custom_property 'rclone_modified' of object to modTime
err := o.setCustomProperty("rclone_modified", modTime.Format(time.RFC3339Nano))
err := o.setCustomProperty(ctx, "rclone_modified", modTime.Format(time.RFC3339Nano))
if err != nil {
return err
}
@@ -1023,7 +1025,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
opts.Parameters.Set("path", o.filePath())
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &dl)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &dl)
return shouldRetry(resp, err)
})
@@ -1038,7 +1040,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
if err != nil {
@@ -1047,7 +1049,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
return resp.Body, err
}
func (o *Object) upload(in io.Reader, overwrite bool, mimeType string) (err error) {
func (o *Object) upload(ctx context.Context, in io.Reader, overwrite bool, mimeType string) (err error) {
// prepare upload
var resp *http.Response
var ur api.AsyncInfo
@@ -1061,7 +1063,7 @@ func (o *Object) upload(in io.Reader, overwrite bool, mimeType string) (err erro
opts.Parameters.Set("overwrite", strconv.FormatBool(overwrite))
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.CallJSON(&opts, nil, &ur)
resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &ur)
return shouldRetry(resp, err)
})
@@ -1079,7 +1081,7 @@ func (o *Object) upload(in io.Reader, overwrite bool, mimeType string) (err erro
}
err = o.fs.pacer.Call(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
resp, err = o.fs.srv.Call(ctx, &opts)
return shouldRetry(resp, err)
})
@@ -1097,13 +1099,13 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
remote := o.filePath()
//create full path to file before upload.
err := o.fs.mkParentDirs(remote)
err := o.fs.mkParentDirs(ctx, remote)
if err != nil {
return err
}
//upload file
err = o.upload(in1, true, fs.MimeType(ctx, src))
err = o.upload(ctx, in1, true, fs.MimeType(ctx, src))
if err != nil {
return err
}
@@ -1120,7 +1122,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
// Remove an object
func (o *Object) Remove(ctx context.Context) error {
return o.fs.delete(o.filePath(), false)
return o.fs.delete(ctx, o.filePath(), false)
}
// MimeType of an Object if known, "" otherwise
+3 -3
View File
@@ -1,4 +1,4 @@
#!/usr/bin/env python
#!/usr/bin/env python3
"""
This is a tool to decrypt file names in rclone logs.
@@ -43,13 +43,13 @@ def map_log_file(crypt_map, log_file):
"""
with open(log_file) as fd:
for line in fd:
for cipher, plain in crypt_map.iteritems():
for cipher, plain in crypt_map.items():
line = line.replace(cipher, plain)
sys.stdout.write(line)
def main():
if len(sys.argv) < 3:
print "Syntax: %s <crypt-mapping-file> <log-file>" % sys.argv[0]
print("Syntax: %s <crypt-mapping-file> <log-file>" % sys.argv[0])
raise SystemExit(1)
mapping_file, log_file = sys.argv[1:]
crypt_map = read_crypt_map(mapping_file)
+4 -4
View File
@@ -1,4 +1,4 @@
#!/usr/bin/env python2
#!/usr/bin/env python3
"""
Make backend documentation
"""
@@ -52,9 +52,9 @@ if __name__ == "__main__":
for backend in find_backends():
try:
alter_doc(backend)
except Exception, e:
print "Failed adding docs for %s backend: %s" % (backend, e)
except Exception as e:
print("Failed adding docs for %s backend: %s" % (backend, e))
failed += 1
else:
success += 1
print "Added docs for %d backends with %d failures" % (success, failed)
print("Added docs for %d backends with %d failures" % (success, failed))
+5 -4
View File
@@ -1,4 +1,4 @@
#!/usr/bin/python
#!/usr/bin/python3
"""
Generate a markdown changelog for the rclone project
"""
@@ -99,10 +99,11 @@ def process_log(log):
def main():
if len(sys.argv) != 3:
print >>sys.stderr, "Syntax: %s vX.XX vX.XY" % sys.argv[0]
print("Syntax: %s vX.XX vX.XY" % sys.argv[0], file=sys.stderr)
sys.exit(1)
version, next_version = sys.argv[1], sys.argv[2]
log = subprocess.check_output(["git", "log", '''--pretty=format:%H|%an|%aI|%s'''] + [version+".."+next_version])
log = log.decode("utf-8")
by_category = process_log(log)
# Output backends first so remaining in by_category are core items
@@ -112,7 +113,7 @@ def main():
out("local", title="Local")
out("cache", title="Cache")
out("crypt", title="Crypt")
backend_names = sorted(x for x in by_category.keys() if x in backends)
backend_names = sorted(x for x in list(by_category.keys()) if x in backends)
for backend_name in backend_names:
if backend_name in backend_titles:
backend_title = backend_titles[backend_name]
@@ -123,7 +124,7 @@ def main():
# Split remaining in by_category into new features and fixes
new_features = defaultdict(list)
bugfixes = defaultdict(list)
for name, messages in by_category.iteritems():
for name, messages in by_category.items():
for message in messages:
if IS_FIX_RE.search(message):
bugfixes[name].append(message)
+6 -5
View File
@@ -1,4 +1,4 @@
#!/usr/bin/env python2
#!/usr/bin/env python3
"""
Make single page versions of the documentation for release and
conversion into man pages etc.
@@ -41,6 +41,7 @@ docs = [
"hubic.md",
"jottacloud.md",
"koofr.md",
"mailru.md",
"mega.md",
"azureblob.md",
"onedrive.md",
@@ -118,8 +119,8 @@ def check_docs(docpath):
docs_set = set(docs)
if files == docs_set:
return
print "Files on disk but not in docs variable: %s" % ", ".join(files - docs_set)
print "Files in docs variable but not on disk: %s" % ", ".join(docs_set - files)
print("Files on disk but not in docs variable: %s" % ", ".join(files - docs_set))
print("Files in docs variable but not on disk: %s" % ", ".join(docs_set - files))
raise ValueError("Missing files")
def read_command(command):
@@ -142,7 +143,7 @@ def read_commands(docpath):
def main():
check_docs(docpath)
command_docs = read_commands(docpath)
command_docs = read_commands(docpath).replace("\\", "\\\\") # escape \ so we can use command_docs in re.sub
with open(outfile, "w") as out:
out.write("""\
%% rclone(1) User Manual
@@ -156,7 +157,7 @@ def main():
if doc == "docs.md":
contents = re.sub(r"The main rclone commands.*?for the full list.", command_docs, contents, 0, re.S)
out.write(contents)
print "Written '%s'" % outfile
print("Written '%s'" % outfile)
if __name__ == "__main__":
main()
+4 -3
View File
@@ -1,4 +1,4 @@
#!/usr/bin/env python
#!/usr/bin/env python3
"""
Update the authors.md file with the authors from the git log
"""
@@ -23,13 +23,14 @@ def add_email(name, email):
"""
adds the email passed in to the end of authors.md
"""
print "Adding %s <%s>" % (name, email)
print("Adding %s <%s>" % (name, email))
with open(AUTHORS, "a+") as fd:
print >>fd, " * %s <%s>" % (name, email)
print(" * %s <%s>" % (name, email), file=fd)
subprocess.check_call(["git", "commit", "-m", "Add %s to contributors" % name, AUTHORS])
def main():
out = subprocess.check_output(["git", "log", '--reverse', '--format=%an|%ae', "master"])
out = out.decode("utf-8")
previous = load()
for line in out.split("\n"):
+9 -2
View File
@@ -174,7 +174,11 @@ func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs
// If file exists then srcFileName != "", however if the file
// doesn't exist then we assume it is a directory...
if srcFileName != "" {
dstRemote, dstFileName = fspath.Split(dstRemote)
var err error
dstRemote, dstFileName, err = fspath.Split(dstRemote)
if err != nil {
log.Fatalf("Parsing %q failed: %v", args[1], err)
}
if dstRemote == "" {
dstRemote = "."
}
@@ -197,7 +201,10 @@ func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs
// NewFsDstFile creates a new dst fs with a destination file name from the arguments
func NewFsDstFile(args []string) (fdst fs.Fs, dstFileName string) {
dstRemote, dstFileName := fspath.Split(args[0])
dstRemote, dstFileName, err := fspath.Split(args[0])
if err != nil {
log.Fatalf("Parsing %q failed: %v", args[0], err)
}
if dstRemote == "" {
dstRemote = "."
}
+5 -2
View File
@@ -267,8 +267,8 @@ func (fsys *FS) Statfs(path string, stat *fuse.Statfs_t) (errc int) {
stat.Blocks = fsBlocks // Total data blocks in file system.
stat.Bfree = fsBlocks // Free blocks in file system.
stat.Bavail = fsBlocks // Free blocks in file system if you're not root.
stat.Files = 1E9 // Total files in file system.
stat.Ffree = 1E9 // Free files in file system.
stat.Files = 1e9 // Total files in file system.
stat.Ffree = 1e9 // Free files in file system.
stat.Bsize = blockSize // Block size
stat.Namemax = 255 // Maximum file name length?
stat.Frsize = blockSize // Fragment size, smallest addressable data size in the file system.
@@ -299,6 +299,9 @@ func (fsys *FS) Open(path string, flags int) (errc int, fh uint64) {
return translateError(err), fhUnset
}
// FIXME add support for unknown length files setting direct_io
// See: https://github.com/billziss-gh/cgofuse/issues/38
return 0, fsys.openHandle(handle)
}
+17 -2
View File
@@ -4,12 +4,18 @@ import (
"context"
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/operations"
"github.com/spf13/cobra"
)
var (
autoFilename = false
)
func init() {
cmd.Root.AddCommand(commandDefintion)
commandDefintion.Flags().BoolVarP(&autoFilename, "auto-filename", "a", autoFilename, "Get the file name from the url and use it for destination file path")
}
var commandDefintion = &cobra.Command{
@@ -18,13 +24,22 @@ var commandDefintion = &cobra.Command{
Long: `
Download urls content and copy it to destination
without saving it in tmp storage.
Setting --auto-filename flag will cause retrieving file name from url and using it in destination path.
`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(2, 2, command, args)
fsdst, dstFileName := cmd.NewFsDstFile(args[1:])
var dstFileName string
var fsdst fs.Fs
if autoFilename {
fsdst = cmd.NewFsDir(args[1:])
} else {
fsdst, dstFileName = cmd.NewFsDstFile(args[1:])
}
cmd.Run(true, true, command, func() error {
_, err := operations.CopyURL(context.Background(), fsdst, dstFileName, args[0])
_, err := operations.CopyURL(context.Background(), fsdst, dstFileName, args[0], autoFilename)
return err
})
},
+19 -7
View File
@@ -47,10 +47,14 @@ __rclone_custom_func() {
__rclone_init_completion -n : || return
fi
if [[ $cur != *:* ]]; then
local ifs=$IFS
IFS=$'\n'
local remotes=($(command rclone listremotes))
IFS=$ifs
local remote
while IFS= read -r remote; do
for remote in "${remotes[@]}"; do
[[ $remote != $cur* ]] || COMPREPLY+=("$remote")
done < <(command rclone listremotes)
done
if [[ ${COMPREPLY[@]} ]]; then
local paths=("$cur"*)
[[ ! -f ${paths[0]} ]] || COMPREPLY+=("${paths[@]}")
@@ -62,14 +66,18 @@ __rclone_custom_func() {
else
local prefix=
fi
local ifs=$IFS
IFS=$'\n'
local lines=($(rclone lsf "${cur%%:*}:$prefix" 2>/dev/null))
IFS=$ifs
local line
while IFS= read -r line; do
for line in "${lines[@]}"; do
local reply=${prefix:+$prefix/}$line
[[ $reply != $path* ]] || COMPREPLY+=("$reply")
done < <(rclone lsf "${cur%%:*}:$prefix" 2>/dev/null)
[[ ! ${COMPREPLY[@]} ]] || compopt -o filenames
done
[[ ! ${COMPREPLY[@]} || $(type -t compopt) != builtin ]] || compopt -o filenames
fi
[[ ! ${COMPREPLY[@]} ]] || compopt -o nospace
[[ ! ${COMPREPLY[@]} || $(type -t compopt) != builtin ]] || compopt -o nospace
fi
}
`
@@ -308,7 +316,11 @@ func showBackend(name string) {
optionsType = "advanced"
for _, opt := range opts {
done[opt.Name] = struct{}{}
fmt.Printf("#### --%s\n\n", opt.FlagName(backend.Prefix))
shortOpt := ""
if opt.ShortOpt != "" {
shortOpt = fmt.Sprintf(" / -%s", opt.ShortOpt)
}
fmt.Printf("#### --%s%s\n\n", opt.FlagName(backend.Prefix), shortOpt)
fmt.Printf("%s\n\n", opt.Help)
fmt.Printf("- Config: %s\n", opt.Name)
fmt.Printf("- Env Var: %s\n", opt.EnvVarName(backend.Prefix))
+1 -1
View File
@@ -58,7 +58,7 @@ a bit of go code for each one.
`,
Hidden: true,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(1, 1E6, command, args)
cmd.CheckArgs(1, 1e6, command, args)
for i := range args {
f := cmd.NewFsDir(args[i : i+1])
cmd.Run(false, false, command, func() error {
+5
View File
@@ -73,6 +73,11 @@ func (f *File) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenR
return nil, translateError(err)
}
// If size unknown then use direct io to read
if handle.Node().DirEntry().Size() < 0 {
resp.Flags |= fuse.OpenDirectIO
}
return &FileHandle{handle}, nil
}
+2 -2
View File
@@ -58,8 +58,8 @@ func (f *FS) Statfs(ctx context.Context, req *fuse.StatfsRequest, resp *fuse.Sta
resp.Blocks = fsBlocks // Total data blocks in file system.
resp.Bfree = fsBlocks // Free blocks in file system.
resp.Bavail = fsBlocks // Free blocks in file system if you're not root.
resp.Files = 1E9 // Total files in file system.
resp.Ffree = 1E9 // Free files in file system.
resp.Files = 1e9 // Total files in file system.
resp.Ffree = 1e9 // Free files in file system.
resp.Bsize = blockSize // Block size
resp.Namelen = 255 // Maximum file name length?
resp.Frsize = blockSize // Fragment size, smallest addressable data size in the file system.
+4
View File
@@ -3,11 +3,15 @@
package mount
import (
"runtime"
"testing"
"github.com/rclone/rclone/cmd/mountlib/mounttest"
)
func TestMount(t *testing.T) {
if runtime.NumCPU() <= 2 {
t.Skip("FIXME skipping mount tests as they lock up on <= 2 CPUs - See: https://github.com/rclone/rclone/issues/3154")
}
mounttest.RunTests(t, mount)
}
+1 -1
View File
@@ -16,7 +16,7 @@ import (
var (
// Flags
iterations = flag.Int("n", 1E6, "Iterations to try")
iterations = flag.Int("n", 1e6, "Iterations to try")
maxBlockSize = flag.Int("b", 1024*1024, "Max block size to read")
)
+1 -1
View File
@@ -17,7 +17,7 @@ import (
var (
// Flags
iterations = flag.Int("n", 1E6, "Iterations to try")
iterations = flag.Int("n", 1e6, "Iterations to try")
maxBlockSize = flag.Int("b", 1024*1024, "Max block size to read")
simultaneous = flag.Int("transfers", 16, "Number of simultaneous files to open")
seeksPerFile = flag.Int("seeks", 8, "Seeks per file")
+19
View File
@@ -23,10 +23,29 @@ const (
logTimeFormat = "2006-01-02 15:04:05"
)
var (
initTerminal func() error
writeToTerminal func([]byte)
)
// Initialise the VT100 terminal
func initTerminalVT100() error {
return nil
}
// Write to the VT100 terminal
func writeToTerminalVT100(b []byte) {
_, _ = os.Stdout.Write(b)
}
// startProgress starts the progress bar printing
//
// It returns a func which should be called to stop the stats.
func startProgress() func() {
if os.Getenv("TERM") != "" {
initTerminal = initTerminalVT100
writeToTerminal = writeToTerminalVT100
}
err := initTerminal()
if err != nil {
fs.Errorf(nil, "Failed to start progress: %v", err)
+4 -8
View File
@@ -2,12 +2,8 @@
package cmd
import "os"
func initTerminal() error {
return nil
}
func writeToTerminal(b []byte) {
_, _ = os.Stdout.Write(b)
func init() {
// Default terminal is VT100 for non Windows
initTerminal = initTerminalVT100
writeToTerminal = writeToTerminalVT100
}
+8 -2
View File
@@ -16,7 +16,13 @@ var (
ansiParser *ansiterm.AnsiParser
)
func initTerminal() error {
func init() {
// Default terminal is Windows console for Windows
initTerminal = initTerminalWindows
writeToTerminal = writeToTerminalWindows
}
func initTerminalWindows() error {
winEventHandler := winterm.CreateWinEventHandler(os.Stdout.Fd(), os.Stdout)
if winEventHandler == nil {
err := syscall.GetLastError()
@@ -29,7 +35,7 @@ func initTerminal() error {
return nil
}
func writeToTerminal(b []byte) {
func writeToTerminalWindows(b []byte) {
// Remove all non-ASCII characters until this is fixed
// https://github.com/Azure/go-ansiterm/issues/26
r := []rune(string(b))
+10 -8
View File
@@ -69,13 +69,14 @@ rclone rc server, eg:
Use "rclone rc" to see a list of all possible commands.`,
Run: func(command *cobra.Command, args []string) {
cmd.CheckArgs(0, 1E9, command, args)
cmd.CheckArgs(0, 1e9, command, args)
cmd.Run(false, false, command, func() error {
ctx := context.Background()
parseFlags()
if len(args) == 0 {
return list()
return list(ctx)
}
return run(args)
return run(ctx, args)
})
},
}
@@ -110,7 +111,7 @@ func setAlternateFlag(flagName string, output *string) {
// do a call from (path, in) to (out, err).
//
// if err is set, out may be a valid error return or it may be nil
func doCall(path string, in rc.Params) (out rc.Params, err error) {
func doCall(ctx context.Context, path string, in rc.Params) (out rc.Params, err error) {
// If loopback set, short circuit HTTP request
if loopback {
call := rc.Calls.Get(path)
@@ -141,6 +142,7 @@ func doCall(path string, in rc.Params) (out rc.Params, err error) {
if err != nil {
return nil, errors.Wrap(err, "failed to make request")
}
req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
req.Header.Set("Content-Type", "application/json")
if authUser != "" || authPass != "" {
@@ -182,7 +184,7 @@ func doCall(path string, in rc.Params) (out rc.Params, err error) {
}
// Run the remote control command passed in
func run(args []string) (err error) {
func run(ctx context.Context, args []string) (err error) {
path := strings.Trim(args[0], "/")
// parse input
@@ -208,7 +210,7 @@ func run(args []string) (err error) {
}
// Do the call
out, callErr := doCall(path, in)
out, callErr := doCall(ctx, path, in)
// Write the JSON blob to stdout if required
if out != nil && !noOutput {
@@ -222,8 +224,8 @@ func run(args []string) (err error) {
}
// List the available commands to stdout
func list() error {
list, err := doCall("rc/list", nil)
func list(ctx context.Context) error {
list, err := doCall(ctx, "rc/list", nil)
if err != nil {
return errors.Wrap(err, "failed to list")
}
+2 -2
View File
@@ -61,7 +61,7 @@ See the [rc documentation](/rc/) for more info on the rc flags.
}
if rcflags.Opt.HTTPOptions.BasicUser == "" {
rcflags.Opt.HTTPOptions.BasicUser = "gui"
fs.Infof("Using default username: %s \n", rcflags.Opt.HTTPOptions.BasicUser)
fs.Infof(nil, "Using default username: %s \n", rcflags.Opt.HTTPOptions.BasicUser)
}
if rcflags.Opt.HTTPOptions.BasicPass == "" {
randomPass, err := random.Password(128)
@@ -69,7 +69,7 @@ See the [rc documentation](/rc/) for more info on the rc flags.
log.Fatalf("Failed to make password: %v", err)
}
rcflags.Opt.HTTPOptions.BasicPass = randomPass
fs.Infof("No password specified. Using random password: %s \n", randomPass)
fs.Infof(nil, "No password specified. Using random password: %s \n", randomPass)
}
rcflags.Opt.Serve = true
}
+18 -3
View File
@@ -77,10 +77,17 @@ func (cds *contentDirectoryService) cdsObjectToUpnpavObject(cdsObject object, fi
}
if fileInfo.IsDir() {
children, err := cds.readContainer(cdsObject, host)
if err != nil {
return nil, err
}
obj.Class = "object.container.storageFolder"
obj.Title = fileInfo.Name()
ret = upnpav.Container{Object: obj}
return
return upnpav.Container{
Object: obj,
ChildCount: len(children),
}, nil
}
if !fileInfo.Mode().IsRegular() {
@@ -245,7 +252,15 @@ func (cds *contentDirectoryService) Handle(action string, argsXML []byte, r *htt
"UpdateID": cds.updateIDString(),
}, nil
case "BrowseMetadata":
result, err := xml.Marshal(obj)
node, err := cds.vfs.Stat(obj.Path)
if err != nil {
return nil, err
}
upnpObject, err := cds.cdsObjectToUpnpavObject(obj, node, host)
if err != nil {
return nil, err
}
result, err := xml.Marshal(upnpObject)
if err != nil {
return nil, err
}
+14 -6
View File
@@ -21,39 +21,46 @@ var Assets = func() http.FileSystem {
fs := vfsgen۰FS{
"/": &vfsgen۰DirInfo{
name: "/",
modTime: time.Date(2019, 5, 26, 18, 31, 10, 658546797, time.UTC),
modTime: time.Date(2019, 9, 15, 16, 40, 10, 576397038, time.UTC),
},
"/ConnectionManager.xml": &vfsgen۰CompressedFileInfo{
name: "ConnectionManager.xml",
modTime: time.Date(2019, 5, 26, 18, 33, 10, 628803626, time.UTC),
modTime: time.Date(2019, 5, 27, 15, 4, 10, 117829131, time.UTC),
uncompressedSize: 5505,
compressedContent: []byte("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xd4\x57\x4d\x6f\xdb\x3c\x0c\x3e\x3b\xbf\x22\xf0\x3d\xaf\x5b\xe0\x3d\x0c\x85\xe2\xa2\x4b\x3f\x10\x6c\x45\x83\x36\x0d\xb0\x53\xa1\xc9\x6c\xaa\xd5\xa6\x0c\x89\xee\xc7\xbf\x1f\x62\xc7\xb5\xdd\x38\x8b\xe3\xc8\x59\x76\x93\x68\x91\xcf\x63\x92\x22\x29\x76\xfa\x16\x85\xfd\x17\xd0\x46\x2a\x1c\xba\xc7\xff\x1d\xb9\x7d\x40\xa1\x02\x89\xf3\xa1\x7b\x3f\xbd\x1c\x7c\x71\x4f\xfd\x1e\x33\x22\x0e\xfa\x6f\x51\x88\x66\xe8\x26\x1a\x4f\x8c\x78\x82\x88\x9b\x41\x12\x63\x3c\x50\x7a\x7e\x62\x40\xbf\x48\x01\x83\xe3\xc1\x91\xeb\xf7\x1c\x66\x62\x10\xb3\xcc\xac\xdf\x73\x1c\x16\xf1\x5f\x4a\xfb\xc7\xcc\xcb\x16\xa9\x48\xa2\xd2\xfe\x11\xf3\xb2\x45\xcf\x61\x5e\x55\x8b\x71\x41\x52\xe1\x77\x69\x28\x55\xc8\xb6\x8b\xa5\xc3\x90\x47\xe0\x5f\x01\x4d\xb4\x22\x25\x54\x38\xc6\x47\xc5\xbc\x54\x9a\x7e\xe7\x7a\x9e\x44\x80\x94\x2b\x97\x44\xd9\x76\x69\xe2\x4e\x25\x5a\x40\x49\xd3\x71\x58\x20\x35\x64\x50\x2a\x21\xe6\x15\xdb\xe5\x77\x0d\x21\x27\x08\xee\x88\x13\xcc\xb8\x96\xfc\x67\x98\x1b\xaa\xd2\xa9\x3d\x98\x91\xf1\xaa\x6c\xd6\x90\x93\xf8\x6c\x83\x9a\xc4\xe7\x96\xc4\x8a\xed\x47\x14\xbc\x22\x0c\xab\x11\x99\x68\x88\xb9\x86\x4b\xa5\x47\x0a\x31\xe3\xd6\x26\x2c\xb7\x10\x29\x82\x35\xc1\xad\xf8\x41\x62\x53\x37\x9c\x3d\x9c\xdd\x5e\x3d\x4c\x7f\x4c\x2e\x1e\xec\x86\x69\x02\x50\xfa\xdd\x6b\x8e\x7c\x0e\xda\x2a\xdf\x1a\xeb\x76\x49\x8f\xcf\x3b\xe2\xbb\x30\xbc\x2b\xd5\xf3\x1c\xde\x2a\xc7\x92\xd5\x5d\x09\x36\xf1\xe3\x16\xf7\xb5\x33\x47\x9e\xcd\xa6\x9a\xa3\x89\x95\x26\xdb\x44\x3f\x99\xde\x95\xe9\xad\x30\xb6\x19\x2e\x4d\x76\x56\xfa\x8a\x50\x8d\x54\x14\x87\x40\xd0\xa6\xf0\x1d\xda\x95\xdc\xd6\x0b\x57\x40\xa3\x44\x6b\x40\x2a\x03\x9a\x5d\x5d\x61\x2c\xe4\x42\x3d\xaf\xfd\x7a\xa2\xe5\x94\x72\x68\x59\x71\xb8\xb7\xf6\x9f\xaf\x7c\x4d\x66\x9e\x76\x44\xff\xe2\xd0\xb3\x6b\xf3\xdb\xfb\xd4\x73\x08\xdd\x7a\xe3\xd8\xd3\x8e\xa4\xc5\xb9\x67\xa1\x98\xd8\x28\xcd\xb5\x3e\xcc\xad\xdb\xa9\xd0\xf9\x72\xf9\x89\x2d\x1f\xac\xa9\xd5\x69\x6e\x92\x99\x32\x48\xdf\x00\x06\x17\x2f\x80\x64\x86\xee\x3b\x18\xb7\x54\xdd\xeb\x9e\x7b\x45\x5d\x0f\x38\xf1\xe9\x7b\x0c\xbe\x21\x2d\x71\xce\xbc\x0f\x41\x4a\xca\x7c\xfe\x95\x2d\x70\x57\xde\x72\xfb\x40\xdd\xd4\xd2\x2d\x22\xa3\x2a\x03\xff\x31\x31\x1a\xe2\x3b\x8c\x87\xa1\x7a\x85\x60\xc6\xc3\x04\xca\xad\xb6\x24\xf6\x6f\xbe\x31\xaf\x22\xa8\x39\x33\x52\x48\x80\x74\xa9\x74\xc4\xe9\x5a\x9a\x88\x93\x78\xda\xac\x36\x46\x93\x3c\x3e\x4a\x21\x01\xe9\x2b\xc7\xe0\x55\x06\xd4\x40\xed\x1e\x35\x84\xa9\x83\x46\x4f\x1c\x11\xc2\x26\x2a\xcf\xa8\x5e\xb1\xe6\x60\x55\x54\xdc\x0f\xcb\xa1\x59\xed\x03\x7b\xc9\x8d\xba\x52\x69\x23\x29\xc6\x18\x2f\x8a\xd8\x26\xb7\xdf\x24\x54\x7f\xae\x53\xaf\xef\xa1\x0c\x34\x88\x78\xa5\x87\x16\xd8\xf2\xff\x4e\x70\xd7\xcd\x71\x9d\x03\x7f\x1e\x6d\xb7\x01\x64\x5e\x4d\xb3\x61\x9e\x11\x71\xe0\xff\x0e\x00\x00\xff\xff\x2a\x62\x9d\xe1\x81\x15\x00\x00"),
},
"/ContentDirectory.xml": &vfsgen۰CompressedFileInfo{
name: "ContentDirectory.xml",
modTime: time.Date(2019, 5, 26, 18, 33, 13, 318736031, time.UTC),
modTime: time.Date(2019, 5, 27, 15, 4, 10, 118094855, time.UTC),
uncompressedSize: 14527,
compressedContent: []byte("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xec\x5a\x5f\x6f\xa3\x38\x10\x7f\x4e\x3f\x45\x95\xf7\x1e\xad\x74\x4f\x2b\x37\xab\xdd\x24\xad\x22\xb5\x4d\x04\xd9\x6a\xef\x29\x72\x61\x9a\x78\x17\x6c\xce\x1e\xda\xe4\xdb\x9f\x80\x90\x40\x0a\xf9\x03\x86\x65\xf7\xf2\xc6\x3f\xff\xe6\x37\xc3\x78\xc6\x1e\x0f\xf9\xbc\xf4\xdc\xcb\x37\x90\x8a\x09\x7e\xdb\xbd\xf9\xeb\xba\xfb\xb9\x77\x41\x94\xed\x3b\x97\x4b\xcf\xe5\xea\xb6\x1b\x48\xfe\x49\xd9\x0b\xf0\xa8\xba\x0a\x7c\xee\x5f\x09\x39\xff\xa4\x40\xbe\x31\x1b\xae\x6e\xae\xae\xbb\xbd\x8b\x0e\x51\x3e\xd8\xcf\x31\x4a\xef\xa2\xd3\x21\x1e\xfd\x21\x64\xef\x86\x18\xf1\x45\xf4\x88\x71\x21\x7b\xd7\xc4\x88\x2f\x2e\x3a\xc4\xc8\x8e\x22\xd4\x46\x26\xf8\x03\x53\x18\x0d\x88\x6f\xc3\xcb\x0e\xe1\xd4\x83\xde\x3d\xa0\x05\x54\xda\x8b\x3e\xf5\xe9\x0b\x73\x19\x32\x50\xc4\x88\xde\x45\x5f\x51\x39\x0f\x3c\xe0\x98\x40\xa4\x1e\xc5\xb7\x6b\xa0\x0d\x4a\x7a\x74\xa7\x43\x1c\x26\x21\x16\x2a\x02\x24\xc6\xf6\x76\xfd\x5e\x82\x4b\x11\x1c\x0b\x29\xc2\x33\x95\x8c\xbe\xb8\x29\xb0\x14\xa5\xdc\x0f\x63\x42\x46\x86\xd1\xf6\x76\xa3\xb6\xb1\xd5\x3b\xdf\x04\x42\x62\x65\x03\xc4\x18\x5a\xd4\xff\x40\xa7\x6e\xe5\x87\x4b\x04\x1e\xfa\x8c\x0e\x2b\xa4\xc1\x74\x99\xa3\x80\x60\x9d\x76\xb9\x03\x8a\x81\x84\xf0\xfb\x32\x96\xc8\x1f\x5e\xd6\x06\x19\xb4\x5a\xbd\x61\xa5\x10\xbc\x6f\xbe\x43\x11\x46\x83\x32\x8a\x8f\x1c\x1d\xff\x7c\x87\x46\x6d\x2a\x7f\x95\xe2\x5d\x41\x19\x3d\xc7\x2f\x3f\xc0\xc6\x8c\x8d\x32\xda\x32\x7e\xac\xb2\x5f\x66\x5f\xcc\xfb\xd9\xf4\x9f\xc9\x70\xb6\x05\x3d\x5a\xe3\x02\x7a\xb1\x62\x77\x2e\x9d\x6b\x25\x98\x86\xad\x4a\xf1\x8e\xb9\x08\x52\x2b\xbd\x04\xb2\x2a\x35\x0b\xa9\x44\xc6\xe7\x23\xee\xc0\x52\x2b\xc3\x35\x62\x55\x82\x26\xfc\x1b\x80\x42\x70\xfa\x22\xe0\x85\x11\xa6\x14\xc3\x35\x62\x65\x13\x86\x59\x4c\x32\x04\xc9\xa8\x56\x7e\x59\xe0\xea\x86\x54\x81\xab\x23\x44\xa7\x18\x26\x98\x55\xb9\x3d\x05\xde\x0b\x48\x13\x30\x90\x1c\x74\x84\x55\xfd\x7f\x79\x2a\x90\xba\x8f\x14\xed\x05\xe8\xc8\xf5\xfa\x09\xe6\xa4\x32\x0d\xe4\x1a\xc8\x4c\xf1\x22\xb8\x4c\x66\xea\x0b\x8e\x94\x71\x90\xad\x4d\x4e\xeb\x05\x7e\x2d\xd1\x61\x07\xfa\x9c\xa4\xce\x49\xea\x9c\xa4\xce\x49\xea\x9c\xa4\xea\x48\x52\x7d\x09\x14\x21\x4e\x0c\x7f\x66\xaa\x1a\xba\x10\x3e\x2b\xf4\x9b\x52\xf4\x74\xcd\xbd\x43\x7b\xd0\x72\x7e\xa3\xcf\x78\xad\x88\x5b\xa7\xfa\xf4\x00\x14\x4a\xb1\x2a\xef\xd4\x6d\xa9\x0c\x9c\xaa\x78\x1c\x2f\x7e\x7f\xbd\x8b\x62\x4d\x20\x25\x70\x9c\xd2\xf9\x33\x75\x03\xd0\xca\x32\x01\x3d\xb1\x40\x57\x94\x52\xe1\xbd\x4d\x2c\x4f\xf5\xa3\x47\xf1\xf6\xe7\x7a\xd1\x13\xbc\x4f\x68\xe8\x47\x6d\x66\xd8\x9a\xbc\x70\xaa\xeb\x8c\x3c\x5f\x48\x34\x41\x89\x40\xda\xa5\xca\xb2\x56\x34\xf2\x9b\x39\xd2\xfa\x77\x22\xbc\xaa\x3f\x26\x4c\x2c\x8c\xd3\x50\x6e\x2b\xf9\x4d\x25\xe5\xea\x75\xdf\x5a\xac\x9c\xdf\xa4\x71\x6b\xf3\x9c\xe1\xf2\xec\x39\x67\xcf\x29\xe3\x39\x16\x0a\x3f\x11\x54\xc5\x7f\x0e\x1b\xa1\x5c\xd2\x6e\xc2\x06\x03\x70\x01\xa1\x8a\xf6\xc9\xd8\x5f\xeb\x9f\x25\x4e\x3c\x13\xfb\x4e\xa4\x98\x4b\x50\xa5\x8e\xbd\xdb\xf4\xeb\x0f\x50\x0c\x01\x02\xcd\x55\x90\x5d\x6c\x5d\x5c\x1f\x80\xcf\x71\x51\x0f\xd7\x04\x5b\x17\xd7\xa8\xc6\x54\x0f\xd5\x35\x74\xcd\x45\x1c\x13\x5e\x41\x02\x2f\x37\xfb\xdb\x5f\xc7\x69\xff\xb6\xe2\x37\x5c\xae\x7f\x9f\xb5\xad\x51\x26\x7d\xcc\x12\x03\xbb\xf5\x6e\x76\xbf\xcf\x2c\xc0\xaf\x42\xfc\xf4\xa8\xfc\x59\x6a\xea\x50\x84\xb9\x90\xab\xe9\xca\xd7\xbb\xdb\xcf\x02\x57\x2e\xe5\x69\x9e\x3a\xe6\xff\x60\x52\x4f\x84\xb2\xc0\x16\xbc\xf0\x54\xa4\x14\xbf\x18\x55\x97\x4b\x27\x97\xeb\x57\x64\xdd\x86\x1a\xa1\x4e\x13\x48\xa2\xd2\x42\x2e\x15\x70\x67\xf8\x06\x1c\xd5\x6d\x97\x8b\x6e\x7a\x2d\xbd\xb7\x99\xd4\xa1\x48\x43\x6f\xec\x29\x94\x8c\xcf\x89\xb1\x79\x10\x71\x52\xbb\x9a\x1c\x2f\x76\x4f\x03\x67\xad\x42\x0f\x36\x4e\x6a\x94\xbe\x02\x95\x11\x5f\xd8\xa9\xb7\x11\x11\xb0\xbf\x35\x0a\xdc\xe4\xf8\x44\x66\x43\x8a\x6e\x57\xc1\xcd\xfc\xd7\x82\x7c\x56\x9b\xbc\xdc\xd0\xd3\xac\xdc\x0f\xe7\x34\x8d\x48\x2d\xec\x02\x69\x44\x7a\x6e\x83\xe4\x7e\xc9\x1d\x42\x5d\x57\xbc\x83\xb3\xa9\xa2\x27\xd1\x3f\xf5\x78\xdd\x79\xf9\x08\x48\xc3\xb1\xc4\xc8\xbc\x2c\xfc\x7e\x10\xa5\x81\xfe\x82\xb9\x8e\x04\x9e\x33\x2a\xfb\x68\x1b\xc9\xb5\x18\xe3\x43\x97\x4b\x33\x0e\x90\xdf\x7d\xd1\x88\xec\xdd\xa6\x19\x6d\x51\xb3\x50\xe2\x6e\x13\x4c\xfd\x12\xeb\x4c\x0e\x85\x42\x73\x6b\x21\xcd\x89\xfd\x50\xdf\xd0\x31\xa5\xfb\xe3\xc7\xc9\xc3\x70\x3a\x1c\x1c\x9e\xcd\x43\xd3\x1c\x9b\x87\x3f\x1b\x3d\xcd\x26\xe6\xf8\xde\x1c\x5a\xd6\xe1\x8f\xad\xe9\x78\x32\xc9\x15\x5e\x6b\x50\x28\x2c\xc3\x34\x32\x41\x8b\x0a\x2b\xcd\x08\xcf\x9c\x94\x36\x2b\x3b\x5b\x41\x4d\xcd\x1c\xc9\xea\x89\x4b\xf9\x1b\xd0\x3d\x53\xb6\x43\x1c\x78\xa5\x81\x8b\x91\x8d\x2e\x0d\x9d\x2b\x91\x63\xc3\x46\x8d\x1c\x92\x0d\xd6\x2f\xa6\x91\xa9\x5f\x1c\x1d\xce\x8e\xa1\x43\x8c\x9c\x5d\x1e\x31\x94\xed\x3b\xbd\xff\x02\x00\x00\xff\xff\x1a\x57\x6a\x92\xbf\x38\x00\x00"),
},
"/X_MS_MediaReceiverRegistrar.xml": &vfsgen۰CompressedFileInfo{
name: "X_MS_MediaReceiverRegistrar.xml",
modTime: time.Date(2019, 5, 26, 18, 29, 30, 798626046, time.UTC),
modTime: time.Date(2019, 5, 27, 15, 4, 10, 118232995, time.UTC),
uncompressedSize: 2485,
compressedContent: []byte("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xec\x56\xc1\x6a\xdb\x40\x10\x3d\xcb\x5f\x61\x74\x77\x36\x86\xd2\x43\x58\x6f\x30\x38\x04\x43\x0b\x45\x71\x0d\x3d\x99\xb5\x76\x50\xa6\x95\x66\xd5\x9d\x95\x49\xfa\xf5\x45\x5a\x39\x92\x13\x25\x14\x27\x4e\x2e\xbd\xcd\xcc\x7a\xde\x7b\xde\xa7\x19\x49\x5e\xde\x15\xf9\x78\x07\x8e\xd1\xd2\x2c\x9e\x9e\x9d\xc7\xe3\x4b\x35\x92\x9c\x96\x66\x7c\x57\xe4\xc4\xb3\xb8\x72\x74\xc1\xe9\x2d\x14\x9a\x27\x55\x49\xe5\xc4\xba\xec\x82\xc1\xed\x30\x85\xc9\x74\x72\x1e\xab\x51\x24\xb9\x84\x74\x1d\x60\xd4\x28\x8a\x64\xa1\x7f\x5a\xa7\xa6\x52\x84\xa0\x29\x21\x59\xa7\xce\xa5\x08\xc1\x28\x92\xe2\xb0\x4b\xea\xd4\xa3\xa5\x2f\xc8\xbe\x69\x08\x69\x1d\x46\x92\x74\x01\x6a\xc9\xf3\xca\xdf\x5a\x87\x7f\xc0\x48\xd1\x94\x9a\x43\xed\xb2\xaa\x00\xf2\xfb\xce\x5e\x29\xa4\x6d\xff\x02\x6a\xcd\xcb\x45\xaf\x37\x8a\xa4\x41\x07\x81\x09\x49\x8a\x2e\x6b\x8f\x1d\xe4\xda\x83\xb9\xf1\xda\xc3\x5a\x3b\xd4\xdb\x1c\xd4\x7c\x33\x4f\xae\x37\xab\x1f\xdf\xae\x36\x1d\xe8\xe0\x2f\x83\x1c\x71\xa8\x67\x58\x5e\x02\x5c\xe5\xfe\x39\x71\xb6\xf2\x47\xa8\xdb\x63\xfe\xb3\xb6\x2e\x7d\xf0\x41\x74\x46\x3c\xf5\x24\x81\x0c\xd9\x83\x0b\xd7\x70\x8c\x2b\x01\xc1\xe9\x1a\x38\x81\xdf\x5f\x39\x7b\x53\x7f\x86\xe0\x5f\xef\x54\x1f\x93\xcb\x17\x34\x1f\x6b\xdb\x00\xc1\xc9\x3c\x5c\xf2\x5a\xe7\x68\x6a\xf0\xff\x63\xf5\x01\x63\xb5\x0f\xdb\x23\xd9\x2e\xd7\x06\x75\xb5\x87\x94\xdc\x27\x19\x33\x90\xb9\xda\x01\x79\x9e\xc5\x64\xe3\x9e\x9b\x83\x97\xd8\xb9\x6a\xb4\xd7\xab\xfb\x12\x14\x7b\x87\x94\x49\xf1\x50\x68\x44\xf1\xe3\xbf\x72\x0c\xef\x93\x2b\xef\x58\x91\xfc\x89\x28\x5f\xd8\x22\x1d\xfd\x16\xe9\x6c\xab\x19\x3e\x7f\x7a\x07\x15\x8f\x17\xc3\x5b\xcb\xb8\x07\x3e\xd0\xd1\xbe\x1c\x1b\xf6\x6b\xa7\xc9\x83\xf9\x5e\xd6\x63\xfd\xcc\x13\x50\xe1\xc9\xe8\x17\x40\xf8\xae\xec\xed\x06\x43\x4b\x37\x55\x9a\x02\x98\x0f\x62\x4f\x60\x67\x7f\xbd\x9e\x5b\x8a\x81\x25\x20\x45\xfd\x5d\xa6\xfe\x06\x00\x00\xff\xff\xfa\x10\x27\xa0\xb5\x09\x00\x00"),
},
"/rclone-120x120.png": &vfsgen۰FileInfo{
name: "rclone-120x120.png",
modTime: time.Date(2019, 5, 26, 18, 29, 30, 798942692, time.UTC),
modTime: time.Date(2019, 5, 27, 15, 4, 10, 118634349, time.UTC),
content: []byte("\x89\x50\x4e\x47\x0d\x0a\x1a\x0a\x00\x00\x00\x0d\x49\x48\x44\x52\x00\x00\x00\x78\x00\x00\x00\x78\x08\x06\x00\x00\x00\x39\x64\x36\xd2\x00\x00\x00\x04\x67\x41\x4d\x41\x00\x00\xb1\x8f\x0b\xfc\x61\x05\x00\x00\x00\x20\x63\x48\x52\x4d\x00\x00\x7a\x26\x00\x00\x80\x84\x00\x00\xfa\x00\x00\x00\x80\xe8\x00\x00\x75\x30\x00\x00\xea\x60\x00\x00\x3a\x98\x00\x00\x17\x70\x9c\xba\x51\x3c\x00\x00\x00\x06\x62\x4b\x47\x44\x00\xff\x00\xff\x00\xff\xa0\xbd\xa7\x93\x00\x00\x00\x09\x70\x48\x59\x73\x00\x00\x0b\x13\x00\x00\x0b\x13\x01\x00\x9a\x9c\x18\x00\x00\x00\x07\x74\x49\x4d\x45\x07\xe3\x03\x0b\x05\x07\x12\x76\x96\x1b\x39\x00\x00\x50\x82\x49\x44\x41\x54\x78\xda\x9d\xbd\x77\x7c\x1c\xc5\xfd\xff\xff\x9c\xdd\xbb\x93\x74\xea\xd5\x92\x2c\x57\xd9\x96\xbb\x71\xc1\xc6\x60\x30\x18\x30\x98\x5e\x42\x0f\x04\x42\xe0\x13\x92\x40\xc2\x87\x34\x42\x12\x48\x03\xd2\x49\x08\x9f\x84\x10\x02\x84\xd0\x7b\x09\xc5\x98\xe2\x42\x31\xd8\xb8\x77\x5b\xee\xb6\x9a\xd5\xbb\x6e\x77\x7e\x7f\x6c\x9b\xd9\x5b\x99\x7c\x7f\xeb\xc7\xfa\x4e\x77\xb7\xbb\x33\xf3\x9a\xf7\x7b\xde\x6d\xde\x6f\xf1\xc0\x8e\x2e\x29\x01\x5b\x7a\xa7\xc4\x72\xff\xb6\x94\xcf\x6c\xf7\x33\x09\x48\xf7\xd5\x3b\x04\x20\x84\xfb\xea\xbe\xf7\x0e\xe9\xfe\x27\xd1\xaf\xd1\xae\x75\xaf\x31\x00\x53\x08\x4c\x01\xa6\x01\x31\x21\x88\x09\x9c\xbf\x05\x08\x21\x10\xca\x7d\xa5\x94\x4e\xfb\xb4\xf6\x06\x6d\xf5\xbe\xf3\xda\x80\xdb\x46\x43\x6d\xaf\xd2\xee\xa8\x43\x86\xde\x0b\xe5\x34\x84\xc0\x10\x28\x6d\x74\xdb\x6b\x38\xef\x0d\xc0\x10\xce\xe9\xf5\x55\x86\xda\x66\x49\x89\x25\x21\x65\x43\x4a\x4a\x52\xee\x7b\xef\x73\x1f\x03\xa4\x3f\xfe\xea\xd8\x19\x02\x0c\x04\xc2\x6d\x83\x21\xc0\x54\xdb\x96\x30\x84\x33\x50\xc2\xbb\x99\xc0\x90\x60\x21\x11\x80\xe5\xdd\x4a\x4a\x10\xce\xc3\xa2\x3a\xee\x7e\xed\x80\xab\x7c\x29\x43\xbf\x0f\x83\xec\x0f\x96\xf7\x87\x3b\x8c\xc2\xeb\x14\x01\xa8\x86\x72\xb5\x37\xc9\x54\x30\xad\x30\xb8\xee\x20\x82\x0c\xda\x87\x40\x0a\x30\xa4\x0e\x96\xff\x78\x11\x4c\x48\xbe\xa0\xcd\x12\x89\x54\xda\x27\x90\x58\x42\x04\x33\xca\x7f\x1f\x80\xab\x4e\x74\xe9\xfe\x21\x23\x1e\x16\xfe\xad\xfa\x6c\xbd\x1d\xc2\x27\x8e\x60\xf2\x0a\x17\x78\x88\x25\x0c\x90\x08\x9f\x02\x2c\x29\x31\x24\x18\xd2\x19\x64\x61\x7b\x40\x2b\x20\x03\x22\x82\x2a\xfd\x7e\x85\x3b\xe1\x37\x56\xf9\xc3\x9f\xd5\x22\xbd\xf3\x52\x22\x8c\x80\x56\x25\x02\xd3\x05\xce\xbb\xce\x07\x58\x99\xdd\x1a\xd7\x51\x66\xbc\xd7\x16\x21\x1d\x10\x0c\x9c\x3e\x0b\xe1\x7d\xa6\x0c\xdc\x20\x13\x32\x8d\x63\xb9\x83\x69\xba\x9d\xf1\xbf\xb7\x65\x40\xb2\x6e\xdb\x7d\xca\x57\x26\x8f\x37\x1e\x83\x4d\x7c\xf5\x59\xd2\x1f\xa7\xe0\x57\x0e\xe7\x11\x3e\x07\xf2\x38\x85\xa1\x80\x6b\x08\x88\x65\x18\xce\x65\x01\x4b\x10\xa4\x24\x18\x2e\xb0\xc2\x70\x81\xb5\x9d\x41\x91\x52\xa6\xb1\x33\x3b\xa2\x75\xd2\xed\x49\xe4\xac\x55\x26\xb8\x70\x39\x85\x0d\x98\x08\xe7\x7b\xa1\x0f\x94\x74\x29\xd9\x08\x01\x60\x4b\xe9\x83\x1c\x06\x56\x05\xd7\x7b\xa6\x37\xd3\x3d\x48\x0c\x77\xe0\xc2\x94\x81\xd6\x66\x99\xc6\x85\xfc\x25\x45\x0a\x6c\x01\x52\x48\xa4\x11\xdc\x41\xda\xd2\x7f\x98\x07\xa2\x21\x1c\x82\x51\xc7\x2c\x6d\x6c\x22\x26\x91\xf7\x87\x90\xce\x3d\x02\xb8\x83\xe5\xc5\x0c\x81\x6b\x2a\x80\xc7\x32\x4c\x8f\x9d\x39\xc0\x7a\x14\x9c\x22\x00\xd6\x7b\x82\xda\x20\xe1\x51\x91\x74\x3b\xa1\xb2\x65\x75\x90\xc2\x1d\x09\x4d\x06\x95\x1a\xa4\x94\x18\x42\x04\xbf\xb5\x9d\x81\x93\x12\x0c\x11\x4c\x2c\xef\x16\xbe\x5c\xe0\x02\x6b\x29\xc0\xaa\xf2\x42\x3a\x35\xe8\x40\x87\x01\x8e\x6a\x7f\x1a\xc0\x12\x84\x90\x98\xb8\x13\xdf\x96\x01\x29\x79\x13\xc3\x76\xe4\x09\x6f\x92\x7a\x2c\x35\x4c\x00\x47\x3b\xfc\xe5\xc0\x05\xd9\x13\x24\x34\x59\xc2\x93\x5f\x08\x28\xd7\x74\x01\x8f\x25\x0c\xe1\xcf\x78\x43\x4a\x52\x52\x20\x6c\x10\x86\x04\x5b\x01\xd6\x65\xcd\x86\x4b\xc5\x76\x44\x63\x54\x8a\x19\xfc\xbd\xc2\x66\x64\xc0\x62\x54\xb6\x85\x3b\x68\x1e\x35\xd8\x2e\xb5\x78\xc2\x5b\x70\x4f\xa9\x0b\x53\x8a\xc0\x15\x9e\x74\x01\xc0\x42\x01\x17\x0c\xc3\x70\x26\x95\xd0\x29\xd0\xb6\x6d\x6c\x29\xd3\xfa\xe5\xdf\xcb\x5d\xc7\xa5\x4b\xc1\x3e\xb0\x2e\xe7\x09\xae\x74\xde\x1b\x1e\x20\xa4\xdf\x73\xd0\xc9\x2f\x82\xc9\xa4\xce\xc2\x40\xc8\x73\x7e\xa3\x0a\x55\x1a\xab\xc6\x5d\x83\x3d\x16\x67\x48\xe1\xac\xb9\x06\x60\x0b\xa4\x21\xfd\xd9\x29\x0d\x81\x6d\x3b\x83\x26\x85\xc0\x70\x07\x57\x5d\x57\xc2\x6c\xd1\x26\x00\x55\x1d\xa0\xa0\xbd\xd2\x6f\xac\x2d\x83\xb5\x43\x02\xa6\x0c\x06\xca\x70\xa9\x45\x28\x33\x5e\x46\x81\xea\x4e\xbc\x30\xe5\x79\x83\x62\x78\x13\x48\x08\x4c\xc3\xc4\x4a\x0d\xd0\x5e\x7f\x88\xb6\x43\xfb\xe8\x3a\xd2\x40\xaa\xbf\x8f\x58\x46\x16\x85\xc3\x46\x51\x30\xbc\x9a\x78\x56\xb6\x0b\xb4\xad\xb5\xdd\x1b\x74\x3b\xc4\x6d\xc2\x20\xeb\xb4\xeb\x5e\x9b\x36\xa1\x3d\x61\x4b\x86\xc6\x46\x69\xb7\x08\xfa\xe3\x0b\x9c\x0a\xe5\x7a\x42\x95\xf7\xb7\x27\x70\x09\x01\xb1\xb8\x10\x2e\x85\xc8\x80\x05\xd8\x12\x69\x80\xb4\xbd\x19\xea\x0e\xa2\x10\xd8\xc2\x59\xbb\x0c\xb7\x0f\xe1\xf5\x57\x05\xd7\x5b\x23\x55\x30\xd4\x41\x52\x67\xa3\x10\xee\xe4\x51\x06\xc3\xa1\x5e\xb7\xf1\x32\xfd\x19\xe1\x75\xd8\x93\xaa\xa3\x58\x9f\x37\x48\xa6\x61\xd0\xdf\xd7\xc7\x81\xd5\xcb\xd9\xfc\xc6\x73\x1c\x5c\xf7\x09\x9d\x8d\x75\xa4\xfa\x7a\x91\xb6\x8d\x30\x4c\x32\xf3\x0b\xa9\x98\x38\x9d\xf1\x67\x5c\x4c\xf5\xfc\x45\x64\x16\x14\x63\x49\x2b\x8d\xdd\x1b\x0Line truncated
},
"/rclone-48x48.png": &vfsgen۰FileInfo{
name: "rclone-48x48.png",
modTime: time.Date(2019, 5, 26, 18, 29, 30, 799532805, time.UTC),
modTime: time.Date(2019, 5, 27, 15, 4, 10, 118821186, time.UTC),
content: []byte("\x89\x50\x4e\x47\x0d\x0a\x1a\x0a\x00\x00\x00\x0d\x49\x48\x44\x52\x00\x00\x00\x30\x00\x00\x00\x30\x08\x06\x00\x00\x00\x57\x02\xf9\x87\x00\x00\x00\x04\x67\x41\x4d\x41\x00\x00\xb1\x8f\x0b\xfc\x61\x05\x00\x00\x00\x20\x63\x48\x52\x4d\x00\x00\x7a\x26\x00\x00\x80\x84\x00\x00\xfa\x00\x00\x00\x80\xe8\x00\x00\x75\x30\x00\x00\xea\x60\x00\x00\x3a\x98\x00\x00\x17\x70\x9c\xba\x51\x3c\x00\x00\x00\x06\x62\x4b\x47\x44\x00\xff\x00\xff\x00\xff\xa0\xbd\xa7\x93\x00\x00\x00\x09\x70\x48\x59\x73\x00\x00\x0b\x13\x00\x00\x0b\x13\x01\x00\x9a\x9c\x18\x00\x00\x00\x07\x74\x49\x4d\x45\x07\xe3\x03\x0b\x05\x07\x12\x76\x96\x1b\x39\x00\x00\x15\x39\x49\x44\x41\x54\x68\xde\x6d\x9a\x79\x78\x54\xd5\xdd\xc7\x3f\xe7\xdc\x7b\x67\xcb\x9e\x10\x92\x10\x48\xc2\x8e\x06\x35\xb2\x28\x9b\x68\x5d\x00\x8b\xb8\x61\x71\xa3\xf6\xd5\x56\x5f\xb5\xae\xd5\x5a\x5b\xb5\x5a\xeb\x6e\xab\x52\x95\xd6\x22\xad\x88\x4a\x45\x2d\x95\x4a\x5b\x51\xc4\x6a\x6b\x40\x04\xc1\x40\x58\xc3\x92\x04\x08\x84\x84\xac\xb3\xde\x7b\xce\xfb\xc7\xbd\x77\x66\x68\xdf\x79\x9e\xfb\xcc\x3c\x33\xf3\x9c\x7b\x7e\xdb\xf7\xf7\xfd\x7d\xcf\x15\xaf\xec\xe9\xd7\x29\x0d\x09\x47\x93\x50\x90\x54\x1a\x5b\x81\x06\x04\x20\x04\x00\x28\xed\x7e\x07\x60\x08\xb0\x04\x84\x0c\x41\xd8\x10\x98\xd2\xfd\xde\x51\x90\xd4\xd9\x6b\x68\x24\x02\x43\x82\xf4\xd6\xd3\x64\xd6\x91\x02\x82\x52\xb8\xeb\x48\xb0\xa4\xc0\x10\xa0\x70\xd7\x88\x3b\x10\x75\x34\x71\x47\x93\x54\x1a\xa5\xdd\x35\x4c\x29\x08\x48\x08\x48\x81\x59\x60\x49\x92\x4a\x13\x13\x20\x1d\x77\x61\xad\x35\x29\x0d\x8e\x76\x3f\x6b\xed\x2e\xaa\x75\xc6\x00\x25\x45\x7a\x2b\xa6\x16\x68\xad\x71\x34\xa4\x14\xa4\xb4\x6b\x00\x80\x21\x35\x16\x06\x52\x4a\xd0\xa0\x94\x83\xa3\x14\xda\x5f\xc7\x5b\x53\x20\xd0\x68\x4c\x29\x40\xbb\xf7\xf2\x0d\x05\x90\x08\x84\x70\x8d\xb6\x04\x58\x42\x60\x09\x30\x0b\x2c\x41\x42\x81\x21\x04\xa0\x50\x1a\x6c\x0d\x29\x07\x1c\x6f\x53\xee\xe5\x79\x40\xb8\x8b\xd9\x5a\xa3\xb4\x40\x09\x81\x81\x44\x39\x0e\xb6\xd6\xa4\x54\xe6\xbf\x52\x0a\x64\xd2\xa1\x6d\x7f\x23\x9d\xfb\x77\x23\x0d\x83\xd2\x31\xa7\x91\x57\x51\x85\xd2\x1a\x01\xd8\x52\xa7\x1d\xa3\xb5\xc0\xd2\x1a\xe9\x19\xa6\xbd\x1f\xa4\x67\xac\x6f\xb4\x29\x05\x96\x17\x31\x33\xd7\x14\x98\x5e\xb8\x1d\x2d\x48\xf9\x5e\x14\x9a\x14\xee\xe6\x53\x5a\xe3\x28\xf7\xb3\x1b\x7a\x8d\x92\x06\x4a\x69\x3a\x9b\xf7\xd3\xdd\xba\x8f\xd2\x93\xc7\x61\x44\x72\xb1\xfd\x50\x4b\x81\xd3\xdd\xcd\xc6\xc5\xcf\xb2\x6d\xe5\x9b\xc4\x7b\xba\x40\x08\x0a\x06\x55\x31\xe1\x7f\xee\xe2\xa4\x39\xf3\x31\x4c\x03\x8d\x70\x0d\xf5\x16\x57\xb8\xce\xd4\xf8\x19\xe0\xe6\x9e\xc8\x32\xc4\xf0\x22\x21\x05\x98\x01\x43\xa0\x80\xa0\x84\xa4\x01\x21\x05\x29\xc3\x4f\x03\x81\x2d\xdc\xfc\x51\x7e\x1d\x08\x70\x6c\x9b\x63\x3b\xbf\x61\xff\x9a\x15\xec\xfb\x64\x25\xc9\x58\x94\x99\xbf\x7a\x8b\x92\xda\x09\x38\xb6\x0d\x42\x60\x00\x0d\xcb\x5f\x65\xc3\x92\x05\x94\x0c\x1b\xc3\xf4\xbb\x1e\x05\x0d\x1b\x96\xbe\xc8\xda\x67\xee\xc3\xc8\x2d\x64\xd4\x8c\xcb\x49\x26\xe3\x24\xbb\x7a\x30\x4a\x06\x60\x48\x89\x56\x1a\x43\xb8\xc6\x64\x47\xde\x4f\x27\xed\xa5\xb2\xf2\x2e\x33\xdb\x32\x4b\x40\x40\x42\x48\x82\x2d\x05\x8e\xe1\xa6\x90\x2d\x40\x09\x09\x52\x90\x4a\x24\xf8\xfa\xd5\x27\xd9\xf1\xee\x62\x12\x3d\xc7\x09\x16\x14\x33\xf5\xa7\x2f\x10\x19\x7a\x32\xd1\x94\x8d\xd6\xae\x67\x54\x6f\x17\x4d\x9f\xfc\x15\x21\x0d\xa6\xdc\xf2\x33\x6a\x67\x5f\x89\x06\x02\xf9\x05\xfc\xf5\x9e\xef\xb2\x73\xd5\x32\x46\x9d\x3b\x87\xc3\x0d\x5f\xb1\xf6\x99\x1f\x33\xfa\xac\x0b\x98\x72\xcd\x4d\x94\x54\x0c\x46\x78\xdb\x55\x5e\xfa\xfa\x06\x08\xe1\x45\x44\x6a\x50\x6e\xcd\x48\x3f\xc7\xdd\xd4\x10\x58\x52\x10\xf4\xd0\x25\x6c\xb8\x9f\x43\x96\x49\xfc\x68\x2b\x47\x36\x7e\x8e\xa1\x1d\x4c\xd3\x62\xcc\xc5\xf3\x19\x38\x76\x22\x76\x2c\x8a\x91\x93\x8f\x0e\x46\x48\x39\x5e\xf1\x0b\x49\x2a\x99\x20\xd1\xd7\x45\xa4\x78\x00\x65\x63\x4e\x43\xdb\x29\x84\x63\x53\x50\x56\x89\x19\x0a\x11\xeb\x38\x82\x48\xc5\x29\x2c\x1f\x44\xb2\xbf\x8f\xcf\x16\xfd\x9a\x0f\x5f\xf8\x05\xf1\xa4\x4d\x02\x83\x98\x87\x40\x31\x47\x13\x57\x9a\x84\xd2\x1e\x52\xba\xe8\x14\xf3\x7e\x93\x49\xa5\xdd\x1c\xf7\x8c\x30\xbd\x28\x04\x0d\x17\xde\x22\x96\x81\xd3\xd5\xce\xc6\x57\x9e\x60\xf5\x7d\xf3\x39\xb6\xa5\x9e\xc9\x37\xff\x8c\xe9\xf7\x3e\x45\xdd\xd5\x37\xa3\x52\x09\x0e\x7c\xf2\x3e\xca\x4e\xa1\x85\x20\xd1\x7d\x9c\x86\x57\x9f\x24\x7a\xe4\x20\xe1\xe2\x81\x08\x21\x90\x86\x81\x14\x02\x53\x4a\xba\xf6\xef\x22\x15\x8b\x52\x54\x59\x4d\x4e\x4e\x0e\x3d\xcd\x4d\x44\x3b\xdb\x01\x48\xc6\x63\x44\xe3\x31\xf6\x37\x7c\x4d\x77\x5f\x94\xa8\x23\xd2\x46\xc4\x3c\x38\x8d\x65\x6d\x3e\x63\x80\x72\x91\x47\x7b\xa1\x33\x84\x5b\xe5\x41\x43\xd2\xfa\xd5\xe7\x2c\xbf\x7d\x1e\xdb\x3e\xf8\x13\xf1\xae\x0e\xd6\x2f\x7c\x14\xa7\xab\x9d\x80\xd0\x0c\x3f\x6b\x06\x15\x75\x93\x38\xb8\x6e\x0d\xb1\x23\x2d\x98\xa6\x85\x4e\xc5\xd9\xf7\xc1\x9b\x74\xef\xdd\x4e\xd9\xa9\x93\x88\x75\x75\xd0\xb9\x77\x07\x96\x65\x91\xec\xe9\xa4\x71\xd5\xdb\x48\xc3\x60\xec\xcc\xcb\x90\xa9\x04\x1b\xde\x5c\x88\x9d\x88\x23\xa5\x41\x61\xd5\x70\xda\x5b\xf6\xb3\xfc\xae\x6b\xf8\xe4\xa5\x5f\xd2\xd5\xd7\xe7\x1a\x61\x9f\x68\x44\xdc\x8f\x8a\x03\x66\x52\x71\x02\xf4\x79\xf0\x8d\x69\x48\x3a\x9a\x76\xf2\xee\x43\xb7\x81\x61\x70\xf9\x2f\x5f\xa6\xbf\xab\x93\x63\x07\x9a\x08\x48\xb7\x5Line truncated
},
"/rootDesc.xml.tmpl": &vfsgen۰CompressedFileInfo{
name: "rootDesc.xml.tmpl",
modTime: time.Date(2019, 9, 15, 16, 40, 10, 576918038, time.UTC),
uncompressedSize: 2521,
compressedContent: []byte("\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\xff\xcc\x56\xd1\x6a\xe3\x3a\x10\x7d\xef\x57\x84\x3c\xdd\x0b\x8d\x65\x97\x5e\x08\x46\x55\x29\x31\x17\x02\x75\xb7\x24\x9b\xa5\x6f\x41\x95\x27\x8e\x76\x6d\xc9\x48\x72\x92\x52\xfa\xef\x8b\x64\x3b\xb6\x93\x36\x5b\x36\xb0\x6c\x5e\x2a\xcd\x9c\x73\x34\x3a\xd2\xa8\xc6\xb7\xbb\x3c\x1b\x6c\x40\x69\x2e\xc5\xcd\x30\xf0\xfc\xe1\x2d\xb9\xc0\x4a\x4a\x33\xd8\xe5\x99\xd0\x37\xc3\x52\x89\x50\xb3\x35\xe4\x54\x8f\xca\x42\x14\x23\xa9\xd2\x30\x81\x0d\x67\x30\x0a\x46\xfe\xf0\x62\xe0\x7e\x0e\x1d\x26\x99\xa0\x7d\x8a\x8d\x9c\xa4\x68\x60\x37\xc3\xb5\x31\x45\x88\xd0\x76\xbb\xf5\x34\x30\x8f\x49\xef\x87\x42\x96\x3a\x24\x17\x83\x01\xd6\x05\xb0\x6f\x55\x91\xc4\x91\x71\x4e\xbf\x4b\x45\x02\x8c\xaa\x41\x1d\xe4\x42\x2a\xe2\x63\x54\x0d\x2c\x13\x1d\x50\x71\x55\x46\x4d\xa8\x26\x5f\x5f\x0a\x20\x27\xb6\x19\xc6\x90\x70\x3a\x07\xb5\x01\x15\x06\x18\x75\x58\x95\xcc\x4a\x71\x10\x49\xf6\xf2\x40\x73\x20\xaf\xaf\xde\xff\x9d\xf9\xdb\x1b\x46\xbd\x7c\x53\xbf\x28\x57\x94\x99\x52\x81\x22\x8a\x65\x52\xc0\xe0\x9f\xea\xaf\x27\x55\xfa\xaf\xdd\x58\x07\x71\x4c\x5a\xcc\xee\x89\x75\x4d\x87\x08\xb5\x3c\xd4\xe7\x59\x50\x4d\x95\x09\x64\x11\x68\xa6\x78\x61\xac\x17\x15\x07\xa3\xa3\x44\x07\xef\xea\xed\x01\xbb\x3b\x70\xf3\x32\x7f\x06\x65\xf7\x1c\xb7\x53\xbb\xe5\x6e\xb6\x83\xff\xb8\xea\x26\x5b\x81\x35\x28\x4e\x1b\xbe\x5f\xff\x30\xea\x85\x2b\xe4\x22\x7a\xb0\xcb\xcf\xa4\x34\x91\x3b\x97\xc5\x62\x1a\xd9\x0a\x6c\xa2\x3e\xe5\x4c\xd0\xf0\x69\x19\xdd\x3f\xdc\x4d\xee\x1e\xd1\x71\x34\xfa\x32\x21\x51\x3c\x1f\x05\xde\x7f\x3e\x46\x07\x89\x77\xd1\xf1\xe8\x17\x78\x0d\x2c\x7c\x54\x32\x29\x99\x99\xd0\x82\xe8\x9c\x5f\x46\x93\x38\xf0\x2f\x53\x30\xee\x3a\x4d\xc5\x4a\xda\xbb\x6e\x03\x13\xea\xac\x6f\x42\x76\x9f\x3d\x76\x2b\xf9\xb4\x3c\x4f\xb4\xc7\xaf\x64\x39\x93\xe2\x9e\x6b\x43\xea\xae\x74\x81\x66\xe2\x9a\x2a\x07\x63\xef\x3a\xcf\x69\x0a\xa8\x10\xa9\xed\xaf\x3a\xd6\xc2\xb6\x3c\x31\x6b\x72\x3d\xc6\xa8\x1a\xb5\x99\x35\xf0\x74\x6d\x5c\xaa\x1e\xb6\xb9\x04\x0a\xb3\x26\x63\xdb\x53\x45\x8f\x54\xaa\x8c\x20\x6d\xa8\xe1\xac\xbe\x28\xa3\xeb\xf1\xee\x7a\xec\xb9\xf5\x6d\xb6\x29\x17\x75\xeb\x3d\xab\xf8\xe0\xca\xff\xb0\x7a\x97\x3b\xab\xfc\xe0\xca\xdf\x05\x57\xfe\xc9\x0d\x54\xe3\xf6\x34\x6c\x1f\xd8\x3b\xdd\x3b\x9f\x3a\xd6\x59\xad\x8e\x7c\xfc\x8e\xd5\x80\x70\x22\x85\x01\x61\x22\xae\x80\x19\xa9\x5e\xec\x6b\xd6\x25\x1f\x49\x4e\x13\x27\x78\x28\x34\x4d\x8e\xa4\xf6\x42\xd3\xa4\x23\x33\x9f\x3c\x46\xb6\xab\x1b\x2f\x0e\x59\xde\x2e\xcf\x30\x6a\x50\x2d\x8f\x49\x61\x94\x74\x0f\x02\x62\x26\xc3\xa8\x13\x68\x51\xb0\x01\x61\xe6\xe5\xb3\x8d\x62\xd4\x9d\xed\xbd\x3d\x30\xeb\x4c\xf3\x04\x30\xdb\x53\x31\x15\x34\xad\xfe\x17\xfc\xbe\x7b\x7d\xad\x4f\xdb\xd7\xa7\xfd\x85\xfe\xe5\x9c\x29\xa9\xe5\xca\x78\x4c\xe6\x7b\xf3\x9e\x96\xf1\x7c\xe9\xde\xa8\x19\x30\xe0\x1b\x50\x33\x48\xb9\x36\x8a\x7e\xda\xc6\x77\x85\xa7\xc9\x29\xe9\xcf\xb9\x7a\x42\xe0\xcf\xfa\xbb\x9f\x76\x5e\x80\x42\x81\x06\x61\x0b\x95\xc2\x2d\x87\xd1\x61\xc8\x7d\xe7\x34\xdf\x35\x18\xd9\x8f\x37\xf2\x33\x00\x00\xff\xff\xb4\xae\xdd\x58\xd9\x09\x00\x00"),
},
}
fs["/"].(*vfsgen۰DirInfo).entries = []os.FileInfo{
fs["/ConnectionManager.xml"].(os.FileInfo),
@@ -61,6 +68,7 @@ var Assets = func() http.FileSystem {
fs["/X_MS_MediaReceiverRegistrar.xml"].(os.FileInfo),
fs["/rclone-120x120.png"].(os.FileInfo),
fs["/rclone-48x48.png"].(os.FileInfo),
fs["/rootDesc.xml.tmpl"].(os.FileInfo),
}
return fs
+32
View File
@@ -2,3 +2,35 @@
// The "go:generate" directive compiles static assets by running assets_generate.go
package data
import (
"io/ioutil"
"text/template"
"github.com/pkg/errors"
"github.com/rclone/rclone/fs"
)
// GetTemplate returns the rootDesc XML template
func GetTemplate() (tpl *template.Template, err error) {
templateFile, err := Assets.Open("rootDesc.xml.tmpl")
if err != nil {
return nil, errors.Wrap(err, "get template open")
}
defer fs.CheckClose(templateFile, &err)
templateBytes, err := ioutil.ReadAll(templateFile)
if err != nil {
return nil, errors.Wrap(err, "get template read")
}
var templateString = string(templateBytes)
tpl, err = template.New("rootDesc").Parse(templateString)
if err != nil {
return nil, errors.Wrap(err, "get template parse")
}
return
}
@@ -0,0 +1,66 @@
<?xml version="1.0"?>
<root xmlns="urn:schemas-upnp-org:device-1-0"
xmlns:dlna="urn:schemas-dlna-org:device-1-0"
xmlns:sec="http://www.sec.co.kr/dlna">
<specVersion>
<major>1</major>
<minor>0</minor>
</specVersion>
<device>
<deviceType>urn:schemas-upnp-org:device:MediaServer:1</deviceType>
<friendlyName>{{.FriendlyName}}</friendlyName>
<manufacturer>rclone (rclone.org)</manufacturer>
<manufacturerURL>https://rclone.org/</manufacturerURL>
<modelDescription>rclone</modelDescription>
<modelName>rclone</modelName>
<modelNumber>{{.ModelNumber}}</modelNumber>
<modelURL>https://rclone.org/</modelURL>
<serialNumber>00000000</serialNumber>
<UDN>{{.RootDeviceUUID}}</UDN>
<dlna:X_DLNACAP/>
<dlna:X_DLNADOC>DMS-1.50</dlna:X_DLNADOC>
<dlna:X_DLNADOC>M-DMS-1.50</dlna:X_DLNADOC>
<sec:ProductCap>smi,DCM10,getMediaInfo.sec,getCaptionInfo.sec</sec:ProductCap>
<sec:X_ProductCap>smi,DCM10,getMediaInfo.sec,getCaptionInfo.sec</sec:X_ProductCap>
<iconList>
<icon>
<mimetype>image/png</mimetype>
<width>48</width>
<height>48</height>
<depth>8</depth>
<url>/static/rclone-48x48.png</url>
</icon>
<icon>
<mimetype>image/png</mimetype>
<width>120</width>
<height>120</height>
<depth>8</depth>
<url>/static/rclone-120x120.png</url>
</icon>
</iconList>
<serviceList>
<service>
<serviceType>urn:schemas-upnp-org:service:ContentDirectory:1</serviceType>
<serviceId>urn:upnp-org:serviceId:ContentDirectory</serviceId>
<SCPDURL>/static/ContentDirectory.xml</SCPDURL>
<controlURL>/ctl</controlURL>
<eventSubURL></eventSubURL>
</service>
<service>
<serviceType>urn:schemas-upnp-org:service:ConnectionManager:1</serviceType>
<serviceId>urn:upnp-org:serviceId:ConnectionManager</serviceId>
<SCPDURL>/static/ConnectionManager.xml</SCPDURL>
<controlURL>/ctl</controlURL>
<eventSubURL></eventSubURL>
</service>
<service>
<serviceType>urn:microsoft.com:service:X_MS_MediaReceiverRegistrar:1</serviceType>
<serviceId>urn:microsoft.com:serviceId:X_MS_MediaReceiverRegistrar</serviceId>
<SCPDURL>/static/X_MS_MediaReceiverRegistrar.xml</SCPDURL>
<controlURL>/ctl</controlURL>
<eventSubURL></eventSubURL>
</service>
</serviceList>
<presentationURL>/</presentationURL>
</device>
</root>
+13 -78
View File
@@ -10,7 +10,6 @@ import (
"os"
"strconv"
"strings"
"text/template"
"time"
dms_dlna "github.com/anacrolix/dms/dlna"
@@ -116,6 +115,9 @@ func newServer(f fs.Fs, opt *dlnaflags.Options) *server {
"ConnectionManager": &connectionManagerService{
server: s,
},
"X_MS_MediaReceiverRegistrar": &mediaReceiverRegistrarService{
server: s,
},
}
// Setup the various http routes.
@@ -153,85 +155,18 @@ func (s *server) ModelNumber() string {
return fs.Version
}
// Template used to generate the root device XML descriptor.
//
// Due to the use of namespaces and various subtleties with device compatibility,
// it turns out to be easier to use a template than to marshal XML.
//
// For rendering, it is passed the server object for context.
var rootDescTmpl = template.Must(template.New("rootDesc").Parse(`<?xml version="1.0"?>
<root xmlns="urn:schemas-upnp-org:device-1-0"
xmlns:dlna="urn:schemas-dlna-org:device-1-0"
xmlns:sec="http://www.sec.co.kr/dlna">
<specVersion>
<major>1</major>
<minor>0</minor>
</specVersion>
<device>
<deviceType>urn:schemas-upnp-org:device:MediaServer:1</deviceType>
<friendlyName>{{.FriendlyName}}</friendlyName>
<manufacturer>rclone (rclone.org)</manufacturer>
<manufacturerURL>https://rclone.org/</manufacturerURL>
<modelDescription>rclone</modelDescription>
<modelName>rclone</modelName>
<modelNumber>{{.ModelNumber}}</modelNumber>
<modelURL>https://rclone.org/</modelURL>
<serialNumber>00000000</serialNumber>
<UDN>{{.RootDeviceUUID}}</UDN>
<dlna:X_DLNACAP/>
<dlna:X_DLNADOC>DMS-1.50</dlna:X_DLNADOC>
<dlna:X_DLNADOC>M-DMS-1.50</dlna:X_DLNADOC>
<sec:ProductCap>smi,DCM10,getMediaInfo.sec,getCaptionInfo.sec</sec:ProductCap>
<sec:X_ProductCap>smi,DCM10,getMediaInfo.sec,getCaptionInfo.sec</sec:X_ProductCap>
<iconList>
<icon>
<mimetype>image/png</mimetype>
<width>48</width>
<height>48</height>
<depth>8</depth>
<url>/static/rclone-48x48.png</url>
</icon>
<icon>
<mimetype>image/png</mimetype>
<width>120</width>
<height>120</height>
<depth>8</depth>
<url>/static/rclone-120x120.png</url>
</icon>
</iconList>
<serviceList>
<service>
<serviceType>urn:schemas-upnp-org:service:ContentDirectory:1</serviceType>
<serviceId>urn:upnp-org:serviceId:ContentDirectory</serviceId>
<SCPDURL>/static/ContentDirectory.xml</SCPDURL>
<controlURL>/ctl</controlURL>
<eventSubURL></eventSubURL>
</service>
<service>
<serviceType>urn:schemas-upnp-org:service:ConnectionManager:1</serviceType>
<serviceId>urn:upnp-org:serviceId:ConnectionManager</serviceId>
<SCPDURL>/static/ConnectionManager.xml</SCPDURL>
<controlURL>/ctl</controlURL>
<eventSubURL></eventSubURL>
</service>
<service>
<serviceType>urn:microsoft.com:service:X_MS_MediaReceiverRegistrar:1</serviceType>
<serviceId>urn:microsoft.com:serviceId:X_MS_MediaReceiverRegistrar</serviceId>
<SCPDURL>/static/X_MS_MediaReceiverRegistrar.xml</SCPDURL>
<controlURL>/ctl</controlURL>
<eventSubURL></eventSubURL>
</service>
</serviceList>
<presentationURL>/</presentationURL>
</device>
</root>`))
// Renders the root device descriptor.
func (s *server) rootDescHandler(w http.ResponseWriter, r *http.Request) {
buffer := new(bytes.Buffer)
err := rootDescTmpl.Execute(buffer, s)
tmpl, err := data.GetTemplate()
if err != nil {
serveError(s, w, "Failed to create root descriptor XML", err)
serveError(s, w, "Failed to load root descriptor template", err)
return
}
buffer := new(bytes.Buffer)
err = tmpl.Execute(buffer, s)
if err != nil {
serveError(s, w, "Failed to render root descriptor XML", err)
return
}
@@ -248,7 +183,7 @@ func (s *server) rootDescHandler(w http.ResponseWriter, r *http.Request) {
// Handle a service control HTTP request.
func (s *server) serviceControlHandler(w http.ResponseWriter, r *http.Request) {
soapActionString := r.Header.Get("SOAPACTION")
soapAction, err := upnp.ParseActionHTTPHeader(soapActionString)
soapAction, err := parseActionHTTPHeader(soapActionString)
if err != nil {
serveError(s, w, "Could not parse SOAPACTION header", err)
return
+55
View File
@@ -1,14 +1,19 @@
package dlna
import (
"bytes"
"context"
"fmt"
"html"
"io/ioutil"
"net/http"
"net/url"
"os"
"strings"
"testing"
"github.com/anacrolix/dms/soap"
"github.com/rclone/rclone/vfs"
_ "github.com/rclone/rclone/backend/local"
@@ -91,3 +96,53 @@ func TestServeContent(t *testing.T) {
require.Equal(t, goldenContents, actualContents)
}
// Check that ContentDirectory#Browse returns appropriate metadata on the root container.
func TestContentDirectoryBrowseMetadata(t *testing.T) {
// Sample from: https://github.com/rclone/rclone/issues/3253#issuecomment-524317469
req, err := http.NewRequest("POST", testURL+"ctl", strings.NewReader(`
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"
s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:Browse xmlns:u="urn:schemas-upnp-org:service:ContentDirectory:1">
<ObjectID>0</ObjectID>
<BrowseFlag>BrowseMetadata</BrowseFlag>
<Filter>*</Filter>
<StartingIndex>0</StartingIndex>
<RequestedCount>0</RequestedCount>
<SortCriteria></SortCriteria>
</u:Browse>
</s:Body>
</s:Envelope>`))
require.NoError(t, err)
req.Header.Set("SOAPACTION", `"urn:schemas-upnp-org:service:ContentDirectory:1#Browse"`)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
body, err := ioutil.ReadAll(resp.Body)
require.NoError(t, err)
// expect a <container> element
require.Contains(t, string(body), html.EscapeString("<container "))
require.NotContains(t, string(body), html.EscapeString("<item "))
// with a non-zero childCount
require.Contains(t, string(body), html.EscapeString(`childCount="1"`))
}
// Check that the X_MS_MediaReceiverRegistrar is faked out properly.
func TestMediaReceiverRegistrarService(t *testing.T) {
env := soap.Envelope{
Body: soap.Body{
Action: []byte("RegisterDevice"),
},
}
req, err := http.NewRequest("POST", testURL+"ctl", bytes.NewReader(mustMarshalXML(env)))
require.NoError(t, err)
req.Header.Set("SOAPACTION", `"urn:microsoft.com:service:X_MS_MediaReceiverRegistrar:1#RegisterDevice"`)
resp, err := http.DefaultClient.Do(req)
require.NoError(t, err)
assert.Equal(t, http.StatusOK, resp.StatusCode)
body, err := ioutil.ReadAll(resp.Body)
require.NoError(t, err)
require.Contains(t, string(body), "<RegistrationRespMsg>")
}
+34
View File
@@ -3,6 +3,7 @@ package dlna
import (
"crypto/md5"
"encoding/xml"
"errors"
"fmt"
"io"
"log"
@@ -11,6 +12,9 @@ import (
"net/http/httptest"
"net/http/httputil"
"os"
"regexp"
"strconv"
"strings"
"github.com/anacrolix/dms/soap"
"github.com/anacrolix/dms/upnp"
@@ -85,6 +89,36 @@ func marshalSOAPResponse(sa upnp.SoapAction, args map[string]string) []byte {
sa.Action, sa.ServiceURN.String(), mustMarshalXML(soapArgs)))
}
var serviceURNRegexp = regexp.MustCompile(`:service:(\w+):(\d+)$`)
func parseServiceType(s string) (ret upnp.ServiceURN, err error) {
matches := serviceURNRegexp.FindStringSubmatch(s)
if matches == nil {
err = errors.New(s)
return
}
if len(matches) != 3 {
log.Panicf("Invalid serviceURNRegexp ?")
}
ret.Type = matches[1]
ret.Version, err = strconv.ParseUint(matches[2], 0, 0)
return
}
func parseActionHTTPHeader(s string) (ret upnp.SoapAction, err error) {
if s[0] != '"' || s[len(s)-1] != '"' {
return
}
s = s[1 : len(s)-1]
hashIndex := strings.LastIndex(s, "#")
if hashIndex == -1 {
return
}
ret.Action = s[hashIndex+1:]
ret.ServiceURN, err = parseServiceType(s[:hashIndex])
return
}
type loggingResponseWriter struct {
http.ResponseWriter
request *http.Request
+27
View File
@@ -0,0 +1,27 @@
package dlna
import (
"net/http"
"github.com/anacrolix/dms/upnp"
)
type mediaReceiverRegistrarService struct {
*server
upnp.Eventing
}
func (mrrs *mediaReceiverRegistrarService) Handle(action string, argsXML []byte, r *http.Request) (map[string]string, error) {
switch action {
case "IsAuthorized", "IsValidated":
return map[string]string{
"Result": "1",
}, nil
case "RegisterDevice":
return map[string]string{
"RegistrationRespMsg": mrrs.RootDeviceUUID,
}, nil
default:
return nil, upnp.InvalidActionError
}
}
-3
View File
@@ -1,7 +1,4 @@
// Package restic serves a remote suitable for use with restic
// +build go1.9
package restic
import (
@@ -1,5 +1,3 @@
// +build go1.9
package restic
import (
@@ -1,5 +1,3 @@
// +build go1.9
package restic
import (
-2
View File
@@ -1,8 +1,6 @@
// Serve restic tests set up a server and run the integration tests
// for restic against it.
// +build go1.9
package restic
import (
-11
View File
@@ -1,11 +0,0 @@
// Build for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build !go1.9
package restic
import "github.com/spf13/cobra"
// Command definition is nil to show not implemented
var Command *cobra.Command = nil
-2
View File
@@ -1,5 +1,3 @@
// +build go1.9
package restic
import (
+21 -2
View File
@@ -1,10 +1,9 @@
// +build go1.9
package restic
import (
"net"
"os"
"time"
)
// Addr implements net.Addr for stdin/stdout.
@@ -52,3 +51,23 @@ func (s *StdioConn) LocalAddr() net.Addr {
func (s *StdioConn) RemoteAddr() net.Addr {
return Addr{}
}
// SetDeadline sets the read/write deadline.
func (s *StdioConn) SetDeadline(t time.Time) error {
err1 := s.stdin.SetReadDeadline(t)
err2 := s.stdout.SetWriteDeadline(t)
if err1 != nil {
return err1
}
return err2
}
// SetReadDeadline sets the read/write deadline.
func (s *StdioConn) SetReadDeadline(t time.Time) error {
return s.stdin.SetReadDeadline(t)
}
// SetWriteDeadline sets the read/write deadline.
func (s *StdioConn) SetWriteDeadline(t time.Time) error {
return s.stdout.SetWriteDeadline(t)
}
-27
View File
@@ -1,27 +0,0 @@
//+build go1.10
// Deadline setting for go1.10+
package restic
import "time"
// SetDeadline sets the read/write deadline.
func (s *StdioConn) SetDeadline(t time.Time) error {
err1 := s.stdin.SetReadDeadline(t)
err2 := s.stdout.SetWriteDeadline(t)
if err1 != nil {
return err1
}
return err2
}
// SetReadDeadline sets the read/write deadline.
func (s *StdioConn) SetReadDeadline(t time.Time) error {
return s.stdin.SetReadDeadline(t)
}
// SetWriteDeadline sets the read/write deadline.
func (s *StdioConn) SetWriteDeadline(t time.Time) error {
return s.stdout.SetWriteDeadline(t)
}
-22
View File
@@ -1,22 +0,0 @@
//+build go1.9,!go1.10
// Fallback deadline setting for pre go1.10
package restic
import "time"
// SetDeadline sets the read/write deadline.
func (s *StdioConn) SetDeadline(t time.Time) error {
return nil
}
// SetReadDeadline sets the read/write deadline.
func (s *StdioConn) SetReadDeadline(t time.Time) error {
return nil
}
// SetWriteDeadline sets the read/write deadline.
func (s *StdioConn) SetWriteDeadline(t time.Time) error {
return nil
}
+1 -2
View File
@@ -1,5 +1,4 @@
//+build go1.9
// Package webdav implements a WebDAV server backed by rclone VFS
package webdav
import (
+1 -1
View File
@@ -3,7 +3,7 @@
//
// We skip tests on platforms with troublesome character mappings
//+build !windows,!darwin,go1.9
//+build !windows,!darwin
package webdav
-11
View File
@@ -1,11 +0,0 @@
// Build for webdav for unsupported platforms to stop go complaining
// about "no buildable Go source files "
// +build !go1.9
package webdav
import "github.com/spf13/cobra"
// Command definition is nil to show not implemented
var Command *cobra.Command = nil
@@ -0,0 +1,30 @@
rclone-dlna-server:
container_name: rclone-dlna-server
image: rclone/rclone
command:
# Tweak here rclone's command line switches:
# - "--config"
# - "/path/to/mounted/rclone.conf"
- "--verbose"
- "serve"
- "dlna"
- "remote:/"
- "--name"
- "myDLNA server"
- "--read-only"
# - "--no-modtime"
# - "--no-checksum"
restart: unless-stopped
# Use host networking for simplicity with DLNA broadcasts
# and to avoid having to do port mapping.
net: host
# Here you have to map your host's rclone.conf directory to
# container's /root/.config/rclone/ dir (R/O).
# If you have any remote referencing local files, you have to
# map them here, too.
volumes:
- ~/.config/rclone/:/root/.config/rclone/:ro
@@ -0,0 +1,35 @@
rclone-webdav-server:
container_name: rclone-webdav-server
image: rclone/rclone
command:
# Tweak here rclone's command line switches:
# - "--config"
# - "/path/to/mounted/rclone.conf"
- "--verbose"
- "serve"
- "webdav"
- "remote:/"
# - "--addr"
# - "0.0.0.0:8080"
- "--read-only"
# - "--no-modtime"
# - "--no-checksum"
restart: unless-stopped
# Use host networking for simplicity.
# It also enables server's default listen on 127.0.0.1 to work safely.
net: host
# If you want to use port mapping instead of host networking,
# make sure to make rclone listen on 0.0.0.0.
#ports:
# - "127.0.0.1:8080:8080"
# Here you have to map your host's rclone.conf directory to
# container's /root/.config/rclone/ dir (R/O).
# If you have any remote referencing local files, you have to
# map them here, too.
volumes:
- ~/.config/rclone/:/root/.config/rclone/:ro
+1
View File
@@ -30,6 +30,7 @@ Rclone is a command line program to sync files and directories to and from:
* {{< provider name="Jottacloud" home="https://www.jottacloud.com/en/" config="/jottacloud/" >}}
* {{< provider name="IBM COS S3" home="http://www.ibm.com/cloud/object-storage" config="/s3/#ibm-cos-s3" >}}
* {{< provider name="Koofr" home="https://koofr.eu/" config="/koofr/" >}}
* {{< provider name="Mail.ru Cloud" home="https://cloud.mail.ru/" config="/mailru/" >}}
* {{< provider name="Memset Memstore" home="https://www.memset.com/cloud/storage/" config="/swift/" >}}
* {{< provider name="Mega" home="https://mega.nz/" config="/mega/" >}}
* {{< provider name="Microsoft Azure Blob Storage" home="https://azure.microsoft.com/en-us/services/storage/blobs/" config="/azureblob/" >}}
+7
View File
@@ -284,3 +284,10 @@ Contributors
* Patrick Wang <mail6543210@yahoo.com.tw>
* Cenk Alti <cenkalti@gmail.com>
* Andreas Chlupka <andy@chlupka.com>
* Alfonso Montero <amontero@tinet.org>
* Ivan Andreev <ivandeex@gmail.com>
* David Baumgold <david@davidbaumgold.com>
* Lars Lehtonen <lars.lehtonen@gmail.com>
* Matei David <matei.david@gmail.com>
* David <david.bramwell@endemolshine.com>
* Anthony Rusdi <33247310+antrusd@users.noreply.github.com>
+10 -2
View File
@@ -12,7 +12,8 @@ Paths are specified as `remote:path`
Paths may be as deep as required, eg `remote:directory/subdirectory`.
The initial setup for Box involves getting a token from Box which you
need to do in your browser. `rclone config` walks you through it.
can do either in your browser, or with a config.json downloaded from Box
to use JWT authentication. `rclone config` walks you through it.
Here is an example of how to make a remote called `remote`. First run:
@@ -37,7 +38,14 @@ Storage> box
Box App Client Id - leave blank normally.
client_id>
Box App Client Secret - leave blank normally.
client_secret>
client_secret>
Box App config.json location
Leave blank normally.
Enter a string value. Press Enter for the default ("").
config_json>
'enterprise' or 'user' depending on the type of token being requested.
Enter a string value. Press Enter for the default ("user").
box_sub_type>
Remote config
Use auto config?
* Say Y if not sure
+1 -1
View File
@@ -35,7 +35,7 @@ costs more. It may do in future (probably with a flag).
## Bugs
Bugs are stored in rclone's Github project:
Bugs are stored in rclone's GitHub project:
* [Reported bugs](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug)
* [Known issues](https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Known+Problem%22)
+32 -1
View File
@@ -1,11 +1,42 @@
---
title: "Documentation"
description: "Rclone Changelog"
date: "2019-08-26"
date: "2019-09-15"
---
# Changelog
## v1.49.3 - 2019-09-15
* Bug Fixes
* accounting
* Fix total duration calculation (Aleksandar Jankovic)
* Fix "file already closed" on transfer retries (Nick Craig-Wood)
## v1.49.2 - 2019-09-08
* New Features
* build: Add Docker workflow support (Alfonso Montero)
* Bug Fixes
* accounting: Fix locking in Transfer to avoid deadlock with --progress (Nick Craig-Wood)
* docs: Fix template argument for mktemp in install.sh (Cnly)
* operations: Fix -u/--update with google photos / files of unknown size (Nick Craig-Wood)
* rc: Fix docs for config/create /update /password (Nick Craig-Wood)
* Google Cloud Storage
* Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
## v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
* Bug Fixes
* config: Fix generated passwords being stored as empty password (Nick Craig-Wood)
* rcd: Added missing parameter for web-gui info logs. (Chaitanya)
* Googlephotos
* Fix crash on error response (Nick Craig-Wood)
* Onedrive
* Fix crash on error response (Nick Craig-Wood)
## v1.49.0 - 2019-08-26
* New backends
+2 -1
View File
@@ -22,7 +22,8 @@ rclone copyurl https://example.com dest:path [flags]
### Options
```
-h, --help help for copyurl
-a, --auto-filename Get the file name from the url and use it for destination file path
-h, --help help for copyurl
```
See the [global flags page](/flags/) for global options not listed here.
+37 -1
View File
@@ -287,11 +287,46 @@ This mode should support all normal file system operations.
If an upload or download fails it will be retried up to
--low-level-retries times.
```
rclone mount remote:path /path/to/mountpoint [flags]
```
### Case Sensitivity
Linux file systems are case-sensitive: two files can differ only
by case, and the exact case must be used when opening a file.
Windows is not like most other operating systems supported by rclone.
File systems in modern Windows are case-insensitive but case-preserving:
although existing files can be opened using any case, the exact case used
to create the file is preserved and available for programs to query.
It is not allowed for two files in the same directory to differ only by case.
Usually file systems on MacOS are case-insensitive. It is possible to make MacOS
file systems case-sensitive but that is not the default
The `--vfs-case-insensitive` mount flag controls how rclone handles these
two cases. If its value is `false`, rclone passes file names to the mounted
file system as is. If the flag is `true` (or appears without a value on
command line), rclone may perform a "fixup" as explained below.
The user may specify a file name to open/delete/rename/etc with a case
different than what is stored on mounted file system. If an argument refers
to an existing file with exactly the same name, then the case of the existing
file on the disk will be used. However, if a file name with exactly the same
name is not found but a name differing only by case exists, rclone will
transparently fixup the name. This fixup happens only when an existing file
is requested. Case sensitivity of file names created anew by rclone is
controlled by an underlying mounted file system.
Note that case sensitivity of the operating system running rclone (the target)
may differ from case sensitivity of a file system mounted by rclone (the source).
The flag controls whether "fixup" is performed to satisfy the target.
If the flag is not provided on command line, then its default value depends
on the operating system where rclone runs: `true` on Windows and MacOS, `false`
otherwise. If the flag is provided without a value, then it is `true`.
### Options
```
@@ -322,6 +357,7 @@ rclone mount remote:path /path/to/mountpoint [flags]
--vfs-cache-max-size SizeSuffix Max total size of objects in the cache. (default off)
--vfs-cache-mode CacheMode Cache mode off|minimal|writes|full (default off)
--vfs-cache-poll-interval duration Interval to poll the cache for stale objects. (default 1m0s)
--vfs-case-insensitive [bool] Case insensitive mount true|false (default depends on operating system)
--vfs-read-chunk-size SizeSuffix Read the source objects in chunks. (default 128M)
--vfs-read-chunk-size-limit SizeSuffix If greater than --vfs-read-chunk-size, double the chunk size after each chunk read, until the limit is reached. 'off' is unlimited. (default off)
--volname string Set the volume name (not supported by all OSes).
+1
View File
@@ -37,6 +37,7 @@ See the following for detailed instructions for
* [Hubic](/hubic/)
* [Jottacloud](/jottacloud/)
* [Koofr](/koofr/)
* [Mail.ru Cloud](/mailru/)
* [Mega](/mega/)
* [Microsoft Azure Blob Storage](/azureblob/)
* [Microsoft OneDrive](/onedrive/)
Loaded 100 of 598 files, more files were not shown because too many files have changed in this diff. Show more