Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
25bf29db20 | ||
|
|
1680e76ff5 | ||
|
|
962cf6d53b | ||
|
|
fde3a4e2ee | ||
|
|
1f3770a57f | ||
|
|
829b06d39b | ||
|
|
0db3e7a2a0 | ||
|
|
654df91375 | ||
|
|
f079361a01 | ||
|
|
3677b7ff77 | ||
|
|
4a00a4dc4b | ||
|
|
2610beb18d | ||
|
|
35e4f60548 | ||
|
|
839a3b92e0 | ||
|
|
492e0bd616 | ||
|
|
d4894db765 | ||
|
|
8d09f16681 | ||
|
|
550889c096 | ||
|
|
896961404b | ||
|
|
5eabfc74f4 | ||
|
|
787d033376 | ||
|
|
db258d404a | ||
|
|
3cdf680a15 | ||
|
|
bc57110920 | ||
|
|
133eae4d5b | ||
|
|
24c40463eb | ||
|
|
e9fddaabeb | ||
|
|
c49015552c | ||
|
|
ffdd043b95 | ||
|
|
434edba275 | ||
|
|
da7a3d8d64 | ||
|
|
2362b4ca50 | ||
|
|
5e3d783b3e | ||
|
|
7a63990df2 | ||
|
|
7ca667d35d | ||
|
|
1e22af8358 | ||
|
|
589d45a0bf | ||
|
|
d5deec3534 | ||
|
|
53c19e04ba | ||
|
|
e7fadc7e3d | ||
|
|
7dfc5b3076 | ||
|
|
523a29a4e9 | ||
|
|
a3e1312d9d | ||
|
|
e987d4f351 | ||
|
|
639bd88959 | ||
|
|
bfa2fbb78e | ||
|
|
69ccbacf30 | ||
|
|
5d6690eb20 | ||
|
|
c7b8654b17 | ||
|
|
906b27d8b9 | ||
|
|
e233f8b0a7 | ||
|
|
17a547a980 | ||
|
|
cdb3ecb0fb | ||
|
|
2d7f84576a | ||
|
|
ab234dcfd7 | ||
|
|
c5bcfbac8f | ||
|
|
88ed8e2b43 | ||
|
|
b097454f83 | ||
|
|
2f890114b6 | ||
|
|
78a7d9bb8e | ||
|
|
72c561d209 | ||
|
|
b864c4f9c9 | ||
|
|
bb78eb8ab2 |
No files matched your search
@@ -283,7 +283,7 @@ jobs:
|
||||
run: govulncheck ./...
|
||||
|
||||
- name: Check Markdown format
|
||||
uses: DavidAnson/markdownlint-cli2-action@v20
|
||||
uses: DavidAnson/markdownlint-cli2-action@v23
|
||||
with:
|
||||
globs: |
|
||||
CONTRIBUTING.md
|
||||
|
||||
@@ -100,7 +100,7 @@ jobs:
|
||||
|
||||
- name: Extract Metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
uses: docker/metadata-action@v6
|
||||
env:
|
||||
DOCKER_METADATA_ANNOTATIONS_LEVELS: manifest,manifest-descriptor # Important for digest annotation (used by Github packages)
|
||||
with:
|
||||
@@ -122,10 +122,10 @@ jobs:
|
||||
type=raw,value=beta,enable={{is_default_branch}}
|
||||
|
||||
- name: Setup QEMU
|
||||
uses: docker/setup-qemu-action@v3
|
||||
uses: docker/setup-qemu-action@v4
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
uses: docker/setup-buildx-action@v4
|
||||
|
||||
- name: Load Go Build Cache for Docker
|
||||
id: go-cache
|
||||
@@ -148,7 +148,7 @@ jobs:
|
||||
skip-extraction: ${{ steps.go-cache.outputs.cache-hit }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: ghcr.io
|
||||
# This is the user that triggered the Workflow. In this case, it will
|
||||
@@ -159,7 +159,7 @@ jobs:
|
||||
|
||||
- name: Build and Publish Image Digest
|
||||
id: build
|
||||
uses: docker/build-push-action@v6
|
||||
uses: docker/build-push-action@v7
|
||||
with:
|
||||
file: Dockerfile
|
||||
context: .
|
||||
@@ -210,7 +210,7 @@ jobs:
|
||||
|
||||
- name: Extract Metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
uses: docker/metadata-action@v6
|
||||
env:
|
||||
DOCKER_METADATA_ANNOTATIONS_LEVELS: index
|
||||
with:
|
||||
@@ -261,16 +261,16 @@ jobs:
|
||||
env.write(f"ANNOTATIONS={annotations_string}\n")
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
uses: docker/setup-buildx-action@v4
|
||||
|
||||
- name: Login to Docker Hub
|
||||
uses: docker/login-action@v3
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- name: Login to GitHub Container Registry
|
||||
uses: docker/login-action@v3
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: ghcr.io
|
||||
# This is the user that triggered the Workflow. In this case, it will
|
||||
|
||||
+7
-7
@@ -412,8 +412,8 @@ the source file in the `Help:` field:
|
||||
- The `backenddocs` make target runs the Python script `bin/make_backend_docs.py`,
|
||||
and you can also run this directly, optionally with the name of a backend
|
||||
as argument to only update the docs for a specific backend.
|
||||
- **Do not** commit the updated Markdown files. This operation is run as part of
|
||||
the release process. Since any manual changes in the autogenerated sections
|
||||
- **Do not** commit the updated Markdown files. This operation is run as part
|
||||
of the release process. Since any manual changes in the autogenerated sections
|
||||
of the Markdown files will then be lost, we have a pull request check that
|
||||
reports error for any changes within the autogenerated sections. Should you
|
||||
have done manual changes outside of the autogenerated sections they must be
|
||||
@@ -580,7 +580,8 @@ remote or an fs.
|
||||
make sure we can encode any path name and `rclone info` to help determine the
|
||||
encodings needed
|
||||
- `rclone purge -v TestRemote:rclone-info`
|
||||
- `rclone test info --all --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info`
|
||||
- `rclone test info --all --remote-encoding None -vv --write-json remote.json
|
||||
TestRemote:rclone-info`
|
||||
- `go run cmd/test/info/internal/build_csv/main.go -o remote.csv remote.json`
|
||||
- open `remote.csv` in a spreadsheet and examine
|
||||
|
||||
@@ -627,10 +628,9 @@ See the [testing](#testing) section for more information on integration tests.
|
||||
|
||||
### Backend documentation
|
||||
|
||||
Add your backend to the docs - you'll need to pick an icon for it from
|
||||
[fontawesome](http://fontawesome.io/icons/). Keep lists of remotes in
|
||||
alphabetical order of full name of remote (e.g. `drive` is ordered as
|
||||
`Google Drive`) but with the local file system last.
|
||||
Add your backend to the docs. Keep lists of remotes in alphabetical
|
||||
order of full name of remote (e.g. `drive` is ordered as `Google
|
||||
Drive`) but with the local file system last.
|
||||
|
||||
First add a data file about your backend in
|
||||
`docs/data/backends/remote.yaml` - this is used to build the overview
|
||||
|
||||
+20
-20
@@ -2,27 +2,27 @@
|
||||
|
||||
Current active maintainers of rclone are:
|
||||
|
||||
| Name | GitHub ID | Specific Responsibilities |
|
||||
| :--------------- | :---------------- | :-------------------------- |
|
||||
| Nick Craig-Wood | @ncw | overall project health |
|
||||
| Stefan Breunig | @breunigs | |
|
||||
| Ishuah Kariuki | @ishuah | |
|
||||
| Remus Bunduc | @remusb | cache backend |
|
||||
| Fabian Möller | @B4dM4n | |
|
||||
| Alex Chen | @Cnly | onedrive backend |
|
||||
| Sandeep Ummadi | @sandeepkru | azureblob backend |
|
||||
| Name | GitHub ID | Specific Responsibilities |
|
||||
| :--------------- | :---------------- | :------------------------------------- |
|
||||
| Nick Craig-Wood | @ncw | overall project health |
|
||||
| Stefan Breunig | @breunigs | |
|
||||
| Ishuah Kariuki | @ishuah | |
|
||||
| Remus Bunduc | @remusb | cache backend |
|
||||
| Fabian Möller | @B4dM4n | |
|
||||
| Alex Chen | @Cnly | onedrive backend |
|
||||
| Sandeep Ummadi | @sandeepkru | azureblob backend |
|
||||
| Sebastian Bünger | @buengese | jottacloud, yandex & compress backends |
|
||||
| Ivan Andreev | @ivandeex | chunker & mailru backends |
|
||||
| Max Sum | @Max-Sum | union backend |
|
||||
| Fred | @creativeprojects | seafile backend |
|
||||
| Caleb Case | @calebcase | storj backend |
|
||||
| wiserain | @wiserain | pikpak backend |
|
||||
| albertony | @albertony | |
|
||||
| Chun-Hung Tseng | @henrybear327 | Proton Drive Backend |
|
||||
| Hideo Aoyama | @boukendesho | snap packaging |
|
||||
| nielash | @nielash | bisync |
|
||||
| Dan McArdle | @dmcardle | gitannex |
|
||||
| Sam Harrison | @childish-sambino | filescom |
|
||||
| Ivan Andreev | @ivandeex | chunker & mailru backends |
|
||||
| Max Sum | @Max-Sum | union backend |
|
||||
| Fred | @creativeprojects | seafile backend |
|
||||
| Caleb Case | @calebcase | storj backend |
|
||||
| wiserain | @wiserain | pikpak backend |
|
||||
| albertony | @albertony | |
|
||||
| Chun-Hung Tseng | @henrybear327 | Proton Drive Backend |
|
||||
| Hideo Aoyama | @boukendesho | snap packaging |
|
||||
| nielash | @nielash | bisync |
|
||||
| Dan McArdle | @dmcardle | gitannex |
|
||||
| Sam Harrison | @childish-sambino | filescom |
|
||||
|
||||
## This is a work in progress draft
|
||||
|
||||
|
||||
@@ -146,6 +146,7 @@ MANUAL.txt: MANUAL.md
|
||||
|
||||
commanddocs: rclone
|
||||
go generate ./lib/transform
|
||||
go generate ./cmd/bisync
|
||||
-@rmdir -p '$$HOME/.config/rclone'
|
||||
XDG_CACHE_HOME="" XDG_CONFIG_HOME="" HOME="\$$HOME" USER="\$$USER" rclone gendocs --config=/notfound docs/content/
|
||||
@[ ! -e '$$HOME' ] || (echo 'Error: created unwanted directory named $$HOME' && exit 1)
|
||||
|
||||
@@ -123,6 +123,7 @@ directories to and from different cloud storage providers.
|
||||
- Synology C2 Object Storage [:page_facing_up:](https://rclone.org/s3/#synology-c2)
|
||||
- Tencent Cloud Object Storage (COS) [:page_facing_up:](https://rclone.org/s3/#tencent-cos)
|
||||
- Uloz.to [:page_facing_up:](https://rclone.org/ulozto/)
|
||||
- US3 Object Storage [:page_facing_up:](https://rclone.org/s3/#us3)
|
||||
- Wasabi [:page_facing_up:](https://rclone.org/s3/#wasabi)
|
||||
- WebDAV [:page_facing_up:](https://rclone.org/webdav/)
|
||||
- Yandex Disk [:page_facing_up:](https://rclone.org/yandex/)
|
||||
|
||||
+2
-1
@@ -32,10 +32,11 @@ This file describes how to make the various kinds of releases
|
||||
- make sign_upload
|
||||
- make check_sign
|
||||
- make upload
|
||||
- make upload_test_website # check sponsors are correct/unchanged
|
||||
- make upload_website
|
||||
- make upload_github
|
||||
- make startdev # make startstable for stable branch
|
||||
- \# announce with forum post, twitter post, patreon post
|
||||
- \# announce with forum post, twitter post, and post to annoucements list
|
||||
|
||||
## Update dependencies
|
||||
|
||||
|
||||
@@ -118,12 +118,19 @@ func (c *Client) Authenticate(ctx context.Context) error {
|
||||
|
||||
fs.Debugf("icloud", "Authenticating as %s\n", c.appleID)
|
||||
err := c.Session.SignIn(ctx, c.appleID, c.password)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err == nil {
|
||||
err = c.Session.AuthWithToken(ctx)
|
||||
if err == nil && c.sessionSaveCallback != nil {
|
||||
c.sessionSaveCallback(c.Session)
|
||||
}
|
||||
// If 2FA is required, don't try AuthWithToken yet — the caller
|
||||
// must complete 2FA first, then call AuthWithToken.
|
||||
if c.Session.Requires2FA() {
|
||||
return nil
|
||||
}
|
||||
|
||||
err = c.Session.AuthWithToken(ctx)
|
||||
if err == nil && c.sessionSaveCallback != nil {
|
||||
c.sessionSaveCallback(c.Session)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/http"
|
||||
@@ -9,11 +10,15 @@ import (
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/oracle/oci-go-sdk/v65/common"
|
||||
"github.com/rclone/rclone/fs"
|
||||
"github.com/rclone/rclone/fs/fshttp"
|
||||
"github.com/rclone/rclone/lib/rest"
|
||||
)
|
||||
|
||||
const iCloudUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3.1 Safari/605.1.15"
|
||||
|
||||
// Session represents an iCloud session
|
||||
type Session struct {
|
||||
SessionToken string `json:"session_token"`
|
||||
@@ -22,10 +27,22 @@ type Session struct {
|
||||
AccountCountry string `json:"account_country"`
|
||||
TrustToken string `json:"trust_token"`
|
||||
ClientID string `json:"client_id"`
|
||||
AuthAttributes string `json:"auth_attributes"`
|
||||
FrameID string `json:"frame_id"`
|
||||
Cookies []*http.Cookie `json:"cookies"`
|
||||
AccountInfo AccountInfo `json:"account_info"`
|
||||
|
||||
srv *rest.Client `json:"-"`
|
||||
srv *rest.Client `json:"-"`
|
||||
needs2FA bool `json:"-"` // set when SRP signin returns 409
|
||||
}
|
||||
|
||||
// srpInitResponse is the server response from /auth/signin/init
|
||||
type srpInitResponse struct {
|
||||
Iteration int `json:"iteration"`
|
||||
Salt string `json:"salt"`
|
||||
Protocol string `json:"protocol"`
|
||||
B string `json:"b"`
|
||||
C string `json:"c"`
|
||||
}
|
||||
|
||||
// String returns the session as a string
|
||||
@@ -57,24 +74,222 @@ func (s *Session) Request(ctx context.Context, opts rest.Opts, request any, resp
|
||||
if val := resp.Header.Get("scnt"); val != "" {
|
||||
s.Scnt = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
|
||||
s.AuthAttributes = val
|
||||
}
|
||||
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// Requires2FA returns true if the session requires 2FA
|
||||
func (s *Session) Requires2FA() bool {
|
||||
return s.AccountInfo.DsInfo.HsaVersion == 2 && s.AccountInfo.HsaChallengeRequired
|
||||
if s.needs2FA {
|
||||
return true
|
||||
}
|
||||
return s.AccountInfo.DsInfo != nil && s.AccountInfo.DsInfo.HsaVersion == 2 && s.AccountInfo.HsaChallengeRequired
|
||||
}
|
||||
|
||||
// SignIn signs in the session
|
||||
// SignIn performs SRP-based authentication against Apple's idmsa endpoint.
|
||||
func (s *Session) SignIn(ctx context.Context, appleID, password string) error {
|
||||
// Apple's SRP implementation expects a lowercase account name.
|
||||
// The old plaintext flow didn't need this because the server normalized
|
||||
// it, but SRP uses the username in client-side proof computation (M1).
|
||||
appleID = strings.ToLower(appleID)
|
||||
|
||||
// Step 1: Initialize the auth session
|
||||
if err := s.authStart(ctx); err != nil {
|
||||
return fmt.Errorf("authStart: %w", err)
|
||||
}
|
||||
|
||||
// Step 2: Federate (submit account name)
|
||||
if err := s.authFederate(ctx, appleID); err != nil {
|
||||
return fmt.Errorf("authFederate: %w", err)
|
||||
}
|
||||
|
||||
// Step 3: SRP init — send client public value A, get salt + B
|
||||
client := newSRPClient()
|
||||
aBase64 := base64.StdEncoding.EncodeToString(client.getABytes())
|
||||
|
||||
initResp, err := s.authSRPInit(ctx, aBase64, appleID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("authSRPInit: %w", err)
|
||||
}
|
||||
|
||||
// Decode server values
|
||||
serverB, err := base64.StdEncoding.DecodeString(initResp.B)
|
||||
if err != nil {
|
||||
return fmt.Errorf("decode B: %w", err)
|
||||
}
|
||||
salt, err := base64.StdEncoding.DecodeString(initResp.Salt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("decode salt: %w", err)
|
||||
}
|
||||
|
||||
// Step 4: Derive password key and process the SRP challenge
|
||||
derivedKey, err := derivePassword(password, salt, initResp.Iteration, initResp.Protocol)
|
||||
if err != nil {
|
||||
return fmt.Errorf("derivePassword: %w", err)
|
||||
}
|
||||
client.processChallenge([]byte(appleID), derivedKey, salt, serverB)
|
||||
|
||||
// Step 5: Complete — send M1, M2 proofs
|
||||
m1Base64 := base64.StdEncoding.EncodeToString(client.M1)
|
||||
m2Base64 := base64.StdEncoding.EncodeToString(client.M2)
|
||||
|
||||
if err := s.authSRPComplete(ctx, appleID, m1Base64, m2Base64, initResp.C); err != nil {
|
||||
return fmt.Errorf("authSRPComplete: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// authStart initializes the SRP auth session by hitting the authorize/signin endpoint.
|
||||
func (s *Session) authStart(ctx context.Context) error {
|
||||
if s.FrameID == "" {
|
||||
s.FrameID = strings.ToLower(uuid.New().String())
|
||||
}
|
||||
frameTag := "auth-" + s.FrameID
|
||||
|
||||
params := url.Values{}
|
||||
params.Set("frame_id", frameTag)
|
||||
params.Set("language", "en_US")
|
||||
params.Set("skVersion", "7")
|
||||
params.Set("iframeId", frameTag)
|
||||
params.Set("client_id", s.ClientID)
|
||||
params.Set("redirect_uri", "https://www.icloud.com")
|
||||
params.Set("response_type", "code")
|
||||
params.Set("response_mode", "web_message")
|
||||
params.Set("state", frameTag)
|
||||
params.Set("authVersion", "latest")
|
||||
|
||||
opts := rest.Opts{
|
||||
Method: "GET",
|
||||
Path: "/authorize/signin",
|
||||
Parameters: params,
|
||||
ExtraHeaders: map[string]string{
|
||||
"Accept": "*/*",
|
||||
"User-Agent": iCloudUserAgent,
|
||||
},
|
||||
RootURL: authEndpoint,
|
||||
NoResponse: true,
|
||||
}
|
||||
|
||||
resp, err := s.srv.Call(ctx, &opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("authStart: unexpected status %s", resp.Status)
|
||||
}
|
||||
|
||||
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
|
||||
s.AuthAttributes = val
|
||||
}
|
||||
if val := resp.Header.Get("scnt"); val != "" {
|
||||
s.Scnt = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
|
||||
s.SessionID = val
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// authFederate submits the account name to Apple's federate endpoint.
|
||||
func (s *Session) authFederate(ctx context.Context, accountName string) error {
|
||||
values := map[string]any{
|
||||
"accountName": accountName,
|
||||
"rememberMe": true,
|
||||
}
|
||||
body, err := IntoReader(values)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/federate",
|
||||
Parameters: url.Values{"isRememberMeEnabled": {"true"}},
|
||||
ExtraHeaders: s.getSRPAuthHeaders(),
|
||||
RootURL: authEndpoint,
|
||||
Body: body,
|
||||
NoResponse: true,
|
||||
}
|
||||
|
||||
resp, err := s.srv.Call(ctx, &opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
|
||||
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
|
||||
s.AuthAttributes = val
|
||||
}
|
||||
if val := resp.Header.Get("scnt"); val != "" {
|
||||
s.Scnt = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
|
||||
s.SessionID = val
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("authFederate: unexpected status %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// authSRPInit sends the client's public value A to the server and retrieves
|
||||
// the salt, server public value B, iteration count, protocol, and challenge.
|
||||
func (s *Session) authSRPInit(ctx context.Context, aBase64, accountName string) (*srpInitResponse, error) {
|
||||
values := map[string]any{
|
||||
"a": aBase64,
|
||||
"accountName": accountName,
|
||||
"protocols": []string{"s2k", "s2k_fo"},
|
||||
}
|
||||
body, err := IntoReader(values)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/signin/init",
|
||||
ExtraHeaders: s.getSRPAuthHeaders(),
|
||||
RootURL: authEndpoint,
|
||||
Body: body,
|
||||
}
|
||||
|
||||
var initResp srpInitResponse
|
||||
resp, err := s.srv.CallJSON(ctx, &opts, nil, &initResp)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if val := resp.Header.Get("scnt"); val != "" {
|
||||
s.Scnt = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
|
||||
s.SessionID = val
|
||||
}
|
||||
|
||||
return &initResp, nil
|
||||
}
|
||||
|
||||
// authSRPComplete sends the SRP proofs M1 and M2 to complete authentication.
|
||||
// Returns nil on success (200 or 409/2FA needed).
|
||||
func (s *Session) authSRPComplete(ctx context.Context, accountName, m1Base64, m2Base64, c string) error {
|
||||
trustTokens := []string{}
|
||||
if s.TrustToken != "" {
|
||||
trustTokens = []string{s.TrustToken}
|
||||
}
|
||||
|
||||
values := map[string]any{
|
||||
"accountName": appleID,
|
||||
"password": password,
|
||||
"accountName": accountName,
|
||||
"m1": m1Base64,
|
||||
"m2": m2Base64,
|
||||
"c": c,
|
||||
"rememberMe": true,
|
||||
"trustTokens": trustTokens,
|
||||
}
|
||||
@@ -82,21 +297,100 @@ func (s *Session) SignIn(ctx context.Context, appleID, password string) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/signin",
|
||||
Parameters: url.Values{},
|
||||
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
|
||||
Path: "/signin/complete",
|
||||
Parameters: url.Values{"isRememberMeEnabled": {"true"}},
|
||||
ExtraHeaders: s.getSRPAuthHeaders(),
|
||||
RootURL: authEndpoint,
|
||||
IgnoreStatus: true, // need to handle 409 for hsa2
|
||||
IgnoreStatus: true,
|
||||
NoResponse: true,
|
||||
Body: body,
|
||||
}
|
||||
opts.Parameters.Set("isRememberMeEnabled", "true")
|
||||
_, err = s.Request(ctx, opts, nil, nil)
|
||||
|
||||
return err
|
||||
resp, err := s.srv.Call(ctx, &opts)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_ = resp.Body.Close()
|
||||
|
||||
// Extract updated headers
|
||||
if val := resp.Header.Get("X-Apple-Auth-Attributes"); val != "" {
|
||||
s.AuthAttributes = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-Session-Token"); val != "" {
|
||||
s.SessionToken = val
|
||||
}
|
||||
if val := resp.Header.Get("scnt"); val != "" {
|
||||
s.Scnt = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
|
||||
s.SessionID = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-ID-Account-Country"); val != "" {
|
||||
s.AccountCountry = val
|
||||
}
|
||||
if val := resp.Header.Get("X-Apple-TwoSV-Trust-Token"); val != "" {
|
||||
s.TrustToken = val
|
||||
}
|
||||
|
||||
switch resp.StatusCode {
|
||||
case http.StatusOK:
|
||||
fs.Debugf("icloud", "SRP sign in successful")
|
||||
return nil
|
||||
case http.StatusConflict:
|
||||
// 409 = 2FA required, this is expected
|
||||
fs.Debugf("icloud", "SRP sign in requires 2FA")
|
||||
s.needs2FA = true
|
||||
return nil
|
||||
case http.StatusForbidden:
|
||||
return fmt.Errorf("sign in failed: incorrect username or password")
|
||||
default:
|
||||
return fmt.Errorf("sign in failed: %s", resp.Status)
|
||||
}
|
||||
}
|
||||
|
||||
// getAuthOrigin returns the origin URL for auth requests.
|
||||
// Supports both global (idmsa.apple.com) and China (idmsa.apple.com.cn) endpoints.
|
||||
func getAuthOrigin() string {
|
||||
return strings.TrimSuffix(authEndpoint, "/appleauth/auth")
|
||||
}
|
||||
|
||||
// getSRPAuthHeaders returns headers needed for SRP auth requests.
|
||||
func (s *Session) getSRPAuthHeaders() map[string]string {
|
||||
frameTag := "auth-" + s.FrameID
|
||||
authOrigin := getAuthOrigin()
|
||||
headers := map[string]string{
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
"User-Agent": iCloudUserAgent,
|
||||
"Origin": authOrigin,
|
||||
"Referer": authOrigin + "/",
|
||||
"X-Apple-Widget-Key": s.ClientID,
|
||||
"X-Apple-OAuth-Client-Id": s.ClientID,
|
||||
"X-Apple-OAuth-Client-Type": "firstPartyAuth",
|
||||
"X-Apple-OAuth-Redirect-URI": "https://www.icloud.com",
|
||||
"X-Apple-OAuth-Require-Grant-Code": "true",
|
||||
"X-Apple-OAuth-Response-Mode": "web_message",
|
||||
"X-Apple-OAuth-Response-Type": "code",
|
||||
"X-Apple-OAuth-State": frameTag,
|
||||
"X-Apple-Frame-Id": frameTag,
|
||||
"X-Requested-With": "XMLHttpRequest",
|
||||
"X-Apple-Mandate-Security-Upgrade": "0",
|
||||
"X-Apple-I-Require-UE": "true",
|
||||
"X-Apple-I-FD-Client-Info": `{"U":"` + iCloudUserAgent + `","L":"en-US","Z":"GMT-05:00","V":"1.1","F":""}`,
|
||||
}
|
||||
if s.AuthAttributes != "" {
|
||||
headers["X-Apple-Auth-Attributes"] = s.AuthAttributes
|
||||
}
|
||||
if s.Scnt != "" {
|
||||
headers["scnt"] = s.Scnt
|
||||
}
|
||||
if s.SessionID != "" {
|
||||
headers["X-Apple-ID-Session-Id"] = s.SessionID
|
||||
}
|
||||
return headers
|
||||
}
|
||||
|
||||
// AuthWithToken authenticates the session
|
||||
@@ -135,14 +429,10 @@ func (s *Session) Validate2FACode(ctx context.Context, code string) error {
|
||||
return err
|
||||
}
|
||||
|
||||
headers := s.GetAuthHeaders(map[string]string{})
|
||||
headers["scnt"] = s.Scnt
|
||||
headers["X-Apple-ID-Session-Id"] = s.SessionID
|
||||
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/verify/trusteddevice/securitycode",
|
||||
ExtraHeaders: headers,
|
||||
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
|
||||
RootURL: authEndpoint,
|
||||
Body: body,
|
||||
NoResponse: true,
|
||||
@@ -162,14 +452,10 @@ func (s *Session) Validate2FACode(ctx context.Context, code string) error {
|
||||
|
||||
// TrustSession trusts the session
|
||||
func (s *Session) TrustSession(ctx context.Context) error {
|
||||
headers := s.GetAuthHeaders(map[string]string{})
|
||||
headers["scnt"] = s.Scnt
|
||||
headers["X-Apple-ID-Session-Id"] = s.SessionID
|
||||
|
||||
opts := rest.Opts{
|
||||
Method: "GET",
|
||||
Path: "/2sv/trust",
|
||||
ExtraHeaders: headers,
|
||||
ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
|
||||
RootURL: authEndpoint,
|
||||
NoResponse: true,
|
||||
ContentLength: common.Int64(0),
|
||||
@@ -201,25 +487,9 @@ func (s *Session) ValidateSession(ctx context.Context) error {
|
||||
}
|
||||
|
||||
// GetAuthHeaders returns the authentication headers for the session.
|
||||
//
|
||||
// It takes an `overwrite` map[string]string parameter which allows
|
||||
// overwriting the default headers. It returns a map[string]string.
|
||||
// Used for 2FA validation and trust requests to idmsa.apple.com.
|
||||
func (s *Session) GetAuthHeaders(overwrite map[string]string) map[string]string {
|
||||
headers := map[string]string{
|
||||
"Accept": "application/json",
|
||||
"Content-Type": "application/json",
|
||||
"X-Apple-OAuth-Client-Id": s.ClientID,
|
||||
"X-Apple-OAuth-Client-Type": "firstPartyAuth",
|
||||
"X-Apple-OAuth-Redirect-URI": "https://www.icloud.com",
|
||||
"X-Apple-OAuth-Require-Grant-Code": "true",
|
||||
"X-Apple-OAuth-Response-Mode": "web_message",
|
||||
"X-Apple-OAuth-Response-Type": "code",
|
||||
"X-Apple-OAuth-State": s.ClientID,
|
||||
"X-Apple-Widget-Key": s.ClientID,
|
||||
"Origin": homeEndpoint,
|
||||
"Referer": fmt.Sprintf("%s/", homeEndpoint),
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0",
|
||||
}
|
||||
headers := s.getSRPAuthHeaders()
|
||||
maps.Copy(headers, overwrite)
|
||||
return headers
|
||||
}
|
||||
@@ -248,7 +518,7 @@ func GetCommonHeaders(overwrite map[string]string) map[string]string {
|
||||
"Content-Type": "application/json",
|
||||
"Origin": baseEndpoint,
|
||||
"Referer": fmt.Sprintf("%s/", baseEndpoint),
|
||||
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0",
|
||||
"User-Agent": iCloudUserAgent,
|
||||
}
|
||||
maps.Copy(headers, overwrite)
|
||||
return headers
|
||||
@@ -281,9 +551,16 @@ func GetCookiesForDomain(url *url.URL, cookies []*http.Cookie) ([]*http.Cookie,
|
||||
|
||||
// NewSession creates a new Session instance with default values.
|
||||
func NewSession() *Session {
|
||||
session := &Session{}
|
||||
session.srv = rest.NewClient(fshttp.NewClient(context.Background())).SetRoot(baseEndpoint)
|
||||
//session.ClientID = "auth-" + uuid.New().String()
|
||||
session := &Session{
|
||||
FrameID: strings.ToLower(uuid.New().String()),
|
||||
}
|
||||
httpClient := fshttp.NewClient(context.Background())
|
||||
if tr, ok := httpClient.Transport.(*fshttp.Transport); ok {
|
||||
tr.SetRequestFilter(func(req *http.Request) {
|
||||
req.Header.Set("User-Agent", iCloudUserAgent)
|
||||
})
|
||||
}
|
||||
session.srv = rest.NewClient(httpClient).SetRoot(baseEndpoint)
|
||||
return session
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"crypto/pbkdf2"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"hash"
|
||||
"math/big"
|
||||
)
|
||||
|
||||
// RFC 5054 2048-bit group parameters for SRP-6a (g=2, SHA-256).
|
||||
var (
|
||||
srpN = mustParseBigHex(
|
||||
"AC6BDB41324A9A9BF166DE5E1389582FAF72B6651987EE07FC3192943DB56050" +
|
||||
"A37329CBB4A099ED8193E0757767A13DD52312AB4B03310DCD7F48A9DA04FD50" +
|
||||
"E8083969EDB767B0CF6095179A163AB3661A05FBD5FAAAE82918A9962F0B93B8" +
|
||||
"55F97993EC975EEAA80D740ADBF4FF747359D041D5C33EA71D281E446B14773B" +
|
||||
"CA97B43A23FB801676BD207A436C6481F1D2B9078717461A5B9D32E688F87748" +
|
||||
"544523B524B0D57D5EA77A2775D2ECFA032CFBDBF52FB37861602790" +
|
||||
"04E57AE6AF874E7303CE53299CCC041C7BC308D82A5698F3A8D0C38271AE35F8" +
|
||||
"E9DBFBB694B5C803D89F7AE435DE236D525F54759B65E372FCD68EF20FA7111F" +
|
||||
"9E4AFF73")
|
||||
srpG = big.NewInt(2)
|
||||
srpNLenBytes = 2048 / 8
|
||||
srpHashFunc = sha256.New
|
||||
)
|
||||
|
||||
func mustParseBigHex(s string) *big.Int {
|
||||
b, ok := new(big.Int).SetString(s, 16)
|
||||
if !ok {
|
||||
panic("srp: bad hex constant for N")
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
// srpClient implements the client side of SRP-6a for Apple's iCloud auth.
|
||||
type srpClient struct {
|
||||
a *big.Int // client secret
|
||||
A *big.Int // client public value
|
||||
k *big.Int // multiplier
|
||||
M1 []byte
|
||||
M2 []byte
|
||||
K []byte
|
||||
}
|
||||
|
||||
// newSRPClient generates a random 32-byte secret and computes the public value A.
|
||||
func newSRPClient() *srpClient {
|
||||
secret := make([]byte, 32)
|
||||
if _, err := rand.Read(secret); err != nil {
|
||||
panic(fmt.Sprintf("srp: rand.Read failed: %v", err))
|
||||
}
|
||||
|
||||
a := new(big.Int).SetBytes(secret)
|
||||
A := new(big.Int).Exp(srpG, a, srpN)
|
||||
k := getMultiplier()
|
||||
|
||||
return &srpClient{
|
||||
a: a,
|
||||
A: A,
|
||||
k: k,
|
||||
}
|
||||
}
|
||||
|
||||
// getABytes returns the padded public value A.
|
||||
func (c *srpClient) getABytes() []byte {
|
||||
return padToN(c.A)
|
||||
}
|
||||
|
||||
// processChallenge computes the session key and proof values from the server challenge.
|
||||
// username is the Apple ID, derivedKey is the output of derivePassword, salt and B come
|
||||
// from the server's init response (raw bytes, already base64-decoded).
|
||||
func (c *srpClient) processChallenge(username, derivedKey, salt, serverB []byte) {
|
||||
B := new(big.Int).SetBytes(serverB)
|
||||
|
||||
// Validate B
|
||||
if B.Cmp(big.NewInt(0)) <= 0 || B.Cmp(srpN) >= 0 {
|
||||
panic("srp: invalid server-supplied B, must be 1..N-1")
|
||||
}
|
||||
|
||||
x := calculateX(salt, derivedKey)
|
||||
u := calculateU(c.A, B)
|
||||
S := calculateS(c.k, x, c.a, B, u)
|
||||
c.K = calculateK(S)
|
||||
|
||||
aBytes := padToN(c.A)
|
||||
bBytes := padToN(B)
|
||||
c.M1 = calculateM1(username, salt, aBytes, bBytes, c.K)
|
||||
c.M2 = calculateM2(aBytes, c.M1, c.K)
|
||||
}
|
||||
|
||||
// derivePassword performs Apple's password key derivation.
|
||||
// For "s2k": PBKDF2(SHA256(password), salt, iterations, 32, SHA256)
|
||||
// For "s2k_fo": PBKDF2(hex(SHA256(password)), salt, iterations, 32, SHA256)
|
||||
func derivePassword(password string, salt []byte, iterations int, protocol string) ([]byte, error) {
|
||||
passHash := sha256.Sum256([]byte(password))
|
||||
|
||||
var passInput string
|
||||
switch protocol {
|
||||
case "s2k_fo":
|
||||
passInput = hex.EncodeToString(passHash[:])
|
||||
default: // "s2k"
|
||||
passInput = string(passHash[:])
|
||||
}
|
||||
return pbkdf2.Key(sha256.New, passInput, salt, iterations, 32)
|
||||
}
|
||||
|
||||
// padToN pads a big.Int to the SRP group size (256 bytes for 2048-bit).
|
||||
func padToN(n *big.Int) []byte {
|
||||
b := n.Bytes()
|
||||
if len(b) >= srpNLenBytes {
|
||||
return b
|
||||
}
|
||||
padded := make([]byte, srpNLenBytes)
|
||||
copy(padded[srpNLenBytes-len(b):], b)
|
||||
return padded
|
||||
}
|
||||
|
||||
func srpHash(data []byte) []byte {
|
||||
h := srpHashFunc()
|
||||
h.Write(data)
|
||||
return h.Sum(nil)
|
||||
}
|
||||
|
||||
func hashToInt(h hash.Hash) *big.Int {
|
||||
return new(big.Int).SetBytes(h.Sum(nil))
|
||||
}
|
||||
|
||||
// getMultiplier computes k = H(N | pad(g))
|
||||
func getMultiplier() *big.Int {
|
||||
h := srpHashFunc()
|
||||
nBytes := srpN.Bytes()
|
||||
gBytes := srpG.Bytes()
|
||||
// pad g to same length as N
|
||||
for len(gBytes) < len(nBytes) {
|
||||
gBytes = append([]byte{0}, gBytes...)
|
||||
}
|
||||
h.Write(nBytes)
|
||||
h.Write(gBytes)
|
||||
return hashToInt(h)
|
||||
}
|
||||
|
||||
// calculateX computes x = H(salt | H(":" | password))
|
||||
// Apple's variant: NoUserNameInX — username is omitted from the inner hash.
|
||||
// The "password" here is actually the derived key from derivePassword.
|
||||
func calculateX(salt, derivedKey []byte) *big.Int {
|
||||
h := srpHashFunc()
|
||||
// Apple omits username (NoUserNameInX), only writes ":" + derivedKey
|
||||
h.Write([]byte(":"))
|
||||
h.Write(derivedKey)
|
||||
digest := h.Sum(nil)
|
||||
|
||||
h2 := srpHashFunc()
|
||||
h2.Write(salt)
|
||||
h2.Write(digest)
|
||||
return new(big.Int).SetBytes(h2.Sum(nil))
|
||||
}
|
||||
|
||||
// calculateU computes u = H(pad(A) | pad(B))
|
||||
func calculateU(A, B *big.Int) *big.Int {
|
||||
h := srpHashFunc()
|
||||
h.Write(padToN(A))
|
||||
h.Write(padToN(B))
|
||||
return hashToInt(h)
|
||||
}
|
||||
|
||||
// calculateS computes the client-side shared secret:
|
||||
// S = (B - k * g^x) ^ (a + u*x) mod N
|
||||
func calculateS(k, x, a, B, u *big.Int) []byte {
|
||||
// g^x mod N
|
||||
gx := new(big.Int).Exp(srpG, x, srpN)
|
||||
|
||||
// k * g^x
|
||||
kgx := new(big.Int).Mul(k, gx)
|
||||
|
||||
// B - k*g^x
|
||||
diff := new(big.Int).Sub(B, kgx)
|
||||
|
||||
// u*x
|
||||
ux := new(big.Int).Mul(u, x)
|
||||
|
||||
// a + u*x
|
||||
exp := new(big.Int).Add(a, ux)
|
||||
|
||||
// (B - k*g^x) ^ (a + u*x) mod N
|
||||
S := new(big.Int).Exp(diff, exp, srpN)
|
||||
|
||||
// Ensure positive
|
||||
S.Mod(S, srpN)
|
||||
|
||||
return padToN(S)
|
||||
}
|
||||
|
||||
// calculateK computes K = H(S)
|
||||
func calculateK(S []byte) []byte {
|
||||
return srpHash(S)
|
||||
}
|
||||
|
||||
// calculateM1 computes the client proof (Apple/iCloud variant):
|
||||
// M1 = H(H(g) XOR H(N) | H(username) | salt | A | B | K)
|
||||
func calculateM1(username, salt, A, B, K []byte) []byte {
|
||||
hg := srpHash(padToN(srpG))
|
||||
hn := srpHash(srpN.Bytes())
|
||||
hi := srpHash(username)
|
||||
|
||||
hxor := make([]byte, len(hg))
|
||||
for i := range hg {
|
||||
hxor[i] = hg[i] ^ hn[i]
|
||||
}
|
||||
|
||||
h := srpHashFunc()
|
||||
h.Write(hxor)
|
||||
h.Write(hi)
|
||||
h.Write(salt)
|
||||
h.Write(A)
|
||||
h.Write(B)
|
||||
h.Write(K)
|
||||
return h.Sum(nil)
|
||||
}
|
||||
|
||||
// calculateM2 computes the server proof: M2 = H(A | M1 | K)
|
||||
func calculateM2(A, M1, K []byte) []byte {
|
||||
h := srpHashFunc()
|
||||
h.Write(A)
|
||||
h.Write(M1)
|
||||
h.Write(K)
|
||||
return h.Sum(nil)
|
||||
}
|
||||
@@ -0,0 +1,379 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"math/big"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestPadToN(t *testing.T) {
|
||||
t.Run("pads small value to 256 bytes", func(t *testing.T) {
|
||||
result := padToN(big.NewInt(1))
|
||||
assert.Equal(t, srpNLenBytes, len(result))
|
||||
assert.Equal(t, byte(0), result[0])
|
||||
assert.Equal(t, byte(1), result[srpNLenBytes-1])
|
||||
})
|
||||
|
||||
t.Run("pads zero to 256 bytes", func(t *testing.T) {
|
||||
result := padToN(big.NewInt(0))
|
||||
assert.Equal(t, srpNLenBytes, len(result))
|
||||
for _, b := range result {
|
||||
assert.Equal(t, byte(0), b)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("preserves value that fills N", func(t *testing.T) {
|
||||
// Use N itself — already 256 bytes
|
||||
result := padToN(srpN)
|
||||
assert.Equal(t, srpNLenBytes, len(result))
|
||||
assert.Equal(t, srpN.Bytes(), result)
|
||||
})
|
||||
|
||||
t.Run("round-trips through big.Int", func(t *testing.T) {
|
||||
original := big.NewInt(123456789)
|
||||
padded := padToN(original)
|
||||
recovered := new(big.Int).SetBytes(padded)
|
||||
assert.Equal(t, 0, original.Cmp(recovered))
|
||||
})
|
||||
}
|
||||
|
||||
func TestGetMultiplier(t *testing.T) {
|
||||
k := getMultiplier()
|
||||
|
||||
// k must be non-zero and less than N
|
||||
assert.True(t, k.Sign() > 0, "multiplier must be positive")
|
||||
assert.True(t, k.Cmp(srpN) < 0, "multiplier must be less than N")
|
||||
|
||||
// k is deterministic
|
||||
k2 := getMultiplier()
|
||||
assert.Equal(t, 0, k.Cmp(k2), "multiplier must be deterministic")
|
||||
|
||||
// k = H(N | pad(g)) — verify by manual computation
|
||||
h := srpHashFunc()
|
||||
nBytes := srpN.Bytes()
|
||||
gBytes := srpG.Bytes()
|
||||
for len(gBytes) < len(nBytes) {
|
||||
gBytes = append([]byte{0}, gBytes...)
|
||||
}
|
||||
h.Write(nBytes)
|
||||
h.Write(gBytes)
|
||||
expected := new(big.Int).SetBytes(h.Sum(nil))
|
||||
assert.Equal(t, 0, expected.Cmp(k), "multiplier must equal H(N|pad(g))")
|
||||
}
|
||||
|
||||
func TestDerivePassword(t *testing.T) {
|
||||
salt := []byte("test_salt_value!")
|
||||
password := "test_password"
|
||||
|
||||
t.Run("s2k produces 32-byte key", func(t *testing.T) {
|
||||
key, err := derivePassword(password, salt, 1000, "s2k")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 32, len(key))
|
||||
})
|
||||
|
||||
t.Run("s2k_fo produces 32-byte key", func(t *testing.T) {
|
||||
key, err := derivePassword(password, salt, 1000, "s2k_fo")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 32, len(key))
|
||||
})
|
||||
|
||||
t.Run("s2k and s2k_fo produce different keys", func(t *testing.T) {
|
||||
key1, err := derivePassword(password, salt, 1000, "s2k")
|
||||
require.NoError(t, err)
|
||||
key2, err := derivePassword(password, salt, 1000, "s2k_fo")
|
||||
require.NoError(t, err)
|
||||
assert.NotEqual(t, key1, key2, "s2k and s2k_fo must produce different keys")
|
||||
})
|
||||
|
||||
t.Run("deterministic output", func(t *testing.T) {
|
||||
key1, err := derivePassword(password, salt, 1000, "s2k")
|
||||
require.NoError(t, err)
|
||||
key2, err := derivePassword(password, salt, 1000, "s2k")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, key1, key2)
|
||||
})
|
||||
|
||||
t.Run("different passwords produce different keys", func(t *testing.T) {
|
||||
key1, err := derivePassword("password1", salt, 1000, "s2k")
|
||||
require.NoError(t, err)
|
||||
key2, err := derivePassword("password2", salt, 1000, "s2k")
|
||||
require.NoError(t, err)
|
||||
assert.NotEqual(t, key1, key2)
|
||||
})
|
||||
|
||||
t.Run("s2k uses raw SHA256 bytes", func(t *testing.T) {
|
||||
// Verify that s2k uses SHA256(password) as the PBKDF2 input.
|
||||
// Two passwords with the same SHA256 hash would produce the same key,
|
||||
// while two passwords with different SHA256 hashes must produce different keys.
|
||||
key1, err := derivePassword("hello", salt, 10, "s2k")
|
||||
require.NoError(t, err)
|
||||
key2, err := derivePassword("world", salt, 10, "s2k")
|
||||
require.NoError(t, err)
|
||||
assert.NotEqual(t, key1, key2)
|
||||
})
|
||||
|
||||
t.Run("s2k_fo uses hex-encoded SHA256", func(t *testing.T) {
|
||||
// For s2k_fo, the PBKDF2 input is hex(SHA256(password)), which is
|
||||
// a 64-character ASCII string, not raw 32-byte hash.
|
||||
key, err := derivePassword("hello", salt, 10, "s2k_fo")
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 32, len(key))
|
||||
})
|
||||
}
|
||||
|
||||
func TestCalculateX(t *testing.T) {
|
||||
salt := []byte("some_salt")
|
||||
derivedKey := []byte("some_derived_key_value_32_bytes!")
|
||||
|
||||
x := calculateX(salt, derivedKey)
|
||||
|
||||
// x must be positive and less than the hash output space
|
||||
assert.True(t, x.Sign() > 0, "x must be positive")
|
||||
|
||||
// x is deterministic
|
||||
x2 := calculateX(salt, derivedKey)
|
||||
assert.Equal(t, 0, x.Cmp(x2), "x must be deterministic")
|
||||
|
||||
// Verify manually: x = H(salt | H(":" | derivedKey))
|
||||
h1 := srpHashFunc()
|
||||
h1.Write([]byte(":"))
|
||||
h1.Write(derivedKey)
|
||||
inner := h1.Sum(nil)
|
||||
|
||||
h2 := srpHashFunc()
|
||||
h2.Write(salt)
|
||||
h2.Write(inner)
|
||||
expected := new(big.Int).SetBytes(h2.Sum(nil))
|
||||
assert.Equal(t, 0, expected.Cmp(x), "x must equal H(salt | H(\":\" | derivedKey))")
|
||||
}
|
||||
|
||||
func TestCalculateU(t *testing.T) {
|
||||
A := big.NewInt(12345)
|
||||
B := big.NewInt(67890)
|
||||
|
||||
u := calculateU(A, B)
|
||||
assert.True(t, u.Sign() > 0, "u must be positive")
|
||||
|
||||
// u is deterministic
|
||||
u2 := calculateU(A, B)
|
||||
assert.Equal(t, 0, u.Cmp(u2))
|
||||
|
||||
// u changes when inputs change
|
||||
u3 := calculateU(A, big.NewInt(99999))
|
||||
assert.NotEqual(t, 0, u.Cmp(u3), "u must change when B changes")
|
||||
|
||||
// Verify: u = H(pad(A) | pad(B))
|
||||
h := srpHashFunc()
|
||||
h.Write(padToN(A))
|
||||
h.Write(padToN(B))
|
||||
expected := new(big.Int).SetBytes(h.Sum(nil))
|
||||
assert.Equal(t, 0, expected.Cmp(u))
|
||||
}
|
||||
|
||||
func TestCalculateK(t *testing.T) {
|
||||
S := []byte("some_shared_secret_value_32bytes")
|
||||
|
||||
K := calculateK(S)
|
||||
|
||||
// K is a SHA-256 hash, so 32 bytes
|
||||
assert.Equal(t, sha256.Size, len(K))
|
||||
|
||||
// K is deterministic
|
||||
K2 := calculateK(S)
|
||||
assert.Equal(t, K, K2)
|
||||
|
||||
// Verify: K = H(S)
|
||||
expected := srpHash(S)
|
||||
assert.Equal(t, expected, K)
|
||||
}
|
||||
|
||||
func TestCalculateM1(t *testing.T) {
|
||||
username := []byte("test@example.com")
|
||||
salt := []byte("test_salt")
|
||||
A := make([]byte, srpNLenBytes)
|
||||
B := make([]byte, srpNLenBytes)
|
||||
K := make([]byte, 32)
|
||||
|
||||
// Fill with non-zero values
|
||||
A[srpNLenBytes-1] = 0x42
|
||||
B[srpNLenBytes-1] = 0x43
|
||||
K[0] = 0x44
|
||||
|
||||
m1 := calculateM1(username, salt, A, B, K)
|
||||
|
||||
// M1 is a SHA-256 hash
|
||||
assert.Equal(t, sha256.Size, len(m1))
|
||||
|
||||
// M1 is deterministic
|
||||
m1b := calculateM1(username, salt, A, B, K)
|
||||
assert.Equal(t, m1, m1b)
|
||||
|
||||
// M1 changes when username changes
|
||||
m1c := calculateM1([]byte("other@example.com"), salt, A, B, K)
|
||||
assert.NotEqual(t, m1, m1c)
|
||||
|
||||
// Verify: M1 = H(H(g) XOR H(N) | H(username) | salt | A | B | K)
|
||||
hg := srpHash(padToN(srpG))
|
||||
hn := srpHash(srpN.Bytes())
|
||||
hi := srpHash(username)
|
||||
hxor := make([]byte, len(hg))
|
||||
for i := range hg {
|
||||
hxor[i] = hg[i] ^ hn[i]
|
||||
}
|
||||
h := srpHashFunc()
|
||||
h.Write(hxor)
|
||||
h.Write(hi)
|
||||
h.Write(salt)
|
||||
h.Write(A)
|
||||
h.Write(B)
|
||||
h.Write(K)
|
||||
expected := h.Sum(nil)
|
||||
assert.Equal(t, expected, m1)
|
||||
}
|
||||
|
||||
func TestCalculateM2(t *testing.T) {
|
||||
A := make([]byte, srpNLenBytes)
|
||||
M1 := make([]byte, 32)
|
||||
K := make([]byte, 32)
|
||||
|
||||
A[srpNLenBytes-1] = 0x01
|
||||
M1[0] = 0x02
|
||||
K[0] = 0x03
|
||||
|
||||
m2 := calculateM2(A, M1, K)
|
||||
assert.Equal(t, sha256.Size, len(m2))
|
||||
|
||||
// Verify: M2 = H(A | M1 | K)
|
||||
h := srpHashFunc()
|
||||
h.Write(A)
|
||||
h.Write(M1)
|
||||
h.Write(K)
|
||||
assert.Equal(t, h.Sum(nil), m2)
|
||||
}
|
||||
|
||||
func TestNewSRPClient(t *testing.T) {
|
||||
client := newSRPClient()
|
||||
|
||||
// A must be non-zero and in range [1, N-1]
|
||||
assert.True(t, client.A.Sign() > 0, "A must be positive")
|
||||
assert.True(t, client.A.Cmp(srpN) < 0, "A must be less than N")
|
||||
|
||||
// A = g^a mod N
|
||||
expectedA := new(big.Int).Exp(srpG, client.a, srpN)
|
||||
assert.Equal(t, 0, expectedA.Cmp(client.A), "A must equal g^a mod N")
|
||||
|
||||
// getABytes returns padded A
|
||||
aBytes := client.getABytes()
|
||||
assert.Equal(t, srpNLenBytes, len(aBytes))
|
||||
recoveredA := new(big.Int).SetBytes(aBytes)
|
||||
assert.Equal(t, 0, client.A.Cmp(recoveredA))
|
||||
|
||||
// Two clients should have different secrets (probabilistic, but 2^256 collision chance)
|
||||
client2 := newSRPClient()
|
||||
assert.NotEqual(t, 0, client.a.Cmp(client2.a), "two clients must have different secrets")
|
||||
}
|
||||
|
||||
func TestProcessChallenge(t *testing.T) {
|
||||
// Use a fixed secret for deterministic testing
|
||||
fixedSecret := make([]byte, 32)
|
||||
fixedSecret[31] = 0x42 // small non-zero value
|
||||
|
||||
a := new(big.Int).SetBytes(fixedSecret)
|
||||
A := new(big.Int).Exp(srpG, a, srpN)
|
||||
k := getMultiplier()
|
||||
|
||||
client := &srpClient{
|
||||
a: a,
|
||||
A: A,
|
||||
k: k,
|
||||
}
|
||||
|
||||
username := []byte("testuser@apple.com")
|
||||
derivedKey := make([]byte, 32)
|
||||
for i := range derivedKey {
|
||||
derivedKey[i] = byte(i)
|
||||
}
|
||||
salt := []byte("fixed_salt_for_test!")
|
||||
|
||||
// Compute a valid B: B = (k*v + g^b) mod N where v = g^x mod N
|
||||
x := calculateX(salt, derivedKey)
|
||||
v := new(big.Int).Exp(srpG, x, srpN)
|
||||
|
||||
bSecret := make([]byte, 32)
|
||||
bSecret[31] = 0x99
|
||||
b := new(big.Int).SetBytes(bSecret)
|
||||
|
||||
// B = (k*v + g^b) mod N
|
||||
kv := new(big.Int).Mul(k, v)
|
||||
gb := new(big.Int).Exp(srpG, b, srpN)
|
||||
B := new(big.Int).Add(kv, gb)
|
||||
B.Mod(B, srpN)
|
||||
|
||||
serverB := padToN(B)
|
||||
|
||||
client.processChallenge(username, derivedKey, salt, serverB)
|
||||
|
||||
// M1, M2, K must be populated
|
||||
assert.NotNil(t, client.M1)
|
||||
assert.NotNil(t, client.M2)
|
||||
assert.NotNil(t, client.K)
|
||||
assert.Equal(t, sha256.Size, len(client.M1))
|
||||
assert.Equal(t, sha256.Size, len(client.M2))
|
||||
assert.Equal(t, sha256.Size, len(client.K))
|
||||
|
||||
// M1 and M2 must be different
|
||||
assert.NotEqual(t, client.M1, client.M2)
|
||||
|
||||
// Results must be deterministic — run again with same inputs
|
||||
client2 := &srpClient{
|
||||
a: new(big.Int).Set(a),
|
||||
A: new(big.Int).Set(A),
|
||||
k: new(big.Int).Set(k),
|
||||
}
|
||||
client2.processChallenge(username, derivedKey, salt, serverB)
|
||||
|
||||
assert.Equal(t, client.M1, client2.M1, "M1 must be deterministic")
|
||||
assert.Equal(t, client.M2, client2.M2, "M2 must be deterministic")
|
||||
assert.Equal(t, client.K, client2.K, "K must be deterministic")
|
||||
|
||||
// Verify the server could also compute S and get the same K.
|
||||
// Server computes: S_server = (A * v^u)^b mod N
|
||||
u := calculateU(A, B)
|
||||
vu := new(big.Int).Exp(v, u, srpN)
|
||||
avu := new(big.Int).Mul(A, vu)
|
||||
avu.Mod(avu, srpN)
|
||||
sServer := new(big.Int).Exp(avu, b, srpN)
|
||||
sServer.Mod(sServer, srpN)
|
||||
kServer := calculateK(padToN(sServer))
|
||||
assert.Equal(t, client.K, kServer, "client and server must derive the same session key")
|
||||
}
|
||||
|
||||
func TestSRPGroupParameters(t *testing.T) {
|
||||
// Verify the 2048-bit prime starts with known hex prefix
|
||||
nHex := hex.EncodeToString(srpN.Bytes())
|
||||
assert.True(t, len(nHex) > 0)
|
||||
assert.Equal(t, "ac6bdb41", nHex[:8], "N must start with known prefix")
|
||||
|
||||
// g must be 2
|
||||
assert.Equal(t, int64(2), srpG.Int64())
|
||||
|
||||
// N byte length must match expected
|
||||
assert.Equal(t, 256, srpNLenBytes)
|
||||
assert.Equal(t, 256, len(srpN.Bytes()))
|
||||
}
|
||||
|
||||
func TestSRPHash(t *testing.T) {
|
||||
data := []byte("hello world")
|
||||
hash := srpHash(data)
|
||||
|
||||
// Must be SHA-256 output
|
||||
assert.Equal(t, sha256.Size, len(hash))
|
||||
|
||||
// Must equal standard sha256
|
||||
expected := sha256.Sum256(data)
|
||||
assert.Equal(t, expected[:], hash)
|
||||
}
|
||||
@@ -1,12 +0,0 @@
|
||||
//go:build !linux
|
||||
|
||||
package local
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
)
|
||||
|
||||
func newFadviseReadCloser(o *Object, f *os.File, offset, limit int64) io.ReadCloser {
|
||||
return f
|
||||
}
|
||||
@@ -1,165 +0,0 @@
|
||||
//go:build linux
|
||||
|
||||
package local
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
|
||||
"github.com/rclone/rclone/fs"
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// fadvise provides means to automate freeing pages in kernel page cache for
|
||||
// a given file descriptor as the file is sequentially processed (read or
|
||||
// written).
|
||||
//
|
||||
// When copying a file to a remote backend all the file content is read by
|
||||
// kernel and put to page cache to make future reads faster.
|
||||
// This causes memory pressure visible in both memory usage and CPU consumption
|
||||
// and can even cause OOM errors in applications consuming large amounts memory.
|
||||
//
|
||||
// In case of an upload to a remote backend, there is no benefits from caching.
|
||||
//
|
||||
// fadvise would orchestrate calling POSIX_FADV_DONTNEED
|
||||
//
|
||||
// POSIX_FADV_DONTNEED attempts to free cached pages associated
|
||||
// with the specified region. This is useful, for example, while
|
||||
// streaming large files. A program may periodically request the
|
||||
// kernel to free cached data that has already been used, so that
|
||||
// more useful cached pages are not discarded instead.
|
||||
//
|
||||
// Requests to discard partial pages are ignored. It is
|
||||
// preferable to preserve needed data than discard unneeded data.
|
||||
// If the application requires that data be considered for
|
||||
// discarding, then offset and len must be page-aligned.
|
||||
//
|
||||
// The implementation may attempt to write back dirty pages in
|
||||
// the specified region, but this is not guaranteed. Any
|
||||
// unwritten dirty pages will not be freed. If the application
|
||||
// wishes to ensure that dirty pages will be released, it should
|
||||
// call fsync(2) or fdatasync(2) first.
|
||||
type fadvise struct {
|
||||
o *Object
|
||||
fd int
|
||||
lastPos int64
|
||||
curPos int64
|
||||
windowSize int64
|
||||
|
||||
freePagesCh chan offsetLength
|
||||
doneCh chan struct{}
|
||||
}
|
||||
|
||||
type offsetLength struct {
|
||||
offset int64
|
||||
length int64
|
||||
}
|
||||
|
||||
const (
|
||||
defaultAllowPages = 32
|
||||
defaultWorkerQueueSize = 64
|
||||
)
|
||||
|
||||
func newFadvise(o *Object, fd int, offset int64) *fadvise {
|
||||
f := &fadvise{
|
||||
o: o,
|
||||
fd: fd,
|
||||
lastPos: offset,
|
||||
curPos: offset,
|
||||
windowSize: int64(os.Getpagesize()) * defaultAllowPages,
|
||||
|
||||
freePagesCh: make(chan offsetLength, defaultWorkerQueueSize),
|
||||
doneCh: make(chan struct{}),
|
||||
}
|
||||
go f.worker()
|
||||
|
||||
return f
|
||||
}
|
||||
|
||||
// sequential configures readahead strategy in Linux kernel.
|
||||
//
|
||||
// Under Linux, POSIX_FADV_NORMAL sets the readahead window to the
|
||||
// default size for the backing device; POSIX_FADV_SEQUENTIAL doubles
|
||||
// this size, and POSIX_FADV_RANDOM disables file readahead entirely.
|
||||
func (f *fadvise) sequential(limit int64) bool {
|
||||
l := int64(0)
|
||||
if limit > 0 {
|
||||
l = limit
|
||||
}
|
||||
if err := unix.Fadvise(f.fd, f.curPos, l, unix.FADV_SEQUENTIAL); err != nil {
|
||||
fs.Debugf(f.o, "fadvise sequential failed on file descriptor %d: %s", f.fd, err)
|
||||
return false
|
||||
}
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
func (f *fadvise) next(n int) {
|
||||
f.curPos += int64(n)
|
||||
f.freePagesIfNeeded()
|
||||
}
|
||||
|
||||
func (f *fadvise) freePagesIfNeeded() {
|
||||
if f.curPos >= f.lastPos+f.windowSize {
|
||||
f.freePages()
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fadvise) freePages() {
|
||||
f.freePagesCh <- offsetLength{f.lastPos, f.curPos - f.lastPos}
|
||||
f.lastPos = f.curPos
|
||||
}
|
||||
|
||||
func (f *fadvise) worker() {
|
||||
for p := range f.freePagesCh {
|
||||
if err := unix.Fadvise(f.fd, p.offset, p.length, unix.FADV_DONTNEED); err != nil {
|
||||
fs.Debugf(f.o, "fadvise dontneed failed on file descriptor %d: %s", f.fd, err)
|
||||
}
|
||||
}
|
||||
|
||||
close(f.doneCh)
|
||||
}
|
||||
|
||||
func (f *fadvise) wait() {
|
||||
close(f.freePagesCh)
|
||||
<-f.doneCh
|
||||
}
|
||||
|
||||
type fadviseReadCloser struct {
|
||||
*fadvise
|
||||
inner io.ReadCloser
|
||||
}
|
||||
|
||||
// newFadviseReadCloser wraps os.File so that reading from that file would
|
||||
// remove already consumed pages from kernel page cache.
|
||||
// In addition to that it instructs kernel to double the readahead window to
|
||||
// make sequential reads faster.
|
||||
// See also fadvise.
|
||||
func newFadviseReadCloser(o *Object, f *os.File, offset, limit int64) io.ReadCloser {
|
||||
r := fadviseReadCloser{
|
||||
fadvise: newFadvise(o, int(f.Fd()), offset),
|
||||
inner: f,
|
||||
}
|
||||
|
||||
// If syscall failed it's likely that the subsequent syscalls to that
|
||||
// file descriptor would also fail. In that case return the provided os.File
|
||||
// pointer.
|
||||
if !r.sequential(limit) {
|
||||
r.wait()
|
||||
return f
|
||||
}
|
||||
|
||||
return r
|
||||
}
|
||||
|
||||
func (f fadviseReadCloser) Read(p []byte) (n int, err error) {
|
||||
n, err = f.inner.Read(p)
|
||||
f.next(n)
|
||||
return
|
||||
}
|
||||
|
||||
func (f fadviseReadCloser) Close() error {
|
||||
f.freePages()
|
||||
f.wait()
|
||||
return f.inner.Close()
|
||||
}
|
||||
@@ -1168,7 +1168,7 @@ func (o *Object) Hash(ctx context.Context, r hash.Type) (string, error) {
|
||||
var fd *os.File
|
||||
fd, err = file.Open(o.path)
|
||||
if fd != nil {
|
||||
in = newFadviseReadCloser(o, fd, 0, 0)
|
||||
in = fd
|
||||
}
|
||||
} else {
|
||||
in, err = o.openTranslatedLink(0, -1)
|
||||
@@ -1374,7 +1374,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
wrappedFd := readers.NewLimitedReadCloser(newFadviseReadCloser(o, fd, offset, limit), limit)
|
||||
wrappedFd := readers.NewLimitedReadCloser(fd, limit)
|
||||
if offset != 0 {
|
||||
// seek the object
|
||||
_, err = fd.Seek(offset, io.SeekStart)
|
||||
|
||||
@@ -16,11 +16,12 @@ type Authenticator interface {
|
||||
}
|
||||
|
||||
// IbmIamSigner is a structure for signing requests using IBM IAM.
|
||||
// Requeres APIKey and Resource InstanceID
|
||||
// Requires APIKey and Resource InstanceID
|
||||
type IbmIamSigner struct {
|
||||
APIKey string
|
||||
InstanceID string
|
||||
Auth Authenticator
|
||||
APIKey string
|
||||
InstanceID string
|
||||
IAMEndpoint string
|
||||
Auth Authenticator
|
||||
}
|
||||
|
||||
// SignHTTP signs requests using IBM IAM token.
|
||||
@@ -29,7 +30,7 @@ func (signer *IbmIamSigner) SignHTTP(ctx context.Context, credentials aws.Creden
|
||||
if signer.Auth != nil {
|
||||
authenticator = signer.Auth
|
||||
} else {
|
||||
authenticator = &core.IamAuthenticator{ApiKey: signer.APIKey}
|
||||
authenticator = &core.IamAuthenticator{ApiKey: signer.APIKey, URL: signer.IAMEndpoint}
|
||||
}
|
||||
token, err := authenticator.GetToken()
|
||||
if err != nil {
|
||||
|
||||
@@ -23,14 +23,16 @@ func TestSignHTTP(t *testing.T) {
|
||||
apiKey := "mock-api-key"
|
||||
instanceID := "mock-instance-id"
|
||||
token := "mock-iam-token"
|
||||
iamEndpoint := "https://iamendpoint.com"
|
||||
mockAuth := &MockAuthenticator{
|
||||
Token: token,
|
||||
Error: nil,
|
||||
}
|
||||
signer := &IbmIamSigner{
|
||||
APIKey: apiKey,
|
||||
InstanceID: instanceID,
|
||||
Auth: mockAuth,
|
||||
APIKey: apiKey,
|
||||
InstanceID: instanceID,
|
||||
IAMEndpoint: iamEndpoint,
|
||||
Auth: mockAuth,
|
||||
}
|
||||
req, err := http.NewRequest("GET", "https://example.com", nil)
|
||||
if err != nil {
|
||||
|
||||
@@ -4,6 +4,7 @@ region:
|
||||
eu-west-1: Europe West
|
||||
endpoint:
|
||||
s3.cubbit.eu: Cubbit DS3 Object Storage endpoint
|
||||
s3.{tenant_name}.cubbit.eu: Multi-tenant endpoint - replace {tenant_name} with your actual tenant name. Do not select this directly
|
||||
acl: {}
|
||||
bucket_acl: true
|
||||
quirks:
|
||||
|
||||
@@ -18,3 +18,7 @@ quirks:
|
||||
# See: https://issuetracker.google.com/issues/323465186
|
||||
# So make cutoff very large which it does seem to support
|
||||
copy_cutoff: 9223372036854775807
|
||||
# GCS S3 API doesn't fully support Object Lock:
|
||||
# - Uses x-goog-bypass-governance-retention instead of x-amz-bypass-governance-retention
|
||||
# - PutObjectLegalHold / GetObjectLegalHold not implemented
|
||||
object_lock_supported: false
|
||||
@@ -117,6 +117,7 @@ acl:
|
||||
This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
|
||||
ibm_api_key: true
|
||||
ibm_resource_instance_id: true
|
||||
ibm_iam_endpoint: true
|
||||
bucket_acl: true
|
||||
quirks:
|
||||
list_version: 1
|
||||
|
||||
@@ -32,5 +32,15 @@ endpoint:
|
||||
s3.us-east-va.io.cloud.ovh.us: OVHcloud Vint Hill, Virginia, USA
|
||||
s3.us-west-or.io.cloud.ovh.us: OVHcloud Hillsboro, Oregon, USA
|
||||
s3.rbx-archive.io.cloud.ovh.net: OVHcloud Roubaix, France (Cold Archive)
|
||||
storage_class:
|
||||
'': Default
|
||||
EXPRESS_ONEZONE: High Performance storage class
|
||||
STANDARD: Standard storage class
|
||||
STANDARD_IA: Standard Infrequent Access storage class
|
||||
ONEZONE_IA: Standard Infrequent Access storage class
|
||||
GLACIER: Active Archive storage class
|
||||
GLACIER_IR: Active Archive storage class
|
||||
DEEP_ARCHIVE: Cold Archive storage class
|
||||
INTELLIGENT_TIERING: Standard storage class
|
||||
acl: {}
|
||||
bucket_acl: true
|
||||
@@ -0,0 +1,44 @@
|
||||
name: US3
|
||||
description: US3 Object Storage
|
||||
endpoint:
|
||||
s3-cn-bj.ufileos.com: North China (Beijing 2)
|
||||
s3-cn-wlcb.ufileos.com: North China (Ulanqab)
|
||||
s3-cn-sh2.ufileos.com: Shanghai
|
||||
s3-cn-gd.ufileos.com: South China (Guangzhou)
|
||||
s3-hk.ufileos.com: Hongkong
|
||||
s3-us-ca.ufileos.com: United States (Los Angeles)
|
||||
s3-sg.ufileos.com: Singapore
|
||||
s3-idn-jakarta.ufileos.com: Indonesia (Jakarta)
|
||||
s3-tw-tp.ufileos.com: Taiwan (Taipei)
|
||||
s3-afr-nigeria.ufileos.com: Nigeria (Lagos)
|
||||
s3-bra-saopaulo.ufileos.com: Brazil (São Paulo)
|
||||
s3-uae-dubai.ufileos.com: United Arab Emirates (Dubai)
|
||||
s3-ge-fra.ufileos.com: Germany (Frankfurt)
|
||||
s3-vn-sng.ufileos.com: Vietnam (Ho Chi Minh City)
|
||||
s3-us-ws.ufileos.com: United States (Washington)
|
||||
s3-ind-mumbai.ufileos.com: India (Mumbai)
|
||||
s3-kr-seoul.ufileos.com: Seoul (South Korea)
|
||||
s3-jpn-tky.ufileos.com: Japan (Tokyo)
|
||||
s3-th-bkk.ufileos.com: Bangkok (Thailand)
|
||||
s3-uk-london.ufileos.com: United Kingdom (London)
|
||||
s3-rus-mosc.ufileos.com: Russia (Moscow)
|
||||
s3-cn-guiyang1.ufileos.com: Guiyang 1
|
||||
s3-pk-khi.ufileos.com: Pakistan
|
||||
acl:
|
||||
private: |-
|
||||
Owner gets FULL_CONTROL.
|
||||
No one else has access rights (default).
|
||||
public-read: |-
|
||||
Owner gets FULL_CONTROL.
|
||||
The AllUsers group gets READ access.
|
||||
storage_class:
|
||||
'': Default
|
||||
STANDARD: Standard storage class
|
||||
ARCHIVE: Archive storage mode
|
||||
STANDARD_IA: Infrequent access storage mode
|
||||
bucket_acl: true
|
||||
quirks:
|
||||
list_version: 1
|
||||
use_multipart_etag: false
|
||||
use_already_exists: false
|
||||
list_url_encode: true
|
||||
@@ -33,6 +33,7 @@ type Quirks struct {
|
||||
UseXID *bool `yaml:"use_x_id,omitempty"`
|
||||
SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"`
|
||||
EtagIsNotMD5 *bool `yaml:"etag_is_not_md5,omitempty"`
|
||||
ObjectLockSupported *bool `yaml:"object_lock_supported,omitempty"`
|
||||
CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"`
|
||||
MaxUploadParts *int `yaml:"max_upload_parts,omitempty"`
|
||||
MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"`
|
||||
@@ -52,6 +53,7 @@ type Provider struct {
|
||||
// other
|
||||
IBMApiKey bool `yaml:"ibm_api_key,omitempty"`
|
||||
IBMResourceInstanceID bool `yaml:"ibm_resource_instance_id,omitempty"`
|
||||
IBMIAMEndpoint bool `yaml:"ibm_iam_endpoint,omitempty"`
|
||||
|
||||
// advanced
|
||||
BucketACL bool `yaml:"bucket_acl,omitempty"`
|
||||
@@ -224,6 +226,8 @@ func constructProviders(options fs.Options, providerMap map[string]*Provider) st
|
||||
addBool(opt, p, p.IBMApiKey)
|
||||
case "ibm_resource_instance_id":
|
||||
addBool(opt, p, p.IBMResourceInstanceID)
|
||||
case "ibm_iam_endpoint":
|
||||
addBool(opt, p, p.IBMIAMEndpoint)
|
||||
case "leave_parts_on_error":
|
||||
addBool(opt, p, p.LeavePartsOnError)
|
||||
case "sts_endpoint":
|
||||
|
||||
+49
-1
@@ -4,6 +4,7 @@ package s3
|
||||
//go:generate go run gen_setfrom.go -o setfrom.go
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/md5"
|
||||
"crypto/tls"
|
||||
@@ -829,6 +830,10 @@ use |-vv| to see the debug level logs.
|
||||
}, {
|
||||
Name: "ibm_resource_instance_id",
|
||||
Help: "IBM service instance id",
|
||||
}, {
|
||||
Name: "ibm_iam_endpoint",
|
||||
Help: "IBM IAM Endpoint to use for authentication.\n\nLeave blank to use the default public endpoint.",
|
||||
Advanced: true,
|
||||
}, {
|
||||
Name: "object_lock_mode",
|
||||
Help: `Object Lock mode to apply when uploading or copying objects.
|
||||
@@ -930,6 +935,18 @@ PutObjectRetention and PutObjectLegalHold API calls after the upload completes.
|
||||
This adds extra API calls per object, so only enable if your provider requires it.`,
|
||||
Default: false,
|
||||
Advanced: true,
|
||||
}, {
|
||||
Name: "object_lock_supported",
|
||||
Help: `Whether the provider supports S3 Object Lock.
|
||||
|
||||
This should be true, false or left unset to use the default for the provider.
|
||||
|
||||
Set to false for providers that don't fully support the S3 Object Lock API
|
||||
(e.g. GCS which uses non-standard headers for bypass governance retention
|
||||
and doesn't implement Legal Hold via the S3 API).
|
||||
`,
|
||||
Default: fs.Tristate{},
|
||||
Advanced: true,
|
||||
},
|
||||
}}))
|
||||
}
|
||||
@@ -1107,6 +1124,7 @@ type Options struct {
|
||||
DirectoryBucket bool `config:"directory_bucket"`
|
||||
IBMAPIKey string `config:"ibm_api_key"`
|
||||
IBMInstanceID string `config:"ibm_resource_instance_id"`
|
||||
IBMIAMEndpoint string `config:"ibm_iam_endpoint"`
|
||||
UseXID fs.Tristate `config:"use_x_id"`
|
||||
SignAcceptEncoding fs.Tristate `config:"sign_accept_encoding"`
|
||||
ObjectLockMode string `config:"object_lock_mode"`
|
||||
@@ -1115,6 +1133,7 @@ type Options struct {
|
||||
BypassGovernanceRetention bool `config:"bypass_governance_retention"`
|
||||
BucketObjectLockEnabled bool `config:"bucket_object_lock_enabled"`
|
||||
ObjectLockSetAfterUpload bool `config:"object_lock_set_after_upload"`
|
||||
ObjectLockSupported fs.Tristate `config:"object_lock_supported"`
|
||||
}
|
||||
|
||||
// Fs represents a remote s3 server
|
||||
@@ -1541,7 +1560,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
|
||||
fs.Debugf(nil, "Using v2 auth")
|
||||
if opt.Provider == "IBMCOS" && opt.IBMAPIKey != "" && opt.IBMInstanceID != "" {
|
||||
options = append(options, func(s3Opt *s3.Options) {
|
||||
s3Opt.HTTPSignerV4 = &IbmIamSigner{APIKey: opt.IBMAPIKey, InstanceID: opt.IBMInstanceID}
|
||||
s3Opt.HTTPSignerV4 = &IbmIamSigner{APIKey: opt.IBMAPIKey, InstanceID: opt.IBMInstanceID, IAMEndpoint: opt.IBMIAMEndpoint}
|
||||
})
|
||||
} else {
|
||||
options = append(options, func(s3Opt *s3.Options) {
|
||||
@@ -1721,6 +1740,7 @@ func setQuirks(opt *Options, provider *Provider) {
|
||||
set(&opt.UseUnsignedPayload, true, provider.Quirks.UseUnsignedPayload)
|
||||
set(&opt.UseXID, true, provider.Quirks.UseXID)
|
||||
set(&opt.SignAcceptEncoding, true, provider.Quirks.SignAcceptEncoding)
|
||||
set(&opt.ObjectLockSupported, true, provider.Quirks.ObjectLockSupported)
|
||||
}
|
||||
|
||||
// setRoot changes the root of the Fs
|
||||
@@ -4603,8 +4623,31 @@ func (o *Object) uploadMultipart(ctx context.Context, src fs.ObjectInfo, in io.R
|
||||
return wantETag, gotETag, versionID, s3cw.ui, nil
|
||||
}
|
||||
|
||||
// bufferForObjectLockMD5 buffers the body and computes Content-MD5 when
|
||||
// Object Lock parameters are set on the request. AWS S3 requires Content-MD5
|
||||
// for PutObject with Object Lock params and cannot compute it automatically
|
||||
// from a non-seekable io.Reader.
|
||||
// See: https://github.com/aws/aws-sdk-go-v2/discussions/2960
|
||||
func bufferForObjectLockMD5(req *s3.PutObjectInput, in io.Reader) (io.Reader, error) {
|
||||
if req.ObjectLockMode == "" && req.ObjectLockRetainUntilDate == nil && req.ObjectLockLegalHoldStatus == "" {
|
||||
return in, nil
|
||||
}
|
||||
buf, err := io.ReadAll(in)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read body for Content-MD5: %w", err)
|
||||
}
|
||||
md5sum := md5.Sum(buf)
|
||||
md5base64 := base64.StdEncoding.EncodeToString(md5sum[:])
|
||||
req.ContentMD5 = &md5base64
|
||||
return bytes.NewReader(buf), nil
|
||||
}
|
||||
|
||||
// Upload a single part using PutObject
|
||||
func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
|
||||
in, err = bufferForObjectLockMD5(req, in)
|
||||
if err != nil {
|
||||
return etag, lastModified, nil, err
|
||||
}
|
||||
req.Body = io.NopCloser(in)
|
||||
var options = []func(*s3.Options){}
|
||||
if o.fs.opt.UseUnsignedPayload.Value {
|
||||
@@ -4635,6 +4678,11 @@ func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjec
|
||||
|
||||
// Upload a single part using a presigned request
|
||||
func (o *Object) uploadSinglepartPresignedRequest(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
|
||||
// Content-MD5 must be set before signing so it's included in the presigned URL.
|
||||
in, err = bufferForObjectLockMD5(req, in)
|
||||
if err != nil {
|
||||
return etag, lastModified, nil, err
|
||||
}
|
||||
// Create the presigned request
|
||||
putReq, err := s3.NewPresignClient(o.fs.c).PresignPutObject(ctx, req, s3.WithPresignExpires(15*time.Minute))
|
||||
if err != nil {
|
||||
|
||||
@@ -499,6 +499,9 @@ func (f *Fs) InternalTestVersions(t *testing.T) {
|
||||
}
|
||||
|
||||
func (f *Fs) InternalTestObjectLock(t *testing.T) {
|
||||
if !f.opt.ObjectLockSupported.Value {
|
||||
t.Skip("Object Lock not supported by this provider (quirk object_lock_supported = false)")
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
// Create a temporary bucket with Object Lock enabled to test on.
|
||||
@@ -721,6 +724,55 @@ func (f *Fs) InternalTestObjectLock(t *testing.T) {
|
||||
assert.WithinDuration(t, retainUntilDate, gotRetainDate, time.Second)
|
||||
assert.Equal(t, "ON", gotMetadata["object-lock-legal-hold-status"])
|
||||
})
|
||||
|
||||
t.Run("Multipart", func(t *testing.T) {
|
||||
// Force multipart upload by setting a very low cutoff
|
||||
oldCutoff := f.opt.UploadCutoff
|
||||
f.opt.UploadCutoff = fs.SizeSuffix(1)
|
||||
f.opt.ObjectLockMode = "GOVERNANCE"
|
||||
f.opt.ObjectLockRetainUntilDate = retainUntilDate.Format(time.RFC3339)
|
||||
defer func() {
|
||||
f.opt.UploadCutoff = oldCutoff
|
||||
f.opt.ObjectLockMode = ""
|
||||
f.opt.ObjectLockRetainUntilDate = ""
|
||||
}()
|
||||
|
||||
contents := random.String(100)
|
||||
item := fstest.NewItem("test-object-lock-multipart", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
|
||||
obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
|
||||
defer func() {
|
||||
removeLocked(t, obj)
|
||||
}()
|
||||
|
||||
o := obj.(*Object)
|
||||
gotMetadata, err := o.Metadata(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "GOVERNANCE", gotMetadata["object-lock-mode"])
|
||||
})
|
||||
|
||||
t.Run("Presigned", func(t *testing.T) {
|
||||
// Use presigned request upload path
|
||||
f.opt.UsePresignedRequest = true
|
||||
f.opt.ObjectLockMode = "GOVERNANCE"
|
||||
f.opt.ObjectLockRetainUntilDate = retainUntilDate.Format(time.RFC3339)
|
||||
defer func() {
|
||||
f.opt.UsePresignedRequest = false
|
||||
f.opt.ObjectLockMode = ""
|
||||
f.opt.ObjectLockRetainUntilDate = ""
|
||||
}()
|
||||
|
||||
contents := random.String(100)
|
||||
item := fstest.NewItem("test-object-lock-presigned", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
|
||||
obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
|
||||
defer func() {
|
||||
removeLocked(t, obj)
|
||||
}()
|
||||
|
||||
o := obj.(*Object)
|
||||
gotMetadata, err := o.Metadata(ctx)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "GOVERNANCE", gotMetadata["object-lock-mode"])
|
||||
})
|
||||
}
|
||||
|
||||
func (f *Fs) InternalTest(t *testing.T) {
|
||||
|
||||
@@ -225,6 +225,7 @@ type Fs struct {
|
||||
hasOCMD5 bool // set if can use owncloud style checksums for MD5
|
||||
hasOCSHA1 bool // set if can use owncloud style checksums for SHA1
|
||||
hasMESHA1 bool // set if can use fastmail style checksums for SHA1
|
||||
useStandardProps bool // set if should use standard props for PROPFIND
|
||||
ntlmAuthMu sync.Mutex // mutex to serialize NTLM auth roundtrips
|
||||
chunksUploadURL string // upload URL for nextcloud chunked
|
||||
canChunk bool // set if nextcloud and nextcloud_chunk_size is set
|
||||
@@ -352,7 +353,10 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string, depth string)
|
||||
}
|
||||
if f.hasOCMD5 || f.hasOCSHA1 {
|
||||
opts.Body = bytes.NewBuffer(owncloudProps)
|
||||
} else if f.useStandardProps {
|
||||
opts.Body = bytes.NewBuffer(standardProps)
|
||||
}
|
||||
// Note: According to WebDAV RFC 4918, empty PROPFIND body defaults to allprop
|
||||
var result api.Multistatus
|
||||
var resp *http.Response
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
@@ -711,6 +715,7 @@ func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
|
||||
f.precision = time.Second
|
||||
f.useOCMtime = true
|
||||
case "other":
|
||||
f.useStandardProps = true
|
||||
default:
|
||||
fs.Debugf(f, "Unknown vendor %q", vendor)
|
||||
}
|
||||
@@ -759,13 +764,23 @@ var owncloudProps = []byte(`<?xml version="1.0"?>
|
||||
<d:getlastmodified />
|
||||
<d:getcontentlength />
|
||||
<d:resourcetype />
|
||||
<d:getcontenttype />
|
||||
<oc:checksums />
|
||||
<oc:permissions />
|
||||
</d:prop>
|
||||
</d:propfind>
|
||||
`)
|
||||
|
||||
var standardProps = []byte(`<?xml version="1.0"?>
|
||||
<d:propfind xmlns:d="DAV:">
|
||||
<d:prop>
|
||||
<d:displayname/>
|
||||
<d:getlastmodified/>
|
||||
<d:getcontentlength/>
|
||||
<d:resourcetype/>
|
||||
</d:prop>
|
||||
</d:propfind>
|
||||
`)
|
||||
|
||||
// list the objects into the function supplied
|
||||
//
|
||||
// If directories is set it only sends directories
|
||||
@@ -787,7 +802,10 @@ func (f *Fs) listAll(ctx context.Context, dir string, directoriesOnly bool, file
|
||||
}
|
||||
if f.hasOCMD5 || f.hasOCSHA1 {
|
||||
opts.Body = bytes.NewBuffer(owncloudProps)
|
||||
} else if f.useStandardProps {
|
||||
opts.Body = bytes.NewBuffer(standardProps)
|
||||
}
|
||||
// Note: According to WebDAV RFC 4918, empty PROPFIND body defaults to `allprop`
|
||||
var result api.Multistatus
|
||||
var resp *http.Response
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
|
||||
+4
-3
@@ -127,14 +127,14 @@ func init() {
|
||||
// and the Command line syntax section of docs/content/bisync.md (it doesn't update automatically)
|
||||
flags.BoolVarP(cmdFlags, &Opt.Resync, "resync", "1", Opt.Resync, "Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first.", "")
|
||||
flags.FVarP(cmdFlags, &Opt.ResyncMode, "resync-mode", "", "During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.)", "")
|
||||
flags.BoolVarP(cmdFlags, &Opt.CheckAccess, "check-access", "", Opt.CheckAccess, makeHelp("Ensure expected {CHECKFILE} files are found on both Path1 and Path2 filesystems, else abort."), "")
|
||||
flags.StringVarP(cmdFlags, &Opt.CheckFilename, "check-filename", "", Opt.CheckFilename, makeHelp("Filename for --check-access (default: {CHECKFILE})"), "")
|
||||
flags.BoolVarP(cmdFlags, &Opt.CheckAccess, "check-access", "", Opt.CheckAccess, MakeHelp("Ensure expected {CHECKFILE} files are found on both Path1 and Path2 filesystems, else abort."), "")
|
||||
flags.StringVarP(cmdFlags, &Opt.CheckFilename, "check-filename", "", Opt.CheckFilename, MakeHelp("Filename for --check-access (default: {CHECKFILE})"), "")
|
||||
flags.BoolVarP(cmdFlags, &Opt.Force, "force", "", Opt.Force, "Bypass --max-delete safety check and run the sync. Consider using with --verbose", "")
|
||||
flags.FVarP(cmdFlags, &Opt.CheckSync, "check-sync", "", "Controls comparison of final listings: true|false|only (default: true)", "")
|
||||
flags.BoolVarP(cmdFlags, &Opt.CreateEmptySrcDirs, "create-empty-src-dirs", "", Opt.CreateEmptySrcDirs, "Sync creation and deletion of empty directories. (Not compatible with --remove-empty-dirs)", "")
|
||||
flags.BoolVarP(cmdFlags, &Opt.RemoveEmptyDirs, "remove-empty-dirs", "", Opt.RemoveEmptyDirs, "Remove ALL empty directories at the final cleanup step.", "")
|
||||
flags.StringVarP(cmdFlags, &Opt.FiltersFile, "filters-file", "", Opt.FiltersFile, "Read filtering patterns from a file", "")
|
||||
flags.StringVarP(cmdFlags, &Opt.Workdir, "workdir", "", Opt.Workdir, makeHelp("Use custom working dir - useful for testing. (default: {WORKDIR})"), "")
|
||||
flags.StringVarP(cmdFlags, &Opt.Workdir, "workdir", "", Opt.Workdir, MakeHelp("Use custom working dir - useful for testing. (default: {WORKDIR})"), "")
|
||||
flags.StringVarP(cmdFlags, &Opt.BackupDir1, "backup-dir1", "", Opt.BackupDir1, "--backup-dir for Path1. Must be a non-overlapping path on the same remote.", "")
|
||||
flags.StringVarP(cmdFlags, &Opt.BackupDir2, "backup-dir2", "", Opt.BackupDir2, "--backup-dir for Path2. Must be a non-overlapping path on the same remote.", "")
|
||||
flags.StringVarP(cmdFlags, &Opt.DebugName, "debugname", "", Opt.DebugName, "Debug by tracking one file at various points throughout a bisync run (when -v or -vv)", "")
|
||||
@@ -153,6 +153,7 @@ func init() {
|
||||
flags.StringVarP(cmdFlags, &Opt.ConflictSuffixFlag, "conflict-suffix", "", Opt.ConflictSuffixFlag, "Suffix to use when renaming a --conflict-loser. Can be either one string or two comma-separated strings to assign different suffixes to Path1/Path2. (default: 'conflict')", "")
|
||||
_ = cmdFlags.MarkHidden("debugname")
|
||||
_ = cmdFlags.MarkHidden("localtime")
|
||||
addRC()
|
||||
}
|
||||
|
||||
// bisync command definition
|
||||
|
||||
+65
-53
@@ -1,65 +1,77 @@
|
||||
package bisync
|
||||
//go:build none
|
||||
|
||||
// Create the help text for the rc
|
||||
//
|
||||
// Run with go generate ./cmd/bisync (defined in rc.go)
|
||||
package main
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"github.com/muesli/reflow/wordwrap"
|
||||
"github.com/rclone/rclone/cmd"
|
||||
"github.com/rclone/rclone/cmd/bisync"
|
||||
"github.com/rclone/rclone/fs"
|
||||
"github.com/spf13/pflag"
|
||||
"golang.org/x/text/cases"
|
||||
"golang.org/x/text/language"
|
||||
)
|
||||
|
||||
func makeHelp(help string) string {
|
||||
replacer := strings.NewReplacer(
|
||||
"|", "`",
|
||||
"{MAXDELETE}", strconv.Itoa(DefaultMaxDelete),
|
||||
"{CHECKFILE}", DefaultCheckFilename,
|
||||
// "{WORKDIR}", DefaultWorkdir,
|
||||
)
|
||||
return replacer.Replace(help)
|
||||
// Output the help to stdout
|
||||
func main() {
|
||||
out := os.Stdout
|
||||
if len(os.Args) > 1 {
|
||||
var err error
|
||||
out, err = os.Create(os.Args[1])
|
||||
if err != nil {
|
||||
fs.Fatalf(nil, "Open output failed: %v", err)
|
||||
}
|
||||
defer out.Close()
|
||||
}
|
||||
fmt.Fprintf(out, "<!--- Docs generated by help.go - use go generate to rebuild - DO NOT EDIT --->\n\n")
|
||||
fmt.Fprint(out, RcHelp())
|
||||
}
|
||||
|
||||
var shortHelp = `Perform bidirectional synchronization between two paths.`
|
||||
|
||||
var rcHelp = makeHelp(`This takes the following parameters
|
||||
|
||||
- path1 - a remote directory string e.g. |drive:path1|
|
||||
- path2 - a remote directory string e.g. |drive:path2|
|
||||
- dryRun - dry-run mode
|
||||
- resync - performs the resync run
|
||||
- checkAccess - abort if {CHECKFILE} files are not found on both filesystems
|
||||
- checkFilename - file name for checkAccess (default: {CHECKFILE})
|
||||
- maxDelete - abort sync if percentage of deleted files is above
|
||||
this threshold (default: {MAXDELETE})
|
||||
- force - Bypass maxDelete safety check and run the sync
|
||||
- checkSync - |true| by default, |false| disables comparison of final listings,
|
||||
|only| will skip sync, only compare listings from the last run
|
||||
- createEmptySrcDirs - Sync creation and deletion of empty directories.
|
||||
(Not compatible with --remove-empty-dirs)
|
||||
- removeEmptyDirs - remove empty directories at the final cleanup step
|
||||
- filtersFile - read filtering patterns from a file
|
||||
- ignoreListingChecksum - Do not use checksums for listings
|
||||
- resilient - Allow future runs to retry after certain less-serious errors, instead of requiring resync.
|
||||
- workdir - server directory for history files (default: |~/.cache/rclone/bisync|)
|
||||
- backupdir1 - --backup-dir for Path1. Must be a non-overlapping path on the same remote.
|
||||
- backupdir2 - --backup-dir for Path2. Must be a non-overlapping path on the same remote.
|
||||
- noCleanup - retain working files
|
||||
// RcHelp returns the rc help
|
||||
func RcHelp() string {
|
||||
return wordwrap.String(bisync.MakeHelp(`This takes the following parameters:
|
||||
|
||||
- path1 (required) - (string) a remote directory string e.g. ||drive:path1||
|
||||
- path2 (required) - (string) a remote directory string e.g. ||drive:path2||
|
||||
- dryRun - (bool) dry-run mode
|
||||
`+GenerateParams()+`
|
||||
See [bisync command help](https://rclone.org/commands/rclone_bisync/)
|
||||
and [full bisync description](https://rclone.org/bisync/)
|
||||
for more information.`)
|
||||
for more information.
|
||||
`), 76)
|
||||
}
|
||||
|
||||
var longHelp = shortHelp + makeHelp(`
|
||||
// example: "create-empty-src-dirs" -> "createEmptySrcDirs"
|
||||
func toCamel(s string) string {
|
||||
split := strings.Split(s, "-")
|
||||
builder := strings.Builder{}
|
||||
for i, word := range split {
|
||||
if i == 0 { // first word always all lowercase
|
||||
builder.WriteString(strings.ToLower(word))
|
||||
continue
|
||||
}
|
||||
builder.WriteString(cases.Title(language.AmericanEnglish).String(word))
|
||||
}
|
||||
return builder.String()
|
||||
}
|
||||
|
||||
[Bisync](https://rclone.org/bisync/) provides a
|
||||
bidirectional cloud sync solution in rclone.
|
||||
It retains the Path1 and Path2 filesystem listings from the prior run.
|
||||
On each successive run it will:
|
||||
|
||||
- list files on Path1 and Path2, and check for changes on each side.
|
||||
Changes include |New|, |Newer|, |Older|, and |Deleted| files.
|
||||
- Propagate changes on Path1 to Path2, and vice-versa.
|
||||
|
||||
Bisync is considered an **advanced command**, so use with care.
|
||||
Make sure you have read and understood the entire [manual](https://rclone.org/bisync)
|
||||
(especially the [Limitations](https://rclone.org/bisync/#limitations) section)
|
||||
before using, or data loss can result. Questions can be asked in the
|
||||
[Rclone Forum](https://forum.rclone.org/).
|
||||
|
||||
See [full bisync description](https://rclone.org/bisync/) for details.`)
|
||||
// GenerateParams automatically generates the param list from commandDefinition.Flags
|
||||
func GenerateParams() string {
|
||||
builder := strings.Builder{}
|
||||
fn := func(flag *pflag.Flag) {
|
||||
if flag.Hidden {
|
||||
return
|
||||
}
|
||||
builder.WriteString(fmt.Sprintf("- %s - (%s) %s \n", toCamel(flag.Name), flag.Value.Type(), flag.Usage))
|
||||
}
|
||||
commandDefinition, _, _ := cmd.Root.Find([]string{"bisync"})
|
||||
commandDefinition.Flags().VisitAll(fn)
|
||||
return builder.String()
|
||||
}
|
||||
+12
-1
@@ -98,15 +98,26 @@ func (b *bisyncRun) lockFileIsExpired() bool {
|
||||
rdf, err := os.Open(b.lockFile)
|
||||
b.handleErr(b.lockFile, "error reading lock file", err, true, true)
|
||||
dec := json.NewDecoder(rdf)
|
||||
var decodeErr error
|
||||
for {
|
||||
if err := dec.Decode(&b.lockFileOpt.data); err != nil {
|
||||
if err != io.EOF {
|
||||
fs.Errorf(b.lockFile, "err: %v", err)
|
||||
decodeErr = err
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
b.handleErr(b.lockFile, "error closing file", rdf.Close(), true, true)
|
||||
if decodeErr != nil {
|
||||
if b.opt.MaxLock < basicallyforever {
|
||||
fs.Infof(b.lockFile, Color(terminal.YellowFg, "Lock file is unreadable (decode error: %v) and --max-lock is set. Treating as expired."), decodeErr)
|
||||
markFailed(b.listing1)
|
||||
markFailed(b.listing2)
|
||||
return true
|
||||
}
|
||||
fs.Errorf(b.lockFile, Color(terminal.RedFg, "Lock file exists, but contents are unreadable. (decode error: %v)"), decodeErr)
|
||||
return false
|
||||
}
|
||||
if !b.lockFileOpt.data.TimeExpires.IsZero() && b.lockFileOpt.data.TimeExpires.Before(time.Now()) {
|
||||
fs.Infof(b.lockFile, Color(terminal.GreenFg, "Lock file found, but it expired at %v. Will delete it and proceed."), b.lockFileOpt.data.TimeExpires)
|
||||
markFailed(b.listing1) // listing is untrusted so force revert to prior (if --recover) or create new ones (if --resync)
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package bisync
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/rclone/rclone/fs"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func newTestLockfileBisyncRun(t *testing.T, lockContent string, maxLock fs.Duration) *bisyncRun {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
lockPath := filepath.Join(dir, "test.lck")
|
||||
require.NoError(t, os.WriteFile(lockPath, []byte(lockContent), 0600))
|
||||
|
||||
listing1 := filepath.Join(dir, "listing1")
|
||||
listing2 := filepath.Join(dir, "listing2")
|
||||
require.NoError(t, os.WriteFile(listing1, []byte(""), 0600))
|
||||
require.NoError(t, os.WriteFile(listing2, []byte(""), 0600))
|
||||
|
||||
return &bisyncRun{
|
||||
lockFile: lockPath,
|
||||
opt: &Options{MaxLock: maxLock},
|
||||
listing1: listing1,
|
||||
listing2: listing2,
|
||||
}
|
||||
}
|
||||
|
||||
func TestLockfileIsExpired_UnreadableWithMaxLock(t *testing.T) {
|
||||
b := newTestLockfileBisyncRun(t, "not json!!!", fs.Duration(5*time.Minute))
|
||||
assert.True(t, b.lockFileIsExpired(), "unreadable lockfile with --max-lock set should be treated as expired")
|
||||
}
|
||||
|
||||
func TestLockfileIsExpired_UnreadableWithoutMaxLock(t *testing.T) {
|
||||
b := newTestLockfileBisyncRun(t, "not json!!!", basicallyforever)
|
||||
assert.False(t, b.lockFileIsExpired(), "unreadable lockfile without --max-lock should not be treated as expired")
|
||||
}
|
||||
|
||||
func TestLockfileIsExpired_ValidExpired(t *testing.T) {
|
||||
data := struct {
|
||||
Session string
|
||||
PID string
|
||||
TimeRenewed time.Time
|
||||
TimeExpires time.Time
|
||||
}{
|
||||
Session: "test",
|
||||
PID: "12345",
|
||||
TimeRenewed: time.Now().Add(-10 * time.Minute),
|
||||
TimeExpires: time.Now().Add(-5 * time.Minute),
|
||||
}
|
||||
content, err := json.Marshal(data)
|
||||
require.NoError(t, err)
|
||||
|
||||
b := newTestLockfileBisyncRun(t, string(content), fs.Duration(5*time.Minute))
|
||||
assert.True(t, b.lockFileIsExpired(), "valid lockfile with past expiry should be expired")
|
||||
}
|
||||
|
||||
func TestLockfileIsExpired_ValidNotExpired(t *testing.T) {
|
||||
data := struct {
|
||||
Session string
|
||||
PID string
|
||||
TimeRenewed time.Time
|
||||
TimeExpires time.Time
|
||||
}{
|
||||
Session: "test",
|
||||
PID: "12345",
|
||||
TimeRenewed: time.Now(),
|
||||
TimeExpires: time.Now().Add(10 * time.Minute),
|
||||
}
|
||||
content, err := json.Marshal(data)
|
||||
require.NoError(t, err)
|
||||
|
||||
b := newTestLockfileBisyncRun(t, string(content), fs.Duration(5*time.Minute))
|
||||
assert.False(t, b.lockFileIsExpired(), "valid lockfile with future expiry should not be expired")
|
||||
}
|
||||
+124
-25
@@ -1,16 +1,22 @@
|
||||
//go:generate go run help.go rc.md
|
||||
package bisync
|
||||
|
||||
import (
|
||||
"context"
|
||||
_ "embed"
|
||||
"errors"
|
||||
"log"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/rclone/rclone/cmd/bisync/bilib"
|
||||
"github.com/rclone/rclone/fs"
|
||||
fslog "github.com/rclone/rclone/fs/log"
|
||||
"github.com/rclone/rclone/fs/rc"
|
||||
)
|
||||
|
||||
func init() {
|
||||
func addRC() {
|
||||
rc.Add(rc.Call{
|
||||
Path: "sync/bisync",
|
||||
AuthRequired: true,
|
||||
@@ -20,6 +26,42 @@ func init() {
|
||||
})
|
||||
}
|
||||
|
||||
//go:embed rc.md
|
||||
var rcHelp string
|
||||
|
||||
var shortHelp = `Perform bidirectional synchronization between two paths.`
|
||||
|
||||
var longHelp = shortHelp + MakeHelp(`
|
||||
|
||||
[Bisync](https://rclone.org/bisync/) provides a
|
||||
bidirectional cloud sync solution in rclone.
|
||||
It retains the Path1 and Path2 filesystem listings from the prior run.
|
||||
On each successive run it will:
|
||||
|
||||
- list files on Path1 and Path2, and check for changes on each side.
|
||||
Changes include ||New||, ||Newer||, ||Older||, and ||Deleted|| files.
|
||||
- Propagate changes on Path1 to Path2, and vice-versa.
|
||||
|
||||
Bisync is considered an **advanced command**, so use with care.
|
||||
Make sure you have read and understood the entire [manual](https://rclone.org/bisync)
|
||||
(especially the [Limitations](https://rclone.org/bisync/#limitations) section)
|
||||
before using, or data loss can result. Questions can be asked in the
|
||||
[Rclone Forum](https://forum.rclone.org/).
|
||||
|
||||
See [full bisync description](https://rclone.org/bisync/) for details.
|
||||
`)
|
||||
|
||||
// MakeHelp replaces some dynamic variables for the help docs
|
||||
func MakeHelp(help string) string {
|
||||
replacer := strings.NewReplacer(
|
||||
"||", "`",
|
||||
"{MAXDELETE}", strconv.Itoa(DefaultMaxDelete),
|
||||
"{CHECKFILE}", DefaultCheckFilename,
|
||||
"{WORKDIR}", DefaultWorkdir,
|
||||
)
|
||||
return replacer.Replace(help)
|
||||
}
|
||||
|
||||
func rcBisync(ctx context.Context, in rc.Params) (out rc.Params, err error) {
|
||||
opt := &Options{}
|
||||
octx, ci := fs.AddConfig(ctx)
|
||||
@@ -41,56 +83,86 @@ func rcBisync(ctx context.Context, in rc.Params) (out rc.Params, err error) {
|
||||
}
|
||||
|
||||
if opt.Resync, err = in.GetBool("resync"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("resync", "optional parameter is missing. using default value: %v", opt.Resync)
|
||||
}
|
||||
if opt.CheckAccess, err = in.GetBool("checkAccess"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("checkAccess", "optional parameter is missing. using default value: %v", opt.CheckAccess)
|
||||
}
|
||||
if opt.Force, err = in.GetBool("force"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("force", "optional parameter is missing. using default value: %v", opt.Force)
|
||||
}
|
||||
if opt.CreateEmptySrcDirs, err = in.GetBool("createEmptySrcDirs"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("createEmptySrcDirs", "optional parameter is missing. using default value: %v", opt.CreateEmptySrcDirs)
|
||||
}
|
||||
if opt.RemoveEmptyDirs, err = in.GetBool("removeEmptyDirs"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("removeEmptyDirs", "optional parameter is missing. using default value: %v", opt.RemoveEmptyDirs)
|
||||
}
|
||||
if opt.NoCleanup, err = in.GetBool("noCleanup"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("noCleanup", "optional parameter is missing. using default value: %v", opt.NoCleanup)
|
||||
}
|
||||
if opt.IgnoreListingChecksum, err = in.GetBool("ignoreListingChecksum"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("ignoreListingChecksum", "optional parameter is missing. using default value: %v", opt.IgnoreListingChecksum)
|
||||
}
|
||||
if opt.Resilient, err = in.GetBool("resilient"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("resilient", "optional parameter is missing. using default value: %v", opt.Resilient)
|
||||
}
|
||||
|
||||
if opt.CheckFilename, err = in.GetString("checkFilename"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
opt.CheckFilename = DefaultCheckFilename
|
||||
fs.Debugf("checkFilename", "optional parameter is missing. using default value: %v", opt.CheckFilename)
|
||||
}
|
||||
if opt.FiltersFile, err = in.GetString("filtersFile"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
fs.Debugf("filtersFile", "optional parameter is missing. using default value: %v", opt.FiltersFile)
|
||||
}
|
||||
if opt.Workdir, err = in.GetString("workdir"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
// "" sets correct default later
|
||||
fs.Debugf("workdir", "optional parameter is missing. using default value: %v", opt.Workdir)
|
||||
}
|
||||
if opt.BackupDir1, err = in.GetString("backupdir1"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
if opt.BackupDir1, err = in.GetString("backupDir1"); rc.NotErrParamNotFound(err) {
|
||||
// we accept an alternate capitalization here for backward compatibility.
|
||||
if opt.BackupDir1, err = in.GetString("backupdir1"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("backupDir1", "optional parameter is missing. using default value: %v", opt.BackupDir1)
|
||||
}
|
||||
}
|
||||
if opt.BackupDir2, err = in.GetString("backupdir2"); rc.NotErrParamNotFound(err) {
|
||||
return
|
||||
if opt.BackupDir2, err = in.GetString("backupDir2"); rc.NotErrParamNotFound(err) {
|
||||
// we accept an alternate capitalization here for backward compatibility.
|
||||
if opt.BackupDir2, err = in.GetString("backupdir2"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("backupDir2", "optional parameter is missing. using default value: %v", opt.BackupDir2)
|
||||
}
|
||||
}
|
||||
|
||||
checkSync, err := in.GetString("checkSync")
|
||||
if rc.NotErrParamNotFound(err) {
|
||||
if err = setEnum(in, "checkSync", "true", opt.CheckSync.Set); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if checkSync == "" {
|
||||
checkSync = "true"
|
||||
}
|
||||
if err := opt.CheckSync.Set(checkSync); err != nil {
|
||||
if err = setEnum(in, "resyncMode", opt.ResyncMode.String(), opt.ResyncMode.Set); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = setEnum(in, "conflictResolve", opt.ConflictResolve.String(), opt.ConflictResolve.Set); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = setEnum(in, "conflictLoser", opt.ConflictLoser.String(), opt.ConflictLoser.Set); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if opt.ConflictSuffixFlag, err = in.GetString("conflictSuffix"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("conflictSuffix", "optional parameter is missing. using default value: %v", opt.ConflictSuffixFlag)
|
||||
}
|
||||
if opt.Recover, err = in.GetBool("recover"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("recover", "optional parameter is missing. using default value: %v", opt.Recover)
|
||||
}
|
||||
if opt.CompareFlag, err = in.GetString("compare"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("compare", "optional parameter is missing. using default value: %v", opt.CompareFlag)
|
||||
}
|
||||
if opt.Compare.NoSlowHash, err = in.GetBool("noSlowHash"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("noSlowHash", "optional parameter is missing. using default value: %v", opt.Compare.NoSlowHash)
|
||||
}
|
||||
if opt.Compare.SlowHashSyncOnly, err = in.GetBool("slowHashSyncOnly"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("slowHashSyncOnly", "optional parameter is missing. using default value: %v", opt.Compare.SlowHashSyncOnly)
|
||||
}
|
||||
if opt.Compare.DownloadHash, err = in.GetBool("downloadHash"); rc.NotErrParamNotFound(err) {
|
||||
fs.Debugf("downloadHash", "optional parameter is missing. using default value: %v", opt.Compare.DownloadHash)
|
||||
}
|
||||
if opt.MaxLock, err = in.GetFsDuration("maxLock"); rc.NotErrParamNotFound(err) {
|
||||
opt.MaxLock = 0
|
||||
fs.Debugf("maxLock", "optional parameter is missing. using default value: %v", opt.MaxLock)
|
||||
}
|
||||
|
||||
fs1, err := rc.GetFsNamed(octx, in, "path1")
|
||||
if err != nil {
|
||||
@@ -105,6 +177,33 @@ func rcBisync(ctx context.Context, in rc.Params) (out rc.Params, err error) {
|
||||
output := bilib.CaptureOutput(func() {
|
||||
err = Bisync(octx, fs1, fs2, opt)
|
||||
})
|
||||
|
||||
workDir, _ := filepath.Abs(DefaultWorkdir)
|
||||
if opt.Workdir != "" {
|
||||
workDir, _ = filepath.Abs(opt.Workdir)
|
||||
}
|
||||
basePath := bilib.BasePath(ctx, workDir, fs1, fs2)
|
||||
|
||||
_, _ = log.Writer().Write(output)
|
||||
return rc.Params{"output": string(output)}, err
|
||||
return rc.Params{
|
||||
"output": string(output),
|
||||
"session": bilib.SessionName(fs1, fs2),
|
||||
"workDir": workDir,
|
||||
"basePath": basePath,
|
||||
"listing1": basePath + ".path1.lst",
|
||||
"listing2": basePath + ".path2.lst",
|
||||
"logFile": fslog.Opt.File,
|
||||
}, err
|
||||
}
|
||||
|
||||
func setEnum(in rc.Params, name string, defaultVal string, set func(s string) error) error {
|
||||
v, err := in.GetString(name)
|
||||
if rc.NotErrParamNotFound(err) || v == "" {
|
||||
v = defaultVal
|
||||
fs.Debugf(name, "optional parameter is missing. using default value: %v", v)
|
||||
}
|
||||
if err := set(v); err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
<!--- Docs generated by help.go - use go generate to rebuild - DO NOT EDIT --->
|
||||
|
||||
This takes the following parameters:
|
||||
|
||||
- path1 (required) - (string) a remote directory string e.g. `drive:path1`
|
||||
- path2 (required) - (string) a remote directory string e.g. `drive:path2`
|
||||
- dryRun - (bool) dry-run mode
|
||||
- backupDir1 - (string) --backup-dir for Path1. Must be a non-overlapping path on
|
||||
the same remote.
|
||||
- backupDir2 - (string) --backup-dir for Path2. Must be a non-overlapping path on
|
||||
the same remote.
|
||||
- checkAccess - (bool) Ensure expected RCLONE_TEST files are found on both
|
||||
Path1 and Path2 filesystems, else abort.
|
||||
- checkFilename - (string) Filename for --check-access (default: RCLONE_TEST)
|
||||
- checkSync - (string) Controls comparison of final listings: true|false|only
|
||||
(default: true)
|
||||
- compare - (string) Comma-separated list of bisync-specific compare options ex.
|
||||
'size,modtime,checksum' (default: 'size,modtime')
|
||||
- conflictLoser - (ConflictLoserAction) Action to take on the loser of a sync
|
||||
conflict (when there is a winner) or on both files (when there is no
|
||||
winner): , num, pathname, delete (default: num)
|
||||
- conflictResolve - (string) Automatically resolve conflicts by preferring the
|
||||
version that is: none, path1, path2, newer, older, larger, smaller (default:
|
||||
none)
|
||||
- conflictSuffix - (string) Suffix to use when renaming a --conflict-loser. Can
|
||||
be either one string or two comma-separated strings to assign different
|
||||
suffixes to Path1/Path2. (default: 'conflict')
|
||||
- createEmptySrcDirs - (bool) Sync creation and deletion of empty directories.
|
||||
(Not compatible with --remove-empty-dirs)
|
||||
- downloadHash - (bool) Compute hash by downloading when otherwise
|
||||
unavailable. (warning: may be slow and use lots of data!)
|
||||
- filtersFile - (string) Read filtering patterns from a file
|
||||
- force - (bool) Bypass --max-delete safety check and run the sync. Consider
|
||||
using with --verbose
|
||||
- ignoreListingChecksum - (bool) Do not use checksums for listings (add --ignore-
|
||||
checksum to additionally skip post-copy checksum checks)
|
||||
- maxLock - (Duration) Consider lock files older than this to be expired
|
||||
(default: 0 (never expire)) (minimum: 2m)
|
||||
- noCleanup - (bool) Retain working files (useful for troubleshooting and
|
||||
testing).
|
||||
- noSlowHash - (bool) Ignore listing checksums only on backends where they are
|
||||
slow
|
||||
- recover - (bool) Automatically recover from interruptions without requiring --
|
||||
resync.
|
||||
- removeEmptyDirs - (bool) Remove ALL empty directories at the final cleanup
|
||||
step.
|
||||
- resilient - (bool) Allow future runs to retry after certain less-serious
|
||||
errors, instead of requiring --resync.
|
||||
- resync - (bool) Performs the resync run. Equivalent to --resync-mode path1.
|
||||
Consider using --verbose or --dry-run first.
|
||||
- resyncMode - (string) During resync, prefer the version that is: path1,
|
||||
path2, newer, older, larger, smaller (default: path1 if --resync, otherwise
|
||||
none for no resync.)
|
||||
- slowHashSyncOnly - (bool) Ignore slow checksums for listings and deltas, but
|
||||
still consider them during sync calls.
|
||||
- workdir - (string) Use custom working dir - useful for testing. (default:
|
||||
~/.cache/rclone/bisync)
|
||||
|
||||
See [bisync command help](https://rclone.org/commands/rclone_bisync/)
|
||||
and [full bisync description](https://rclone.org/bisync/)
|
||||
for more information.
|
||||
+16
-14
@@ -1,7 +1,8 @@
|
||||
Rclone @ allows Linux, FreeBSD, macOS and Windows to
|
||||
mount any of Rclone's cloud storage systems as a file system with FUSE.
|
||||
|
||||
First set up your remote using `rclone config`. Check it works with `rclone ls` etc.
|
||||
First set up your remote using `rclone config`. Check it works with `rclone ls`
|
||||
etc.
|
||||
|
||||
On Linux and macOS, you can run mount in either foreground or background (aka
|
||||
daemon) mode. Mount runs in foreground mode by default. Use the `--daemon` flag
|
||||
@@ -29,8 +30,8 @@ The following examples will mount to an automatically assigned drive,
|
||||
to specific drive letter `X:`, to path `C:\path\parent\mount`
|
||||
(where parent directory or drive must exist, and mount must **not** exist,
|
||||
and is not supported when [mounting as a network drive](#mounting-modes-on-windows)),
|
||||
and the last example will mount as network share `\\cloud\remote` and map it to an
|
||||
automatically assigned drive:
|
||||
and the last example will mount as network share `\\cloud\remote` and map it to
|
||||
an automatically assigned drive:
|
||||
|
||||
```console
|
||||
rclone @ remote:path/to/files *
|
||||
@@ -136,8 +137,8 @@ rclone @ remote:path/to/files X: --volname \\server\share
|
||||
|
||||
You may also specify the network share UNC path as the mountpoint itself. Then rclone
|
||||
will automatically assign a drive letter, same as with `*` and use that as
|
||||
mountpoint, and instead use the UNC path specified as the volume name, as if it were
|
||||
specified with the `--volname` option. This will also implicitly set
|
||||
mountpoint, and instead use the UNC path specified as the volume name, as if it
|
||||
were specified with the `--volname` option. This will also implicitly set
|
||||
the `--network-mode` option. This means the following two examples have same result:
|
||||
|
||||
```console
|
||||
@@ -268,7 +269,7 @@ does not suffer from the same limitations.
|
||||
|
||||
Mounting on macOS can be done either via [built-in NFS server](/commands/rclone_serve_nfs/),
|
||||
[macFUSE](https://osxfuse.github.io/) (also known as osxfuse) or
|
||||
[FUSE-T](https://www.fuse-t.org/).macFUSE is a traditional FUSE driver utilizing
|
||||
[FUSE-T](https://www.fuse-t.org/). macFUSE is a traditional FUSE driver utilizing
|
||||
a macOS kernel extension (kext). FUSE-T is an alternative FUSE system which
|
||||
"mounts" via an NFSv4 local server.
|
||||
|
||||
@@ -311,13 +312,13 @@ current as of FUSE-T version 1.0.14.
|
||||
|
||||
As per the [FUSE-T wiki](https://github.com/macos-fuse-t/fuse-t/wiki#caveats):
|
||||
|
||||
> File access and modification times cannot be set separately as it seems to be an
|
||||
> issue with the NFS client which always modifies both. Can be reproduced with
|
||||
> 'touch -m' and 'touch -a' commands
|
||||
> File access and modification times cannot be set separately as it seems to be
|
||||
> an issue with the NFS client which always modifies both. Can be reproduced
|
||||
> with `touch -m` and `touch -a` commands
|
||||
|
||||
This means that viewing files with various tools, notably macOS Finder, will cause
|
||||
rlcone to update the modification time of the file. This may make rclone upload a
|
||||
full new copy of the file.
|
||||
This means that viewing files with various tools, notably macOS Finder, will
|
||||
cause rlcone to update the modification time of the file. This may make rclone
|
||||
upload a full new copy of the file.
|
||||
|
||||
##### Read Only mounts
|
||||
|
||||
@@ -331,9 +332,10 @@ On newer versions of Ubuntu, you may encounter the following error when running
|
||||
|
||||
> NOTICE: mount helper error: fusermount3: mount failed: Permission denied
|
||||
> CRITICAL: Fatal error: failed to mount FUSE fs: fusermount: exit status 1
|
||||
|
||||
This may be due to newer [Apparmor](https://wiki.ubuntu.com/AppArmor) restrictions,
|
||||
which can be disabled with `sudo aa-disable /usr/bin/fusermount3` (you may need to
|
||||
`sudo apt install apparmor-utils` beforehand).
|
||||
which can be disabled with `sudo aa-disable /usr/bin/fusermount3` (you may need
|
||||
to `sudo apt install apparmor-utils` beforehand).
|
||||
|
||||
### Limitations
|
||||
|
||||
|
||||
+1
-1
@@ -34,7 +34,7 @@
|
||||
}
|
||||
},
|
||||
"highlight": {
|
||||
"style": "monokailight"
|
||||
"noClasses": false
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -8,7 +8,7 @@ notoc: true
|
||||
# Rclone syncs your files to cloud storage
|
||||
|
||||
<!-- markdownlint-disable-next-line line-length -->
|
||||
{{< img width="50%" src="/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" >}}
|
||||
{{< img width="50%" src="/img/logo_on_light__horizontal_color.svg" dark-src="/img/logo_on_dark__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" class="hide-mobile" >}}
|
||||
|
||||
- [About rclone](#about)
|
||||
- [What can rclone do for you?](#what)
|
||||
@@ -216,6 +216,7 @@ WebDAV or S3, that work out of the box.)
|
||||
{{< provider name="SugarSync" home="https://sugarsync.com/" config="/sugarsync/" >}}
|
||||
{{< provider name="Tencent Cloud Object Storage (COS)" home="https://intl.cloud.tencent.com/product/cos" config="/s3/#tencent-cos" >}}
|
||||
{{< provider name="Uloz.to" home="https://uloz.to" config="/ulozto/" >}}
|
||||
{{< provider name="US3" home="https://www.ucloud.cn/site/product/ufile.html" config="/s3/#us3" >}}
|
||||
{{< provider name="Wasabi" home="https://wasabi.com/" config="/s3/#wasabi" >}}
|
||||
{{< provider name="WebDAV" home="https://en.wikipedia.org/wiki/WebDAV" config="/webdav/" >}}
|
||||
{{< provider name="Yandex Disk" home="https://disk.yandex.com/" config="/yandex/" >}}
|
||||
@@ -248,7 +249,7 @@ These backends adapt or modify other storage providers:
|
||||
|
||||
## Links
|
||||
|
||||
- {{< icon "fa fa-home fa-fw" >}} [Home page](https://rclone.org/)
|
||||
- {{< icon "fab fa-github fa-fw" >}} [GitHub project page for source and bug tracker](https://github.com/rclone/rclone)
|
||||
- {{< icon "fa fa-comments fa-fw" >}} [Rclone Forum](https://forum.rclone.org)
|
||||
- {{< icon "fas fa-cloud-download-alt fa-fw" >}}[Downloads](/downloads/)
|
||||
- [Home page](https://rclone.org/)
|
||||
- [GitHub project page for source and bug tracker](https://github.com/rclone/rclone)
|
||||
- [Rclone Forum](https://forum.rclone.org)
|
||||
- [Downloads](/downloads/)
|
||||
@@ -4,7 +4,7 @@ description: "Remote Aliases"
|
||||
versionIntroduced: "v1.40"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-link" >}} Alias
|
||||
# Alias
|
||||
|
||||
The `alias` remote provides a new name for another remote.
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ versionIntroduced: "v1.20"
|
||||
status: Removed
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-amazon" >}} Amazon Drive
|
||||
# Amazon Drive
|
||||
|
||||
Amazon Drive, formerly known as Amazon Cloud Drive, was a cloud storage
|
||||
service run by Amazon for consumers.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Archive Remote"
|
||||
versionIntroduced: "v1.72"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-archive" >}} Archive
|
||||
# Archive
|
||||
|
||||
The Archive backend allows read only access to the content of archive
|
||||
files on cloud storage without downloading the complete archive. This
|
||||
|
||||
@@ -1094,3 +1094,18 @@ put them back in again. -->
|
||||
- Bjoern Franke <bjo@schafweide.org>
|
||||
- razorloves <razorloves@gmail.com>
|
||||
- Duncan F <131309315+duncanaf@users.noreply.github.com>
|
||||
- Billy Hughes <w.g.h.hughes@gmail.com>
|
||||
- Bhagyashreek8 <57988406+Bhagyashreek8@users.noreply.github.com>
|
||||
- Andriy Senyshyn <135705861+sam-mt@users.noreply.github.com>
|
||||
- Andrew Furman <7343697+furmandev@users.noreply.github.com>
|
||||
- ZRHan <56144550+ZRHann@users.noreply.github.com>
|
||||
- Jason <jasonbabo@gmail.com>
|
||||
- Patrick Farrell <pfarrell@whamcloud.com>
|
||||
- BizaNator <17164422+BizaNator@users.noreply.github.com>
|
||||
- lif <1835304752@qq.com>
|
||||
- jinkeyuu <44105232+jinkeyuu@users.noreply.github.com>
|
||||
- jinyu.han <jinyu.han@ucloud.cn>
|
||||
- Claude Opus 4.6 <noreply@anthropic.com>
|
||||
- Andriy Senyshyn <andriy.senyshyn@kiteworks.com>
|
||||
- Mike GIllan <mike.gillan@gmail.com>
|
||||
- Xiangzhe <xiangzhedev@gmail.com>
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Microsoft Azure Blob Storage"
|
||||
versionIntroduced: "v1.38"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-windows" >}} Microsoft Azure Blob Storage
|
||||
# Microsoft Azure Blob Storage
|
||||
|
||||
Paths are specified as `remote:container` (or `remote:` for the `lsd`
|
||||
command.) You may put subdirectories in too, e.g.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Microsoft Azure Files Storage"
|
||||
versionIntroduced: "v1.65"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-windows" >}} Microsoft Azure Files Storage
|
||||
# Microsoft Azure Files Storage
|
||||
|
||||
Paths are specified as `remote:` You may put subdirectories in too,
|
||||
e.g. `remote:path/to/dir`.
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ description: "Backblaze B2"
|
||||
versionIntroduced: "v1.26"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-fire" >}} Backblaze B2
|
||||
# Backblaze B2
|
||||
|
||||
B2 is [Backblaze's cloud storage system](https://www.backblaze.com/cloud-storage).
|
||||
|
||||
|
||||
@@ -820,6 +820,12 @@ running -- and you can therefore be reasonably sure that any *expired* lock
|
||||
file you may find was left there by an interrupted run, not one that is still
|
||||
running and just taking awhile.
|
||||
|
||||
If a lock file exists but its contents are unreadable (for example, due to an
|
||||
incomplete write or disk error), it is treated as expired when `--max-lock` is
|
||||
set to a value greater than `0`. If `--max-lock` is `0` or not set, an
|
||||
unreadable lock file will produce an error and block future runs until removed
|
||||
manually.
|
||||
|
||||
If `--max-lock` is `0` or not set, the default is that lock files will never
|
||||
expire, and will block future runs (of these same two bisync paths)
|
||||
indefinitely.
|
||||
@@ -1897,6 +1903,12 @@ about *Unison* and synchronization in general.
|
||||
|
||||
## Changelog
|
||||
|
||||
### `v1.74`
|
||||
|
||||
- Added several missing `rc` parameters.
|
||||
- Optional `rc` parameters are now truly optional.
|
||||
- `rc` output now provides more structured information.
|
||||
|
||||
### `v1.71`
|
||||
|
||||
- `bisync` is now officially released from beta.
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ description: "Rclone docs for Box"
|
||||
versionIntroduced: "v1.38"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-archive" >}} Box
|
||||
# Box
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for cache remote"
|
||||
versionIntroduced: "v1.39"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-archive" >}} Cache
|
||||
# Cache
|
||||
|
||||
The `cache` remote wraps another existing remote and stores file structure
|
||||
and its data for long running tasks like `rclone mount`.
|
||||
|
||||
@@ -6,6 +6,47 @@ description: "Rclone Changelog"
|
||||
|
||||
# Changelog
|
||||
|
||||
## v1.73.3 - 2026-03-23
|
||||
|
||||
[See commits](https://github.com/rclone/rclone/compare/v1.73.2...v1.73.3)
|
||||
|
||||
- Bug Fixes
|
||||
- build
|
||||
- Update to google.golang.org/grpc 1.79.3 to fix CVE-2026-33186 (dependabot[bot])
|
||||
- Update to github.com/buger/jsonparser 1.1.2 to fix GHSA-6g7g-w4f8-9c9x (dependabot[bot])
|
||||
- doc fixes
|
||||
- Added text to the label showing version-introduced info (Jan-Philipp Reßler)
|
||||
- Clarify Filen password change requires updating both password and API key in rclone config (Jason)
|
||||
- s3: clarify multi tenant support for Cubbit (Marco Ferretti)
|
||||
- jottacloud: Fix broken link (albertony)
|
||||
- lib/rest: Fix URLPathEscapeAll breaking WebDAV servers (eg nzbdav) with strict path matching (Andrew Furman)
|
||||
- list: Fix nil pointer panic in Sorter when temp file creation fails (Nick Craig-Wood)
|
||||
|
||||
## v1.73.2 - 2026-03-06
|
||||
|
||||
[See commits](https://github.com/rclone/rclone/compare/v1.73.1...v1.73.2)
|
||||
|
||||
- Bug Fixes
|
||||
- build
|
||||
- Update to go 1.25.8 to fix CVE-2026-27137 CVE-2026-27138 CVE-2026-25679 CVE-2026-27142 (Nick Craig-Wood)
|
||||
- Update github.com/cloudflare/circl to v1.6.3 to fix CVE-2026-1229 (Nick Craig-Wood)
|
||||
- Update to golang.org/x/net v0.51.0 to fix CVE-2026-27141 (Nick Craig-Wood)
|
||||
- docs fixes:
|
||||
- bisync: Add group Sync to the bisync command (Jan-Philipp Reßler)
|
||||
- Note that --use-server-modtime only works on some backends (Nick Craig-Wood)
|
||||
- Document unsupported S3 object keys with double slashes (Adam Kasztenny)
|
||||
- Fix headers hierarchy for mount.md (Dark Dragon)
|
||||
- Fix new drive flag typo in changelog (razorloves)
|
||||
- Archive
|
||||
- Extract: fix extraction with "./" prefix from tar entry paths (Varun Chawla)
|
||||
- Drime
|
||||
- Fix chunk-uploaded files ignoring workspace ID (a1pcm)
|
||||
- Internxt
|
||||
- Fix Entry doesn't belong in directory errors on windows (jzunigax2)
|
||||
- WebDAV
|
||||
- Escape reserved characters in URL path segments (Varun Chawla)
|
||||
- Add missing headers for CORS (Romāns Potašovs)
|
||||
|
||||
## v1.73.1 - 2026-02-17
|
||||
|
||||
[See commits](https://github.com/rclone/rclone/compare/v1.73.0...v1.73.1)
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Split-chunking overlay remote"
|
||||
versionIntroduced: "v1.50"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cut" >}} Chunker
|
||||
# Chunker
|
||||
|
||||
The `chunker` overlay transparently splits large files into smaller chunks
|
||||
during upload to wrapped remote and transparently assembles them back
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Cloudinary backend"
|
||||
versionIntroduced: "v1.69"
|
||||
|
||||
---
|
||||
# {{< icon "fa fa-cloud" >}} Cloudinary
|
||||
# Cloudinary
|
||||
|
||||
This is a backend for the [Cloudinary](https://cloudinary.com/) platform
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Combine several remotes into one"
|
||||
versionIntroduced: "v1.59"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-folder-plus" >}} Combine
|
||||
# Combine
|
||||
|
||||
The `combine` backend joins remotes together into a single directory
|
||||
tree.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Compression Remote"
|
||||
versionIntroduced: "v1.54"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-compress" >}} Compress
|
||||
# Compress
|
||||
|
||||
## Warning
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Encryption overlay remote"
|
||||
versionIntroduced: "v1.33"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-lock" >}} Crypt
|
||||
# Crypt
|
||||
|
||||
Rclone `crypt` remotes encrypt and decrypt other remotes.
|
||||
|
||||
|
||||
+18
-17
@@ -753,21 +753,21 @@ object also.
|
||||
Here is a table of standard system metadata which, if appropriate, a
|
||||
backend may implement.
|
||||
|
||||
| key | description | example |
|
||||
|---------------------|-------------|---------|
|
||||
| mode | File type and mode: octal, unix style | 0100664 |
|
||||
| uid | User ID of owner: decimal number | 500 |
|
||||
| gid | Group ID of owner: decimal number | 500 |
|
||||
| rdev | Device ID (if special file) => hexadecimal | 0 |
|
||||
| atime | Time of last access: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| mtime | Time of last modification: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| btime | Time of file creation (birth): RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| utime | Time of file upload: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| cache-control | Cache-Control header | no-cache |
|
||||
| key | description | example |
|
||||
| --- | ----------- | ------- |
|
||||
| mode | File type and mode: octal, unix style | 0100664 |
|
||||
| uid | User ID of owner: decimal number | 500 |
|
||||
| gid | Group ID of owner: decimal number | 500 |
|
||||
| rdev | Device ID (if special file) => hexadecimal | 0 |
|
||||
| atime | Time of last access: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| mtime | Time of last modification: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| btime | Time of file creation (birth): RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| utime | Time of file upload: RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 |
|
||||
| cache-control | Cache-Control header | no-cache |
|
||||
| content-disposition | Content-Disposition header | inline |
|
||||
| content-encoding | Content-Encoding header | gzip |
|
||||
| content-language | Content-Language header | en-US |
|
||||
| content-type | Content-Type header | text/plain |
|
||||
| content-encoding | Content-Encoding header | gzip |
|
||||
| content-language | Content-Language header | en-US |
|
||||
| content-type | Content-Type header | text/plain |
|
||||
|
||||
The metadata keys `mtime` and `content-type` will take precedence if
|
||||
supplied in the metadata over reading the `Content-Type` or
|
||||
@@ -1193,7 +1193,8 @@ on any OS, and the value is defined as following:
|
||||
|
||||
- On Windows: `%HOME%` if defined, else `%USERPROFILE%`, or else `%HOMEDRIVE%\%HOMEPATH%`.
|
||||
- On Unix: `$HOME` if defined, else by looking up current user in OS-specific user
|
||||
database (e.g. passwd file), or else use the result from shell command `cd && pwd`.
|
||||
database (e.g. passwd file), or else use the result from shell command
|
||||
`cd && pwd`.
|
||||
|
||||
If you run `rclone config file` you will see where the default location is for
|
||||
you. Running `rclone config touch` will ensure a configuration file exists,
|
||||
@@ -2059,7 +2060,7 @@ This is the default.
|
||||
`CAUTIOUS` will try to prevent rclone from reaching the limit. Only applicable
|
||||
for `--max-transfer`.
|
||||
|
||||
## -M, --metadata
|
||||
### -M, --metadata
|
||||
|
||||
Setting this flag enables rclone to copy the metadata from the source
|
||||
to the destination. For local backends this is ownership, permissions,
|
||||
@@ -3444,7 +3445,7 @@ many items, the input is treated as a [CSV encoded](https://godoc.org/encoding/c
|
||||
string. For example
|
||||
|
||||
| Environment variable | Equivalent options |
|
||||
|----------------------|--------------------|
|
||||
| -------------------- | ------------------ |
|
||||
| `RCLONE_EXCLUDE="*.jpg"` | `--exclude "*.jpg"` |
|
||||
| `RCLONE_EXCLUDE="*.jpg,*.png"` | `--exclude "*.jpg"` `--exclude "*.png"` |
|
||||
| `RCLONE_EXCLUDE='"*.jpg","*.png"'` | `--exclude "*.jpg"` `--exclude "*.png"` |
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ description: "Rclone docs for DOI"
|
||||
versionIntroduced: "v1.70"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-building-columns" >}} DOI
|
||||
# DOI
|
||||
|
||||
The DOI remote is a read only remote for reading files from digital object
|
||||
identifiers (DOI).
|
||||
|
||||
@@ -16,7 +16,7 @@ image](https://securebuild.com/images/rclone) through our partner
|
||||
## Release {{% version %}} OS requirements {#osrequirements}
|
||||
|
||||
| OS | Minimum Version |
|
||||
|:-------:|:-------:|
|
||||
| :---: | :---: |
|
||||
| Linux | Kernel 3.2 |
|
||||
| macOS | 12 (Monterey) |
|
||||
| Windows | 10, Server 2016 |
|
||||
@@ -31,7 +31,7 @@ in the Go Wiki.
|
||||
## Release {{% version %}} {#release}
|
||||
|
||||
| Arch-OS | Windows | macOS | Linux | .deb | .rpm | FreeBSD | NetBSD | OpenBSD | Plan9 | Solaris |
|
||||
|:-------:|:-------:|:-----:|:-----:|:----:|:----:|:-------:|:------:|:-------:|:-----:|:-------:|
|
||||
| :-----: | :-----: | :---: | :---: | :--: | :--: | :-----: | :----: | :-----: | :---: | :-----: |
|
||||
| Intel/AMD - 64 Bit | {{< download windows amd64 >}} | {{< download osx amd64 >}} | {{< download linux amd64 >}} | {{< download linux amd64 deb >}} | {{< download linux amd64 rpm >}} | {{< download freebsd amd64 >}} | {{< download netbsd amd64 >}} | {{< download openbsd amd64 >}} | {{< download plan9 amd64 >}} | {{< download solaris amd64 >}} |
|
||||
| Intel/AMD - 32 Bit | {{< download windows 386 >}} | - | {{< download linux 386 >}} | {{< download linux 386 deb >}} | {{< download linux 386 rpm >}} | {{< download freebsd 386 >}} | {{< download netbsd 386 >}} | {{< download openbsd 386 >}} | {{< download plan9 386 >}} | - |
|
||||
| ARMv5 - 32 Bit NOHF | - | - | {{< download linux arm >}} | {{< download linux arm deb >}} | {{< download linux arm rpm >}} | {{< download freebsd arm >}} | {{< download netbsd arm >}} | - | - | - |
|
||||
@@ -120,7 +120,7 @@ If you would like to download the current version (maybe from a
|
||||
script) from a URL which doesn't change then you can use these links.
|
||||
|
||||
| Arch-OS | Windows | macOS | Linux | .deb | .rpm | FreeBSD | NetBSD | OpenBSD | Plan9 | Solaris |
|
||||
|:-------:|:-------:|:-----:|:-----:|:----:|:----:|:-------:|:------:|:-------:|:-----:|:-------:|
|
||||
| :-----: | :-----: | :---: | :---: | :--: | :--: | :-----: | :----: | :-----: | :---: | :-----: |
|
||||
| Intel/AMD - 64 Bit | {{< cdownload windows amd64 >}} | {{< cdownload osx amd64 >}} | {{< cdownload linux amd64 >}} | {{< cdownload linux amd64 deb >}} | {{< cdownload linux amd64 rpm >}} | {{< cdownload freebsd amd64 >}} | {{< cdownload netbsd amd64 >}} | {{< cdownload openbsd amd64 >}} | {{< cdownload plan9 amd64 >}} | {{< cdownload solaris amd64 >}} |
|
||||
| Intel/AMD - 32 Bit | {{< cdownload windows 386 >}} | - | {{< cdownload linux 386 >}} | {{< cdownload linux 386 deb >}} | {{< cdownload linux 386 rpm >}} | {{< cdownload freebsd 386 >}} | {{< cdownload netbsd 386 >}} | {{< cdownload openbsd 386 >}} | {{< cdownload plan9 386 >}} | - |
|
||||
| ARMv5 - 32 Bit NOHF | - | - | {{< cdownload linux arm >}} | {{< cdownload linux arm deb >}} | {{< cdownload linux arm rpm >}} | {{< cdownload freebsd arm >}} | {{< cdownload netbsd arm >}} | - | - | - |
|
||||
@@ -137,12 +137,13 @@ Older downloads can be found at <https://downloads.rclone.org/>
|
||||
The latest `rclone` version working for:
|
||||
|
||||
| OS | Maximum rclone version |
|
||||
|:-------:|:-------:|
|
||||
| :---: | :---: |
|
||||
| Windows 7 | v1.63.1 |
|
||||
| Windows Server 2008 | v1.63.1 |
|
||||
| Windows Server 2012 | v1.63.1 |
|
||||
| Windows XP | v1.42 |
|
||||
| Windows Vista | v1.42 |
|
||||
| macOS 10.15 (Catalina) | v1.70.3 |
|
||||
| macOS 10.14 (Mojave) | v1.63.1 |
|
||||
| macOS 10.13 (High Sierra) | v1.63.1 |
|
||||
| macOS 10.12 (Sierra) | v1.56.0 |
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Drime"
|
||||
versionIntroduced: "v1.73"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} Drime
|
||||
# Drime
|
||||
|
||||
[Drime](https://drime.cloud/) is a cloud storage and transfer service focused
|
||||
on fast, resilient file delivery. It offers both free and paid tiers with
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Google drive"
|
||||
versionIntroduced: "v0.91"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-google" >}} Google Drive
|
||||
# Google Drive
|
||||
|
||||
Paths are specified as `drive:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Dropbox"
|
||||
versionIntroduced: "v1.02"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-dropbox" >}} Dropbox
|
||||
# Dropbox
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for 1Fichier"
|
||||
versionIntroduced: "v1.49"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-archive" >}} 1Fichier
|
||||
# 1Fichier
|
||||
|
||||
This is a backend for the [1fichier](https://1fichier.com) cloud
|
||||
storage service. Note that a Premium subscription is required to use
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for the Enterprise File Fabric backend"
|
||||
versionIntroduced: "v1.54"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} Enterprise File Fabric
|
||||
# Enterprise File Fabric
|
||||
|
||||
This backend supports [Storage Made Easy's Enterprise File
|
||||
Fabric™](https://storagemadeeasy.com/about/) which provides a software
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for FileLu"
|
||||
versionIntroduced: "v1.70"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-folder" >}} FileLu
|
||||
# FileLu
|
||||
|
||||
[FileLu](https://filelu.com/) is a reliable cloud storage provider
|
||||
offering features like secure file uploads, downloads, flexible
|
||||
|
||||
@@ -4,12 +4,14 @@ description: "Rclone docs for Filen"
|
||||
versionIntroduced: "1.73"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-solid fa-f" >}} Filen
|
||||
# Filen
|
||||
## Configuration
|
||||
The initial setup for Filen requires that you get an API key for your account,
|
||||
currently this is only possible using the [Filen CLI](https://github.com/FilenCloudDienste/filen-cli).
|
||||
This means you must first download the CLI, login, and then run the `export-api-key` command.
|
||||
|
||||
**Note:** If you change your Filen account password, you will need to update your rclone configuration with the new password and a re-exported API key (via `export-api-key`).
|
||||
|
||||
Here is an example of how to make a remote called `FilenRemote`. First run:
|
||||
|
||||
rclone config
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Files.com"
|
||||
versionIntroduced: "v1.68"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-brands fa-files-pinwheel" >}} Files.com
|
||||
# Files.com
|
||||
|
||||
[Files.com](https://www.files.com/) is a cloud storage service that provides a
|
||||
secure and easy way to store and share files.
|
||||
|
||||
+22
-22
@@ -202,28 +202,28 @@ them into regular expressions.
|
||||
|
||||
## Filter pattern examples {#examples}
|
||||
|
||||
| Description | Pattern | Matches | Does not match |
|
||||
| ----------- |-------- | ------- | -------------- |
|
||||
| Wildcard | `*.jpg` | `/file.jpg` | `/file.png` |
|
||||
| | | `/dir/file.jpg` | `/dir/file.png` |
|
||||
| Rooted | `/*.jpg` | `/file.jpg` | `/file.png` |
|
||||
| | | `/file2.jpg` | `/dir/file.jpg` |
|
||||
| Alternates | `*.{jpg,png}` | `/file.jpg` | `/file.gif` |
|
||||
| | | `/dir/file.png` | `/dir/file.gif` |
|
||||
| Path Wildcard | `dir/**` | `/dir/anyfile` | `file.png` |
|
||||
| | | `/subdir/dir/subsubdir/anyfile` | `/subdir/file.png` |
|
||||
| Any Char | `*.t?t` | `/file.txt` | `/file.qxt` |
|
||||
| | | `/dir/file.tzt` | `/dir/file.png` |
|
||||
| Range | `*.[a-z]` | `/file.a` | `/file.0` |
|
||||
| | | `/dir/file.b` | `/dir/file.1` |
|
||||
| Escape | `*.\?\?\?` | `/file.???` | `/file.abc` |
|
||||
| | | `/dir/file.???` | `/dir/file.def` |
|
||||
| Class | `*.\d\d\d` | `/file.012` | `/file.abc` |
|
||||
| | | `/dir/file.345` | `/dir/file.def` |
|
||||
| Regexp | `*.{{jpe?g}}` | `/file.jpeg` | `/file.png` |
|
||||
| | | `/dir/file.jpg` | `/dir/file.jpeeg` |
|
||||
| Rooted Regexp | `/{{.*\.jpe?g}}` | `/file.jpeg` | `/file.png` |
|
||||
| | | `/file.jpg` | `/dir/file.jpg` |
|
||||
| Description | Pattern | Matches | Does not match |
|
||||
| ----------- | ---------------- | ------------------------------- | ------------------ |
|
||||
| Wildcard | `*.jpg` | `/file.jpg` | `/file.png` |
|
||||
| | | `/dir/file.jpg` | `/dir/file.png` |
|
||||
| Rooted | `/*.jpg` | `/file.jpg` | `/file.png` |
|
||||
| | | `/file2.jpg` | `/dir/file.jpg` |
|
||||
| Alternates | `*.{jpg,png}` | `/file.jpg` | `/file.gif` |
|
||||
| | | `/dir/file.png` | `/dir/file.gif` |
|
||||
| Path Wildcard | `dir/**` | `/dir/anyfile` | `file.png` |
|
||||
| | | `/subdir/dir/subsubdir/anyfile` | `/subdir/file.png` |
|
||||
| Any Char | `*.t?t` | `/file.txt` | `/file.qxt` |
|
||||
| | | `/dir/file.tzt` | `/dir/file.png` |
|
||||
| Range | `*.[a-z]` | `/file.a` | `/file.0` |
|
||||
| | | `/dir/file.b` | `/dir/file.1` |
|
||||
| Escape | `*.\?\?\?` | `/file.???` | `/file.abc` |
|
||||
| | | `/dir/file.???` | `/dir/file.def` |
|
||||
| Class | `*.\d\d\d` | `/file.012` | `/file.abc` |
|
||||
| | | `/dir/file.345` | `/dir/file.def` |
|
||||
| Regexp | `*.{{jpe?g}}` | `/file.jpeg` | `/file.png` |
|
||||
| | | `/dir/file.jpg` | `/dir/file.jpeeg` |
|
||||
| Rooted Regexp | `/{{.*\.jpe?g}}` | `/file.jpeg` | `/file.png` |
|
||||
| | | `/file.jpg` | `/dir/file.jpg` |
|
||||
|
||||
## How filter rules are applied to files {#how-filter-rules-work}
|
||||
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ description: "Rclone docs for FTP backend"
|
||||
versionIntroduced: "v1.37"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-file" >}} FTP
|
||||
# FTP
|
||||
|
||||
FTP is the File Transfer Protocol. Rclone FTP support is provided using the
|
||||
[github.com/jlaffaye/ftp](https://godoc.org/github.com/jlaffaye/ftp)
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Gofile"
|
||||
versionIntroduced: "v1.68"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-folder" >}} Gofile
|
||||
# Gofile
|
||||
|
||||
[Gofile](https://gofile.io) is a content storage and distribution
|
||||
platform. Its aim is to provide as much service as possible for free
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Google Cloud Storage"
|
||||
versionIntroduced: "v1.02"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-google" >}} Google Cloud Storage
|
||||
# Google Cloud Storage
|
||||
|
||||
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
|
||||
command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Google Photos"
|
||||
versionIntroduced: "v1.49"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-images" >}} Google Photos
|
||||
# Google Photos
|
||||
|
||||
The rclone backend for [Google Photos](https://www.google.com/photos/about/) is
|
||||
a specialized backend for transferring photos and videos to and from
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Better checksums for other remotes"
|
||||
versionIntroduced: "v1.57"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-check-double" >}} Hasher
|
||||
# Hasher
|
||||
|
||||
Hasher is a special overlay backend to create remotes which handle
|
||||
checksums for other remotes. It's main functions include:
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Remote for Hadoop Distributed Filesystem"
|
||||
versionIntroduced: "v1.54"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-globe" >}} HDFS
|
||||
# HDFS
|
||||
|
||||
[HDFS](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html)
|
||||
is a distributed file-system, part of the [Apache Hadoop](https://hadoop.apache.org/)
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for HiDrive"
|
||||
versionIntroduced: "v1.59"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} HiDrive
|
||||
# HiDrive
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Read only remote for HTTP servers"
|
||||
versionIntroduced: "v1.37"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-globe" >}} HTTP
|
||||
# HTTP
|
||||
|
||||
The HTTP remote is a read only remote for reading files of a
|
||||
webserver. The webserver should provide file listings which rclone
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for iCloud Drive"
|
||||
versionIntroduced: "v1.69"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} iCloud Drive
|
||||
# iCloud Drive
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -12,13 +12,27 @@ The initial setup for an iCloud Drive backend involves getting a trust token/ses
|
||||
This can be done by simply using the regular iCloud password, and accepting the code
|
||||
prompt on another iCloud connected device.
|
||||
|
||||
**IMPORTANT**: At the moment an app specific password won't be accepted. Only
|
||||
use your regular password and 2FA.
|
||||
**IMPORTANT**: App-specific passwords are not accepted. Only use your
|
||||
regular Apple ID password and 2FA.
|
||||
|
||||
`rclone config` walks you through the token creation. The trust token is valid
|
||||
for 30 days. After which you will have to reauthenticate with `rclone reconnect`
|
||||
or `rclone config`.
|
||||
|
||||
## Authentication
|
||||
|
||||
rclone authenticates with Apple using [SRP (Secure Remote Password)](https://en.wikipedia.org/wiki/Secure_Remote_Password_protocol),
|
||||
the same protocol used by the iCloud web interface. Your password is never
|
||||
sent to Apple's servers -- instead, a cryptographic proof is exchanged that
|
||||
verifies you know the password without revealing it.
|
||||
|
||||
The authentication flow is:
|
||||
|
||||
1. rclone initiates a session with Apple's identity service
|
||||
2. An SRP key exchange takes place (your password is used locally to derive a key)
|
||||
3. Apple sends a 2FA prompt to your trusted devices
|
||||
4. After you enter the 2FA code, rclone receives a trust token for future sessions
|
||||
|
||||
Here is an example of how to make a remote called `iclouddrive`. First run:
|
||||
|
||||
```console
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for ImageKit backend."
|
||||
versionIntroduced: "v1.63"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} ImageKit
|
||||
# ImageKit
|
||||
|
||||
This is a backend for the [ImageKit.io](https://imagekit.io/) storage service.
|
||||
|
||||
|
||||
+22
-22
@@ -285,8 +285,8 @@ rclone v1.49.1
|
||||
- go version: go1.12.9
|
||||
```
|
||||
|
||||
There are a few command line options to consider when starting an rclone Docker container
|
||||
from the rclone image.
|
||||
There are a few command line options to consider when starting an rclone Docker
|
||||
container from the rclone image.
|
||||
|
||||
- You need to mount the host rclone config dir at `/config/rclone` into the Docker
|
||||
container. Due to the fact that rclone updates tokens inside its config file,
|
||||
@@ -300,8 +300,8 @@ from the rclone image.
|
||||
data files reside on the host with a non-root UID:GID, you need to pass these
|
||||
on the container start command line.
|
||||
|
||||
- If you want to access the RC interface (either via the API or the Web UI), it is
|
||||
required to set the `--rc-addr` to `:5572` in order to connect to it from outside
|
||||
- If you want to access the RC interface (either via the API or the Web UI), it
|
||||
is required to set the `--rc-addr` to `:5572` in order to connect to it from outside
|
||||
the container. An explanation about why this is necessary can be found in an old
|
||||
[pythonspeed.com](https://web.archive.org/web/20200808071950/https://pythonspeed.com/articles/docker-connection-refused/)
|
||||
article.
|
||||
@@ -309,9 +309,9 @@ from the rclone image.
|
||||
probably set it to listen to localhost only, with `127.0.0.1:5572` as the
|
||||
value for `--rc-addr`
|
||||
|
||||
- It is possible to use `rclone mount` inside a userspace Docker container, and expose
|
||||
the resulting fuse mount to the host. The exact `docker run` options to do that
|
||||
might vary slightly between hosts. See, e.g. the discussion in this
|
||||
- It is possible to use `rclone mount` inside a userspace Docker container, and
|
||||
expose the resulting fuse mount to the host. The exact `docker run` options to
|
||||
do that might vary slightly between hosts. See, e.g. the discussion in this
|
||||
[thread](https://github.com/moby/moby/issues/9448).
|
||||
|
||||
You also need to mount the host `/etc/passwd` and `/etc/group` for fuse to work
|
||||
@@ -542,8 +542,8 @@ To override them set the corresponding options (as command-line arguments, or as
|
||||
|
||||
After installing and configuring rclone, as described above, you are ready to use
|
||||
rclone as an interactive command line utility. If your goal is to perform *periodic*
|
||||
operations, such as a regular [sync](https://rclone.org/commands/rclone_sync/), you
|
||||
will probably want to configure your rclone command in your operating system's
|
||||
operations, such as a regular [sync](https://rclone.org/commands/rclone_sync/),
|
||||
you will probably want to configure your rclone command in your operating system's
|
||||
scheduler. If you need to expose *service*-like features, such as
|
||||
[remote control](https://rclone.org/rc/), [GUI](https://rclone.org/gui/),
|
||||
[serve](https://rclone.org/commands/rclone_serve/) or [mount](https://rclone.org/commands/rclone_mount/),
|
||||
@@ -583,9 +583,9 @@ c:\rclone\rclone.exe sync c:\files remote:/files --no-console --log-file c:\rclo
|
||||
|
||||
As mentioned in the [mount](https://rclone.org/commands/rclone_mount/) documentation,
|
||||
mounted drives created as Administrator are not visible to other accounts, not even
|
||||
the account that was elevated as Administrator. By running the mount command as the
|
||||
built-in `SYSTEM` user account, it will create drives accessible for everyone on
|
||||
the system. Both scheduled task and Windows service can be used to achieve this.
|
||||
the account that was elevated as Administrator. By running the mount command as
|
||||
the built-in `SYSTEM` user account, it will create drives accessible for everyone
|
||||
on the system. Both scheduled task and Windows service can be used to achieve this.
|
||||
|
||||
NOTE: Remember that when rclone runs as the `SYSTEM` user, the user profile
|
||||
that it sees will not be yours. This means that if you normally run rclone with
|
||||
@@ -615,8 +615,8 @@ will often give you better results.
|
||||
|
||||
#### Start from Task Scheduler
|
||||
|
||||
Task Scheduler is an administrative tool built into Windows, and it can be used to
|
||||
configure rclone to be started automatically in a highly configurable way, e.g.
|
||||
Task Scheduler is an administrative tool built into Windows, and it can be used
|
||||
to configure rclone to be started automatically in a highly configurable way, e.g.
|
||||
periodically on a schedule, on user log on, or at system startup. It can run
|
||||
be configured to run as the current user, or for a mount command that needs to
|
||||
be available to all users it can run as the `SYSTEM` user.
|
||||
@@ -656,18 +656,18 @@ To Windows service running any rclone command, the excellent third-party utility
|
||||
[NSSM](http://nssm.cc), the "Non-Sucking Service Manager", can be used.
|
||||
It includes some advanced features such as adjusting process priority, defining
|
||||
process environment variables, redirect to file anything written to stdout, and
|
||||
customized response to different exit codes, with a GUI to configure everything from
|
||||
(although it can also be used from command line ).
|
||||
customized response to different exit codes, with a GUI to configure everything
|
||||
from (although it can also be used from command line ).
|
||||
|
||||
There are also several other alternatives. To mention one more,
|
||||
[WinSW](https://github.com/winsw/winsw), "Windows Service Wrapper", is worth checking
|
||||
out. It requires .NET Framework, but it is preinstalled on newer versions of Windows,
|
||||
and it also provides alternative standalone distributions which includes necessary
|
||||
runtime (.NET 5). WinSW is a command-line only utility, where you have to manually
|
||||
create an XML file with service configuration. This may be a drawback for some, but
|
||||
it can also be an advantage as it is easy to back up and reuse the configuration
|
||||
settings, without having go through manual steps in a GUI. One thing to note is that
|
||||
by default it does not restart the service on error, one have to explicit enable
|
||||
create an XML file with service configuration. This may be a drawback for some,
|
||||
but it can also be an advantage as it is easy to back up and reuse the configuration
|
||||
settings, without having go through manual steps in a GUI. One thing to note is
|
||||
that by default it does not restart the service on error, one have to explicit enable
|
||||
this in the configuration file (via the "onfailure" parameter).
|
||||
|
||||
### Autostart on Linux
|
||||
@@ -676,8 +676,8 @@ this in the configuration file (via the "onfailure" parameter).
|
||||
|
||||
To always run rclone in background, relevant for mount commands etc,
|
||||
you can use systemd to set up rclone as a system or user service. Running as a
|
||||
system service ensures that it is run at startup even if the user it is running as
|
||||
has no active session. Running rclone as a user service ensures that it only
|
||||
system service ensures that it is run at startup even if the user it is running
|
||||
as has no active session. Running rclone as a user service ensures that it only
|
||||
starts after the configured user has logged into the system.
|
||||
|
||||
#### Run periodically from cron
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Internet Archive"
|
||||
versionIntroduced: "v1.59"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-archive" >}} Internet Archive
|
||||
# Internet Archive
|
||||
|
||||
The Internet Archive backend utilizes Items on [archive.org](https://archive.org/)
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Internxt Drive"
|
||||
versionIntroduced: "v1.73"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-cloud" >}} Internxt Drive
|
||||
# Internxt Drive
|
||||
|
||||
[Internxt Drive](https://internxt.com) is a zero-knowledge encrypted cloud storage service.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Jottacloud"
|
||||
versionIntroduced: "v1.43"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} Jottacloud
|
||||
# Jottacloud
|
||||
|
||||
Jottacloud is a cloud storage service provider from a Norwegian company, using
|
||||
its own datacenters in Norway.
|
||||
@@ -12,7 +12,7 @@ its own datacenters in Norway.
|
||||
In addition to the official service at [jottacloud.com](https://www.jottacloud.com/),
|
||||
it also provides white-label solutions to different companies. The following
|
||||
are currently supported by this backend, using a different authentication setup
|
||||
as described [below](#whitelabel-authentication):
|
||||
as described [below](#traditional):
|
||||
|
||||
- Elkjøp (with subsidiaries):
|
||||
- Elkjøp Cloud (cloud.elkjop.no)
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Koofr"
|
||||
versionIntroduced: "v1.47"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-suitcase" >}} Koofr
|
||||
# Koofr
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Linkbox"
|
||||
versionIntroduced: "v1.65"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-infinity" >}} Linkbox
|
||||
# Linkbox
|
||||
|
||||
Linkbox is [a private cloud drive](https://linkbox.to/).
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for the local filesystem"
|
||||
versionIntroduced: "v0.91"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-hdd" >}} Local Filesystem
|
||||
# Local Filesystem
|
||||
|
||||
Local paths are specified as normal filesystem paths, e.g. `/path/to/wherever`, so
|
||||
|
||||
@@ -62,7 +62,7 @@ handling file names.
|
||||
| / | 0x2F | / |
|
||||
|
||||
When running on Windows the following characters are replaced. This
|
||||
list is based on the [Windows file naming conventions](https://docs.microsoft.com/de-de/windows/desktop/FileIO/naming-a-file#naming-conventions).
|
||||
list is based on the [Windows file naming conventions](https://docs.microsoft.com/windows/desktop/FileIO/naming-a-file#naming-conventions).
|
||||
|
||||
| Character | Value | Replacement |
|
||||
| --------- |:-----:|:-----------:|
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Mail.ru Cloud"
|
||||
versionIntroduced: "v1.50"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-at" >}} Mail.ru Cloud
|
||||
# Mail.ru Cloud
|
||||
|
||||
[Mail.ru Cloud](https://cloud.mail.ru/) is a cloud storage provided by a
|
||||
Russian internet company [Mail.Ru Group](https://mail.ru). The official
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Mega"
|
||||
versionIntroduced: "v1.41"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-archive" >}} Mega
|
||||
# Mega
|
||||
|
||||
[Mega](https://mega.nz/) is a cloud storage and file hosting service
|
||||
known for its security feature where all files are encrypted locally
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Memory backend"
|
||||
versionIntroduced: "v1.51"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-memory" >}} Memory
|
||||
# Memory
|
||||
|
||||
The memory backend is an in RAM backend. It does not persist its
|
||||
data - use the local backend for that.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Akamai NetStorage"
|
||||
versionIntroduced: "v1.58"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-database" >}} Akamai NetStorage
|
||||
# Akamai NetStorage
|
||||
|
||||
Paths are specified as `remote:`
|
||||
You may put subdirectories in too, e.g. `remote:/path/to/dir`.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Microsoft OneDrive"
|
||||
versionIntroduced: "v1.24"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-windows" >}} Microsoft OneDrive
|
||||
# Microsoft OneDrive
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for OpenDrive"
|
||||
versionIntroduced: "v1.42"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-file" >}} OpenDrive
|
||||
# OpenDrive
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ type: page
|
||||
versionIntroduced: "v1.60"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} Oracle Object Storage
|
||||
# Oracle Object Storage
|
||||
|
||||
Object Storage provided by the Oracle Cloud Infrastructure (OCI).
|
||||
Read more at <oracle.com>:
|
||||
|
||||
@@ -3,7 +3,7 @@ title: "Oracle Object Storage Mount"
|
||||
description: "Oracle Object Storage mounting tutorial"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} Mount Buckets and Expose via NFS Tutorial
|
||||
# Mount Buckets and Expose via NFS Tutorial
|
||||
|
||||
This runbook shows how to [mount](/commands/rclone_mount/) *Oracle Object Storage*
|
||||
buckets as local file system in OCI compute Instance using rclone tool.
|
||||
|
||||
+12
-12
@@ -39,7 +39,7 @@ size by default, though can be configured to check the file hash
|
||||
change the timestamp of an existing file without having to re-upload it.
|
||||
|
||||
| Key | Explanation |
|
||||
|-----|-------------|
|
||||
| --- | ----------- |
|
||||
| `-` | ModTimes not supported - times likely the upload time |
|
||||
| `R` | ModTimes supported on files but can't be changed without re-upload |
|
||||
| `R/W` | Read and Write ModTimes fully supported on files |
|
||||
@@ -186,8 +186,8 @@ will be escaped with the `‛` character to avoid ambiguous file names.
|
||||
Each cloud storage backend can use a different set of characters,
|
||||
which will be specified in the documentation for each backend.
|
||||
|
||||
| Character | Value | Replacement |
|
||||
| --------- |:-----:|:-----------:|
|
||||
| Character | Value | Replacement |
|
||||
| --------- | :---: | :---------- :|
|
||||
| NUL | 0x00 | ␀ |
|
||||
| SOH | 0x01 | ␁ |
|
||||
| STX | 0x02 | ␂ |
|
||||
@@ -227,9 +227,9 @@ The default encoding will also encode these file names as they are
|
||||
problematic with many cloud storage systems.
|
||||
|
||||
| File name | Replacement |
|
||||
| --------- |:-----------:|
|
||||
| --------- | :--------- :|
|
||||
| . | . |
|
||||
| .. | .. |
|
||||
| .. | .. |
|
||||
|
||||
#### Invalid UTF-8 bytes {#invalid-utf8}
|
||||
|
||||
@@ -269,8 +269,8 @@ list of all possible values by passing an invalid value to this
|
||||
flag, e.g. `--local-encoding "help"`. The command `rclone help flags encoding`
|
||||
will show you the defaults for the backends.
|
||||
|
||||
| Encoding | Characters | Encoded as |
|
||||
| --------- | ---------- | ---------- |
|
||||
| Encoding | Characters | Encoded as |
|
||||
| -------- | ---------- | ---------- |
|
||||
| Asterisk | `*` | `*` |
|
||||
| BackQuote | `` ` `` | ``` |
|
||||
| BackSlash | `\` | `\` |
|
||||
@@ -395,12 +395,12 @@ that backend) and/or user metadata (general purpose metadata).
|
||||
The levels of metadata support are
|
||||
|
||||
| Key | Explanation |
|
||||
|-----|-------------|
|
||||
| `R` | Read only System Metadata on files only|
|
||||
| `RW` | Read and write System Metadata on files only|
|
||||
| `RWU` | Read and write System Metadata and read and write User Metadata on files only|
|
||||
| --- | ----------- |
|
||||
| `R` | Read only System Metadata on files only |
|
||||
| `RW` | Read and write System Metadata on files only |
|
||||
| `RWU` | Read and write System Metadata and read and write User Metadata on files only |
|
||||
| `DR` | Read only System Metadata on files and directories |
|
||||
| `DRW` | Read and write System Metadata on files and directories|
|
||||
| `DRW` | Read and write System Metadata on files and directories |
|
||||
| `DRWU` | Read and write System Metadata and read and write User Metadata on files and directories |
|
||||
|
||||
See [the metadata docs](/docs/#metadata) for more info.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for pCloud"
|
||||
versionIntroduced: "v1.39"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} pCloud
|
||||
# pCloud
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for PikPak"
|
||||
versionIntroduced: "v1.62"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-cloud" >}} PikPak
|
||||
# PikPak
|
||||
|
||||
PikPak is [a private cloud drive](https://mypikpak.com/).
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Pixeldrain"
|
||||
versionIntroduced: "v1.68"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-circle" >}} Pixeldrain
|
||||
# Pixeldrain
|
||||
|
||||
This is the backend for Pixeldrain's premium filesystem feature. This is not the
|
||||
same as pixeldrain's free file sharing product. The filesystem requires either a
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for premiumize.me"
|
||||
versionIntroduced: "v1.49"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-user" >}} premiumize.me
|
||||
# premiumize.me
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Proton Drive"
|
||||
versionIntroduced: "v1.64.0"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-folder" >}} Proton Drive
|
||||
# Proton Drive
|
||||
|
||||
[Proton Drive](https://proton.me/drive) is an end-to-end encrypted Swiss vault
|
||||
for your files that protects your data.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for put.io"
|
||||
versionIntroduced: "v1.49"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-parking" >}} put.io
|
||||
# put.io
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for QingStor Object Storage"
|
||||
versionIntroduced: "v1.38"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-hdd" >}} QingStor
|
||||
# QingStor
|
||||
|
||||
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
|
||||
command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Rclone docs for Quatrix"
|
||||
versionIntroduced: "v1.63.2"
|
||||
---
|
||||
|
||||
# {{< icon "fas fa-shield-alt" >}} Quatrix
|
||||
# Quatrix
|
||||
|
||||
Quatrix by Maytech is [Quatrix Secure Compliant File Sharing | Maytech](https://www.maytech.net/products/quatrix-business).
|
||||
|
||||
|
||||
+145
-7
@@ -4,7 +4,7 @@ description: "Rclone docs for Amazon S3"
|
||||
versionIntroduced: "v0.91"
|
||||
---
|
||||
|
||||
# {{< icon "fab fa-amazon" >}} Amazon S3 Storage Providers
|
||||
# Amazon S3 Storage Providers
|
||||
|
||||
The S3 backend can be used with a number of different providers:
|
||||
|
||||
@@ -55,6 +55,7 @@ The S3 backend can be used with a number of different providers:
|
||||
{{< provider name="Storj" home="https://storj.io/" config="/s3/#storj" >}}
|
||||
{{< provider name="Synology C2 Object Storage" home="https://c2.synology.com/en-global/object-storage/overview" config="/s3/#synology-c2" >}}
|
||||
{{< provider name="Tencent Cloud Object Storage (COS)" home="https://intl.cloud.tencent.com/product/cos" config="/s3/#tencent-cos" >}}
|
||||
{{< provider name="US3" home="https://www.ucloud.cn/site/product/ufile.html" config="/s3/#us3" >}}
|
||||
{{< provider name="Wasabi" home="https://wasabi.com/" config="/s3/#wasabi" >}}
|
||||
{{< provider name="Zadara Object Storage" home="https://www.zadara.com" config="/s3/#zadara" >}}
|
||||
{{< provider name="Zata" home="https://zata.ai/" config="/s3/#Zata" end="true" >}}
|
||||
@@ -913,10 +914,9 @@ According to AWS's [documentation on S3 Object Lock](https://docs.aws.amazon.com
|
||||
> If you configure a default retention period on a bucket, requests to upload
|
||||
objects in such a bucket must include the Content-MD5 header.
|
||||
|
||||
As mentioned in the [Modification times and hashes](#modification-times-and-hashes)
|
||||
section, small files that are not uploaded as multipart, use a different tag, causing
|
||||
the upload to fail. A simple solution is to set the `--s3-upload-cutoff 0` and force
|
||||
all the files to be uploaded as multipart.
|
||||
Rclone automatically computes and includes the Content-MD5 header when
|
||||
uploading objects with Object Lock parameters set. No special
|
||||
configuration is needed.
|
||||
|
||||
#### Setting Object Lock retention
|
||||
|
||||
@@ -5132,7 +5132,9 @@ can follow this [guide](https://docs.cubbit.io/getting-started/quickstart#api-ke
|
||||
to retrieve these keys. They will be needed when prompted by `rclone config`.
|
||||
|
||||
Default region will correspond to `eu-west-1` and the endpoint has to be specified
|
||||
as `s3.cubbit.eu`.
|
||||
as `s3.cubbit.eu`. If you set a custom tenant endpoint, you must use
|
||||
`s3.{tenant_name}.cubbit.eu`, because the generic `s3.cubbit.eu` endpoint will not
|
||||
work in this case.
|
||||
|
||||
Going through the whole process of creating a new remote by running `rclone config`,
|
||||
each prompt should be answered as shown below:
|
||||
@@ -5145,7 +5147,7 @@ env_auth> false
|
||||
access_key_id> YOUR_ACCESS_KEY
|
||||
secret_access_key> YOUR_SECRET_KEY
|
||||
region> eu-west-1 (or leave empty)
|
||||
endpoint> s3.cubbit.eu
|
||||
endpoint> s3.cubbit.eu (or s3.{tenant_name}.cubbit.eu)
|
||||
acl>
|
||||
```
|
||||
|
||||
@@ -9117,6 +9119,142 @@ To configure access to Tencent COS, follow the steps below:
|
||||
==== ====
|
||||
cos s3
|
||||
```
|
||||
### US3 {#us3}
|
||||
[US3](https://www.ucloud.cn/site/product/ufile.html) Object Storage is a service that provides cloud storage for unstructured files for internet applications.
|
||||
|
||||
To configure access to US3, follow the steps below:
|
||||
1. First run:
|
||||
```console
|
||||
rclone config
|
||||
No remotes found, make a new one?
|
||||
n) New remote
|
||||
s) Set configuration password
|
||||
q) Quit config
|
||||
n/s/q> n
|
||||
```
|
||||
2. Give the name of the configuration. For example, name it 'US3'.
|
||||
|
||||
```text
|
||||
name> US3
|
||||
```
|
||||
3. Select `s3` storage.
|
||||
|
||||
```text
|
||||
Choose a number from below, or type in your own value
|
||||
[snip]
|
||||
XX / Amazon S3 Compliant Storage Providers including AWS, ...
|
||||
\ "s3"
|
||||
[snip]
|
||||
Storage> s3
|
||||
```
|
||||
4. Select `US3` provider.
|
||||
|
||||
```text
|
||||
Choose a number from below, or type in your own value.
|
||||
XX / Amazon Web Services (AWS) S3
|
||||
\ (AWS)
|
||||
[snip]
|
||||
XX / UCloud Object Storage (US3)
|
||||
\ (US3)
|
||||
[snip]
|
||||
provider> US3
|
||||
```
|
||||
5. Enter your SecretId and SecretKey of US3.
|
||||
|
||||
```text
|
||||
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
|
||||
Only applies if access_key_id and secret_access_key is blank.
|
||||
Choose a number from below, or type in your own boolean value (true or false).
|
||||
Press Enter for the default (false).
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Enter AWS credentials in the next step
|
||||
\ "false"
|
||||
2 / Get AWS credentials from the environment (env vars or IAM)
|
||||
\ "true"
|
||||
env_auth> 1
|
||||
Option access_key_id.
|
||||
AWS Access Key ID.
|
||||
Leave blank for anonymous access or runtime credentials.
|
||||
Enter a value. Press Enter to leave empty.
|
||||
access_key_id> AKIDxxxxxxxxxx
|
||||
AWS Secret Access Key (password)
|
||||
Leave blank for anonymous access or runtime credentials.
|
||||
Enter a string value. Press Enter for the default ("").
|
||||
secret_access_key> xxxxxxxxxxx
|
||||
```
|
||||
|
||||
6. Select endpoint.
|
||||
```text
|
||||
Option endpoint.
|
||||
Endpoint for S3 API.
|
||||
Required when using an S3 clone.
|
||||
Choose a number from below, or type in your own value.
|
||||
Press Enter to leave empty.
|
||||
1 / North China (Beijing 2).
|
||||
\ "s3-cn-bj.ufileos.com"
|
||||
2 / North China (Ulanqab).
|
||||
\ "s3-cn-wlcb.ufileos.com"
|
||||
3 / Shanghai.
|
||||
\ "s3-cn-sh2.ufileos.com"
|
||||
4 / South China (Guangzhou).
|
||||
\ "s3-cn-gd.ufileos.com"
|
||||
[snip]
|
||||
endpoint> 1
|
||||
```
|
||||
|
||||
7. Choose acl and storage class.
|
||||
|
||||
```text
|
||||
Option acl.
|
||||
Canned ACL used when creating buckets and storing or copying objects.
|
||||
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
|
||||
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
|
||||
Note that this ACL is applied when server-side copying objects as S3
|
||||
doesn't copy the ACL from the source but rather writes a fresh one.
|
||||
If the acl is an empty string then no X-Amz-Acl: header is added and
|
||||
the default (private) will be used.
|
||||
Choose a number from below, or type in your own value.
|
||||
Press Enter to leave empty.
|
||||
/ Owner gets FULL_CONTROL.
|
||||
1 | No one else has access rights (default).
|
||||
\ (private)
|
||||
/ Owner gets FULL_CONTROL.
|
||||
2 | The AllUsers group gets READ access.
|
||||
\ (public-read)
|
||||
acl> 1
|
||||
|
||||
The storage class to use when storing new objects in US3.
|
||||
Enter a string value. Press Enter for the default ("").
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Default
|
||||
\ ""
|
||||
[snip]
|
||||
storage_class> 1
|
||||
Edit advanced config? (y/n)
|
||||
y) Yes
|
||||
n) No (default)
|
||||
y/n> n
|
||||
|
||||
Configuration complete.
|
||||
Options:
|
||||
- type: s3
|
||||
- provider: US3
|
||||
- access_key_id: xx
|
||||
- secret_access_key: xx
|
||||
- endpoint: s3-cn-bj.ufileos.com
|
||||
- acl: private
|
||||
|
||||
Keep this "US3" remote?
|
||||
y) Yes this is OK (default)
|
||||
e) Edit this remote
|
||||
d) Delete this remote
|
||||
y/e/d> y
|
||||
Current remotes:
|
||||
|
||||
Name Type
|
||||
==== ====
|
||||
US3 s3
|
||||
```
|
||||
|
||||
### Wasabi
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Seafile"
|
||||
versionIntroduced: "v1.52"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-server" >}} Seafile
|
||||
# Seafile
|
||||
|
||||
This is a backend for the [Seafile](https://www.seafile.com/) storage service:
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "SFTP"
|
||||
versionIntroduced: "v1.36"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-server" >}} SFTP
|
||||
# SFTP
|
||||
|
||||
SFTP is the [Secure (or SSH) File Transfer
|
||||
Protocol](https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol).
|
||||
|
||||
@@ -4,7 +4,7 @@ description: "Shade Backend Docs"
|
||||
versionIntroduced: "v1.73"
|
||||
---
|
||||
|
||||
# {{< icon "fa fa-moon" >}} Shade
|
||||
# Shade
|
||||
|
||||
This is a backend for the [Shade](https://shade.inc/) platform
|
||||
|
||||
|
||||
Loaded 100 of 165 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user