Compare commits

...
Author SHA1 Message Date
Nick Craig-Wood 07313554bf b2: reduce default chunk size to 16M and increase upload concurrency to 8
The 96M default chunk size dates from 2016 when 100,000,000 bytes was
the minimum part size B2 allowed. The minimum is now 5,000,000 bytes.

Measurements against B2 EU on a 1 Gbit/s link show that each part has
a fixed overhead of about a second regardless of size, so throughput
is governed by chunk size multiplied by upload concurrency rather than
by chunk size alone. 16M chunks with concurrency 8 upload a 1 GiB file
as fast as 96M chunks with concurrency 4 while using a third of the
memory per transfer, and retry much less data when a part fails.

This also means that files between --multi-thread-cutoff and the chunk
size are no longer uploaded as a single part large file, which B2
rejects with "large files must have at least 2 parts", unless the
cutoff is set below 16M.
2026-09-10 15:57:25 +01:00
Nick Craig-Wood 417ecce222 docs: update sponsors 2026-09-08 15:27:44 +01:00
4 changed files with 13 additions and 7 deletions

No files matched your search

+8 -3
View File
@@ -64,7 +64,7 @@ const (
maxParts = 10000
maxVersions = 100 // maximum number of versions we search in --b2-versions mode
minChunkSize = 5 * fs.Mebi
defaultChunkSize = 96 * fs.Mebi
defaultChunkSize = 16 * fs.Mebi
defaultUploadCutoff = 200 * fs.Mebi
largeFileCopyCutoff = 4 * fs.Gibi // 5E9 is the max
defaultMaxAge = 24 * time.Hour
@@ -153,7 +153,12 @@ The minimum is 0 and the maximum is 4.6 GiB.`,
When uploading large files, chunk the file into this size.
Must fit in memory. These chunks are buffered in memory and there
might a maximum of "--transfers" chunks in progress at once.
might be a maximum of "--transfers" * "--b2-upload-concurrency"
chunks in progress at once.
B2 has a fixed per part overhead of about a second, so upload speed
is roughly proportional to the chunk size multiplied by
"--b2-upload-concurrency".
5,000,000 Bytes is the minimum size.`,
Default: defaultChunkSize,
@@ -168,7 +173,7 @@ concurrently.
Note that chunks are stored in memory and there may be up to
"--transfers" * "--b2-upload-concurrency" chunks stored at once
in memory.`,
Default: 4,
Default: 8,
Advanced: true,
}, {
Name: "disable_checksum",
+3 -2
View File
@@ -169,8 +169,9 @@ to load your computer, etc. The default of `--transfers 4` is
definitely too low for Backblaze B2 though.
Note that uploading big files (bigger than 200 MiB by default) will use
a 96 MiB RAM buffer by default. There can be at most `--transfers` of
these in use at any moment, so this sets the upper limit on the memory
`--b2-upload-concurrency` (8 by default) buffers of `--b2-chunk-size`
(16 MiB by default) each. There can be at most `--transfers` files
uploading at any moment, so this sets the upper limit on the memory
used.
### Versions
+1 -1
View File
@@ -22,7 +22,7 @@ Thank you to our sponsors:
{{< sponsor src="/img/logos/sia.svg" width="200" height="200" title="Visit our sponsor sia" link="https://sia.tech">}}
{{< sponsor src="/img/logos/route4me.svg" width="400" height="200" title="Visit our sponsor Route4Me" link="https://route4me.com/">}}
{{< sponsor src="/img/logos/rcloneview.svg" width="300" height="200" title="Visit our sponsor RcloneView" link="https://rcloneview.com/">}}
{{< sponsor src="/img/logos/rcloneui.svg" width="300" height="200" title="Visit our sponsor RcloneUI" link="https://rcloneui.com">}}
{{< sponsor src="/img/logos/rcloneui.svg" width="300" height="200" title="Visit our sponsor RcloneUI" link="https://rcloneui.com/web">}}
{{< sponsor src="/img/logos/data-raven.svg" width="300" height="200" title="Data Raven: rclone, managed. Unlimited transfers, no per-GB fees." link="https://dataraven.io/r/rclone">}}
{{< sponsor src="/img/logos/shade.svg" width="300" height="200" title="Visit our sponsor Shade" link="https://shade.inc">}}
{{< sponsor src="/img/logos/filelu-rclone.svg" width="300" height="200" title="Visit our sponsor FileLu" link="https://filelu.com/">}}
+1 -1
View File
@@ -43,7 +43,7 @@
<div class="card">
<div class="card-header">Silver Sponso⁠r</div>
<div class="card-body">
<a href="https://rcloneui.com" target="_blank" rel="noopener" title="Visit our sponsor RcloneUI"><img src="/img/logos/rcloneui.svg"></a><br />
<a href="https://rcloneui.com/web" target="_blank" rel="noopener" title="Visit our sponsor RcloneUI"><img src="/img/logos/rcloneui.svg"></a><br />
</div>
</div>
</div>