Add wdsexport: Linux directory scan to WinDirStat 2.x saved-results CSV

Single-file, stdlib-only CLI that writes the CSV format WinDirStat's
CsvLoader accepts (parent-before-child rows, subtree totals, localized
headers for all 25 UI languages). Includes install.sh, pyproject entry
point, unit tests and a generator for the language table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Lucas WintherandClaude Opus 5.5 committed 2026-11-12 22:01:02 +01:00
commit e28fbeb0b3
8 files changed
+637

No files matched your search

+8
View File
@@ -0,0 +1,8 @@
__pycache__/
*.egg-info/
build/
dist/
*.csv
*.zip
*.part
.wdsexport-*.tmp
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Lucas Winther
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+85
View File
@@ -0,0 +1,85 @@
# WinDirStat-Export
`wdsexport` scans a directory tree on Linux (or macOS) and writes a
[WinDirStat](https://windirstat.net) 2.x **saved-results CSV**. Copy it to a
Windows machine and open it with **File → Open saved results** to browse the
remote disk with WinDirStat's treemap.
Single Python file, standard library only, Python 3.8+.
## Install
```sh
git clone https://gitea.lucaswinther.info/lucasw89/WinDirStat-Export.git
cd WinDirStat-Export
./install.sh # -> ~/.local/bin/wdsexport
# or: ./install.sh /usr/local (with sudo)
# or: pipx install .
```
## Usage
```sh
wdsexport # scan /, write ./<hostname>-root.csv
wdsexport / --zip # also write a .zip (CSVs compress ~15x)
sudo wdsexport / # include root-only dirs (/var/lib/docker, ...)
wdsexport ~/projects -o proj.csv # scan one directory
wdsexport / --exclude /home/me/vms --lang da
wdsexport / -o - | gzip > scan.csv.gz
```
| Argument | Default | Meaning |
|---|---|---|
| `PATH` | `/` | Directory to scan |
| `-o, --output FILE` | `./<name>.csv` | Output CSV; `-` writes to stdout |
| `--name NAME` | `<hostname>-<dirname>` | Root folder name shown in WinDirStat |
| `--exclude DIR` | — | Skip a directory (repeatable) |
| `--zip` | off | Also write `<output>.zip` |
| `--lang CODE` | `en` | Column-name language (see below) |
| `--cross-fs` | off | Descend into other mounted filesystems (default stays on one, like `du -x`) |
| `-q, --quiet` | off | No progress output |
### `--lang` must match WinDirStat's UI language
WinDirStat identifies CSV columns by their *localized* header names. If your
WinDirStat runs in Danish, export with `--lang da`; otherwise the file is
rejected. Supported: ar, cs, da, de, en, es, et, fi, fr, hi, hu, it, ja, ko,
nb, nl, pl, pt, ru, sl, sv, tr, uk, zh, zh-HK.
## What gets recorded
- **Logical size** = file length; **physical size** = allocated blocks
(`st_blocks × 512`), so sparse files and filesystem overhead show correctly.
- Hard-linked files are counted once (first path seen).
- Symlinks are recorded as small files and never followed.
- Dotfiles get the Hidden attribute, so enable *show hidden files* in WinDirStat.
- Directories that can't be read are marked *partial* (WinDirStat shows them as
incomplete). Run with `sudo` to include them.
- Characters WinDirStat's CSV parser can't handle in names are replaced:
`\` → `_`, `"` → `'`, newlines → space. Invalid UTF-8 becomes `�`.
- The tool skips its own output and temporary files when they sit inside the
scanned tree.
Disk space: the scan writes a temporary file about the size of the final CSV
next to the output (in `$TMPDIR` for `-o -`). Roughly 200 bytes per file, so
8 million files ≈ 1.5 GB CSV, ≈ 100 MB zipped.
## How it works
The format follows WinDirStat's own loader
([`CsvLoader.cpp`](https://github.com/windirstat/windirstat/blob/master/windirstat/CsvLoader.cpp)):
rows must list each parent before its children, paths use `\` separators, and
folder rows carry subtree totals. `wdsexport` writes rows in post-order (when
folder totals are known) to a temp file and then copies it out reversed, which
gives a parent-first order without holding the tree in memory.
## Development
```sh
python3 -m unittest discover -s tests
python3 tools/gen_langs.py # refresh column names from WinDirStat's language files
```
## License
MIT
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/bin/sh
# Install wdsexport as a single executable file (no dependencies beyond python3).
# Usage: ./install.sh [PREFIX] default PREFIX: ~/.local (installs to PREFIX/bin/wdsexport)
set -eu
prefix="${1:-$HOME/.local}"
src="$(cd "$(dirname "$0")" && pwd)/wdsexport.py"
command -v python3 >/dev/null || { echo "python3 not found" >&2; exit 1; }
mkdir -p "$prefix/bin"
install -m 755 "$src" "$prefix/bin/wdsexport"
echo "installed $prefix/bin/wdsexport"
case ":$PATH:" in
*":$prefix/bin:"*) ;;
*) echo "note: $prefix/bin is not on your PATH" >&2 ;;
esac
+21
View File
@@ -0,0 +1,21 @@
[build-system]
requires = ["setuptools>=61"]
build-backend = "setuptools.build_meta"
[project]
name = "wdsexport"
dynamic = ["version"]
description = "Scan a Linux/macOS directory tree and write a WinDirStat 2.x saved-results CSV"
readme = "README.md"
license = { text = "MIT" }
requires-python = ">=3.8"
dependencies = []
[project.scripts]
wdsexport = "wdsexport:main"
[tool.setuptools]
py-modules = ["wdsexport"]
[tool.setuptools.dynamic]
version = { attr = "wdsexport.__version__" }
+147
View File
@@ -0,0 +1,147 @@
import contextlib
import csv
import io
import os
import subprocess
import sys
import tempfile
import unittest
import zipfile
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
import wdsexport # noqa: E402
SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "wdsexport.py")
def read_rows(path_or_text):
text = path_or_text if "\n" in path_or_text else open(path_or_text, encoding="utf-8").read()
assert text.startswith("\ufeff"), "missing UTF-8 BOM"
rows = list(csv.reader(io.StringIO(text[1:])))
return rows[0], rows[1:]
def check_loader_invariants(testcase, rows):
"""Mirror the constraints of WinDirStat's CsvLoader: one root container first,
every other row's parent (prefix before last backslash) already seen as a container."""
containers = set()
for i, r in enumerate(rows):
name, itype = r[0], int(r[7], 16)
if i == 0:
testcase.assertTrue(itype & wdsexport.ITF_ROOTITEM)
else:
testcase.assertFalse(itype & wdsexport.ITF_ROOTITEM)
testcase.assertIn(name.rsplit("\\", 1)[0], containers, f"orphan row {name}")
if itype & wdsexport.IT_DIRECTORY:
containers.add(name)
class ExportTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.root = os.path.join(self.tmp.name, "tree")
os.makedirs(os.path.join(self.root, "a", "b"))
os.makedirs(os.path.join(self.root, "skipme"))
os.makedirs(os.path.join(self.root, ".hidden"))
self.write("a/one.bin", 1000)
self.write("a/b/two.bin", 5000)
self.write('a/b/we,ird "na\\me".txt', 10)
self.write("skipme/big.bin", 100000)
self.write(".hidden/x", 7)
os.link(os.path.join(self.root, "a/one.bin"), os.path.join(self.root, "a/b/hardlink.bin"))
os.symlink("a/one.bin", os.path.join(self.root, "link"))
self.locked = os.path.join(self.root, "locked")
os.makedirs(self.locked)
self.write("locked/secret", 3)
os.chmod(self.locked, 0)
self.out = os.path.join(self.tmp.name, "out.csv")
def tearDown(self):
os.chmod(self.locked, 0o755)
self.tmp.cleanup()
def write(self, rel, size):
with open(os.path.join(self.root, rel), "wb") as f:
f.write(b"x" * size)
def run_main(self, *args):
err = io.StringIO()
with contextlib.redirect_stderr(err):
rc = wdsexport.main(list(args))
self.assertEqual(rc, 0, err.getvalue())
return err.getvalue()
def test_structure_and_totals(self):
self.run_main(self.root, "-o", self.out, "--name", "test", "--exclude",
os.path.join(self.root, "skipme"), "-q")
header, rows = read_rows(self.out)
self.assertEqual(header, wdsexport.header("en"))
check_loader_invariants(self, rows)
by_name = {r[0]: r for r in rows}
root = rows[0]
self.assertEqual(root[0], "test")
symlink_size = os.lstat(os.path.join(self.root, "link")).st_size
# one.bin counted once despite the hard link; symlink counted as itself
self.assertEqual(int(root[3]), 1000 + 5000 + 10 + 7 + symlink_size)
# files: one.bin, two.bin, weird, x, link (hardlink and locked contents not counted)
self.assertEqual(int(root[1]), 5)
# folders: a, a/b, .hidden, locked
self.assertEqual(int(root[2]), 4)
self.assertNotIn("test\\skipme", by_name)
# which hard-link path gets the bytes depends on scandir order; both live under a/
self.assertEqual(by_name["test\\a"][3], "6010")
self.assertIn("test\\a\\b\\we,ird 'na_me'.txt", by_name)
self.assertEqual(by_name["test\\.hidden"][5], "H")
self.assertEqual(by_name["test\\locked"][9], "partial")
self.assertTrue(int(by_name["test\\locked"][7], 16) & wdsexport.ITF_SCANERROR)
self.assertRegex(root[6], r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$")
# each CSV line must have exactly 8 double quotes (loader has no escaping)
with open(self.out, encoding="utf-8") as f:
next(f)
for line in f:
self.assertEqual(line.count('"'), 8, line)
def test_language_headers(self):
self.run_main(self.root, "-o", self.out, "--lang", "da", "-q")
header, _ = read_rows(self.out)
self.assertEqual(header[:3], ["Navn", "Filer", "Mapper"])
self.assertEqual(header[7], "WinDirStat Attributter")
def test_stdout_matches_file(self):
self.run_main(self.root, "-o", self.out, "--name", "n", "-q")
res = subprocess.run([sys.executable, SCRIPT, self.root, "-o", "-", "--name", "n", "-q"],
capture_output=True, check=True)
self.assertEqual(res.stdout.decode("utf-8"), open(self.out, encoding="utf-8").read())
def test_zip(self):
self.run_main(self.root, "-o", self.out, "--zip", "-q")
with zipfile.ZipFile(self.out[:-4] + ".zip") as z:
self.assertEqual(z.read("out.csv"), open(self.out, "rb").read())
def test_output_inside_scanned_tree_is_skipped(self):
out = os.path.join(self.root, "inside.csv")
self.run_main(self.root, "-o", out, "--name", "t", "-q")
names = [r[0] for r in read_rows(out)[1]]
self.assertNotIn("t\\inside.csv", names)
self.assertFalse([n for n in names if ".wdsexport-" in n or n.endswith(".part")])
self.assertFalse([f for f in os.listdir(self.root) if f.startswith(".wdsexport-")])
def test_default_name(self):
self.assertTrue(wdsexport.default_name("/").endswith("-root"))
self.assertTrue(wdsexport.default_name(self.root).endswith("-tree"))
class ReverseLinesTest(unittest.TestCase):
def test_small_blocks(self):
lines = [f"line {i} {'x' * (i % 7)}".encode() for i in range(500)]
src = io.BytesIO(b"\n".join(lines) + b"\n")
dst = io.BytesIO()
wdsexport.copy_reversed_lines(src, dst, block=13)
self.assertEqual(dst.getvalue(), b"\n".join(reversed(lines)) + b"\n")
if __name__ == "__main__":
unittest.main()
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""Regenerate the LANGS table in wdsexport.py from WinDirStat's language files.
Usage: python3 tools/gen_langs.py [REF] (REF defaults to master)
"""
import json
import os
import re
import sys
import urllib.request
REPO = "windirstat/windirstat"
KEYS = ["IDS_COL_NAME", "IDS_COL_FILES", "IDS_COL_FOLDERS", "IDS_COL_SIZE_LOGICAL",
"IDS_COL_SIZE_PHYSICAL", "IDS_COL_ATTRIBUTES", "IDS_COL_LAST_CHANGE", "IDS_COL_INDEX"]
TARGET = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "wdsexport.py")
def fetch(url):
req = urllib.request.Request(url, headers={"User-Agent": "wdsexport-gen-langs"})
with urllib.request.urlopen(req, timeout=30) as r:
return r.read()
def main():
ref = sys.argv[1] if len(sys.argv) > 1 else "master"
tree = json.loads(fetch(f"https://api.github.com/repos/{REPO}/git/trees/{ref}?recursive=1"))
paths = sorted(t["path"] for t in tree["tree"]
if re.fullmatch(r"windirstat/res/langs/lang_[\w-]+\.txt", t["path"]))
langs = {}
for path in paths:
code = re.search(r"lang_([\w-]+)\.txt$", path).group(1)
text = fetch(f"https://raw.githubusercontent.com/{REPO}/{ref}/{path}").decode("utf-8-sig")
values = dict(line.split("=", 1) for line in text.splitlines() if "=" in line)
missing = [k for k in KEYS if k not in values]
if missing:
print(f"skipping {code}: missing {missing}", file=sys.stderr)
continue
row = [values[k] for k in KEYS]
if any('"' in v for v in row):
print(f"skipping {code}: column name contains a double quote", file=sys.stderr)
continue
langs[code] = row
body = "LANGS = {\n" + "".join(f" {c!r}: {v!r},\n" for c, v in langs.items()) + "}\n"
src = open(TARGET, encoding="utf-8").read()
new = re.sub(r"(# @@LANGS-BEGIN@@\n).*?(# @@LANGS-END@@)",
lambda m: m.group(1) + body + m.group(2), src, flags=re.S)
open(TARGET, "w", encoding="utf-8").write(new)
print(f"wrote {len(langs)} languages to {os.path.normpath(TARGET)}", file=sys.stderr)
if __name__ == "__main__":
main()
Executable
+288
View File
@@ -0,0 +1,288 @@
#!/usr/bin/env python3
"""wdsexport - scan a Linux/macOS directory tree and write a WinDirStat 2.x
"saved results" CSV, to be opened on Windows via File > Open saved results.
Format notes (from windirstat/CsvLoader.cpp):
* Header names must match WinDirStat's UI language exactly.
* Rows must be ordered parent-before-child; parents are found by the path
prefix before the last backslash.
* Container rows carry subtree totals (the loader does not sum children).
Rows are written in post-order (when subtree totals are known) to a temp file,
which is then copied out in reverse: reversed post-order is parent-first.
"""
import argparse
import os
import socket
import stat
import sys
import tempfile
import time
import zipfile
__version__ = "1.0.0"
IT_DIRECTORY = 1 << 2
IT_FILE = 1 << 3
ITF_SCANERROR = 1 << 14
ITF_ROOTITEM = 1 << 28
# Localized column names from windirstat/res/langs/lang_*.txt, in the order:
# Name, Files, Folders, Logical Size, Physical Size, Attributes, Last Change, Index.
# Regenerate with tools/gen_langs.py.
# @@LANGS-BEGIN@@
LANGS = {
'ar': ['الاسم', 'الملفات', 'المجلدات الفرعية', 'الحجم المنطقي', 'الحجم الفعلي على القرص', 'السمات', 'آخر تعديل', 'الفهرس'],
'cs': ['Název', 'Soubory', 'Složky', 'Logická velikost', 'Fyzická velikost', 'Atributy', 'Poslední změna', 'Index'],
'da': ['Navn', 'Filer', 'Mapper', 'Logisk størrelse', 'Fysisk størrelse', 'Attributter', 'Sidste ændring', 'Indeks'],
'de': ['Name', 'Dateien', 'Verzeichnisse', 'Logische Größe', 'Physische Größe', 'Attribute', 'Letzte Änderung', 'Index'],
'en': ['Name', 'Files', 'Folders', 'Logical Size', 'Physical Size', 'Attributes', 'Last Change', 'Index'],
'es': ['Nombre', 'Archivos', 'Subdirectorios', 'Tamaño Lógico', 'Tamaño Físico', 'Atributos', 'Último Cambio', 'Índice'],
'et': ['Nimi', 'Failid', 'Alamkataloogid', 'Loogiline suurus', 'Füüsiline suurus', 'Atribuudid', 'Viimane muudatus', 'Indeks'],
'fi': ['Nimi', 'Tiedostoja', 'Alikansioita', 'Looginen koko', 'Fyysinen koko', 'Määritteet', 'Edellinen muutos', 'Indeksi'],
'fr': ['Nom', 'Fichiers', 'Sous-dossiers', 'Taille Logique', 'Taille Physique', 'Attributs', 'Dernier changement', 'Index'],
'hi': ['नाम', 'फ़ाइलें', 'फ़ोल्डर', 'तार्किक आकार', 'भौतिक आकार', 'विशेषताएँ', 'अंतिम बदलाव', 'सूचकांक'],
'hu': ['Név', 'Fájlok', 'Alkönyvtárak', 'Logikai méret', 'Fizikai méret', 'Attribútumok', 'Utoljára módosítva', 'Index'],
'it': ['Nome', 'File', 'Sottocartella', 'Dimensione logica', 'Dimensione fisica', 'Attributi', 'Ultima modifica', 'Indice'],
'ja': ['名前', 'ファイル', 'フォルダー', '論理サイズ', '物理サイズ', '属性', '最終更新日時', 'インデックス'],
'ko': ['이름', '파일', '폴더', '논리적 크기', '물리적 크기', '속성', '최근 변경', '인덱스'],
'nb': ['Navn', 'Filer', 'Mapper', 'Størrelse', 'Diskbruk', 'Attributter', 'Endringsdato', 'Indeks'],
'nl': ['Naam', 'Bestanden', 'Onderliggende mappen', 'Logische Grootte', 'Fysieke Grootte', 'Kenmerken', 'Laatste wijziging', 'Index'],
'pl': ['Nazwa', 'Plików', 'Podkatalogi', 'Rozmiar logiczny', 'Rozmiar fizyczny', 'Atrybuty', 'Ostatnia modyfikacja', 'Indeks'],
'pt': ['Nome', 'Arquivos', 'Subdiretórios', 'Tamanho Lógico', 'Tamanho Físico', 'Atributos', 'Última Alteração', 'Índice'],
'ru': ['Имя', 'Файлы', 'Подкаталоги', 'Логический размер', 'Физический размер', 'Атрибуты', 'Дата изменения', 'Индекс'],
'sl': ['Ime', 'Datoteke', 'Mape', 'Logična velikost', 'Fizična velikost', 'Atributi', 'Zadnja sprememba', 'Indeks'],
'sv': ['Namn', 'Filer', 'Mappar', 'Logisk storlek', 'Fysisk storlek', 'Attribut', 'Senast ändrad', 'Index'],
'tr': ['Ad', 'Dosyalar', 'Dizinler', 'Mantıksal Boyut', 'Fiziksel Boyut', 'Özellikler', 'Son Değişiklik', 'Dizin'],
'uk': ['Ім’я', 'Файли', 'Підкаталоги', 'Логічний розмір', 'Фізичний розмір', 'Атрибути', 'Дата зміни', 'Індекс'],
'zh-HK': ['名稱', '檔案數目', '資料夾數目', '邏輯大小', '實際大小', '檔案屬性', '最後修改時間', '索引'],
'zh': ['名称', '文件数', '文件夹数', '逻辑大小', '物理大小', '属性', '修改时间', '索引'],
}
# @@LANGS-END@@
def header(lang):
name, files, folders, logical, physical, attrs, last_change, index = LANGS[lang]
return [name, files, folders, logical, physical, attrs, last_change,
"WinDirStat " + attrs, index, "ScanOutcome", "ScanError"]
def clean(name):
"""Make a path component safe for WinDirStat's minimal CSV parser."""
s = os.fsdecode(name).encode("utf-8", "replace").decode("utf-8")
for bad, good in (("\\", "_"), ('"', "'"), ("\n", " "), ("\r", " ")):
s = s.replace(bad, good)
return s
def timestamp(mtime):
try:
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(mtime))
except (OverflowError, OSError, ValueError):
return ""
class Scanner:
def __init__(self, out, cross_fs=False, exclude=(), skip_inodes=(), progress=None):
self.out = out
self.cross_fs = cross_fs
self.exclude = {os.path.realpath(p) for p in exclude}
self.skip_inodes = set(skip_inodes)
self.seen_links = set()
self.progress = progress
self.entries = 0
self.errors = 0
self.root_dev = None
def row(self, path, files, folders, logical, physical, attrs, mtime, itype,
outcome="complete", err=""):
self.out.write(
f'"{path}",{files},{folders},{logical},{physical},"{attrs}",'
f'{timestamp(mtime)},0x{itype:08X},0x{0:016X},"{outcome}","{err}"\n')
def scan(self, root, root_name):
st = os.stat(root)
self.root_dev = st.st_dev
return self._walk(root, clean(root_name), st.st_mtime, "", True)
def _walk(self, dirpath, wpath, mtime, attrs, is_root=False):
# One stack frame per directory level; main() raises the recursion limit.
files = folders = logical = physical = 0
outcome, err = "complete", ""
try:
it = os.scandir(dirpath)
except OSError as e:
it, outcome, err = None, "partial", clean(e.strerror or type(e).__name__)
self.errors += 1
if it is not None:
with it:
for entry in it:
self.entries += 1
if self.progress and self.entries % 100000 == 0:
self.progress(self.entries)
try:
st = entry.stat(follow_symlinks=False)
except OSError:
self.errors += 1
continue
name = clean(entry.name)
child = wpath + "\\" + name
a = "H" if name.startswith(".") else ""
if stat.S_ISDIR(st.st_mode):
full = os.path.join(dirpath, entry.name)
if (not self.cross_fs and st.st_dev != self.root_dev) or \
os.path.realpath(full) in self.exclude:
continue
r = self._walk(full, child, st.st_mtime, a)
files += r[0]
folders += r[1] + 1
logical += r[2]
physical += r[3]
continue
key = (st.st_dev, st.st_ino)
if key in self.skip_inodes:
continue # our own output/temp files
if st.st_nlink > 1:
if key in self.seen_links:
continue # count each hard-linked inode once
self.seen_links.add(key)
lsz, psz = st.st_size, st.st_blocks * 512
self.row(child, 0, 0, lsz, psz, a, st.st_mtime, IT_FILE)
files += 1
logical += lsz
physical += psz
itype = IT_DIRECTORY | (ITF_ROOTITEM if is_root else 0) | \
(ITF_SCANERROR if outcome != "complete" else 0)
self.row(wpath, files, folders, logical, physical, attrs, mtime, itype, outcome, err)
return files, folders, logical, physical
def copy_reversed_lines(src, dst, block=1 << 20):
"""Write the lines of binary file `src` to `dst` in reverse order."""
src.seek(0, os.SEEK_END)
pos = src.tell()
tail = b""
while pos > 0:
step = min(block, pos)
pos -= step
src.seek(pos)
buf = src.read(step) + tail
lines = buf.split(b"\n")
tail = lines[0] # may be incomplete; joined with the previous block
for line in reversed(lines[1:]):
if line:
dst.write(line + b"\n")
if tail:
dst.write(tail + b"\n")
def default_name(path):
host = socket.gethostname().split(".")[0] or "host"
real = os.path.realpath(path)
base = os.path.basename(real.rstrip(os.sep)) or "root"
return f"{host}-{base}"
def human(n):
for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
if n < 1024 or unit == "TiB":
return f"{n:.1f} {unit}" if unit != "B" else f"{n} B"
n /= 1024
def parse_args(argv):
p = argparse.ArgumentParser(
prog="wdsexport",
description="Scan a directory tree and write a WinDirStat 2.x saved-results CSV "
"(open it in WinDirStat via File > Open saved results).")
p.add_argument("path", nargs="?", default="/", help="directory to scan (default: /)")
p.add_argument("-o", "--output",
help="output CSV path, or - for stdout (default: ./<name>.csv)")
p.add_argument("--name", help="root folder name shown in WinDirStat "
"(default: <hostname>-<dirname>, e.g. myhost-root)")
p.add_argument("--exclude", action="append", default=[], metavar="DIR",
help="skip this directory (repeatable)")
p.add_argument("--zip", action="store_true",
help="also write <output>.zip (CSVs compress ~15x)")
p.add_argument("--lang", default="en", choices=sorted(LANGS), metavar="CODE",
help="column-name language; must match WinDirStat's UI language "
f"(default: en; choices: {', '.join(sorted(LANGS))})")
p.add_argument("--cross-fs", action="store_true",
help="descend into other mounted filesystems (default: stay on one, like du -x)")
p.add_argument("-q", "--quiet", action="store_true", help="no progress output")
p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
args = p.parse_args(argv)
if not os.path.isdir(args.path):
p.error(f"not a directory: {args.path}")
if args.zip and args.output == "-":
p.error("--zip cannot be used with --output -")
return args
def main(argv=None):
args = parse_args(argv)
sys.setrecursionlimit(max(sys.getrecursionlimit(), 100000))
name = args.name or default_name(args.path)
to_stdout = args.output == "-"
output = None if to_stdout else os.path.abspath(args.output or f"{clean(name)}.csv")
log = (lambda *a: None) if args.quiet else (lambda msg: print(msg, file=sys.stderr, flush=True))
tty = sys.stderr.isatty() and not args.quiet
t0 = time.monotonic()
def progress(n):
if tty:
print(f"\r{n:,} entries, {time.monotonic() - t0:.0f}s", end="", file=sys.stderr, flush=True)
tmp_dir = tempfile.gettempdir() if to_stdout else os.path.dirname(output)
tmp = tempfile.NamedTemporaryFile("w+b", dir=tmp_dir, prefix=".wdsexport-", suffix=".tmp", delete=False)
skip = set()
for f in [tmp.name] + ([output] if output and os.path.exists(output) else []):
st = os.stat(f)
skip.add((st.st_dev, st.st_ino))
try:
text = open(tmp.fileno(), "w", encoding="utf-8", newline="\n", buffering=1 << 20, closefd=False)
scanner = Scanner(text, cross_fs=args.cross_fs, exclude=args.exclude,
skip_inodes=skip, progress=progress)
files, folders, logical, physical = scanner.scan(args.path, name)
text.flush()
if tty:
print(file=sys.stderr)
log(f"Scanned {scanner.entries:,} entries in {time.monotonic() - t0:.0f}s: "
f"{files:,} files, {folders:,} folders, {human(physical)} on disk")
if scanner.errors:
log(f"{scanner.errors:,} entries could not be read (marked as partial); "
"run with sudo to include them")
head = ("" + ",".join(f'"{h}"' for h in header(args.lang)) + "\n").encode("utf-8")
if to_stdout:
dst = sys.stdout.buffer
dst.write(head)
copy_reversed_lines(tmp, dst)
dst.flush()
else:
# Write next to the target, then rename, so a failed run never
# leaves a truncated CSV behind under the final name.
part = output + ".part"
with open(part, "wb", buffering=1 << 20) as dst:
dst.write(head)
copy_reversed_lines(tmp, dst)
os.replace(part, output)
log(f"Wrote {output} ({human(os.path.getsize(output))})")
if args.zip:
zpath = os.path.splitext(output)[0] + ".zip"
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_DEFLATED, compresslevel=6) as z:
z.write(output, os.path.basename(output))
log(f"Wrote {zpath} ({human(os.path.getsize(zpath))})")
except KeyboardInterrupt:
print("\ninterrupted", file=sys.stderr)
return 130
finally:
tmp.close()
os.unlink(tmp.name)
if output and os.path.exists(output + ".part"):
os.unlink(output + ".part")
return 0
if __name__ == "__main__":
sys.exit(main())