Add wdsexport: Linux directory scan to WinDirStat 2.x saved-results CSV
Single-file, stdlib-only CLI that writes the CSV format WinDirStat's CsvLoader accepts (parent-before-child rows, subtree totals, localized headers for all 25 UI languages). Includes install.sh, pyproject entry point, unit tests and a generator for the language table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
e28fbeb0b3
8 files changed
+637
No files matched your search
@@ -0,0 +1,8 @@
|
||||
__pycache__/
|
||||
*.egg-info/
|
||||
build/
|
||||
dist/
|
||||
*.csv
|
||||
*.zip
|
||||
*.part
|
||||
.wdsexport-*.tmp
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Lucas Winther
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,85 @@
|
||||
# WinDirStat-Export
|
||||
|
||||
`wdsexport` scans a directory tree on Linux (or macOS) and writes a
|
||||
[WinDirStat](https://windirstat.net) 2.x **saved-results CSV**. Copy it to a
|
||||
Windows machine and open it with **File → Open saved results** to browse the
|
||||
remote disk with WinDirStat's treemap.
|
||||
|
||||
Single Python file, standard library only, Python 3.8+.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
git clone https://gitea.lucaswinther.info/lucasw89/WinDirStat-Export.git
|
||||
cd WinDirStat-Export
|
||||
./install.sh # -> ~/.local/bin/wdsexport
|
||||
# or: ./install.sh /usr/local (with sudo)
|
||||
# or: pipx install .
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
```sh
|
||||
wdsexport # scan /, write ./<hostname>-root.csv
|
||||
wdsexport / --zip # also write a .zip (CSVs compress ~15x)
|
||||
sudo wdsexport / # include root-only dirs (/var/lib/docker, ...)
|
||||
wdsexport ~/projects -o proj.csv # scan one directory
|
||||
wdsexport / --exclude /home/me/vms --lang da
|
||||
wdsexport / -o - | gzip > scan.csv.gz
|
||||
```
|
||||
|
||||
| Argument | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `PATH` | `/` | Directory to scan |
|
||||
| `-o, --output FILE` | `./<name>.csv` | Output CSV; `-` writes to stdout |
|
||||
| `--name NAME` | `<hostname>-<dirname>` | Root folder name shown in WinDirStat |
|
||||
| `--exclude DIR` | — | Skip a directory (repeatable) |
|
||||
| `--zip` | off | Also write `<output>.zip` |
|
||||
| `--lang CODE` | `en` | Column-name language (see below) |
|
||||
| `--cross-fs` | off | Descend into other mounted filesystems (default stays on one, like `du -x`) |
|
||||
| `-q, --quiet` | off | No progress output |
|
||||
|
||||
### `--lang` must match WinDirStat's UI language
|
||||
|
||||
WinDirStat identifies CSV columns by their *localized* header names. If your
|
||||
WinDirStat runs in Danish, export with `--lang da`; otherwise the file is
|
||||
rejected. Supported: ar, cs, da, de, en, es, et, fi, fr, hi, hu, it, ja, ko,
|
||||
nb, nl, pl, pt, ru, sl, sv, tr, uk, zh, zh-HK.
|
||||
|
||||
## What gets recorded
|
||||
|
||||
- **Logical size** = file length; **physical size** = allocated blocks
|
||||
(`st_blocks × 512`), so sparse files and filesystem overhead show correctly.
|
||||
- Hard-linked files are counted once (first path seen).
|
||||
- Symlinks are recorded as small files and never followed.
|
||||
- Dotfiles get the Hidden attribute, so enable *show hidden files* in WinDirStat.
|
||||
- Directories that can't be read are marked *partial* (WinDirStat shows them as
|
||||
incomplete). Run with `sudo` to include them.
|
||||
- Characters WinDirStat's CSV parser can't handle in names are replaced:
|
||||
`\` → `_`, `"` → `'`, newlines → space. Invalid UTF-8 becomes `�`.
|
||||
- The tool skips its own output and temporary files when they sit inside the
|
||||
scanned tree.
|
||||
|
||||
Disk space: the scan writes a temporary file about the size of the final CSV
|
||||
next to the output (in `$TMPDIR` for `-o -`). Roughly 200 bytes per file, so
|
||||
8 million files ≈ 1.5 GB CSV, ≈ 100 MB zipped.
|
||||
|
||||
## How it works
|
||||
|
||||
The format follows WinDirStat's own loader
|
||||
([`CsvLoader.cpp`](https://github.com/windirstat/windirstat/blob/master/windirstat/CsvLoader.cpp)):
|
||||
rows must list each parent before its children, paths use `\` separators, and
|
||||
folder rows carry subtree totals. `wdsexport` writes rows in post-order (when
|
||||
folder totals are known) to a temp file and then copies it out reversed, which
|
||||
gives a parent-first order without holding the tree in memory.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s tests
|
||||
python3 tools/gen_langs.py # refresh column names from WinDirStat's language files
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/bin/sh
|
||||
# Install wdsexport as a single executable file (no dependencies beyond python3).
|
||||
# Usage: ./install.sh [PREFIX] default PREFIX: ~/.local (installs to PREFIX/bin/wdsexport)
|
||||
set -eu
|
||||
prefix="${1:-$HOME/.local}"
|
||||
src="$(cd "$(dirname "$0")" && pwd)/wdsexport.py"
|
||||
command -v python3 >/dev/null || { echo "python3 not found" >&2; exit 1; }
|
||||
mkdir -p "$prefix/bin"
|
||||
install -m 755 "$src" "$prefix/bin/wdsexport"
|
||||
echo "installed $prefix/bin/wdsexport"
|
||||
case ":$PATH:" in
|
||||
*":$prefix/bin:"*) ;;
|
||||
*) echo "note: $prefix/bin is not on your PATH" >&2 ;;
|
||||
esac
|
||||
@@ -0,0 +1,21 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=61"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "wdsexport"
|
||||
dynamic = ["version"]
|
||||
description = "Scan a Linux/macOS directory tree and write a WinDirStat 2.x saved-results CSV"
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
requires-python = ">=3.8"
|
||||
dependencies = []
|
||||
|
||||
[project.scripts]
|
||||
wdsexport = "wdsexport:main"
|
||||
|
||||
[tool.setuptools]
|
||||
py-modules = ["wdsexport"]
|
||||
|
||||
[tool.setuptools.dynamic]
|
||||
version = { attr = "wdsexport.__version__" }
|
||||
@@ -0,0 +1,147 @@
|
||||
import contextlib
|
||||
import csv
|
||||
import io
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
import zipfile
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||
import wdsexport # noqa: E402
|
||||
|
||||
SCRIPT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "wdsexport.py")
|
||||
|
||||
|
||||
def read_rows(path_or_text):
|
||||
text = path_or_text if "\n" in path_or_text else open(path_or_text, encoding="utf-8").read()
|
||||
assert text.startswith("\ufeff"), "missing UTF-8 BOM"
|
||||
rows = list(csv.reader(io.StringIO(text[1:])))
|
||||
return rows[0], rows[1:]
|
||||
|
||||
|
||||
def check_loader_invariants(testcase, rows):
|
||||
"""Mirror the constraints of WinDirStat's CsvLoader: one root container first,
|
||||
every other row's parent (prefix before last backslash) already seen as a container."""
|
||||
containers = set()
|
||||
for i, r in enumerate(rows):
|
||||
name, itype = r[0], int(r[7], 16)
|
||||
if i == 0:
|
||||
testcase.assertTrue(itype & wdsexport.ITF_ROOTITEM)
|
||||
else:
|
||||
testcase.assertFalse(itype & wdsexport.ITF_ROOTITEM)
|
||||
testcase.assertIn(name.rsplit("\\", 1)[0], containers, f"orphan row {name}")
|
||||
if itype & wdsexport.IT_DIRECTORY:
|
||||
containers.add(name)
|
||||
|
||||
|
||||
class ExportTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
self.root = os.path.join(self.tmp.name, "tree")
|
||||
os.makedirs(os.path.join(self.root, "a", "b"))
|
||||
os.makedirs(os.path.join(self.root, "skipme"))
|
||||
os.makedirs(os.path.join(self.root, ".hidden"))
|
||||
self.write("a/one.bin", 1000)
|
||||
self.write("a/b/two.bin", 5000)
|
||||
self.write('a/b/we,ird "na\\me".txt', 10)
|
||||
self.write("skipme/big.bin", 100000)
|
||||
self.write(".hidden/x", 7)
|
||||
os.link(os.path.join(self.root, "a/one.bin"), os.path.join(self.root, "a/b/hardlink.bin"))
|
||||
os.symlink("a/one.bin", os.path.join(self.root, "link"))
|
||||
self.locked = os.path.join(self.root, "locked")
|
||||
os.makedirs(self.locked)
|
||||
self.write("locked/secret", 3)
|
||||
os.chmod(self.locked, 0)
|
||||
self.out = os.path.join(self.tmp.name, "out.csv")
|
||||
|
||||
def tearDown(self):
|
||||
os.chmod(self.locked, 0o755)
|
||||
self.tmp.cleanup()
|
||||
|
||||
def write(self, rel, size):
|
||||
with open(os.path.join(self.root, rel), "wb") as f:
|
||||
f.write(b"x" * size)
|
||||
|
||||
def run_main(self, *args):
|
||||
err = io.StringIO()
|
||||
with contextlib.redirect_stderr(err):
|
||||
rc = wdsexport.main(list(args))
|
||||
self.assertEqual(rc, 0, err.getvalue())
|
||||
return err.getvalue()
|
||||
|
||||
def test_structure_and_totals(self):
|
||||
self.run_main(self.root, "-o", self.out, "--name", "test", "--exclude",
|
||||
os.path.join(self.root, "skipme"), "-q")
|
||||
header, rows = read_rows(self.out)
|
||||
self.assertEqual(header, wdsexport.header("en"))
|
||||
check_loader_invariants(self, rows)
|
||||
by_name = {r[0]: r for r in rows}
|
||||
|
||||
root = rows[0]
|
||||
self.assertEqual(root[0], "test")
|
||||
symlink_size = os.lstat(os.path.join(self.root, "link")).st_size
|
||||
# one.bin counted once despite the hard link; symlink counted as itself
|
||||
self.assertEqual(int(root[3]), 1000 + 5000 + 10 + 7 + symlink_size)
|
||||
# files: one.bin, two.bin, weird, x, link (hardlink and locked contents not counted)
|
||||
self.assertEqual(int(root[1]), 5)
|
||||
# folders: a, a/b, .hidden, locked
|
||||
self.assertEqual(int(root[2]), 4)
|
||||
|
||||
self.assertNotIn("test\\skipme", by_name)
|
||||
# which hard-link path gets the bytes depends on scandir order; both live under a/
|
||||
self.assertEqual(by_name["test\\a"][3], "6010")
|
||||
self.assertIn("test\\a\\b\\we,ird 'na_me'.txt", by_name)
|
||||
self.assertEqual(by_name["test\\.hidden"][5], "H")
|
||||
self.assertEqual(by_name["test\\locked"][9], "partial")
|
||||
self.assertTrue(int(by_name["test\\locked"][7], 16) & wdsexport.ITF_SCANERROR)
|
||||
self.assertRegex(root[6], r"^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$")
|
||||
|
||||
# each CSV line must have exactly 8 double quotes (loader has no escaping)
|
||||
with open(self.out, encoding="utf-8") as f:
|
||||
next(f)
|
||||
for line in f:
|
||||
self.assertEqual(line.count('"'), 8, line)
|
||||
|
||||
def test_language_headers(self):
|
||||
self.run_main(self.root, "-o", self.out, "--lang", "da", "-q")
|
||||
header, _ = read_rows(self.out)
|
||||
self.assertEqual(header[:3], ["Navn", "Filer", "Mapper"])
|
||||
self.assertEqual(header[7], "WinDirStat Attributter")
|
||||
|
||||
def test_stdout_matches_file(self):
|
||||
self.run_main(self.root, "-o", self.out, "--name", "n", "-q")
|
||||
res = subprocess.run([sys.executable, SCRIPT, self.root, "-o", "-", "--name", "n", "-q"],
|
||||
capture_output=True, check=True)
|
||||
self.assertEqual(res.stdout.decode("utf-8"), open(self.out, encoding="utf-8").read())
|
||||
|
||||
def test_zip(self):
|
||||
self.run_main(self.root, "-o", self.out, "--zip", "-q")
|
||||
with zipfile.ZipFile(self.out[:-4] + ".zip") as z:
|
||||
self.assertEqual(z.read("out.csv"), open(self.out, "rb").read())
|
||||
|
||||
def test_output_inside_scanned_tree_is_skipped(self):
|
||||
out = os.path.join(self.root, "inside.csv")
|
||||
self.run_main(self.root, "-o", out, "--name", "t", "-q")
|
||||
names = [r[0] for r in read_rows(out)[1]]
|
||||
self.assertNotIn("t\\inside.csv", names)
|
||||
self.assertFalse([n for n in names if ".wdsexport-" in n or n.endswith(".part")])
|
||||
self.assertFalse([f for f in os.listdir(self.root) if f.startswith(".wdsexport-")])
|
||||
|
||||
def test_default_name(self):
|
||||
self.assertTrue(wdsexport.default_name("/").endswith("-root"))
|
||||
self.assertTrue(wdsexport.default_name(self.root).endswith("-tree"))
|
||||
|
||||
|
||||
class ReverseLinesTest(unittest.TestCase):
|
||||
def test_small_blocks(self):
|
||||
lines = [f"line {i} {'x' * (i % 7)}".encode() for i in range(500)]
|
||||
src = io.BytesIO(b"\n".join(lines) + b"\n")
|
||||
dst = io.BytesIO()
|
||||
wdsexport.copy_reversed_lines(src, dst, block=13)
|
||||
self.assertEqual(dst.getvalue(), b"\n".join(reversed(lines)) + b"\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Executable
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regenerate the LANGS table in wdsexport.py from WinDirStat's language files.
|
||||
|
||||
Usage: python3 tools/gen_langs.py [REF] (REF defaults to master)
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
REPO = "windirstat/windirstat"
|
||||
KEYS = ["IDS_COL_NAME", "IDS_COL_FILES", "IDS_COL_FOLDERS", "IDS_COL_SIZE_LOGICAL",
|
||||
"IDS_COL_SIZE_PHYSICAL", "IDS_COL_ATTRIBUTES", "IDS_COL_LAST_CHANGE", "IDS_COL_INDEX"]
|
||||
TARGET = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "wdsexport.py")
|
||||
|
||||
|
||||
def fetch(url):
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "wdsexport-gen-langs"})
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
return r.read()
|
||||
|
||||
|
||||
def main():
|
||||
ref = sys.argv[1] if len(sys.argv) > 1 else "master"
|
||||
tree = json.loads(fetch(f"https://api.github.com/repos/{REPO}/git/trees/{ref}?recursive=1"))
|
||||
paths = sorted(t["path"] for t in tree["tree"]
|
||||
if re.fullmatch(r"windirstat/res/langs/lang_[\w-]+\.txt", t["path"]))
|
||||
langs = {}
|
||||
for path in paths:
|
||||
code = re.search(r"lang_([\w-]+)\.txt$", path).group(1)
|
||||
text = fetch(f"https://raw.githubusercontent.com/{REPO}/{ref}/{path}").decode("utf-8-sig")
|
||||
values = dict(line.split("=", 1) for line in text.splitlines() if "=" in line)
|
||||
missing = [k for k in KEYS if k not in values]
|
||||
if missing:
|
||||
print(f"skipping {code}: missing {missing}", file=sys.stderr)
|
||||
continue
|
||||
row = [values[k] for k in KEYS]
|
||||
if any('"' in v for v in row):
|
||||
print(f"skipping {code}: column name contains a double quote", file=sys.stderr)
|
||||
continue
|
||||
langs[code] = row
|
||||
|
||||
body = "LANGS = {\n" + "".join(f" {c!r}: {v!r},\n" for c, v in langs.items()) + "}\n"
|
||||
src = open(TARGET, encoding="utf-8").read()
|
||||
new = re.sub(r"(# @@LANGS-BEGIN@@\n).*?(# @@LANGS-END@@)",
|
||||
lambda m: m.group(1) + body + m.group(2), src, flags=re.S)
|
||||
open(TARGET, "w", encoding="utf-8").write(new)
|
||||
print(f"wrote {len(langs)} languages to {os.path.normpath(TARGET)}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+288
@@ -0,0 +1,288 @@
|
||||
#!/usr/bin/env python3
|
||||
"""wdsexport - scan a Linux/macOS directory tree and write a WinDirStat 2.x
|
||||
"saved results" CSV, to be opened on Windows via File > Open saved results.
|
||||
|
||||
Format notes (from windirstat/CsvLoader.cpp):
|
||||
* Header names must match WinDirStat's UI language exactly.
|
||||
* Rows must be ordered parent-before-child; parents are found by the path
|
||||
prefix before the last backslash.
|
||||
* Container rows carry subtree totals (the loader does not sum children).
|
||||
|
||||
Rows are written in post-order (when subtree totals are known) to a temp file,
|
||||
which is then copied out in reverse: reversed post-order is parent-first.
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import socket
|
||||
import stat
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import zipfile
|
||||
|
||||
__version__ = "1.0.0"
|
||||
|
||||
IT_DIRECTORY = 1 << 2
|
||||
IT_FILE = 1 << 3
|
||||
ITF_SCANERROR = 1 << 14
|
||||
ITF_ROOTITEM = 1 << 28
|
||||
|
||||
# Localized column names from windirstat/res/langs/lang_*.txt, in the order:
|
||||
# Name, Files, Folders, Logical Size, Physical Size, Attributes, Last Change, Index.
|
||||
# Regenerate with tools/gen_langs.py.
|
||||
# @@LANGS-BEGIN@@
|
||||
LANGS = {
|
||||
'ar': ['الاسم', 'الملفات', 'المجلدات الفرعية', 'الحجم المنطقي', 'الحجم الفعلي على القرص', 'السمات', 'آخر تعديل', 'الفهرس'],
|
||||
'cs': ['Název', 'Soubory', 'Složky', 'Logická velikost', 'Fyzická velikost', 'Atributy', 'Poslední změna', 'Index'],
|
||||
'da': ['Navn', 'Filer', 'Mapper', 'Logisk størrelse', 'Fysisk størrelse', 'Attributter', 'Sidste ændring', 'Indeks'],
|
||||
'de': ['Name', 'Dateien', 'Verzeichnisse', 'Logische Größe', 'Physische Größe', 'Attribute', 'Letzte Änderung', 'Index'],
|
||||
'en': ['Name', 'Files', 'Folders', 'Logical Size', 'Physical Size', 'Attributes', 'Last Change', 'Index'],
|
||||
'es': ['Nombre', 'Archivos', 'Subdirectorios', 'Tamaño Lógico', 'Tamaño Físico', 'Atributos', 'Último Cambio', 'Índice'],
|
||||
'et': ['Nimi', 'Failid', 'Alamkataloogid', 'Loogiline suurus', 'Füüsiline suurus', 'Atribuudid', 'Viimane muudatus', 'Indeks'],
|
||||
'fi': ['Nimi', 'Tiedostoja', 'Alikansioita', 'Looginen koko', 'Fyysinen koko', 'Määritteet', 'Edellinen muutos', 'Indeksi'],
|
||||
'fr': ['Nom', 'Fichiers', 'Sous-dossiers', 'Taille Logique', 'Taille Physique', 'Attributs', 'Dernier changement', 'Index'],
|
||||
'hi': ['नाम', 'फ़ाइलें', 'फ़ोल्डर', 'तार्किक आकार', 'भौतिक आकार', 'विशेषताएँ', 'अंतिम बदलाव', 'सूचकांक'],
|
||||
'hu': ['Név', 'Fájlok', 'Alkönyvtárak', 'Logikai méret', 'Fizikai méret', 'Attribútumok', 'Utoljára módosítva', 'Index'],
|
||||
'it': ['Nome', 'File', 'Sottocartella', 'Dimensione logica', 'Dimensione fisica', 'Attributi', 'Ultima modifica', 'Indice'],
|
||||
'ja': ['名前', 'ファイル', 'フォルダー', '論理サイズ', '物理サイズ', '属性', '最終更新日時', 'インデックス'],
|
||||
'ko': ['이름', '파일', '폴더', '논리적 크기', '물리적 크기', '속성', '최근 변경', '인덱스'],
|
||||
'nb': ['Navn', 'Filer', 'Mapper', 'Størrelse', 'Diskbruk', 'Attributter', 'Endringsdato', 'Indeks'],
|
||||
'nl': ['Naam', 'Bestanden', 'Onderliggende mappen', 'Logische Grootte', 'Fysieke Grootte', 'Kenmerken', 'Laatste wijziging', 'Index'],
|
||||
'pl': ['Nazwa', 'Plików', 'Podkatalogi', 'Rozmiar logiczny', 'Rozmiar fizyczny', 'Atrybuty', 'Ostatnia modyfikacja', 'Indeks'],
|
||||
'pt': ['Nome', 'Arquivos', 'Subdiretórios', 'Tamanho Lógico', 'Tamanho Físico', 'Atributos', 'Última Alteração', 'Índice'],
|
||||
'ru': ['Имя', 'Файлы', 'Подкаталоги', 'Логический размер', 'Физический размер', 'Атрибуты', 'Дата изменения', 'Индекс'],
|
||||
'sl': ['Ime', 'Datoteke', 'Mape', 'Logična velikost', 'Fizična velikost', 'Atributi', 'Zadnja sprememba', 'Indeks'],
|
||||
'sv': ['Namn', 'Filer', 'Mappar', 'Logisk storlek', 'Fysisk storlek', 'Attribut', 'Senast ändrad', 'Index'],
|
||||
'tr': ['Ad', 'Dosyalar', 'Dizinler', 'Mantıksal Boyut', 'Fiziksel Boyut', 'Özellikler', 'Son Değişiklik', 'Dizin'],
|
||||
'uk': ['Ім’я', 'Файли', 'Підкаталоги', 'Логічний розмір', 'Фізичний розмір', 'Атрибути', 'Дата зміни', 'Індекс'],
|
||||
'zh-HK': ['名稱', '檔案數目', '資料夾數目', '邏輯大小', '實際大小', '檔案屬性', '最後修改時間', '索引'],
|
||||
'zh': ['名称', '文件数', '文件夹数', '逻辑大小', '物理大小', '属性', '修改时间', '索引'],
|
||||
}
|
||||
# @@LANGS-END@@
|
||||
|
||||
|
||||
def header(lang):
|
||||
name, files, folders, logical, physical, attrs, last_change, index = LANGS[lang]
|
||||
return [name, files, folders, logical, physical, attrs, last_change,
|
||||
"WinDirStat " + attrs, index, "ScanOutcome", "ScanError"]
|
||||
|
||||
|
||||
def clean(name):
|
||||
"""Make a path component safe for WinDirStat's minimal CSV parser."""
|
||||
s = os.fsdecode(name).encode("utf-8", "replace").decode("utf-8")
|
||||
for bad, good in (("\\", "_"), ('"', "'"), ("\n", " "), ("\r", " ")):
|
||||
s = s.replace(bad, good)
|
||||
return s
|
||||
|
||||
|
||||
def timestamp(mtime):
|
||||
try:
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(mtime))
|
||||
except (OverflowError, OSError, ValueError):
|
||||
return ""
|
||||
|
||||
|
||||
class Scanner:
|
||||
def __init__(self, out, cross_fs=False, exclude=(), skip_inodes=(), progress=None):
|
||||
self.out = out
|
||||
self.cross_fs = cross_fs
|
||||
self.exclude = {os.path.realpath(p) for p in exclude}
|
||||
self.skip_inodes = set(skip_inodes)
|
||||
self.seen_links = set()
|
||||
self.progress = progress
|
||||
self.entries = 0
|
||||
self.errors = 0
|
||||
self.root_dev = None
|
||||
|
||||
def row(self, path, files, folders, logical, physical, attrs, mtime, itype,
|
||||
outcome="complete", err=""):
|
||||
self.out.write(
|
||||
f'"{path}",{files},{folders},{logical},{physical},"{attrs}",'
|
||||
f'{timestamp(mtime)},0x{itype:08X},0x{0:016X},"{outcome}","{err}"\n')
|
||||
|
||||
def scan(self, root, root_name):
|
||||
st = os.stat(root)
|
||||
self.root_dev = st.st_dev
|
||||
return self._walk(root, clean(root_name), st.st_mtime, "", True)
|
||||
|
||||
def _walk(self, dirpath, wpath, mtime, attrs, is_root=False):
|
||||
# One stack frame per directory level; main() raises the recursion limit.
|
||||
files = folders = logical = physical = 0
|
||||
outcome, err = "complete", ""
|
||||
try:
|
||||
it = os.scandir(dirpath)
|
||||
except OSError as e:
|
||||
it, outcome, err = None, "partial", clean(e.strerror or type(e).__name__)
|
||||
self.errors += 1
|
||||
if it is not None:
|
||||
with it:
|
||||
for entry in it:
|
||||
self.entries += 1
|
||||
if self.progress and self.entries % 100000 == 0:
|
||||
self.progress(self.entries)
|
||||
try:
|
||||
st = entry.stat(follow_symlinks=False)
|
||||
except OSError:
|
||||
self.errors += 1
|
||||
continue
|
||||
name = clean(entry.name)
|
||||
child = wpath + "\\" + name
|
||||
a = "H" if name.startswith(".") else ""
|
||||
if stat.S_ISDIR(st.st_mode):
|
||||
full = os.path.join(dirpath, entry.name)
|
||||
if (not self.cross_fs and st.st_dev != self.root_dev) or \
|
||||
os.path.realpath(full) in self.exclude:
|
||||
continue
|
||||
r = self._walk(full, child, st.st_mtime, a)
|
||||
files += r[0]
|
||||
folders += r[1] + 1
|
||||
logical += r[2]
|
||||
physical += r[3]
|
||||
continue
|
||||
key = (st.st_dev, st.st_ino)
|
||||
if key in self.skip_inodes:
|
||||
continue # our own output/temp files
|
||||
if st.st_nlink > 1:
|
||||
if key in self.seen_links:
|
||||
continue # count each hard-linked inode once
|
||||
self.seen_links.add(key)
|
||||
lsz, psz = st.st_size, st.st_blocks * 512
|
||||
self.row(child, 0, 0, lsz, psz, a, st.st_mtime, IT_FILE)
|
||||
files += 1
|
||||
logical += lsz
|
||||
physical += psz
|
||||
itype = IT_DIRECTORY | (ITF_ROOTITEM if is_root else 0) | \
|
||||
(ITF_SCANERROR if outcome != "complete" else 0)
|
||||
self.row(wpath, files, folders, logical, physical, attrs, mtime, itype, outcome, err)
|
||||
return files, folders, logical, physical
|
||||
|
||||
|
||||
def copy_reversed_lines(src, dst, block=1 << 20):
|
||||
"""Write the lines of binary file `src` to `dst` in reverse order."""
|
||||
src.seek(0, os.SEEK_END)
|
||||
pos = src.tell()
|
||||
tail = b""
|
||||
while pos > 0:
|
||||
step = min(block, pos)
|
||||
pos -= step
|
||||
src.seek(pos)
|
||||
buf = src.read(step) + tail
|
||||
lines = buf.split(b"\n")
|
||||
tail = lines[0] # may be incomplete; joined with the previous block
|
||||
for line in reversed(lines[1:]):
|
||||
if line:
|
||||
dst.write(line + b"\n")
|
||||
if tail:
|
||||
dst.write(tail + b"\n")
|
||||
|
||||
|
||||
def default_name(path):
|
||||
host = socket.gethostname().split(".")[0] or "host"
|
||||
real = os.path.realpath(path)
|
||||
base = os.path.basename(real.rstrip(os.sep)) or "root"
|
||||
return f"{host}-{base}"
|
||||
|
||||
|
||||
def human(n):
|
||||
for unit in ("B", "KiB", "MiB", "GiB", "TiB"):
|
||||
if n < 1024 or unit == "TiB":
|
||||
return f"{n:.1f} {unit}" if unit != "B" else f"{n} B"
|
||||
n /= 1024
|
||||
|
||||
|
||||
def parse_args(argv):
|
||||
p = argparse.ArgumentParser(
|
||||
prog="wdsexport",
|
||||
description="Scan a directory tree and write a WinDirStat 2.x saved-results CSV "
|
||||
"(open it in WinDirStat via File > Open saved results).")
|
||||
p.add_argument("path", nargs="?", default="/", help="directory to scan (default: /)")
|
||||
p.add_argument("-o", "--output",
|
||||
help="output CSV path, or - for stdout (default: ./<name>.csv)")
|
||||
p.add_argument("--name", help="root folder name shown in WinDirStat "
|
||||
"(default: <hostname>-<dirname>, e.g. myhost-root)")
|
||||
p.add_argument("--exclude", action="append", default=[], metavar="DIR",
|
||||
help="skip this directory (repeatable)")
|
||||
p.add_argument("--zip", action="store_true",
|
||||
help="also write <output>.zip (CSVs compress ~15x)")
|
||||
p.add_argument("--lang", default="en", choices=sorted(LANGS), metavar="CODE",
|
||||
help="column-name language; must match WinDirStat's UI language "
|
||||
f"(default: en; choices: {', '.join(sorted(LANGS))})")
|
||||
p.add_argument("--cross-fs", action="store_true",
|
||||
help="descend into other mounted filesystems (default: stay on one, like du -x)")
|
||||
p.add_argument("-q", "--quiet", action="store_true", help="no progress output")
|
||||
p.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
|
||||
args = p.parse_args(argv)
|
||||
if not os.path.isdir(args.path):
|
||||
p.error(f"not a directory: {args.path}")
|
||||
if args.zip and args.output == "-":
|
||||
p.error("--zip cannot be used with --output -")
|
||||
return args
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
args = parse_args(argv)
|
||||
sys.setrecursionlimit(max(sys.getrecursionlimit(), 100000))
|
||||
name = args.name or default_name(args.path)
|
||||
to_stdout = args.output == "-"
|
||||
output = None if to_stdout else os.path.abspath(args.output or f"{clean(name)}.csv")
|
||||
log = (lambda *a: None) if args.quiet else (lambda msg: print(msg, file=sys.stderr, flush=True))
|
||||
tty = sys.stderr.isatty() and not args.quiet
|
||||
t0 = time.monotonic()
|
||||
|
||||
def progress(n):
|
||||
if tty:
|
||||
print(f"\r{n:,} entries, {time.monotonic() - t0:.0f}s", end="", file=sys.stderr, flush=True)
|
||||
|
||||
tmp_dir = tempfile.gettempdir() if to_stdout else os.path.dirname(output)
|
||||
tmp = tempfile.NamedTemporaryFile("w+b", dir=tmp_dir, prefix=".wdsexport-", suffix=".tmp", delete=False)
|
||||
skip = set()
|
||||
for f in [tmp.name] + ([output] if output and os.path.exists(output) else []):
|
||||
st = os.stat(f)
|
||||
skip.add((st.st_dev, st.st_ino))
|
||||
try:
|
||||
text = open(tmp.fileno(), "w", encoding="utf-8", newline="\n", buffering=1 << 20, closefd=False)
|
||||
scanner = Scanner(text, cross_fs=args.cross_fs, exclude=args.exclude,
|
||||
skip_inodes=skip, progress=progress)
|
||||
files, folders, logical, physical = scanner.scan(args.path, name)
|
||||
text.flush()
|
||||
if tty:
|
||||
print(file=sys.stderr)
|
||||
log(f"Scanned {scanner.entries:,} entries in {time.monotonic() - t0:.0f}s: "
|
||||
f"{files:,} files, {folders:,} folders, {human(physical)} on disk")
|
||||
if scanner.errors:
|
||||
log(f"{scanner.errors:,} entries could not be read (marked as partial); "
|
||||
"run with sudo to include them")
|
||||
|
||||
head = ("" + ",".join(f'"{h}"' for h in header(args.lang)) + "\n").encode("utf-8")
|
||||
if to_stdout:
|
||||
dst = sys.stdout.buffer
|
||||
dst.write(head)
|
||||
copy_reversed_lines(tmp, dst)
|
||||
dst.flush()
|
||||
else:
|
||||
# Write next to the target, then rename, so a failed run never
|
||||
# leaves a truncated CSV behind under the final name.
|
||||
part = output + ".part"
|
||||
with open(part, "wb", buffering=1 << 20) as dst:
|
||||
dst.write(head)
|
||||
copy_reversed_lines(tmp, dst)
|
||||
os.replace(part, output)
|
||||
log(f"Wrote {output} ({human(os.path.getsize(output))})")
|
||||
if args.zip:
|
||||
zpath = os.path.splitext(output)[0] + ".zip"
|
||||
with zipfile.ZipFile(zpath, "w", zipfile.ZIP_DEFLATED, compresslevel=6) as z:
|
||||
z.write(output, os.path.basename(output))
|
||||
log(f"Wrote {zpath} ({human(os.path.getsize(zpath))})")
|
||||
except KeyboardInterrupt:
|
||||
print("\ninterrupted", file=sys.stderr)
|
||||
return 130
|
||||
finally:
|
||||
tmp.close()
|
||||
os.unlink(tmp.name)
|
||||
if output and os.path.exists(output + ".part"):
|
||||
os.unlink(output + ".part")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user