fix: bind deployment state to local data directories

.gitignore

@@ -26,6 +26,7 @@ coverage.xml
 htmlcov/
 
 # Local runtime state
+data/
 *.sqlite3
 *.sqlite3-shm
 *.sqlite3-wal

README.md

@@ -44,6 +44,7 @@ RECOLL_MODERN_ADMIN_PASSWORD=use-a-long-unique-password
 RECOLL_MODERN_IMAGE=gitea.lucaswinther.info/lucasw89/recoll-modern:main
 EOF
 docker compose -f compose.published.yaml pull
+sudo install -d -m 0700 -o 10001 -g 10001 data/service-state data/recoll-index
 docker compose -f compose.published.yaml up -d --no-build
 docker compose -f compose.published.yaml ps
 docker compose -f compose.published.yaml logs -f app
@@ -54,8 +55,8 @@ Once a version appears in Gitea Packages, edit `RECOLL_MODERN_IMAGE` in `.env` t
 `RECOLL_MODERN_IMAGE` to use `:latest` after the first version tag is published. The image is published
 only after the CI checks pass; a tag that has not been published cannot be pulled. To update, set the
 desired tag and run `docker compose -f compose.published.yaml pull` followed
-by `docker compose -f compose.published.yaml up -d --no-build` again. Keep the same Compose project
+by `docker compose -f compose.published.yaml up -d --no-build` again. Keep the same Compose file
-directory so its named volumes are reused.
+directory so the local data paths are reused.
 
 For a trusted local HTTP trial with `docker run`, export the settings for that shell:
 
@@ -65,13 +66,14 @@ export RECOLL_MODERN_ADMIN_USERNAME=admin
 export RECOLL_MODERN_ADMIN_PASSWORD='use-a-long-unique-password'
 export RECOLL_MODERN_IMAGE=gitea.lucaswinther.info/lucasw89/recoll-modern:main
 docker pull "$RECOLL_MODERN_IMAGE"
+sudo install -d -m 0700 -o 10001 -g 10001 data/service-state data/recoll-index
 docker run -d --name recoll-modern --restart unless-stopped -p 127.0.0.1:8000:8000 \
   -e RECOLL_MODERN_ADMIN_USERNAME -e RECOLL_MODERN_ADMIN_PASSWORD \
   -e RECOLL_MODERN_HOST=0.0.0.0 -e RECOLL_MODERN_PORT=8000 \
   -e RECOLL_MODERN_DEVELOPMENT_MODE=true \
   --mount "type=bind,src=$RECOLL_MODERN_DOCUMENTS,dst=/documents,readonly" \
-  --mount type=volume,src=recoll-modern-service-state,dst=/var/lib/recoll-modern \
+  --mount "type=bind,src=$PWD/data/service-state,dst=/var/lib/recoll-modern" \
-  --mount type=volume,src=recoll-modern-index,dst=/var/lib/recoll \
+  --mount "type=bind,src=$PWD/data/recoll-index,dst=/var/lib/recoll" \
   "$RECOLL_MODERN_IMAGE"
 ```
 
@@ -88,19 +90,21 @@ docker compose up -d --build
 
 The service listens on host port 8000 by default; set `RECOLL_MODERN_PUBLISH_PORT` to change it. The
 document directory is mounted read-only. SQLite, the membership map and generated Recoll configuration
-live in the `service-state` volume; the index lives in the separate `recoll-index` volume. Back up both
+live in `./data/service-state`; the index lives in `./data/recoll-index`, relative to the Compose file
-volumes together and keep a separate copy of the documents. `docker compose down` preserves the volumes;
+directory. Both directories must be writable by container UID/GID `10001:10001`; create them with the
-never pass `-v` unless you intentionally want to delete the index and service state.
+command above before starting either deployment. Back up both directories together and keep a separate
+copy of the documents. Container replacement does not delete these bind-mounted directories. If you
+already deployed with the earlier named volumes, copy both volumes' contents into these directories
+while the old container is stopped, preserving ownership, before starting the new configuration. Do not
+start against empty directories in place of existing data.
 
-For the direct `docker run` example, the volumes are named `recoll-modern-service-state` and
+On first boot, the API is available but `/health` can return 503 with a Recoll database-open error until
-`recoll-modern-index`. Back up both together. Replacing a container does not delete these volumes.
+a membership or reindex job has built the first index. Log in through `POST /auth/login`, create a
-
+collection, then add file or directory sources through `POST /collections/{id}/memberships`. Mutations
-On first boot, the API is available but `/health` may return 503 until a membership or reindex job has
+return a durable job ID; inspect `GET /jobs/{id}` for completion or file errors. `POST /jobs/reindex`
-built the first Recoll index. Log in through `POST /auth/login`, create a collection, then add file or
+refreshes the index without deleting it. The login seed applies only when no administrator exists;
-directory sources through `POST /collections/{id}/memberships`. Mutations return a durable job ID;
+changing the environment value later does not rotate an existing password. Protect and back up
-inspect `GET /jobs/{id}` for completion or file errors. `POST /jobs/reindex` refreshes the index without
+`./data/service-state`.
-deleting it. The login seed applies only when no administrator exists; changing the environment value
-later does not rotate an existing password. Protect and back up the service-state volume.
 
 Production sessions use Secure cookies, so serve the API over HTTPS. For a trusted local HTTP trial,
 set `RECOLL_MODERN_DEVELOPMENT_MODE=true`; do not use that setting on a public endpoint. Set

compose.published.yaml

@@ -14,8 +14,8 @@ services:
       RECOLL_MODERN_MEMBERSHIP_MAP: /var/lib/recoll-modern/membership.map
     volumes:
       - ${RECOLL_MODERN_DOCUMENTS:?Set documents directory}:/documents:ro
-      - service-state:/var/lib/recoll-modern
+      - ./data/service-state:/var/lib/recoll-modern
-      - recoll-index:/var/lib/recoll
+      - ./data/recoll-index:/var/lib/recoll
     healthcheck:
       test: ["CMD", "python", "-c", "import os,urllib.request; p=os.environ.get('RECOLL_MODERN_ROOT_PATH','/').rstrip('/'); urllib.request.urlopen('http://127.0.0.1:8000'+p+'/health',timeout=5)"]
       interval: 30s
@@ -23,6 +23,3 @@ services:
       retries: 3
       start_period: 30s
     restart: unless-stopped
-volumes:
-  service-state:
-  recoll-index:

compose.yaml

@@ -15,8 +15,8 @@ services:
       RECOLL_MODERN_MEMBERSHIP_MAP: /var/lib/recoll-modern/membership.map
     volumes:
       - ${RECOLL_MODERN_DOCUMENTS:?Set documents directory}:/documents:ro
-      - service-state:/var/lib/recoll-modern
+      - ./data/service-state:/var/lib/recoll-modern
-      - recoll-index:/var/lib/recoll
+      - ./data/recoll-index:/var/lib/recoll
     healthcheck:
       test: ["CMD", "python", "-c", "import os,urllib.request; p=os.environ.get('RECOLL_MODERN_ROOT_PATH','/').rstrip('/'); urllib.request.urlopen('http://127.0.0.1:8000'+p+'/health',timeout=5)"]
       interval: 30s
@@ -24,6 +24,3 @@ services:
       retries: 3
       start_period: 30s
     restart: unless-stopped
-volumes:
-  service-state:
-  recoll-index:

docs/ARCHITECTURE.md

@@ -30,7 +30,7 @@ web UI and (b) an MCP server, running in Docker, with several independent collec
 | `src/recoll_modern/indexing/handlers.py` | Scans membership sources before strict-mode mutations and records file errors |
 | `src/recoll_modern/indexing/map_export.py` | Writes a durable atomic map from canonical paths to opaque collection IDs |
 | `helper/` | C reader run by Recoll's `metadatacmds` for each indexed file |
-| `Dockerfile`, `compose.yaml`, `docker/` | Non-root Recoll runtime, read-only document mount, persistent state/index volumes and config templates |
+| `Dockerfile`, `compose.yaml`, `docker/` | Non-root Recoll runtime, read-only document mount, persistent local state/index directories and config templates |
 | `compose.published.yaml` | Pull-only deployment with the same runtime settings and persistent mounts |
 | `.gitea/workflows/image.yaml`, `scripts/image_tags.py`, `scripts/publish_image.sh` | CI checks and Gitea registry publication; Git refs select published tags |
 | `src/recoll_modern/engine/` | Shared Protocol, deterministic fake, and isolated Recoll Python/indexer adapter |
@@ -65,7 +65,7 @@ future distribution track and starts only after the hosted product through 004 i
 - A Python service owns Recoll through its native Python bindings (`from recoll import recoll`).
 - SQLite holds collections, membership, indexing jobs, users and sessions.
 - An HTTP API consumed by both the web UI and the MCP server, so the two never diverge.
-- A Docker image containing Recoll itself, plus compose wiring for document volumes and index storage.
+- A Docker image containing Recoll itself, plus compose wiring for read-only documents and local index storage.
 - A packaging seam for a future desktop shell: application logic does not depend on Docker, state paths
   and listen addresses are configurable, process startup/shutdown is explicit, and every UI operation
   crosses the same HTTP API. Docker remains the only supported distribution through spec 004.
@@ -129,9 +129,9 @@ display is limited to safe media types.
 The image installs Debian trixie's headless `recollcmd` package and Python binding.
 It syncs the application into a system-site-packages venv so the binding remains
 importable. A separate build stage compiles the C helper. The non-root process
-copies Recoll config templates into the persistent state volume only when absent;
+copies Recoll config templates into the persistent state directory only when absent;
 Recoll may generate additional support files there. Compose mounts documents
-read-only and keeps the SQLite/config and Xapian index volumes separate. Initial
+read-only and keeps the SQLite/config and Xapian index bind mounts separate. Initial
 health can be 503 until an indexing job creates the first index.
 
 Gitea Actions tests and builds on pull requests, main pushes and version tags. A dependent job publishes
@@ -207,8 +207,8 @@ semantics and test strategy — are closed in the approved `docs/specs/001-searc
 |---|---|
 | Recoll (in-container) | All indexing and search. Not installed on the host — see `AGENTS.md` Rules. |
 | Docker + compose | The only supported way to run the service. |
-| Document volumes | Nothing to index; collections resolve to empty. |
+| Document bind mount | Nothing to index; collections resolve to empty. |
-| A writable volume for the index and SQLite | All state. The document volume itself may be mounted read-only. |
+| Writable local directories for the index and SQLite | All state. The document directory itself may be mounted read-only. |
 
 Docker is the only supported deployment for specs 001–004, not a permanent application-layer
 dependency. Spec 005 will design cross-platform packaging of the web UI, API service, Recoll and its

docs/DECISIONS.md

@@ -442,3 +442,18 @@ layers. A split-package probe still produced a 253 MB compressed layer.
 **Consequences:** the service remains headless. The full image's largest compressed layer is 39.02 MB,
 and a local container confirms both Recoll commands and the Python binding are present. Registry
 publishing and pulling still require a successful Gitea Actions run.
+
+---
+
+## 2026-09-28 — Keep Docker state beside the Compose file
+**Status:** accepted
+**Context:** named Docker volumes hide the index and service database from operators managing a
+deployment from its Compose directory. The non-root container needs writable state owned by UID/GID
+`10001:10001`.
+**Decision:** both Compose variants bind-mount `./data/service-state` and `./data/recoll-index`.
+The direct `docker run` example uses the same local directories. Ignore `data/` in Git and require
+operators to create the directories with the container's ownership before startup.
+**Alternatives:** retain Docker-managed named volumes and require volume-specific backup commands.
+**Consequences:** moving the Compose file changes the resolved data paths. Existing named-volume
+deployments must copy both stores into the local directories before switching mounts; neither store
+is deleted or rebuilt by this change.