Refresh deploy/ and bin/ from the boilerplate at 09e144d (the release split between the distro and Windows, wsl-setup, the server config, USER 1000, python:3.15-slim) with roledesk's facts put back: no frontend stage, desk, plugins and seed copied, the extras passkeys, embeddings and llm, roles as the table the backup and the importer require, bin/pull.sh and bin/worker.sh with their profile commands, and Install Server.md and Install WSL.md as copies in place of the five old deployment docs

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-10-27 11:11:19 +01:00
1 parent 83e965dfeb
commit a1f281a23a
40 files changed
+2226 -1287

No files matched your search

+213
View File
@@ -0,0 +1,213 @@
> A copy of the boilerplate's `documentation/Install Server.md` at commit 09e144d, unedited below this line; the fix lands upstream first.
# Deploying a service
`<app>` below is your *service's* name: the hostname, the data directory, the image tag, the compose service. It is the one thing you choose. The Python package stays `app` and the environment variables stay `APP_`, here and in every project started from this repository — nothing inside the checkout is renamed, and the files these steps need are already in place under `deploy/`.
## What the box is
A single Hetzner server running Ubuntu 24.04 on x86_64, with a 37 GB root filesystem and no separate data disk. One Docker Compose project holds everything.
/srv/swag/docker-compose.yml every service, one file, project name `swag`
/srv/swag/.env every secret, chmod 600, never in the compose file
/srv/data/<app>/ every service's data, owned 1000:1000
/etc/config/swag/ SWAG's /config: nginx confs, certificates, logs
A snapshot of that whole compose file, every neighbour included, is [example-docker-compose.yml](example-docker-compose.yml): what a new service block lands beside, and where the SWAG subdomain list lives.
`swag` is nginx, certbot and fail2ban in one container. It terminates TLS for every hostname and proxies to the others **by container name** over the project's default network. That shared network is the only reason all these services live in one compose file — nothing else about them is related.
Images are published to the Gitea instance on the same box, at `git.rittershofer.org/wolf/<app>`. Everything runs as uid/gid 1000 and `TZ=Europe/Brussels`.
Which compose file that is comes off the running container rather than out of memory:
```bash
docker inspect swag --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}'
```
Declaring the network `external` from a second Compose project would work too, and would hardcode the first project's name into the second — a rename away from a confusing failure.
## The seven steps
**1. DNS.** `<app>.rittershofer.org` must resolve to the server *before* anything else. Confirm with `getent hosts <app>.rittershofer.org`.
**2. Certificate.** Add `<app>` to `SUBDOMAINS` in the swag service's environment (docker compose yaml). That list separates on commas and nothing else. SWAG runs certbot on every container start and requests **one certificate covering every name in the list**. If SWAG already holds a wildcard certificate covering the name, reuse it and skip this step entirely.
**3. Registry login.** On the build machine a Gitea token with package **write**; on the server, package **read**. Same command both ends, token at the password prompt:
```bash
docker login git.rittershofer.org -u wolf
```
Git credentials and Docker registry credentials are separate. A
**4. Data directory.** Root's `mkdir` produces a root-owned directory, and the container runs as 1000, so the chown is not optional:
```bash
sudo mkdir -p /srv/data/<app>/home && sudo chown -R 1000:1000 /srv/data/<app>
```
**5. Service block** from [../deploy/server/compose.yaml](../deploy/server/compose.yaml) into `/srv/swag/docker-compose.yml`, the variables from [../deploy/server/env.example](../deploy/server/env.example) appended to `/srv/swag/.env`, and [../deploy/server/backup.sh](../deploy/server/backup.sh) next to the compose file for the nightly cron. Copy [../deploy/server/config.example.yaml](../deploy/server/config.example.yaml) to `$APP_DATA/config.yaml` with `rp_id` and `origin` set to the hostname: it holds the keys whose defaults are wrong in a container (`server.host: 0.0.0.0`, `database: /data/app.db`, `auth.enabled: always`, `auth.secure_cookie: always`). Without it the app binds loopback inside the container and writes its database into the image's root-owned folder.
**6. Proxy conf** — [../deploy/server/swag/app.subdomain.conf](../deploy/server/swag/app.subdomain.conf) renamed to `<app>.subdomain.conf` in `/etc/config/swag/nginx/proxy-confs/`. Then validate, and reload only if validation passes:
```bash
docker exec swag nginx -t && docker exec swag nginx -s reload
```
Confirm the other sites still respond afterwards.
**7. Deploy.** Pull first, since `up` alone keeps the `:latest` it already has, and name the service, so nothing else in the file is recreated:
```bash
cd /srv/swag && docker compose pull <app> && docker compose up -d <app>
```
[../deploy/server/deploy.sh](../deploy/server/deploy.sh) is the same two commands with a check that `config.yaml` exists and sets `database` and `auth.enabled`, and a wait for the health check before it returns.
## What the templates already decide for you
Both templates are in [../deploy/server/](../deploy/server), commented in place. The decisions worth knowing before you read them:
**No `ports:`.** SWAG reaches the container by name on the project network. That is what keeps the app's port off the internet even if the firewall is wrong.
**No `networks:` either.** Being in that compose file is what puts it on the right one.
**`user: "1000:1000"`.** Container uid is host uid — there is no translation layer. Without this the process runs as whatever the image declares, usually root, and everything it writes into `/srv/data` lands root-owned.
**`mem_limit:`, not `deploy: resources: limits:`.** The latter is Swarm-only and silently ignored by `docker compose up`. 0 is unlimited, which is the honest default until the box has been watched under real load.
**`logging:` with a size cap.** Without it, json-file logs grow until the disk does not.
**`listen 443 ssl` without `http2`.** `http2 on;` is set once at `http` scope in `nginx.conf`, and the `listen ... http2` parameter is deprecated in nginx 1.25 and later.
**`proxy_read_timeout 86400s`** only if the app holds a WebSocket open. SWAG's `proxy.conf` already sets the Upgrade headers; what it also sets is a read timeout of a couple of minutes, which closes an idle socket on the minute. Delete those two lines if there is no socket.
# Publishing a release
The scripts are in place in `deploy/`, with `deploy/server/` beside them. The Dockerfile at `deploy/Dockerfile`, which is where `2-build-image.ps1` looks for it in the tree step 1 exports.
The release is split where the machines are. The checkout, its `.venv` and the tests are Linux, so the gate, step 1, is bash and runs inside the WSL distro. The distro is a sealed sandbox (Windows interop off, no drive mounts, no container engine, see [Install WSL.md, 7. Cut the distro off from Windows](Install%20WSL.md#7-cut-the-distro-off-from-windows)) and stays that way, because a sandbox that can drive containers can mount the host. Step 1 ends by exporting the production tree to `release/<folder>/<sha>/` beside the checkout, and the image is built and pushed from Windows with `wslc`, the container CLI that ships with WSL: steps 2 and 3, two short PowerShell scripts that read that tree through `\\wsl.localhost`. The distro never holds a container socket.
## Every release
From Windows, in PowerShell, once `deploy\wsl-setup.cmd` has installed the profile commands:
```powershell
<folder>-0release
```
where `<folder>` is the checkout folder's name (`boilerplate` here). It runs `<folder>-1test`, `<folder>-2build` and `<folder>-3push` in order and stops at the first that fails; its arguments (`--push-branch`, `--no-tests`) go to step 1. Each step is a command of its own:
```text
<folder>-1test bash deploy/1-test-and-promote.sh in WSL: test local main, fast-forward local production, export it
<folder>-2build deploy\2-build-image.ps1 on Windows: build the exported tree with wslc
<folder>-3push deploy\3-push-image.ps1 on Windows: push the built image to the registry
```
The number is the order. Each script can be run on its own and works out its paths for itself, so the working directory does not matter.
**Step 1 never builds; step 2 never pushes; step 3 never rebuilds or deploys.** That separation is the point of the numbering: a failed push is `3` again, not the whole thing again.
## What each step is actually doing
**1: test and promote.** Resolves `main` to a commit, checks that commit out into a throwaway worktree, and runs the project's checks there. The worktree is what makes this honest: the tests run against exactly the tree that is about to become production, with none of the working copy's uncommitted edits in the way. Worktrees have no `.venv` of their own, so the interpreter comes from the main checkout and the code from the worktree.
The checks are pytest, then `npm ci` and `npm run check` in `frontend/`: lint, format, the generated types, the page's tests and a build. So the machine that runs step 1 needs Node, which `frontend/install.sh` installs once. Step 1 runs the gates through `deploy/test-commit.sh`, which looks for Node from nvm and from fnm, since a non-interactive shell has not read the rc file that puts it on `PATH`. `PYTHON` in `release.env` names the interpreter when it is not `.venv/bin/python`; `--no-tests` skips the gates.
Promotion is a fast-forward only. If `production` has commits `main` does not, that is a release made from somewhere else, and the script stops rather than discarding it.
Git publication is optional: run `git push origin production` yourself, or pass `--push-branch` to step 1 or `<folder>-0release`. `production` identifies the tested candidate; the server's running image identifies the deployed release.
Step 1 then exports the production commit: `git archive` unpacked into `release/<folder>/<sha>/` beside the checkout (the environment variable `RELEASE_DIR` names another folder for `release/`), with `deploy/Dockerfile.dockerignore` copied to `.dockerignore` at its root, and the short sha written to `release/<folder>/production`. Older trees are removed; a tree that already exists for this sha is kept, because `wslc` can build from a stale view of a folder deleted and recreated at the same path ([microsoft/WSL#41287](https://github.com/microsoft/WSL/issues/41287)). When the folder cannot be written, step 1 says so and fails after the promote; run it again.
**2: build.** Reads the sha from `production` and runs `wslc build -t <IMAGE>:<sha> -t <IMAGE>:latest -f <tree>\deploy\Dockerfile <tree>` with the tree's `\\wsl.localhost` path. `-f` is a path `wslc` opens on Windows and hands to the builder on stdin, which is why the ignore rules have to be at the root of the context. The image builds the React page itself, in a Node stage from `frontend/` and its lock file, so neither Node nor a built `app/web/dist/` is needed for this step.
`wslc` builds an **image**. No container starts or restarts on this machine.
**3: push.** Re-tags `latest` onto this candidate with `wslc tag`, then `wslc push`es both. It never rebuilds, so a push that failed on connectivity is just this step again. A refused push prints the `wslc login` line to use.
# Proposal - Operating it
[../deploy/server/backup.sh](../deploy/server/backup.sh), [../deploy/server/import.sh](../deploy/server/import.sh) and [../deploy/server/deploy.sh](../deploy/server/deploy.sh) name the compose service in `SERVICE=` at their top, and `NAME` in `deploy/release.env` names it for [../deploy/db.sh](../deploy/db.sh). Each has to be `<app>`. All of them read `APP_DATA` from `/srv/swag/.env`.
## The nightly backup
`backup.sh` beside the compose file takes one snapshot through the running container. Inside it, the app's own config names the database, which is opened read-only, so a path that names no file is an error rather than a new empty database. `VACUUM INTO` writes `/data/backups/<app>-<stamp>.db` as one consistent, compacted file without blocking a writer for long. The snapshot is reopened read-only and must hold every table in `TABLES` (`jobs ticks`; a test keeps it equal to `REQUIRED_TABLES` in [../deploy/server/import_database.py](../deploy/server/import_database.py)); otherwise it is deleted and the run fails. A project that replaces `ticks` with its own first table changes both. The snapshot is gzipped under `umask 077`, so every backup is owner-only, and everything past the newest `KEEP` (14 by default) is deleted.
Schedule it in a crontab that can run `docker compose`:
```
0 4 * * * bash /srv/swag/backup.sh >> /srv/swag/backup.log 2>&1
```
The container has to be running, since the snapshot is taken through `docker compose exec`. A backup on the same disk is not an off-server copy; `db.sh pull` below makes one.
## Moving the database between laptop and server
`deploy/db.sh` runs in the checkout, with the `ssh` and `scp` of that shell, and reads `SSH_HOST`, `REMOTE_DIR` (`/srv/swag`), `REMOTE_DATA` (`/srv/data/<app>`) and `NAME` from `deploy/release.env`.
`bash deploy/db.sh pull` runs the server's own `backup.sh` over ssh and downloads the newest `<app>-*.db.gz` into `backups/` in the checkout. Use it as a read-only copy. Pushing it back after the server has written discards those writes, and nothing merges two copies.
Pushing the laptop's database onto the server replaces the server's, including the password hash, sessions, passkeys and API tokens, because the `auth_*` tables live in the same file. Set the account up against the final database, not before the push.
**1. Stop the app.** The import is a file swap under SQLite and `import.sh` refuses a running service:
```bash
cd /srv/swag && docker compose stop <app>
```
**2. Push.** From the checkout:
```bash
bash deploy/db.sh push
```
It snapshots the local database (the one the checkout's config names, or `LOCAL_DB`) with `VACUUM INTO`, gzips it, asks before going on unless given `--yes`, uploads it as `$REMOTE_DATA/incoming.db.gz` and `import_database.py` into `$REMOTE_DIR`, and streams `import.sh` over ssh. On the server, `import.sh` runs `import_database.py` in a one-off container of the image already there (`--pull never`). The importer decompresses into a temporary file beside the database and requires `PRAGMA integrity_check` to answer `ok` and every table in `REQUIRED_TABLES` to exist. Only then does it snapshot the current database with `VACUUM INTO` into `/data/backups/<database name>-<stamp>.db.gz`, owner-only, refuse if `wal_checkpoint(TRUNCATE)` finds another connection, delete the `-wal` and `-shm` files and move the new file into place with one `os.replace`. It prints the snapshot's path. For `/data/app.db` the snapshot is `app-<stamp>.db.gz`, which `backup.sh` prunes with its own only while the service is also called `app`.
**3. Start the app.**
```bash
cd /srv/swag && docker compose up -d --wait <app>
```
## Restoring a backup on the server
The same import, by hand, with a file from `$APP_DATA/backups` in place of the pushed one. `import.sh` and `import_database.py` from `deploy/server/` have to be beside the compose file; `db.sh push` leaves the importer there but streams `import.sh`.
**1. Stop the app.** `cd /srv/swag && docker compose stop <app>`
**2. Stage the backup** as the incoming file:
```bash
sudo -u '#1000' cp /srv/data/<app>/backups/<app>-<stamp>.db.gz /srv/data/<app>/incoming.db.gz
```
The backups are owner-only and belong to uid 1000, which the importer runs as; a copy made as root is one it cannot read.
**3. Import.** `bash /srv/swag/import.sh /srv/swag` checks it, snapshots the current database and replaces it, as in step 2 above.
**4. Start the app.** `cd /srv/swag && docker compose up -d --wait <app>`
## Rolling back to a pinned image
Every release pushes `<IMAGE>:<sha>` and `<IMAGE>:latest`, where `<sha>` is the short hash of the promoted commit. `latest` moves on every release, so it cannot name an earlier one; the compose file runs `${APP_IMAGE}:${APP_TAG:-latest}`, and `APP_TAG` in `.env` pins a commit tag.
**1. Find the tag.** The image's package page in Gitea lists every pushed tag with its date. A tag removed by the package cleanup rule cannot be rolled back to.
**2. Pin it.** Set `APP_TAG=<sha>` in `/srv/swag/.env`.
**3. Deploy.** `bash /srv/swag/deploy.sh` pulls the pinned image, which the weekly prune removes from the box once no container runs it, and recreates `<app>` with it.
**4. Unpin** with `APP_TAG=latest` and deploy again once a fixed release is out.
A rollback replaces the code, not the database. The queue's tables only ever gain columns, each nullable or with a default; `pipeline/table.py` adds them and never drops one. A project whose migration changed the schema in a way the older code cannot read restores a backup taken before that release, with the app stopped, as above.
# Pitfalls
**Disk.** 37 GB, and it has been full. Four causes, all recurring. `:latest` orphans the previous image on every pull and nothing cleans them up; root's crontab runs `docker image prune -af` every Sunday at 04:00, and the `-a` also removes a pinned image that is not running, so a rollback tag must be pulled again. Gitea's package store under `/srv/data/gitea/gitea/packages` keeps every pushed `:sha` and `:latest` version of every image forever: it was 11 GB on 16 October 2026 after a week of newsdesk releases at 2.6 GB each, and the prune cron never touches it. A cleanup rule per owner at `/user/settings/packages/rules` (keep the most recent n, **container** type) is applied by Gitea's `cleanup_packages` cron at midnight, which also drops blobs unreferenced for 24 hours; `/-/admin/packages` lists every version with a delete button and a Cleanup expired data button runs the same job now. Journald defaults to 10% of the filesystem and is capped here at 200 MB. VS Code Remote installs a fresh server per version under `~/.vscode-server/cli/servers` forever. [disk_usage.sh](disk_usage.sh) prints all of it, read-only.
**SQLite is never safe to copy live.** Stop the container, or snapshot with `VACUUM INTO` through the running one. A `cp` of a WAL-mode database is a coin toss.
**Secrets in `.env`, never in the compose file.** It is easy to paste a compose file into a chat window; it is harder to do that by accident with a file you had to `cat` deliberately.