18 KiB
A copy of the boilerplate's
documentation/Install Server.mdat commit 09e144d, unedited below this line; the fix lands upstream first.
Deploying a service
<app> below is your service's name: the hostname, the data directory, the image tag, the compose service. It is the one thing you choose. The Python package stays app and the environment variables stay APP_, here and in every project started from this repository — nothing inside the checkout is renamed, and the files these steps need are already in place under deploy/.
What the box is
A single Hetzner server running Ubuntu 24.04 on x86_64, with a 37 GB root filesystem and no separate data disk. One Docker Compose project holds everything.
/srv/swag/docker-compose.yml every service, one file, project name `swag`
/srv/swag/.env every secret, chmod 600, never in the compose file
/srv/data/<app>/ every service's data, owned 1000:1000
/etc/config/swag/ SWAG's /config: nginx confs, certificates, logs
A snapshot of that whole compose file, every neighbour included, is example-docker-compose.yml: what a new service block lands beside, and where the SWAG subdomain list lives.
swag is nginx, certbot and fail2ban in one container. It terminates TLS for every hostname and proxies to the others by container name over the project's default network. That shared network is the only reason all these services live in one compose file — nothing else about them is related.
Images are published to the Gitea instance on the same box, at git.rittershofer.org/wolf/<app>. Everything runs as uid/gid 1000 and TZ=Europe/Brussels.
Which compose file that is comes off the running container rather than out of memory:
docker inspect swag --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}'
Declaring the network external from a second Compose project would work too, and would hardcode the first project's name into the second — a rename away from a confusing failure.
The seven steps
1. DNS. <app>.rittershofer.org must resolve to the server before anything else. Confirm with getent hosts <app>.rittershofer.org.
2. Certificate. Add <app> to SUBDOMAINS in the swag service's environment (docker compose yaml). That list separates on commas and nothing else. SWAG runs certbot on every container start and requests one certificate covering every name in the list. If SWAG already holds a wildcard certificate covering the name, reuse it and skip this step entirely.
3. Registry login. On the build machine a Gitea token with package write; on the server, package read. Same command both ends, token at the password prompt:
docker login git.rittershofer.org -u wolf
Git credentials and Docker registry credentials are separate. A
4. Data directory. Root's mkdir produces a root-owned directory, and the container runs as 1000, so the chown is not optional:
sudo mkdir -p /srv/data/<app>/home && sudo chown -R 1000:1000 /srv/data/<app>
5. Service block from ../deploy/server/compose.yaml into /srv/swag/docker-compose.yml, the variables from ../deploy/server/env.example appended to /srv/swag/.env, and ../deploy/server/backup.sh next to the compose file for the nightly cron. Copy ../deploy/server/config.example.yaml to $APP_DATA/config.yaml with rp_id and origin set to the hostname: it holds the keys whose defaults are wrong in a container (server.host: 0.0.0.0, database: /data/app.db, auth.enabled: always, auth.secure_cookie: always). Without it the app binds loopback inside the container and writes its database into the image's root-owned folder.
6. Proxy conf — ../deploy/server/swag/app.subdomain.conf renamed to <app>.subdomain.conf in /etc/config/swag/nginx/proxy-confs/. Then validate, and reload only if validation passes:
docker exec swag nginx -t && docker exec swag nginx -s reload
Confirm the other sites still respond afterwards.
7. Deploy. Pull first, since up alone keeps the :latest it already has, and name the service, so nothing else in the file is recreated:
cd /srv/swag && docker compose pull <app> && docker compose up -d <app>
../deploy/server/deploy.sh is the same two commands with a check that config.yaml exists and sets database and auth.enabled, and a wait for the health check before it returns.
What the templates already decide for you
Both templates are in ../deploy/server/, commented in place. The decisions worth knowing before you read them:
No ports:. SWAG reaches the container by name on the project network. That is what keeps the app's port off the internet even if the firewall is wrong.
No networks: either. Being in that compose file is what puts it on the right one.
user: "1000:1000". Container uid is host uid — there is no translation layer. Without this the process runs as whatever the image declares, usually root, and everything it writes into /srv/data lands root-owned.
mem_limit:, not deploy: resources: limits:. The latter is Swarm-only and silently ignored by docker compose up. 0 is unlimited, which is the honest default until the box has been watched under real load.
logging: with a size cap. Without it, json-file logs grow until the disk does not.
listen 443 ssl without http2. http2 on; is set once at http scope in nginx.conf, and the listen ... http2 parameter is deprecated in nginx 1.25 and later.
proxy_read_timeout 86400s only if the app holds a WebSocket open. SWAG's proxy.conf already sets the Upgrade headers; what it also sets is a read timeout of a couple of minutes, which closes an idle socket on the minute. Delete those two lines if there is no socket.
Publishing a release
The scripts are in place in deploy/, with deploy/server/ beside them. The Dockerfile at deploy/Dockerfile, which is where 2-build-image.ps1 looks for it in the tree step 1 exports.
The release is split where the machines are. The checkout, its .venv and the tests are Linux, so the gate, step 1, is bash and runs inside the WSL distro. The distro is a sealed sandbox (Windows interop off, no drive mounts, no container engine, see Install WSL.md, 7. Cut the distro off from Windows) and stays that way, because a sandbox that can drive containers can mount the host. Step 1 ends by exporting the production tree to release/<folder>/<sha>/ beside the checkout, and the image is built and pushed from Windows with wslc, the container CLI that ships with WSL: steps 2 and 3, two short PowerShell scripts that read that tree through \\wsl.localhost. The distro never holds a container socket.
Every release
From Windows, in PowerShell, once deploy\wsl-setup.cmd has installed the profile commands:
<folder>-0release
where <folder> is the checkout folder's name (boilerplate here). It runs <folder>-1test, <folder>-2build and <folder>-3push in order and stops at the first that fails; its arguments (--push-branch, --no-tests) go to step 1. Each step is a command of its own:
<folder>-1test bash deploy/1-test-and-promote.sh in WSL: test local main, fast-forward local production, export it
<folder>-2build deploy\2-build-image.ps1 on Windows: build the exported tree with wslc
<folder>-3push deploy\3-push-image.ps1 on Windows: push the built image to the registry
The number is the order. Each script can be run on its own and works out its paths for itself, so the working directory does not matter.
Step 1 never builds; step 2 never pushes; step 3 never rebuilds or deploys. That separation is the point of the numbering: a failed push is 3 again, not the whole thing again.
What each step is actually doing
1: test and promote. Resolves main to a commit, checks that commit out into a throwaway worktree, and runs the project's checks there. The worktree is what makes this honest: the tests run against exactly the tree that is about to become production, with none of the working copy's uncommitted edits in the way. Worktrees have no .venv of their own, so the interpreter comes from the main checkout and the code from the worktree.
The checks are pytest, then npm ci and npm run check in frontend/: lint, format, the generated types, the page's tests and a build. So the machine that runs step 1 needs Node, which frontend/install.sh installs once. Step 1 runs the gates through deploy/test-commit.sh, which looks for Node from nvm and from fnm, since a non-interactive shell has not read the rc file that puts it on PATH. PYTHON in release.env names the interpreter when it is not .venv/bin/python; --no-tests skips the gates.
Promotion is a fast-forward only. If production has commits main does not, that is a release made from somewhere else, and the script stops rather than discarding it.
Git publication is optional: run git push origin production yourself, or pass --push-branch to step 1 or <folder>-0release. production identifies the tested candidate; the server's running image identifies the deployed release.
Step 1 then exports the production commit: git archive unpacked into release/<folder>/<sha>/ beside the checkout (the environment variable RELEASE_DIR names another folder for release/), with deploy/Dockerfile.dockerignore copied to .dockerignore at its root, and the short sha written to release/<folder>/production. Older trees are removed; a tree that already exists for this sha is kept, because wslc can build from a stale view of a folder deleted and recreated at the same path (microsoft/WSL#41287). When the folder cannot be written, step 1 says so and fails after the promote; run it again.
2: build. Reads the sha from production and runs wslc build -t <IMAGE>:<sha> -t <IMAGE>:latest -f <tree>\deploy\Dockerfile <tree> with the tree's \\wsl.localhost path. -f is a path wslc opens on Windows and hands to the builder on stdin, which is why the ignore rules have to be at the root of the context. The image builds the React page itself, in a Node stage from frontend/ and its lock file, so neither Node nor a built app/web/dist/ is needed for this step.
wslc builds an image. No container starts or restarts on this machine.
3: push. Re-tags latest onto this candidate with wslc tag, then wslc pushes both. It never rebuilds, so a push that failed on connectivity is just this step again. A refused push prints the wslc login line to use.
Proposal - Operating it
../deploy/server/backup.sh, ../deploy/server/import.sh and ../deploy/server/deploy.sh name the compose service in SERVICE= at their top, and NAME in deploy/release.env names it for ../deploy/db.sh. Each has to be <app>. All of them read APP_DATA from /srv/swag/.env.
The nightly backup
backup.sh beside the compose file takes one snapshot through the running container. Inside it, the app's own config names the database, which is opened read-only, so a path that names no file is an error rather than a new empty database. VACUUM INTO writes /data/backups/<app>-<stamp>.db as one consistent, compacted file without blocking a writer for long. The snapshot is reopened read-only and must hold every table in TABLES (jobs ticks; a test keeps it equal to REQUIRED_TABLES in ../deploy/server/import_database.py); otherwise it is deleted and the run fails. A project that replaces ticks with its own first table changes both. The snapshot is gzipped under umask 077, so every backup is owner-only, and everything past the newest KEEP (14 by default) is deleted.
Schedule it in a crontab that can run docker compose:
0 4 * * * bash /srv/swag/backup.sh >> /srv/swag/backup.log 2>&1
The container has to be running, since the snapshot is taken through docker compose exec. A backup on the same disk is not an off-server copy; db.sh pull below makes one.
Moving the database between laptop and server
deploy/db.sh runs in the checkout, with the ssh and scp of that shell, and reads SSH_HOST, REMOTE_DIR (/srv/swag), REMOTE_DATA (/srv/data/<app>) and NAME from deploy/release.env.
bash deploy/db.sh pull runs the server's own backup.sh over ssh and downloads the newest <app>-*.db.gz into backups/ in the checkout. Use it as a read-only copy. Pushing it back after the server has written discards those writes, and nothing merges two copies.
Pushing the laptop's database onto the server replaces the server's, including the password hash, sessions, passkeys and API tokens, because the auth_* tables live in the same file. Set the account up against the final database, not before the push.
1. Stop the app. The import is a file swap under SQLite and import.sh refuses a running service:
cd /srv/swag && docker compose stop <app>
2. Push. From the checkout:
bash deploy/db.sh push
It snapshots the local database (the one the checkout's config names, or LOCAL_DB) with VACUUM INTO, gzips it, asks before going on unless given --yes, uploads it as $REMOTE_DATA/incoming.db.gz and import_database.py into $REMOTE_DIR, and streams import.sh over ssh. On the server, import.sh runs import_database.py in a one-off container of the image already there (--pull never). The importer decompresses into a temporary file beside the database and requires PRAGMA integrity_check to answer ok and every table in REQUIRED_TABLES to exist. Only then does it snapshot the current database with VACUUM INTO into /data/backups/<database name>-<stamp>.db.gz, owner-only, refuse if wal_checkpoint(TRUNCATE) finds another connection, delete the -wal and -shm files and move the new file into place with one os.replace. It prints the snapshot's path. For /data/app.db the snapshot is app-<stamp>.db.gz, which backup.sh prunes with its own only while the service is also called app.
3. Start the app.
cd /srv/swag && docker compose up -d --wait <app>
Restoring a backup on the server
The same import, by hand, with a file from $APP_DATA/backups in place of the pushed one. import.sh and import_database.py from deploy/server/ have to be beside the compose file; db.sh push leaves the importer there but streams import.sh.
1. Stop the app. cd /srv/swag && docker compose stop <app>
2. Stage the backup as the incoming file:
sudo -u '#1000' cp /srv/data/<app>/backups/<app>-<stamp>.db.gz /srv/data/<app>/incoming.db.gz
The backups are owner-only and belong to uid 1000, which the importer runs as; a copy made as root is one it cannot read.
3. Import. bash /srv/swag/import.sh /srv/swag checks it, snapshots the current database and replaces it, as in step 2 above.
4. Start the app. cd /srv/swag && docker compose up -d --wait <app>
Rolling back to a pinned image
Every release pushes <IMAGE>:<sha> and <IMAGE>:latest, where <sha> is the short hash of the promoted commit. latest moves on every release, so it cannot name an earlier one; the compose file runs ${APP_IMAGE}:${APP_TAG:-latest}, and APP_TAG in .env pins a commit tag.
1. Find the tag. The image's package page in Gitea lists every pushed tag with its date. A tag removed by the package cleanup rule cannot be rolled back to.
2. Pin it. Set APP_TAG=<sha> in /srv/swag/.env.
3. Deploy. bash /srv/swag/deploy.sh pulls the pinned image, which the weekly prune removes from the box once no container runs it, and recreates <app> with it.
4. Unpin with APP_TAG=latest and deploy again once a fixed release is out.
A rollback replaces the code, not the database. The queue's tables only ever gain columns, each nullable or with a default; pipeline/table.py adds them and never drops one. A project whose migration changed the schema in a way the older code cannot read restores a backup taken before that release, with the app stopped, as above.
Pitfalls
Disk. 37 GB, and it has been full. Four causes, all recurring. :latest orphans the previous image on every pull and nothing cleans them up; root's crontab runs docker image prune -af every Sunday at 04:00, and the -a also removes a pinned image that is not running, so a rollback tag must be pulled again. Gitea's package store under /srv/data/gitea/gitea/packages keeps every pushed :sha and :latest version of every image forever: it was 11 GB on 16 October 2026 after a week of newsdesk releases at 2.6 GB each, and the prune cron never touches it. A cleanup rule per owner at /user/settings/packages/rules (keep the most recent n, container type) is applied by Gitea's cleanup_packages cron at midnight, which also drops blobs unreferenced for 24 hours; /-/admin/packages lists every version with a delete button and a Cleanup expired data button runs the same job now. Journald defaults to 10% of the filesystem and is capped here at 200 MB. VS Code Remote installs a fresh server per version under ~/.vscode-server/cli/servers forever. disk_usage.sh prints all of it, read-only.
SQLite is never safe to copy live. Stop the container, or snapshot with VACUUM INTO through the running one. A cp of a WAL-mode database is a coin toss.
Secrets in .env, never in the compose file. It is easy to paste a compose file into a chat window; it is harder to do that by accident with a file you had to cat deliberately.