Bring in pipeline, auth, deploy and docs from the boilerplate
roledesk is built on the boilerplate from here on: pipeline 2.0 (queue and scheduler), auth (one password, sessions, passkeys), the deploy kit and its docs, plus the coupling and packaging tests. Downstream copies, unedited. Refreshed from boilerplate/d56949f7d0f6c0bfe5745478e09007bfad67ae87 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f436104f3
commit
9d666d0937
76 files changed
+15235
No files matched your search
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# On the server, next to the compose file, or with its directory as $1. One
|
||||
# snapshot of the database, taken through the running container, kept
|
||||
# alongside the others.
|
||||
#
|
||||
# bash backup.sh # into $APP_DATA/backups
|
||||
# 0 4 * * * bash /srv/swag/backup.sh >> /srv/swag/backup.log 2>&1
|
||||
#
|
||||
# `VACUUM INTO` rather than copying the file: SQLite is in WAL mode, so the
|
||||
# .db on disk is missing whatever is still in the -wal, and `cp` of a live
|
||||
# database is a coin toss. VACUUM INTO takes a read lock, writes one
|
||||
# self-contained and compacted file, and never blocks a writer for long enough
|
||||
# to matter on a database of this size.
|
||||
#
|
||||
# Rename `app` and `APP_DATA` to your service's name.
|
||||
|
||||
set -euo pipefail
|
||||
cd "${1:-$(dirname "$(readlink -f "$0")")}"
|
||||
|
||||
# Same source of truth as the compose file and deploy.sh.
|
||||
DATA=$(sed -n 's/^[[:space:]]*APP_DATA=\([^#]*\).*/\1/p' .env | tail -1 | xargs || true)
|
||||
DATA=${DATA:-./data}
|
||||
|
||||
KEEP=${KEEP:-14}
|
||||
stamp=$(date +%Y%m%d-%H%M%S)
|
||||
|
||||
mkdir -p "$DATA/backups"
|
||||
|
||||
docker compose exec -T app python -c "
|
||||
import sqlite3
|
||||
db = sqlite3.connect('/data/app.db')
|
||||
db.execute('VACUUM INTO ?', ('/data/backups/app-$stamp.db',))
|
||||
db.close()
|
||||
"
|
||||
gzip -f "$DATA/backups/app-$stamp.db"
|
||||
echo "$DATA/backups/app-$stamp.db.gz $(du -h "$DATA/backups/app-$stamp.db.gz" | cut -f1)"
|
||||
|
||||
# Oldest first, everything past KEEP goes. Named by timestamp, so sorting the
|
||||
# names sorts them by age.
|
||||
ls -1 "$DATA"/backups/app-*.db.gz 2>/dev/null \
|
||||
| sort \
|
||||
| head -n -"$KEEP" \
|
||||
| while read -r old; do rm -f "$old"; echo "removed $old"; done
|
||||
@@ -0,0 +1,60 @@
|
||||
# The service block, for the compose file that already runs your proxy.
|
||||
#
|
||||
# It needs to be on the same Docker network as SWAG, because that is how nginx
|
||||
# reaches it — by container name, on the network Compose gives every project.
|
||||
# Being in the same file is what puts it there; nothing else about it is
|
||||
# related to the other services, and nothing else needs to be.
|
||||
#
|
||||
# Paste this `app:` block into that file's `services:`, add the variables from
|
||||
# `env.example` to the `.env` next to it, and it is `docker compose up -d app`
|
||||
# from then on — which recreates this container and no other.
|
||||
#
|
||||
# Rename `app` and `APP_` throughout to your service's name.
|
||||
|
||||
services:
|
||||
app:
|
||||
image: ${APP_IMAGE:?set APP_IMAGE in .env}:${APP_TAG:-latest}
|
||||
container_name: app
|
||||
restart: unless-stopped
|
||||
|
||||
# A plain directory on the host rather than a managed volume, so a backup
|
||||
# is `tar` and it is owned by you rather than by root. Relative to this
|
||||
# file by default; set APP_DATA to an absolute path to keep every
|
||||
# container's data under one root, which is the thing a backup wants.
|
||||
volumes:
|
||||
- ${APP_DATA:-./data}:/data
|
||||
|
||||
# The container follows the host directory's uid instead of the other way
|
||||
# round — the same PUID/PGID the rest of the stack already runs with.
|
||||
user: "${PUID:-1000}:${PGID:-1000}"
|
||||
|
||||
environment:
|
||||
TZ: ${TZ:-Europe/Berlin}
|
||||
# Which model answers, decided here rather than in the image. With
|
||||
# `backend: auto` the API key wins when there is one and the
|
||||
# subscription CLI is the fallback, so filling in ANTHROPIC_API_KEY
|
||||
# later switches it over with no rebuild. See docs/llm-backends.md.
|
||||
CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN:-}
|
||||
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
|
||||
# Normally empty. Set it to pin the password — the way back in when a
|
||||
# login is lost — then unset it and recreate to hand control of the
|
||||
# password back to the app. See docs/password-protection.md.
|
||||
APP_PASSWORD: ${APP_PASSWORD:-}
|
||||
|
||||
# No `ports:`. Nothing on the host reaches it; SWAG does, by container
|
||||
# name, on the project's network. That is what keeps the app's port off
|
||||
# the internet even if the firewall is wrong.
|
||||
|
||||
# `deploy: resources: limits:` is Swarm-only and silently ignored by
|
||||
# `docker compose up`. This is the one that works. 0 is unlimited, which
|
||||
# is the honest default until the box has been watched under real load.
|
||||
mem_limit: ${APP_MEM:-0}
|
||||
|
||||
# The log is the record — nothing here writes a second copy of what went
|
||||
# wrong — so it is worth keeping a few rotations of, and worth capping on
|
||||
# a small disk.
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "20m"
|
||||
max-file: "5"
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run on the server, next to the compose file, or with its directory as $1.
|
||||
# No remote shell required: the release scripts on Windows publish an image
|
||||
# and stop there, and this is the other half.
|
||||
#
|
||||
# Rename `app` and `APP_DATA` to your service's name.
|
||||
set -euo pipefail
|
||||
cd "${1:-$(dirname "$(readlink -f "$0")")}"
|
||||
|
||||
[[ -s .env ]] || { echo 'Install .env before deploying.' >&2; exit 1; }
|
||||
|
||||
# Where the data directory is depends on APP_DATA, which the compose file also
|
||||
# reads. Pulled out of .env rather than sourced, because Compose's .env is not
|
||||
# a shell script and running it as one is how a stray quote becomes a surprise.
|
||||
DATA=$(sed -n 's/^[[:space:]]*APP_DATA=\([^#]*\).*/\1/p' .env | tail -1 | xargs || true)
|
||||
DATA=${DATA:-./data}
|
||||
|
||||
[[ -s "$DATA/config.yaml" ]] || {
|
||||
echo "Install $DATA/config.yaml before deploying." >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Pull explicitly, here and nowhere else: the compose file has no pull policy,
|
||||
# so starting or importing never fetches an image on its own. A registry
|
||||
# failure must leave the running service alone, not deploy a cache.
|
||||
echo 'Pulling the configured image…'
|
||||
if ! docker compose pull app; then
|
||||
echo 'Image pull failed; app was not restarted.' >&2
|
||||
exit 1
|
||||
fi
|
||||
# Use the image just pulled. Do not start or recreate other services in the
|
||||
# file. `--wait` holds until the container's own HEALTHCHECK passes.
|
||||
if ! docker compose up -d --no-deps --pull never --wait --wait-timeout 240 app; then
|
||||
docker compose ps app >&2 || true
|
||||
echo 'Deployment failed. Check docker compose logs app.' >&2
|
||||
exit 1
|
||||
fi
|
||||
docker compose ps app
|
||||
@@ -0,0 +1,42 @@
|
||||
# The variables this service adds to the `.env` next to your compose file.
|
||||
# Written by hand once, on the server, and never synced from a repository — it
|
||||
# holds the secrets and it is the record of which image is running.
|
||||
#
|
||||
# chmod 600. Rename APP_ to your service's name throughout.
|
||||
|
||||
APP_IMAGE=git.rittershofer.org/wolf/app
|
||||
APP_TAG=latest # follows published images; pin a commit tag for rollback
|
||||
|
||||
# Where the app keeps its database, its config and its backups. Relative to
|
||||
# the compose file if you leave it out. Set it to an absolute path to put it
|
||||
# under the one root every container on the box writes to — the thing that
|
||||
# makes a backup one target instead of a hunt.
|
||||
APP_DATA=/srv/data/app
|
||||
|
||||
# The same PUID/PGID the rest of the stack runs as. The data directory must be
|
||||
# owned by it.
|
||||
PUID=1000
|
||||
PGID=1000
|
||||
TZ=Europe/Berlin
|
||||
|
||||
# Which model answers. Set one or the other, not both — see
|
||||
# docs/llm-backends.md for what each one means and what it costs.
|
||||
#
|
||||
# CLAUDE_CODE_OAUTH_TOKEN a year-long token for the Claude subscription, from
|
||||
# `claude setup-token` on a machine you are signed in
|
||||
# on. Nothing billed per token.
|
||||
# ANTHROPIC_API_KEY per-token billing, no local CLI, the unattended
|
||||
# path. Preferred by `auto` when both are set.
|
||||
CLAUDE_CODE_OAUTH_TOKEN=
|
||||
ANTHROPIC_API_KEY=
|
||||
|
||||
# Leave empty. The first locked start generates a password and prints it to
|
||||
# `docker compose logs app`; after that it is the app's Account screen's to
|
||||
# change. Fill this in only to force one — it is the way back in — then empty
|
||||
# it again and recreate, or the Account screen refuses to change what the
|
||||
# environment pins. See docs/password-protection.md.
|
||||
APP_PASSWORD=
|
||||
|
||||
# Unlimited until the box has been watched through the heaviest thing the app
|
||||
# does. Then something like 1500m.
|
||||
APP_MEM=0
|
||||
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# Called on the server after uploading incoming.db.gz and import_database.py.
|
||||
# `db.ps1 push` does both and then runs this; by hand it is the same two files
|
||||
# in the same two places.
|
||||
#
|
||||
# This replaces the database and does nothing else. It does not stop the app
|
||||
# and it does not start it: whoever is running a deployment knows what else is
|
||||
# moving and in what order, and a tool that helpfully restarts things underneath
|
||||
# that is a tool fighting its caller. By hand it is
|
||||
# `docker compose stop app` and `docker compose up -d --wait app` around this.
|
||||
#
|
||||
# What it will not do is import into a running app. The replacement is a file
|
||||
# swap under SQLite's feet, and the live database is checkpointed first, which
|
||||
# needs every writer gone — `import_database.py` refuses a busy database, and
|
||||
# this refuses an app that is up, because the clearer error is the earlier one.
|
||||
set -euo pipefail
|
||||
cd "${1:?server directory required}"
|
||||
|
||||
# Same source of truth as the compose file, deploy.sh and backup.sh.
|
||||
DATA=$(sed -n 's/^[[:space:]]*APP_DATA=\([^#]*\).*/\1/p' .env | tail -1 | xargs || true)
|
||||
DATA=${DATA:-./data}
|
||||
|
||||
[[ -s "$DATA/incoming.db.gz" && -s import_database.py ]]
|
||||
|
||||
if [[ -n "$(docker compose ps -q --status running app 2>/dev/null)" ]]; then
|
||||
echo 'app is running. Stop it before importing:' >&2
|
||||
echo ' docker compose stop app' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The one-off container uses compose's uid and its mounted /data. `--pull
|
||||
# never`: an import runs on the image already here, whatever the installed
|
||||
# compose file says about pulling. Getting a new image is deploy.sh's job.
|
||||
docker compose run --rm --no-deps --pull never -T --entrypoint python app - < import_database.py
|
||||
rm -f "$DATA/incoming.db.gz"
|
||||
@@ -0,0 +1,77 @@
|
||||
"""Offline database replacement, run as the compose service's uid.
|
||||
|
||||
The app container must be stopped. Only stdlib is used so this can be sent
|
||||
over stdin to either the old or the new image during a deployment.
|
||||
|
||||
Two things to edit for your service: `DATABASE` and `REQUIRED_TABLES`. The
|
||||
table check is what stops a right-shaped-but-wrong database — last project's,
|
||||
or a half-downloaded one — from landing on top of a good one.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import os
|
||||
import shutil
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from contextlib import closing
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
DATABASE = Path("/data/app.db")
|
||||
INCOMING = Path("/data/incoming.db.gz")
|
||||
BACKUPS = Path("/data/backups")
|
||||
|
||||
#: Any one of these missing means this is not our database. Name tables that
|
||||
#: have existed since the first schema, not ones a recent migration added.
|
||||
REQUIRED_TABLES = {"items"}
|
||||
|
||||
|
||||
def replace_database(database: Path, archive: Path, backups: Path) -> Path | None:
|
||||
"""Validate first, snapshot the stopped database, then atomically replace it."""
|
||||
fd, name = tempfile.mkstemp(prefix=".incoming-", suffix=".db", dir=database.parent)
|
||||
staged = Path(name)
|
||||
backup = None
|
||||
try:
|
||||
with os.fdopen(fd, "wb") as output, gzip.open(archive, "rb") as source:
|
||||
shutil.copyfileobj(source, output)
|
||||
output.flush()
|
||||
os.fsync(output.fileno())
|
||||
with closing(sqlite3.connect(staged)) as candidate:
|
||||
if candidate.execute("PRAGMA integrity_check").fetchall() != [("ok",)]:
|
||||
raise ValueError("Incoming database failed integrity_check")
|
||||
tables = {row[0] for row in candidate.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'")}
|
||||
if not REQUIRED_TABLES <= tables:
|
||||
raise ValueError("Incoming file is not this application's database")
|
||||
|
||||
if database.exists():
|
||||
backups.mkdir(parents=True, exist_ok=True)
|
||||
stamp = datetime.now(UTC).strftime("%Y%m%d-%H%M%S-%f")
|
||||
snapshot = backups / f"{database.stem}-{stamp}.db"
|
||||
backup = snapshot.with_suffix(".db.gz")
|
||||
with closing(sqlite3.connect(database)) as current:
|
||||
current.execute("VACUUM INTO ?", (str(snapshot),))
|
||||
# Refuse a competing connection instead of removing a live WAL.
|
||||
busy, _, _ = current.execute("PRAGMA wal_checkpoint(TRUNCATE)").fetchone()
|
||||
if busy:
|
||||
raise RuntimeError("Database is busy; stop every writer before importing")
|
||||
# Keep the uncompressed recovery copy if compression fails.
|
||||
with snapshot.open("rb") as source, gzip.open(backup, "xb") as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
with backup.open("r+b") as saved:
|
||||
os.fsync(saved.fileno())
|
||||
snapshot.unlink()
|
||||
|
||||
for suffix in ("-wal", "-shm"):
|
||||
Path(str(database) + suffix).unlink(missing_ok=True)
|
||||
os.replace(staged, database)
|
||||
return backup
|
||||
finally:
|
||||
staged.unlink(missing_ok=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
saved = replace_database(DATABASE, INCOMING, BACKUPS)
|
||||
print(f"Database replaced; previous snapshot: {saved or 'none (first import)'}")
|
||||
@@ -0,0 +1,64 @@
|
||||
## Rename to <app>.subdomain.conf, copy to SWAG's /config/nginx/proxy-confs/,
|
||||
## validate, reload.
|
||||
##
|
||||
## The subdomain has to exist before this does anything: add <app> to SWAG's
|
||||
## SUBDOMAINS (or run it with a wildcard cert) and point the DNS record at the
|
||||
## same address as the rest of the stack. See docs/hosting.md.
|
||||
|
||||
## Per-IP throttle for the login endpoints, if the app has a password. Declared
|
||||
## here at `http` scope — SWAG includes proxy-confs outside any server block,
|
||||
## which is what makes this legal in this file. 1 MB of zone holds about 16,000
|
||||
## addresses. Delete both this and the `location` below if there is no login.
|
||||
limit_req_zone $binary_remote_addr zone=app_login:1m rate=6r/m;
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
listen [::]:443 ssl;
|
||||
|
||||
server_name app.rittershofer.org;
|
||||
|
||||
include /config/nginx/ssl.conf;
|
||||
# Put proxy defaults at server scope so location timeouts override them
|
||||
# through inheritance, rather than duplicate directives from proxy.conf.
|
||||
include /config/nginx/proxy.conf;
|
||||
|
||||
# 0 is no limit. Set a real one if strangers can post to this app; leave it
|
||||
# off when the only uploads are yours and can be large.
|
||||
client_max_body_size 0;
|
||||
|
||||
# Guessing the password is already pointless if the app locks out after a
|
||||
# handful of wrong answers (see docs/password-protection.md). This is the
|
||||
# cheap half of the rest: attempts are slowed per address, at the proxy,
|
||||
# before they reach the app at all. `burst=5 nodelay` lets a person who
|
||||
# fat-fingers it a few times through immediately and then holds the rate.
|
||||
location ~ ^/login(/passkey)?$ {
|
||||
limit_req zone=app_login burst=5 nodelay;
|
||||
limit_req_status 429;
|
||||
|
||||
include /config/nginx/resolver.conf;
|
||||
set $upstream_app app;
|
||||
set $upstream_port 8787;
|
||||
set $upstream_proto http;
|
||||
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
|
||||
}
|
||||
|
||||
location / {
|
||||
include /config/nginx/resolver.conf;
|
||||
|
||||
set $upstream_app app;
|
||||
set $upstream_port 8787;
|
||||
set $upstream_proto http;
|
||||
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
|
||||
|
||||
# Only if the app holds a WebSocket open. SWAG's proxy.conf already
|
||||
# sets the Upgrade headers; what it also sets is a read timeout of a
|
||||
# couple of minutes, which would close every open tab's socket on the
|
||||
# minute and leave the browser reconnecting for no reason.
|
||||
proxy_read_timeout 86400s;
|
||||
proxy_send_timeout 86400s;
|
||||
}
|
||||
}
|
||||
|
||||
## `listen 443 ssl` without `http2` is correct — `http2 on;` is set once at
|
||||
## `http` scope in nginx.conf, and the `listen ... http2` parameter is
|
||||
## deprecated in nginx 1.25 and later.
|
||||
Reference in new issue
Block a user