Compare commits

...
26 Commits
Author SHA1 Message Date
Wolf 8d90054b09 skills second commit 2026-11-21 18:15:14 +01:00
Wolf 32d95857c7 architecture doc 2026-11-21 18:13:23 +01:00
WolfandClaude Fable 5.1 6e51134477 Every test client speaks to 127.0.0.1, since an open door now answers 421 to a Host that is not loopback
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:52:30 +01:00
WolfandClaude Fable 5.1 6513e2ab95 Merge auth/ of boilerplate 09e144d (lockout, recovery, cookies, tokens, 421 on a foreign Host) and the hardening headers on every answer (wp-up-auth)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:52:30 +01:00
WolfandClaude Fable 5.1 14793f4707 Put nosniff, X-Frame-Options: DENY and Referrer-Policy: same-origin on every HTTP answer through the boilerplate's Hardening middleware, outermost and leaving the socket alone, name the API tokens in the Account page's lede, and reach the open test app as 127.0.0.1, since auth now answers 421 to any other Host
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:51:21 +01:00
WolfandClaude Fable 5.1 244e57ac29 Take auth/ whole from boilerplate 09e144d: the lockout claimed before the scrypt, APP_PASSWORD recovery closing every session, Starlette's cookie parser, the rightmost forwarded hop in the log, revoke_all on a password change, a 421 for a non-loopback Host on an open app, API tokens in auth_tokens behind Authorization: Bearer, and sockets closed with their credential
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:51:21 +01:00
WolfandClaude Fable 5.1 47889ebf66 Merge the deploy kit of boilerplate 09e144d with roledesk's facts put back, bin/ with pull and worker, and the two Install docs in place of the five old copies (wp-up-deploy)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:11:53 +01:00
WolfandClaude Fable 5.1 a1f281a23a Refresh deploy/ and bin/ from the boilerplate at 09e144d (the release split between the distro and Windows, wsl-setup, the server config, USER 1000, python:3.15-slim) with roledesk's facts put back: no frontend stage, desk, plugins and seed copied, the extras passkeys, embeddings and llm, roles as the table the backup and the importer require, bin/pull.sh and bin/worker.sh with their profile commands, and Install Server.md and Install WSL.md as copies in place of the five old deployment docs
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 11:11:19 +01:00
WolfandClaude Fable 5.1 5fe5e00735 CLAUDE.md: llm/ among the copies, desk/ split into news's generic modules and roledesk's own contract, and the UI pointers at the React pages' app.css
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 10:50:32 +01:00
WolfandClaude Fable 5.1 83e965dfeb Plan the refresh from the boilerplate and news of October 2026: pipeline, auth, llm and the generic desk modules, deploy and the toolchain, in five packages
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-27 10:48:18 +01:00
WolfandClaude Fable 5.1 920457d326 Record that the form's gates are the four parts alone, and strike the two decided items from open.md
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:34:12 +02:00
WolfandClaude Fable 5.1 2262cda8f1 Merge the agent form without Advanced rules: the gates are the four parts of the fetch gate, a stored gate the form cannot say is named and kept (wp-advanced)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:34:11 +02:00
WolfandClaude Fable 5.1 1913101061 Drop Advanced › Other rules from the agent form, so an agent's gates are the fetch gate's four parts: a stored gate the parts cannot say is named in a note under the fetch gate and sent back unchanged on Save, and the card still says it in words
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:33:34 +02:00
WolfandClaude Fable 5.1 849adb197d Close by age what an incomplete board read did not list and what a filtered read spares, so no role stays open for good, as ADR 1.3 has it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:27:09 +02:00
WolfandClaude Fable 5.1 136047023e Merge the runner dropping and counting rows without an id, title or link, so no reader skips a listing (wp-rows)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:05:15 +02:00
WolfandClaude Fable 5.1 911dfe8584 Let the runner drop and count a row without an id, a title or a link, as ADR Plugins 3.8 has it, in one place for whole, paged and search reads: the count is stamped per target, logged, named on the board's row, and over 10% of the rows fails the read; the 28 readers hand every listing over instead of skipping it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:04:33 +02:00
WolfandClaude Fable 5.1 8b875b377b Record the world read, the remote mark, the one-country fold, the judge's languages and the end of must-haves in decisions.md, and cut open.md to what is still open
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:03:59 +02:00
WolfandClaude Fable 5.1 c6730062fd Pin that Phenom is asked for the agents' world when they want remote, with or without All locations
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:03:09 +02:00
WolfandClaude Fable 5.1 49929372c7 Mark a role whose title or text says remote while no field does, in prep with no model, and let it meet a gate on remote until extract has read it (migration 35, roles.remote_text)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 14:02:46 +02:00
WolfandClaude Fable 5.1 f141cffc3c Pin the per-office fold within one country: two offices in two countries are two roles, and a role in several Danish cities leads with the agent's town
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:58:41 +02:00
WolfandClaude Fable 5.1 02ea191b53 Merge the removal of must-haves: the judge reads CV, languages, prompts, likes and dislikes, the CV match is uncapped, migration 34 drops the columns (wp-musts)
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:57:16 +02:00
WolfandClaude Fable 5.1 1976f6a353 Ask a host that cannot filter on remote for the agents' world where it takes the world in one read, and keep the per-office fold within one country, as ADR 1.2 and 1.4 have it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:56:01 +02:00
WolfandClaude Fable 5.1 2159c7925f Remove the must-haves the form dropped on 2026-10-03: the agent's must text, the judge's verdicts and the career cap they set, with migration 34 dropping their columns and the judge's format at 6 so every role is judged again
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:55:59 +02:00
WolfandClaude Fable 5.1 0c4f3c9bbe Add to the flow ADR what the review found missing: the gate's unknown location, prep, the remote read on hosts without a remote filter, the per-office fold within one country, the embedding and its calibration, the models and their inputs, the feed's read state and Show gated, and must-haves taken out
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:49:46 +02:00
WolfandClaude Fable 5.1 38c31c830b Rewrite the plugins ADR to the rules the runner needed and the target where the code falls short: cursors and stalls, the runner dropping and counting bad rows, the guard thresholds, remote not asked, a search never closing by list, provider params declared, what the runner keeps per target
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:36:15 +02:00
WolfandClaude Fable 5.1 4a9bcb2b1c Let the judge read the person's languages beside the CV, as ADR 4.2 has it: a language a posting requires above the level spoken lowers skills and is a gap, and a change to them re-judges the cut
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 13:33:32 +02:00
187 changed files with 5960 additions and 2507 deletions

No files matched your search

+2
View File
@@ -12,3 +12,5 @@
# PowerShell reads LF happily; what it cannot read is a non-ASCII byte, which # PowerShell reads LF happily; what it cannot read is a non-ASCII byte, which
# is a different rule and lives in `tests/test_packaging.py`. # is a different rule and lives in `tests/test_packaging.py`.
deploy/** text eol=lf deploy/** text eol=lf
# cmd.exe misparses LF-only batch files, so the double-click launcher stays CRLF.
*.cmd text eol=crlf
-17
View File
@@ -1,17 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="pull" type="ShConfigurationType">
<option name="SCRIPT_TEXT" value="wsl.exe -d claudebox --cd /home/claude/cboxcode/jobboard -- env PYTHONUNBUFFERED=1 /home/claude/.local/bin/uv run app job pull" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="" />
<option name="SCRIPT_OPTIONS" value="" />
<option name="INDEPENDENT_SCRIPT_WORKING_DIRECTORY" value="true" />
<option name="SCRIPT_WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="INDEPENDENT_INTERPRETER_PATH" value="true" />
<option name="INTERPRETER_PATH" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="EXECUTE_IN_TERMINAL" value="true" />
<option name="EXECUTE_SCRIPT_FILE" value="false" />
<envs />
<method v="2" />
</configuration>
</component>
-17
View File
@@ -1,17 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="pytest" type="ShConfigurationType">
<option name="SCRIPT_TEXT" value="wsl.exe -d claudebox --cd /home/claude/cboxcode/jobboard -- env PYTHONUNBUFFERED=1 /home/claude/.local/bin/uv run pytest -q" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="" />
<option name="SCRIPT_OPTIONS" value="" />
<option name="INDEPENDENT_SCRIPT_WORKING_DIRECTORY" value="true" />
<option name="SCRIPT_WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="INDEPENDENT_INTERPRETER_PATH" value="true" />
<option name="INTERPRETER_PATH" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="EXECUTE_IN_TERMINAL" value="true" />
<option name="EXECUTE_SCRIPT_FILE" value="false" />
<envs />
<method v="2" />
</configuration>
</component>
-17
View File
@@ -1,17 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="seed companies" type="ShConfigurationType">
<option name="SCRIPT_TEXT" value="wsl.exe -d claudebox --cd /home/claude/cboxcode/jobboard -- env PYTHONUNBUFFERED=1 /home/claude/.local/bin/uv run app companies seed" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="" />
<option name="SCRIPT_OPTIONS" value="" />
<option name="INDEPENDENT_SCRIPT_WORKING_DIRECTORY" value="true" />
<option name="SCRIPT_WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="INDEPENDENT_INTERPRETER_PATH" value="true" />
<option name="INTERPRETER_PATH" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="EXECUTE_IN_TERMINAL" value="true" />
<option name="EXECUTE_SCRIPT_FILE" value="false" />
<envs />
<method v="2" />
</configuration>
</component>
-17
View File
@@ -1,17 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="serve" type="ShConfigurationType">
<option name="SCRIPT_TEXT" value="wsl.exe -d claudebox --cd /home/claude/cboxcode/jobboard -- env PYTHONUNBUFFERED=1 /home/claude/.local/bin/uv run app serve --reload" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="" />
<option name="SCRIPT_OPTIONS" value="" />
<option name="INDEPENDENT_SCRIPT_WORKING_DIRECTORY" value="true" />
<option name="SCRIPT_WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="INDEPENDENT_INTERPRETER_PATH" value="true" />
<option name="INTERPRETER_PATH" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="EXECUTE_IN_TERMINAL" value="true" />
<option name="EXECUTE_SCRIPT_FILE" value="false" />
<envs />
<method v="2" />
</configuration>
</component>
-17
View File
@@ -1,17 +0,0 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="worker" type="ShConfigurationType">
<option name="SCRIPT_TEXT" value="wsl.exe -d claudebox --cd /home/claude/cboxcode/jobboard -- env PYTHONUNBUFFERED=1 /home/claude/.local/bin/uv run app worker" />
<option name="INDEPENDENT_SCRIPT_PATH" value="true" />
<option name="SCRIPT_PATH" value="" />
<option name="SCRIPT_OPTIONS" value="" />
<option name="INDEPENDENT_SCRIPT_WORKING_DIRECTORY" value="true" />
<option name="SCRIPT_WORKING_DIRECTORY" value="$PROJECT_DIR$" />
<option name="INDEPENDENT_INTERPRETER_PATH" value="true" />
<option name="INTERPRETER_PATH" value="" />
<option name="INTERPRETER_OPTIONS" value="" />
<option name="EXECUTE_IN_TERMINAL" value="true" />
<option name="EXECUTE_SCRIPT_FILE" value="false" />
<envs />
<method v="2" />
</configuration>
</component>
+2 -2
View File
@@ -31,11 +31,11 @@ The package is `app`, following the boilerplate: the deploy kit, `APP_CONFIG`, `
# Rule: # Rule:
`pipeline/`, `auth/` and `deploy/` are downstream copies of `../boilerplate`, `desk/` of `../news`. They are refreshed with `git archive`, and a fix lands upstream first, otherwise the next refresh reverts it. `pipeline/`, `auth/`, `llm/` and `deploy/` are downstream copies of `../boilerplate`, taken whole with `git archive`; the generic modules of `desk/` (`modeltask`, `notices`, `config`, `tools/*`) are copies of `../news`'s, taken file by file; `desk/sources/*` and `desk/models.py` are roledesk's own. A fix to a copy lands upstream first, otherwise the next refresh reverts it. The drift check is `diff -rq --strip-trailing-cr -x __pycache__ <folder> ../boilerplate/<folder>`.
# Rule: # Rule:
Phone and UI patterns come from `../news` (`newsdesk/web/templates/index.html`, `newsdesk/web/static/app.css`) and `../health` (`webapp/`), which went through the phone fixes already; the boilerplate has none. Copy their fragments, do not write new ones. Phone and UI patterns come from `../news` (`frontend/src/app.css`, a React page since October 2026) and `../health` (`webapp/static/app.css`), which went through the phone fixes already; the boilerplate's are in its `frontend/src/app.css` and its Kit tab. roledesk's page is Jinja templates and plain JS: copy their CSS fragments and layout rules, do not write new ones, and do not port components.
# Rule: # Rule:
+1 -1
View File
@@ -76,4 +76,4 @@ The reader, `/roles/{uid}`, shows the full posting: the same vote and Save contr
- [`plugins/README.md`](plugins/README.md): every reader, its endpoints and what each states; [`docs/filter-survey.md`](docs/filter-survey.md): the filter parameters each host takes at the source. - [`plugins/README.md`](plugins/README.md): every reader, its endpoints and what each states; [`docs/filter-survey.md`](docs/filter-survey.md): the filter parameters each host takes at the source.
- `docs/plan-*.md`: plans in progress, folded into `decisions.md` and deleted when their work lands. - `docs/plan-*.md`: plans in progress, folded into `decisions.md` and deleted when their work lands.
- [`desk/README.md`](desk/README.md), [`pipeline/README.md`](pipeline/README.md), [`auth/README.md`](auth/README.md), [`deploy/README.md`](deploy/README.md): the copied modules, where each comes from and how to refresh it. - [`desk/README.md`](desk/README.md), [`pipeline/README.md`](pipeline/README.md), [`auth/README.md`](auth/README.md), [`deploy/README.md`](deploy/README.md): the copied modules, where each comes from and how to refresh it.
- [`docs/hosting.md`](docs/hosting.md), [`docs/release.md`](docs/release.md), [`docs/server.md`](docs/server.md), [`docs/password-protection.md`](docs/password-protection.md), [`docs/llm-backends.md`](docs/llm-backends.md): the boilerplate's deployment docs. - [`docs/Install Server.md`](<docs/Install Server.md>) and [`docs/Install WSL.md`](<docs/Install WSL.md>): the server and the Windows machine the release runs on, copies of the boilerplate's files; the release steps are in [`deploy/README.md`](deploy/README.md).
+1 -2
View File
@@ -4,7 +4,7 @@ https://elevenlabs.io/careers/6b84bd1f-05cb-4b54-b0db-ff64c84217ae/deployment-st
https://jobs.ashbyhq.com/cohere/2d256112-b336-4539-8133-a0bf7f6698f0?locationId=f15fd0a7-62c8-4e26-a06d-f396a4e21ffb https://jobs.ashbyhq.com/cohere/2d256112-b336-4539-8133-a0bf7f6698f0?locationId=f15fd0a7-62c8-4e26-a06d-f396a4e21ffb
https://jobs.ashbyhq.com/mistral.ai/2c1ff089-db74-467c-b17d-608713e43699?departmentId=cdcd26c0-0c1e-4aff-91f7-5f6306b726b3
https://jobs.ashbyhq.com/Sierra/44120ba5-4cf7-48a1-86af-7981c41c179e https://jobs.ashbyhq.com/Sierra/44120ba5-4cf7-48a1-86af-7981c41c179e
@@ -274,7 +274,6 @@ I need a mailbox and an app password from you.
- Ashby marks hybrid roles as remote. - Ashby marks hybrid roles as remote.
- Lever keeps only the intro paragraph. - Lever keeps only the intro paragraph.
- Roles with a place but no country fail the gates (Zeiss "Oberkochen; Jena").
- The top search box doesn't match `Orsted` or `Skane` without the special letters. - The top search box doesn't match `Orsted` or `Skane` without the special letters.
**6. Housekeeping.** **6. Housekeeping.**
+13 -3
View File
@@ -111,6 +111,12 @@ def same_posting(a: Copy, b: Copy) -> bool:
return not ta or not tb or bool(ta & tb) return not ta or not tb or bool(ta & tb)
def _same_country(a: Copy, b: Copy) -> bool:
"""Whether two rows can be in one country: equal, or either unknown."""
ca, cb = _country(a), _country(b)
return not ca or not cb or ca == cb
def keep_order(c: Copy) -> tuple[bool, bool, bool, str, str]: def keep_order(c: Copy) -> tuple[bool, bool, bool, str, str]:
"""The sort key that picks the kept copy of a group, smallest first: a """The sort key that picks the kept copy of a group, smallest first: a
board over a search provider, then one with a description, then the board over a search provider, then one with a description, then the
@@ -193,12 +199,16 @@ def fold(rows: Iterable[Copy]) -> Folded:
Within a `(company_key, title_key)` the kept rows of `assign` are grouped Within a `(company_key, title_key)` the kept rows of `assign` are grouped
again, in `keep_order`: a row joins the first group whose members are all again, in `keep_order`: a row joins the first group whose members are all
of its host, `similar` to it and remote as it is or not. Only a row with of its host, in its country (equal, or either unknown, as `same_posting`
reads it), `similar` to it and remote as it is or not. Only a row with
a link can be folded, as its link is what the kept role keeps. Every row a link can be folded, as its link is what the kept role keeps. Every row
of a merged group points at its first row, the rows that pointed at a of a merged group points at its first row, the rows that pointed at a
folded one included. folded one included.
A remote listing and an office listing stay two roles: the kept role One country, as ADR Fetch prep and score 1.4 has it: a job posted once
per office in several countries is several roles, one per country, which
keeps a different opening with an alike text (a federal sales role in
the US and one in Australia) from merging. A remote listing and an office listing stay two roles: the kept role
takes a copy's places and not its remote, so the fetch gate, which reads takes a copy's places and not its remote, so the fetch gate, which reads
a remote role by its area, would reject the remote listing with an a remote role by its area, would reject the remote listing with an
office role elsewhere, and the office one with a remote role elsewhere.""" office role elsewhere, and the office one with a remote role elsewhere."""
@@ -216,7 +226,7 @@ def fold(rows: Iterable[Copy]) -> Folded:
for row in sorted(part, key=keep_order): for row in sorted(part, key=keep_order):
for group in groups: for group in groups:
if all(m.host == row.host and m.remote == row.remote if all(m.host == row.host and m.remote == row.remote
and similar(m.text, row.text) for m in group): and _same_country(m, row) and similar(m.text, row.text) for m in group):
group.append(row) group.append(row)
break break
else: else:
+25 -70
View File
@@ -7,18 +7,14 @@ to three strengths and gaps. A model's rating is an estimate, so the scale is
coarse. The total is `app.models.cv_score`, the three numbers under the coarse. The total is `app.models.cv_score`, the three numbers under the
agent's `cv_*` weights times 10, so 0 to 100, worked out when a role is served agent's `cv_*` weights times 10, so 0 to 100, worked out when a role is served
and not stored, so the weights are no part of `input_key`; it is what the and not stored, so the weights are no part of `input_key`; it is what the
feed's CV match sorts by. The model also reports a verdict on each of the feed's CV match sorts by. Culture and location are left out: the first is
agent's must-haves (one per line of its must text, numbered M1..Mn in the guessed from ad text, the second is what the fetch gate does.
prompt); it does not apply them. The rule is code's: a "broken" one makes
career count as at most `app.models.BROKEN_CAP` (4), when the role is
served. Culture and location are left out: the first is guessed from ad text,
the second is what the fetch gate does.
The work is pairs, `(agent, role)`: an enabled agent with a CV, with or The work is pairs, `(agent, role)`: an enabled agent with a CV, with or
without a prompt, and an open role in its cut (`scores.cut`, see `app.jobs.score`: the best by without a prompt, and an open role in its cut (`scores.cut`, see `app.jobs.score`: the best by
fit, the best by CV closeness that those leave out, and the liked and saved) fit, the best by CV closeness that those leave out, and the liked and saved)
with some text, that has no judgement yet, or one read from another CV, with some text, that has no judgement yet, or one read from another CV,
prompts, like, avoid or must text or role text than today's (`input_key`). prompts, languages, like or avoid text or role text than today's (`input_key`).
The cut is the only selection: the pull judges it after extract and the The cut is the only selection: the pull judges it after extract and the
rescore that follows, so it reads the cut as recomputed. A pair is named by the composite key `"<agent_id>:<uid>"`, rescore that follows, so it reads the cut as recomputed. A pair is named by the composite key `"<agent_id>:<uid>"`,
which is what `desk.modeltask` hands round as a uid, so `desk/` is untouched. which is what `desk.modeltask` hands round as a uid, so `desk/` is untouched.
@@ -51,9 +47,7 @@ from pipeline import NULL, RunContext, Task, job
from pipeline.task import Batch from pipeline.task import Batch
from app.config import Config from app.config import Config
from app.jobs import stages from app.jobs import stages
from app.models import ( from app.models import LANGUAGE_LEVELS, Agent, CvLanguage, _language_name, cv_languages
LANGUAGE_LEVELS, MUST_VERDICTS, Agent, CvLanguage, cv_languages, must_lines,
)
from app.storage import Store from app.storage import Store
#: How much of a posting goes in one item: summarise's 8000, since the #: How much of a posting goes in one item: summarise's 8000, since the
@@ -66,22 +60,28 @@ EXCERPT = 8000
#: 3: what the person likes (`like_text`) beside what they do not, and the #: 3: what the person likes (`like_text`) beside what they do not, and the
#: career rubric reading both. #: career rubric reading both.
#: 4: the three numbers 0 to 10 rather than 0 to 100. #: 4: the three numbers 0 to 10 rather than 0 to 100.
FORMAT = 4 #: 5: the person's languages, read into skills.
#: 6: no must-haves: the profile is the CV, languages, likes and dislikes.
FORMAT = 6
#: How many strengths and gaps are kept, and how long each may be. #: How many strengths and gaps are kept, and how long each may be.
MOST, LONGEST = 3, 300 MOST, LONGEST = 3, 300
SYSTEM = """You judge job postings for one person: how well each fits their CV and what they want. SYSTEM = """You judge job postings for one person: how well each fits their CV and what they want.
Below the rules is the person: their CV, what they want, what they like and Below the rules is the person: their CV, the languages they speak and how
what they do not like (the work, the workplace, situations), and their well, what they want, what they like and what they do not like (the work,
must-haves and deal-breakers when they gave any. Each item is one posting with a key, its title, company, place the workplace, situations). Each item is one posting with a key, its title,
and text. Answer one judgement per item. company, place and text. Answer one judgement per item.
Score three dimensions, each a whole number from 0 to 10: Score three dimensions, each a whole number from 0 to 10:
- skills: 8-10 the core requirements are the person's primary skills; 6-7 - skills: 8-10 the core requirements are the person's primary skills; 6-7
most requirements match, with 1-2 gaps that are learnable; 4-5 a partial most requirements match, with 1-2 gaps that are learnable; 4-5 a partial
match, significant upskilling needed; 0-3 a fundamental mismatch. match, significant upskilling needed; 0-3 a fundamental mismatch. A
language the posting requires counts as a requirement: one the person
speaks below the level asked, or not at all when their languages are
listed, lowers skills and is a gap. A language the posting does not
mention changes nothing.
- experience: match on the function and nature of the work performed, not the - experience: match on the function and nature of the work performed, not the
literal job title, and on its level. 8-10 direct experience in the same literal job title, and on its level. 8-10 direct experience in the same
domain and role type; 6-7 related, with transferable skills clear; 4-5 domain and role type; 6-7 related, with transferable skills clear; 4-5
@@ -93,16 +93,6 @@ Score three dimensions, each a whole number from 0 to 10:
whether the tasks and the setting would energise them, not only whether whether the tasks and the setting would energise them, not only whether
they could do them. they could do them.
Their must-haves and deal-breakers are numbered M1, M2 and so on. Give a
verdict on every one of them, in `musts`, for each item:
- "met": the posting text says it is met.
- "broken": only when the posting text clearly contradicts it.
- "unclear": the text does not say. Never infer from silence: a posting that
is silent on a must-have is "unclear", not "broken".
Judge a must-have by its own words, not by how well the role fits otherwise.
Answer musts as [] when there are none. The verdicts do not change the three
scores, which follow their rubrics; name a broken must-have as a gap.
Then 1 to 3 strengths and 1 to 3 gaps, each one short sentence grounded in Then 1 to 3 strengths and 1 to 3 gaps, each one short sentence grounded in
the posting text. State the gaps honestly: a prestigious role that fits the posting text. State the gaps honestly: a prestigious role that fits
poorly scores low. poorly scores low.
@@ -126,21 +116,8 @@ SCHEMA: dict[str, Any] = {
"career": {"type": "integer"}, "career": {"type": "integer"},
"strengths": {"type": "array", "items": {"type": "string"}}, "strengths": {"type": "array", "items": {"type": "string"}},
"gaps": {"type": "array", "items": {"type": "string"}}, "gaps": {"type": "array", "items": {"type": "string"}},
"musts": {
"type": "array",
"items": {
"type": "object",
"properties": {
"id": {"type": "string"},
"verdict": {"type": "string", "enum": list(MUST_VERDICTS)},
},
"required": ["id", "verdict"],
"additionalProperties": False,
},
},
}, },
"required": ["key", "skills", "experience", "career", "strengths", "gaps", "required": ["key", "skills", "experience", "career", "strengths", "gaps"],
"musts"],
"additionalProperties": False, "additionalProperties": False,
}, },
}, },
@@ -181,10 +158,11 @@ def _text(row: Any) -> str:
def input_key(agent: Agent, text: str) -> str: def input_key(agent: Agent, text: str) -> str:
"""A hash of everything a judgement reads: the CV, the prompts, the """A hash of everything a judgement reads: the CV, the prompts, the
like, avoid and must texts, and the role's text as sent. A judgement stored languages, the like and avoid texts, and the role's text as sent. A judgement
under another key is read again.""" stored under another key is read again."""
blob = json.dumps([FORMAT, agent.cv_text, [[p.name, p.text] for p in agent.prompts], blob = json.dumps([FORMAT, agent.cv_text, [[p.name, p.text] for p in agent.prompts],
agent.like_text, agent.avoid_text, agent.must_text, text], ensure_ascii=False) [[l.language, l.level] for l in agent.cv_languages],
agent.like_text, agent.avoid_text, text], ensure_ascii=False)
return hashlib.sha1(blob.encode()).hexdigest()[:16] return hashlib.sha1(blob.encode()).hexdigest()[:16]
@@ -230,6 +208,9 @@ def pending(store: Store) -> int:
def system(agent: Agent) -> str: def system(agent: Agent) -> str:
"""The rubric, and the person as this agent describes them.""" """The rubric, and the person as this agent describes them."""
parts = [SYSTEM, "", "The person.", "", f"<cv>\n{agent.cv_text.strip()}\n</cv>"] parts = [SYSTEM, "", "The person.", "", f"<cv>\n{agent.cv_text.strip()}\n</cv>"]
if agent.cv_languages:
spoken = ", ".join(f"{_language_name(l.language)} {l.level}" for l in agent.cv_languages)
parts.append(f"\n<languages title=\"the languages they speak, and how well\">\n{spoken}\n</languages>")
wants = "\n".join(f"- {p.name}: {' '.join(p.text.split())}" wants = "\n".join(f"- {p.name}: {' '.join(p.text.split())}"
for p in agent.prompts if p.text.strip()) for p in agent.prompts if p.text.strip())
if wants: if wants:
@@ -240,9 +221,6 @@ def system(agent: Agent) -> str:
if agent.avoid_text.strip(): if agent.avoid_text.strip():
parts.append("\n<dislikes title=\"what they do not like\">\n" parts.append("\n<dislikes title=\"what they do not like\">\n"
f"{agent.avoid_text.strip()}\n</dislikes>") f"{agent.avoid_text.strip()}\n</dislikes>")
if musts := must_lines(agent.must_text):
lines = "\n".join(f"M{n}: {line}" for n, line in enumerate(musts, start=1))
parts.append(f"\n<must title=\"must-haves and deal-breakers\">\n{lines}\n</must>")
return "\n".join(parts) return "\n".join(parts)
@@ -290,25 +268,6 @@ def _lines(value: Any) -> list[str]:
return [line for line in found if line][:MOST] return [line for line in found if line][:MOST]
def _musts(agent: Agent | None, value: Any) -> list[dict[str, str]]:
"""The verdicts on the agent's must-haves as they are stored: one
`{id, text, verdict}` per line of its must text, in order. A must-have
the model left out, or gave a verdict outside "met", "broken", "unclear",
is "unclear": nothing is broken by silence. An id that is not one of the
agent's is ignored, the first answer for an id counts."""
said: dict[str, str] = {}
for one in value if isinstance(value, list) else []:
if isinstance(one, dict):
said.setdefault(str(one.get("id") or "").strip().upper(),
str(one.get("verdict") or "").strip().lower())
found = []
for n, line in enumerate(must_lines(agent.must_text if agent else ""), start=1):
verdict = said.get(f"M{n}", "")
found.append({"id": f"M{n}", "text": line,
"verdict": verdict if verdict in MUST_VERDICTS else "unclear"})
return found
def apply_batch(store: Store, answer: dict[str, Any], def apply_batch(store: Store, answer: dict[str, Any],
batch: dict[str, Any] | None = None, batch: dict[str, Any] | None = None,
config: Config | None = None) -> JudgeResult: config: Config | None = None) -> JudgeResult:
@@ -317,7 +276,6 @@ def apply_batch(store: Store, answer: dict[str, Any],
all three numbers; the pair stays waiting.""" all three numbers; the pair stays waiting."""
keys: dict[str, str] = dict((batch or {}).get("keys") or {}) keys: dict[str, str] = dict((batch or {}).get("keys") or {})
stats = JudgeResult() stats = JudgeResult()
agents: dict[int, Agent | None] = {}
for one in answer.get("judgements") or []: for one in answer.get("judgements") or []:
key = str(one.get("key") or "") key = str(one.get("key") or "")
pair = pair_of(key) pair = pair_of(key)
@@ -326,11 +284,8 @@ def apply_batch(store: Store, answer: dict[str, Any],
stats.rejected.append(key or "?") stats.rejected.append(key or "?")
continue continue
skills, experience, career = (int(m) for m in marks if m is not None) skills, experience, career = (int(m) for m in marks if m is not None)
if pair[0] not in agents:
agents[pair[0]] = store.agent(pair[0])
if not store.set_judgement(pair[0], pair[1], keys.pop(key), skills, experience, career, if not store.set_judgement(pair[0], pair[1], keys.pop(key), skills, experience, career,
_lines(one.get("strengths")), _lines(one.get("gaps")), _lines(one.get("strengths")), _lines(one.get("gaps"))):
_musts(agents[pair[0]], one.get("musts"))):
stats.rejected.append(key) stats.rejected.append(key)
continue continue
stats.judged += 1 stats.judged += 1
+17 -3
View File
@@ -10,7 +10,10 @@ both; nothing here decides how a board is read, so the words never disagree
with the pull as long as they read the keys the pull writes. with the pull as long as they read the keys the pull writes.
A word with a number carries it after a colon ("without text: 12"), and the A word with a number carries it after a colon ("without text: 12"), and the
filter matches the phrase before the colon (`WORDS`). filter matches the phrase before the colon (`WORDS`). One word is outside
`WORDS`: the listings the last read dropped for want of an id, a title or a
link ("3 listings dropped: no title or link", `_runner.valid`), shown on the
board's row and not offered by the filter.
""" """
from __future__ import annotations from __future__ import annotations
@@ -49,7 +52,7 @@ WORDS: dict[str, str] = {
#: The `state` keys `words` reads, `<prefix>:<key>:<target>`. #: The `state` keys `words` reads, `<prefix>:<key>:<target>`.
KEYS = ("failing", "ok", "complete", "swept", "sweep", "filtered", "remote_unasked", KEYS = ("failing", "ok", "complete", "swept", "sweep", "filtered", "remote_unasked",
"narrowed", "detailfailed") "narrowed", "detailfailed", "dropped")
class Texts(NamedTuple): class Texts(NamedTuple):
@@ -84,7 +87,8 @@ def texts(store: Store) -> dict[int, Texts]:
def words(module: ModuleType, state: Mapping[str, str], target: str, def words(module: ModuleType, state: Mapping[str, str], target: str,
sub: Subscription | None, size: int | None, *, plan: Plan | None = None, sub: Subscription | None, size: int | None, *, plan: Plan | None = None,
line: str = "", counts: Texts | None = None) -> list[str]: line: str = "", counts: Texts | None = None) -> list[str]:
"""The limits of one board, in the order of `WORDS`, those that apply. """The limits of one board, in the order of `WORDS`, those that apply, the
rows the last read dropped after "read incomplete".
`state` holds the board's keys (`state_of`), `size` its kept size, `state` holds the board's keys (`state_of`), `size` its kept size,
`plan` and `sub` the reach it is read with (for "read whole"), `line` the `plan` and `sub` the reach it is read with (for "read whole"), `line` the
@@ -105,6 +109,8 @@ def words(module: ModuleType, state: Mapping[str, str], target: str,
out.append("failing") out.append("failing")
if state.get(key("ok")) and not state.get(key("complete")): if state.get(key("ok")) and not state.get(key("complete")):
out.append("read incomplete") out.append("read incomplete")
if dropped := _count(state.get(key("dropped"))):
out.append(f"{dropped} listing{'' if dropped == 1 else 's'} dropped: no title or link")
if state.get(key("swept")) == "1" or key("sweep") in state: if state.get(key("swept")) == "1" or key("sweep") in state:
out.append("swept") out.append("swept")
if state.get(key("filtered")) == "1": if state.get(key("filtered")) == "1":
@@ -130,6 +136,14 @@ def words(module: ModuleType, state: Mapping[str, str], target: str,
return out return out
def _count(value: str | None) -> int:
"""A number kept in state, 0 for none or one that does not parse."""
try:
return max(0, int(value or 0))
except ValueError:
return 0
def _whole(reads: Reads, sub: Subscription | None, size: int | None, plan: Plan | None) -> bool: def _whole(reads: Reads, sub: Subscription | None, size: int | None, plan: Plan | None) -> bool:
"""Whether the reach leaves the board unread by place: its reader cannot """Whether the reach leaves the board unread by place: its reader cannot
filter, or the board is under `SMALL` roles (the line's two "read whole" filter, or the board is under `SMALL` roles (the line's two "read whole"
+21 -14
View File
@@ -37,7 +37,12 @@ not swept clears them.
A read narrowed by the agents' fetch terms (`ats:narrowed:<target>`) lists A read narrowed by the agents' fetch terms (`ats:narrowed:<target>`) lists
the roles matching a word, which says nothing of the roles it did not list, the roles matching a word, which says nothing of the roles it did not list,
so it closes by list nothing: a role of such a board closes once the board so it closes by list nothing: a role of such a board closes once the board
has not listed it for `STALE_DAYS`, the rule searches have. has not listed it for `STALE_DAYS`, the rule searches have. The same age
rule is the floor under every board (ADR Fetch prep and score 1.3): an
incomplete read (a ceiling, a page cap without a sweep) and a filtered read,
which spares the roles acted on and the remote ones it could not ask for,
close by age what no read has listed for `STALE_DAYS` (`_close_aged`), so no
role stays open for good.
A search never closes by its list, even one stamped complete because it A search never closes by its list, even one stamped complete because it
read as many results as its provider stated: a search is never a company's read as many results as its provider stated: a search is never a company's
@@ -365,7 +370,10 @@ def close_boards(config: Config, store: Store, tally: Tally, *,
incomplete += 1 incomplete += 1
if not store.state.get(f"ats:sweep:{sub.target}"): if not store.state.get(f"ats:sweep:{sub.target}"):
partial.append(sub.target) partial.append(sub.target)
continue # An incomplete read closes nothing by its list; what no
# read has listed for STALE_DAYS still goes.
read = _close_aged(store, source, sub.id, json.loads(
store.state.get(f"{source}:ids:{sub.target}", "[]")), stale)
shut = read shut = read
closed += shut closed += shut
before = open_before.get(sub.id, 0) before = open_before.get(sub.id, 0)
@@ -394,8 +402,9 @@ def _close_read(store: Store, source: str, sub: Any, stale: str) -> int | None:
clears that record and closes what it did not list, all of it unfiltered clears that record and closes what it did not list, all of it unfiltered
(`Store.mark_closed`), filtered sparing the roles acted on (`Store.mark_closed`), filtered sparing the roles acted on
(`_close_unlisted`). A filtered read that could not ask for remote roles (`_close_unlisted`). A filtered read that could not ask for remote roles
(`remote_unasked`) spares the remote ones as well, and closes those by (`remote_unasked`) spares the remote ones as well. What a filtered read
age instead.""" spares closes by age (`_close_aged`), as on an incomplete read
(`close_boards`)."""
state, target = store.state, sub.target state, target = store.state, sub.target
if not state.get(f"{source}:complete:{target}", ""): if not state.get(f"{source}:complete:{target}", ""):
return None return None
@@ -404,8 +413,7 @@ def _close_read(store: Store, source: str, sub: Any, stale: str) -> int | None:
return _close_aged(store, source, sub.id, seen_ids, stale) return _close_aged(store, source, sub.id, seen_ids, stale)
is_filtered = bool(state.get(f"{source}:filtered:{target}", "")) is_filtered = bool(state.get(f"{source}:filtered:{target}", ""))
spare_remote = is_filtered and bool(state.get(f"{source}:remote_unasked:{target}", "")) spare_remote = is_filtered and bool(state.get(f"{source}:remote_unasked:{target}", ""))
aged = (_close_aged(store, source, sub.id, seen_ids, stale, remote_only=True) aged = _close_aged(store, source, sub.id, seen_ids, stale) if is_filtered else 0
if spare_remote else 0)
if state.get(f"{source}:swept:{target}", ""): if state.get(f"{source}:swept:{target}", ""):
return aged + _close_swept(store, source, sub, seen_ids, is_filtered, return aged + _close_swept(store, source, sub, seen_ids, is_filtered,
spare_remote=spare_remote) spare_remote=spare_remote)
@@ -484,20 +492,19 @@ def _close_unlisted(store: Store, source: str, seen_ids: list[str], subscription
def _close_aged(store: Store, source: str, subscription_id: int, seen_ids: list[str], def _close_aged(store: Store, source: str, subscription_id: int, seen_ids: list[str],
stale: str, *, remote_only: bool = False) -> int: stale: str) -> int:
"""Close the open roles of one board that its read did not list and that """Close the open roles of one board that its read did not list and that
no read has listed since `stale` (`roles.last_seen`, which `upsert_role` no read has listed since `stale` (`roles.last_seen`, which `upsert_role`
writes on every listing): the rule for what a read cannot speak for. A writes on every listing): the rule for what a read cannot speak for, a
read narrowed by terms closes all such roles, as a search does; with read narrowed by terms, an incomplete one, or what a filtered read
`remote_only`, a filtered read that could not ask for remote closes only spares. A role acted on is not spared here: three weeks unlisted is the
its remote roles so, sparing those acted on as its list does.""" one evidence a filtered or incomplete read gives."""
where = " AND remote = 'remote'" + _ACTED_ON if remote_only else ""
with store.transaction() as conn: with store.transaction() as conn:
return conn.execute( return conn.execute(
f"""UPDATE roles SET closed_at = ? """UPDATE roles SET closed_at = ?
WHERE source = ? AND subscription_id = ? AND closed_at IS NULL WHERE source = ? AND subscription_id = ? AND closed_at IS NULL
AND last_seen < ? AND last_seen < ?
AND source_id NOT IN (SELECT value FROM json_each(?)){where}""", AND source_id NOT IN (SELECT value FROM json_each(?))""",
(store.now(), source, subscription_id, stale, (store.now(), source, subscription_id, stale,
json.dumps(list(seen_ids)))).rowcount json.dumps(list(seen_ids)))).rowcount
+39 -8
View File
@@ -283,9 +283,18 @@ class Plan:
return self._big(sub) return self._big(sub)
def _big(self, sub: Subscription) -> tuple[Reach, str]: def _big(self, sub: Subscription) -> tuple[Reach, str]:
"""The reach of `sub` as a board of `SMALL` roles or more.""" """The reach of `sub` as a board of `SMALL` roles or more. A host
that cannot filter on remote while the agents want remote roles is
asked for the wide reach, the agents' world (ADR Fetch prep and
score 1.2): asked for "DK, SE" alone it would never list a remote
role filed under Germany, and the gate sorts the rest out. Only
where the host takes the world in one read, a list of countries or
their ids: on a host that takes one country a request, Europe is
the whole board, so the location read stays and remote is not asked."""
if company_key(sub.company or sub.label) in self.all_countries: if company_key(sub.company or sub.label) in self.all_countries:
return self.wide, "countries" return self.wide, "countries"
if self.base.remote and not _filters_remote(sub) and _takes_world(sub, self.wide):
return self.wide, "world"
return self.base, "agents" return self.base, "agents"
def narrows_roles(self, sub: Subscription) -> bool: def narrows_roles(self, sub: Subscription) -> bool:
@@ -305,15 +314,34 @@ class Plan:
if why == "unknown": if why == "unknown":
found, why = self._big(sub) found, why = self._big(sub)
out = found.params() out = found.params()
if self.terms and why in ("countries", "agents") and self.narrows_roles(sub): if self.terms and why in ("countries", "agents", "world") and self.narrows_roles(sub):
out = (out or {}) | {"terms": list(self.terms)} out = (out or {}) | {"terms": list(self.terms)}
if why == "countries": if why in ("countries", "world"):
out["wide"] = True out["wide"] = True
if sub.target not in self.sizes: if sub.target not in self.sizes:
out = (out or {}) | {"unsized": True, "small": SMALL} out = (out or {}) | {"unsized": True, "small": SMALL}
return out return out
def _filters_remote(sub: Subscription) -> bool:
"""Whether `sub`'s host declares a remote filter at all. A host that
declares one by ids may still lack the facet on a site, which the
runner finds out when it asks (`_runner.ask_for`)."""
module = readers.HOSTS.get(sub.target.partition(":")[0])
return module is None or module.READS.filters.remote != "none"
def _takes_world(sub: Subscription, wide: Reach) -> bool:
"""Whether `sub`'s host can be asked for the wide reach in one read: a
country filter that takes a list (`many`, `ids`), or one value a request
with the countries within `MAX_SERIES`; never everywhere."""
module = readers.HOSTS.get(sub.target.partition(":")[0])
if module is None or wide.everywhere or not wide.countries:
return False
reach = module.READS.filters.country
return reach in ("many", "ids") or (reach == "one" and len(wide.countries) <= _runner.MAX_SERIES)
def sizes(store: Store, subs: Iterable[Subscription]) -> dict[str, int]: def sizes(store: Store, subs: Iterable[Subscription]) -> dict[str, int]:
"""Each board's size as kept (`ats:size:<target>`); a board never sized """Each board's size as kept (`ats:size:<target>`); a board never sized
has none.""" has none."""
@@ -381,7 +409,7 @@ def reaches(store: Store, subs: Iterable[Subscription]) -> Mapping[str, dict[str
#: Why a board is read whole, as the Companies list says it. #: Why a board is read whole, as the Companies list says it.
_WHOLE = {"small": f"under {SMALL} roles", "unknown": "size not known yet", _WHOLE = {"small": f"under {SMALL} roles", "unknown": "size not known yet",
"override": "set here", "countries": "all locations (everywhere)", "override": "set here", "countries": "all locations (everywhere)",
"agents": "an agent takes any place"} "world": "the agents' world (everywhere)", "agents": "an agent takes any place"}
def _codes(codes: Iterable[str]) -> list[str]: def _codes(codes: Iterable[str]) -> list[str]:
@@ -486,6 +514,9 @@ def line(found: Plan, sub: Subscription) -> str:
if why == "countries": if why == "countries":
named = found.wide_names or f"{len(found.wide.countries)} countries" named = found.wide_names or f"{len(found.wide.countries)} countries"
return f"all locations ({named}){unasked} · {roles}" return f"all locations ({named}){unasked} · {roles}"
if why == "world":
named = found.wide_names or f"{len(found.wide.countries)} countries"
return f"follows agents' world ({named}){unasked} · {roles}"
if why == "override": if why == "override":
return f"set here: {_places(ask)}{unasked} · {roles}" return f"set here: {_places(ask)}{unasked} · {roles}"
return f"follows agents: {_places(ask)}{unasked} · {roles}" return f"follows agents: {_places(ask)}{unasked} · {roles}"
@@ -494,14 +525,14 @@ def line(found: Plan, sub: Subscription) -> str:
what = "all locations (everywhere)" if why == "countries" else _WHOLE[why] what = "all locations (everywhere)" if why == "countries" else _WHOLE[why]
return f"{what} · {roles}" return f"{what} · {roles}"
return f"all places, too many countries to ask · {roles}" return f"all places, too many countries to ask · {roles}"
if why in ("agents", "countries") and given.everywhere: if why in ("agents", "countries", "world") and given.everywhere:
return f"{_WHOLE[why]}: read whole" return f"{_WHOLE[why]}: read whole"
if why in ("agents", "countries", "override") and not module.READS.filters.declared(): if why in ("agents", "countries", "world", "override") and not module.READS.filters.declared():
return f"{module.LABEL} cannot filter: read whole" return f"{module.LABEL} cannot filter: read whole"
reads = module.READS.filters reads = module.READS.filters
if why in ("agents", "countries") and reads.country == "one" \ if why in ("agents", "countries", "world") and reads.country == "one" \
and len(given.countries) > _runner.MAX_SERIES: and len(given.countries) > _runner.MAX_SERIES:
return "too many countries to ask: read whole" return "too many countries to ask: read whole"
if why in ("agents", "countries"): if why in ("agents", "countries", "world"):
return f"{module.LABEL} cannot ask for these: read whole" return f"{module.LABEL} cannot ask for these: read whole"
return f"{_WHOLE[why]}: read whole" return f"{_WHOLE[why]}: read whole"
+47 -4
View File
@@ -4,8 +4,9 @@
uv run uvicorn app.main:app --reload uv run uvicorn app.main:app --reload
`create_app()` is the whole of the wiring, and the order in it is the point: `create_app()` is the whole of the wiring, and the order in it is the point:
config and store, the door, the queue, the pages. Everything before the queue config and store, the door, the queue, the pages, and around all of them the
is something the queue needs; everything after it is something that enqueues. hardening headers. Everything before the queue is something the queue needs;
everything after it is something that enqueues.
""" """
from __future__ import annotations from __future__ import annotations
@@ -17,7 +18,9 @@ from fastapi import FastAPI, Request
from fastapi.exceptions import RequestValidationError from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse from fastapi.responses import JSONResponse
from fastapi.staticfiles import StaticFiles from fastapi.staticfiles import StaticFiles
from starlette.datastructures import MutableHeaders
from starlette.exceptions import HTTPException as StarletteHTTPException from starlette.exceptions import HTTPException as StarletteHTTPException
from starlette.types import ASGIApp, Message, Receive, Scope, Send
import auth import auth
from app import jobs from app import jobs
@@ -33,6 +36,39 @@ log = logging.getLogger(__name__)
#: data gets `{"detail": ...}`; a browser that asked for a page gets one. #: data gets `{"detail": ...}`; a browser that asked for a page gets one.
API_PREFIXES = ("/api", "/ws", "/_jobs", "/_auth", "/static", "/health") API_PREFIXES = ("/api", "/ws", "/_jobs", "/_auth", "/static", "/health")
#: On every HTTP answer, the door's refusals and the error pages included. No
#: CSP yet: the templates carry inline scripts and styles, which would need a
#: nonce or a hash each.
HARDENING = {
"X-Content-Type-Options": "nosniff",
"X-Frame-Options": "DENY",
"Referrer-Policy": "same-origin",
}
class Hardening:
"""`HARDENING` on every response that does not set the header itself.
Pure ASGI rather than `BaseHTTPMiddleware`, so a streamed answer and the
socket pass through untouched."""
def __init__(self, app: ASGIApp) -> None:
self.app = app
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
async def harden(message: Message) -> None:
if message["type"] == "http.response.start":
headers = MutableHeaders(scope=message)
for name, value in HARDENING.items():
headers.setdefault(name, value)
await send(message)
await self.app(scope, receive, harden)
def create_app(config: Config | None = None) -> FastAPI: def create_app(config: Config | None = None) -> FastAPI:
"""Build the application. A test passes its own config; nobody else does.""" """Build the application. A test passes its own config; nobody else does."""
@@ -59,8 +95,12 @@ def create_app(config: Config | None = None) -> FastAPI:
# The door, first, because it is the door. The middleware covers every # The door, first, because it is the door. The middleware covers every
# route whenever it was added, `/ws/jobs` included; written first, nobody # route whenever it was added, `/ws/jobs` included; written first, nobody
# has to work that out. `auto` locks unless the bind address is loopback. # has to work that out. `auto` locks unless the bind address is loopback,
# Auth's tables are the `auth_*` ones in the same file. # and an app it leaves open answers 421 to a request that names any host
# but a loopback one or comes through a proxy. Auth's tables are the
# `auth_*` ones in the same file, `auth_tokens` for the API tokens among
# them. No `session_only=`: no route here hands out or replaces the
# database as a file.
auth.connect(app, auth.AuthStore(config.database), config.auth, auth.connect(app, auth.AuthStore(config.database), config.auth,
host=config.server.host, open=("/static/",), host=config.server.host, open=("/static/",),
stylesheet="/static/app.css") stylesheet="/static/app.css")
@@ -84,6 +124,9 @@ def create_app(config: Config | None = None) -> FastAPI:
--wait` waits on this, so it has to prove the database answers.""" --wait` waits on this, so it has to prove the database answers."""
return {"ok": True, "open_roles": sum(store.open_role_counts().values())} return {"ok": True, "open_roles": sum(store.open_role_counts().values())}
# Last, so outermost: around the door and the error pages.
app.add_middleware(Hardening)
@app.exception_handler(StarletteHTTPException) @app.exception_handler(StarletteHTTPException)
async def errors(request: Request, exc: StarletteHTTPException) -> Any: async def errors(request: Request, exc: StarletteHTTPException) -> Any:
"""JSON for a caller, a page for a browser. One shape for every API """JSON for a caller, a page for a browser. One shape for every API
+9 -62
View File
@@ -1008,79 +1008,30 @@ def language_flag(required: list[RoleLanguage],
return "", "" return "", ""
#: What career counts as at most, on the judge's 0 to 10, when a posting
#: breaks one of the agent's must-haves: the foot of the rubric's 4-5 band,
#: a decent job that does not build toward the person's goals. The model
#: reports whether each must-have is met; this cap is applied in code, when a
#: role is served (`cv_score`, `Store.judgement`, `store._cv_sql`).
BROKEN_CAP = 4
#: A must-have's verdict, as the judge answers it.
MUST_VERDICTS = ("met", "broken", "unclear")
_BULLET = re.compile(r"^\s*(?:[-*\u2022]+|\(?\d+[.)])\s+")
def must_lines(text: str) -> list[str]:
"""The agent's must-haves text read as one must-have per line: a leading
bullet (`-`, `*`, `•`) or number (`1.`, `2)`) is dropped, so are empty
lines."""
found = []
for line in text.splitlines():
line = " ".join(_BULLET.sub("", line, count=1).split())
if line:
found.append(line)
return found
def capped_career(career: int, broken: int = 0) -> int:
"""`career` as it counts: at most `BROKEN_CAP` when any must-have is
broken."""
return min(career, BROKEN_CAP) if broken else career
def cv_score(skills: int, experience: int, career: int, def cv_score(skills: int, experience: int, career: int,
weights: Weights | None = None, broken: int = 0) -> float | None: weights: Weights | None = None) -> float | None:
"""The one number of a CV match, 0 to 100 for sorting and display: the """The one number of a CV match, 0 to 100 for sorting and display: the
judge's three numbers, each 0 to 10, averaged by the agent's judge's three numbers, each 0 to 10, averaged by the agent's
`cv_skills`, `cv_experience` and `cv_career` weights (30, 25 and 30 by `cv_skills`, `cv_experience` and `cv_career` weights (30, 25 and 30 by
default), career as `capped_career` has it, times 10. None when all default), times 10. None when all three weights are 0."""
three weights are 0."""
w = weights or Weights() w = weights or Weights()
total = w.cv_skills + w.cv_experience + w.cv_career total = w.cv_skills + w.cv_experience + w.cv_career
if not total: if not total:
return None return None
career = capped_career(career, broken)
return round(10 * (w.cv_skills * skills + w.cv_experience * experience return round(10 * (w.cv_skills * skills + w.cv_experience * experience
+ w.cv_career * career) / total, 1) + w.cv_career * career) / total, 1)
class Must(Model):
"""One must-have of an agent as the judge read it against a role: its
`id` (`M1`, by line), the line's `text` and the `verdict`."""
id: str
text: str = ""
verdict: Literal["met", "broken", "unclear"] = "unclear"
class Judgement(Model): class Judgement(Model):
"""What `judge` read of one role for one agent (`role_judgements`): the """What `judge` read of one role for one agent (`role_judgements`): the
three dimensions 0 to 10 as the model gave them, `cv_score` three dimensions 0 to 10 as the model gave them, `cv_score`
(`cv_score`, 0 to 100) and 1 to 3 strengths and gaps grounded in the (`cv_score`, 0 to 100) and 1 to 3 strengths and gaps grounded in the
posting. posting. `cv_score` is not read from the table: `Store.judgement` sets
`musts` is the verdict on each must-have and `broken` how many are it from the agent's weights; it is None when all three are 0."""
broken; any makes `career_capped` the lower of `career` and
`BROKEN_CAP`. `cv_score` and `career_capped` are not read from the
table: `Store.judgement` sets them from the agent's weights; `cv_score`
is None when all three are 0."""
skills: int skills: int
experience: int experience: int
career: int career: int
career_capped: int | None = None
broken: int = 0
musts: Annotated[list[Must], BeforeValidator(lambda v: _json(v, "[]"))] = []
cv_score: float | None = None cv_score: float | None = None
strengths: JsonStrings = [] strengths: JsonStrings = []
gaps: JsonStrings = [] gaps: JsonStrings = []
@@ -1574,14 +1525,11 @@ class Agent(Model):
are the union over the enabled agents, and an agent with none means no are the union over the enabled agents, and an agent with none means no
board is narrowed by words. They decide what is fetched, not what ranks. board is narrowed by words. They decide what is fetched, not what ranks.
`cv_text`, `cv_languages`, `like_text`, `avoid_text` and `must_text` are `cv_text`, `cv_languages`, `like_text` and `avoid_text` are what `judge`
what `judge` reads a role against: the CV, the languages it states with reads a role against: the CV, the languages it states with a level, and
a level, what the person likes and does not like (the work, the what the person likes and does not like (the work, the workplace,
workplace, situations; both feed the career score), and their situations; both feed the career score). An agent without a CV is not
must-haves and deal-breakers (set through the API only, not on the judged, and one without languages flags none (`language_flag`)."""
form; soft, read by the model only; a hard rule is a gate). An agent without
a CV is not judged, and one without languages flags none
(`language_flag`)."""
id: int id: int
name: str name: str
@@ -1594,7 +1542,6 @@ class Agent(Model):
cv_languages: CvLanguages = [] cv_languages: CvLanguages = []
like_text: str = "" like_text: str = ""
avoid_text: str = "" avoid_text: str = ""
must_text: str = ""
enabled: bool = True enabled: bool = True
created_at: str created_at: str
+6 -2
View File
@@ -37,8 +37,9 @@ from app.storage import Store
#: Raised when the rules in this package change the way a stored word maps, #: Raised when the rules in this package change the way a stored word maps,
#: so every open role is mapped again as it is after a table changes. 3: the #: so every open role is mapped again as it is after a table changes. 3: the
#: remote area, and remote as the model's before the host's word. 4: the #: remote area, and remote as the model's before the host's word. 4: the
#: deadline, the host's before the model's. #: deadline, the host's before the model's. 5: `remote_text`, the title or
VERSION = 4 #: text saying remote while no field does.
VERSION = 5
#: The state key the last run's `fingerprint` is kept under. #: The state key the last run's `fingerprint` is kept under.
STATE = "prep:tables" STATE = "prep:tables"
@@ -61,6 +62,7 @@ def fingerprint(tables: Tables = TABLES) -> str:
"""A hash of every host table, the shared remote words, the gazetteer the """A hash of every host table, the shared remote words, the gazetteer the
code describes and `VERSION`.""" code describes and `VERSION`."""
blob = json.dumps({"version": VERSION, "tables": tables, "remote": words.REMOTE_WORDS, blob = json.dumps({"version": VERSION, "tables": tables, "remote": words.REMOTE_WORDS,
"remote_text": [words.REMOTE_TEXT, words.REMOTE_TEXT_NOT],
"places": repr(places.seed())}, sort_keys=True) "places": repr(places.seed())}, sort_keys=True)
return hashlib.sha1(blob.encode()).hexdigest() return hashlib.sha1(blob.encode()).hexdigest()
@@ -114,6 +116,7 @@ def map_role(row: Mapping[str, Any], gaz: places.Gazetteer, folds: Mapping[str,
"seniority_title": titled, "seniority_title": titled,
"employment": words.employment(host, row["employment_host"], tables), "employment": words.employment(host, row["employment_host"], tables),
"remote": remote, "remote": remote,
"remote_text": int(words.remote_in_text(row["title"], row["text"] or "")),
"remote_area": list(area(gaz, texts, place_ids, _model_names(row["remote_where_model"])) "remote_area": list(area(gaz, texts, place_ids, _model_names(row["remote_where_model"]))
if remote == "remote" else ()), if remote == "remote" else ()),
"country": country(gaz, place_ids, row["country_host"]), "country": country(gaz, place_ids, row["country_host"]),
@@ -125,6 +128,7 @@ def _changed(row: Mapping[str, Any], found: Mapping[str, Any]) -> bool:
level = row["seniority_model"] or found["seniority_stated"] or found["seniority_title"] level = row["seniority_model"] or found["seniority_stated"] or found["seniority_title"]
return bool(row["prep_due"]) or level != row["seniority"] \ return bool(row["prep_due"]) or level != row["seniority"] \
or found["remote"] != row["remote"] \ or found["remote"] != row["remote"] \
or found["remote_text"] != row["remote_text"] \
or found["remote_area"] != json.loads(row["remote_area"] or "[]") \ or found["remote_area"] != json.loads(row["remote_area"] or "[]") \
or (found["country"] and found["country"] != row["country"]) \ or (found["country"] and found["country"] != row["country"]) \
or found["deadline"] != row["deadline"] \ or found["deadline"] != row["deadline"] \
+33
View File
@@ -58,6 +58,39 @@ _NOT_REMOTE_WORK = re.compile(r"\bremote[\s-]+(?:sensing|control\w*|monitoring)\
re.IGNORECASE) re.IGNORECASE)
#: Phrases in a title or text that say the role is worked remotely, and the
#: phrases that take one back. Both are read on the lowercased text with its
#: hyphens and runs of spaces squashed to one space. A match marks the role as
#: possibly remote (`remote_in_text`); it decides nothing by itself.
REMOTE_TEXT: tuple[str, ...] = (
"fully remote", "100% remote", "100 % remote", "remote first", "remote work",
"remote working", "work remotely", "working remotely", "remote position", "remote role",
"remote job", "remote team", "remote friendly", "remote possible", "remote option",
"remote ok", "work from home", "working from home", "work from anywhere", "home based",
"home office", "homeoffice", "hjemmearbejde", "fjernarbejde", "distansarbete",
)
REMOTE_TEXT_NOT: tuple[str, ...] = (
"no remote", "not remote", "not a remote", "non remote", "not be remote", "cannot be remote",
"no home office", "kein homeoffice", "not available remotely", "not open to remote",
"remote work is not", "remote working is not", "is not a remote", "remote sensing",
"remote control", "remote monitoring",
)
_REMOTE_TEXT = re.compile("|".join(rf"\b{re.escape(p)}\b" for p in REMOTE_TEXT))
_REMOTE_TEXT_NOT = re.compile("|".join(rf"\b{re.escape(p)}\b" for p in REMOTE_TEXT_NOT))
_SQUASHED = re.compile(r"[\s-]+")
def remote_in_text(title: str, text: str) -> bool:
"""Whether the title or the text says the role is worked remotely: a
phrase of `REMOTE_TEXT` left once those of `REMOTE_TEXT_NOT` are taken
out. Cheap evidence, no model: a role marked so passes a location gate
on remote until extract has read it (ADR Fetch prep and score 1.1), so a
false mark costs a cut slot at most, and a missed one nothing new."""
folded = _SQUASHED.sub(" ", f"{title or ''} {text or ''}".lower())
return bool(_REMOTE_TEXT.search(_REMOTE_TEXT_NOT.sub(" ", folded)))
def _line_word(text: str) -> str: def _line_word(text: str) -> str:
"""`remote_of`'s reading of a line, else its Danish and Swedish words.""" """`remote_of`'s reading of a line, else its Danish and Swedish words."""
if found := remote_of(text): if found := remote_of(text):
+55 -87
View File
@@ -9,13 +9,15 @@
* ADR's order: the world (the scope select), the location (a short list of * ADR's order: the world (the scope select), the location (a short list of
* lines a role passes by matching any one of: countries, towns, remote), the * lines a role passes by matching any one of: countries, towns, remote), the
* role exclusion (the titles to leave out) and the role fetch (the fetch * role exclusion (the titles to leave out) and the role fetch (the fetch
* terms), over a row per remaining rule under Advanced; a gate the four parts * terms), and nothing else: seniority, pay and the like are filters on Roles.
* cannot say lands in a row, so a load and a save lose nothing. A gate is a * A gate is a field, values and a mode (must, one of, not), with no weight.
* field, values and a mode (must, one of, not), with no weight. After it come * A stored gate the four parts cannot say, which only the config or the API
* the analytics filter (the best N by prompt closeness and the best M by CV * writes, is named in a note under the fetch gate and sent back as it came,
* closeness, each with a minimum level), the want feed score weights (prompt * so a load and a save lose nothing. After the fetch gate come the analytics
* closeness, newness, company like, job like), the CV match weights and the * filter (the best N by prompt closeness and the best M by CV closeness, each
* size of summarise. Edit turns a card * with a minimum level), the want feed score weights (prompt closeness,
* newness, company like, job like), the CV match weights and the size of
* summarise. Edit turns a card
* into the form from the page's `<template>`, in place, and New agent puts a * into the form from the page's `<template>`, in place, and New agent puts a
* blank card on top in the same state. One card edits at a time, so the * blank card on top in the same state. One card edits at a time, so the
* form's ids stay unique. Votes, liked companies and scores are the agent's, * form's ids stay unique. Votes, liked companies and scores are the agent's,
@@ -30,10 +32,7 @@
* text (typed, or read from a .md/.txt file by the browser), the languages it * text (typed, or read from a .md/.txt file by the browser), the languages it
* states as rows of a code and a level, and the two soft texts, what the * states as rows of a code and a level, and the two soft texts, what the
* person likes and what they do not like (the work, the workplace, * person likes and what they do not like (the work, the workplace,
* situations), which the judge reads for the career score. The must-have text * situations), which the judge reads for the career score.
* is no longer on the form: a save leaves `must_text` out, so the stored one
* is kept (it is set through the API, and still caps career at 4 in code
* when the judge reports one broken).
* Every save sends the CV, languages and the two texts, and the card's switch * Every save sends the CV, languages and the two texts, and the card's switch
* re-posts the row as the server gave it, so neither drops them. * re-posts the row as the server gave it, so neither drops them.
* *
@@ -64,9 +63,8 @@
const NEW = "new"; const NEW = "new";
// Europe as the starter agents spell it (static/remote.js). // Europe as the starter agents spell it (static/remote.js).
const EUROPE = rh.EUROPE; const EUROPE = rh.EUROPE;
const FIELDS = ["place", "country", "remote", "remote_from", "seniority", "function", "salary_min", // A gate's mode in the words of the note on the gates the form keeps.
"currency", "permit", "languages", "company", "title"]; const MODES = { require: "must", any: "one of", exclude: "not" };
const MODES = [["require", "must"], ["any", "one of"], ["exclude", "not"]];
/** @type {object[]} every agent, as the server lists them */ /** @type {object[]} every agent, as the server lists them */
let agents = []; let agents = [];
@@ -266,8 +264,8 @@
} }
// The region and place in words, one sentence led by the region, then the lines a // The region and place in words, one sentence led by the region, then the lines a
// role may match, with the one-of rows among them since they are the same // role may match, with any other one-of gates among them since they are the
// group: "Within Europe: keeps roles anywhere in Denmark, in Malmö, Lund or // same group: "Within Europe: keeps roles anywhere in Denmark, in Malmö, Lund or
// Helsingborg, or remote, and leaves out the rest." The lines add up, which // Helsingborg, or remote, and leaves out the rest." The lines add up, which
// the first wording ("Roles in ...") left the owner to guess. // the first wording ("Roles in ...") left the owner to guess.
function whereSentence(where, others) { function whereSentence(where, others) {
@@ -384,7 +382,7 @@
// ---------------------------------------------------------------- the form // ---------------------------------------------------------------- the form
// Tell the form something changed that no field typed: a chip or a row // Tell the form something changed that no field typed: a chip or a row
// added or removed. It marks the form dirty and redraws the gates view. // added or removed. It marks the form dirty and redraws the gates' sentence.
function changed(node) { function changed(node) {
node.dispatchEvent(new Event("input", { bubbles: true })); node.dispatchEvent(new Event("input", { bubbles: true }));
} }
@@ -559,9 +557,9 @@
return box; return box;
} }
// A language as a row, after the rule rows: a code, a level, Remove. The // A language as a row: a code, a level, Remove. The code is two letters,
// code is two letters, which the server turns into the language; its name // which the server turns into the language; its name is the row's title
// is the row's title where the browser knows it. // where the browser knows it.
function languageRow(language, remove) { function languageRow(language, remove) {
const row = el("div", "rule-row"); const row = el("div", "rule-row");
const code = el("input"); const code = el("input");
@@ -587,32 +585,32 @@
return { node: row, read: () => ({ language: code.value.trim().toLowerCase(), level: level.value }) }; return { node: row, read: () => ({ language: code.value.trim().toLowerCase(), level: level.value }) };
} }
// A gate as a row: field, mode, values and Remove. Values untouched since the load are sent as they came, so a // The gates the fetch gate's four parts say, in the order the starter
// value with a comma in it survives a save. // agents have them: the world, the location lines and the titles left out,
function ruleRow(gate, remove) { // then `kept`, the stored gates the parts cannot say, as they came. `parts`
const row = el("div", "rule-row"); // holds what the form reads: `scope` (europe, countries, everywhere) and
const field = select(FIELDS.map((f) => [f, f.replace("_", " ")]), gate.field, "Field"); // its `scoped` countries, the `within` countries, the `towns`, `remote`
const mode = select(MODES, gate.mode, "Mode"); // (none, scope, from) and its `from` areas, and the `titles`.
const values = el("input"); function formGates(parts, kept = []) {
values.type = "text"; const out = [];
values.required = true; const { scope, scoped, within, towns: lines, remote, from, titles } = parts;
values.spellcheck = false; if (scope === "europe") out.push({ field: "country", values: rh.europeWith(within), mode: "require" });
values.placeholder = "senior, staff"; if (scope === "countries" && scoped.length) out.push({ field: "country", values: scoped, mode: "require" });
values.value = gate.values.join(", "); if (within.length) out.push({ field: "country", values: within, mode: "any" });
values.setAttribute("aria-label", "Values, separated by commas"); if (remote === "scope") out.push({ field: "remote", values: ["remote"], mode: "any" });
const loaded = values.value; if (remote === "from" && from.length) out.push({ field: "remote_from", values: from, mode: "any" });
const drop = linky("Remove"); if (lines.length) out.push({ field: "place", values: lines, mode: "any" });
drop.addEventListener("click", () => remove(row)); if (titles.length) out.push({ field: "title", values: titles, mode: "exclude" });
row.append(field, mode, values, drop); return out.concat(kept);
}
function read() { // The note on the gates the form keeps: "The form cannot edit these rules,
const parts = values.value === loaded // and Save keeps them as stored: must: with seniority senior; not: at Acme."
? gate.values function keptNote(kept) {
: values.value.split(",").map((v) => v.trim()).filter(Boolean) if (!kept.length) return "";
.map((v) => (field.value === "salary_min" ? Number(v) : v)); const said = kept.map((g) => `${MODES[g.mode] || g.mode}: ${phrase(g)}`).join("; ");
return { field: field.value, values: parts, mode: mode.value }; return `The form cannot edit ${kept.length > 1 ? "these rules" : "this rule"}, and Save keeps `
} + `${kept.length > 1 ? "them" : "it"} as stored: ${said}.`;
return { node: row, read };
} }
function form(row) { function form(row) {
@@ -713,51 +711,18 @@
remoteFrom.node.hidden = remoteMode.value !== "from"; remoteFrom.node.hidden = remoteMode.value !== "from";
}; };
offer(); offer();
$("details.advanced", f).open = rest.length > 0; const kept = $("#a-kept", f);
kept.textContent = keptNote(rest);
kept.hidden = !rest.length;
const rules = []; // The gates Save sends: the four parts', then the kept ones. The sentence
const dropRule = (node) => { // under them says them all, as the card will, the kept ones included.
rules.splice(rules.findIndex((r) => r.node === node), 1); const gates = () => formGates({
node.remove(); scope: scope.value, scoped: scoped.values(), within: countries.values(), towns: places.values(),
changed(f); remote: remoteMode.value, from: remoteFrom.values(), titles: titles.values(),
}; }, rest);
const addRule = (gate) => {
const entry = ruleRow(gate, dropRule);
rules.push(entry);
$("#a-rules", f).appendChild(entry.node);
return entry;
};
rest.forEach(addRule);
$("#a-add-rule", f).addEventListener("click", () => {
$("select", addRule({ field: "seniority", values: [], mode: "require" }).node).focus();
changed(f);
});
// The gates the fetch gate's three parts say, in the order the starter
// agents have them: the region, the place lines, and the titles left out.
function whereGates() {
const out = [];
const within = countries.values();
const towns = places.values();
const from = remoteFrom.values();
if (scope.value === "europe") out.push({ field: "country", values: rh.europeWith(within), mode: "require" });
if (scope.value === "countries" && scoped.values().length) {
out.push({ field: "country", values: scoped.values(), mode: "require" });
}
if (within.length) out.push({ field: "country", values: within, mode: "any" });
if (remoteMode.value === "scope") out.push({ field: "remote", values: ["remote"], mode: "any" });
if (remoteMode.value === "from" && from.length) out.push({ field: "remote_from", values: from, mode: "any" });
if (towns.length) out.push({ field: "place", values: towns, mode: "any" });
const left = titles.values();
if (left.length) out.push({ field: "title", values: left, mode: "exclude" });
return out;
}
// The gates Save sends: the fetch gate's, then the rows. The sentence
// under it says them all, as the card will, rows included.
const gates = () => whereGates().concat(rules.map((r) => r.read()));
const view = () => { const view = () => {
offer(); offer();
$("#a-gates", f).value = JSON.stringify(gates(), null, 2);
$("#a-where-sum", f).textContent = rulesLine(gates()) || "Passes every role."; $("#a-where-sum", f).textContent = rulesLine(gates()) || "Passes every role.";
}; };
view(); view();
@@ -981,6 +946,9 @@
} }
} }
// For tests/js/agents.test.mjs: what Save sends and how it is said.
window.agentGates = { formGates, keptNote, rulesLine };
document.addEventListener("DOMContentLoaded", async () => { document.addEventListener("DOMContentLoaded", async () => {
if (!$("#agents-list")) return; if (!$("#agents-list")) return;
$("#a-new").addEventListener("click", () => open(null)); $("#a-new").addEventListener("click", () => open(null));
+3 -17
View File
@@ -978,14 +978,6 @@ body[data-reader="open"] .company-strip { display: none; }
.reader-cv .cv-total { margin: 0 0 .45rem; font-variant-numeric: tabular-nums; } .reader-cv .cv-total { margin: 0 0 .45rem; font-variant-numeric: tabular-nums; }
.reader-cv .cv-total b { font-size: 1.25rem; } .reader-cv .cv-total b { font-size: 1.25rem; }
.reader-cv .cv-total span { display: inline-block; margin-left: .4rem; color: var(--dim); font-size: .88rem; } .reader-cv .cv-total span { display: inline-block; margin-left: .4rem; color: var(--dim); font-size: .88rem; }
/* The must-haves, one line each with its verdict's mark. */
.cv-musts { list-style: none; margin: 0 0 .55rem; padding: 0; font-size: .92rem; line-height: 1.4; }
.cv-musts li { display: flex; gap: .45rem; }
.cv-musts li + li { margin-top: .15rem; }
.cv-musts .mark { flex: none; width: 1rem; text-align: center; font-weight: 700; }
.cv-musts .must-met .mark { color: var(--good); }
.cv-musts .must-broken .mark { color: var(--error); }
.cv-musts .must-unclear .mark { color: var(--faint); }
.cv-lists { display: grid; grid-template-columns: 1fr 1fr; gap: .5rem 1.2rem; } .cv-lists { display: grid; grid-template-columns: 1fr 1fr; gap: .5rem 1.2rem; }
.cv-lists ul { margin: 0; padding-left: 1.1rem; font-size: .92rem; line-height: 1.45; } .cv-lists ul { margin: 0; padding-left: 1.1rem; font-size: .92rem; line-height: 1.45; }
.cv-lists li + li { margin-top: .2rem; } .cv-lists li + li { margin-top: .2rem; }
@@ -1308,8 +1300,6 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
background: var(--bg); border: 1px solid var(--line); background: var(--bg); border: 1px solid var(--line);
border-radius: .3rem; padding: .35rem .5rem; width: 100%; border-radius: .3rem; padding: .35rem .5rem; width: 100%;
} }
/* Prose in a textarea is read in the page face; JSON is in the mono one. */
.agent-form textarea.code { font-family: var(--mono); font-size: .85rem; }
.agent-form .hint { color: var(--dim); font-size: .8rem; overflow-wrap: anywhere; } .agent-form .hint { color: var(--dim); font-size: .8rem; overflow-wrap: anywhere; }
.weights { display: flex; gap: .75rem; flex-wrap: wrap; border: 1px solid var(--line); border-radius: .3rem; } .weights { display: flex; gap: .75rem; flex-wrap: wrap; border: 1px solid var(--line); border-radius: .3rem; }
/* A label that wraps to two lines pushes its box down; the boxes stay in line. */ /* A label that wraps to two lines pushes its box down; the boxes stay in line. */
@@ -1324,14 +1314,13 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
} }
/* The fieldsets that hold rows: a column, the Add button under the rows. */ /* The fieldsets that hold rows: a column, the Add button under the rows. */
.agent-form fieldset.prompts, .agent-form fieldset.where, .agent-form fieldset.titles, .agent-form fieldset.prompts, .agent-form fieldset.where, .agent-form fieldset.titles,
.agent-form fieldset.terms, .agent-form fieldset.terms {
.agent-form fieldset.rules {
display: flex; flex-direction: column; align-items: stretch; gap: .5rem; min-width: 0; display: flex; flex-direction: column; align-items: stretch; gap: .5rem; min-width: 0;
border: 1px solid var(--line); border-radius: .3rem; border: 1px solid var(--line); border-radius: .3rem;
} }
.agent-form fieldset > button { align-self: flex-start; } .agent-form fieldset > button { align-self: flex-start; }
/* A prompt or a rule as a row of a list, after `.crow`: the name and Remove /* A prompt or a language as a row of a list, after `.crow`: the name and Remove
on line one, the text spanning the row under them. A rule row wraps like on line one, the text spanning the row under them. A language row wraps like
`.crow .board-edit`, so a phone takes it in two lines rather than scrolling. */ `.crow .board-edit`, so a phone takes it in two lines rather than scrolling. */
.prompt-row { .prompt-row {
display: grid; grid-template-columns: 1fr auto; align-items: center; display: grid; grid-template-columns: 1fr auto; align-items: center;
@@ -1361,9 +1350,6 @@ table.applications td[data-label="Saved"] { white-space: nowrap; color: var(--di
.agent-form .chips input[type="text"] { flex: 1 1 6rem; width: auto; min-width: 0; } .agent-form .chips input[type="text"] { flex: 1 1 6rem; width: auto; min-width: 0; }
.agent-form .chips select { flex: 0 1 12rem; min-width: 0; max-width: 100%; } .agent-form .chips select { flex: 0 1 12rem; min-width: 0; max-width: 100%; }
.chips .tag span { color: var(--faint); } .chips .tag span { color: var(--faint); }
.agent-form details summary { cursor: pointer; color: var(--dim); font-size: .9rem; }
.agent-form details[open] summary { margin-bottom: .5rem; }
.agent-form .advanced-body { display: flex; flex-direction: column; gap: .6rem; }
/* ================================================================ */ /* ================================================================ */
/* companies and sources — the Sources page's sections, */ /* companies and sources — the Sources page's sections, */
+12 -10
View File
@@ -1,7 +1,8 @@
/* Where a remote role may be worked from, as the pages say it: the words of /* Where a remote role may be worked from, as the pages say it: the words of
* the badge, the picks of the agent's "Remote, open to someone in" chips, and * the badge, the picks of the agent's "Remote, open to someone in" chips, and
* how an agent's gates share out between the fetch gate's blocks and Other rules. No * how an agent's gates share out between the fetch gate's blocks and the rest
* DOM in it, so `tests/js/remote.test.mjs` can walk it without a browser. * the form keeps as stored. No DOM in it, so `tests/js/remote.test.mjs` can
* walk it without a browser.
* *
* The area is a place token in three spellings, one per layer: `remote:<place * The area is a place token in three spellings, one per layer: `remote:<place
* token>` on the Roles rail and in the address, `<place token>` in a * token>` on the Roles rail and in the address, `<place token>` in a
@@ -61,18 +62,19 @@
} }
/** Share out an agent's gates between the fetch gate's blocks (region, /** Share out an agent's gates between the fetch gate's blocks (region,
place, titles to leave out) and Other rules. Each line takes the first place, titles to leave out) and the rest, which the form cannot edit
gate it can say whole; the rest are rows. A gate has a field, values and and keeps as stored. Each line takes the first gate it can say whole.
a mode (require, any, exclude) and no weight; a gate with a mode the A gate has a field, values and a mode (require, any, exclude) and no
form does not know, such as an old prefer, stays a row. weight; a gate with a mode the form does not know, such as an old
prefer, is in the rest.
The scope line is the first plain require gate on country: `europe` The scope line is the first plain require gate on country: `europe`
when it is exactly Europe and the countries line (`europeWith`), when it is exactly Europe and the countries line (`europeWith`),
`countries` with its codes as chips otherwise, `everywhere` when there `countries` with its codes as chips otherwise, `everywhere` when there
is none. Any other require gate on country is a row. The remote line is is none. Any other require gate on country is in the rest. The remote
`scope` for the one-of on `remote: [remote]` (remote, within the line is `scope` for the one-of on `remote: [remote]` (remote, within
scope), `from` for a one-of on `remote_from`, else `none`; when both the scope), `from` for a one-of on `remote_from`, else `none`; when
are there `remote_from` wins and the plain one stays a row. */ both are there `remote_from` wins and the plain one is in the rest. */
function distribute(gates) { function distribute(gates) {
const where = { scope: null, countries: null, places: null, remote: { mode: "none", values: [] }, titles: null }; const where = { scope: null, countries: null, places: null, remote: { mode: "none", values: [] }, titles: null };
const rest = []; const rest = [];
+4 -18
View File
@@ -1744,34 +1744,20 @@
try { return languageNames?.of(code) || code; } catch (err) { return code; } try { return languageNames?.of(code) || code; } catch (err) { return code; }
} }
const MUST_MARKS = { met: ["✓", "met"], broken: ["✗", "broken"], unclear: ["?", "unclear"] };
/* The agent's `judge` reading against its CV: the total and its three /* The agent's `judge` reading against its CV: the total and its three
numbers, each of 10, career as the server counted it (capped at 4 when a must-have numbers, each of 10, then the strengths and gaps it named. A box as the
is broken, with what the model said), the must-haves with a verdict summary is, since it is the same kind of thing: a model's reading, not
each, then the strengths and gaps it named. A box as the summary is, the posting. */
since it is the same kind of thing: a model's reading, not the posting. */
function cvMatch(r) { function cvMatch(r) {
const j = r.judgement; const j = r.judgement;
if (!j) return ""; if (!j) return "";
const list = (items, none) => (items?.length const list = (items, none) => (items?.length
? `<ul>${items.map((x) => `<li>${esc(x)}</li>`).join("")}</ul>` ? `<ul>${items.map((x) => `<li>${esc(x)}</li>`).join("")}</ul>`
: `<p class="muted small">${none}</p>`); : `<p class="muted small">${none}</p>`);
const counted = j.career_capped ?? j.career;
const career = counted < j.career
? `career ${counted}/10 (capped: breaks a must-have; model said ${j.career}/10)`
: `career ${j.career}/10`;
const musts = Array.isArray(j.musts) && j.musts.length
? `<ul class="cv-musts">${j.musts.map((m) => {
const [mark, word] = MUST_MARKS[m.verdict] || MUST_MARKS.unclear;
return `<li class="must-${word}"><span class="mark" role="img"
aria-label="${word}" title="${word}">${mark}</span><span>${esc(m.text)}</span></li>`;
}).join("")}</ul>` : "";
return `<section class="reader-summary reader-cv"> return `<section class="reader-summary reader-cv">
<h2 class="reader-label">CV match</h2> <h2 class="reader-label">CV match</h2>
<p class="cv-total"><b>${Math.round(j.cv_score)}</b> of 100 <p class="cv-total"><b>${Math.round(j.cv_score)}</b> of 100
<span>skills ${j.skills}/10 · experience ${j.experience}/10 · ${career}</span></p> <span>skills ${j.skills}/10 · experience ${j.experience}/10 · career ${j.career}/10</span></p>
${musts}
<div class="cv-lists"> <div class="cv-lists">
<div><h3 class="reader-label">Strengths</h3>${list(j.strengths, "None named.")}</div> <div><h3 class="reader-label">Strengths</h3>${list(j.strengths, "None named.")}</div>
<div><h3 class="reader-label">Gaps</h3>${list(j.gaps, "None named.")}</div> <div><h3 class="reader-label">Gaps</h3>${list(j.gaps, "None named.")}</div>
+43 -16
View File
@@ -55,6 +55,10 @@ CREATE TABLE IF NOT EXISTS roles (
country TEXT NOT NULL DEFAULT '', country TEXT NOT NULL DEFAULT '',
remote TEXT NOT NULL DEFAULT '' remote TEXT NOT NULL DEFAULT ''
CHECK (remote IN ('', 'remote', 'hybrid', 'onsite')), CHECK (remote IN ('', 'remote', 'hybrid', 'onsite')),
-- 1 when the title or text says the role is worked remotely while no
-- field does (prep, app.prep.words.remote_in_text): the location gate
-- lets such a role through until extract has read it.
remote_text INTEGER NOT NULL DEFAULT 0,
salary_min INTEGER, salary_min INTEGER,
salary_max INTEGER, salary_max INTEGER,
currency TEXT NOT NULL DEFAULT '', currency TEXT NOT NULL DEFAULT '',
@@ -328,9 +332,8 @@ END;
-- as JSON: a key it lacks means the `Limits` default. fetch_terms is a JSON -- as JSON: a key it lacks means the `Limits` default. fetch_terms is a JSON
-- list of words that narrow what big boards send (app/jobs/reach.py); empty -- list of words that narrow what big boards send (app/jobs/reach.py); empty
-- means no narrowing. cv_text, cv_languages (a JSON list of {language, -- means no narrowing. cv_text, cv_languages (a JSON list of {language,
-- level}), like_text (what they like: the work, the workplace, situations), -- level}), like_text (what they like: the work, the workplace, situations)
-- avoid_text (what they do not like) and must_text (must-haves and -- and avoid_text (what they do not like) are what judge reads a role
-- deal-breakers, set through the API only) are what judge reads a role
-- against; an agent with no CV is not judged. -- against; an agent with no CV is not judged.
CREATE TABLE IF NOT EXISTS agents ( CREATE TABLE IF NOT EXISTS agents (
id INTEGER PRIMARY KEY, id INTEGER PRIMARY KEY,
@@ -344,7 +347,6 @@ CREATE TABLE IF NOT EXISTS agents (
cv_text TEXT NOT NULL DEFAULT '', cv_text TEXT NOT NULL DEFAULT '',
cv_languages TEXT NOT NULL DEFAULT '[]', cv_languages TEXT NOT NULL DEFAULT '[]',
avoid_text TEXT NOT NULL DEFAULT '', avoid_text TEXT NOT NULL DEFAULT '',
must_text TEXT NOT NULL DEFAULT '',
like_text TEXT NOT NULL DEFAULT '' like_text TEXT NOT NULL DEFAULT ''
); );
@@ -435,11 +437,8 @@ CREATE TABLE IF NOT EXISTS agent_cv_vectors (
-- 100 until migration 33), cv_score their weighted total times 10 at the -- 100 until migration 33), cv_score their weighted total times 10 at the
-- default weights (not read: app.models.cv_score is worked out when a role -- default weights (not read: app.models.cv_score is worked out when a role
-- is served), strengths and gaps JSON lists. input_key -- is served), strengths and gaps JSON lists. input_key
-- is a hash of the CV, prompts, avoid text and the role's text it read: a -- is a hash of the CV, prompts, languages, like and avoid texts and the
-- different one is read again. musts is a JSON list of {id, text, verdict}, -- role's text it read: a different one is read again. tries counts
-- the model's verdict on each of the agent's must-haves (one per line of its
-- must text), broken how many are "broken": career counts as at most 4 then
-- (app.models.BROKEN_CAP), applied when the role is served. tries counts
-- replies that left the role without one since; at 3 it is given up until the agent is saved again. A -- replies that left the role without one since; at 3 it is given up until the agent is saved again. A
-- row with judged_at NULL holds only tries. -- row with judged_at NULL holds only tries.
CREATE TABLE IF NOT EXISTS role_judgements ( CREATE TABLE IF NOT EXISTS role_judgements (
@@ -451,8 +450,6 @@ CREATE TABLE IF NOT EXISTS role_judgements (
cv_score REAL, cv_score REAL,
strengths TEXT NOT NULL DEFAULT '[]', strengths TEXT NOT NULL DEFAULT '[]',
gaps TEXT NOT NULL DEFAULT '[]', gaps TEXT NOT NULL DEFAULT '[]',
musts TEXT NOT NULL DEFAULT '[]',
broken INTEGER NOT NULL DEFAULT 0,
input_key TEXT NOT NULL DEFAULT '', input_key TEXT NOT NULL DEFAULT '',
tries INTEGER NOT NULL DEFAULT 0, tries INTEGER NOT NULL DEFAULT 0,
judged_at TEXT, judged_at TEXT,
@@ -763,6 +760,12 @@ UPDATE roles SET seniority = COALESCE(NULLIF(seniority_model, ''),
# A Python step: JSON limits, and a count for the log. # A Python step: JSON limits, and a count for the log.
(33, "the judge on 0 to 10: every judgement cleared to be made again, and summarise_top " (33, "the judge on 0 to 10: every judgement cleared to be made again, and summarise_top "
"50 becomes 30", lambda conn: _judge_out_of_ten(conn)), "50 becomes 30", lambda conn: _judge_out_of_ten(conn)),
# A Python step: it drops the columns only where they are there. The
# judge reads every role again (judge.FORMAT).
(34, "agents.must_text, role_judgements.musts and broken dropped: the profile is the "
"CV, languages, likes and dislikes", lambda conn: _drop_musts(conn)),
(35, "roles.remote_text: the text says remote while no field does, read by prep for the "
"location gate", lambda conn: _add_remote_text(conn)),
) )
#: What a file created from `SCHEMA` today is already at. #: What a file created from `SCHEMA` today is already at.
@@ -1012,6 +1015,15 @@ def _add_posting_url(conn: sqlite3.Connection) -> None:
conn.execute("ALTER TABLE roles ADD COLUMN posting_url TEXT NOT NULL DEFAULT ''") conn.execute("ALTER TABLE roles ADD COLUMN posting_url TEXT NOT NULL DEFAULT ''")
def _add_remote_text(conn: sqlite3.Connection) -> None:
"""Migration 35's step: `roles.remote_text`, 0 until the next prep,
which maps every open role again since its rules changed. Runs twice
without harm."""
columns = {r[1] for r in conn.execute("PRAGMA table_info(roles)")}
if "remote_text" not in columns:
conn.execute("ALTER TABLE roles ADD COLUMN remote_text INTEGER NOT NULL DEFAULT 0")
def _add_remote_area(conn: sqlite3.Connection) -> None: def _add_remote_area(conn: sqlite3.Connection) -> None:
"""Migration 27's step: `roles.remote_model` empty and """Migration 27's step: `roles.remote_model` empty and
`roles.remote_where_model` NULL for every role, so `extract` reads the `roles.remote_where_model` NULL for every role, so `extract` reads the
@@ -1203,12 +1215,14 @@ def _judge_out_of_ten(conn: sqlite3.Connection) -> None:
old number is shown on the new scale. The rows stay, as rows of no old number is shown on the new scale. The rows stay, as rows of no
judgement, and the judge reads every pair again. An agent whose judgement, and the judge reads every pair again. An agent whose
`summarise_top` is 50, migration 32's default, takes 30, the new one. `summarise_top` is 50, migration 32's default, takes 30, the new one.
Runs twice without harm.""" Runs twice without harm, also after migration 34 dropped the verdicts."""
columns = {r[1] for r in conn.execute("PRAGMA table_info(role_judgements)")}
verdicts = ", musts = '[]', broken = 0" if {"musts", "broken"} <= columns else ""
cleared = conn.execute( cleared = conn.execute(
"""UPDATE role_judgements f"""UPDATE role_judgements
SET skills = NULL, experience = NULL, career = NULL, cv_score = NULL, SET skills = NULL, experience = NULL, career = NULL, cv_score = NULL,
strengths = '[]', gaps = '[]', musts = '[]', broken = 0, strengths = '[]', gaps = '[]'{verdicts},
input_key = '', tries = 0, judged_at = NULL""").rowcount input_key = '', tries = 0, judged_at = NULL""").rowcount
lowered = conn.execute( lowered = conn.execute(
"""UPDATE agents SET limits = json_set(limits, '$.summarise_top', 30) """UPDATE agents SET limits = json_set(limits, '$.summarise_top', 30)
WHERE json_valid(limits) AND json_type(limits) = 'object' WHERE json_valid(limits) AND json_type(limits) = 'object'
@@ -1217,6 +1231,19 @@ def _judge_out_of_ten(conn: sqlite3.Connection) -> None:
"summarise_top 50 set to 30", cleared, lowered) "summarise_top 50 set to 30", cleared, lowered)
def _drop_musts(conn: sqlite3.Connection) -> None:
"""Migration 34's step: the must-haves are gone, the agent's text
(`agents.must_text`) and the judge's verdicts on it
(`role_judgements.musts`, `broken`), and with them the cap on career.
Runs twice without harm."""
for table, columns in (("agents", ("must_text",)),
("role_judgements", ("musts", "broken"))):
present = {r[1] for r in conn.execute(f"PRAGMA table_info({table})")}
for column in columns:
if column in present:
conn.execute(f"ALTER TABLE {table} DROP COLUMN {column}")
def _add_all_roles(conn: sqlite3.Connection) -> None: def _add_all_roles(conn: sqlite3.Connection) -> None:
"""Migration 21's step. The column; the tag for every company that is a """Migration 21's step. The column; the tag for every company that is a
favourite of any agent, since favourite no longer widens what a pull favourite of any agent, since favourite no longer widens what a pull
+32 -39
View File
@@ -11,7 +11,7 @@ from __future__ import annotations
import json import json
import sqlite3 import sqlite3
from collections import Counter from collections import Counter
from collections.abc import Callable, Iterable, Iterator, Mapping, MutableMapping, Sequence from collections.abc import Callable, Iterable, Iterator, MutableMapping, Sequence
from contextlib import closing, contextmanager from contextlib import closing, contextmanager
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from pathlib import Path from pathlib import Path
@@ -26,7 +26,7 @@ from app.models import (
AFFINITY, HAND_REASONS, LEVELS, STATUSES, Agent, Application, AFFINITY, HAND_REASONS, LEVELS, STATUSES, Agent, Application,
ApplicationEvent, Candidate, Company, CvLanguage, Gate, Judgement, Limits, Prompt, Role, ApplicationEvent, Candidate, Company, CvLanguage, Gate, Judgement, Limits, Prompt, Role,
RoleCopy, RolePlace, RoleRow, SavedFilter, ScoredRole, cv_languages as normal_languages, cv_score, RoleCopy, RolePlace, RoleRow, SavedFilter, ScoredRole, cv_languages as normal_languages, cv_score,
BROKEN_CAP, capped_career, language_flag, role_languages, language_flag, role_languages,
Subscription, Weights, check_prompts, company_fold, company_key, country_of, excerpt, place_key, Subscription, Weights, check_prompts, company_fold, company_key, country_of, excerpt, place_key,
fetch_terms as normal_terms, host_word, iso_day, posted_at, posted_host, remote_of, title_key, fetch_terms as normal_terms, host_word, iso_day, posted_at, posted_host, remote_of, title_key,
) )
@@ -140,7 +140,7 @@ _VOTED_SCORE = ("MIN(1.0, s.score * CASE v.vote WHEN -1 THEN 0.5 "
_WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?" _WITH_VOTES = "LEFT JOIN role_votes v ON v.uid = roles.uid AND v.agent_id = ?"
#: The agent's `judge` readings beside every role, the same way: `j.skills`, #: The agent's `judge` readings beside every role, the same way: `j.skills`,
#: `j.experience`, `j.career` and `j.broken` are NULL for a role not judged and with no #: `j.experience` and `j.career` are NULL for a role not judged and with no
#: agent. #: agent.
_WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agent_id = ? " _WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agent_id = ? "
"AND j.judged_at IS NOT NULL") "AND j.judged_at IS NOT NULL")
@@ -150,19 +150,15 @@ _WITH_JUDGEMENTS = ("LEFT JOIN role_judgements j ON j.uid = roles.uid AND j.agen
def _cv_sql(weights: Weights | None) -> str: def _cv_sql(weights: Weights | None) -> str:
"""`cv_score` as SQL for one agent's weights, 0 to 100 from the judge's """`cv_score` as SQL for one agent's weights, 0 to 100 from the judge's
three numbers 0 to 10, unrounded (`app.models.cv_score` rounds it): NULL three numbers 0 to 10, unrounded (`app.models.cv_score` rounds it): NULL
for a role not judged, and for all three weights 0. Career for a role not judged, and for all three weights 0. The weights are
counts as at most `BROKEN_CAP` when the judgement holds a broken validated, finite floats, so they are written in rather than bound, the
must-have (`j.broken`), as `app.models.capped_career` has it. The select's placeholders being then no concern."""
numbers are the model's validated, finite floats, so they are written in
rather than bound, the select's placeholders being then no concern."""
w = weights or Weights() w = weights or Weights()
total = w.cv_skills + w.cv_experience + w.cv_career total = w.cv_skills + w.cv_experience + w.cv_career
if not total: if not total:
return "NULL" return "NULL"
career = (f"(CASE WHEN j.broken > 0 AND j.career > {BROKEN_CAP} THEN {BROKEN_CAP} "
"ELSE j.career END)")
return (f"(10.0 * ({w.cv_skills!r} * j.skills + {w.cv_experience!r} * j.experience " return (f"(10.0 * ({w.cv_skills!r} * j.skills + {w.cv_experience!r} * j.experience "
f"+ {w.cv_career!r} * {career}) / {total!r})") f"+ {w.cv_career!r} * j.career) / {total!r})")
def _by_cv(expression: str) -> str: def _by_cv(expression: str) -> str:
@@ -359,6 +355,12 @@ def _gate_sql(gate: Gate, gaz: places.Gazetteer) -> tuple[str, str, list[Any]]:
return ("r.languages IN ('', '[]')", return ("r.languages IN ('', '[]')",
f"NOT EXISTS (SELECT 1 FROM json_each(r.languages) " f"NOT EXISTS (SELECT 1 FROM json_each(r.languages) "
f"WHERE LOWER(value) NOT IN ({marks}))", values) f"WHERE LOWER(value) NOT IN ({marks}))", values)
if gate.field == "remote" and "remote" in values:
# A role whose text says remote while no field does is possibly
# remote (`remote_text`, prep): it meets a gate on remote until
# extract has read it, and is not empty for it.
return (f"({column} = '' AND r.remote_text = 0)",
f"(LOWER({column}) IN ({marks}) OR ({column} = '' AND r.remote_text = 1))", values)
return f"{column} = ''", f"LOWER({column}) IN ({marks})", values return f"{column} = ''", f"LOWER({column}) IN ({marks})", values
@@ -1629,8 +1631,9 @@ class Store:
f"""SELECT uid, prep_due, source, {_HOST} AS host, title, company, company_key, place, f"""SELECT uid, prep_due, source, {_HOST} AS host, title, company, company_key, place,
seniority_host, employment_host, remote_host, country_host, seniority_host, employment_host, remote_host, country_host,
seniority_model, seniority_stated, seniority_title, seniority, seniority_model, seniority_stated, seniority_title, seniority,
employment, remote, country, remote_model, remote_where_model, employment, remote, remote_text, country, remote_model,
deadline_host, deadline_model, deadline, remote_where_model, deadline_host, deadline_model, deadline,
SUBSTR(COALESCE(NULLIF(description, ''), teaser), 1, 6000) AS text,
json_extract(raw, '$.places') AS raw_places, json_extract(raw, '$.places') AS raw_places,
(SELECT json_group_array(place_id) FROM ( (SELECT json_group_array(place_id) FROM (
SELECT place_id FROM role_remote rr WHERE rr.uid = roles.uid SELECT place_id FROM role_remote rr WHERE rr.uid = roles.uid
@@ -1656,8 +1659,8 @@ class Store:
def write_prep(self, mapped: Sequence[dict[str, Any]]) -> int: def write_prep(self, mapped: Sequence[dict[str, Any]]) -> int:
"""Write what prep mapped, one dict per role with `uid`, `due` (the """Write what prep mapped, one dict per role with `uid`, `due` (the
`prep_due` it read), `company_key`, `seniority_stated`, `prep_due` it read), `company_key`, `seniority_stated`,
`seniority_title`, `employment`, `remote`, `remote_area`, `country` `seniority_title`, `employment`, `remote`, `remote_text`, `remote_area`,
and `deadline` (`YYYY-MM-DD`, None for none, written as it is). `seniority` is the model's level, else the stated, else `country` and `deadline` (`YYYY-MM-DD`, None for none, written as it is). `seniority` is the model's level, else the stated, else
the title's. `remote` is written as prep composed it (the model's, the title's. `remote` is written as prep composed it (the model's,
else the host's word, else the place line) and `remote_area`, place else the host's word, else the place line) and `remote_area`, place
ids, replaces the role's `role_remote`. An empty `country` leaves the ids, replaces the role's `role_remote`. An empty `country` leaves the
@@ -1673,6 +1676,7 @@ class Store:
NULLIF(:seniority_stated, ''), :seniority_title), NULLIF(:seniority_stated, ''), :seniority_title),
employment = :employment, employment = :employment,
remote = :remote, remote = :remote,
remote_text = :remote_text,
country = COALESCE(NULLIF(:country, ''), country), country = COALESCE(NULLIF(:country, ''), country),
deadline = :deadline, deadline = :deadline,
prep_due = CASE WHEN prep_due = :due THEN 0 ELSE prep_due END prep_due = CASE WHEN prep_due = :due THEN 0 ELSE prep_due END
@@ -2021,33 +2025,28 @@ class Store:
def set_judgement(self, agent_id: int, uid: str, input_key: str, skills: int, def set_judgement(self, agent_id: int, uid: str, input_key: str, skills: int,
experience: int, career: int, strengths: Sequence[str], experience: int, career: int, strengths: Sequence[str],
gaps: Sequence[str], musts: Sequence[Mapping[str, str]] = ()) -> bool: gaps: Sequence[str]) -> bool:
"""Store `judge`'s reading of one role for one agent, replacing the """Store `judge`'s reading of one role for one agent, replacing the
one before: the three numbers as the model gave them, the verdict on one before: the three numbers as the model gave them, `cv_score`
each of the agent's must-haves (`musts`, `{id, text, verdict}`) and how (`app.models.cv_score`, at the default weights: the table's column is
many are broken, `cv_score` (`app.models.cv_score`, at the default not read, see `judgement`) and its tries back at 0. False for an
weights and with the cap: the table's column is not read, see unknown role or agent."""
`judgement`) and its tries back at 0. False for an unknown role or
agent."""
broken = sum(1 for m in musts if m.get("verdict") == "broken")
try: try:
with self.transaction() as conn: with self.transaction() as conn:
conn.execute( conn.execute(
"""INSERT INTO role_judgements (uid, agent_id, skills, experience, career, """INSERT INTO role_judgements (uid, agent_id, skills, experience, career,
cv_score, strengths, gaps, musts, broken, cv_score, strengths, gaps,
input_key, tries, judged_at) input_key, tries, judged_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?,0,?) VALUES (?,?,?,?,?,?,?,?,?,0,?)
ON CONFLICT(uid, agent_id) DO UPDATE SET ON CONFLICT(uid, agent_id) DO UPDATE SET
skills=excluded.skills, experience=excluded.experience, skills=excluded.skills, experience=excluded.experience,
career=excluded.career, cv_score=excluded.cv_score, career=excluded.career, cv_score=excluded.cv_score,
strengths=excluded.strengths, gaps=excluded.gaps, strengths=excluded.strengths, gaps=excluded.gaps,
musts=excluded.musts, broken=excluded.broken,
input_key=excluded.input_key, tries=0, judged_at=excluded.judged_at""", input_key=excluded.input_key, tries=0, judged_at=excluded.judged_at""",
(uid, agent_id, skills, experience, career, (uid, agent_id, skills, experience, career,
cv_score(skills, experience, career, broken=broken), cv_score(skills, experience, career),
json.dumps(list(strengths), ensure_ascii=False), json.dumps(list(strengths), ensure_ascii=False),
json.dumps(list(gaps), ensure_ascii=False), json.dumps(list(gaps), ensure_ascii=False),
json.dumps([dict(m) for m in musts], ensure_ascii=False), broken,
input_key, self.now())) input_key, self.now()))
except sqlite3.IntegrityError: except sqlite3.IntegrityError:
return False return False
@@ -2075,8 +2074,7 @@ class Store:
def judgement(self, agent_id: int | None, uid: str) -> Judgement | None: def judgement(self, agent_id: int | None, uid: str) -> Judgement | None:
"""The agent's judgement of one role, None when it has none, its """The agent's judgement of one role, None when it has none, its
`cv_score` the three numbers under the agent's weights as they are `cv_score` the three numbers under the agent's weights as they are
now, not as stored, and `career_capped` the model's career as the now, not as stored."""
broken must-haves leave it (`BROKEN_CAP`)."""
if agent_id is None: if agent_id is None:
return None return None
row = self._row("SELECT * FROM role_judgements WHERE agent_id = ? AND uid = ? " row = self._row("SELECT * FROM role_judgements WHERE agent_id = ? AND uid = ? "
@@ -2085,9 +2083,8 @@ class Store:
return None return None
found = Judgement.from_row(row) found = Judgement.from_row(row)
agent = self.agent(agent_id) agent = self.agent(agent_id)
found.career_capped = capped_career(found.career, found.broken)
found.cv_score = cv_score(found.skills, found.experience, found.career, found.cv_score = cv_score(found.skills, found.experience, found.career,
agent.weights if agent else None, found.broken) agent.weights if agent else None)
return found return found
def set_article_text(self, uid: str, text: str) -> Role | None: def set_article_text(self, uid: str, text: str) -> Role | None:
@@ -2209,14 +2206,13 @@ class Store:
cv_text: str | None = None, cv_text: str | None = None,
cv_languages: Sequence[CvLanguage] | None = None, cv_languages: Sequence[CvLanguage] | None = None,
avoid_text: str | None = None, avoid_text: str | None = None,
must_text: str | None = None,
like_text: str | None = None) -> Agent: like_text: str | None = None) -> Agent:
"""Create an agent, or with `agent_id` replace that one's fields and """Create an agent, or with `agent_id` replace that one's fields and
prompts. Without `prompts`, a non-empty `intent` is its one prompt, prompts. Without `prompts`, a non-empty `intent` is its one prompt,
named after the agent. Without `fetch_terms` an update keeps the named after the agent. Without `fetch_terms` an update keeps the
stored ones and a new agent has none, so a caller that does not know stored ones and a new agent has none, so a caller that does not know
them (the CLI's prompt edit) loses none; `cv_text`, `cv_languages`, them (the CLI's prompt edit) loses none; `cv_text`, `cv_languages`,
`like_text`, `avoid_text` and `must_text` are kept the same way. A prompt whose text is `like_text` and `avoid_text` are kept the same way. A prompt whose text is
unchanged keeps its cached vector, whatever its name or place, so an unchanged keeps its cached vector, whatever its name or place, so an
edit re-embeds only the prompts it changed. A save gives every role edit re-embeds only the prompts it changed. A save gives every role
`judge` gave up on for the agent its tries back, since what it is `judge` gave up on for the agent its tries back, since what it is
@@ -2237,7 +2233,6 @@ class Store:
(limits or Limits()).model_dump_json(), int(enabled), terms, (limits or Limits()).model_dump_json(), int(enabled), terms,
None if cv_text is None else cv_text.strip(), spoken, None if cv_text is None else cv_text.strip(), spoken,
None if avoid_text is None else avoid_text.strip(), None if avoid_text is None else avoid_text.strip(),
None if must_text is None else must_text.strip(),
None if like_text is None else like_text.strip()) None if like_text is None else like_text.strip())
try: try:
with self.transaction() as conn: with self.transaction() as conn:
@@ -2245,10 +2240,9 @@ class Store:
agent_id = conn.execute( agent_id = conn.execute(
"""INSERT INTO agents (name, gates, weights, limits, enabled, """INSERT INTO agents (name, gates, weights, limits, enabled,
fetch_terms, cv_text, cv_languages, avoid_text, fetch_terms, cv_text, cv_languages, avoid_text,
must_text, like_text, created_at) like_text, created_at)
VALUES (?,?,?,?,?,COALESCE(?, '[]'),COALESCE(?, ''), VALUES (?,?,?,?,?,COALESCE(?, '[]'),COALESCE(?, ''),
COALESCE(?, '[]'),COALESCE(?, ''),COALESCE(?, ''), COALESCE(?, '[]'),COALESCE(?, ''),COALESCE(?, ''),?)""",
COALESCE(?, ''),?)""",
(*fields, self.now())).lastrowid (*fields, self.now())).lastrowid
elif not conn.execute( elif not conn.execute(
"""UPDATE agents SET name=?, gates=?, weights=?, limits=?, enabled=?, """UPDATE agents SET name=?, gates=?, weights=?, limits=?, enabled=?,
@@ -2256,7 +2250,6 @@ class Store:
cv_text=COALESCE(?, cv_text), cv_text=COALESCE(?, cv_text),
cv_languages=COALESCE(?, cv_languages), cv_languages=COALESCE(?, cv_languages),
avoid_text=COALESCE(?, avoid_text), avoid_text=COALESCE(?, avoid_text),
must_text=COALESCE(?, must_text),
like_text=COALESCE(?, like_text) like_text=COALESCE(?, like_text)
WHERE id=?""", WHERE id=?""",
(*fields, agent_id)).rowcount: (*fields, agent_id)).rowcount:
+1 -1
View File
@@ -3,7 +3,7 @@
{% block content %} {% block content %}
<h1>Account</h1> <h1>Account</h1>
<p class="lede">The one password, the devices signed in with it, and the passkeys that sign in without it.</p> <p class="lede">The one password, the devices signed in with it, the passkeys that sign in without it, and the API tokens a script sends instead.</p>
<section class="card"> <section class="card">
{# As in Sources' Runs: the element and its script belong to {# As in Sources' Runs: the element and its script belong to
+2 -15
View File
@@ -74,6 +74,7 @@
</div> </div>
<span class="hint">Fetch terms, separated by a comma or Enter: <code>machine learning, data science</code> is two terms, and <code>machine learning</code> is searched as one. A host that takes a query sends only roles with one of the terms anywhere in the ad, but only for boards of 300 or more roles; smaller boards and other hosts are read whole. The terms are ignored after the fetch, since prompt closeness is the finer filter. Empty means no narrowing.</span> <span class="hint">Fetch terms, separated by a comma or Enter: <code>machine learning, data science</code> is two terms, and <code>machine learning</code> is searched as one. A host that takes a query sends only roles with one of the terms anywhere in the ad, but only for boards of 300 or more roles; smaller boards and other hosts are read whole. The terms are ignored after the fetch, since prompt closeness is the finer filter. Empty means no narrowing.</span>
<p id="a-where-sum" class="where-sum" role="status"></p> <p id="a-where-sum" class="where-sum" role="status"></p>
<p id="a-kept" class="hint" hidden></p>
</fieldset> </fieldset>
<fieldset class="terms cv"> <fieldset class="terms cv">
<legend>Your CV</legend> <legend>Your CV</legend>
@@ -94,7 +95,7 @@
<textarea id="a-like-text" rows="4" placeholder="Building a team from scratch, close work with the product side, small companies, a say in the roadmap"></textarea></label> <textarea id="a-like-text" rows="4" placeholder="Building a team from scratch, close work with the product side, small companies, a say in the roadmap"></textarea></label>
<label>What I don't like <label>What I don't like
<textarea id="a-avoid-text" rows="4" placeholder="Constant on-call, sales targets, large consultancies, open-plan offices"></textarea></label> <textarea id="a-avoid-text" rows="4" placeholder="Constant on-call, sales targets, large consultancies, open-plan offices"></textarea></label>
<span class="hint">The work, the workplace, situations; read by the judge for the career rating. Rules that must hold go under the fetch gate or Other rules instead.</span> <span class="hint">The work, the workplace, situations; read by the judge for the career rating. Rules that must hold go under the fetch gate instead.</span>
</fieldset> </fieldset>
<fieldset class="weights limits"> <fieldset class="weights limits">
<legend>Analytics filter</legend> <legend>Analytics filter</legend>
@@ -124,20 +125,6 @@
<label>Summarise the top <input type="number" id="a-summarise-top" min="0" step="5" value="30"> of each feed order</label> <label>Summarise the top <input type="number" id="a-summarise-top" min="0" step="5" value="30"> of each feed order</label>
</fieldset> </fieldset>
<span class="hint">Summarise writes a short summary and tags for the top N of What I'm looking for and the top N of What fits me, at most twice N roles, and for every role you liked or saved. It is no part of any score.</span> <span class="hint">Summarise writes a short summary and tags for the top N of What I'm looking for and the top N of What fits me, at most twice N roles, and for every role you liked or saved. It is no part of any score.</span>
<details class="advanced">
<summary>Advanced</summary>
<div class="advanced-body">
<fieldset class="rules">
<legend>Other rules</legend>
<div id="a-rules" class="rule-rows"></div>
<button type="button" id="a-add-rule" class="secondary">Add rule</button>
<span class="hint">Must: a role must meet it. One of: a role must meet one of these or a Place line. Not: a role that meets it is left out. Values are separated by commas; salary is a number.</span>
</fieldset>
<label>Rules as sent
<textarea id="a-gates" class="code" rows="6" spellcheck="false" readonly>[]</textarea>
<span class="hint">Read only: what Save sends as the agent's gates, from the fetch gate and Other rules.</span></label>
</div>
</details>
<div class="agent-actions"> <div class="agent-actions">
<button type="submit">Save and score</button> <button type="submit">Save and score</button>
<button type="button" id="a-cancel" class="secondary">Cancel</button> <button type="button" id="a-cancel" class="secondary">Cancel</button>
+2 -4
View File
@@ -1251,7 +1251,7 @@ class AgentBody(BaseModel):
it is stored as one prompt named after the agent. `fetch_terms` narrow it is stored as one prompt named after the agent. `fetch_terms` narrow
what big boards send (`app/jobs/reach.py`); `[]` narrows nothing, and what big boards send (`app/jobs/reach.py`); `[]` narrows nothing, and
leaving the key out keeps an existing agent's. `cv_text`, leaving the key out keeps an existing agent's. `cv_text`,
`cv_languages`, `like_text`, `avoid_text` and `must_text` are what `judge` reads a `cv_languages`, `like_text` and `avoid_text` are what `judge` reads a
role against (`app/jobs/judge.py`), kept the same way when left out or role against (`app/jobs/judge.py`), kept the same way when left out or
null.""" null."""
@@ -1267,7 +1267,6 @@ class AgentBody(BaseModel):
cv_languages: CvLanguages | None = None cv_languages: CvLanguages | None = None
like_text: str | None = None like_text: str | None = None
avoid_text: str | None = None avoid_text: str | None = None
must_text: str | None = None
enabled: bool = True enabled: bool = True
@@ -1307,8 +1306,7 @@ def save_agent(body: AgentBody, store: StoreDep, queue: QueueDep) -> Agent:
body.enabled, agent_id=body.id, limits=body.limits, body.enabled, agent_id=body.id, limits=body.limits,
prompts=body.prompts, fetch_terms=body.fetch_terms, prompts=body.prompts, fetch_terms=body.fetch_terms,
cv_text=body.cv_text, cv_languages=body.cv_languages, cv_text=body.cv_text, cv_languages=body.cv_languages,
like_text=body.like_text, avoid_text=body.avoid_text, like_text=body.like_text, avoid_text=body.avoid_text)
must_text=body.must_text)
except ValueError as exc: except ValueError as exc:
raise HTTPException(409, str(exc)) from exc raise HTTPException(409, str(exc)) from exc
except KeyError: except KeyError:
+19 -9
View File
@@ -1,6 +1,6 @@
# auth # auth
One password on a public hostname. scrypt, a lockout counted in the database, sessions you can revoke, passkeys, and one middleware in front of everything — the WebSocket included. No user management: there is one user, they are the person who deployed it, and the only questions are whether the door is locked and what opens it. One password on a public hostname. scrypt, a lockout counted in the database, sessions you can revoke, passkeys, API tokens for scripts, and one middleware in front of everything — the WebSocket included. No user management: there is one user, they are the person who deployed it, and the only questions are whether the door is locked and what opens it: the password, a passkey, or a token.
Needs Python 3.12+, `fastapi` and `pydantic`. `webauthn` is optional and only for passkeys; every import of it is inside a function, and the passkey routes answer 501 with a sentence saying what to install when it is missing. Needs Python 3.12+, `fastapi` and `pydantic`. `webauthn` is optional and only for passkeys; every import of it is inside a function, and the passkey routes answer 501 with a sentence saying what to install when it is missing.
@@ -11,7 +11,7 @@ cp -r boilerplate/auth yourproject/auth
python -m pytest yourproject/auth/tests python -m pytest yourproject/auth/tests
``` ```
Then add `auth*` to that project's `pyproject.toml` packages, `"auth" = ["templates/*.html", "static/*"]` to its package data, and `auth/tests` to its testpaths. `docs/password-protection.md` in this repository is the reasoning behind every decision below, at length. Then add `auth*` to that project's `pyproject.toml` packages, `"auth" = ["templates/*.html", "static/*"]` to its package data, and `auth/tests` to its testpaths. The boilerplate's `documentation/Password Protection.md` is the reasoning behind every decision below, at length.
## Five lines ## Five lines
@@ -27,7 +27,7 @@ auth.connect(app, auth.AuthStore(config.database), config.auth,
Call it **first**, before the app's own routers. Starlette wraps the whole router in its middleware whenever the routes were added, so the socket and every page are covered either way — but written first it is also read first, and nobody reviewing the file has to work out whether the socket is behind the password. Call it **first**, before the app's own routers. Starlette wraps the whole router in its middleware whenever the routes were added, so the socket and every page are covered either way — but written first it is also read first, and nobody reviewing the file has to work out whether the socket is behind the password.
That gives you `/login` (the page and the POST), `/logout`, `/api/account`, `/api/account/password`, `/api/account/sessions/{id}` and `/api/account/sessions/close`, the four passkey routes, `/_auth/passkey.js` and `/_auth/account.js` — and a startup bootstrap that makes sure there is a password at all. That gives you `/login` (the page and the POST), `/logout`, `/api/account`, `/api/account/password`, `/api/account/sessions/{id}` and `/api/account/sessions/close`, the four passkey routes, the three token routes (`POST /api/account/tokens`, `POST /api/account/tokens/{id}/label`, `DELETE /api/account/tokens/{id}`), `/_auth/passkey.js` and `/_auth/account.js` — and a startup bootstrap that makes sure there is a password at all.
| `connect(app, store, config, ...)` | | | `connect(app, store, config, ...)` | |
|---|---| |---|---|
@@ -35,6 +35,7 @@ That gives you `/login` (the page and the POST), `/logout`, `/api/account`, `/ap
| `config` | an `AuthConfig`. The app's own config model carries it as a field; see below. | | `config` | an `AuthConfig`. The app's own config model carries it as a field; see below. |
| `host=` | the address the server binds — `config.server.host`. What `enabled: auto` decides from, and deliberately not taken from a request, since a `Host` header is written by whoever is talking. | | `host=` | the address the server binds — `config.server.host`. What `enabled: auto` decides from, and deliberately not taken from a request, since a `Host` header is written by whoever is talking. |
| `open=` | extra paths reachable without a session, added to `middleware.OPEN`. A name ending in `/` is a prefix; any other is matched whole. `("/static/",)` is what an app with a stylesheet passes. | | `open=` | extra paths reachable without a session, added to `middleware.OPEN`. A name ending in `/` is a prefix; any other is matched whole. `("/static/",)` is what an app with a stylesheet passes. |
| `session_only=` | extra paths an API token may not reach, added to `middleware.ACCOUNT` and matched the same way. Anything that reads or replaces the database as a file belongs here, since the file holds the password hash and the token rows. |
| `stylesheet=` | a URL the login page links, so it looks like the rest of the app. Optional: the page carries enough inline CSS to be presentable on its own. | | `stylesheet=` | a URL the login page links, so it looks like the rest of the app. Optional: the page carries enough inline CSS to be presentable on its own. |
The account screen is two lines in a template, the same deal `<desk-jobs>` makes: The account screen is two lines in a template, the same deal `<desk-jobs>` makes:
@@ -44,7 +45,7 @@ The account screen is two lines in a template, the same deal `<desk-jobs>` makes
<auth-account></auth-account> <auth-account></auth-account>
``` ```
It draws the lot — locked or not, change the password, the sessions with revoke and close-others, the passkeys with register, rename and revoke, and sign out — and it is themed from outside by the same `--desk-jobs-*` custom properties the queue's elements read, so `desk-jobs, desk-schedules, auth-account { ... }` in one stylesheet is all three. It draws the lot — locked or not, change the password, the sessions with revoke and close-others, the passkeys with register, rename and revoke, the API tokens with create, rename and revoke, and sign out — and it is themed from outside by the same `--desk-jobs-*` custom properties the queue's elements read, so `desk-jobs, desk-schedules, auth-account { ... }` in one stylesheet is all three. Under `40rem` a row's buttons drop under its name when they do not fit beside it and a form's fields take the width; under a coarse pointer every button is a thumb's height and no field is under 16px. The login page does the same for itself, since it is served before any of the app's CSS.
## The config block ## The config block
@@ -69,7 +70,7 @@ auth:
cookie: app_session cookie: app_session
``` ```
`enabled: auto` is the setting that matters: locked unless the server binds a loopback address. Deciding from the bind address rather than defaulting to off prevents the failure that actually happens — you deploy, it works, and nothing anywhere tells you the door is open. Production says `always` anyway, because behind a TLS-terminating proxy every request looks local to the app and is not. `enabled: auto` is the setting that matters: locked unless the server binds a loopback address. Deciding from the bind address rather than defaulting to off prevents the failure that actually happens — you deploy, it works, and nothing anywhere tells you the door is open. Production says `always` anyway, so the lock does not depend on the bind address or on how the process was started. Under `auto` an app left open on loopback answers any request carrying `X-Forwarded-For`, `X-Forwarded-Proto`, `X-Forwarded-Host` or `Forwarded` with 421 and logs a warning once, since a proxy on the same machine would otherwise serve it to the network without a password. The Vite dev server's proxy sends none of these headers.
`password_env` and `cookie` are settings because this folder is copied into projects that are not called `app`, and two of those on one hostname would otherwise share a cookie and sign each other out. `password_env` and `cookie` are settings because this folder is copied into projects that are not called `app`, and two of those on one hostname would otherwise share a cookie and sign each other out.
@@ -77,33 +78,42 @@ auth:
The password is never in the config file — only ever an scrypt hash in the database. The first locked start takes `$APP_PASSWORD` if it is set, and otherwise invents one and prints it to the log once, in a plain ASCII banner you can read back off `docker compose logs`. The password is never in the config file — only ever an scrypt hash in the database. The first locked start takes `$APP_PASSWORD` if it is set, and otherwise invents one and prints it to the log once, in a plain ASCII banner you can read back off `docker compose logs`.
`APP_PASSWORD` is authoritative while it is set: the Account screen says so and declines to change it. That is the way back in when a login is lost — set it, get in, empty it, recreate the container, then change the password in the app. A variable that silently stopped applying the moment you changed the password in the browser would be worse than either rule. `APP_PASSWORD` is authoritative while it is set: the Account screen says so and declines to change it. That is the way back in when a login is lost — set it, get in, empty it, recreate the container, then change the password in the app. A variable that silently stopped applying the moment you changed the password in the browser would be worse than either rule. When it replaces a stored password, the start closes every session, since whoever had the old password may be signed in, and logs at WARNING how many API tokens and passkeys remain; review them under Account, or tick the revoke box on the next password change.
## What the app has to provide ## What the app has to provide
Almost nothing, and that is the point. Almost nothing, and that is the point.
- A SQLite path. `AuthStore` creates four `auth_*` tables in whatever file it is given, so it can share the app's own. - A SQLite path. `AuthStore` creates five `auth_*` tables in whatever file it is given, so it can share the app's own.
- The bind host, for `enabled: auto`. - The bind host, for `enabled: auto`.
- Nothing for the open list, as a rule. `/login`, `/login/`, `/_auth/` and `/health` are opened by this package; the app adds `/static/` if it serves one, or the login page arrives unstyled. Everything else stays shut — a module's own scripts and sockets included, so in this repository `/_jobs/jobs.js` and `/ws/jobs` need a session like the pages that load them. - Nothing for the open list, as a rule. `/login`, `/login/`, `/_auth/` and `/health` are opened by this package; the app adds `/static/` if it serves one, or the login page arrives unstyled. Everything else stays shut — a module's own scripts and sockets included, so in this repository `/_jobs/jobs.js` and `/ws/jobs` need a session like the pages that load them.
- An error handler that keeps `{"detail": "..."}` for `/api`, `/ws` and `/_auth`, if it has one. Every refusal here is already that shape. - An error handler that keeps `{"detail": "..."}` for `/api`, `/ws` and `/_auth`, if it has one. Every refusal here is already that shape.
- `app.state.bus`, if it holds sockets open: an object with a `sockets` tuple and an async `close(sockets, code, reason)`, which is what `pipeline.web.connect` puts there. Without one, a socket stays open after its credential ends until the tab reconnects.
What it must **not** do is protect its own routes. Everything is shut except the open list, which is four entries long and asserted in a test — so a router added next month is safe because it exists, not because somebody remembered. What it must **not** do is protect its own routes. Everything is shut except the open list, which is four entries long and asserted in a test — so a router added next month is safe because it exists, not because somebody remembered.
## The decisions, in one paragraph each ## The decisions, in one paragraph each
**One door, in front of everything.** The door is one middleware, installed once by `connect`, in front of every route and the socket. A route checks no credential itself, and the only exceptions are the open list. A route that needs the password again, such as a restore whose body is the database, calls `auth.confirm_password(request, current)`, which is counted and gated like the front door; when the body is a file, it calls `auth.confirm_current(request)` instead, which reads `current` from an `X-Current-Password` header before the body, and answers 403 without counting when the header is missing. A socket is checked at its handshake and closed with 1008 when its credential ends: on a revoke by `close_ended_sockets`, which `/logout`, the password change and the session and token revokes run after their answer, and on expiry by `sweep_ended_sockets`, which `connect` runs on a locked app every `SWEEP` (60) seconds.
**scrypt, not SHA-256.** Memory-hard rather than merely slow: one guess costs 16 MB, which is what takes a stolen database file out of range of a GPU. A wrong answer costs a full derivation too, so the delay an attacker meets is the one the honest user meets. **scrypt, not SHA-256.** Memory-hard rather than merely slow: one guess costs 16 MB, which is what takes a stolen database file out of range of a GPU. A wrong answer costs a full derivation too, so the delay an attacker meets is the one the honest user meets.
**A session is 32 random bytes, and the table holds only its SHA-256.** No signing key to generate, store, keep out of backups and rotate; signing out everywhere is one `DELETE`, which no self-contained token can offer without keeping exactly this state. **A session is 32 random bytes, and the table holds only its SHA-256.** No signing key to generate, store, keep out of backups and rotate; signing out everywhere is one `DELETE`, which no self-contained token can offer without keeping exactly this state.
**The lockout is counted in the file and read before the check.** A counter on the process is a free reset for whoever is guessing, on a server that reloads. Read after instead of before, and hammering the button extends your own lockout — which punishes the owner far harder than the attacker who is not watching. Five wrong answers are free, then 30 s doubling to a quarter of an hour, with the exponent capped so a bot cannot turn arithmetic into a 500. **The lockout is counted in the file and claimed before the check.** A counter on the process is a free reset for whoever is guessing, on a server that reloads. Each guess is claimed before its derivation, in one `BEGIN IMMEDIATE` that reads `locked_until` and counts the attempt, and a right answer clears the count afterwards; read before the derivation and written after it, thirty concurrent guesses all found the door open. A claim refused while locked counts nothing, so hammering the button does not extend your own lockout, which would punish the owner far harder than the attacker who is not watching. Five wrong answers are free, then 30 s doubling to a quarter of an hour, with the exponent capped so a bot cannot turn arithmetic into a 500.
**One counter, not one per address.** Behind a proxy the only address the app sees is one the caller wrote. Keyed on that, an attacker gets a fresh counter per request while the owner still gets locked out by their own typos. So the address is logged — one WARNING per refusal, `auth.router: wrong password from <addr>`, stripped to the characters an address is made of — and decided from at the layer that knows who connected. `docs/password-protection.md` has the nginx and fail2ban half. **One counter, not one per address.** Behind a proxy the only address the app sees is one the caller wrote. Keyed on that, an attacker gets a fresh counter per request while the owner still gets locked out by their own typos. So the address is logged — one WARNING per refusal, `auth.middleware: wrong password from <addr>`, stripped to the characters an address is made of — and decided from at the layer that knows who connected. The address is the rightmost hop of `X-Forwarded-For`, which the proxy appended, else `X-Real-IP`, else the peer; the hops left of it are the caller's own writing. `documentation/Password Protection.md` has the nginx and fail2ban half.
**No CSRF token.** `SameSite=Lax` refuses to send this cookie on any cross-site request that is not a top-level navigation, and no `GET` here changes anything, so the attack a token would stop cannot be mounted. A second secret to rotate, and a second way for a request to fail, buys nothing. **No CSRF token.** `SameSite=Lax` refuses to send this cookie on any cross-site request that is not a top-level navigation, and no `GET` here changes anything, so the attack a token would stop cannot be mounted. A second secret to rotate, and a second way for a request to fail, buys nothing.
**Passkeys need a real hostname.** A passkey is bound to an origin, and that binding is what makes it unphishable — so it needs HTTPS, with `localhost` the one exception, which is what lets it be tried before deploying. Behind a proxy, set `rp_id` and `origin` explicitly or every ceremony fails as an origin mismatch. Changing `rp_id` later invalidates every key registered against the old one. The password never goes away, which is why revoking a key can never lock you out. **Passkeys need a real hostname.** A passkey is bound to an origin, and that binding is what makes it unphishable — so it needs HTTPS, with `localhost` the one exception, which is what lets it be tried before deploying. Behind a proxy, set `rp_id` and `origin` explicitly or every ceremony fails as an origin mismatch. Changing `rp_id` later invalidates every key registered against the old one. The password never goes away, which is why revoking a key can never lock you out.
**A passkey is added only with the password.** A session is not enough: a passkey stays valid after every session ends and after a password change, so a borrowed signed-in browser must not be able to add one. `POST /api/account/passkeys/options` takes `current`, checked behind the same lockout as the front door. An app with no password yet (loopback, never locked) registers without it. Changing the password leaves the passkeys registered; revoke any you do not recognise.
**An API token is a session for a script.** `POST /api/account/tokens` takes the password like a passkey does, and answers with the token once; the table keeps its SHA-256, a label, the first eight characters so the owner can tell the rows apart, and an optional expiry. A script sends it as `Authorization: Bearer <token>` and it is looked up on every request, so revoking the row takes effect on the next call. It opens everything a session opens except `/api/account`, `/logout` and what the app names in `connect(session_only=...)`, which answer 403 to a token: a token that leaked cannot mint another, revoke the owner's sessions or read the list of ways in. A route that downloads or restores the database belongs on that list, because the file holds the password hash and the token rows, and a token that could fetch or swap it would hold the account after all. A token the store does not know is a 401 and a WARNING line like a wrong password, and is not counted against the lockout, since 256 random bits are not guessed and the counter would only hand a stranger a way to lock the owner out. A password change leaves tokens in place, as it does passkeys, unless `revoke_all` is sent with it, which deletes every token and passkey; the answer counts the sessions closed and the tokens and passkeys revoked and left.
**Another site's page changes nothing.** `SameSite=Lax` covers a locked app, and an unlocked one on loopback has no cookie to withhold: a page open in the same browser could `POST` a `text/plain` body to any route that takes none, and open `/ws/jobs`, which CORS does not cover. So the middleware refuses a request that changes something when `Sec-Fetch-Site`, or `Origin` against `Host` or `auth.origin`, says it came from another site, refuses a body of a type a page can send without a preflight (the API is JSON), and refuses a socket handshake from another origin. A read is never refused, and a request with no `Origin` at all (`curl`, a script with a token) passes. The comparison is against the request's own `Host`, which a DNS rebinding page matches, so an app `auto` left open also answers 421 to any request whose `Host` is not a loopback name, a `.localhost` name or the host of `auth.origin`, whatever other headers it carries: a loopback app is reached as `localhost`, and nothing else should be naming it. A test client of an unlocked app therefore names one, e.g. `TestClient(app, base_url="http://127.0.0.1")`.
## Tests ## Tests
```bash ```bash
+58 -19
View File
@@ -9,27 +9,30 @@ That is the middleware in front of everything (the WebSocket included), the
login page, the account API, the `<auth-account>` element, and the startup login page, the account API, the `<auth-account>` element, and the startup
bootstrap that makes sure there is a password at all. No user management: there bootstrap that makes sure there is a password at all. No user management: there
is one user, they are the person who deployed it, and the only questions are is one user, they are the person who deployed it, and the only questions are
whether the door is locked and what opens it. whether the door is locked and what opens it: the password, a passkey, or an
API token carried by a script as `Authorization: Bearer`.
| module | | | module | |
|---|---| |---|---|
| `config.py` | `AuthConfig`: the `auth:` block of the app's `config.yaml` | | `config.py` | `AuthConfig`: the `auth:` block of the app's `config.yaml` |
| `store.py` | `AuthStore`: scrypt, the lockout, sessions, passkeys, four `auth_*` tables in whatever SQLite file it is given | | `store.py` | `AuthStore`: scrypt, the lockout, sessions, passkeys, API tokens, five `auth_*` tables in whatever SQLite file it is given |
| `models.py` | `Account`, `SessionInfo`, `Passkey` — what the account screen is told, which carries no credential | | `models.py` | `Account`, `SessionInfo`, `Passkey`, `ApiToken` — what the account screen is told, which carries no credential |
| `middleware.py` | `RequireLogin`, `locked()`, the open list, the cookie | | `middleware.py` | `RequireLogin`, `locked()`, the open list, the cookie, the bearer header, the same-site rule, `confirm_password`, `confirm_current` and `warn`, and `close_ended_sockets` with its sweep |
| `passkeys.py` | the two WebAuthn ceremonies; `webauthn` is an optional extra, imported lazily | | `passkeys.py` | the two WebAuthn ceremonies; `webauthn` is an optional extra, imported lazily |
| `schemas.py` | the five request bodies | | `schemas.py` | the seven request bodies |
| `router.py` | `/login`, `/logout`, `/api/account/*`, the passkey routes, `/_auth/*.js` | | `router.py` | `/login`, `/logout`, `/api/account/*`, the passkey and token routes, `/_auth/*.js` |
| `templates/login.html` | the one page served without a session | | `templates/login.html` | the one page served without a session |
| `static/` | `passkey.js` (the browser half of WebAuthn) and `account.js` (`<auth-account>`) | | `static/` | `passkey.js` (the browser half of WebAuthn) and `account.js` (`<auth-account>`) |
The reasoning is `docs/password-protection.md`, and this package is that The reasoning is the boilerplate's `documentation/Password Protection.md`, and this package is that
document in code. `auth/README.md` is how to copy the folder into a project document in code. `auth/README.md` is how to copy the folder into a project
that has never heard of this one. that has never heard of this one.
""" """
from __future__ import annotations from __future__ import annotations
import asyncio
import contextlib
import logging import logging
import os import os
import secrets import secrets
@@ -41,23 +44,29 @@ from fastapi import FastAPI
from auth import passkeys from auth import passkeys
from auth.config import AuthConfig from auth.config import AuthConfig
from auth.middleware import OPEN, RequireLogin, client_address, locked, token_from from auth.middleware import (
from auth.models import Account, Passkey, SessionInfo ACCOUNT, CURRENT_HEADER, OPEN, RequireLogin, bearer_from, client_address,
close_ended_sockets, confirm_current, confirm_password, credential_live, locked,
sweep_ended_sockets, token_from, warn,
)
from auth.models import Account, ApiToken, Passkey, SessionInfo
from auth.router import router from auth.router import router
from auth.store import AuthStore from auth.store import AuthStore
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
__all__ = [ __all__ = [
"OPEN", "Account", "AuthConfig", "AuthStore", "Passkey", "RequireLogin", "ACCOUNT", "CURRENT_HEADER", "OPEN", "Account", "ApiToken", "AuthConfig", "AuthStore",
"SessionInfo", "bootstrap", "client_address", "connect", "locked", "Passkey", "RequireLogin", "SessionInfo", "bearer_from", "bootstrap", "client_address",
"passkeys", "router", "token_from", "close_ended_sockets", "confirm_current", "confirm_password", "connect",
"credential_live", "locked", "passkeys", "router", "sweep_ended_sockets", "token_from",
"warn",
] ]
def connect(app: FastAPI, store: AuthStore, config: AuthConfig, *, def connect(app: FastAPI, store: AuthStore, config: AuthConfig, *,
open: Iterable[str] = (), stylesheet: str | None = None, open: Iterable[str] = (), session_only: Iterable[str] = (),
host: str = "") -> AuthStore: stylesheet: str | None = None, host: str = "") -> AuthStore:
"""Put `app` behind a password: middleware, routes, element, bootstrap. """Put `app` behind a password: middleware, routes, element, bootstrap.
Call it first, before the app's own routers. Starlette wraps the whole Call it first, before the app's own routers. Starlette wraps the whole
@@ -72,13 +81,17 @@ def connect(app: FastAPI, store: AuthStore, config: AuthConfig, *,
| keyword | | | keyword | |
|---|---| |---|---|
| `open` | extra paths reachable without a session, added to `middleware.OPEN`. A name ending in `/` is a prefix; any other is matched whole. `("/static/",)` is what an app with a stylesheet passes. | | `open` | extra paths reachable without a session, added to `middleware.OPEN`. A name ending in `/` is a prefix; any other is matched whole. `("/static/",)` is what an app with a stylesheet passes. |
| `session_only` | extra paths an API token may not reach, added to `middleware.ACCOUNT`, matched the same way. Anything that reads or replaces the database as a file belongs here: it holds the password hash and the tokens, so a token that could fetch or swap it would hold the account. A route listed here should also take the current password; a route that hands out or replaces the database as a file takes it in an `X-Current-Password` header, through `auth.confirm_current`, and any other through `auth.confirm_password`. |
| `stylesheet` | a URL the login page links, so it looks like the rest of the app. Optional: the page carries enough inline CSS to be presentable on its own. | | `stylesheet` | a URL the login page links, so it looks like the rest of the app. Optional: the page carries enough inline CSS to be presentable on its own. |
| `host` | the address the server binds — `config.server.host`. What `enabled: auto` decides from, and deliberately not taken from a request, since a `Host` header is written by whoever is talking. | | `host` | the address the server binds — `config.server.host`. What `enabled: auto` decides from, and deliberately not taken from a request, since a `Host` header is written by whoever is talking. |
The lifespan is wrapped rather than replaced, the same way The lifespan is wrapped rather than replaced, the same way
`pipeline.web.connect` does it, so an app's own lifespan still runs inside `pipeline.web.connect` does it, so an app's own lifespan still runs inside
this one: the password exists before the app's startup code, which may want this one: the password exists before the app's startup code, which may want
to say something about it. to say something about it. On a locked app it also runs
`sweep_ended_sockets` for as long as the app is up, as the task on
`app.state.auth_sweep` (None on an unlocked app), so a socket whose session
or token expires is closed within `middleware.SWEEP` seconds.
Returns the store, for the app to keep if it wants one. Returns the store, for the app to keep if it wants one.
""" """
@@ -88,10 +101,12 @@ def connect(app: FastAPI, store: AuthStore, config: AuthConfig, *,
#: the bind address, which only the caller knows. #: the bind address, which only the caller knows.
app.state.auth_locked = locked(config, host) app.state.auth_locked = locked(config, host)
app.state.auth_stylesheet = stylesheet or "" app.state.auth_stylesheet = stylesheet or ""
#: The `sweep_ended_sockets` task while a locked app is up; see below.
app.state.auth_sweep = None
app.include_router(router) app.include_router(router)
app.add_middleware(RequireLogin, store=store, config=config, app.add_middleware(RequireLogin, store=store, config=config,
open=tuple(open), host=host) open=tuple(open), host=host, session_only=tuple(session_only))
inner = app.router.lifespan_context inner = app.router.lifespan_context
@@ -104,8 +119,18 @@ def connect(app: FastAPI, store: AuthStore, config: AuthConfig, *,
# a loopback app before it is ever deployed, which is the order this # a loopback app before it is ever deployed, which is the order this
# encourages, so an unlocked one can still have keys it cannot use. # encourages, so an unlocked one can still have keys it cannot use.
passkeys.warn_if_missing(store) passkeys.warn_if_missing(store)
async with inner(this) as state: # Expiry ends no request, so no route closes the socket it ended.
yield state sweep = (asyncio.create_task(sweep_ended_sockets(app))
if app.state.auth_locked else None)
app.state.auth_sweep = sweep
try:
async with inner(this) as state:
yield state
finally:
if sweep is not None:
sweep.cancel()
with contextlib.suppress(asyncio.CancelledError):
await sweep
app.router.lifespan_context = lifespan app.router.lifespan_context = lifespan
return store return store
@@ -124,12 +149,26 @@ def bootstrap(store: AuthStore, config: AuthConfig) -> None:
Re-derived only when the stored hash does not already match, so a restart is Re-derived only when the stored hash does not already match, so a restart is
not a write and "password last changed" keeps meaning something. not a write and "password last changed" keeps meaning something.
When the variable replaces a password that was there, every session is
closed: that is the recovery, and whoever had the old password may be
signed in. API tokens and passkeys stay, since the environment cannot say
which of them are the owner's, and their counts are logged at WARNING so
the owner knows to review them under Account.
""" """
pinned = os.environ.get(config.password_env, "") pinned = os.environ.get(config.password_env, "")
if pinned: if pinned:
if not store.check_password(pinned): if not store.check_password(pinned):
replaced = store.has_password()
store.set_password(pinned) store.set_password(pinned)
log.info("password set from %s", config.password_env) if not replaced:
log.info("password set from %s", config.password_env)
return
closed = store.close_all()
tokens, keys = store.counts()
log.warning("password reset from %s: closed %d sessions; %d API tokens and "
"%d passkeys remain, review them under Account",
config.password_env, closed, tokens, keys)
return return
if store.has_password(): if store.has_password():
+8 -8
View File
@@ -22,7 +22,7 @@ from pydantic import BaseModel, ConfigDict, Field
class AuthConfig(BaseModel): class AuthConfig(BaseModel):
"""The `auth:` block of `config.yaml`. See docs/password-protection.md.""" """The `auth:` block of `config.yaml`. See documentation/Password Protection.md."""
#: Unknown keys are refused rather than ignored, the same rule the app's #: Unknown keys are refused rather than ignored, the same rule the app's
#: own sections keep: a setting you believe you have changed and have not #: own sections keep: a setting you believe you have changed and have not
@@ -38,9 +38,9 @@ class AuthConfig(BaseModel):
#: Deciding from the bind address rather than defaulting to off is the #: Deciding from the bind address rather than defaulting to off is the
#: whole point. The failure this prevents is the one that actually happens: #: whole point. The failure this prevents is the one that actually happens:
#: you deploy, it works, and nothing anywhere tells you the door is open. #: you deploy, it works, and nothing anywhere tells you the door is open.
#: `always` locks it on localhost too and is what a deployment behind a #: `always` locks it on localhost too and is what a deployment sets, so the
#: TLS-terminating proxy needs, since there every request looks local to #: lock does not depend on the bind address or on how the process was
#: the app and is not. `never` is for putting some other authentication in #: started. `never` is for putting some other authentication in
#: front of this one, and means exactly what it says. #: front of this one, and means exactly what it says.
enabled: Literal["auto", "always", "never"] = "auto" enabled: Literal["auto", "always", "never"] = "auto"
@@ -52,10 +52,10 @@ class AuthConfig(BaseModel):
#: Whether the session cookie is marked `Secure`, so a browser will only #: Whether the session cookie is marked `Secure`, so a browser will only
#: ever send it back over HTTPS. `auto` sets it when the request itself #: ever send it back over HTTPS. `auto` sets it when the request itself
#: arrived over HTTPS, which is right when the app terminates TLS and wrong #: arrived over HTTPS, which behind a reverse proxy that terminates TLS
#: behind a reverse proxy that terminates it for you -- there the app sees #: holds only while the server trusts `X-Forwarded-Proto` from it; the app
#: plain HTTP and `auto` reads as "no". Set `always` when something else is #: alone cannot tell. Set `always` when something else is doing TLS.
#: doing TLS. `never` exists for testing over http on a LAN and is the #: `never` exists for testing over http on a LAN and is the
#: setting to be uncomfortable with. #: setting to be uncomfortable with.
secure_cookie: Literal["auto", "always", "never"] = "auto" secure_cookie: Literal["auto", "always", "never"] = "auto"
+511 -28
View File
@@ -1,8 +1,10 @@
"""The lock on the door: is there a session, and what happens when there isn't. """The lock on the door: is there a session, and what happens when there isn't.
Three things live here — when the door is locked at all, who gets past it, and Six things live here — when the door is locked at all, who gets past it, the
the cookie that says so. The password itself is `auth/store.py`; this module cookie and the bearer header that say so, the rule that another site's page
never sees a hash. gets nothing changed, the password asked again from behind a session
(`confirm_password`, and `confirm_current` for a header), and the closing of a socket whose credential has ended.
The password itself is `auth/store.py`; this module never sees a hash.
## Why a middleware and not a dependency ## Why a middleware and not a dependency
@@ -30,6 +32,66 @@ password is a second lock on it; bound to anything else, the app is on a
network and needs its own. The alternative — default off, switch it on when you network and needs its own. The alternative — default off, switch it on when you
deploy — fails silently in the exact case it exists for, because an app serving deploy — fails silently in the exact case it exists for, because an app serving
the internet with no password looks and behaves precisely like one that is fine. the internet with no password looks and behaves precisely like one that is fine.
A reverse proxy on the same machine defeats the bind address, so an app `auto`
left open refuses any request a proxy forwarded; see
`RequireLogin._refuse_proxied`.
## Three credentials, two carriers
A session cookie is what the password and a passkey open, and it is what a
browser carries. An API token (`AuthStore.create_token`) is what a script
carries, as `Authorization: Bearer <token>`, and it is looked up on every
request the way a session is, so revoking it takes effect on the next call.
The cookie is read first and wins; the header is read only when there is no
live session. A token opens every door the session does except the account:
a request carrying one for `/api/account` or `/logout` is answered 403
(`ACCOUNT`), so a token that leaked cannot mint another, revoke the owner's
sessions or read the list of ways in. An app adds its own session-only routes
through `connect(session_only=...)`: anything that reads or replaces the
database as a file belongs there, since the file holds the password hash and
these rows, and a token that could fetch or swap it would hold the account
after all. A token that is not live is a 401 and a
WARNING line, like a wrong password; it is not counted against the lockout,
since 256 random bits are not guessed and the counter would only hand a
stranger a way to lock the owner out.
## Another site's page
`SameSite=Lax` keeps the session cookie off a cross-site `POST`, which covers
a locked app. An unlocked one on loopback has no cookie to withhold, so a page
open in the same browser could `POST` to `http://127.0.0.1:8787/api/...` with
a `text/plain` body, which needs no preflight, and every route that takes no
body acted on it; a WebSocket handshake is not subject to CORS at all, so the
same page could open `/ws/jobs` and read every job and notice. So a request
that changes something is refused when the browser says it came from another
site (`Sec-Fetch-Site`, else `Origin` against `Host` or `AuthConfig.origin`),
and a body of a type another site could send without a preflight is refused
whoever sent it: the API is JSON. Locked or not, because the rule costs two
header reads and a locked app with `secure_cookie: never` on a LAN is a real
deployment. See `cross_site`.
That comparison is against the request's own `Host`, and a page served from
`rebind.example` whose name is pointed at 127.0.0.1 arrives with a `Host` and
an `Origin` that agree. So an app `auto` left open also refuses any request
whose `Host` is not a loopback name, a `.localhost` name or the host of
`AuthConfig.origin`, with the same 421 as a forwarded one, whatever else it
carries: a browser that sends no fetch metadata and no `Origin` on a read is
still a rebound page. A loopback app is reached as `localhost` or
`127.0.0.1`, and a request that names another host for it came through a
proxy or a rebinding, neither of which the bind address can see. A test
client names the host it is given (`base_url="http://127.0.0.1"`).
## A socket outlives its handshake
The door is checked when a socket opens, and the socket stays open for the
life of the tab. So a route that ends a credential (`/logout`, a password
change, a revoked session or token) runs `close_ended_sockets` after its
answer, which asks the store about every open socket's cookie or bearer header
and closes the ones it no longer knows with 1008. Expiry ends no request, so
`connect` also runs `sweep_ended_sockets`, which does the same every `SWEEP`
seconds. A tab's reconnect then meets the door. The sockets are the app's:
`app.state.bus`, which `pipeline.web.connect` sets, and an app without one has
nothing to close.
""" """
from __future__ import annotations from __future__ import annotations
@@ -37,11 +99,11 @@ from __future__ import annotations
import asyncio import asyncio
import ipaddress import ipaddress
import logging import logging
from http.cookies import SimpleCookie
from typing import Any from typing import Any
from urllib.parse import quote from urllib.parse import quote, urlsplit
from starlette.requests import HTTPConnection from fastapi import HTTPException
from starlette.requests import HTTPConnection, cookie_parser
from starlette.responses import JSONResponse, RedirectResponse, Response from starlette.responses import JSONResponse, RedirectResponse, Response
from starlette.types import ASGIApp, Receive, Scope, Send from starlette.types import ASGIApp, Receive, Scope, Send
@@ -61,6 +123,22 @@ log = logging.getLogger(__name__)
#: against the path only — a query string cannot widen this. #: against the path only — a query string cannot widen this.
OPEN: tuple[str, ...] = ("/login", "/login/", "/_auth/", "/health") OPEN: tuple[str, ...] = ("/login", "/login/", "/_auth/", "/health")
#: What a bearer token cannot reach: the account, which is managed from a
#: browser with a session, and the sign-out of a session a token has not got.
#: Matched as `OPEN` is. Everything else a session opens, a token opens.
ACCOUNT: tuple[str, ...] = ("/api/account", "/api/account/", "/logout")
#: What a request carrying a token for the account is told.
TOKEN_NOT_HERE = ("An API token opens the API, not the account. Sign in with the "
"password or a passkey to manage it.")
def matches(path: str, names: tuple[str, ...]) -> bool:
"""Whether `path` is one of `names`: a name ending in `/` is a prefix, any
other is matched whole. The one rule for `OPEN` and `ACCOUNT`."""
return any(path == name.rstrip("/") or (name.endswith("/") and path.startswith(name))
for name in names)
def locked(config: AuthConfig, host: str) -> bool: def locked(config: AuthConfig, host: str) -> bool:
"""Whether this app should ask for a password. See `AuthConfig.enabled`. """Whether this app should ask for a password. See `AuthConfig.enabled`.
@@ -126,47 +204,113 @@ def token_from(conn: HTTPConnection | Scope, config: AuthConfig) -> str:
Both cookie names are looked for, so flipping `secure_cookie` or moving Both cookie names are looked for, so flipping `secure_cookie` or moving
behind TLS does not strand a live session under the other name. behind TLS does not strand a live session under the other name.
Parsed per `;`-separated pair by Starlette's `cookie_parser`, as a browser
does, and not by `http.cookies.SimpleCookie`, whose `load` returns an
empty jar for the whole header when one cookie in it has a space or JSON
in its value: a sibling app on the parent domain setting `theme=dark mode`
signed the owner out on every request. A pair that is not a cookie is
skipped, so a garbage header is no cookie rather than a 500.
""" """
scope: Any = conn.scope if isinstance(conn, HTTPConnection) else conn scope: Any = conn.scope if isinstance(conn, HTTPConnection) else conn
for key, value in scope.get("headers", ()): for key, value in scope.get("headers", ()):
if key == b"cookie": if key == b"cookie":
jar = SimpleCookie() jar = cookie_parser(value.decode("latin-1"))
try:
jar.load(value.decode("latin-1"))
except Exception: # a cookie header that is not one
return ""
for name in cookie_names(config): for name in cookie_names(config):
if name in jar: if jar.get(name):
return jar[name].value return jar[name]
return "" return ""
def bearer_from(conn: HTTPConnection | Scope) -> str:
"""The API token this request carries in `Authorization: Bearer`, or "".
Only that header and only that scheme: a token in a query string would
land in every access log between the script and the app, and `Basic` is
a password, which has its own door.
"""
value = _header(conn, b"authorization")
scheme, _, token = value.strip().partition(" ")
if scheme.lower() != "bearer":
return ""
return token.strip()
def _header(conn: HTTPConnection | Scope, name: bytes) -> str:
scope: Any = conn.scope if isinstance(conn, HTTPConnection) else conn
for key, value in scope.get("headers", ()):
if key == name:
return value.decode("latin-1")
return ""
def credential_live(store: AuthStore, config: AuthConfig,
conn: HTTPConnection | Scope) -> bool:
"""Whether what this request carries still opens the door: its cookie's
session, else its bearer token. What `RequireLogin` decides from, asked
again of an open socket by `close_ended_sockets`."""
token = token_from(conn, config)
if token and store.touch(token) is not None:
return True
bearer = bearer_from(conn)
return bool(bearer) and store.touch_token(bearer) is not None
def client_address(conn: HTTPConnection | Scope) -> str: def client_address(conn: HTTPConnection | Scope) -> str:
"""Who is asking, as well as it can be known, for the log to say so. """Who is asking, as well as it can be known, for the log to say so.
Behind a reverse proxy the peer is a container on the same Docker network Behind a reverse proxy the peer is a container on the same Docker network
and says nothing about who is at the other end; the address worth writing and says nothing about who is at the other end. The address worth writing
down is the first hop of `X-Forwarded-For`, which the proxy sets. Both are down is the rightmost hop of `X-Forwarded-For`, the one the proxy in front
reported when they differ, because the header is only as trustworthy as of this app appended for the connection it accepted; every hop left of it
whoever sent it — which is why nothing is *decided* from this, only logged. was in the request when it arrived and is whatever the caller wrote. nginx's
See `AuthStore.check_password` for that argument in full. `$proxy_add_x_forwarded_for` appends rather than replaces, so the first hop
let an attacker name the address a fail2ban filter on this line bans.
Without that header, `X-Real-IP`, which such a proxy sets outright. The
claimed address and the peer are both reported when they differ, because a
header is only as trustworthy as the proxy that set it, which is why
nothing is *decided* from this, only logged. See
`AuthStore.check_password` for that argument in full.
That header is also the one piece of a stranger's own writing that ends up The header is also the one piece of a stranger's writing that can end up
in the log, so only the characters an address is made of survive the trip: in the log, so only the characters an address is made of survive the trip:
a line in this log is something fail2ban reads and a person believes, and a line in this log is something fail2ban reads and a person believes, and
neither should be forgeable by whoever is knocking. neither should be forgeable by whoever is knocking. The address stays the
first word after `from`, so a filter is one regex.
""" """
scope: Any = conn.scope if isinstance(conn, HTTPConnection) else conn scope: Any = conn.scope if isinstance(conn, HTTPConnection) else conn
peer = (scope.get("client") or ("", 0))[0] or "unknown" peer = (scope.get("client") or ("", 0))[0] or "unknown"
forwarded = real = ""
for key, value in scope.get("headers", ()): for key, value in scope.get("headers", ()):
# The last of each, should a header arrive twice: a proxy that adds
# one rather than appending writes it after the caller's.
if key == b"x-forwarded-for": if key == b"x-forwarded-for":
first = value.decode("latin-1").split(",")[0] forwarded = value.decode("latin-1")
claimed = "".join(c for c in first if c.isalnum() or c in ".:%-")[:45] elif key == b"x-real-ip":
if claimed and claimed != peer: real = value.decode("latin-1")
return f"{claimed} via {peer}" hop = forwarded.split(",")[-1] if forwarded.strip() else real
claimed = "".join(c for c in hop if c.isalnum() or c in ".:%-")[:45]
if claimed and claimed != peer:
return f"{claimed} via {peer}"
return peer return peer
#: Headers a reverse proxy adds. Any one of them on a request to an app that
#: `auto` left open because it binds loopback means the bind address is not the
#: whole story: a proxy on this machine is passing on someone else.
FORWARDED = frozenset({b"forwarded", b"x-forwarded-for", b"x-forwarded-proto",
b"x-forwarded-host"})
#: What such a request is answered with, for whoever runs the proxy.
PROXIED = ("This app binds loopback and so asks for no password, but this request "
"came through a proxy. Set auth.enabled: always in config.yaml and restart.")
#: What a browser script's request naming another host is answered with.
REBOUND = ("This app binds loopback and so asks for no password, but this request "
"names a host that is not localhost. Open the app as http://localhost, or "
"set auth.enabled: always in config.yaml and restart.")
#: Prefixes whose refusal is JSON rather than a redirect. A caller that asked #: Prefixes whose refusal is JSON rather than a redirect. A caller that asked
#: for data and is sent a login page reads it as a 200 full of HTML where its #: for data and is sent a login page reads it as a 200 full of HTML where its
#: JSON should be; `{"detail": ...}` with a 401 is something it can act on. #: JSON should be; `{"detail": ...}` with a 401 is something it can act on.
@@ -187,20 +331,35 @@ class RequireLogin:
""" """
def __init__(self, app: ASGIApp, store: AuthStore, config: AuthConfig, def __init__(self, app: ASGIApp, store: AuthStore, config: AuthConfig,
open: tuple[str, ...] = (), host: str = "") -> None: open: tuple[str, ...] = (), host: str = "",
session_only: tuple[str, ...] = ()) -> None:
self.app = app self.app = app
self.store = store self.store = store
self.config = config self.config = config
self.open = OPEN + tuple(open) self.open = OPEN + tuple(open)
#: What a token may not reach: the account, and whatever the app adds.
self.session_only = (*ACCOUNT, *session_only)
self.locked = locked(config, host) self.locked = locked(config, host)
#: Open only because `auto` saw loopback; see `_refuse_proxied`.
self.watch_proxy = config.enabled == "auto" and not self.locked
self.warned = False
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] not in ("http", "websocket") or not self.locked: if scope["type"] not in ("http", "websocket"):
return await self.app(scope, receive, send)
# Another site's page, locked or not: see the module docstring.
refused = cross_site(scope, self.config.origin)
if refused is not None:
return await self._answer(scope, receive, send, *refused)
if not self.locked:
if self.watch_proxy and _forwarded(scope):
return await self._refuse_proxied(scope, receive, send, PROXIED)
if self.watch_proxy and _rebound(scope, self.config.origin):
return await self._refuse_proxied(scope, receive, send, REBOUND)
return await self.app(scope, receive, send) return await self.app(scope, receive, send)
path = scope.get("path", "") path = scope.get("path", "")
if any(path == p.rstrip("/") or (p.endswith("/") and path.startswith(p)) if matches(path, self.open):
for p in self.open):
return await self.app(scope, receive, send) return await self.app(scope, receive, send)
token = token_from(scope, self.config) token = token_from(scope, self.config)
@@ -214,8 +373,34 @@ class RequireLogin:
scope["auth_token"] = token scope["auth_token"] = token
return await self.app(scope, receive, send) return await self.app(scope, receive, send)
bearer = bearer_from(scope)
if bearer:
held = await asyncio.to_thread(self.store.touch_token, bearer)
if held is not None:
if matches(path, self.session_only):
return await self._answer(scope, receive, send, 403, TOKEN_NOT_HERE)
# The row, for a route that wants to know it was a script.
scope["auth_bearer"] = held.id
return await self.app(scope, receive, send)
# The same line a wrong password writes, for the same reader. A
# script with a bad token gets JSON whatever it asked for: a
# redirect to the login page is for a browser, and a browser does
# not send this header.
log.warning("bad token from %s", client_address(scope))
return await self._answer(scope, receive, send, 401,
"That token is not valid here.")
await self._refuse(scope, receive, send, path) await self._refuse(scope, receive, send, path)
async def _answer(self, scope: Scope, receive: Receive, send: Send,
status: int, detail: str) -> None:
"""A refusal in the shape the caller can act on: `{"detail"}` with the
status, or a 1008 close before the socket is accepted."""
if scope["type"] == "websocket":
await send({"type": "websocket.close", "code": 1008})
return
await JSONResponse({"detail": detail}, status_code=status)(scope, receive, send)
async def _refuse(self, scope: Scope, receive: Receive, send: Send, async def _refuse(self, scope: Scope, receive: Receive, send: Send,
path: str) -> None: path: str) -> None:
"""No session. Say so in whatever the caller can understand.""" """No session. Say so in whatever the caller can understand."""
@@ -244,3 +429,301 @@ class RequireLogin:
else: else:
response = JSONResponse({"detail": "Not signed in"}, status_code=401) response = JSONResponse({"detail": "Not signed in"}, status_code=401)
await response(scope, receive, send) await response(scope, receive, send)
async def _refuse_proxied(self, scope: Scope, receive: Receive, send: Send,
detail: str) -> None:
"""421 for a forwarded request to an app `auto` left open, and for a
request that names a host it is not reached under (`_rebound`).
`auto` reads loopback as "this machine only", and a reverse proxy on
the same machine makes that false while every request still arrives
from 127.0.0.1. The peer cannot tell the proxy from a local browser,
only the headers the proxy adds can, so a request carrying one is
refused rather than served unlocked. A client can add the headers
itself, which only gets it refused. The Vite dev server's proxy adds
none (`xfwd` is off) and forwards `Host` unchanged, so `npm run dev`
is unaffected. Logged once at WARNING, since every request through
the proxy would repeat it.
"""
if not self.warned:
self.warned = True
log.warning("refused a proxied or rebound request to an unlocked app (%s): %s",
client_address(scope), detail)
if scope["type"] == "websocket":
await send({"type": "websocket.close", "code": 1008})
return
await JSONResponse({"detail": detail}, status_code=421)(scope, receive, send)
def _forwarded(scope: Scope) -> bool:
return any(key in FORWARDED for key, _ in scope.get("headers", ()))
def _rebound(scope: Scope, configured_origin: str) -> bool:
"""Whether a request reaches this loopback app under a name that is not
loopback: the shape of a DNS rebinding, where the page's own `Origin` and
the `Host` it sends agree and `_from_here` cannot tell.
Every request is judged, whatever headers it carries. A browser that sends
neither `Sec-Fetch-Site` nor `Origin` on a read is still a page on the
rebound name reading the API, and the only names a loopback app is reached
under are a loopback address, `localhost` and its subdomains, and the host
of `AuthConfig.origin`. A missing or unparseable `Host` names none of
them and is refused too.
"""
try:
host = urlsplit("//" + _header(scope, b"host")).hostname or ""
except ValueError: # an unparseable authority, e.g. `[`
return True
if _loopback(host) or host.endswith(".localhost"):
return False
allowed = urlsplit(configured_origin).hostname if configured_origin else ""
return not host or host != (allowed or "").lower()
# ------------------------------------------------------ the password again
def claim_attempt(conn: HTTPConnection) -> None:
"""The lockout, in front of every place a password can be guessed: 429
while it holds, else the guess is counted before it is derived.
`AuthStore.claim_attempt` decides and counts in one write transaction, so
concurrent guesses each see the ones before them and a window admits five
of them, not a threadpool's worth. A refused claim counts nothing, so
hammering while locked out does not extend the lockout. The caller then
checks with `check_password(..., claimed=True)`, which clears the count
when the guess was right. Called from both doors rather than living inside
`check_password`, because the same comparison is a counted guess here and
a bootstrap comparison in `auth.bootstrap`, which must not be able to lock
anyone out of their own restart.
"""
wait = conn.app.state.auth.claim_attempt()
if wait:
warn(conn, "attempt while locked out")
raise HTTPException(429, f"Too many attempts. Try again in {_spell(wait)}.",
headers={"Retry-After": str(wait)})
def confirm_password(conn: HTTPConnection, current: str) -> None:
"""The password again, from someone already signed in: changing it, adding
a passkey, minting a token, and any app route that must not be one
borrowed browser away from the account (`connect(session_only=...)`).
Counted and gated like the front door, because `current` is a guess at the
password made from behind a session: 429 while locked out, 403 when wrong.
It has nothing to confirm on an app with no password yet, which refuses
every guess, so a route that can run there asks `has_password()` first, as
the account routes do."""
claim_attempt(conn)
if not conn.app.state.auth.check_password(current, claimed=True):
warn(conn, "wrong current password")
raise HTTPException(403, "That is not the current password.")
#: Where a route whose body is a file takes the current password. A header and
#: not a field, because the body is the file itself; read before the first byte
#: of that body, so a refusal costs no upload.
CURRENT_HEADER = "x-current-password"
def confirm_current(conn: HTTPConnection) -> None:
"""The current password from the `X-Current-Password` header, for a route
that hands out or replaces the database as a file (`connect(session_only=...)`).
A session is not enough for a route that replaces the credentials: the
restored file brings its own password hash, passkeys and tokens, so a
borrowed signed-in browser that could restore a doctored backup would own
the account. Call it before reading the body. A missing or empty header is
a WARNING line and a 403 that counts nothing against the lockout, since a
missing field is not a guess; a wrong one goes through `confirm_password`,
which counts it, warns and answers 403, or 429 while locked out. An app
that never had a password (loopback, never locked) has nothing to confirm.
The check is a deliberately slow derivation, so a coroutine route runs it
in the threadpool. newsdesk's `/api/restore` is the caller.
"""
if not conn.app.state.auth.has_password():
return
current = conn.headers.get(CURRENT_HEADER, "")
if not current:
warn(conn, f"no current password for {conn.scope.get('method', '')} "
f"{conn.scope.get('path', '')}")
raise HTTPException(403, "This takes the current password in X-Current-Password.")
confirm_password(conn, current)
def warn(conn: HTTPConnection, what: str) -> None:
"""Write down that someone got it wrong, and who.
The whole record of who is knocking. There is no access log worth the name —
uvicorn is run quietly — and no table of attempts, because the log is the
record and a second copy of it would only ever disagree. WARNING so it
survives that level, one line per refusal, and the address first after
`from` so a fail2ban filter is one regex; see documentation/Password Protection.md.
The message does not name the package, because the logger already does:
with the usual `%(name)s: %(message)s` this module is `auth.middleware`,
and a line reading `auth.middleware: auth: wrong password` says it twice
and reads like a bug in the logging.
Nothing is *decided* from the address — `AuthStore.check_password` says why
it cannot be — so this is where it earns its keep: at the layer that can
drop the connection instead of answering it.
"""
log.warning("%s from %s", what, client_address(conn))
def _spell(seconds: int) -> str:
if seconds < 60:
return f"{seconds} seconds"
return f"{round(seconds / 60)} minutes"
# --------------------------------------------- sockets whose credential ended
#: How often, in seconds, `sweep_ended_sockets` asks about every open socket:
#: the longest an expired session or token keeps the socket it opened.
SWEEP = 60.0
async def close_ended_sockets(app: Any) -> int:
"""Close every open socket whose session or token has ended. Returns how
many it closed.
The door is checked at the handshake, and a socket stays open for the
life of the tab. So signing out, changing the password or revoking a
session or a token left the socket it had opened streaming to whoever
held it. Run after each of those, as a background task of the route, and
on a clock by `sweep_ended_sockets`, this asks the store about every
socket's cookie or bearer header (`credential_live`) and closes the ones
it no longer knows with 1008; a tab's reconnect then meets the door.
The sockets are `app.state.bus`: what `pipeline.web.connect` puts there,
or anything with a `sockets` tuple and an async `close(sockets, code,
reason)`. Nothing to do on an unlocked app, which has no sessions to
end, or on one without a bus.
"""
if not getattr(app.state, "auth_locked", False):
return 0
bus = getattr(app.state, "bus", None)
if bus is None:
return 0
store, config = app.state.auth, app.state.auth_config
sockets = bus.sockets
def ended() -> list[Any]:
return [ws for ws in sockets if not credential_live(store, config, ws)]
# `to_thread`, because the store is SQLite and this is the event loop.
gone = await asyncio.to_thread(ended)
return await bus.close(gone, 1008, "signed out") if gone else 0
async def sweep_ended_sockets(app: Any, every: float = SWEEP) -> None:
"""`close_ended_sockets` every `every` seconds, until cancelled.
The door is checked at the handshake only, and the routes that end a
credential close its sockets themselves. Expiry is seen by no route: a
session or token that runs out ends no request, so its socket streamed
until the tab reconnected. This closes it within `every` seconds.
`connect` starts it on a locked app and cancels it at shutdown. A sweep
that fails is logged and the next one runs, since a loop that died on
one busy database would leave every later expiry open.
"""
while True:
await asyncio.sleep(every)
try:
await close_ended_sockets(app)
except Exception:
log.warning("the socket sweep failed", exc_info=True)
# ------------------------------------------------------------ another site
#: The methods that change something. No `GET` here changes anything (see the
#: module docstring of `auth/router.py`), so these are the ones to guard.
UNSAFE = frozenset({"POST", "PUT", "PATCH", "DELETE"})
#: The bodies another site's page may send without the browser asking first:
#: the three CORS-safelisted types, and a body with no type at all. A
#: cross-site request with any other type is preflighted, and nothing here
#: answers a preflight, so it never leaves the browser.
SIMPLE_BODIES = frozenset({"", "text/plain", "application/x-www-form-urlencoded",
"multipart/form-data"})
CROSS_SITE = ("This request came from another site's page, and nothing here "
"changes on another site's say-so.")
def cross_site(scope: Scope, configured_origin: str = "") -> tuple[int, str] | None:
"""Why a request that changes something is refused, or None.
Two checks, the first for the socket too:
- **Where it came from.** `Sec-Fetch-Site` when the browser sends it,
which a page cannot set; otherwise `Origin`, which must name this host
or `configured_origin` (`AuthConfig.origin`, for a proxy that rewrites
`Host`). No `Origin` at all is a client that is not a browser page
(`curl`, a script with a token, a test) and passes.
- **What the body is.** A body of a type another site could send without
a preflight is refused, whoever sent it: the API is JSON.
A read is never refused, wherever it came from.
"""
if scope["type"] == "websocket":
if not _from_here(scope, configured_origin):
return 403, "another site's page cannot open this socket"
return None
if scope.get("method") not in UNSAFE:
return None
if not _from_here(scope, configured_origin):
return 403, CROSS_SITE
kind = _header(scope, b"content-type").split(";")[0].strip().lower()
if _has_body(scope) and kind in SIMPLE_BODIES:
return 415, ("Send the body as application/json, not "
f"{kind or 'without a content type'}.")
return None
def _from_here(scope: Scope, configured: str) -> bool:
"""Whether the page that sent this is this app's own, as far as the
browser says.
`Sec-Fetch-Site` decides when present: the browser sets it and a page
cannot, and it stays right behind a proxy that rewrites `Host`, where an
`Origin` comparison would refuse the app's own page. `none` is a request
the user made by typing or from a bookmark. Without it, `Origin` against
`Host` (the Vite dev server forwards `Host` unchanged) or against the
configured origin. `null` is a sandboxed frame or a `file://` page and is
never this app.
"""
site = _header(scope, b"sec-fetch-site")
if site:
return site in ("same-origin", "none")
origin = _header(scope, b"origin")
if not origin:
return True
if origin == "null":
return False
if configured and origin.rstrip("/") == configured.rstrip("/"):
return True
parts = urlsplit(origin)
host = _header(scope, b"host")
return bool(host) and (_authority(parts.scheme, parts.netloc)
== _authority(parts.scheme, host))
def _authority(scheme: str, netloc: str) -> str:
"""`host[:port]`, lowercased, without the scheme's default port, so
`Host: app.example.com:443` and `Origin: https://app.example.com` agree."""
netloc = netloc.lower()
default = {"http": ":80", "ws": ":80", "https": ":443", "wss": ":443"}.get(scheme, "")
return netloc[: -len(default)] if default and netloc.endswith(default) else netloc
def _has_body(scope: Scope) -> bool:
length = _header(scope, b"content-length")
# The header's presence, not its value, says a body follows; an empty value
# is malformed and is not given the benefit of the doubt.
chunked = any(key == b"transfer-encoding" for key, _ in scope.get("headers", ()))
return (length.isdigit() and int(length) > 0) or chunked
+25 -2
View File
@@ -1,8 +1,8 @@
"""What the Account tab is told, and nothing more. """What the Account tab is told, and nothing more.
Three models, and the thing worth noticing about all three is what they cannot Four models, and the thing worth noticing about all four is what they cannot
carry. No token, no hash, no key material — not even the public half of a carry. No token, no hash, no key material — not even the public half of a
passkey. The list of ways into an app is the one page most worth reading and passkey; of an API token, the first eight characters and never the rest. The list of ways into an app is the one page most worth reading and
least worth leaking, so the shape it is rendered from is one that could not least worth leaking, so the shape it is rendered from is one that could not
leak a credential if it were logged whole. leak a credential if it were logged whole.
""" """
@@ -47,6 +47,27 @@ class Passkey(BaseModel):
transports: list[str] = Field(default_factory=list) transports: list[str] = Field(default_factory=list)
class ApiToken(BaseModel):
"""One token a script signs in with, as the account screen sees it.
The token itself is shown once, when it is made (`TokenCreated` in
`auth/schemas.py`), and is not here: this is the row, which is what the
revoke button sends back. `hint` is the token's opening, enough to match
the row to the environment variable that holds the whole.
"""
id: int
#: What you called it. Named for the machine or the job that holds it, so
#: the one to revoke can be told from the others.
label: str = ""
#: The token's first characters. See `auth.store.HINT`.
hint: str = ""
created_at: str = ""
last_used_at: str = ""
#: When it stops working, or "" for a token that does not expire.
expires_at: str = ""
class Account(BaseModel): class Account(BaseModel):
"""What the account screen knows. Deliberately almost nothing. """What the account screen knows. Deliberately almost nothing.
@@ -79,3 +100,5 @@ class Account(BaseModel):
#: honest common case is an insecure origin: WebAuthn refuses anything but #: honest common case is an insecure origin: WebAuthn refuses anything but
#: HTTPS, localhost excepted, so this reads "you need TLS first". #: HTTPS, localhost excepted, so this reads "you need TLS first".
passkeys_blocked: str = "" passkeys_blocked: str = ""
#: The API tokens, expired ones included, each without its secret.
tokens: list[ApiToken] = Field(default_factory=list)
+9 -1
View File
@@ -291,11 +291,19 @@ def login_verify(config: AuthConfig, request: Request, store: AuthStore,
) )
challenge = _challenge_from(credential) challenge = _challenge_from(credential)
# Read before the challenge is spent, and inside a `try`: outside one, a
# `rawId` that is not base64url was a 500 from an endpoint anyone can
# reach, after the challenge had already been used up.
try:
raw_id = base64url_to_bytes(credential.get("rawId", ""))
except (TypeError, ValueError) as exc:
raise PasskeyMalformed("That response was not a passkey: its rawId is "
"not base64url.") from exc
if not store.take_challenge(challenge, "login"): if not store.take_challenge(challenge, "login"):
raise PasskeyError("That sign-in took too long, or was already used. " raise PasskeyError("That sign-in took too long, or was already used. "
"Try again.") "Try again.")
known = store.passkey_for(base64url_to_bytes(credential.get("rawId", ""))) known = store.passkey_for(raw_id)
if known is None: if known is None:
# Deliberately the same sentence as a bad signature. There is one user, # Deliberately the same sentence as a bad signature. There is one user,
# so this leaks little, but "that key is not registered" is an oracle # so this leaks little, but "that key is not registered" is an oracle
+119 -83
View File
@@ -29,18 +29,20 @@ import os
from pathlib import Path from pathlib import Path
from urllib.parse import unquote, urlsplit from urllib.parse import unquote, urlsplit
from fastapi import APIRouter, HTTPException, Request, Response from fastapi import APIRouter, BackgroundTasks, HTTPException, Request, Response
from fastapi.responses import FileResponse, HTMLResponse, RedirectResponse from fastapi.responses import FileResponse, HTMLResponse, RedirectResponse
from starlette.requests import HTTPConnection from starlette.requests import HTTPConnection
from auth import passkeys as pk from auth import passkeys as pk
from auth.config import AuthConfig from auth.config import AuthConfig
from auth.middleware import ( from auth.middleware import (
client_address, cookie_name, cookie_names, is_secure, token_from, claim_attempt, close_ended_sockets, confirm_password, cookie_name, cookie_names,
is_secure, token_from, warn,
) )
from auth.models import Account from auth.models import Account
from auth.schemas import ( from auth.schemas import (
LabelBody, LoginBody, PasskeyBody, PasskeyLoginBody, PasswordBody, LabelBody, LoginBody, Ok, PasskeyAdded, PasskeyBody, PasskeyLoginBody,
PasskeyOptionsBody, PasswordBody, PasswordChanged, SignedOut, TokenBody, TokenCreated,
) )
from auth.store import AuthStore from auth.store import AuthStore
@@ -63,7 +65,7 @@ NO_STORE = {"Cache-Control": "no-store"}
#: The shortest new password the account screen will take. Low, deliberately: #: The shortest new password the account screen will take. Low, deliberately:
#: it is a floor against a typo, not a password policy. The honest advice is in #: it is a floor against a typo, not a password policy. The honest advice is in
#: docs/password-protection.md — the generated password is sixteen random #: documentation/Password Protection.md — the generated password is sixteen random
#: characters and anything you choose yourself is the weak half. #: characters and anything you choose yourself is the weak half.
MIN_LENGTH = 8 MIN_LENGTH = 8
@@ -156,19 +158,20 @@ def login_page(request: Request) -> Response:
def sign_in(request: Request, response: Response, body: LoginBody) -> dict[str, str]: def sign_in(request: Request, response: Response, body: LoginBody) -> dict[str, str]:
"""Check the password, and if it is right, open a session. """Check the password, and if it is right, open a session.
The lockout is read *before* the password is checked, so hammering the The attempt is claimed *before* the password is derived, which reads the
button while locked out does not extend the lockout — see lockout and counts the guess in one write, so hammering the button while
`AuthStore.locked_for`. A wrong password and an unknown one are the same locked out does not extend the lockout and concurrent guesses cannot all
answer here for the usual reason, which is thin cover with one account but slip in under one reading of it; see `AuthStore.claim_attempt`. A wrong
costs nothing to keep. password and an unknown one are the same answer here for the usual
reason, which is thin cover with one account but costs nothing to keep.
""" """
if not _locked(request): if not _locked(request):
raise HTTPException(400, "This app is not password-protected.") raise HTTPException(400, "This app is not password-protected.")
_refuse_while_locked(request) claim_attempt(request)
if not _store(request).check_password(body.password): if not _store(request).check_password(body.password, claimed=True):
_warn(request, "wrong password") warn(request, "wrong password")
raise HTTPException(401, "That is not the password.") raise HTTPException(401, "That is not the password.")
_open_session(request, response) _open_session(request, response)
@@ -176,17 +179,22 @@ def sign_in(request: Request, response: Response, body: LoginBody) -> dict[str,
@router.post("/logout", include_in_schema=False) @router.post("/logout", include_in_schema=False)
def sign_out(request: Request, response: Response) -> dict[str, bool]: def sign_out(request: Request, response: Response, tasks: BackgroundTasks) -> Ok:
"""End this session and clear the cookie. Other devices are untouched.""" """End this session and clear the cookie. Other devices are untouched.
The socket this tab opened is closed too, after the answer; see
`auth.close_ended_sockets`. The same for the four routes below that end
a session or a token."""
token = token_from(request, _config(request)) token = token_from(request, _config(request))
if token: if token:
_store(request).close_session(token) _store(request).close_session(token)
tasks.add_task(close_ended_sockets, request.app)
# Both names, because which one was set depends on `secure_cookie` and on # Both names, because which one was set depends on `secure_cookie` and on
# the scheme the login happened over — and clearing only one of them leaves # the scheme the login happened over — and clearing only one of them leaves
# a cookie the browser will keep offering. # a cookie the browser will keep offering.
for name in cookie_names(_config(request)): for name in cookie_names(_config(request)):
response.delete_cookie(name, path="/") response.delete_cookie(name, path="/")
return {"ok": True} return Ok()
# ---------------------------------------------------------------- the account # ---------------------------------------------------------------- the account
@@ -205,11 +213,13 @@ def account(request: Request) -> Account:
passkeys=store.passkeys(), passkeys=store.passkeys(),
passkeys_available=pk.available(), passkeys_available=pk.available(),
passkeys_blocked=pk.blocked(config, request), passkeys_blocked=pk.blocked(config, request),
tokens=store.tokens(),
) )
@router.post("/api/account/password") @router.post("/api/account/password")
def change_password(request: Request, body: PasswordBody) -> dict[str, int]: def change_password(request: Request, body: PasswordBody,
tasks: BackgroundTasks) -> PasswordChanged:
"""Change it, having proved you know the current one — or set the first. """Change it, having proved you know the current one — or set the first.
Asking for the current password matters even though you are already signed Asking for the current password matters even though you are already signed
@@ -223,6 +233,13 @@ def change_password(request: Request, body: PasswordBody) -> dict[str, int]:
the change ceremonial — and the cost when that is not the reason is signing the change ceremonial — and the cost when that is not the reason is signing
in again on your phone, which is the right way round. in again on your phone, which is the right way round.
API tokens and passkeys stay by default, because the cron job holding a
token is not what the password was changed against. `revoke_all` deletes
every one of them too, for the case the route exists for: an intruder
who had the password could have minted a token that never expires or
registered a passkey, and either outlives the change. The answer counts
what was closed and what remains, so the screen can say so.
Which is also why this is behind the same lockout as the front door, and not Which is also why this is behind the same lockout as the front door, and not
only behind a session. `current` is the password, guessed here rather than only behind a session. `current` is the password, guessed here rather than
at `/login`, and a borrowed unlocked laptop is exactly the situation the at `/login`, and a borrowed unlocked laptop is exactly the situation the
@@ -244,32 +261,82 @@ def change_password(request: Request, body: PasswordBody) -> dict[str, int]:
# app never lands here: `auth.bootstrap` gives it a password at startup, # app never lands here: `auth.bootstrap` gives it a password at startup,
# so `current` is always required for the app that has a door. # so `current` is always required for the app that has a door.
if store.has_password(): if store.has_password():
_refuse_while_locked(request) confirm_password(request, body.current)
if not store.check_password(body.current):
_warn(request, "wrong current password")
raise HTTPException(403, "That is not the current password.")
if len(body.new) < MIN_LENGTH: if len(body.new) < MIN_LENGTH:
raise HTTPException(400, f"Use at least {MIN_LENGTH} characters.") raise HTTPException(400, f"Use at least {MIN_LENGTH} characters.")
store.set_password(body.new) store.set_password(body.new)
closed = store.close_all(keep=token_from(request, config)) closed = store.close_all(keep=token_from(request, config))
return {"signed_out": closed} tokens, keys = store.revoke_credentials() if body.revoke_all else (0, 0)
tokens_left, keys_left = store.counts()
tasks.add_task(close_ended_sockets, request.app)
return PasswordChanged(signed_out=closed, tokens_revoked=tokens, passkeys_revoked=keys,
tokens_left=tokens_left, passkeys_left=keys_left)
@router.delete("/api/account/sessions/{session_id}") @router.delete("/api/account/sessions/{session_id}")
def revoke(session_id: int, request: Request) -> dict[str, bool]: def revoke(session_id: int, request: Request, tasks: BackgroundTasks) -> Ok:
"""Drop one session. Revoking your own is allowed — it is a logout with an """Drop one session. Revoking your own is allowed — it is a logout with an
extra click, and refusing it would be a rule to explain for no gain.""" extra click, and refusing it would be a rule to explain for no gain."""
if not _store(request).close_by_id(session_id): if not _store(request).close_by_id(session_id):
raise HTTPException(404, "That session is already gone.") raise HTTPException(404, "That session is already gone.")
return {"ok": True} tasks.add_task(close_ended_sockets, request.app)
return Ok()
@router.post("/api/account/sessions/close") @router.post("/api/account/sessions/close")
def close_others(request: Request) -> dict[str, int]: def close_others(request: Request, tasks: BackgroundTasks) -> SignedOut:
"""Sign out everywhere but here. The reason sessions are rows.""" """Sign out everywhere but here. The reason sessions are rows."""
closed = _store(request).close_all(keep=token_from(request, _config(request)))
tasks.add_task(close_ended_sockets, request.app)
return SignedOut(signed_out=closed)
# --------------------------------------------------------------------- tokens
#
# Three routes, all behind a session and none reachable with a token: the
# middleware answers a bearer request under `/api/account` with 403
# (`auth.middleware.ACCOUNT`), so a leaked token cannot mint another, revoke
# the owner's sessions or read the list of ways in. The token is used
# anywhere else as `Authorization: Bearer <token>`.
@router.post("/api/account/tokens")
def create_token(request: Request, body: TokenBody) -> TokenCreated:
"""Mint an API token for a script, once the password is confirmed.
A session is not enough, for the reason it is not enough for a passkey: a
token outlives every session and a password change, so a borrowed
signed-in browser that could mint one would keep a way in for good.
`current` is checked as `change_password` checks it, behind the same
lockout, and on the same condition. The token is in this answer and
nowhere else; the store keeps its hash.
"""
store = _store(request) store = _store(request)
return {"signed_out": store.close_all(keep=token_from(request, _config(request)))} if store.has_password():
confirm_password(request, body.current)
token_id, token = store.create_token(body.label, body.days)
made = next(t for t in store.tokens() if t.id == token_id)
return TokenCreated(id=token_id, token=token, hint=made.hint, label=made.label,
expires_at=made.expires_at)
@router.post("/api/account/tokens/{token_id}/label")
def rename_token(token_id: int, body: LabelBody, request: Request) -> Ok:
if not _store(request).rename_token(token_id, body.label):
raise HTTPException(404, "That token is already gone.")
return Ok()
@router.delete("/api/account/tokens/{token_id}")
def drop_token(token_id: int, request: Request, tasks: BackgroundTasks) -> Ok:
"""Revoke one. The next request carrying it is refused, whatever it holds
open: a token is looked up on every request, like a session, and a socket
it opened is closed as a session's is."""
if not _store(request).delete_token(token_id):
raise HTTPException(404, "That token is already gone.")
tasks.add_task(close_ended_sockets, request.app)
return Ok()
# ------------------------------------------------------------------- passkeys # ------------------------------------------------------------------- passkeys
@@ -281,24 +348,37 @@ def close_others(request: Request) -> dict[str, int]:
@router.post("/api/account/passkeys/options") @router.post("/api/account/passkeys/options")
def passkey_options(request: Request) -> dict[str, object]: def passkey_options(request: Request, body: PasskeyOptionsBody) -> dict[str, object]:
"""Step one of registering: a challenge for `navigator.credentials.create`.""" """Step one of registering: a challenge for `navigator.credentials.create`,
once the password is confirmed.
A session is not enough. A passkey outlives every session and a password
change, so a borrowed signed-in browser that could add one would keep a way
in for good. `current` is checked as `change_password` checks it, behind the
same lockout, and on the same condition: an app with no password (loopback,
never locked) has nothing to confirm and no session to borrow, so
registering there before deploying still works.
"""
_require_passkeys(request) _require_passkeys(request)
return pk.register_options(_config(request), request, _store(request)) store = _store(request)
if store.has_password():
confirm_password(request, body.current)
return pk.register_options(_config(request), request, store)
@router.post("/api/account/passkeys") @router.post("/api/account/passkeys")
def add_passkey(request: Request, body: PasskeyBody) -> dict[str, int]: def add_passkey(request: Request, body: PasskeyBody) -> PasskeyAdded:
"""Step two: check what the authenticator signed, and keep the public key. """Step two: check what the authenticator signed, and keep the public key.
Requires a session, like everything else under `/api/account`. Registering a Requires a session, like everything else under `/api/account`, and a
passkey is granting a new way in, so it is gated on already being in — the challenge from `passkey_options`, which only the password gets. The
password is what proves that, and the ceremony does not replace it. challenge is single-use and bound to registering, so this step cannot be
reached without the password having been typed for it.
""" """
_require_passkeys(request) _require_passkeys(request)
try: try:
return {"id": pk.register_verify(_config(request), request, _store(request), return PasskeyAdded(id=pk.register_verify(_config(request), request, _store(request),
body.credential, body.label)} body.credential, body.label))
except pk.PasskeyError as exc: except pk.PasskeyError as exc:
# `PasskeyMalformed` is one of these and answers the same way: from # `PasskeyMalformed` is one of these and answers the same way: from
# behind a session, a body that is not a ceremony and a ceremony that # behind a session, a body that is not a ceremony and a ceremony that
@@ -307,20 +387,20 @@ def add_passkey(request: Request, body: PasskeyBody) -> dict[str, int]:
@router.post("/api/account/passkeys/{passkey_id}/label") @router.post("/api/account/passkeys/{passkey_id}/label")
def rename_passkey(passkey_id: int, body: LabelBody, request: Request) -> dict[str, bool]: def rename_passkey(passkey_id: int, body: LabelBody, request: Request) -> Ok:
"""Name it. A key you cannot name is one you will never dare revoke.""" """Name it. A key you cannot name is one you will never dare revoke."""
if not _store(request).rename_passkey(passkey_id, body.label): if not _store(request).rename_passkey(passkey_id, body.label):
raise HTTPException(404, "That passkey is already gone.") raise HTTPException(404, "That passkey is already gone.")
return {"ok": True} return Ok()
@router.delete("/api/account/passkeys/{passkey_id}") @router.delete("/api/account/passkeys/{passkey_id}")
def drop_passkey(passkey_id: int, request: Request) -> dict[str, bool]: def drop_passkey(passkey_id: int, request: Request) -> Ok:
"""Revoke one. The password is still there, so this cannot lock you out — """Revoke one. The password is still there, so this cannot lock you out —
which is exactly why the password is still there.""" which is exactly why the password is still there."""
if not _store(request).delete_passkey(passkey_id): if not _store(request).delete_passkey(passkey_id):
raise HTTPException(404, "That passkey is already gone.") raise HTTPException(404, "That passkey is already gone.")
return {"ok": True} return Ok()
@router.post("/login/passkey/options", include_in_schema=False) @router.post("/login/passkey/options", include_in_schema=False)
@@ -363,7 +443,7 @@ def passkey_login(request: Request, response: Response,
# been wrong about. # been wrong about.
raise HTTPException(400, str(exc)) from None raise HTTPException(400, str(exc)) from None
except pk.PasskeyError as exc: except pk.PasskeyError as exc:
_warn(request, "passkey refused") warn(request, "passkey refused")
raise HTTPException(401, str(exc)) from None raise HTTPException(401, str(exc)) from None
_open_session(request, response) _open_session(request, response)
@@ -406,44 +486,6 @@ def _open_session(request: Request, response: Response) -> None:
) )
def _refuse_while_locked(request: Request) -> None:
"""The lockout, in front of every place a password can be guessed.
Read *before* the guess is looked at, so hammering while locked out does not
extend the lockout — see `AuthStore.locked_for`. Called from both doors
rather than living inside `check_password`, because the store counts and
this decides: the same wrong answer is a refusal here and a bootstrap
comparison in `auth.bootstrap`, which must not be able to lock anyone out of
their own restart.
"""
wait = _store(request).locked_for()
if wait:
_warn(request, "attempt while locked out")
raise HTTPException(429, f"Too many attempts. Try again in {_spell(wait)}.",
headers={"Retry-After": str(wait)})
def _warn(request: Request, what: str) -> None:
"""Write down that someone got it wrong, and who.
The whole record of who is knocking. There is no access log worth the name —
uvicorn is run quietly — and no table of attempts, because the log is the
record and a second copy of it would only ever disagree. WARNING so it
survives that level, one line per refusal, and the address first after
`from` so a fail2ban filter is one regex; see docs/password-protection.md.
The message does not name the package, because the logger already does:
with the usual `%(name)s: %(message)s` this module is `auth.router`, and a
line reading `auth.router: auth: wrong password` says it twice and reads
like a bug in the logging.
Nothing is *decided* from the address — `AuthStore.check_password` says why
it cannot be — so this is where it earns its keep: at the layer that can
drop the connection instead of answering it.
"""
log.warning("%s from %s", what, client_address(request))
def _safe_next(target: str) -> str: def _safe_next(target: str) -> str:
r"""Where to go after signing in — only ever inside this app. r"""Where to go after signing in — only ever inside this app.
@@ -481,9 +523,3 @@ def _local(target: str) -> bool:
except ValueError: # an unparseable authority, e.g. `//[` except ValueError: # an unparseable authority, e.g. `//[`
return False return False
return not parts.scheme and not parts.netloc return not parts.scheme and not parts.netloc
def _spell(seconds: int) -> str:
if seconds < 60:
return f"{seconds} seconds"
return f"{round(seconds / 60)} minutes"
+84 -2
View File
@@ -1,10 +1,15 @@
"""The five bodies this package accepts. Small on purpose. """The seven bodies this package accepts, and the five small answers it gives.
Small on purpose.
A ceremony's `credential` is passed through as an opaque dict — it is the A ceremony's `credential` is passed through as an opaque dict — it is the
browser's `PublicKeyCredential` serialised by `toJSON()`, and its shape is the browser's `PublicKeyCredential` serialised by `toJSON()`, and its shape is the
WebAuthn spec's business rather than ours. Restating it as a model here would WebAuthn spec's business rather than ours. Restating it as a model here would
be a second, staler copy of a structure the library already parses and be a second, staler copy of a structure the library already parses and
validates properly. validates properly.
The answers are models rather than dicts so that the app's OpenAPI document,
which a client generates its types from, names every one of them. The JSON is
what the dicts were.
""" """
from __future__ import annotations from __future__ import annotations
@@ -34,6 +39,22 @@ class PasswordBody(BaseModel):
current: str = Field(max_length=1000) current: str = Field(max_length=1000)
new: str = Field(max_length=1000) new: str = Field(max_length=1000)
#: Also delete every API token and passkey. Off by default, because a
#: token's cron job is not what a routine change is against; on when the
#: password may have been someone else's, since what they minted with it
#: outlives the change. None is off, and is the default rather than False
#: so the generated client type leaves the field optional, as it does
#: `TokenBody.days`: openapi-typescript makes a field with a non-null
#: default required, and a client that never asks should not have to say so.
revoke_all: bool | None = None
class PasskeyOptionsBody(BaseModel):
"""Starting to register a passkey. `current` is the password, required for
the same reason as in `PasswordBody`: a passkey is a permanent way in, and
a borrowed session must not be able to mint one."""
current: str = Field(max_length=1000)
class PasskeyBody(BaseModel): class PasskeyBody(BaseModel):
@@ -51,6 +72,67 @@ class PasskeyLoginBody(BaseModel):
class LabelBody(BaseModel): class LabelBody(BaseModel):
"""Renaming a passkey.""" """Renaming a passkey or a token."""
label: str = Field(max_length=100)
class TokenBody(BaseModel):
"""Minting an API token. `current` is the password, required for the same
reason as in `PasskeyOptionsBody`: a token is a standing way in, and a
borrowed session must not be able to mint one."""
current: str = Field(max_length=1000)
#: What to call it: the machine or the job that will hold it.
label: str = Field("", max_length=100)
#: How long it lives, in days. None is a token that does not expire, which
#: is what a cron job wants; bounded above so a typo cannot ask for a
#: thousand years and still be called dated.
days: int | None = Field(None, ge=1, le=3650)
class Ok(BaseModel):
"""A change that has nothing to report beyond having happened."""
ok: bool = True
class SignedOut(BaseModel):
"""How many other sessions "sign out everywhere else" closed."""
signed_out: int
class PasswordChanged(BaseModel):
"""What a password change closed and what it left, for the screen to say.
`signed_out` is the other sessions, always closed. The tokens and
passkeys are deleted only when `revoke_all` was asked for, and the
`_left` counts are what is still registered afterwards, expired tokens
included, so an owner who did not ask knows what to review."""
signed_out: int
tokens_revoked: int
passkeys_revoked: int
tokens_left: int
passkeys_left: int
class PasskeyAdded(BaseModel):
"""The row a registered passkey was given, which rename and revoke take."""
id: int
class TokenCreated(BaseModel):
"""A minted token: the one answer in this package that carries a credential.
`token` is shown once and stored nowhere but as a hash, so a caller that
loses it mints another. The rest is the row as `ApiToken` lists it.
"""
id: int
token: str
hint: str
label: str label: str
expires_at: str
+209 -12
View File
@@ -4,9 +4,9 @@
* <auth-account></auth-account> * <auth-account></auth-account>
* *
* That is the whole of it. The element asks `/api/account` what there is to * That is the whole of it. The element asks `/api/account` what there is to
* know, and draws the four things there are to do about it: change the * know, and draws the five things there are to do about it: change the
* password, look at what is signed in and revoke any of it, register and * password, look at what is signed in and revoke any of it, register and
* revoke passkeys, and sign out. * revoke passkeys, make and revoke API tokens for scripts, and sign out.
* *
* One file, and the styles are in it rather than beside it, because the shadow * One file, and the styles are in it rather than beside it, because the shadow
* root scopes them anyway and a second file is a second route, a second cache * root scopes them anyway and a second file is a second route, a second cache
@@ -79,6 +79,18 @@ button {
padding: 2px 9px; padding: 2px 9px;
cursor: pointer; cursor: pointer;
} }
select {
font: inherit; color: inherit; background: transparent;
border: 1px solid color-mix(in srgb, var(--desk-jobs-fg) 25%, transparent);
border-radius: 6px; padding: 3px 7px;
}
select option { color: initial; }
label.check { display: flex; align-items: center; gap: 6px; flex: 1 1 100%; }
label.check input { min-width: 0; margin: 0; }
input.token { font-family: ui-monospace, "Cascadia Mono", Consolas, monospace; min-width: 20rem; }
.made { display: flex; flex-direction: column; gap: var(--desk-jobs-gap); }
.made .bar { display: flex; flex-wrap: wrap; gap: var(--desk-jobs-gap); align-items: center; }
button:hover { color: var(--desk-jobs-accent); } button:hover { color: var(--desk-jobs-accent); }
button.danger:hover { color: var(--desk-jobs-error); } button.danger:hover { color: var(--desk-jobs-error); }
button[disabled] { opacity: .4; cursor: default; } button[disabled] { opacity: .4; cursor: default; }
@@ -87,6 +99,25 @@ button[disabled] { opacity: .4; cursor: default; }
font-size: 12px; padding: 0 6px; border-radius: 999px; font-size: 12px; padding: 0 6px; border-radius: 999px;
border: 1px solid currentColor; color: var(--desk-jobs-dim); border: 1px solid currentColor; color: var(--desk-jobs-dim);
} }
/* A finger: every button a thumb's height, and no field under 16px, which
* is where iOS zooms the page in on focus and never back out. Sizes only;
* the layout is the width query below, and neither reads the other. */
@media (pointer: coarse) {
button { min-height: 2.5rem; padding: 4px 12px; }
input, select { font-size: 16px; padding: 7px 9px; }
}
/* A phone: a session's name and its buttons share a line while they fit and
* the buttons drop under it when they do not, still on the right; a form's
* fields take the width and its buttons share what is left. 40rem is
* app.css's number, on purpose. */
@media (max-width: 40rem) {
.row { flex-wrap: wrap; align-items: center; }
.row .acts { margin-left: auto; }
form input:not([type="checkbox"]), form select, .bar input { flex: 1 1 100%; min-width: 0; }
form button, .bar button { flex: 1 1 auto; }
}
`; `;
/** What the server says when this build has no `webauthn` installed. The /** What the server says when this build has no `webauthn` installed. The
@@ -110,6 +141,11 @@ class AuthAccount extends HTMLElement {
* the other way -- from an app that was restarted without the extra while * the other way -- from an app that was restarted without the extra while
* this page was open. */ * this page was open. */
#passkeysGone = false; #passkeysGone = false;
/** The token just made, held until Done: the server shows it once and keeps
* only its hash, so nothing reloaded from `/api/account` can bring it back. */
#made = null;
/** What Copy last did: "", "done" or "failed". */
#copy = "";
constructor() { constructor() {
super(); super();
@@ -211,6 +247,7 @@ class AuthAccount extends HTMLElement {
this.#panel.append(this.#password(account)); this.#panel.append(this.#password(account));
if (account.sessions.length) this.#panel.append(this.#sessions(account)); if (account.sessions.length) this.#panel.append(this.#sessions(account));
if (this.#showPasskeys(account)) this.#panel.append(this.#passkeys(account)); if (this.#showPasskeys(account)) this.#panel.append(this.#passkeys(account));
this.#panel.append(this.#tokens(account));
if (account.locked) this.#panel.append(this.#signOut()); if (account.locked) this.#panel.append(this.#signOut());
} }
@@ -255,22 +292,30 @@ class AuthAccount extends HTMLElement {
const form = document.createElement("form"); const form = document.createElement("form");
const current = field("password", "Current password"); const current = field("password", "Current password");
const next = field("password", first ? "Choose a password" : "New password"); const next = field("password", first ? "Choose a password" : "New password");
// Tokens and passkeys survive a change unless this is ticked: the cron
// job holding a token is not what a routine change is against, but an
// intruder who had the password may have minted one.
const revoke = document.createElement("input");
revoke.type = "checkbox";
const revokeLabel = document.createElement("label");
revokeLabel.className = "check";
revokeLabel.append(revoke, "Also revoke every API token and passkey");
const go = document.createElement("button"); const go = document.createElement("button");
go.textContent = first ? "Set password" : "Change password"; go.textContent = first ? "Set password" : "Change password";
go.disabled = this.#busy; go.disabled = this.#busy;
form.append(...(first ? [next, go] : [current, next, go])); form.append(...(first ? [next, go] : [current, next, revokeLabel, go]));
form.addEventListener("submit", (event) => { form.addEventListener("submit", (event) => {
event.preventDefault(); event.preventDefault();
this.#act(async () => { this.#act(async () => {
const result = await this.#api("/api/account/password", { const result = await this.#api("/api/account/password", {
method: "POST", method: "POST",
body: JSON.stringify({ current: current.value, new: next.value }), body: JSON.stringify({
current: current.value, new: next.value, revoke_all: !first && revoke.checked,
}),
}); });
current.value = next.value = ""; current.value = next.value = "";
if (first) return this.#say("Password set."); if (first) return this.#say("Password set.");
this.#say(result.signed_out this.#say(changed(result));
? `Password changed. ${count(result.signed_out, "other session")} signed out.`
: "Password changed.");
}); });
}); });
section.append(form); section.append(form);
@@ -279,7 +324,7 @@ class AuthAccount extends HTMLElement {
// makes the change ceremonial. Said here so it is not a surprise. // makes the change ceremonial. Said here so it is not a surprise.
section.append(this.#line(first section.append(this.#line(first
? "Nothing asks for it until the app binds an address that is not loopback -- or `auth.enabled: always`. Set it before you deploy, not after." ? "Nothing asks for it until the app binds an address that is not loopback -- or `auth.enabled: always`. Set it before you deploy, not after."
: "Changing it signs out every other session, and keeps this one.")); : "Changing it signs out every other session and keeps this one. API tokens and passkeys stay unless you tick the box; tick it if someone else may have had the password."));
return section; return section;
} }
@@ -344,13 +389,142 @@ class AuthAccount extends HTMLElement {
section.append(this.#line(account.passkeys_blocked)); section.append(this.#line(account.passkeys_blocked));
return section; return section;
} }
const add = this.#button("Add a passkey", "", () => this.#register()); // A passkey outlasts every session, so adding one takes the password --
// see `passkey_options` in router.py. With none set there is nothing to
// confirm, and the server does not ask.
const asks = Boolean(account.password_set_at);
const form = document.createElement("form");
const current = field("password", "Password, to add a passkey");
current.autocomplete = "current-password";
const add = document.createElement("button");
add.textContent = "Add a passkey";
add.disabled = this.#busy; add.disabled = this.#busy;
section.append(add); form.append(...(asks ? [current, add] : [add]));
form.addEventListener("submit", (event) => {
event.preventDefault();
this.#register(current.value);
});
section.append(form);
return section; return section;
} }
async #register() { #tokens(account) {
const section = this.#section("API tokens");
for (const token of account.tokens || []) {
const row = this.#row(
token.label || "Token",
[`${token.hint}…`,
`added ${when(token.created_at)}`,
token.last_used_at ? `last used ${when(token.last_used_at)}` : "never used",
lifetime(token),
].join(" · "));
const acts = row.querySelector(".acts");
acts.append(this.#button("Rename", "", () => {
const label = prompt("Call this token what?", token.label || "");
if (label === null) return;
this.#act(async () => {
await this.#api(`/api/account/tokens/${token.id}/label`, {
method: "POST", body: JSON.stringify({ label }),
});
this.#say("Renamed.");
});
}));
acts.append(this.#button("Revoke", "danger", () => this.#act(async () => {
await this.#api(`/api/account/tokens/${token.id}`, { method: "DELETE" });
// The script holding it is locked out on its next request, and
// nothing else changes: the password and the sessions stay.
this.#say("Token revoked.");
})));
section.append(row);
}
// A token outlives every session and a password change, so making one
// takes the password, on the same condition as a passkey -- see
// `create_token` in router.py.
const asks = Boolean(account.password_set_at);
const form = document.createElement("form");
const label = field("text", "Name, e.g. the script that holds it");
label.required = false;
label.maxLength = 100;
const days = document.createElement("select");
days.setAttribute("aria-label", "How long it is valid");
for (const [value, text] of [["", "Does not expire"], ["30", "30 days"],
["90", "90 days"], ["365", "1 year"]]) {
const option = document.createElement("option");
option.value = value;
option.textContent = text;
days.append(option);
}
const current = field("password", "Password, to create a token");
current.autocomplete = "current-password";
const create = document.createElement("button");
create.textContent = "Create a token";
create.disabled = this.#busy;
form.append(label, days, ...(asks ? [current] : []), create);
form.addEventListener("submit", (event) => {
event.preventDefault();
this.#act(async () => {
const made = await this.#api("/api/account/tokens", {
method: "POST",
body: JSON.stringify({
current: current.value, label: label.value.trim(),
days: days.value ? Number(days.value) : null,
}),
});
// In the one answer that made it and nowhere after, so it is kept on
// the element until Done rather than read back from the account.
this.#made = made;
this.#copy = "";
this.#say(`Token "${made.label || "Token"}" created. Copy it below; it is shown only once.`);
});
});
section.append(form);
if (this.#made) section.append(this.#shown(this.#made));
section.append(this.#line(
"A script sends a token as `Authorization: Bearer`. It cannot manage the account, and it is kept through a password change. Revoke it here when the script no longer needs it."));
return section;
}
/** The new token, read-only and selectable, with Copy and a sentence that
* it is the only time. Done drops it from the element, after which the
* page never has it again. */
#shown(made) {
const box = document.createElement("div");
box.className = "made";
const bar = document.createElement("div");
bar.className = "bar";
const value = document.createElement("input");
value.type = "text";
value.readOnly = true;
value.className = "token";
value.value = made.token;
value.setAttribute("aria-label", "The new token");
value.addEventListener("focus", () => value.select());
bar.append(value,
this.#button("Copy", "", async () => {
try {
await navigator.clipboard.writeText(made.token);
this.#copy = "done";
} catch {
// No clipboard on an insecure origin, or the browser refused: the
// field is selectable, so the token is still there to take by hand.
this.#copy = "failed";
}
this.#render();
}),
this.#button("Done", "", () => {
this.#made = null;
this.#copy = "";
this.#render();
}));
box.append(bar, this.#line(
"This is the only time the token is shown. Only a hash is stored, so it cannot be looked up again; if it is lost, revoke it and create another."));
if (this.#copy === "done") box.append(this.#line("Copied."));
if (this.#copy === "failed") box.append(this.#note("Could not copy. Select the token and copy it by hand.", true));
return box;
}
async #register(current) {
const passkey = await loadPasskeyHelper(); const passkey = await loadPasskeyHelper();
if (!passkey || !passkey.supported()) { if (!passkey || !passkey.supported()) {
this.#say("This browser will not create a passkey here. It needs HTTPS, or localhost."); this.#say("This browser will not create a passkey here. It needs HTTPS, or localhost.");
@@ -361,7 +535,9 @@ class AuthAccount extends HTMLElement {
// browser's own prompt happens between them, and a cancelled prompt is an // browser's own prompt happens between them, and a cancelled prompt is an
// exception rather than an error status -- hence `explain`. // exception rather than an error status -- hence `explain`.
await this.#act(async () => { await this.#act(async () => {
const options = await this.#api("/api/account/passkeys/options", { method: "POST" }); const options = await this.#api("/api/account/passkeys/options", {
method: "POST", body: JSON.stringify({ current }),
});
let made; let made;
try { try {
made = await passkey.create(options); made = await passkey.create(options);
@@ -485,6 +661,19 @@ function field(type, placeholder) {
return input; return input;
} }
/** What a password change closed and what it left, in a sentence or three. */
function changed(result) {
const said = ["Password changed."];
if (result.signed_out) said.push(`${count(result.signed_out, "other session")} signed out.`);
if (result.tokens_revoked || result.passkeys_revoked) {
said.push(`${count(result.tokens_revoked, "API token")} and ${count(result.passkeys_revoked, "passkey")} revoked.`);
}
if (result.tokens_left || result.passkeys_left) {
said.push(`${count(result.tokens_left, "API token")} and ${count(result.passkeys_left, "passkey")} remain; revoke any you do not recognise.`);
}
return said.join(" ");
}
function count(n, thing) { function count(n, thing) {
return `${n} ${thing}${n === 1 ? "" : "s"}`; return `${n} ${thing}${n === 1 ? "" : "s"}`;
} }
@@ -498,6 +687,14 @@ function when(stamp) {
return at.toLocaleString(); return at.toLocaleString();
} }
/** What a token's row says about its end: no end, one to come, or one that
* has passed and left the row for the owner to clear. */
function lifetime(token) {
if (!token.expires_at) return "does not expire";
const gone = new Date(token.expires_at).getTime() <= Date.now();
return `${gone ? "expired" : "expires"} ${when(token.expires_at)}`;
}
if (!customElements.get("auth-account")) { if (!customElements.get("auth-account")) {
customElements.define("auth-account", AuthAccount); customElements.define("auth-account", AuthAccount);
} }
+207 -26
View File
@@ -1,6 +1,7 @@
"""One password, and the sessions it opens. Nothing else knows a hash. """One password, the sessions it opens, and the tokens that stand in for it
from a script. Nothing else knows a hash.
Four tables in whatever SQLite file it is given — the app's own, normally, so Five tables in whatever SQLite file it is given — the app's own, normally, so
there is one file to back up and SQLite's writer lock keeps auth honest with there is one file to back up and SQLite's writer lock keeps auth honest with
everything else in it. Every name starts `auth_`, because the file is shared: everything else in it. Every name starts `auth_`, because the file is shared:
a `sessions` table is a name an app is entitled to want for itself. a `sessions` table is a name an app is entitled to want for itself.
@@ -54,7 +55,7 @@ from contextlib import closing
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from pathlib import Path from pathlib import Path
from auth.models import Passkey, SessionInfo from auth.models import ApiToken, Passkey, SessionInfo
#: scrypt cost. `n` is the work factor, and 2**14 with `r=8` is the widely #: scrypt cost. `n` is the work factor, and 2**14 with `r=8` is the widely
#: recommended interactive setting: about 16 MB and ~50 ms per guess, which is #: recommended interactive setting: about 16 MB and ~50 ms per guess, which is
@@ -86,6 +87,12 @@ _MAX_LOCK = timedelta(minutes=15)
#: the ceiling. See `_lockout` for what goes wrong without it. #: the ceiling. See `_lockout` for what goes wrong without it.
_MAX_STEPS = math.ceil(math.log2(_MAX_LOCK / _FIRST_LOCK)) _MAX_STEPS = math.ceil(math.log2(_MAX_LOCK / _FIRST_LOCK))
#: How much of a token the account screen is shown, so the owner can tell
#: which environment variable holds which row. Eight of forty-three characters
#: from `token_urlsafe(32)` leaves 210 bits unknown, which is still past
#: anybody's ability to guess.
HINT = 8
#: How stale `last_seen_at` may get before a request bothers to update it. #: How stale `last_seen_at` may get before a request bothers to update it.
#: Without this, every single request — every socket frame's sibling fetch, #: Without this, every single request — every socket frame's sibling fetch,
#: every poll of a page left open — takes SQLite's write lock to move a #: every poll of a page left open — takes SQLite's write lock to move a
@@ -96,7 +103,7 @@ _SEEN_EVERY = timedelta(minutes=5)
#: The tables. Created on construction rather than by a migration step, because #: The tables. Created on construction rather than by a migration step, because
#: the only schema change this has ever had is "it did not exist yet", and a #: the only schema change this has ever had is "it did not exist yet", and a
#: migration runner for four `CREATE TABLE IF NOT EXISTS` is a thing to #: migration runner for five `CREATE TABLE IF NOT EXISTS` is a thing to
#: maintain in exchange for nothing. #: maintain in exchange for nothing.
SCHEMA = """ SCHEMA = """
CREATE TABLE IF NOT EXISTS auth_credential ( CREATE TABLE IF NOT EXISTS auth_credential (
@@ -130,6 +137,15 @@ CREATE TABLE IF NOT EXISTS auth_challenges (
purpose TEXT NOT NULL, purpose TEXT NOT NULL,
expires_at TEXT NOT NULL expires_at TEXT NOT NULL
); );
CREATE TABLE IF NOT EXISTS auth_tokens (
id INTEGER PRIMARY KEY,
token_hash TEXT NOT NULL UNIQUE,
hint TEXT NOT NULL DEFAULT '',
label TEXT NOT NULL DEFAULT '',
created_at TEXT NOT NULL,
last_used_at TEXT NOT NULL DEFAULT '',
expires_at TEXT NOT NULL DEFAULT ''
);
""" """
@@ -151,7 +167,7 @@ def _derive(password: str, salt: bytes) -> bytes:
class AuthStore: class AuthStore:
"""The four `auth_*` tables, opened on the path it is handed. """The five `auth_*` tables, opened on the path it is handed.
store = AuthStore("data/app.db") store = AuthStore("data/app.db")
@@ -207,13 +223,8 @@ class AuthStore:
) )
def locked_for(self) -> int: def locked_for(self) -> int:
"""Seconds until the next attempt will be looked at, or 0. """Seconds until the next attempt will be looked at, or 0. Reads only;
the doors ask `claim_attempt`, which decides and counts in one step.
Read before checking rather than after, so a locked-out caller is told
to wait instead of having a guess counted — otherwise a fast enough
attacker extends their own lockout forever and never notices, and, more
to the point, an honest person hammering the button digs a deeper hole
with every press.
""" """
with closing(self._connect()) as conn: with closing(self._connect()) as conn:
row = conn.execute( row = conn.execute(
@@ -223,9 +234,61 @@ class AuthStore:
remaining = datetime.fromisoformat(row["locked_until"]) - _now() remaining = datetime.fromisoformat(row["locked_until"]) - _now()
return max(0, int(remaining.total_seconds() + 0.999)) return max(0, int(remaining.total_seconds() + 0.999))
def check_password(self, password: str) -> bool: def claim_attempt(self) -> int:
"""Claim one guess at the password before it is derived: 0 when the
guess may go ahead, else the seconds until one may.
Inside one `BEGIN IMMEDIATE`, `locked_until` is read and, when it has
passed, the attempt is counted as a failure up front and the lockout
it would earn is written; `check_password(..., claimed=True)` then
clears both when the guess was right. Read before scrypt and written
after it, thirty wrong guesses in parallel each found the door open
and each got a 401, so a window admitted a threadpool of guesses
rather than `_GRACE`. Now the sixth concurrent claim finds the lockout
the fifth wrote.
A refused claim writes nothing, so a caller locked out is told to wait
instead of having a guess counted: otherwise a fast enough attacker
extends their own lockout forever and never notices, and an honest
person hammering the button digs a deeper hole with every press. On a
database with no password there is nothing to count and nothing to
lock, and the claim is granted.
"""
with closing(self._connect()) as conn:
conn.execute("BEGIN IMMEDIATE")
try:
row = conn.execute(
"SELECT failures, locked_until FROM auth_credential WHERE id=1"
).fetchone()
wait = 0
if row is not None:
now = _now()
if row["locked_until"]:
remaining = datetime.fromisoformat(row["locked_until"]) - now
wait = max(0, int(remaining.total_seconds() + 0.999))
if not wait:
failures = int(row["failures"]) + 1
conn.execute(
"UPDATE auth_credential SET failures=?, locked_until=? WHERE id=1",
(failures, _stamp(now + _lockout(failures))
if failures >= _GRACE else ""))
conn.commit()
except BaseException:
conn.rollback()
raise
return wait
def check_password(self, password: str, *, claimed: bool = False) -> bool:
"""Right or wrong, and record which. """Right or wrong, and record which.
`claimed` says the caller counted this guess with `claim_attempt`
already, which is what both doors do: a wrong answer then writes
nothing more and a right one clears the count. Without it a wrong
answer is counted here, after the derivation, which is what a caller
that is not a door wants: `auth.bootstrap` compares the pinned
password, and its comparison cannot lock anyone out because it is
followed by `set_password`, which clears the count.
`hmac.compare_digest` rather than `==`, so the comparison takes the same `hmac.compare_digest` rather than `==`, so the comparison takes the same
time whatever the first wrong byte is. That is close to superstition at time whatever the first wrong byte is. That is close to superstition at
this scale — the network jitter in front of it is larger than the signal this scale — the network jitter in front of it is larger than the signal
@@ -247,29 +310,42 @@ class AuthStore:
owner still gets locked out by their own typos: strictly worse than owner still gets locked out by their own typos: strictly worse than
this, and it would look like protection. So the address is written to this, and it would look like protection. So the address is written to
the log instead, at WARNING, where fail2ban can act on it at the layer the log instead, at WARNING, where fail2ban can act on it at the layer
that actually knows who connected — see `_warn` in `auth/router.py`. that actually knows who connected — see `warn` in `auth/middleware.py`.
What is left for this counter to do is make guessing pointless, which it What is left for this counter to do is make guessing pointless, which it
does whoever is guessing. does whoever is guessing.
""" """
with closing(self._connect()) as conn: with closing(self._connect()) as conn:
row = conn.execute( # `fetchall`, so no read is left open on this connection when it
"SELECT salt, hash, failures FROM auth_credential WHERE id=1" # asks for the write lock below.
).fetchone() rows = conn.execute(
if row is None: "SELECT salt, hash FROM auth_credential WHERE id=1").fetchall()
if not rows:
_derive(password, b"no password here") _derive(password, b"no password here")
return False return False
ok = hmac.compare_digest(_derive(password, row["salt"]), row["hash"]) ok = hmac.compare_digest(_derive(password, rows[0]["salt"]), rows[0]["hash"])
with conn: if claimed and not ok:
return False
# The count is raised in the database under the write lock, and
# the lockout is decided from what that raise returned. Read
# before scrypt and written after it, forty guesses in parallel
# each wrote "one more than before" and recorded one failure.
conn.execute("BEGIN IMMEDIATE")
try:
if ok: if ok:
conn.execute( conn.execute(
"UPDATE auth_credential SET failures=0, locked_until='' WHERE id=1") "UPDATE auth_credential SET failures=0, locked_until='' WHERE id=1")
else: elif counted := conn.execute(
failures = int(row["failures"]) + 1 "UPDATE auth_credential SET failures = failures + 1 WHERE id=1 "
"RETURNING failures").fetchall():
failures = int(counted[0][0])
conn.execute( conn.execute(
"UPDATE auth_credential SET failures=?, locked_until=? WHERE id=1", "UPDATE auth_credential SET locked_until=? WHERE id=1",
(failures, _stamp(_now() + _lockout(failures)) (_stamp(_now() + _lockout(failures))
if failures >= _GRACE else ""), if failures >= _GRACE else "",))
) conn.commit()
except BaseException:
conn.rollback()
raise
return ok return ok
def changed_at(self) -> str: def changed_at(self) -> str:
@@ -368,6 +444,104 @@ class AuthStore:
cur = conn.execute("DELETE FROM auth_sessions") cur = conn.execute("DELETE FROM auth_sessions")
return cur.rowcount return cur.rowcount
def revoke_credentials(self) -> tuple[int, int]:
"""Delete every API token and every passkey in one transaction, for a
password change made because the account may be in someone else's
hands. Returns `(tokens, passkeys)` deleted. The password stays, so
this cannot lock the owner out."""
with closing(self._connect()) as conn, conn:
tokens = conn.execute("DELETE FROM auth_tokens").rowcount
keys = conn.execute("DELETE FROM auth_passkeys").rowcount
return tokens, keys
def counts(self) -> tuple[int, int]:
"""`(tokens, passkeys)` registered, expired tokens included, for a
caller that reports what is left after a password change."""
with closing(self._connect()) as conn:
tokens = conn.execute("SELECT COUNT(*) FROM auth_tokens").fetchone()[0]
keys = conn.execute("SELECT COUNT(*) FROM auth_passkeys").fetchone()[0]
return int(tokens), int(keys)
# ---------------------------------------------------------------- tokens
#
# A token is a session for a script: the same 32 random bytes, the same
# SHA-256 in the file, carried as `Authorization: Bearer` instead of a
# cookie. It differs from a session in what it is for. A session is a
# browser that typed the password once and is signed out by a password
# change; a token is minted on purpose, named, optionally dated, and
# survives a password change the way a passkey does, because the cron job
# holding it is not the thing the password was changed against. It is
# revoked by its row, from the account screen, which a token cannot reach
# (`auth.middleware.ACCOUNT`).
def create_token(self, label: str, days: int | None) -> tuple[int, str]:
"""Mint one. Returns `(row id, token)`; the token exists only in that
return value, so the caller shows it once and never again.
`days` of None is a token that does not expire: the cron job that
holds it is not going to type a new one, and the row's last-used
stamp is what tells an abandoned one from one in use.
"""
token = secrets.token_urlsafe(32)
made = _now()
expires = _stamp(made + timedelta(days=days)) if days else ""
with closing(self._connect()) as conn, conn:
cur = conn.execute(
"""INSERT INTO auth_tokens
(token_hash, hint, label, created_at, last_used_at, expires_at)
VALUES (?,?,?,?,'',?)""",
(_token_hash(token), token[:HINT], label[:100], _stamp(made), expires),
)
return int(cur.lastrowid or 0), token
def touch_token(self, token: str) -> ApiToken | None:
"""The token this bearer value names, if it is live, marking it used.
Expiry is in the `WHERE` as it is for a session; a blank `expires_at`
is a token that does not. `last_used_at` is written at most once every
`_SEEN_EVERY`, for the reason `touch` gives.
"""
digest = _token_hash(token)
now = _now()
stamp = _stamp(now)
with closing(self._connect()) as conn:
row = conn.execute(
"SELECT * FROM auth_tokens WHERE token_hash=? "
"AND (expires_at = '' OR expires_at > ?)",
(digest, stamp),
).fetchone()
if row is None:
return None
used = row["last_used_at"]
if not used or now - datetime.fromisoformat(used) >= _SEEN_EVERY:
with conn:
conn.execute(
"UPDATE auth_tokens SET last_used_at=? WHERE token_hash=?",
(stamp, digest))
used = stamp
return _api_token(row, used)
def tokens(self) -> list[ApiToken]:
"""Every token, newest first, the expired ones included: a row that
ran out is still the owner's to recognise and remove. No hash."""
with closing(self._connect()) as conn:
rows = conn.execute(
"SELECT * FROM auth_tokens ORDER BY created_at DESC, id DESC").fetchall()
return [_api_token(r, r["last_used_at"]) for r in rows]
def rename_token(self, token_id: int, label: str) -> bool:
with closing(self._connect()) as conn, conn:
cur = conn.execute("UPDATE auth_tokens SET label=? WHERE id=?",
(label[:100], token_id))
return cur.rowcount > 0
def delete_token(self, token_id: int) -> bool:
"""Revoke one. By row id, as a session is: the screen has never seen
the token after the moment it was minted."""
with closing(self._connect()) as conn, conn:
cur = conn.execute("DELETE FROM auth_tokens WHERE id=?", (token_id,))
return cur.rowcount > 0
# -------------------------------------------------------------- passkeys # -------------------------------------------------------------- passkeys
def add_passkey(self, credential_id: bytes, public_key: bytes, def add_passkey(self, credential_id: bytes, public_key: bytes,
@@ -503,3 +677,10 @@ def _session(row: sqlite3.Row, last_seen: str, current: bool) -> SessionInfo:
id=row["id"], label=row["label"], created_at=row["created_at"], id=row["id"], label=row["label"], created_at=row["created_at"],
last_seen_at=last_seen, expires_at=row["expires_at"], current=current, last_seen_at=last_seen, expires_at=row["expires_at"], current=current,
) )
def _api_token(row: sqlite3.Row, last_used: str) -> ApiToken:
return ApiToken(
id=row["id"], label=row["label"], hint=row["hint"], created_at=row["created_at"],
last_used_at=last_used, expires_at=row["expires_at"],
)
+8 -1
View File
@@ -2,7 +2,7 @@
<html lang="en"> <html lang="en">
<head> <head>
<meta charset="utf-8"> <meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>{name} &mdash; sign in</title> <title>{name} &mdash; sign in</title>
<!-- The app's own stylesheet, if it passed one to `connect`. Optional on <!-- The app's own stylesheet, if it passed one to `connect`. Optional on
purpose: this folder is copied into projects that have no stylesheet yet, purpose: this folder is copied into projects that have no stylesheet yet,
@@ -42,6 +42,13 @@ body.login {
* saying twice. Without it the passkey button is drawn on an app that has no * saying twice. Without it the passkey button is drawn on an app that has no
* passkeys, and pressing it 404s. */ * passkeys, and pressing it 404s. */
[hidden] { display: none !important; } [hidden] { display: none !important; }
/* A phone: the form clears the notch and the home indicator, and a finger
* gets a thumb's height of button and a field that does not make iOS zoom. */
body.login { padding: env(safe-area-inset-top) 0 env(safe-area-inset-bottom); }
@media (pointer: coarse) {
.signin input[type=password] { font-size: 16px; padding: 0.6rem 0.7rem; }
.signin button { min-height: 2.75rem; }
}
</style> </style>
</head> </head>
<body class="login"> <body class="login">
+5 -2
View File
@@ -68,8 +68,11 @@ def locked(make_client: Callable[..., TestClient]) -> TestClient:
@pytest.fixture @pytest.fixture
def open_app(make_client: Callable[..., TestClient]) -> TestClient: def open_app(make_client: Callable[..., TestClient]) -> TestClient:
"""The same app on loopback under `auto` — no password asked for.""" """The same app on loopback under `auto` — no password asked for. Reached
return make_client(enabled="auto", host="127.0.0.1", password=None) as 127.0.0.1, which is the name an unlocked app answers a browser under
(`auth.middleware._rebound`)."""
return make_client(enabled="auto", host="127.0.0.1", password=None,
base_url="http://127.0.0.1")
@pytest.fixture @pytest.fixture
+44 -5
View File
@@ -5,7 +5,8 @@ one, so its tests are not allowed to reach for `app/` or `pipeline/` either --
`tests/test_coupling.py` reads the imports of every file under this folder and `tests/test_coupling.py` reads the imports of every file under this folder and
says so. What they need is a FastAPI app with a page, an API route, a socket, says so. What they need is a FastAPI app with a page, an API route, a socket,
and a couple of neighbours to prove the open list is matched the way it claims. and a couple of neighbours to prove the open list is matched the way it claims.
That is this file, and it is nine routes. That is this file, and it is nine routes and a stand-in for the broadcaster
the socket subscribes to.
A module rather than a conftest fixture because two tests build the same A module rather than a conftest fixture because two tests build the same
database twice, to see what a restart does, and a fixture cannot be asked for database twice, to see what a restart does, and a fixture cannot be asked for
@@ -15,11 +16,32 @@ twice in one test.
from __future__ import annotations from __future__ import annotations
import os import os
from collections.abc import Iterable
from fastapi import FastAPI, WebSocket from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect
from fastapi.responses import PlainTextResponse from fastapi.responses import PlainTextResponse
from auth import AuthConfig, AuthStore, connect from auth import AuthConfig, AuthStore, confirm_current, connect
class Bus:
"""What `auth.close_ended_sockets` reads off `app.state.bus`: the open
sockets and a way to close some. A stand-in for `pipeline.web.broadcaster.
Broadcaster`, which this folder may not import, with its two members."""
def __init__(self) -> None:
self.held: set[WebSocket] = set()
@property
def sockets(self) -> tuple[WebSocket, ...]:
return tuple(self.held)
async def close(self, sockets: Iterable[WebSocket], code: int, reason: str = "") -> int:
gone = [ws for ws in sockets if ws in self.held]
for ws in gone:
self.held.discard(ws)
await ws.close(code, reason)
return len(gone)
def build(db: str | os.PathLike[str], *, host: str = "0.0.0.0", def build(db: str | os.PathLike[str], *, host: str = "0.0.0.0",
@@ -28,6 +50,7 @@ def build(db: str | os.PathLike[str], *, host: str = "0.0.0.0",
app = FastAPI() app = FastAPI()
connect(app, AuthStore(db), AuthConfig(**settings), host=host, connect(app, AuthStore(db), AuthConfig(**settings), host=host,
open=("/static/",), stylesheet="/static/app.css") open=("/static/",), stylesheet="/static/app.css")
app.state.bus = bus = Bus()
@app.get("/", response_class=PlainTextResponse) @app.get("/", response_class=PlainTextResponse)
def front() -> str: def front() -> str:
@@ -41,6 +64,13 @@ def build(db: str | os.PathLike[str], *, host: str = "0.0.0.0",
def thing() -> dict[str, bool]: def thing() -> dict[str, bool]:
return {"ok": True} return {"ok": True}
@app.post("/thing/replace")
def replace(request: Request) -> dict[str, bool]:
"""A route that swaps the database as a file would: the current
password first, in its header, before any body."""
confirm_current(request)
return {"ok": True}
@app.get("/health") @app.get("/health")
def health() -> dict[str, bool]: def health() -> dict[str, bool]:
return {"ok": True} return {"ok": True}
@@ -61,8 +91,17 @@ def build(db: str | os.PathLike[str], *, host: str = "0.0.0.0",
@app.websocket("/ws/jobs") @app.websocket("/ws/jobs")
async def socket(ws: WebSocket) -> None: async def socket(ws: WebSocket) -> None:
"""Held open until the client leaves or the bus closes it, as the
app's `/ws/jobs` is."""
await ws.accept() await ws.accept()
await ws.send_json({"type": "jobs"}) bus.held.add(ws)
await ws.close() try:
await ws.send_json({"type": "jobs"})
while True:
await ws.receive_text()
except WebSocketDisconnect:
pass
finally:
bus.held.discard(ws)
return app return app
+217 -8
View File
@@ -8,15 +8,19 @@ from __future__ import annotations
import logging import logging
import sqlite3 import sqlite3
from collections.abc import Callable from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor
from contextlib import closing from contextlib import closing
from urllib.parse import parse_qs, urlsplit from urllib.parse import parse_qs, urlsplit
import pytest import pytest
from fastapi import Request
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
from starlette.requests import HTTPConnection
from starlette.websockets import WebSocketDisconnect from starlette.websockets import WebSocketDisconnect
from auth import AuthStore from auth import AuthStore
from auth.middleware import OPEN from auth.middleware import OPEN
from auth.store import _GRACE
from auth.tests import PASSWORD, sign_in from auth.tests import PASSWORD, sign_in
from auth.tests.harness import build from auth.tests.harness import build
@@ -27,7 +31,9 @@ def test_loopback_under_auto_needs_no_password(open_app: TestClient):
"""The default on a laptop: the machine's own login is already the door.""" """The default on a laptop: the machine's own login is already the door."""
assert open_app.get("/").status_code == 200 assert open_app.get("/").status_code == 200
assert open_app.get("/api/thing").status_code == 200 assert open_app.get("/api/thing").status_code == 200
with open_app.websocket_connect("/ws/jobs") as socket: # The handshake names `testserver` unless told otherwise, and an unlocked
# app answers only to the names it is reached under.
with open_app.websocket_connect("/ws/jobs", headers={"host": "127.0.0.1"}) as socket:
assert socket.receive_json()["type"] == "jobs" assert socket.receive_json()["type"] == "jobs"
@@ -49,6 +55,49 @@ def test_never_is_open_off_loopback_too(make_client: Callable[..., TestClient]):
assert client.get("/api/thing").status_code == 200 assert client.get("/api/thing").status_code == 200
# ------------------------------------------------ a proxy on the same machine
@pytest.mark.parametrize("header", [
{"x-forwarded-for": "203.0.113.9"}, {"x-forwarded-proto": "https"},
{"x-forwarded-host": "app.example.com"}, {"forwarded": "for=203.0.113.9"},
])
def test_an_open_app_refuses_what_a_proxy_forwarded(open_app: TestClient, header):
"""`auto` on loopback is open because only this machine can reach it. A
reverse proxy on the same machine makes that false while every request
still comes from 127.0.0.1; its headers are the only sign, so a request
carrying one is refused rather than served unlocked."""
refused = open_app.get("/api/thing", headers=header)
assert refused.status_code == 421
assert "auth.enabled: always" in refused.json()["detail"]
assert open_app.get("/api/thing").status_code == 200
def test_the_proxy_refusal_is_logged_once(open_app: TestClient, caplog):
with caplog.at_level(logging.WARNING):
for _ in range(3):
open_app.get("/api/thing", headers={"x-forwarded-for": "203.0.113.9"})
said = [r.getMessage() for r in caplog.records if "proxied" in r.getMessage()]
assert len(said) == 1 and "auth.enabled: always" in said[0]
def test_the_proxy_refusal_covers_the_socket(open_app: TestClient):
with pytest.raises(WebSocketDisconnect), \
open_app.websocket_connect("/ws/jobs", headers={"x-forwarded-for": "203.0.113.9"}):
pass
def test_a_locked_or_never_locked_app_serves_a_proxy(
locked: TestClient, make_client: Callable[..., TestClient]):
"""`always` has its own door and `never` is the operator's word: neither
is second-guessed from the headers."""
forwarded = {"x-forwarded-for": "203.0.113.9", "x-forwarded-proto": "https"}
assert locked.post("/login", json={"password": PASSWORD},
headers=forwarded).status_code == 200
assert locked.get("/api/thing", headers=forwarded).status_code == 200
never = make_client(enabled="never", host="127.0.0.1")
assert never.get("/api/thing", headers=forwarded).status_code == 200
# -------------------------------------------------------- the locked door # -------------------------------------------------------- the locked door
def test_everything_is_shut(locked: TestClient): def test_everything_is_shut(locked: TestClient):
@@ -150,6 +199,21 @@ def test_a_wrong_password_is_refused(locked: TestClient):
assert locked.get("/api/thing").status_code == 401 assert locked.get("/api/thing").status_code == 401
def test_wrong_guesses_in_parallel_meet_the_lockout(locked: TestClient):
"""The lockout was read before the derivation and the failure written
after it, so thirty wrong guesses at once all found the door open and all
got a 401. Each is claimed before it is derived now: the first `_GRACE`
are looked at and the rest are told to wait."""
def guess(_: int) -> int:
# A client per thread, so no two share a cookie jar or a portal.
return sign_in(TestClient(locked.app), "nope").status_code
with ThreadPoolExecutor(max_workers=30) as pool:
said = list(pool.map(guess, range(30)))
assert said.count(401) == _GRACE, said
assert said.count(429) == 30 - _GRACE, said
def test_signing_in_opens_everything(locked: TestClient): def test_signing_in_opens_everything(locked: TestClient):
assert sign_in(locked).status_code == 200 assert sign_in(locked).status_code == 200
assert locked.get("/api/thing").status_code == 200 assert locked.get("/api/thing").status_code == 200
@@ -352,6 +416,84 @@ def test_the_password_change_is_behind_the_same_lockout(signed_in: TestClient):
assert sign_in(signed_in).status_code == 429 assert sign_in(signed_in).status_code == 429
def test_an_app_route_asks_for_the_password_through_the_package(
signed_in: TestClient, caplog):
"""A route outside the package that must take the current password (a
restore, whose body is the database file) calls `confirm_password`
rather than writing its own check, so it is counted and gated like the
front door."""
from auth import confirm_password, warn
@signed_in.app.post("/api/restore")
def restore(request: Request) -> dict[str, bool]:
confirm_password(request, request.headers.get("x-current-password", ""))
return {"ok": True}
with caplog.at_level(logging.WARNING):
wrong = signed_in.post("/api/restore", headers={"x-current-password": "nope"})
assert wrong.status_code == 403
assert "wrong current password from testclient" in caplog.text
assert {r.name for r in caplog.records if "from testclient" in r.getMessage()} == {
"auth.middleware"}
assert signed_in.post("/api/restore",
headers={"x-current-password": PASSWORD}).status_code == 200
for _ in range(5):
signed_in.post("/api/restore", headers={"x-current-password": "nope"})
refused = signed_in.post("/api/restore", headers={"x-current-password": PASSWORD})
assert refused.status_code == 429 and refused.headers["retry-after"]
assert sign_in(signed_in).status_code == 429
caplog.clear()
with caplog.at_level(logging.WARNING):
warn(HTTPConnection({"type": "http", "client": ("203.0.113.9", 1), "headers": []}),
"restore refused")
assert "restore refused from 203.0.113.9" in caplog.text
def test_a_file_route_without_the_header_is_refused_and_counts_nothing(
signed_in: TestClient, caplog):
"""A restore's body is the database, so the password rides in a header,
and a session alone is not enough: the restored file brings its own hash.
A missing header is a missing field and not a guess, so five of them leave
the right password working, at this route and at the front door."""
with caplog.at_level(logging.WARNING):
missing = signed_in.post("/thing/replace")
empty = signed_in.post("/thing/replace", headers={"x-current-password": ""})
for refused in (missing, empty):
assert refused.status_code == 403
assert "X-Current-Password" in refused.json()["detail"]
assert "no current password for POST /thing/replace from testclient" in caplog.text
for _ in range(5):
assert signed_in.post("/thing/replace").status_code == 403
assert signed_in.post("/thing/replace",
headers={"x-current-password": PASSWORD}).status_code == 200
assert sign_in(signed_in).status_code == 200
def test_a_file_route_counts_a_wrong_header(signed_in: TestClient, caplog):
with caplog.at_level(logging.WARNING):
wrong = signed_in.post("/thing/replace", headers={"x-current-password": "nope"})
assert wrong.status_code == 403
assert "wrong current password from testclient" in caplog.text
for _ in range(4):
signed_in.post("/thing/replace", headers={"x-current-password": "nope"})
refused = signed_in.post("/thing/replace", headers={"x-current-password": PASSWORD})
assert refused.status_code == 429 and refused.headers["retry-after"]
assert sign_in(signed_in).status_code == 429
def test_a_file_route_takes_the_right_header(signed_in: TestClient):
right = signed_in.post("/thing/replace", headers={"x-current-password": PASSWORD})
assert right.status_code == 200 and right.json() == {"ok": True}
def test_a_file_route_on_an_app_with_no_password_needs_no_header(open_app: TestClient):
"""The laptop that was never locked has nothing to confirm."""
assert open_app.post("/thing/replace").status_code == 200
# ------------------------------------------------------- the bootstrap # ------------------------------------------------------- the bootstrap
def test_a_generated_password_is_printed_once( def test_a_generated_password_is_printed_once(
@@ -429,6 +571,51 @@ def test_the_environment_variable_is_a_setting(
assert client.get("/api/account").json()["pinned_by"] == "OTHER_PASSWORD" assert client.get("/api/account").json()["pinned_by"] == "OTHER_PASSWORD"
def test_recovery_through_the_environment_signs_everyone_out(
tmp_path, monkeypatch, caplog):
"""`APP_PASSWORD` is the way back in when the password is lost, and the
reason it is lost may be that someone else has it and is signed in. So a
pinned password that replaces the stored one closes every session. Tokens
and passkeys stay, since the environment cannot tell whose they are, and
the log says how many so the owner knows to review them."""
db = tmp_path / "recovered.db"
store = AuthStore(db)
store.set_password("the-one-the-intruder-has")
intruder = store.open_session("intruder", 30)
_, token = store.create_token("intruder's cron", None)
store.add_passkey(b"cred", b"key", 0, "", "intruder's key")
monkeypatch.setenv("APP_PASSWORD", PASSWORD)
with caplog.at_level(logging.WARNING), \
TestClient(build(db, enabled="always"), follow_redirects=False) as client:
assert client.get("/api/thing",
headers={"cookie": f"app_session={intruder}"}).status_code == 401
assert client.get("/api/thing",
headers={"authorization": f"Bearer {token}"}).status_code == 200
assert sign_in(client).status_code == 200
mine = client.cookies["app_session"]
said = [r for r in caplog.records if "password reset from APP_PASSWORD" in r.getMessage()]
assert len(said) == 1 and said[0].levelno == logging.WARNING
assert "closed 1 sessions; 1 API tokens and 1 passkeys remain" in said[0].getMessage()
# A restart with the same variable matches and closes nothing.
caplog.clear()
with caplog.at_level(logging.WARNING), TestClient(build(db, enabled="always")) as client:
assert client.get("/api/thing",
headers={"cookie": f"app_session={mine}"}).status_code == 200
assert "password reset" not in caplog.text
def test_a_first_password_from_the_environment_is_not_a_recovery(
make_client: Callable[..., TestClient], monkeypatch, caplog):
monkeypatch.setenv("APP_PASSWORD", PASSWORD)
with caplog.at_level(logging.INFO):
client = make_client(enabled="always", password=None)
assert "password set from APP_PASSWORD" in caplog.text
assert "password reset" not in caplog.text
assert sign_in(client).status_code == 200
def test_expired_sessions_are_purged_at_startup(tmp_path): def test_expired_sessions_are_purged_at_startup(tmp_path):
"""Housekeeping, not a security boundary -- `touch` already refuses an """Housekeeping, not a security boundary -- `touch` already refuses an
expired row. Without it the table grows forever.""" expired row. Without it the table grows forever."""
@@ -452,8 +639,8 @@ def test_a_refused_login_says_who_it_was(locked: TestClient, caplog):
nothing anywhere knows it happened.""" nothing anywhere knows it happened."""
with caplog.at_level(logging.WARNING): with caplog.at_level(logging.WARNING):
assert sign_in(locked, "nope").status_code == 401 assert sign_in(locked, "nope").status_code == 401
# The logger is `auth.router`, so the line already says which package this # The logger is `auth.middleware`, so the line already says which package
# came from and the message does not say it again. # this came from and the message does not say it again.
assert "wrong password from testclient" in caplog.text assert "wrong password from testclient" in caplog.text
assert "auth: wrong password" not in caplog.text assert "auth: wrong password" not in caplog.text
@@ -465,15 +652,37 @@ def test_a_refused_password_change_says_so_too(signed_in: TestClient, caplog):
assert "wrong current password" in caplog.text assert "wrong current password" in caplog.text
def test_the_log_prefers_the_address_the_proxy_passed_on(locked: TestClient, caplog): def test_the_log_names_the_hop_the_proxy_added(locked: TestClient, caplog):
"""Behind a proxy the peer is a container on its own network and says """Behind a proxy the peer is a container on its own network and says
nothing about who is at the other end. Both are written, because the header nothing about who is at the other end. The proxy appends the address it
is only as good as whoever sent it -- which is also why nothing is decided accepted the connection from to whatever `X-Forwarded-For` the caller
from it.""" sent, so the rightmost hop is the proxy's word and every hop left of it
the caller's: logging the first let an attacker name whom fail2ban bans.
Both the hop and the peer are written, because the header is only as good
as whoever set it -- which is also why nothing is decided from it."""
with caplog.at_level(logging.WARNING): with caplog.at_level(logging.WARNING):
locked.post("/login", json={"password": "nope"}, locked.post("/login", json={"password": "nope"},
headers={"x-forwarded-for": "203.0.113.9, 10.9.9.9"}) headers={"x-forwarded-for": "203.0.113.9, 10.9.9.9"})
assert "203.0.113.9 via testclient" in caplog.text (said,) = [r.getMessage() for r in caplog.records if r.name == "auth.middleware"]
assert said == "wrong password from 10.9.9.9 via testclient"
assert "203.0.113.9" not in caplog.text
def test_the_log_falls_back_to_x_real_ip(locked: TestClient, caplog):
"""A proxy that sets `X-Real-IP` sets it outright. Read only when there is
no `X-Forwarded-For`, which a proxy that sends both keeps current too."""
with caplog.at_level(logging.WARNING):
locked.post("/login", json={"password": "nope"},
headers={"x-real-ip": "198.51.100.7"})
locked.post("/login", json={"password": "nope"},
headers={"x-real-ip": "198.51.100.7",
"x-forwarded-for": "203.0.113.9, 192.0.2.1"})
locked.post("/login", json={"password": "nope"},
headers={"x-forwarded-for": " ", "x-real-ip": "198.51.100.7"})
said = [r.getMessage() for r in caplog.records if r.name == "auth.middleware"]
assert said == ["wrong password from 198.51.100.7 via testclient",
"wrong password from 192.0.2.1 via testclient",
"wrong password from 198.51.100.7 via testclient"]
def test_a_forwarded_address_cannot_write_the_log_itself(locked: TestClient, caplog): def test_a_forwarded_address_cannot_write_the_log_itself(locked: TestClient, caplog):
+63 -6
View File
@@ -32,6 +32,8 @@ from auth.tests.softkey import SoftKey, b64 # noqa: E402 after the skip
#: the real `blocked()` path rather than a configured override. #: the real `blocked()` path rather than a configured override.
RP_ID = "localhost" RP_ID = "localhost"
ORIGIN = "http://localhost" ORIGIN = "http://localhost"
#: What starting a registration takes besides a session: the password.
CONFIRM = {"current": PASSWORD}
def unb64(text: str) -> bytes: def unb64(text: str) -> bytes:
@@ -48,7 +50,7 @@ def client(make_client: Callable[..., TestClient]) -> TestClient:
def register(client: TestClient, key: SoftKey, label: str = "Test key"): def register(client: TestClient, key: SoftKey, label: str = "Test key"):
"""The whole registration ceremony, both round trips.""" """The whole registration ceremony, both round trips."""
options = client.post("/api/account/passkeys/options").json() options = client.post("/api/account/passkeys/options", json=CONFIRM).json()
made = key.register(unb64(options["challenge"]), RP_ID, ORIGIN) made = key.register(unb64(options["challenge"]), RP_ID, ORIGIN)
return client.post("/api/account/passkeys", return client.post("/api/account/passkeys",
json={"credential": made, "label": label}) json={"credential": made, "label": label})
@@ -66,7 +68,45 @@ def login(client: TestClient, key: SoftKey, origin: str = ORIGIN):
def test_registering_needs_a_session(client: TestClient): def test_registering_needs_a_session(client: TestClient):
"""A passkey is a new way in, so adding one is gated on already being in.""" """A passkey is a new way in, so adding one is gated on already being in."""
client.post("/logout") client.post("/logout")
assert client.post("/api/account/passkeys/options").status_code == 401 assert client.post("/api/account/passkeys/options", json=CONFIRM).status_code == 401
def test_registering_needs_the_password_too(client: TestClient, caplog):
"""A passkey outlives every session and a password change, so a borrowed
signed-in browser must not be able to add one: a wrong or missing password
gets no challenge, is logged like the other doors, and is counted."""
with caplog.at_level("WARNING"):
wrong = client.post("/api/account/passkeys/options", json={"current": "nope"})
assert wrong.status_code == 403
assert "wrong current password" in caplog.text
assert client.post("/api/account/passkeys/options", json={}).status_code == 422
assert client.post("/api/account/passkeys/options").status_code == 422
assert client.get("/api/account").json()["passkeys"] == []
def test_registering_is_behind_the_same_lockout(client: TestClient):
"""Guessing at the passkey door locks the front door too, and once locked
the right password is not looked at."""
for _ in range(5):
assert client.post("/api/account/passkeys/options",
json={"current": "nope"}).status_code == 403
refused = client.post("/api/account/passkeys/options", json=CONFIRM)
assert refused.status_code == 429
client.post("/logout")
assert sign_in(client).status_code == 429
def test_an_app_with_no_password_registers_without_one(
make_client: Callable[..., TestClient]):
"""Loopback under `auto` and never locked: there is no password to confirm
and no session to borrow, so registering before deploying still works."""
open_app = make_client(host="127.0.0.1", password=None, base_url=ORIGIN)
assert not open_app.app.state.auth.has_password()
options = open_app.post("/api/account/passkeys/options", json={"current": ""})
assert options.status_code == 200
made = SoftKey().register(unb64(options.json()["challenge"]), RP_ID, ORIGIN)
assert open_app.post("/api/account/passkeys",
json={"credential": made}).status_code == 200
def test_a_passkey_registers_and_is_listed(client: TestClient): def test_a_passkey_registers_and_is_listed(client: TestClient):
@@ -95,14 +135,14 @@ def test_a_second_registration_is_excluded_not_duplicated(client: TestClient):
say "you have one of these" rather than making a second.""" say "you have one of these" rather than making a second."""
key = SoftKey() key = SoftKey()
register(client, key) register(client, key)
options = client.post("/api/account/passkeys/options").json() options = client.post("/api/account/passkeys/options", json=CONFIRM).json()
assert [c["id"] for c in options["excludeCredentials"]] == [b64(key.credential_id)] assert [c["id"] for c in options["excludeCredentials"]] == [b64(key.credential_id)]
def test_a_stale_challenge_is_refused(client: TestClient): def test_a_stale_challenge_is_refused(client: TestClient):
"""A challenge is single-use, so the same response cannot be sent twice.""" """A challenge is single-use, so the same response cannot be sent twice."""
key = SoftKey() key = SoftKey()
options = client.post("/api/account/passkeys/options").json() options = client.post("/api/account/passkeys/options", json=CONFIRM).json()
made = key.register(unb64(options["challenge"]), RP_ID, ORIGIN) made = key.register(unb64(options["challenge"]), RP_ID, ORIGIN)
assert client.post("/api/account/passkeys", json={"credential": made}).status_code == 200 assert client.post("/api/account/passkeys", json={"credential": made}).status_code == 200
@@ -224,6 +264,23 @@ def test_a_malformed_body_is_a_400_and_not_a_traceback(client: TestClient):
assert client.post("/login/passkey", json={"credential": {}}).status_code == 400 assert client.post("/login/passkey", json={"credential": {}}).status_code == 400
def test_a_raw_id_that_is_not_base64url_is_a_400_and_spends_nothing(client: TestClient):
"""`rawId` was decoded unguarded after the challenge was taken, so `"a"`
was a 500 from an endpoint anyone can reach and the challenge was gone.
Now it is refused before the challenge is spent: the same challenge,
answered properly, still signs in."""
key = SoftKey()
register(client, key)
client.post("/logout")
options = client.post("/login/passkey/options").json()
signed = key.sign(unb64(options["challenge"]), RP_ID, ORIGIN)
refused = client.post("/login/passkey", json={"credential": {**signed, "rawId": "a"}})
assert refused.status_code == 400
assert "rawId is not base64url" in refused.json()["detail"]
assert client.post("/login/passkey", json={"credential": signed}).status_code == 200
def test_no_passkeys_registered_says_so(client: TestClient): def test_no_passkeys_registered_says_so(client: TestClient):
client.post("/logout") client.post("/logout")
assert client.post("/login/passkey/options").status_code == 404 assert client.post("/login/passkey/options").status_code == 404
@@ -243,7 +300,7 @@ def test_a_registration_challenge_cannot_be_answered_at_the_login_door(
""" """
key = SoftKey() key = SoftKey()
register(client, key) register(client, key)
theirs = client.post("/api/account/passkeys/options").json() theirs = client.post("/api/account/passkeys/options", json=CONFIRM).json()
client.post("/logout") client.post("/logout")
signed = key.sign(unb64(theirs["challenge"]), RP_ID, ORIGIN) signed = key.sign(unb64(theirs["challenge"]), RP_ID, ORIGIN)
@@ -352,7 +409,7 @@ def test_a_configured_rp_id_and_origin_win(make_client: Callable[..., TestClient
client = make_client(enabled="always", rp_id="app.example.com", client = make_client(enabled="always", rp_id="app.example.com",
origin="https://app.example.com") origin="https://app.example.com")
sign_in(client) sign_in(client)
options = client.post("/api/account/passkeys/options").json() options = client.post("/api/account/passkeys/options", json=CONFIRM).json()
assert options["rp"]["id"] == "app.example.com" assert options["rp"]["id"] == "app.example.com"
assert client.get("/api/account").json()["passkeys_blocked"] == "" assert client.get("/api/account").json()["passkeys_blocked"] == ""
+170
View File
@@ -0,0 +1,170 @@
"""A socket whose credential ends is closed: by the route that ended it, or by
the sweep when it expired.
uv run pytest auth/tests/test_sockets.py -q
The door is checked at the handshake only, so before this a signed-out tab,
a revoked token or an expired session kept its socket streaming until the tab
reconnected. Every socket here is opened on the client the route is called
on: a client that is not entered runs its requests on an event loop of their
own, and the route's background task could not close a socket living there.
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
import sqlite3
from contextlib import closing
from pathlib import Path
from typing import Any
import pytest
from fastapi.testclient import TestClient
from starlette.websockets import WebSocketDisconnect
import auth.middleware
from auth import AuthConfig, close_ended_sockets, sweep_ended_sockets
from auth.tests import PASSWORD
from auth.tests.harness import build
#: The cookie a login over plain http sets.
COOKIE = AuthConfig().cookie
LONG_AGO = "2000-01-01T00:00:00+00:00"
def closed_with(ws: Any) -> int:
"""Read frames until the server closes the socket; the close code.
Blocks on a socket the server left open, since the test client's receive
has no timeout, so each test asks the bus first, which does not block."""
for _ in range(50):
try:
ws.receive_json()
except WebSocketDisconnect as closed:
return closed.code
raise AssertionError("the socket stayed open")
def sweep_now(client: TestClient) -> int:
"""`close_ended_sockets` on the loop the client's sockets live on."""
assert client.portal is not None
return client.portal.call(close_ended_sockets, client.app)
def expire(client: TestClient, table: str) -> None:
with closing(sqlite3.connect(client.app.state.auth.path)) as conn, conn:
conn.execute(f"UPDATE {table} SET expires_at=?", (LONG_AGO,))
# --------------------------------------------------------------- the routes
def test_signing_out_closes_the_socket_the_session_opened(signed_in: TestClient):
with signed_in.websocket_connect("/ws/jobs") as ws:
assert ws.receive_json()["type"] == "jobs"
assert signed_in.post("/logout").status_code == 200
assert signed_in.app.state.bus.sockets == ()
assert closed_with(ws) == 1008
def test_revoking_a_token_closes_its_socket(signed_in: TestClient):
"""The cookie is taken out so the bearer header is what opens the socket;
with the cookie in, the session would be what it stands on."""
made = signed_in.post("/api/account/tokens",
json={"current": PASSWORD, "label": "cron"}).json()
here = signed_in.cookies[COOKIE]
signed_in.cookies.clear()
with signed_in.websocket_connect(
"/ws/jobs", headers={"authorization": f"Bearer {made['token']}"}) as ws:
assert ws.receive_json()["type"] == "jobs"
signed_in.cookies.set(COOKIE, here)
assert signed_in.delete(f"/api/account/tokens/{made['id']}").status_code == 200
assert signed_in.app.state.bus.sockets == ()
assert closed_with(ws) == 1008
def test_closing_the_others_closes_their_sockets_and_keeps_this_one(
signed_in: TestClient):
here = signed_in.cookies[COOKIE]
there = signed_in.app.state.auth.open_session("phone", 1)
def as_(token: str) -> None:
signed_in.cookies.clear()
signed_in.cookies.set(COOKIE, token)
as_(there)
with signed_in.websocket_connect("/ws/jobs") as theirs:
assert theirs.receive_json()["type"] == "jobs"
as_(here)
with signed_in.websocket_connect("/ws/jobs") as mine:
assert mine.receive_json()["type"] == "jobs"
closed = signed_in.post("/api/account/sessions/close")
assert closed.json() == {"signed_out": 1}
assert len(signed_in.app.state.bus.sockets) == 1, "this tab's is open"
assert closed_with(theirs) == 1008
# ---------------------------------------------------------------- expiry
def test_an_expired_session_loses_its_socket(signed_in: TestClient):
"""Expiry ends no request, so no route sees it; the sweep's call does."""
with signed_in.websocket_connect("/ws/jobs") as ws:
assert ws.receive_json()["type"] == "jobs"
assert sweep_now(signed_in) == 0, "a live session keeps its socket"
expire(signed_in, "auth_sessions")
assert sweep_now(signed_in) == 1
assert closed_with(ws) == 1008
def test_nothing_is_closed_on_an_unlocked_app_or_one_without_a_bus(
open_app: TestClient, signed_in: TestClient):
with open_app.websocket_connect("/ws/jobs", headers={"host": "127.0.0.1"}) as ws:
assert ws.receive_json()["type"] == "jobs"
assert sweep_now(open_app) == 0
del signed_in.app.state.bus
expire(signed_in, "auth_sessions")
assert sweep_now(signed_in) == 0
def test_the_sweep_runs_on_its_clock_and_survives_a_failure(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture):
calls: list[object] = []
async def counted(app: object) -> int:
calls.append(app)
if len(calls) == 1:
raise sqlite3.OperationalError("database is locked")
return 0
monkeypatch.setattr(auth.middleware, "close_ended_sockets", counted)
async def run() -> asyncio.Task[None]:
task = asyncio.create_task(sweep_ended_sockets("the app", every=0.01))
for _ in range(200):
if len(calls) >= 2:
break
await asyncio.sleep(0.01)
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await task
return task
with caplog.at_level(logging.WARNING):
task = asyncio.run(run())
assert len(calls) >= 2 and calls[0] == "the app"
assert "the socket sweep failed" in caplog.text
assert task.cancelled()
def test_a_locked_app_sweeps_while_it_is_up(tmp_path: Path):
app = build(tmp_path / "a.db", enabled="always")
assert app.state.auth_sweep is None
with TestClient(app):
sweep = app.state.auth_sweep
assert isinstance(sweep, asyncio.Task) and not sweep.done()
assert sweep.done() and sweep.cancelled()
def test_an_unlocked_app_has_no_sweep(open_app: TestClient):
assert open_app.app.state.auth_sweep is None
+66 -1
View File
@@ -53,7 +53,7 @@ def test_the_tables_are_namespaced(store: AuthStore):
named = {name for (name,) in rows( named = {name for (name,) in rows(
store, "SELECT name FROM sqlite_master WHERE type='table'")} store, "SELECT name FROM sqlite_master WHERE type='table'")}
assert {"auth_credential", "auth_sessions", "auth_passkeys", assert {"auth_credential", "auth_sessions", "auth_passkeys",
"auth_challenges"} <= named "auth_challenges", "auth_tokens"} <= named
assert not {n for n in named if not n.startswith(("auth_", "sqlite_"))} assert not {n for n in named if not n.startswith(("auth_", "sqlite_"))}
@@ -119,6 +119,71 @@ def test_the_delay_doubles_and_then_stops(store: AuthStore):
assert store.check_password(PASSWORD) is True assert store.check_password(PASSWORD) is True
def test_wrong_guesses_in_parallel_are_each_counted(store: AuthStore):
"""`failures` was read, scrypt ran, and the read plus one was written, so
forty wrong guesses at once recorded one failure and no lockout. The raise
is one statement under the write lock now."""
from concurrent.futures import ThreadPoolExecutor
store.set_password(PASSWORD)
with ThreadPoolExecutor(max_workers=16) as pool:
said = list(pool.map(lambda _: store.check_password("wrong"), range(40)))
assert said == [False] * 40
assert rows(store, "SELECT failures FROM auth_credential") == [(40,)]
assert store.locked_for() > 0
def test_parallel_claims_admit_the_grace_and_no_more(store: AuthStore):
"""What both doors do: claim, then derive. A claim reads the lockout and
counts the attempt in one write transaction, so thirty claims at once
admit `_GRACE` guesses and tell the rest to wait, where a read before the
derivation and a write after it admitted all thirty."""
from concurrent.futures import ThreadPoolExecutor
store.set_password(PASSWORD)
def guess(_: int) -> str:
if store.claim_attempt():
return "wait"
return "right" if store.check_password("wrong", claimed=True) else "wrong"
with ThreadPoolExecutor(max_workers=30) as pool:
said = list(pool.map(guess, range(30)))
assert said.count("wrong") == _GRACE and said.count("wait") == 30 - _GRACE
assert rows(store, "SELECT failures FROM auth_credential") == [(_GRACE,)]
def test_a_claimed_right_password_clears_the_count(store: AuthStore):
"""Counted up front, credited back: the fifth claim writes the lockout
before the derivation, and a right answer then clears it."""
store.set_password(PASSWORD)
for _ in range(_GRACE - 1):
assert store.claim_attempt() == 0
assert store.check_password("wrong", claimed=True) is False
assert store.claim_attempt() == 0
assert store.locked_for() > 0, "the fifth claim did not write the lockout"
assert store.check_password(PASSWORD, claimed=True) is True
assert store.locked_for() == 0
assert rows(store, "SELECT failures, locked_until FROM auth_credential") == [(0, "")]
def test_a_claim_while_locked_counts_nothing(store: AuthStore):
"""Hammering while locked out does not extend the lockout."""
store.set_password(PASSWORD)
for _ in range(_GRACE):
assert store.claim_attempt() == 0
store.check_password("wrong", claimed=True)
before = rows(store, "SELECT failures, locked_until FROM auth_credential")
for _ in range(10):
assert store.claim_attempt() > 0
assert rows(store, "SELECT failures, locked_until FROM auth_credential") == before
def test_a_claim_on_a_database_with_no_password_is_granted(store: AuthStore):
assert store.claim_attempt() == 0
assert store.check_password(PASSWORD, claimed=True) is False
# -------------------------------------------------------------- the sessions # -------------------------------------------------------------- the sessions
def test_a_session_is_live_until_it_is_not(store: AuthStore): def test_a_session_is_live_until_it_is_not(store: AuthStore):
+465
View File
@@ -0,0 +1,465 @@
"""API tokens: the third way in, for a script, and what it cannot do.
uv run pytest auth/tests/test_tokens.py -q
A token is minted with the password, carried as `Authorization: Bearer`, looked
up on every request like a session, and shut out of the account routes. The
same-site rule the middleware applies to every request is here too, because a
script's token is the one credential a browser never carries for it.
"""
from __future__ import annotations
import logging
import sqlite3
from collections.abc import Callable
from contextlib import closing
from datetime import UTC, datetime, timedelta
import pytest
from fastapi.testclient import TestClient
from starlette.websockets import WebSocketDisconnect
from auth import AuthStore
from auth.middleware import ACCOUNT, _rebound, cross_site
from auth.store import HINT
from auth.tests import PASSWORD, sign_in
CONFIRM = {"current": PASSWORD}
def mint(client: TestClient, label: str = "cron", **body: object) -> dict:
answer = client.post("/api/account/tokens", json={**CONFIRM, "label": label, **body})
assert answer.status_code == 200, answer.text
return answer.json()
def bearer(token: str) -> dict[str, str]:
return {"authorization": f"Bearer {token}"}
# ------------------------------------------------------------------ the store
def test_the_token_is_not_in_the_file(store: AuthStore):
"""Hashed like a session token, and for the same reason: the file lets
nobody in."""
token_id, token = store.create_token("cron", None)
assert token.encode() not in store.path.read_bytes()
assert store.touch_token(token) is not None
assert store.touch_token(token).id == token_id
def test_the_list_shows_the_opening_and_nothing_else(store: AuthStore):
_, token = store.create_token("cron", None)
(listed,) = store.tokens()
assert listed.hint == token[:HINT] and len(listed.hint) == HINT
assert listed.label == "cron" and listed.expires_at == "" and listed.last_used_at == ""
assert "token" not in listed.model_dump()
def test_a_dated_token_runs_out(store: AuthStore):
_, token = store.create_token("short", 30)
(listed,) = store.tokens()
expires = datetime.fromisoformat(listed.expires_at)
assert timedelta(days=29) < expires - datetime.now(UTC) <= timedelta(days=30)
with closing(sqlite3.connect(store.path)) as conn, conn:
conn.execute("UPDATE auth_tokens SET expires_at='2000-01-01T00:00:00+00:00'")
assert store.touch_token(token) is None
# Still listed, so the owner sees the row that ran out and removes it.
assert len(store.tokens()) == 1
def test_using_a_token_stamps_it_once_in_a_while(store: AuthStore):
_, token = store.create_token("cron", None)
first = store.touch_token(token)
assert first is not None and first.last_used_at
with closing(sqlite3.connect(store.path)) as conn, conn:
conn.execute("UPDATE auth_tokens SET last_used_at='2000-01-01T00:00:00+00:00'")
again = store.touch_token(token)
assert again is not None and again.last_used_at > "2000-01-02"
def test_a_token_is_renamed_and_revoked_by_its_row(store: AuthStore):
token_id, token = store.create_token("cron", None)
assert store.rename_token(token_id, "the laptop") is True
assert store.tokens()[0].label == "the laptop"
assert store.delete_token(token_id) is True
assert store.touch_token(token) is None
assert store.delete_token(token_id) is False
def test_a_token_is_not_a_session(store: AuthStore):
"""Two tables, two lookups: a token in the cookie opens nothing, and a
session token in the header opens nothing."""
_, token = store.create_token("cron", None)
session = store.open_session("browser", 30)
assert store.touch(token) is None
assert store.touch_token(session) is None
# ----------------------------------------------------------------- minting
def test_minting_needs_a_session(locked: TestClient):
assert locked.post("/api/account/tokens", json=CONFIRM).status_code == 401
def test_minting_needs_the_password_too(signed_in: TestClient, caplog):
"""A token outlives every session and a password change, so a borrowed
signed-in browser must not be able to mint one."""
with caplog.at_level(logging.WARNING):
wrong = signed_in.post("/api/account/tokens", json={"current": "nope"})
assert wrong.status_code == 403
assert "wrong current password" in caplog.text
assert signed_in.get("/api/account").json()["tokens"] == []
def test_minting_is_behind_the_same_lockout(signed_in: TestClient):
for _ in range(5):
signed_in.post("/api/account/tokens", json={"current": "nope"})
refused = signed_in.post("/api/account/tokens", json=CONFIRM)
assert refused.status_code == 429
assert sign_in(signed_in).status_code == 429
def test_an_app_with_no_password_mints_without_one(open_app: TestClient):
"""The laptop before it is deployed, as with a passkey."""
made = open_app.post("/api/account/tokens", json={"current": "", "label": "here"})
assert made.status_code == 200 and made.json()["token"]
def test_the_token_is_shown_once_and_listed_by_its_opening(signed_in: TestClient):
made = mint(signed_in, "cron", days=90)
assert made["hint"] == made["token"][:HINT] and made["label"] == "cron"
assert made["expires_at"]
(listed,) = signed_in.get("/api/account").json()["tokens"]
assert listed["id"] == made["id"] and listed["hint"] == made["hint"]
assert "token" not in listed
def test_the_label_and_the_days_are_bounded(signed_in: TestClient):
assert signed_in.post("/api/account/tokens",
json={**CONFIRM, "days": 0}).status_code == 422
assert signed_in.post("/api/account/tokens",
json={**CONFIRM, "days": 4000}).status_code == 422
assert signed_in.post("/api/account/tokens",
json={**CONFIRM, "label": "x" * 101}).status_code == 422
# ------------------------------------------------------------- the bearer
def test_a_token_opens_the_api_and_the_socket(signed_in: TestClient):
token = mint(signed_in)["token"]
script = TestClient(signed_in.app, follow_redirects=False)
assert script.get("/api/thing").status_code == 401
assert script.get("/api/thing", headers=bearer(token)).status_code == 200
with script.websocket_connect("/ws/jobs", headers=bearer(token)) as socket:
assert socket.receive_json()["type"] == "jobs"
def test_a_token_cannot_reach_the_account(signed_in: TestClient):
"""A leaked token must not be able to mint another, revoke the owner's
sessions, or read the list of ways in; and `/logout` is a session's."""
token = mint(signed_in)["token"]
script = TestClient(signed_in.app, follow_redirects=False)
for method, path in (("GET", "/api/account"),
("POST", "/api/account/tokens"),
("POST", "/api/account/sessions/close"),
("DELETE", "/api/account/sessions/1"),
("POST", "/api/account/password"),
("POST", "/logout")):
refused = script.request(method, path, headers=bearer(token), json=CONFIRM)
assert refused.status_code == 403, f"{method} {path}: {refused.status_code}"
assert "not the account" in refused.json()["detail"]
# And the names are matched whole: `/api/accounts` is not the account.
assert script.get("/api/accountx", headers=bearer(token)).status_code == 404
assert ACCOUNT == ("/api/account", "/api/account/", "/logout")
def test_a_cookie_session_wins_over_a_bad_header(signed_in: TestClient):
"""A browser extension that adds a stale header must not sign a browser out."""
assert signed_in.get("/api/account", headers=bearer("stale")).status_code == 200
def test_a_bad_token_is_a_401_and_a_line_in_the_log(locked: TestClient, caplog):
with caplog.at_level(logging.WARNING):
refused = locked.get("/api/thing", headers=bearer("not-a-token"))
assert refused.status_code == 401
assert refused.json()["detail"] == "That token is not valid here."
assert "bad token from testclient" in caplog.text
# JSON whatever was asked for: a script, not a browser, sent this header.
assert locked.get("/deep/page", headers=bearer("not-a-token")).status_code == 401
# And nothing is counted against the password: the right one still opens.
assert sign_in(locked).status_code == 200
def test_only_the_bearer_scheme_is_read(locked: TestClient):
assert locked.get("/api/thing",
headers={"authorization": "Basic dXNlcjpwYXNz"}).status_code == 401
assert locked.get("/api/thing?token=x").status_code == 401
def test_a_revoked_token_is_refused_on_the_next_request(signed_in: TestClient):
made = mint(signed_in)
script = TestClient(signed_in.app, follow_redirects=False)
assert script.get("/api/thing", headers=bearer(made["token"])).status_code == 200
assert signed_in.delete(f"/api/account/tokens/{made['id']}").status_code == 200
assert script.get("/api/thing", headers=bearer(made["token"])).status_code == 401
assert signed_in.delete(f"/api/account/tokens/{made['id']}").status_code == 404
def test_a_token_survives_a_password_change(signed_in: TestClient):
"""Like a passkey: the cron job holding it is not what the password was
changed against. The account screen is where to revoke it, and the answer
says how many remain so the screen can say so."""
token = mint(signed_in)["token"]
changed = signed_in.post("/api/account/password",
json={"current": PASSWORD, "new": "a-much-longer-one"})
assert changed.status_code == 200
assert changed.json() == {"signed_out": 0, "tokens_revoked": 0, "passkeys_revoked": 0,
"tokens_left": 1, "passkeys_left": 0}
script = TestClient(signed_in.app, follow_redirects=False)
assert script.get("/api/thing", headers=bearer(token)).status_code == 200
def test_a_password_change_can_revoke_every_token_and_passkey(signed_in: TestClient):
"""The case the route exists for: whoever had the password may have
minted a token that never expires or registered a passkey, and either
outlives the change. `revoke_all` deletes them with it."""
token = mint(signed_in)["token"]
mint(signed_in, "another")
signed_in.app.state.auth.add_passkey(b"cred", b"key", 0, "", "someone's key")
other = TestClient(signed_in.app, follow_redirects=False)
sign_in(other)
changed = signed_in.post("/api/account/password", json={
"current": PASSWORD, "new": "a-much-longer-one", "revoke_all": True})
assert changed.status_code == 200
assert changed.json() == {"signed_out": 1, "tokens_revoked": 2, "passkeys_revoked": 1,
"tokens_left": 0, "passkeys_left": 0}
script = TestClient(signed_in.app, follow_redirects=False)
assert script.get("/api/thing", headers=bearer(token)).status_code == 401
account = signed_in.get("/api/account").json()
assert account["tokens"] == [] and account["passkeys"] == []
assert other.get("/api/thing").status_code == 401
assert signed_in.get("/api/thing").status_code == 200
def test_a_wrong_password_revokes_nothing(signed_in: TestClient):
token = mint(signed_in)["token"]
refused = signed_in.post("/api/account/password", json={
"current": "nope", "new": "a-much-longer-one", "revoke_all": True})
assert refused.status_code == 403
script = TestClient(signed_in.app, follow_redirects=False)
assert script.get("/api/thing", headers=bearer(token)).status_code == 200
def test_a_token_is_renamed_from_the_account(signed_in: TestClient):
made = mint(signed_in)
renamed = signed_in.post(f"/api/account/tokens/{made['id']}/label",
json={"label": "the pi"})
assert renamed.status_code == 200
assert signed_in.get("/api/account").json()["tokens"][0]["label"] == "the pi"
assert signed_in.post("/api/account/tokens/999/label",
json={"label": "x"}).status_code == 404
def test_an_open_app_ignores_the_header(open_app: TestClient):
"""Unlocked, there is no door for a token to open; a stale one is not an error."""
assert open_app.get("/api/thing", headers=bearer("whatever")).status_code == 200
# ------------------------------------------------------------ another site
EVIL = "http://evil.example"
def test_another_sites_page_cannot_post_to_an_unlocked_app(open_app: TestClient):
"""No session cookie to withhold on loopback, so `SameSite` protected
nothing: a `text/plain` POST needs no preflight and reached every route
that takes no body."""
assert open_app.post("/api/account/sessions/close").status_code == 200
assert open_app.post("/api/account/sessions/close",
headers={"origin": "http://127.0.0.1"}).status_code == 200
assert open_app.post("/api/account/sessions/close",
headers={"sec-fetch-site": "same-origin"}).status_code == 200
for headers in ({"origin": EVIL}, {"origin": "null"},
{"sec-fetch-site": "cross-site"}, {"sec-fetch-site": "same-site"},
{"origin": "http://127.0.0.1", "sec-fetch-site": "cross-site"}):
refused = open_app.post("/api/account/sessions/close", headers=headers)
assert refused.status_code == 403, headers
assert "another site" in refused.json()["detail"]
plain = open_app.post("/api/account/sessions/close", content=b"x",
headers={"content-type": "text/plain"})
assert plain.status_code == 415 and "application/json" in plain.json()["detail"]
form = open_app.post("/api/account/sessions/close", content=b"a=1",
headers={"content-type": "application/x-www-form-urlencoded"})
assert form.status_code == 415
# A read is never refused, wherever it came from.
assert open_app.get("/api/thing", headers={"origin": EVIL}).status_code == 200
def test_a_transfer_encoding_header_is_a_body_whatever_its_value():
"""Its presence says a body follows; an empty value is malformed and is
not given the benefit of the doubt."""
for value in (b"chunked", b""):
scope = {"type": "http", "method": "POST",
"headers": [(b"content-type", b"text/plain"),
(b"transfer-encoding", value)]}
refused = cross_site(scope)
assert refused is not None and refused[0] == 415, value
scope["headers"] = [(b"content-type", b"text/plain")]
assert cross_site(scope) is None
def test_another_sites_page_cannot_open_the_socket(open_app: TestClient):
"""A WebSocket handshake is not subject to CORS, so any page could open
`/ws/jobs` on a loopback app and read every job and notice."""
with pytest.raises(WebSocketDisconnect), \
open_app.websocket_connect("/ws/jobs", headers={"origin": EVIL}):
pass
# The test client's handshake names `testserver` whatever the base URL,
# so the app's own page is spelt out: a loopback host, and its origin.
with open_app.websocket_connect("/ws/jobs", headers={
"host": "127.0.0.1", "origin": "http://127.0.0.1"}) as ws:
assert ws.receive_json()["type"] == "jobs"
def test_the_configured_origin_is_accepted_behind_a_proxy_that_rewrites_host(
make_client: Callable[..., TestClient]):
client = make_client(enabled="auto", host="127.0.0.1", password=None,
origin="https://app.example.com", base_url="http://127.0.0.1")
assert client.post("/api/account/sessions/close",
headers={"origin": "https://app.example.com"}).status_code == 200
assert client.post("/api/account/sessions/close",
headers={"origin": EVIL}).status_code == 403
def test_the_rule_holds_on_a_locked_app_too(signed_in: TestClient):
"""A locked app with `secure_cookie: never` on a LAN is a real deployment,
and the two header reads cost nothing."""
assert signed_in.post("/api/account/sessions/close",
headers={"origin": EVIL}).status_code == 403
assert signed_in.post("/api/account/sessions/close").status_code == 200
# ------------------------------------------------- what a token must not reach
def test_an_app_names_what_a_token_may_not_reach(tmp_path):
"""A route that hands out or replaces the database as a file holds the
password hash and the token rows, so `connect(session_only=...)` shuts it
to a token the way the account is shut."""
from fastapi import FastAPI
from fastapi.responses import PlainTextResponse
from auth import AuthConfig, AuthStore, connect
app = FastAPI()
db = tmp_path / "s.db"
store = AuthStore(db)
store.set_password(PASSWORD)
connect(app, store, AuthConfig(enabled="always"), host="0.0.0.0",
session_only=("/api/backups", "/api/backups/", "/api/restore"))
@app.get("/api/backups/{name}", response_class=PlainTextResponse)
def download(name: str) -> str:
return "the whole database"
@app.post("/api/restore", response_class=PlainTextResponse)
def restore() -> str:
return "swapped"
@app.get("/api/thing")
def thing() -> dict[str, bool]:
return {"ok": True}
_, token = store.create_token("cron", None)
with TestClient(app, follow_redirects=False) as client:
assert client.get("/api/thing", headers=bearer(token)).status_code == 200
assert client.get("/api/backups/x.db.gz", headers=bearer(token)).status_code == 403
assert client.post("/api/restore", headers=bearer(token)).status_code == 403
sign_in(client)
assert client.get("/api/backups/x.db.gz").status_code == 200
# ---------------------------------------------------------- a rebound name
def test_a_rebinding_page_is_refused_on_an_unlocked_app(open_app: TestClient):
"""A page on `rebind.example`, pointed at 127.0.0.1, sends a `Host` and
an `Origin` that agree, and `Sec-Fetch-Site: same-origin`, so the
same-site rule cannot tell it from the app's own page. The name can:
a loopback app is reached as localhost, and a script naming anything
else for it is refused with the proxy's 421."""
page = {"host": "rebind.example:8787", "origin": "http://rebind.example:8787",
"sec-fetch-site": "same-origin"}
refused = open_app.post("/api/account/tokens", json={"current": ""}, headers=page)
assert refused.status_code == 421
assert "localhost" in refused.json()["detail"]
assert open_app.get("/api/thing", headers=page).status_code == 421
with pytest.raises(WebSocketDisconnect), \
open_app.websocket_connect("/ws/jobs", headers=page):
pass
# The app's own page, at any loopback name, is untouched, and so is a
# client that is not a page when it names one.
for host in ("localhost:8787", "127.0.0.1", "[::1]:8787", "app.localhost"):
own = {"host": host, "origin": f"http://{host}", "sec-fetch-site": "same-origin"}
assert open_app.get("/api/thing", headers=own).status_code == 200, host
assert open_app.get("/api/thing", headers={"host": host}).status_code == 200, host
assert open_app.post("/api/account/sessions/close",
headers={"host": "localhost"}).status_code == 200
def test_a_rebound_request_without_page_headers_is_refused_too(open_app: TestClient):
"""A browser that sends no `Sec-Fetch-Site`, and no `Origin` on a read, is
still a rebound page reading the API. The `Host` is judged whatever else
the request carries, so a name that is not loopback is refused, and so is
a `Host` that names nothing."""
for host in ("rebind.example", "rebind.example:8787", "testserver", "["):
refused = open_app.get("/api/thing", headers={"host": host})
assert refused.status_code == 421, host
assert open_app.post("/api/account/sessions/close",
headers={"host": "rebind.example"}).status_code == 421
with pytest.raises(WebSocketDisconnect), \
open_app.websocket_connect("/ws/jobs", headers={"host": "rebind.example"}):
pass
assert _rebound({"type": "http", "headers": []}, "") is True
def test_the_configured_origin_is_a_name_the_unlocked_app_answers_to(
make_client: Callable[..., TestClient]):
client = make_client(enabled="auto", host="127.0.0.1", password=None,
origin="https://app.example.com")
page = {"host": "app.example.com", "origin": "https://app.example.com",
"sec-fetch-site": "same-origin"}
assert client.get("/api/thing", headers=page).status_code == 200
page["host"] = page["origin"] = "other.example.com"
assert client.get("/api/thing", headers=page).status_code == 421
# -------------------------------------------------------- a cookie that is not one
def test_a_cookie_header_that_is_not_one_is_read_as_no_cookie(locked: TestClient):
"""The middleware is in front of every request, so a header it cannot
read must be no cookie and not a 500 on every call that carried it."""
for header in ("a@b=1", ";;;", "=", '"', "app_session=", "app_session"):
refused = locked.get("/api/thing", headers={"cookie": header})
assert refused.status_code == 401, header
def test_a_malformed_neighbour_cookie_does_not_sign_the_owner_out(signed_in: TestClient):
"""`SimpleCookie.load` returned an empty jar for the whole header when any
cookie in it had a space or JSON in its value, so a sibling app on the
parent domain setting `theme=dark mode` locked the owner into a login
loop. Read per pair, the session beside it still opens the door."""
from http.cookies import SimpleCookie
session = signed_in.cookies["app_session"]
jar = SimpleCookie()
jar.load(f"theme=dark mode; app_session={session}")
assert "app_session" not in jar, "the stdlib parser was the bug"
browser = TestClient(signed_in.app, follow_redirects=False)
for neighbour in ("theme=dark mode", 'x={"a":1}'):
for header in (f"{neighbour}; app_session={session}",
f"app_session={session}; {neighbour}"):
answer = browser.get("/api/thing", headers={"cookie": header})
assert answer.status_code == 200, header
+1 -1
View File
@@ -55,7 +55,7 @@ def test_the_passkey_routes_answer_501_rather_than_crashing(
"""501 is what the element watches for -- "this route exists and this build """501 is what the element watches for -- "this route exists and this build
cannot perform it" -- and it takes the passkey section off the screen.""" cannot perform it" -- and it takes the passkey section off the screen."""
for path in ("/api/account/passkeys/options", "/login/passkey/options"): for path in ("/api/account/passkeys/options", "/login/passkey/options"):
answer = signed_in.post(path) answer = signed_in.post(path, json={"current": PASSWORD})
assert answer.status_code == 501, path assert answer.status_code == 501, path
assert "passkeys" in answer.json()["detail"] assert "passkeys" in answer.json()["detail"]
assert "uv sync" in answer.json()["detail"] assert "uv sync" in answer.json()["detail"]
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
# The backend in the foreground until Ctrl-C: uv run app serve, on :8790. The profile's <name>-backend runs it.
# Ours: `-v` and `--config` go before `serve`, where the app takes them, and every other argument
# after it (`--reload`, `--port`), so `.run/serve.run.xml`'s `serve --reload` is `<name>-backend --reload`.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
# A shell started by wsl.exe has not read the rc file that puts uv on PATH.
export PATH="$HOME/.local/bin:$PATH"
before=() after=()
while (($#)); do
case $1 in
-v|--verbose) before+=("$1") ;;
-c|--config) before+=("$1" "${2:?$1 needs a file}"); shift ;;
--config=*) before+=("$1") ;;
*) after+=("$1") ;;
esac
shift
done
exec uv run app "${before[@]}" serve "${after[@]}"
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bash
# The local gate on the working tree, uncommitted edits included, without promoting:
# the steps deploy/test-commit.sh runs on a commit; the profile's <name>-check runs it.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
exec sh deploy/test-commit.sh --here
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
# roledesk's own, beside the boilerplate's backend and check: one pull in the foreground, `app job pull`.
# The profile's <name>-pull runs it; `.run/pull.run.xml` is the same command from PyCharm.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
# A shell started by wsl.exe has not read the rc file that puts uv on PATH.
export PATH="$HOME/.local/bin:$PATH"
exec uv run app job pull "$@"
+8
View File
@@ -0,0 +1,8 @@
#!/usr/bin/env bash
# roledesk's own, beside the boilerplate's backend and check: the queue drained with no web app, `app worker`.
# The profile's <name>-worker runs it; `.run/worker.run.xml` is the same command from PyCharm.
set -euo pipefail
cd "$(dirname "${BASH_SOURCE[0]}")/.."
# A shell started by wsl.exe has not read the rc file that puts uv on PATH.
export PATH="$HOME/.local/bin:$PATH"
exec uv run app worker "$@"
-23
View File
@@ -1,23 +0,0 @@
This skill adapts rules from ai-job-search (https://github.com/MadsLorentzen/ai-job-search), used under the MIT License:
MIT License
Copyright (c) 2026 Mads Lorentzen
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Binary file not shown.
Binary file not shown.
-14
View File
@@ -1,14 +0,0 @@
<#
.SYNOPSIS
Run steps 1, 2 and 3: test/promote, build image, push image.
.DESCRIPTION
Stops on the first failure. Server deployment is a separate operation --
nothing here connects to the server. Git branch publication is optional
through -PushBranch.
#>
[CmdletBinding()]
param([switch] $PushBranch)
$ErrorActionPreference = 'Stop'
& "$PSScriptRoot/1-test-and-promote.ps1" -PushBranch:$PushBranch
& "$PSScriptRoot/2-build-image.ps1"
& "$PSScriptRoot/3-push-image.ps1"
-90
View File
@@ -1,90 +0,0 @@
<#
.SYNOPSIS
Test the latest local main commit and fast-forward production to it.
.DESCRIPTION
Uses a fixed commit and an isolated worktree. PushBranch optionally publishes
production to origin; without it promotion stays local. Does not run Docker.
#>
[CmdletBinding()]
param([switch] $PushBranch, [switch] $NoTests)
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
. "$PSScriptRoot/_common.ps1"
$Source = Setting SOURCE 'main'
$Branch = Setting BRANCH 'production'
$Python = Setting PYTHON (Join-Path $root '.venv\Scripts\python.exe')
Push-Location $root
try {
$commit = git rev-parse --verify "$Source^{commit}"
Assert-Ok 'resolve source commit'
$commit = $commit.Trim()
# ------------------------------------------------------- test the commit
# In a throwaway worktree, not here. The tests then run against exactly the
# tree that is about to become production, with none of the working copy's
# uncommitted edits in the way. Worktrees have no .venv of their own, so the
# interpreter comes from the main checkout and the code from the worktree.
#
# A checkout inside WSL has a Linux .venv, so there the same steps run in
# the distro through deploy/test-commit.sh.
if (-not $NoTests -and $Wsl.Success) {
Write-Step "tests on $Source, in WSL $($Wsl.Groups[1].Value)"
wsl.exe -d $Wsl.Groups[1].Value --cd (ConvertTo-LinuxPath $root) -- sh deploy/test-commit.sh $commit
Assert-Ok 'tests'
} elseif (-not $NoTests) {
Write-Step "tests on $Source"
if (-not (Test-Path $Python)) {
throw "no interpreter at $Python - set PYTHON in deploy\release.env"
}
$base = Join-Path ([IO.Path]::GetTempPath()) ("$Name-test-" + [guid]::NewGuid().ToString('N').Substring(0, 8))
$tree = Join-Path $base $Name
try {
git worktree add --detach $tree $commit | Out-Null
Assert-Ok 'git worktree add'
Push-Location $tree
try {
# Your project's checks. A gate that is red for weeks is not a
# gate -- it is a flag you learn to type -- so take one out when
# it stops meaning anything, and put it back when it is green.
#
# The starter's dev group is pytest and httpx, so pytest is the
# only gate that can run out of the box. Add ruff and mypy to
# [dependency-groups] dev in pyproject.toml, then uncomment the
# two lines below -- a gate whose tool is not installed fails
# every release for a reason that has nothing to do with the code.
& $Python -m pytest -q; Assert-Ok 'pytest'
# & $Python -m ruff check .; Assert-Ok 'ruff'
# & $Python -m mypy --cache-dir (Join-Path $root '.mypy_cache')
# Assert-Ok 'mypy'
} finally { Pop-Location }
} finally {
git worktree remove --force $tree 2>$null | Out-Null
Remove-ReleaseTemp @($base)
}
}
# ------------------------------------------------------------- promote
# Fast-forward only. If production has commits main does not, that is a
# release made from somewhere else, and this stops rather than discarding it.
Write-Step "promote $Source -> $Branch"
git show-ref --verify --quiet "refs/heads/$Branch"
if ($LASTEXITCODE -eq 0) {
git merge-base --is-ancestor $Branch $commit
if ($LASTEXITCODE -ne 0) {
Write-Host "$Branch is not an ancestor of $Source - it has commits of its own." -ForegroundColor Red
throw "look at: git log --oneline $Source..$Branch"
}
}
# Git publication is optional and independent of the image.
git branch -f $Branch $commit; Assert-Ok 'git branch -f'
if ($PushBranch) {
git push origin "${commit}:refs/heads/$Branch"
Assert-Ok 'publish production branch'
}
Write-Host "Promoted $Branch to $commit"
} finally { Pop-Location }
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env bash
# Test the latest local main commit, fast-forward production to it, and export
# it for the image build on Windows.
#
# bash deploy/1-test-and-promote.sh [--push-branch] [--no-tests]
#
# Uses a fixed commit and an isolated worktree. --push-branch optionally
# publishes production to origin; without it promotion stays local. --no-tests
# skips the gates. The export is the production tree in <RELEASE_DIR>/<folder>/<sha>/
# and its sha in <RELEASE_DIR>/<folder>/production, RELEASE_DIR defaulting to
# release/ beside the checkout. Uncommitted work is never included.
set -euo pipefail
# shellcheck source=_common.sh
. "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
push_branch=0 no_tests=0
for arg in "$@"; do
case "$arg" in
--push-branch) push_branch=1 ;;
--no-tests) no_tests=1 ;;
-h|--help) sed -n '2,11p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) die "unknown option $arg" ;;
esac
done
need_git
SOURCE=$(setting SOURCE main)
BRANCH=$(setting BRANCH production)
PYTHON=$(setting PYTHON "$root/.venv/bin/python")
cd "$root"
commit=$(resolve_commit "$SOURCE" source)
# ----------------------------------------------------------- test the commit
# In a throwaway worktree, not here. The tests then run against exactly the tree
# that is about to become production, with none of the working copy's
# uncommitted edits in the way. test-commit.sh makes the worktree, runs pytest,
# ruff, mypy and the client's check in it, and removes it. Worktrees have no
# .venv of their own, so the interpreter comes from this checkout.
if ((! no_tests)); then
step "tests on $SOURCE"
run_or_die tests env PYTHON="$PYTHON" sh "$here/test-commit.sh" "$commit"
fi
# ------------------------------------------------------------------- promote
# Fast-forward only. If production has commits main does not, that is a release
# made from somewhere else, and this stops rather than discarding it.
step "promote $SOURCE -> $BRANCH"
if git show-ref --verify --quiet "refs/heads/$BRANCH"; then
if ! git merge-base --is-ancestor "$BRANCH" "$commit"; then
echo "$BRANCH is not an ancestor of $SOURCE - it has commits of its own." >&2
die "look at: git log --oneline $SOURCE..$BRANCH"
fi
fi
# Git publication is optional and independent of the image.
run_or_die 'git branch -f' git branch -f "$BRANCH" "$commit"
if ((push_branch)); then
run_or_die 'publish production branch' git push origin "$commit:refs/heads/$BRANCH"
fi
echo "Promoted $BRANCH to $commit"
# -------------------------------------------------------------------- export
# The image is built on Windows by wslc (2-build-image.ps1), which reads this
# folder through \\wsl.localhost. A tree rather than a tar, so Windows has
# nothing to unpack. wslc hands the Dockerfile to the builder on stdin, so the
# builder only finds the ignore rules at the root of the context:
# Dockerfile.dockerignore is copied there as .dockerignore. A tree that exists
# for this sha is kept, not rewritten, because wslc can build from a stale view
# of a folder deleted and recreated at the same path (microsoft/WSL#41287).
# The promote has happened by now; when the export fails, rerun this step.
name=$(basename "$root")
release=${RELEASE_DIR:-$(dirname "$root")/release}/$name
sha=$(git rev-parse --short "$commit") || die 'resolve image tag failed'
tree=$release/$sha
step "export $BRANCH ($sha) to $tree"
mkdir -p -- "$release" 2>/dev/null && [[ -w $release ]] ||
die "cannot write $release; $BRANCH is promoted, rerun this step once it is writable"
if [[ ! -d $tree ]]; then
part=$release/.part-$sha
rm -rf -- "$part"
mkdir -- "$part"
git archive --format=tar "$commit" | tar -x -C "$part" || die 'export failed'
if [[ -f $part/deploy/Dockerfile.dockerignore ]]; then
cp -- "$part/deploy/Dockerfile.dockerignore" "$part/.dockerignore"
fi
mv -- "$part" "$tree"
fi
for old in "$release"/*/; do
[[ ${old%/} == "$tree" ]] || rm -rf -- "${old%/}"
done
printf '%s\n' "$sha" > "$release/production"
echo "Exported $tree"
+12 -56
View File
@@ -1,57 +1,13 @@
<# # Step 2, on Windows: build the image from the tree step 1 exported, with wslc.
.SYNOPSIS # Tags <IMAGE>:<sha> and <IMAGE>:latest. Never pushes; 3-push-image.ps1 does.
Build a local Docker image from the committed production branch.
.DESCRIPTION
Does not test, promote, push Git branches, use SSH, or start a container.
The image stays local. Use 3-push-image.ps1 to publish it separately.
#>
[CmdletBinding()]
param()
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop' $ErrorActionPreference = 'Stop'
. "$PSScriptRoot/_common.ps1" . "$PSScriptRoot\_common.ps1"
Assert-Wslc
$Image = Setting IMAGE -Required $sha = Get-ReleaseSha
$Branch = Setting BRANCH 'production' $context = Join-Path $Release $sha
$Platform = Setting PLATFORM 'linux/amd64' Write-Host "== build ${Image}:$sha from $context" -ForegroundColor Cyan
# -f is a path wslc opens itself, not one inside the context.
Push-Location $root wslc build -t "${Image}:$sha" -t "${Image}:latest" -f "$context\deploy\Dockerfile" $context
try { if ($LASTEXITCODE -ne 0) { throw "wslc build failed (exit $LASTEXITCODE)" }
$commit = git rev-parse --verify "$Branch^{commit}" wslc image list
Assert-Ok 'resolve production commit' Write-Host "Image: ${Image}:$sha"
$commit = $commit.Trim()
$sha = (git rev-parse --short $commit).Trim()
Assert-Ok 'resolve image tag'
$tagged = "${Image}:${sha}"
$latest = "${Image}:latest"
# --------------------------------------------------------------- build
# `git archive` to a file and unpack it, rather than piping it. A PowerShell
# pipeline carries text: it would re-encode a tar stream and hand docker a
# corrupted context. Writing the archive out and building from the directory
# keeps the same guarantee -- the context is the committed tree and nothing
# else -- without a byte going through a pipe.
#
# The Dockerfile comes out of that same tree, at deploy/Dockerfile, so the
# recipe is versioned with the code it builds. Docker reads the ignore rules
# from deploy/Dockerfile.dockerignore beside it.
Write-Step "build $tagged"
$ctx = Join-Path ([IO.Path]::GetTempPath()) "$Name-build-$sha"
$tar = "$ctx.tar"
Remove-ReleaseTemp @($ctx, $tar)
New-Item -ItemType Directory -Path $ctx | Out-Null
try {
git archive --format=tar -o $tar $commit; Assert-Ok 'git archive'
tar -xf $tar -C $ctx; Assert-Ok 'tar -xf'
$dockerfile = Join-Path $ctx 'deploy/Dockerfile'
docker build --platform $Platform -f $dockerfile -t $tagged -t $latest $ctx
Assert-Ok 'docker build'
docker image ls $Image --format ' {{.Tag}} {{.Size}}'
} finally {
Remove-ReleaseTemp @($ctx, $tar)
}
Write-Host "Image: $tagged"
} finally { Pop-Location }
+19 -35
View File
@@ -1,36 +1,20 @@
<# # Step 3, on Windows: push the image step 2 built, <IMAGE>:<sha> and :latest,
.SYNOPSIS # with wslc. Never rebuilds: after a failed push, fix the login and run it again.
Publish the already-built production image to the registry.
.DESCRIPTION
Never rebuilds. If the push fails, fix access or connectivity and run this
again; the image is already on disk. `docker login` first, with a token
that has package-write access.
#>
[CmdletBinding()]
param()
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop' $ErrorActionPreference = 'Stop'
. "$PSScriptRoot/_common.ps1" . "$PSScriptRoot\_common.ps1"
Assert-Wslc
$Image = Setting IMAGE -Required $sha = Get-ReleaseSha
$Branch = Setting BRANCH 'production' wslc image inspect "${Image}:$sha" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "no local ${Image}:$sha - run step 2 first" }
Push-Location $root # latest must name this candidate, even if another image was built since.
try { wslc tag "${Image}:$sha" "${Image}:latest"
$commit = git rev-parse --verify "$Branch^{commit}" if ($LASTEXITCODE -ne 0) { throw "wslc tag failed (exit $LASTEXITCODE)" }
Assert-Ok 'resolve production commit' foreach ($tag in $sha, 'latest') {
$sha = (git rev-parse --short $commit.Trim()).Trim() Write-Host "== push ${Image}:$tag" -ForegroundColor Cyan
Assert-Ok 'resolve image tag' wslc push "${Image}:$tag"
$tagged = "${Image}:${sha}" if ($LASTEXITCODE -ne 0) {
$latest = "${Image}:latest" Write-Host "If the push was refused: wslc login $(($Image -split '/')[0]) -u <user>, with a token that has package-write access" -ForegroundColor Yellow
docker image inspect $tagged --format '{{.Id}}' | Out-Null throw "wslc push ${Image}:$tag failed (exit $LASTEXITCODE)"
Assert-Ok "find local $tagged (run step 2 first)" }
# latest must name this candidate, even if another image was built since. }
docker tag $tagged $latest Write-Host "Published ${Image}:$sha and ${Image}:latest"
Assert-Ok 'tag latest image'
docker push $tagged
Assert-Ok 'push commit image'
docker push $latest
Assert-Ok 'push latest image'
Write-Host "Published $tagged and $latest"
} finally { Pop-Location }
+50 -16
View File
@@ -1,21 +1,28 @@
# A Python app, for a server rather than a laptop. # A Python app, for a server rather than a laptop.
# #
# Two stages so the runtime carries no build tooling: `uv` resolves and # Two stages so the runtime carries no build tooling: `uv` resolves and
# installs into /app/.venv, and the second stage copies that venv and the # installs into /app/.venv, and the second stage copies the venv and the source
# source next to it. The install is deliberately editable, which matters if # next to it. The install is deliberately editable, which matters because the
# the app reads a file from beside its package (an example config, a template # app reads files from beside its package (templates, `app/static`, `seed/`):
# directory): that path is true in a checkout and true here, and false in # that path is true in a checkout and true here, and false in site-packages.
# site-packages. # Ours: roledesk has no `frontend/` and no Node stage, since the server renders
# Jinja templates and serves `app/static` as it is.
# #
# Already in place at deploy/Dockerfile, with its ignore file beside it. # Already in place at deploy/Dockerfile, with its ignore file beside it.
# 2-build-image.ps1 reads it out of a `git archive` of the production commit, # 2-build-image.ps1 builds with it from the `git archive` of the production
# so the recipe is versioned with the code it builds. # commit that step 1 exports, so the recipe is versioned with the code it builds.
# #
# The package is `app` and stays `app` — a project started from this repo # The package is `app` and stays `app` (the domain noun is `role`). Edit the
# renames nothing. Edit the two `uv sync` lines when you add extras. # two `uv sync` lines when you add extras.
#
# Each base image is pinned by tag and digest: the tag says which release, the
# digest makes a rebuild next month install the same bytes. The digest is the
# multi-platform index's. To move on, look up the new tag's digest on its
# registry (Docker Hub's tag API for python, ghcr.io for uv) and
# replace both halves.
ARG PYTHON=3.14-slim ARG PYTHON=3.15-slim@sha256:2da676046630d7bf14c6764ae1143dc7fa0377a53f94bf5a642f0b214aca5bce
ARG UV=0.9.17 ARG UV=0.13.1@sha256:4a1f3666816dafadd20ceb71eac9a7d56d7ef30279f7d64048519ab901ada336
FROM ghcr.io/astral-sh/uv:${UV} AS uv FROM ghcr.io/astral-sh/uv:${UV} AS uv
@@ -32,22 +39,38 @@ WORKDIR /app
# Dependencies first and on their own layer, keyed on the lock file alone, so # Dependencies first and on their own layer, keyed on the lock file alone, so
# editing a Python file rebuilds in seconds rather than re-resolving the tree. # editing a Python file rebuilds in seconds rather than re-resolving the tree.
# The extras are the same on both `uv sync` lines: `passkeys`, without which
# the passkey routes answer 501, `embeddings`, the default ranking backend
# (torch comes from the CPU index named in pyproject.toml, [tool.uv.sources],
# not the CUDA build; the model itself is downloaded by the first `embed` job
# into HOME, which is the /data volume), and `llm`, the SDKs the compose file
# passes keys for.
COPY pyproject.toml uv.lock README.md ./ COPY pyproject.toml uv.lock README.md ./
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev --no-install-project uv sync --frozen --no-dev --no-install-project \
--extra passkeys --extra embeddings --extra llm
COPY app app COPY app app
# The two module folders this repo ships beside the app, here and again in the # The module folders this repo ships beside the app, here and again in the
# runtime stage below. `pyproject.toml` lists `pipeline*` and `auth*` in its # runtime stage below. `pyproject.toml` lists `pipeline*`, `auth*`, `llm*` and
# packages as well, or the install leaves them out. # `desk*` in its packages as well, or the install leaves them out.
COPY pipeline pipeline COPY pipeline pipeline
# The door. A project that does not want one drops this line *and* the # The door. A project that does not want one drops this line *and* the
# `auth.connect` call in `app/main.py` together: dropping the folder and # `auth.connect` call in `app/main.py` together: dropping the folder and
# leaving the call is a container that fails at startup, and dropping the call # leaving the call is a container that fails at startup, and dropping the call
# and leaving the folder is a container that starts with the door open. # and leaving the folder is a container that starts with the door open.
COPY auth auth COPY auth auth
# The model call. Nothing imports it until an app adds its SDKs, and then the
# folder has to be here already, so that is not a Dockerfile edit.
COPY llm llm
# Ours: the source contract and the generic desk modules, the plugin folder the
# config's `plugin_paths` names, and the data a first run imports (`seed/`).
COPY desk desk
COPY plugins plugins
COPY seed seed
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev uv sync --frozen --no-dev \
--extra passkeys --extra embeddings --extra llm
# ---------------------------------------------------------------- runtime # ---------------------------------------------------------------- runtime
@@ -69,6 +92,10 @@ COPY pipeline pipeline
# The login page and the two scripts are files inside this folder, served from # The login page and the two scripts are files inside this folder, served from
# routes; see the build stage above. # routes; see the build stage above.
COPY auth auth COPY auth auth
COPY llm llm
COPY desk desk
COPY plugins plugins
COPY seed seed
ENV PATH=/app/.venv/bin:$PATH \ ENV PATH=/app/.venv/bin:$PATH \
PYTHONUNBUFFERED=1 \ PYTHONUNBUFFERED=1 \
@@ -79,6 +106,13 @@ ENV PATH=/app/.venv/bin:$PATH \
# /data, and a CLI the app shells out to — the `claude` binary, for one — # /data, and a CLI the app shells out to — the `claude` binary, for one —
# wants somewhere to write. An image-baked /home/app would be right for uid # wants somewhere to write. An image-baked /home/app would be right for uid
# 1000 and unwritable for anyone else. # 1000 and unwritable for anyone else.
#
# The image runs as 1000:1000, the compose file's default PUID:PGID and the
# owner docs/Install Server.md gives the data folder, so a `docker run` without a
# `user:` does not write root-owned files into it. /data is made here, before
# VOLUME, so an anonymous volume starts out owned by that uid too.
RUN install -d -o 1000 -g 1000 /data /data/home
USER 1000:1000
VOLUME /data VOLUME /data
EXPOSE 8787 EXPOSE 8787
+10 -8
View File
@@ -1,12 +1,13 @@
# Named for the Dockerfile beside it: Docker reads `<dockerfile>.dockerignore` # Named for the Dockerfile beside it. Docker reads `<dockerfile>.dockerignore`
# before falling back to one at the root of the context, which is how the ignore # before falling back to one at the root of the context, but wslc hands the
# rules get to live next to the recipe they belong to instead of at the top of # Dockerfile to the builder on stdin, which leaves only the root one: step 1
# the repository. # (1-test-and-promote.sh) copies this file to `.dockerignore` at the root of
# the tree it exports, so the rules still live next to the recipe here.
# #
# 2-build-image.ps1 already hands docker a context extracted from `git archive`, # That tree is a `git archive` of the production commit, so the untracked
# so the untracked heavyweights below are not there to exclude. They are listed # heavyweights below are not there to exclude. They are listed for the other
# for the other way in — `docker build -f deploy/Dockerfile .` from a checkout — # way in, `docker build -f deploy/Dockerfile .` from a checkout, where .venv
# where .venv alone can be several hundred megabytes. # alone can be several hundred megabytes.
.git .git
.venv .venv
.claude .claude
@@ -19,6 +20,7 @@ docs
# per folder, so a fourth module is a fourth line here. # per folder, so a fourth module is a fourth line here.
tests tests
pipeline/tests pipeline/tests
llm/tests
auth/tests auth/tests
__pycache__ __pycache__
+45 -21
View File
@@ -1,52 +1,76 @@
# Deploy # Deploy
Everything needed to put this app on the rittershofer.org box behind SWAG: build an image on Windows, publish it to Gitea, run it as one more service in the compose file that already runs the proxy, back up its database, and give it a password that survives being on the open internet. Everything needed to put roledesk on a server behind SWAG ([../docs/Install Server.md](../docs/Install%20Server.md) is the box it was written for, a copy of the boilerplate's file): test it in WSL, build an image on Windows with `wslc`, publish it to Gitea, run it as one more service in the compose file that already runs the proxy, back up its database, and give it a password that survives being on the open internet.
Extracted from newsdesk, which is the worked example — every decision here has been run in production at least once, and the comments say why rather than what. This folder is a downstream copy of `../boilerplate/deploy` (commit 09e144d), refreshed with `git archive`; a fix to a shared script lands upstream first. roledesk's own facts are put back after a refresh: the Dockerfile without a frontend stage and with `desk`, `plugins`, `llm` and `seed`, the extras `passkeys`, `embeddings` and `llm`, our image name in `release.env.example`, `server/config.example.yaml` for roledesk's keys, `TABLES` and `REQUIRED_TABLES` naming `roles`, and `bin/pull.sh` and `bin/worker.sh`. `tests/test_deploy.py` pins them.
**It is already in place.** Nothing has to be copied and nothing has to be renamed: the package is `app`, the environment variables are `APP_`, and the files below are where the scripts expect them. `<app>` in the docs is your *service's* name — the hostname, the data directory, the image tag, the compose service — and it is the one thing you choose. The package is `app`, and the files below are where the scripts expect them: the `APP_` variables, the compose service `app` and the hostname `app.example.com` in the templates are placeholders for the server, and `<app>` in the docs is the service's name: the hostname, the data directory, the image tag, the compose service. The domain noun is `role`, so "job" only ever means queued work.
## Where each file is ## Where each file is
| Here | | | | Here | | |
|---|---|---| |---|---|---|
| `deploy/*.ps1` | the release scripts, run on Windows | | | `deploy/1-test-and-promote.sh`, `test-commit.sh`, `_common.sh`, `db.sh` | the gate, the export and the database moves, run in the WSL distro | |
| `deploy/Dockerfile` | read out of the archived commit by step 2 | copies `app`, `pipeline` and `auth` | | `deploy/2-build-image.ps1`, `3-push-image.ps1`, `_common.ps1` | the image build and push, run on Windows with `wslc` | the distro is a sealed sandbox and does not hold a container socket |
| `deploy/Dockerfile.dockerignore` | Docker finds it by that name | | | `deploy/wsl-setup.cmd`, `wsl-setup.ps1` | the profile commands, written once per checkout on Windows | |
| `bin/backend.sh`, `check.sh`, and ours `pull.sh`, `worker.sh` | the dev commands, run in the WSL distro: the backend, the gate on the working tree, a pull, the worker | |
| `.run/serve`, `pull`, `worker` | PyCharm run configurations, left as they are | `serve --reload` = `bin/backend.sh --reload` = `<folder>-backend --reload`; `pull` = `bin/pull.sh` = `<folder>-pull`; `worker` = `bin/worker.sh` = `<folder>-worker` |
| `deploy/Dockerfile` | read out of the tree step 1 exports, by step 2 | no frontend stage; copies `app`, `pipeline`, `auth`, `llm`, `desk`, `plugins` and `seed` |
| `deploy/Dockerfile.dockerignore` | copied to `.dockerignore` at the root of the exported tree | `wslc` streams the Dockerfile, so only a root ignore file applies |
| `deploy/server/*` | committed here, then installed on the server | | | `deploy/server/*` | committed here, then installed on the server | |
| `deploy/server/compose.yaml` | to `/srv/swag/docker-compose.yml`, as one `services:` block | | | `deploy/server/compose.yaml` | to `/srv/swag/docker-compose.yml`, as one `services:` block | |
| `deploy/server/env.example` | to `/srv/swag/.env`, as appended variables, `chmod 600` | | | `deploy/server/env.example` | to `/srv/swag/.env`, as appended variables, `chmod 600` | |
| `deploy/server/config.example.yaml` | to `$APP_DATA/config.yaml`, with the hostname filled in | `deploy.sh` refuses a config without `database` and `auth.enabled` |
| `deploy/server/deploy.sh`, `deploy/server/backup.sh` | to `/srv/swag/`, next to the compose file | | | `deploy/server/deploy.sh`, `deploy/server/backup.sh` | to `/srv/swag/`, next to the compose file | |
| `deploy/server/import.sh`, `deploy/server/import_database.py` | uploaded and run by `db.sh push` | |
| `deploy/server/swag/app.subdomain.conf` | to SWAG's `proxy-confs/`, renamed to `<app>.subdomain.conf` | | | `deploy/server/swag/app.subdomain.conf` | to SWAG's `proxy-confs/`, renamed to `<app>.subdomain.conf` | |
`db.ps1` expects `server/` beside it, which is why `deploy/server/` is committed here as well as installed on the server. `db.sh` expects `server/` beside it, which is why `deploy/server/` is committed here as well as installed on the server.
**Keep the `.ps1` files to ASCII.** Windows PowerShell 5.1 reads a script without a byte-order mark as Windows-1252, so an em dash becomes three characters ending in a closing quote: the string ends in the middle of itself and the script fails to parse before it runs a line. Write `--`. newsdesk hit this on a comment nobody thought was code, and `tests/test_packaging.py` here holds every script in this folder to it. **Keep every `.ps1` to ASCII.** Windows PowerShell 5.1 reads a script without a byte-order mark as Windows-1252, so an em dash becomes three characters ending in a closing quote: the string ends in the middle of itself and the script fails to parse before it runs a line. Write `--`. newsdesk hit this on a comment nobody thought was code, and `tests/test_packaging.py` here holds every `.ps1` in this folder to it, the release scripts included.
`wsl-setup.sh` and `wsl-setup.ps1` are the two halves of setting up a checkout that lives in WSL. `wsl-setup.sh` runs in the distro ([../docs/Install WSL.md, Git](../docs/Install%20WSL.md#git)). `wsl-setup.ps1`, or a double-click on `deploy\wsl-setup.cmd`, writes the commands below into the PowerShell profile, with `<folder>` the checkout folder's name (`jobboard` here) or `-Name`. PowerShell 7, the execution policy, `wslc`, the hand-paste fallback and what to check when a command fails are in [../docs/Install WSL.md, PS Profile commands](../docs/Install%20WSL.md#ps-profile-commands).
| Command | Runs | Does |
|---|---|---|
| `<folder>-cd` | `Set-Location` | PowerShell into the checkout's folder |
| `<folder>-sh` | `wsl.exe` | a shell in the distro, in the checkout |
| `<folder>-backend [-v] [--reload]` | bin/backend.sh in WSL | `uv run app serve` on :8790 |
| `<folder>-pull [args]` | bin/pull.sh in WSL | `uv run app job pull`, one pull in the foreground (roledesk's) |
| `<folder>-worker [args]` | bin/worker.sh in WSL | `uv run app worker`, the queue drained with no web app (roledesk's) |
| `<folder>-check` | bin/check.sh in WSL | the gate on the working tree, without promoting |
| `<folder>-1test [--push-branch] [--no-tests]` | deploy/1-test-and-promote.sh in WSL | test local main, fast-forward `production`, export it |
| `<folder>-2build` | deploy\2-build-image.ps1 on Windows | build the exported tree with `wslc` |
| `<folder>-3push` | deploy\3-push-image.ps1 on Windows | push the built image with `wslc` |
| `<folder>-0release [--push-branch] [--no-tests]` | the three above | steps 1, 2 and 3 in order, stopping at the first failure |
| `<folder>-db pull\|push` | deploy/db.sh in WSL | the database, to and from the server |
Each command is one line in the profile: `wsl.exe -d <DISTRO> --cd <CHECKOUT> -- bash <script> @args` for the ones that run in the distro, and for steps 2 and 3 the `.ps1` with the execution policy lifted for that run. `DISTRO` and `CHECKOUT` are read from `deploy/release.env` when the setup runs, so rerun it after either changes. `-backend`, `-pull` and `-worker` stay in the foreground until Ctrl-C. The `-0release` to `-3push` names carry the number of the script's file name, so `0<Tab>` after `<folder>-` finds them.
`.gitignore` already covers `deploy/release.env` and `deploy/backups/`. `.gitignore` already covers `deploy/release.env` and `deploy/backups/`.
## The docs ## The docs
They are one directory up, in [../docs/](../docs), because they are about the hosting rather than about these scripts. They are one directory up, in [../docs/](../docs), because they are about the hosting rather than about these scripts. `Install Server.md` and `Install WSL.md` are copies of the boilerplate's `documentation/` files; the two proposals stay in the boilerplate checkout.
- **[../docs/hosting.md](../docs/hosting.md)** — what the box is, the seven steps to add a service, what the templates already decide for you, and the things that have already gone wrong on it. Start here for a service that is not on the box yet. - **[../docs/Install Server.md](../docs/Install%20Server.md)** — what the box is, the seven steps to add a service, what the templates already decide for you, and the things that have already gone wrong on it. Start here for a service that is not on the box yet.
- **[../docs/release.md](../docs/release.md)** — the Windows side: `release.env`, the four numbered scripts, what each one refuses to do, and why the tests run in a throwaway worktree. - **[../docs/Install Server.md, Publishing a release](../docs/Install%20Server.md#publishing-a-release)** — the release side: `release.env`, the numbered scripts and where each runs, what each one refuses to do, and why the tests run in a throwaway worktree.
- **[../docs/server.md](../docs/server.md)** — installing, deploying, backups, moving a SQLite database in either direction, rolling back, and what to verify before relying on any of it. - **[../docs/Install Server.md, Operating it](../docs/Install%20Server.md#operating-it)** — installing, deploying, backups, moving a SQLite database in either direction, rolling back, and what to verify before relying on any of it.
- **[../docs/llm-backends.md](../docs/llm-backends.md)** — asking Claude from a deployed app: API key or subscription token, selected by which one is in the environment, with a self-contained reference implementation. - **[Proposal LLM Backends.md](../../boilerplate/documentation/Proposal%20LLM%20Backends.md)** (boilerplate checkout) — asking Claude from a deployed app: API key or subscription token, selected by which one is in the environment. `llm/` is the implementation of it, see [../llm/README.md](../llm/README.md).
- **[../docs/password-protection.md](../docs/password-protection.md)** — one password on a public hostname: the lockout, sessions, passkeys, and how much throttling is actually worth building. `auth/` is the implementation of it. - **[Proposal Password Protection.md](../../boilerplate/documentation/Proposal%20Password%20Protection.md)** (boilerplate checkout) — one password on a public hostname: the lockout, sessions, passkeys, and how much throttling is actually worth building. `auth/` is the implementation of it, see [../auth/README.md](../auth/README.md).
## A new service, in order ## A new service, in order
1. DNS, then certificate, then registry login, then data directory — [../docs/hosting.md](../docs/hosting.md). 1. DNS, then certificate, then registry login, then data directory — [../docs/Install Server.md](../docs/Install%20Server.md).
2. Copy `release.env.example` to `release.env` and fill it in. That is the only file this step touches: everything else is already committed. 2. Copy `release.env.example` to `release.env` and fill it in. That is the only file this step touches: everything else is already committed.
3. `.\deploy\0-release.ps1` to test, promote, build and publish. 3. `<folder>-0release` from Windows to test, promote, build and publish. `wslc` and its registry login are on Windows, [../docs/Install Server.md, Publishing a release](../docs/Install%20Server.md#publishing-a-release).
4. Install the server files, edit `.env`, `bash deploy.sh` — [../docs/server.md](../docs/server.md). 4. Install the server files and `config.yaml`, edit `.env`, `bash deploy.sh` — [../docs/Install Server.md, Operating it](../docs/Install%20Server.md#operating-it).
5. Put the model credential in `.env` if the app needs one — [../docs/llm-backends.md](../docs/llm-backends.md). 5. Put the model credential in `.env` if the app needs one — [../llm/README.md](../llm/README.md).
6. Get in, rotate the bootstrap password, register a passkey — [../docs/password-protection.md](../docs/password-protection.md). 6. Get in, rotate the bootstrap password, register a passkey — [../auth/README.md](../auth/README.md).
7. Schedule `backup.sh`, then restore one of its backups somewhere else to prove it works. 7. Schedule `backup.sh`, then restore one of its backups somewhere else to prove it works.
Step 1 of `0-release.ps1` runs `pytest` in a throwaway worktree, with `ruff` and `mypy` commented out beside it: this repository's dev group is pytest and httpx, and a gate whose tool is not installed fails every release for a reason that has nothing to do with the code. Add the tools, then uncomment the gates. Step 1 runs `pytest` in a throwaway worktree through `test-commit.sh`, with `ruff` and `mypy` commented out beside it, as the release has always run. The dev group has both, so uncommenting them is the owner's call: the tree has to pass them first.
## What this is not ## What this is not
Not a framework and not a generator: nothing here is imported or extended, it is copied and edited. That is the point — three services with three edited copies beats three services sharing one abstraction that has to be right for all of them. What this repository changes is only *where the copy comes from*: `boilerplate/deploy/` is the source of truth, and news and health hold copies of it. Not a framework: nothing here is imported or extended, it is copied and edited. `boilerplate/deploy/` is the source of truth, and news, health, jobboard and pocketmoney hold copies of it.
+22 -66
View File
@@ -1,76 +1,32 @@
<# # Dot-sourced by 2-build-image.ps1 and 3-push-image.ps1. Reads deploy\release.env
.SYNOPSIS # beside it (KEY=VALUE lines, # comments, never run as code) into $Settings,
Shared plumbing for the release scripts. Dot-source it; do not run it. # and works out where step 1 exported the release: release\<folder> beside the
# checkout, in the distro, seen from Windows through \\wsl.localhost.
.DESCRIPTION
Dot-sourcing runs in the caller's scope, so everything defined here --
$root, $settings, Setting, Assert-Ok, Write-Step, Remove-ReleaseTemp -- is
the caller's afterwards. One copy of the release.env reader, because three
copies is three places for it to drift.
#>
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop' $ErrorActionPreference = 'Stop'
#: The repository root, worked out from this file rather than from the working
#: directory, so the scripts can be run from anywhere.
$root = Split-Path -Parent $PSScriptRoot
#: A checkout inside WSL, run from Windows as \\wsl.localhost\<distro>\... Its
#: .venv is a Linux one, so the steps that run Python hand that part to the
#: distro. $Wsl.Groups[1] is the distro; ConvertTo-LinuxPath turns a path under
#: the checkout into the distro's own. See docs/wsl.md.
$WslPattern = '^\\\\wsl(?:\.localhost|\$)\\([^\\]+)(\\.*)$'
$Wsl = [regex]::Match($root, $WslPattern)
function ConvertTo-LinuxPath([string] $Path) {
[regex]::Match($Path, $WslPattern).Groups[2].Value -replace '\\', '/'
}
function Assert-Ok([string] $What) {
if ($LASTEXITCODE -ne 0) { throw "$What failed (exit $LASTEXITCODE)" }
}
function Write-Step([string] $Text) {
Write-Host ''
Write-Host "== $Text" -ForegroundColor Cyan
}
$envFile = Join-Path $PSScriptRoot 'release.env' $envFile = Join-Path $PSScriptRoot 'release.env'
if (-not (Test-Path $envFile)) { if (-not (Test-Path $envFile)) { throw "no $envFile - copy release.env.example and fill it in" }
throw "no deploy\release.env - copy deploy\release.env.example and fill it in" $Settings = @{}
}
$settings = @{}
foreach ($line in Get-Content $envFile) { foreach ($line in Get-Content $envFile) {
if ($line -match '^\s*(#|$)') { continue } if ($line -match '^\s*(#|$)') { continue }
# A line with no `=` is a typo, not a setting. Say so here rather than
# letting it surface later as a null reference three functions away.
if ($line -notmatch '=') { throw "release.env: not a KEY=VALUE line: $line" }
$key, $value = $line -split '=', 2 $key, $value = $line -split '=', 2
$settings[$key.Trim()] = $value.Trim().Trim('"') $Settings[$key.Trim()] = "$value".Trim().Trim('"')
} }
foreach ($key in 'DISTRO', 'CHECKOUT', 'IMAGE') {
function Setting([string] $Name, [string] $Default, [switch] $Required) { if (-not $Settings[$key]) { throw "set $key in $envFile" }
if ($settings.ContainsKey($Name) -and $settings[$Name]) { return $settings[$Name] }
if ($Required) { throw "set $Name in deploy\release.env" }
return $Default
} }
$Image = $Settings.IMAGE
$Checkout = $Settings.CHECKOUT.TrimEnd('/')
$Release = "\\wsl.localhost\$($Settings.DISTRO)" + ($Checkout -replace '/[^/]+$', '' -replace '/', '\') + '\release\' + ($Checkout -replace '^.*/', '')
#: What temporary directories this release is allowed to delete. Everything function Assert-Wslc {
#: below is created by these scripts and named for it. if (-not (Get-Command wslc -ErrorAction SilentlyContinue)) {
$Name = Setting NAME -Required throw 'wslc is not on PATH - run wsl --update (WSL 2.9.3 or later), see https://learn.microsoft.com/windows/wsl/tutorials/wsl-containers'
# Only ever deletes a <name>-* entry directly under the temp directory. A
# guard, not a convenience: these paths are built by string concatenation and
# handed to `Remove-Item -Recurse -Force`, and one empty variable in that
# sentence is a very bad afternoon.
function Remove-ReleaseTemp([string[]] $Paths) {
$tempRoot = [IO.Path]::GetFullPath([IO.Path]::GetTempPath()).TrimEnd('\') + '\'
foreach ($path in $Paths) {
$resolved = [IO.Path]::GetFullPath($path)
if (-not $resolved.StartsWith($tempRoot, [StringComparison]::OrdinalIgnoreCase) -or
-not ([IO.Path]::GetFileName($resolved)).StartsWith("$Name-")) {
throw "Unsafe release temporary path: $resolved"
}
if (Test-Path -LiteralPath $resolved) { Remove-Item -LiteralPath $resolved -Recurse -Force }
} }
} }
# The tag step 1 exported, from release\<folder>\production.
function Get-ReleaseSha {
$file = Join-Path $Release 'production'
if (-not (Test-Path $file)) { throw "no $file - run step 1 first" }
(Get-Content $file -TotalCount 1).Trim()
}
+86
View File
@@ -0,0 +1,86 @@
# shellcheck shell=bash
# Shared plumbing for the release scripts that run in WSL. Source it; do not
# run it.
#
# Sourcing runs in the caller's shell, so everything defined here ($root,
# SETTINGS, setting, require_setting, die, step, run_or_die) is the caller's
# afterwards. The Windows half (steps 2 and 3) reads the same release.env with
# _common.ps1, which parses the same KEY=VALUE lines. The caller sets
# `set -euo pipefail` first.
#: The directory of these scripts and the repository root above it, worked out
#: from this file rather than from the working directory, so the scripts can be
#: run from anywhere.
here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
# shellcheck disable=SC2034 # read by the scripts that source this file
root=$(dirname "$here")
die() { printf 'error: %s\n' "$*" >&2; exit 1; }
step() { printf '\n== %s\n' "$*"; }
# Runs a command and, when it fails, stops with the step's name and its exit
# code: `run_or_die 'git archive' git archive ...`.
run_or_die() {
local what=$1 rc=0
shift
"$@" || rc=$?
((rc == 0)) || die "$what failed (exit $rc)"
}
# --------------------------------------------------------------- release.env
# Read, not sourced: sourcing would run whatever the file says and would break
# on a space in a value. A line is blank, a comment, or KEY=VALUE; anything else
# is a typo and is reported here rather than as an empty variable later. Keys
# and values are trimmed and surrounding double quotes are dropped. A CRLF file
# (saved from a Windows editor) reads the same as an LF one.
trim() {
local s=$1
s=${s#"${s%%[![:space:]]*}"}
s=${s%"${s##*[![:space:]]}"}
printf '%s' "$s"
}
declare -A SETTINGS=()
env_file=$here/release.env
[[ -f $env_file ]] || die "no deploy/release.env - copy deploy/release.env.example and fill it in"
while IFS= read -r line || [[ -n $line ]]; do
line=${line%$'\r'}
[[ $line =~ ^[[:space:]]*(#|$) ]] && continue
[[ $line == *=* ]] || die "release.env: not a KEY=VALUE line: $line"
key=$(trim "${line%%=*}")
value=$(trim "${line#*=}")
while [[ $value == \"* ]]; do value=${value#\"}; done
while [[ $value == *\" ]]; do value=${value%\"}; done
SETTINGS[$key]=$value
done < "$env_file"
unset line key value
# setting NAME [DEFAULT]: the value, or the default when unset or empty.
setting() {
local value=${SETTINGS[$1]-}
if [[ -n $value ]]; then printf '%s' "$value"; else printf '%s' "${2-}"; fi
}
# require_setting NAME: the value, or a stop naming the missing setting.
require_setting() {
local value
value=$(setting "$1")
[[ -n $value ]] || die "set $1 in deploy/release.env"
printf '%s' "$value"
}
# ------------------------------------------------------------- shared checks
need_git() {
command -v git >/dev/null 2>&1 || die "git is not on PATH"
}
# The commit a branch or ref points at, or a stop saying which one did not
# resolve.
resolve_commit() {
local commit
commit=$(git rev-parse --verify "$1^{commit}") || die "resolve $2 commit failed"
printf '%s' "$commit"
}
-150
View File
@@ -1,150 +0,0 @@
<#
.SYNOPSIS
The database, in both directions.
.DESCRIPTION
push this machine's database -> the server
pull a fresh snapshot from the server -> .\backups
`push` is the one-off at the start: production begins knowing what you
already have, rather than starting empty. `pull` is the one you keep
running -- on a schedule from the server itself (server\backup.sh under
cron) and by hand from here whenever you want the copy somewhere the
server is not.
`push` moves the database and nothing else. It does not stop the app before
or start it after: a deployment knows what else is moving and in what order,
and this does not. Stop the app on the server first -- the import is a file
swap under SQLite's feet and is refused while the app is up. Start it again
afterwards.
Both go through SQLite's own `VACUUM INTO`. The database is in WAL mode, so
the .db file on disk is always missing whatever is still in the -wal beside
it, and copying the three files with the app running is a race you lose
quietly -- a database that opens fine and is short a day's writes.
After a push the two copies diverge from the first thing written on either.
There is no merge for that and this does not pretend there is: pick the one
that is authoritative, and use the other read-only or not at all.
.EXAMPLE
.\deploy\db.ps1 push
.EXAMPLE
.\deploy\db.ps1 pull
#>
[CmdletBinding()]
param(
[Parameter(Mandatory)]
[ValidateSet('push', 'pull')]
[string] $Action,
# Skip the confirmation before replacing the server's database.
[switch] $Yes
)
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
. "$PSScriptRoot/_common.ps1"
Set-Location $root
$SshHost = Setting SSH_HOST -Required
$RemoteDir = Setting REMOTE_DIR -Required
# Where the data actually is. Beside the compose file unless APP_DATA put it
# under a shared root; then REMOTE_DATA has to say so.
$RemoteData = Setting REMOTE_DATA "$RemoteDir/data"
$Python = Setting PYTHON (Join-Path $root '.venv\Scripts\python.exe')
$LocalDb = Setting LOCAL_DB (Join-Path $root "$Name.db")
switch ($Action) {
# ---------------------------------------------------------------- push
'push' {
if (-not (Test-Path $LocalDb)) { throw "no $LocalDb" }
$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$snap = Join-Path $root "$Name-$stamp.db.gz"
Write-Host "snapshotting $LocalDb"
# One Python call for both halves. `VACUUM INTO` is the only way to get a
# consistent single file out of a WAL database, and Python is already the
# thing that has to be here -- PowerShell has no gzip that writes the format
# the server's `gunzip` expects.
$snapshot = @'
import gzip, pathlib, shutil, sqlite3, sys, tempfile
source, target = sys.argv[1], pathlib.Path(sys.argv[2])
with tempfile.TemporaryDirectory() as work:
plain = pathlib.Path(work) / "snapshot.db"
db = sqlite3.connect(source)
try:
db.execute("VACUUM INTO ?", (str(plain),))
finally:
db.close()
with plain.open("rb") as raw, gzip.open(target, "wb") as out:
shutil.copyfileobj(raw, out)
'@
# A checkout inside WSL has a Linux .venv, and SQLite cannot lock a WAL
# database through the \\wsl.localhost share, so the snapshot is taken in
# the distro. The script goes in on stdin, past both shells.
if ($Wsl.Success) {
$snapshot | wsl.exe -d $Wsl.Groups[1].Value --cd (ConvertTo-LinuxPath $root) --exec .venv/bin/python - (ConvertTo-LinuxPath $LocalDb) (ConvertTo-LinuxPath $snap)
} else {
& $Python -c $snapshot $LocalDb $snap
}
Assert-Ok 'snapshot'
' {0:N1} MB' -f ((Get-Item $snap).Length / 1MB) | Write-Host
try {
Write-Host ''
Write-Host "This replaces the database on $SshHost with this one." -ForegroundColor Yellow
Write-Host 'Anything written there since is gone.' -ForegroundColor Yellow
Write-Host 'The app must already be stopped there, and stays stopped after.' -ForegroundColor Yellow
if (-not $Yes) {
$reply = Read-Host 'go ahead? [y/N]'
if ($reply -notmatch '^[yY]') { Write-Host 'stopped.'; return }
}
scp -C $snap "${SshHost}:$RemoteData/incoming.db.gz"; Assert-Ok 'scp database'
scp (Join-Path $PSScriptRoot 'server\import_database.py') "${SshHost}:$RemoteDir/import_database.py"
Assert-Ok 'scp importer'
} finally {
Remove-Item -Force $snap -ErrorAction SilentlyContinue
}
# Base64 rather than stdin: PowerShell has no `<` for a native command, and
# piping the text would rewrite the script's line endings on the way.
$b64 = [Convert]::ToBase64String([IO.File]::ReadAllBytes((Join-Path $PSScriptRoot 'server\import.sh')))
ssh $SshHost "echo $b64 | base64 -d | bash -s -- '$RemoteDir'"; Assert-Ok 'import'
Write-Host "pushed. The app is still stopped; start it when the rest of"
Write-Host 'the deployment is in place:'
Write-Host " ssh $SshHost `"cd '$RemoteDir' && docker compose up -d --wait $Name`""
}
# ---------------------------------------------------------------- pull
'pull' {
$into = Join-Path $root 'backups'
New-Item -ItemType Directory -Force -Path $into | Out-Null
# The server's own copy of backup.sh, not a streamed one. It is the same
# script cron runs, it is already there, and it locates its own directory
# through BASH_SOURCE -- which is exactly the thing a script read from stdin
# does not have. import.sh takes its directory as an argument and so can be
# streamed; this one cannot.
Write-Host "snapshotting on $SshHost"
ssh $SshHost "cd '$RemoteDir' && bash backup.sh"; Assert-Ok 'backup'
$newest = (ssh $SshHost "ls -1t '$RemoteData'/backups/$Name-*.db.gz | head -1").Trim()
Assert-Ok 'listing backups'
if (-not $newest) { throw 'no backup appeared on the server' }
scp -C "${SshHost}:$newest" $into; Assert-Ok 'scp'
$file = Split-Path -Leaf $newest
Write-Host "backups\$file"
Write-Host ''
Write-Host 'To read it here, gunzip it, or in Python:'
Write-Host " import gzip, shutil; shutil.copyfileobj(gzip.open(r'backups\$file'), open('read-only.db','wb'))"
}
}
+130
View File
@@ -0,0 +1,130 @@
#!/usr/bin/env bash
# The database, in both directions.
#
# bash deploy/db.sh push [--yes] this machine's database -> the server
# bash deploy/db.sh pull a fresh snapshot from the server -> backups/
#
# `push` is the one-off at the start: production begins knowing what you already
# have, rather than starting empty. `pull` is the one you keep running, on a
# schedule from the server itself (server/backup.sh under cron) and by hand from
# here whenever you want the copy somewhere the server is not.
#
# `push` moves the database and nothing else. It does not stop the app before or
# start it after: a deployment knows what else is moving and in what order, and
# this does not. Stop the app on the server first; the import is a file swap
# under SQLite's feet and is refused while the app is up. Start it again
# afterwards.
#
# Both go through SQLite's own `VACUUM INTO`. The database is in WAL mode, so the
# .db file on disk is always missing whatever is still in the -wal beside it, and
# copying the three files with the app running is a race you lose quietly: a
# database that opens fine and is short a day's writes.
#
# After a push the two copies diverge from the first thing written on either.
# There is no merge for that: pick the one that is authoritative, and use the
# other read-only or not at all.
#
# Settings from release.env: SSH_HOST, REMOTE_DIR and NAME (the compose service)
# are required; REMOTE_DATA (default REMOTE_DIR/data), PYTHON and LOCAL_DB
# (default the database the app's config names) are not.
set -euo pipefail
# shellcheck source=_common.sh
. "$(dirname "${BASH_SOURCE[0]}")/_common.sh"
action='' yes=0
for arg in "$@"; do
case "$arg" in
push|pull) action=$arg ;;
--yes) yes=1 ;;
-h|--help) sed -n '2,29p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) die "unknown option $arg" ;;
esac
done
[[ -n $action ]] || die "say push or pull: bash deploy/db.sh push|pull"
ssh_host=$(require_setting SSH_HOST)
remote_dir=$(require_setting REMOTE_DIR)
# Where the data actually is. Beside the compose file unless APP_DATA put it
# under a shared root; then REMOTE_DATA has to say so.
remote_data=$(setting REMOTE_DATA "$remote_dir/data")
name=$(require_setting NAME)
python=$(setting PYTHON "$root/.venv/bin/python")
cd "$root"
# ---------------------------------------------------------------------- push
if [[ $action == push ]]; then
[[ -x $python ]] || die "no interpreter at $python - run uv sync --all-extras, or set PYTHON in deploy/release.env"
# The file the app writes: `database` in config.yaml (or $APP_CONFIG),
# roledesk.db without one, asked of the app from the checkout's root.
local_db=$(setting LOCAL_DB)
if [[ -z $local_db ]]; then
local_db=$("$python" -c 'from app.config import load; print(load().database.absolute())') \
|| die "could not read the database path from the app's config; set LOCAL_DB in deploy/release.env"
fi
[[ -f $local_db ]] || die "no $local_db"
snap=$root/$name-$(date +%Y%m%d-%H%M%S).db.gz
trap 'rm -f -- "$snap"' EXIT
echo "snapshotting $local_db"
# One Python call for both halves: `VACUUM INTO` is the only way to get a
# consistent single file out of a WAL database, and it writes the gzip the
# server's `gunzip` expects.
run_or_die snapshot "$python" -c '
import gzip, pathlib, shutil, sqlite3, sys, tempfile
source, target = sys.argv[1], pathlib.Path(sys.argv[2])
with tempfile.TemporaryDirectory() as work:
plain = pathlib.Path(work) / "snapshot.db"
db = sqlite3.connect(source)
try:
db.execute("VACUUM INTO ?", (str(plain),))
finally:
db.close()
with plain.open("rb") as raw, gzip.open(target, "wb") as out:
shutil.copyfileobj(raw, out)
' "$local_db" "$snap"
printf ' %s\n' "$(du -h "$snap" | cut -f1)"
echo
echo "This replaces the database on $ssh_host with this one."
echo 'Anything written there since is gone.'
echo 'The app must already be stopped there, and stays stopped after.'
if ((! yes)); then
read -r -p 'go ahead? [y/N] ' reply || reply=''
[[ $reply =~ ^[yY] ]] || { echo 'stopped.'; exit 0; }
fi
run_or_die 'scp database' scp -C "$snap" "$ssh_host:$remote_data/incoming.db.gz"
run_or_die 'scp importer' scp "$here/server/import_database.py" "$ssh_host:$remote_dir/import_database.py"
rm -f -- "$snap"
# import.sh takes its directory as an argument, so it can be streamed.
run_or_die import ssh "$ssh_host" "bash -s -- '$remote_dir'" < "$here/server/import.sh"
echo 'pushed. The app is still stopped; start it when the rest of'
echo 'the deployment is in place:'
echo " ssh $ssh_host \"cd '$remote_dir' && docker compose up -d --wait $name\""
exit 0
fi
# ---------------------------------------------------------------------- pull
into=$root/backups
mkdir -p -- "$into"
# The server's own copy of backup.sh, not a streamed one: it is the same script
# cron runs, and it locates its own directory through BASH_SOURCE, which a
# script read from stdin does not have.
echo "snapshotting on $ssh_host"
run_or_die backup ssh "$ssh_host" "cd '$remote_dir' && bash backup.sh"
# shellcheck disable=SC2029 # the remote command is meant to expand here
newest=$(ssh "$ssh_host" "ls -1t '$remote_data'/backups/$name-*.db.gz | head -1") || die 'listing backups failed'
[[ -n $newest ]] || die 'no backup appeared on the server'
run_or_die scp scp -C "$ssh_host:$newest" "$into/"
file=$(basename "$newest")
echo "backups/$file"
echo
echo 'To read it here, gunzip it, or in Python:'
echo " import gzip, shutil; shutil.copyfileobj(gzip.open('backups/$file'), open('read-only.db','wb'))"
+26 -13
View File
@@ -1,28 +1,41 @@
# Copy to release.env beside these scripts and fill it in. Never committed: # Copy to release.env. One file, read by both halves of the release: the bash
# it names your registry and, for db.ps1, your server. # scripts (_common.sh), which run in the WSL distro that holds the checkout, and
# the PowerShell ones (_common.ps1, wsl-setup.ps1), which run on Windows. Both
# read KEY=VALUE lines and neither runs the file as shell, so $HOME and ~ are
# not expanded. Local main is the source; fetching is your choice. Never
# committed: it names your registry and, for db.sh, your server.
# #
# Local main is the source. Fetching it first is your choice, not the script's. # Where the checkout is: the distro's name and the path inside it, as
# docs/Install WSL.md lays them out, with the checkout folder's name last.
# wsl-setup.ps1 writes them into the profile commands, and steps 2 and 3 find
# step 1's export at \\wsl.localhost\<DISTRO>\<CHECKOUT's parent>\release\<folder>.
DISTRO=claudebox
CHECKOUT=/home/claude/cboxcode/jobboard
# The service's name. Used for the image tag, for the temporary directories # The compose service, SERVICE in deploy/server/*.sh. Used by db.sh, on the
# these scripts create, and for the database file db.ps1 moves. One word, no # Linux side, for the snapshot names (<NAME>-<stamp>.db.gz, as backup.sh writes
# spaces — it ends up in paths. # them) and the start command it prints. One word, no spaces.
NAME=app NAME=app
# Windows side, used by steps 2 and 3. wslc builds for the machine's own
# architecture (linux/amd64 here) and has no --platform.
IMAGE=git.rittershofer.org/wolf/app IMAGE=git.rittershofer.org/wolf/app
PLATFORM=linux/amd64
# Linux side, used by step 1: which branch is tested, and which one it promotes.
BRANCH=production BRANCH=production
SOURCE=main SOURCE=main
# Defaults to the checkout's .venv/Scripts/python.exe. Only needed for the # Linux side, used by step 1 and by db.sh's snapshot, in the distro. Defaults to
# tests in step 1 and for db.ps1's snapshot. # the checkout's .venv/bin/python (a WSL path, /home/<user>/...).
# PYTHON=C:/Users/Coolkau/code/app/.venv/Scripts/python.exe # PYTHON=/home/<user>/cboxcode/jobboard/.venv/bin/python
# db.ps1 uses SSH. The numbered release scripts never do. REMOTE_DIR is the # db.sh uses SSH. The numbered release scripts never do. REMOTE_DIR is the
# directory holding the compose file; REMOTE_DATA is where APP_DATA points, # directory holding the compose file; REMOTE_DATA is where APP_DATA points,
# and defaults to REMOTE_DIR/data when left blank. # and defaults to REMOTE_DIR/data when left blank.
SSH_HOST= SSH_HOST=
REMOTE_DIR=/srv/swag REMOTE_DIR=/srv/swag
REMOTE_DATA=/srv/data/app REMOTE_DATA=/srv/data/app
# The database this machine pushes, if it is not <root>/<NAME>.db. # The database this machine pushes, if it is not the one the app's config names
# LOCAL_DB=C:/Users/Coolkau/code/app/app.db # (`database` in config.yaml, roledesk.db in the checkout without one).
# LOCAL_DB=/home/<user>/cboxcode/jobboard/roledesk.db
+43 -10
View File
@@ -13,9 +13,26 @@
# self-contained and compacted file, and never blocks a writer for long enough # self-contained and compacted file, and never blocks a writer for long enough
# to matter on a database of this size. # to matter on a database of this size.
# #
# Rename `app` and `APP_DATA` to your service's name. # The database is the one the app's own config names, asked of the app inside
# the container, and it is opened read-only, so a path that names no file is
# an error rather than a new empty database and a snapshot of nothing.
#
# SERVICE is the compose service (deploy.sh says why it is here); APP_DATA is
# renamed with the other APP_ variables, and `from app.config` with the package.
SERVICE=app
# Tables every database this app writes has: the same set as REQUIRED_TABLES in
# import_database.py, which a test holds this line to. A snapshot without one
# of them is deleted and the run fails.
# Ours: `roles` is app/storage/schema.py's first table, where the boilerplate has `ticks`.
TABLES="jobs roles"
set -euo pipefail set -euo pipefail
# A snapshot is the whole database, so it is owner only whatever the caller's
# umask is. This covers the folder made below; the umask inside the container
# (set in the snippet) covers the .db, and gzip gives the .gz the mode of the
# file it compresses.
umask 077
cd "${1:-$(dirname "$(readlink -f "$0")")}" cd "${1:-$(dirname "$(readlink -f "$0")")}"
# Same source of truth as the compose file and deploy.sh. # Same source of truth as the compose file and deploy.sh.
@@ -27,18 +44,34 @@ stamp=$(date +%Y%m%d-%H%M%S)
mkdir -p "$DATA/backups" mkdir -p "$DATA/backups"
docker compose exec -T app python -c " # shellcheck disable=SC2086 # TABLES is a list of words on purpose
import sqlite3 docker compose exec -T "$SERVICE" python - "$SERVICE-$stamp" $TABLES <<'PY'
db = sqlite3.connect('/data/app.db') import os, sqlite3, sys
db.execute('VACUUM INTO ?', ('/data/backups/app-$stamp.db',)) from contextlib import closing
db.close() from pathlib import Path
"
gzip -f "$DATA/backups/app-$stamp.db" from app.config import load
echo "$DATA/backups/app-$stamp.db.gz $(du -h "$DATA/backups/app-$stamp.db.gz" | cut -f1)"
os.umask(0o077)
name, required = sys.argv[1], set(sys.argv[2:])
database = load().database.absolute()
snapshot = Path("/data/backups") / f"{name}.db"
with closing(sqlite3.connect(f"{database.as_uri()}?mode=ro", uri=True)) as db:
db.execute("VACUUM INTO ?", (str(snapshot),))
with closing(sqlite3.connect(f"{snapshot.as_uri()}?mode=ro", uri=True)) as db:
tables = {row[0] for row in db.execute("SELECT name FROM sqlite_master WHERE type='table'")}
missing = sorted(required - tables)
if missing:
snapshot.unlink()
sys.exit(f"{database} has no table {', '.join(missing)}: not this app's database")
print(f"snapshot of {database}")
PY
gzip -f "$DATA/backups/$SERVICE-$stamp.db"
echo "$DATA/backups/$SERVICE-$stamp.db.gz $(du -h "$DATA/backups/$SERVICE-$stamp.db.gz" | cut -f1)"
# Oldest first, everything past KEEP goes. Named by timestamp, so sorting the # Oldest first, everything past KEEP goes. Named by timestamp, so sorting the
# names sorts them by age. # names sorts them by age.
ls -1 "$DATA"/backups/app-*.db.gz 2>/dev/null \ ls -1 "$DATA"/backups/"$SERVICE"-*.db.gz 2>/dev/null \
| sort \ | sort \
| head -n -"$KEEP" \ | head -n -"$KEEP" \
| while read -r old; do rm -f "$old"; echo "removed $old"; done | while read -r old; do rm -f "$old"; echo "removed $old"; done
+6 -3
View File
@@ -9,7 +9,10 @@
# `env.example` to the `.env` next to it, and it is `docker compose up -d app` # `env.example` to the `.env` next to it, and it is `docker compose up -d app`
# from then on — which recreates this container and no other. # from then on — which recreates this container and no other.
# #
# Rename `app` and `APP_` throughout to your service's name. # Rename `app` and `APP_` throughout to your service's name. The service and
# `container_name` are the name `SERVICE=` in deploy.sh, backup.sh and
# import.sh, `$upstream_app` in the proxy conf and `NAME=` in release.env
# repeat; tests/test_deploy.py reads it from this file and holds them to it.
services: services:
app: app:
@@ -33,12 +36,12 @@ services:
# Which model answers, decided here rather than in the image. With # Which model answers, decided here rather than in the image. With
# `backend: auto` the API key wins when there is one and the # `backend: auto` the API key wins when there is one and the
# subscription CLI is the fallback, so filling in ANTHROPIC_API_KEY # subscription CLI is the fallback, so filling in ANTHROPIC_API_KEY
# later switches it over with no rebuild. See docs/llm-backends.md. # later switches it over with no rebuild. See llm/README.md.
CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN:-} CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN:-}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-} ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
# Normally empty. Set it to pin the password — the way back in when a # Normally empty. Set it to pin the password — the way back in when a
# login is lost — then unset it and recreate to hand control of the # login is lost — then unset it and recreate to hand control of the
# password back to the app. See docs/password-protection.md. # password back to the app. See auth/README.md.
APP_PASSWORD: ${APP_PASSWORD:-} APP_PASSWORD: ${APP_PASSWORD:-}
# No `ports:`. Nothing on the host reaches it; SWAG does, by container # No `ports:`. Nothing on the host reaches it; SWAG does, by container
+30
View File
@@ -0,0 +1,30 @@
# The server's config.yaml. Copy it to $APP_DATA/config.yaml, which the
# container reads as /data/config.yaml (APP_CONFIG in the Dockerfile), and set
# the hostname. deploy.sh refuses a config without `database` or `auth.enabled`.
# Every key left out keeps its default; ../../config.example.yaml lists them all.
# Ours: the keys are roledesk's (app/config.py); an unknown one is an error.
server:
host: 0.0.0.0 # the container's interface; SWAG is in front of it
port: 8787 # the port the proxy conf and the HEALTHCHECK use
database: /data/roledesk.db # in the volume; /app, the working directory, is not writable
auth:
enabled: always # the lock must not depend on how the process was started
secure_cookie: always # auto needs uvicorn to trust X-Forwarded-Proto, which only `app serve` off loopback sets
rp_id: app.example.com # the hostname passkeys are bound to; changing it voids every passkey
origin: https://app.example.com # what the browser's address bar shows, scheme included
# Who answers when a stage needs a model (extract, summarise, judge). `auto`
# takes ANTHROPIC_API_KEY when the compose file passes one and otherwise the
# subscription CLI with CLAUDE_CODE_OAUTH_TOKEN, so switching is an .env edit.
# The stages themselves are switched under Settings, Sources, Runs.
agent:
backend: auto
# The vectors behind prompt and CV closeness. The model is not in the image:
# the first `embed` job downloads it (~460 MB) into HOME, the /data volume, so
# it happens once per server and not once per release. Changing it re-embeds.
embed:
model: intfloat/multilingual-e5-small
+27 -10
View File
@@ -1,9 +1,14 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Run on the server, next to the compose file, or with its directory as $1. # Run on the server, next to the compose file, or with its directory as $1.
# No remote shell required: the release scripts on Windows publish an image # No remote shell required: the release scripts publish an image
# and stop there, and this is the other half. # and stop there, and this is the other half.
# #
# Rename `app` and `APP_DATA` to your service's name. # SERVICE is the compose service, the one name a project renames in these
# scripts (README, "Starting a project from it"); the tests read it from
# compose.yaml and hold every script to it. APP_DATA is renamed with the other
# APP_ variables.
SERVICE=app
set -euo pipefail set -euo pipefail
cd "${1:-$(dirname "$(readlink -f "$0")")}" cd "${1:-$(dirname "$(readlink -f "$0")")}"
@@ -15,24 +20,36 @@ cd "${1:-$(dirname "$(readlink -f "$0")")}"
DATA=$(sed -n 's/^[[:space:]]*APP_DATA=\([^#]*\).*/\1/p' .env | tail -1 | xargs || true) DATA=$(sed -n 's/^[[:space:]]*APP_DATA=\([^#]*\).*/\1/p' .env | tail -1 | xargs || true)
DATA=${DATA:-./data} DATA=${DATA:-./data}
[[ -s "$DATA/config.yaml" ]] || { config=$DATA/config.yaml
echo "Install $DATA/config.yaml before deploying." >&2 [[ -s $config ]] || {
echo "Install $config before deploying, from config.example.yaml beside this script." >&2
exit 1 exit 1
} }
# The two keys whose defaults are wrong in a container: `auth.enabled: auto`
# ties the lock to how the process was started, and the default
# database is wherever the config file happens to be. Read with grep and awk,
# because the host need not have a Python with a YAML parser.
if ! grep -Eq '^database:[[:space:]]*[^[:space:]#]' "$config" ||
! awk '/^auth:/ { a = 1; next } /^[^[:space:]#]/ { a = 0 }
a && /^[[:space:]]+enabled:[[:space:]]*[^[:space:]#]/ { found = 1 }
END { exit !found }' "$config"; then
echo "$config must set database and auth.enabled; config.example.yaml beside this script has both." >&2
exit 1
fi
# Pull explicitly, here and nowhere else: the compose file has no pull policy, # Pull explicitly, here and nowhere else: the compose file has no pull policy,
# so starting or importing never fetches an image on its own. A registry # so starting or importing never fetches an image on its own. A registry
# failure must leave the running service alone, not deploy a cache. # failure must leave the running service alone, not deploy a cache.
echo 'Pulling the configured image…' echo 'Pulling the configured image…'
if ! docker compose pull app; then if ! docker compose pull "$SERVICE"; then
echo 'Image pull failed; app was not restarted.' >&2 echo "Image pull failed; $SERVICE was not restarted." >&2
exit 1 exit 1
fi fi
# Use the image just pulled. Do not start or recreate other services in the # Use the image just pulled. Do not start or recreate other services in the
# file. `--wait` holds until the container's own HEALTHCHECK passes. # file. `--wait` holds until the container's own HEALTHCHECK passes.
if ! docker compose up -d --no-deps --pull never --wait --wait-timeout 240 app; then if ! docker compose up -d --no-deps --pull never --wait --wait-timeout 240 "$SERVICE"; then
docker compose ps app >&2 || true docker compose ps "$SERVICE" >&2 || true
echo 'Deployment failed. Check docker compose logs app.' >&2 echo "Deployment failed. Check docker compose logs $SERVICE." >&2
exit 1 exit 1
fi fi
docker compose ps app docker compose ps "$SERVICE"
+3 -3
View File
@@ -4,7 +4,7 @@
# #
# chmod 600. Rename APP_ to your service's name throughout. # chmod 600. Rename APP_ to your service's name throughout.
APP_IMAGE=git.rittershofer.org/wolf/app APP_IMAGE=registry.example/you/app
APP_TAG=latest # follows published images; pin a commit tag for rollback APP_TAG=latest # follows published images; pin a commit tag for rollback
# Where the app keeps its database, its config and its backups. Relative to # Where the app keeps its database, its config and its backups. Relative to
@@ -20,7 +20,7 @@ PGID=1000
TZ=Europe/Berlin TZ=Europe/Berlin
# Which model answers. Set one or the other, not both — see # Which model answers. Set one or the other, not both — see
# docs/llm-backends.md for what each one means and what it costs. # llm/README.md for what each one means and what it costs.
# #
# CLAUDE_CODE_OAUTH_TOKEN a year-long token for the Claude subscription, from # CLAUDE_CODE_OAUTH_TOKEN a year-long token for the Claude subscription, from
# `claude setup-token` on a machine you are signed in # `claude setup-token` on a machine you are signed in
@@ -34,7 +34,7 @@ ANTHROPIC_API_KEY=
# `docker compose logs app`; after that it is the app's Account screen's to # `docker compose logs app`; after that it is the app's Account screen's to
# change. Fill this in only to force one — it is the way back in — then empty # change. Fill this in only to force one — it is the way back in — then empty
# it again and recreate, or the Account screen refuses to change what the # it again and recreate, or the Account screen refuses to change what the
# environment pins. See docs/password-protection.md. # environment pins. See auth/README.md.
APP_PASSWORD= APP_PASSWORD=
# Unlimited until the box has been watched through the heaviest thing the app # Unlimited until the box has been watched through the heaviest thing the app
+9 -6
View File
@@ -1,18 +1,21 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Called on the server after uploading incoming.db.gz and import_database.py. # Called on the server after uploading incoming.db.gz and import_database.py.
# `db.ps1 push` does both and then runs this; by hand it is the same two files # `db.sh push` does both and then runs this; by hand it is the same two files
# in the same two places. # in the same two places.
# #
# This replaces the database and does nothing else. It does not stop the app # This replaces the database and does nothing else. It does not stop the app
# and it does not start it: whoever is running a deployment knows what else is # and it does not start it: whoever is running a deployment knows what else is
# moving and in what order, and a tool that helpfully restarts things underneath # moving and in what order, and a tool that helpfully restarts things underneath
# that is a tool fighting its caller. By hand it is # that is a tool fighting its caller. By hand it is
# `docker compose stop app` and `docker compose up -d --wait app` around this. # `docker compose stop app` and `docker compose up -d --wait app` around this,
# with SERVICE (deploy.sh says why it is here) in place of app.
# #
# What it will not do is import into a running app. The replacement is a file # What it will not do is import into a running app. The replacement is a file
# swap under SQLite's feet, and the live database is checkpointed first, which # swap under SQLite's feet, and the live database is checkpointed first, which
# needs every writer gone — `import_database.py` refuses a busy database, and # needs every writer gone — `import_database.py` refuses a busy database, and
# this refuses an app that is up, because the clearer error is the earlier one. # this refuses an app that is up, because the clearer error is the earlier one.
SERVICE=app
set -euo pipefail set -euo pipefail
cd "${1:?server directory required}" cd "${1:?server directory required}"
@@ -22,14 +25,14 @@ DATA=${DATA:-./data}
[[ -s "$DATA/incoming.db.gz" && -s import_database.py ]] [[ -s "$DATA/incoming.db.gz" && -s import_database.py ]]
if [[ -n "$(docker compose ps -q --status running app 2>/dev/null)" ]]; then if [[ -n "$(docker compose ps -q --status running "$SERVICE" 2>/dev/null)" ]]; then
echo 'app is running. Stop it before importing:' >&2 echo "$SERVICE is running. Stop it before importing:" >&2
echo ' docker compose stop app' >&2 echo " docker compose stop $SERVICE" >&2
exit 1 exit 1
fi fi
# The one-off container uses compose's uid and its mounted /data. `--pull # The one-off container uses compose's uid and its mounted /data. `--pull
# never`: an import runs on the image already here, whatever the installed # never`: an import runs on the image already here, whatever the installed
# compose file says about pulling. Getting a new image is deploy.sh's job. # compose file says about pulling. Getting a new image is deploy.sh's job.
docker compose run --rm --no-deps --pull never -T --entrypoint python app - < import_database.py docker compose run --rm --no-deps --pull never -T --entrypoint python "$SERVICE" - < import_database.py
rm -f "$DATA/incoming.db.gz" rm -f "$DATA/incoming.db.gz"
+17 -9
View File
@@ -1,11 +1,13 @@
"""Offline database replacement, run as the compose service's uid. """Offline database replacement, run as the compose service's uid.
The app container must be stopped. Only stdlib is used so this can be sent The app container must be stopped. Sent over stdin to the image on the
over stdin to either the old or the new image during a deployment. server, whose working directory is the app's: the database is the one the
app's config names (`from app.config` follows the package when it is renamed),
and the rest is stdlib, so the old image and the new one both run it.
Two things to edit for your service: `DATABASE` and `REQUIRED_TABLES`. The `REQUIRED_TABLES` is the table check, which stops a right-shaped-but-wrong
table check is what stops a right-shaped-but-wrong database — last project's, database (last project's, or a half-downloaded one) from landing on top of a
or a half-downloaded one — from landing on top of a good one. good one. `tests/test_deploy.py` fails when the app stops creating one of them.
""" """
from __future__ import annotations from __future__ import annotations
@@ -19,13 +21,14 @@ from contextlib import closing
from datetime import UTC, datetime from datetime import UTC, datetime
from pathlib import Path from pathlib import Path
DATABASE = Path("/data/app.db")
INCOMING = Path("/data/incoming.db.gz") INCOMING = Path("/data/incoming.db.gz")
BACKUPS = Path("/data/backups") BACKUPS = Path("/data/backups")
#: Any one of these missing means this is not our database. Name tables that #: Any one of these missing means this is not our database. Name tables that
#: have existed since the first schema, not ones a recent migration added. #: have existed since the first schema, not ones a recent migration added:
REQUIRED_TABLES = {"items"} #: `jobs` is the queue's and `roles` is `app/storage/schema.py`'s first table
#: (the boilerplate has `ticks`). backup.sh's TABLES is the same set.
REQUIRED_TABLES = {"jobs", "roles"}
def replace_database(database: Path, archive: Path, backups: Path) -> Path | None: def replace_database(database: Path, archive: Path, backups: Path) -> Path | None:
@@ -73,5 +76,10 @@ def replace_database(database: Path, archive: Path, backups: Path) -> Path | Non
if __name__ == "__main__": if __name__ == "__main__":
saved = replace_database(DATABASE, INCOMING, BACKUPS) from app.config import load
# The snapshot is the whole database, so it is owner only whatever the
# container's umask is; the gzip of it is made under the same mask.
os.umask(0o077)
saved = replace_database(load().database.absolute(), INCOMING, BACKUPS)
print(f"Database replaced; previous snapshot: {saved or 'none (first import)'}") print(f"Database replaced; previous snapshot: {saved or 'none (first import)'}")
+14 -7
View File
@@ -3,7 +3,7 @@
## ##
## The subdomain has to exist before this does anything: add <app> to SWAG's ## The subdomain has to exist before this does anything: add <app> to SWAG's
## SUBDOMAINS (or run it with a wildcard cert) and point the DNS record at the ## SUBDOMAINS (or run it with a wildcard cert) and point the DNS record at the
## same address as the rest of the stack. See docs/hosting.md. ## same address as the rest of the stack. See docs/Install Server.md.
## Per-IP throttle for the login endpoints, if the app has a password. Declared ## Per-IP throttle for the login endpoints, if the app has a password. Declared
## here at `http` scope — SWAG includes proxy-confs outside any server block, ## here at `http` scope — SWAG includes proxy-confs outside any server block,
@@ -15,23 +15,30 @@ server {
listen 443 ssl; listen 443 ssl;
listen [::]:443 ssl; listen [::]:443 ssl;
server_name app.rittershofer.org; server_name app.example.com;
include /config/nginx/ssl.conf; include /config/nginx/ssl.conf;
# Put proxy defaults at server scope so location timeouts override them # Put proxy defaults at server scope so location timeouts override them
# through inheritance, rather than duplicate directives from proxy.conf. # through inheritance, rather than duplicate directives from proxy.conf.
include /config/nginx/proxy.conf; include /config/nginx/proxy.conf;
# 0 is no limit. Set a real one if strangers can post to this app; leave it # nginx buffers a request body to disk before it hands it on, and the login
# off when the only uploads are yours and can be large. # takes one without a session, so no limit here lets anyone spool gigabytes.
client_max_body_size 0; # This is small. A route that takes a large upload gets its own
# `location = /path { client_max_body_size 0; proxy_request_buffering off;
# ... }` above `location /`, with the same proxy lines as below; streaming
# the body lets the app refuse a request without a session before nginx
# has stored any of it.
client_max_body_size 1m;
# Guessing the password is already pointless if the app locks out after a # Guessing the password is already pointless if the app locks out after a
# handful of wrong answers (see docs/password-protection.md). This is the # handful of wrong answers (see auth/README.md). This is the
# cheap half of the rest: attempts are slowed per address, at the proxy, # cheap half of the rest: attempts are slowed per address, at the proxy,
# before they reach the app at all. `burst=5 nodelay` lets a person who # before they reach the app at all. `burst=5 nodelay` lets a person who
# fat-fingers it a few times through immediately and then holds the rate. # fat-fingers it a few times through immediately and then holds the rate.
location ~ ^/login(/passkey)?$ { # `/login/passkey/options` is open and writes a challenge row per call, so
# it is throttled with the two routes that check a credential.
location ~ ^/login(/passkey(/options)?)?$ {
limit_req zone=app_login burst=5 nodelay; limit_req zone=app_login burst=5 nodelay;
limit_req_status 429; limit_req_status 429;
+16 -9
View File
@@ -1,17 +1,24 @@
#!/bin/sh #!/bin/sh
# Test one commit in a throwaway worktree, with the checkout's Linux .venv. # Test one commit in a throwaway worktree, with the checkout's Linux .venv.
# 1-test-and-promote.ps1 runs this through wsl.exe when the checkout lives # 1-test-and-promote.sh runs this from the checkout root. PYTHON, when set (the
# inside WSL, where the .venv has no Windows python.exe to call. The gates are # PYTHON line of release.env), names the interpreter instead of .venv/bin/python.
# the same as in that script; keep the two lists in step. # The gates are this project's: roledesk has no `frontend/`, so there is no npm
# step, and ruff and mypy stay commented out as the release has run them so far.
# Usage: sh deploy/test-commit.sh <commit> (from the checkout root) # Usage: sh deploy/test-commit.sh <commit> (from the checkout root)
# sh deploy/test-commit.sh --here (the working tree as it is, no worktree: bin/check.sh)
set -eu set -eu
root=$(pwd) root=$(pwd)
python="$root/.venv/bin/python" python="${PYTHON:-$root/.venv/bin/python}"
[ -x "$python" ] || { echo "no interpreter at $python - run uv sync" >&2; exit 1; } [ -x "$python" ] || { echo "no interpreter at $python - run uv sync --all-extras, or set PYTHON in deploy/release.env" >&2; exit 1; }
base=$(mktemp -d /tmp/test-commit-XXXXXXXX) target="${1:?usage: sh deploy/test-commit.sh <commit>|--here}"
tree="$base/tree" if [ "$target" = "--here" ]; then
trap 'git -C "$root" worktree remove --force "$tree" 2>/dev/null; rm -rf "$base"' EXIT tree="$root"
git worktree add --detach "$tree" "$1" >/dev/null else
base=$(mktemp -d /tmp/release-test-XXXXXXXX)
tree="$base/tree"
trap 'git -C "$root" worktree remove --force "$tree" 2>/dev/null; rm -rf "$base"' EXIT
git worktree add --detach "$tree" "$target" >/dev/null
fi
cd "$tree" cd "$tree"
"$python" -m pytest -q "$python" -m pytest -q
# "$python" -m ruff check . # "$python" -m ruff check .
+14
View File
@@ -0,0 +1,14 @@
@echo off
rem Double-click to write this checkout's short commands, folder-cd to
rem folder-3push, into the PowerShell profile. It runs deploy\wsl-setup.ps1
rem beside it with PowerShell 7 (pwsh) when installed, else Windows PowerShell;
rem the profile it writes is that one's. cmd cannot sit in a \\wsl.localhost
rem folder, so the "UNC paths are not supported" line it prints is harmless.
rem No rem line here holds a redirection or pipe character: cmd still applies
rem them after rem, as dostips.com/forum/viewtopic.php?p=15252 shows.
where pwsh >nul 2>nul
if %errorlevel%==0 (set shell=pwsh) else (set shell=powershell)
%shell% -NoProfile -ExecutionPolicy Bypass -File "%~dp0wsl-setup.ps1" %*
echo.
echo Open a new PowerShell for the commands to appear.
pause
+64
View File
@@ -0,0 +1,64 @@
<#
.SYNOPSIS
Once per checkout, on Windows: writes this checkout's short commands into
the PowerShell profile. See docs/Install WSL.md.
.DESCRIPTION
Reads DISTRO and CHECKOUT from deploy\release.env beside this script. The
prefix is the checkout folder's name (news-, boilerplate-), or -Name. Writes
one marked block into $PROFILE and replaces it when run again.
#>
param([string] $Name)
$ErrorActionPreference = 'Stop'
$envFile = Join-Path $PSScriptRoot 'release.env'
if (-not (Test-Path $envFile)) { throw "no $envFile - copy release.env.example and fill it in" }
$settings = @{}
foreach ($line in Get-Content $envFile) {
if ($line -match '^\s*(#|$)') { continue }
$key, $value = $line -split '=', 2
$settings[$key.Trim()] = "$value".Trim().Trim('"')
}
$distro = $settings.DISTRO
$checkout = "$($settings.CHECKOUT)".TrimEnd('/')
if (-not $distro -or -not $checkout) { throw "set DISTRO and CHECKOUT in $envFile" }
if (-not $Name) { $Name = Split-Path -Leaf (Split-Path -Parent $PSScriptRoot) }
# Replaces the block between the two marker lines, or appends it.
function Set-ProfileBlock([string] $Key, [string] $Body) {
$path = $PROFILE.CurrentUserCurrentHost
$begin = "# >>> $Key >>>"
$end = "# <<< $Key <<<"
$block = "$begin`r`n$Body`r`n$end"
$text = if (Test-Path $path) { [IO.File]::ReadAllText($path) } else { '' }
$pattern = '(?s)' + [regex]::Escape($begin) + '.*?' + [regex]::Escape($end)
if ($text -match $pattern) {
$text = [regex]::Replace($text, $pattern, { $block })
} else {
$text = $text.TrimEnd() + "`r`n`r`n$block`r`n"
}
New-Item -ItemType Directory -Force -Path (Split-Path $path) | Out-Null
[IO.File]::WriteAllText($path, $text)
Write-Host "profile: wrote $Key to $path"
}
$unc = "\\wsl.localhost\$distro" + ($checkout -replace '/', '\')
$wsl = "wsl.exe -d $distro --cd $checkout"
$ps = "& (Get-Process -Id `$PID).Path -NoProfile -ExecutionPolicy Bypass -File"
$lines = @(
"function $Name-cd { Set-Location '$unc' }"
"function $Name-sh { $wsl }"
"function $Name-backend { $wsl -- bash bin/backend.sh @args }"
# roledesk's additions: the PyCharm run configurations in .run/, from PowerShell.
"function $Name-pull { $wsl -- bash bin/pull.sh @args }"
"function $Name-worker { $wsl -- bash bin/worker.sh @args }"
"function $Name-check { $wsl -- bash bin/check.sh @args }"
"function $Name-1test { $wsl -- bash deploy/1-test-and-promote.sh @args }"
"function $Name-2build { $ps '$unc\deploy\2-build-image.ps1' }"
"function $Name-3push { $ps '$unc\deploy\3-push-image.ps1' }"
"function $Name-0release { $Name-1test @args; if (`$LASTEXITCODE -ne 0) { return }; $Name-2build; if (`$LASTEXITCODE -ne 0) { return }; $Name-3push }"
)
if (Test-Path (Join-Path $PSScriptRoot 'db.sh')) {
$lines += "function $Name-db { $wsl -- bash deploy/db.sh @args }"
}
Set-ProfileBlock "$Name (deploy/wsl-setup.ps1)" ($lines -join "`r`n")
Write-Host 'Open a new PowerShell window to load the profile.'
+22
View File
@@ -0,0 +1,22 @@
#!/bin/sh
# Once, inside the distro, for a checkout that lives in WSL: line endings, the
# Linux .venv, and a test run to prove it works. deploy/wsl-setup.ps1 is the
# Windows half. See docs/Install WSL.md, section 9 and Working from Windows.
# Usage: sh deploy/wsl-setup.sh (from the checkout root)
set -eu
export PATH="$HOME/.local/bin:$PATH"
# Windows Git (PyCharm) has core.autocrlf=true, so it writes CRLF into the
# working tree, and Linux Git then reports every line of those files changed.
# The repository's own setting wins over the Windows one for both.
git config core.autocrlf false
if [ -n "$(git diff --ignore-cr-at-eol --name-only)" ]; then
echo "uncommitted changes beyond line endings - commit or stash them first:" >&2
git diff --ignore-cr-at-eol --stat >&2
exit 1
fi
# Rewrites the CRLF copies with LF. Nothing else differs, checked above.
git checkout -- .
uv sync --all-extras
uv run pytest -q
+213
View File
@@ -0,0 +1,213 @@
> A copy of the boilerplate's `documentation/Install Server.md` at commit 09e144d, unedited below this line; the fix lands upstream first.
# Deploying a service
`<app>` below is your *service's* name: the hostname, the data directory, the image tag, the compose service. It is the one thing you choose. The Python package stays `app` and the environment variables stay `APP_`, here and in every project started from this repository — nothing inside the checkout is renamed, and the files these steps need are already in place under `deploy/`.
## What the box is
A single Hetzner server running Ubuntu 24.04 on x86_64, with a 37 GB root filesystem and no separate data disk. One Docker Compose project holds everything.
/srv/swag/docker-compose.yml every service, one file, project name `swag`
/srv/swag/.env every secret, chmod 600, never in the compose file
/srv/data/<app>/ every service's data, owned 1000:1000
/etc/config/swag/ SWAG's /config: nginx confs, certificates, logs
A snapshot of that whole compose file, every neighbour included, is [example-docker-compose.yml](example-docker-compose.yml): what a new service block lands beside, and where the SWAG subdomain list lives.
`swag` is nginx, certbot and fail2ban in one container. It terminates TLS for every hostname and proxies to the others **by container name** over the project's default network. That shared network is the only reason all these services live in one compose file — nothing else about them is related.
Images are published to the Gitea instance on the same box, at `git.rittershofer.org/wolf/<app>`. Everything runs as uid/gid 1000 and `TZ=Europe/Brussels`.
Which compose file that is comes off the running container rather than out of memory:
```bash
docker inspect swag --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}'
```
Declaring the network `external` from a second Compose project would work too, and would hardcode the first project's name into the second — a rename away from a confusing failure.
## The seven steps
**1. DNS.** `<app>.rittershofer.org` must resolve to the server *before* anything else. Confirm with `getent hosts <app>.rittershofer.org`.
**2. Certificate.** Add `<app>` to `SUBDOMAINS` in the swag service's environment (docker compose yaml). That list separates on commas and nothing else. SWAG runs certbot on every container start and requests **one certificate covering every name in the list**. If SWAG already holds a wildcard certificate covering the name, reuse it and skip this step entirely.
**3. Registry login.** On the build machine a Gitea token with package **write**; on the server, package **read**. Same command both ends, token at the password prompt:
```bash
docker login git.rittershofer.org -u wolf
```
Git credentials and Docker registry credentials are separate. A
**4. Data directory.** Root's `mkdir` produces a root-owned directory, and the container runs as 1000, so the chown is not optional:
```bash
sudo mkdir -p /srv/data/<app>/home && sudo chown -R 1000:1000 /srv/data/<app>
```
**5. Service block** from [../deploy/server/compose.yaml](../deploy/server/compose.yaml) into `/srv/swag/docker-compose.yml`, the variables from [../deploy/server/env.example](../deploy/server/env.example) appended to `/srv/swag/.env`, and [../deploy/server/backup.sh](../deploy/server/backup.sh) next to the compose file for the nightly cron. Copy [../deploy/server/config.example.yaml](../deploy/server/config.example.yaml) to `$APP_DATA/config.yaml` with `rp_id` and `origin` set to the hostname: it holds the keys whose defaults are wrong in a container (`server.host: 0.0.0.0`, `database: /data/app.db`, `auth.enabled: always`, `auth.secure_cookie: always`). Without it the app binds loopback inside the container and writes its database into the image's root-owned folder.
**6. Proxy conf** — [../deploy/server/swag/app.subdomain.conf](../deploy/server/swag/app.subdomain.conf) renamed to `<app>.subdomain.conf` in `/etc/config/swag/nginx/proxy-confs/`. Then validate, and reload only if validation passes:
```bash
docker exec swag nginx -t && docker exec swag nginx -s reload
```
Confirm the other sites still respond afterwards.
**7. Deploy.** Pull first, since `up` alone keeps the `:latest` it already has, and name the service, so nothing else in the file is recreated:
```bash
cd /srv/swag && docker compose pull <app> && docker compose up -d <app>
```
[../deploy/server/deploy.sh](../deploy/server/deploy.sh) is the same two commands with a check that `config.yaml` exists and sets `database` and `auth.enabled`, and a wait for the health check before it returns.
## What the templates already decide for you
Both templates are in [../deploy/server/](../deploy/server), commented in place. The decisions worth knowing before you read them:
**No `ports:`.** SWAG reaches the container by name on the project network. That is what keeps the app's port off the internet even if the firewall is wrong.
**No `networks:` either.** Being in that compose file is what puts it on the right one.
**`user: "1000:1000"`.** Container uid is host uid — there is no translation layer. Without this the process runs as whatever the image declares, usually root, and everything it writes into `/srv/data` lands root-owned.
**`mem_limit:`, not `deploy: resources: limits:`.** The latter is Swarm-only and silently ignored by `docker compose up`. 0 is unlimited, which is the honest default until the box has been watched under real load.
**`logging:` with a size cap.** Without it, json-file logs grow until the disk does not.
**`listen 443 ssl` without `http2`.** `http2 on;` is set once at `http` scope in `nginx.conf`, and the `listen ... http2` parameter is deprecated in nginx 1.25 and later.
**`proxy_read_timeout 86400s`** only if the app holds a WebSocket open. SWAG's `proxy.conf` already sets the Upgrade headers; what it also sets is a read timeout of a couple of minutes, which closes an idle socket on the minute. Delete those two lines if there is no socket.
# Publishing a release
The scripts are in place in `deploy/`, with `deploy/server/` beside them. The Dockerfile at `deploy/Dockerfile`, which is where `2-build-image.ps1` looks for it in the tree step 1 exports.
The release is split where the machines are. The checkout, its `.venv` and the tests are Linux, so the gate, step 1, is bash and runs inside the WSL distro. The distro is a sealed sandbox (Windows interop off, no drive mounts, no container engine, see [Install WSL.md, 7. Cut the distro off from Windows](Install%20WSL.md#7-cut-the-distro-off-from-windows)) and stays that way, because a sandbox that can drive containers can mount the host. Step 1 ends by exporting the production tree to `release/<folder>/<sha>/` beside the checkout, and the image is built and pushed from Windows with `wslc`, the container CLI that ships with WSL: steps 2 and 3, two short PowerShell scripts that read that tree through `\\wsl.localhost`. The distro never holds a container socket.
## Every release
From Windows, in PowerShell, once `deploy\wsl-setup.cmd` has installed the profile commands:
```powershell
<folder>-0release
```
where `<folder>` is the checkout folder's name (`boilerplate` here). It runs `<folder>-1test`, `<folder>-2build` and `<folder>-3push` in order and stops at the first that fails; its arguments (`--push-branch`, `--no-tests`) go to step 1. Each step is a command of its own:
```text
<folder>-1test bash deploy/1-test-and-promote.sh in WSL: test local main, fast-forward local production, export it
<folder>-2build deploy\2-build-image.ps1 on Windows: build the exported tree with wslc
<folder>-3push deploy\3-push-image.ps1 on Windows: push the built image to the registry
```
The number is the order. Each script can be run on its own and works out its paths for itself, so the working directory does not matter.
**Step 1 never builds; step 2 never pushes; step 3 never rebuilds or deploys.** That separation is the point of the numbering: a failed push is `3` again, not the whole thing again.
## What each step is actually doing
**1: test and promote.** Resolves `main` to a commit, checks that commit out into a throwaway worktree, and runs the project's checks there. The worktree is what makes this honest: the tests run against exactly the tree that is about to become production, with none of the working copy's uncommitted edits in the way. Worktrees have no `.venv` of their own, so the interpreter comes from the main checkout and the code from the worktree.
The checks are pytest, then `npm ci` and `npm run check` in `frontend/`: lint, format, the generated types, the page's tests and a build. So the machine that runs step 1 needs Node, which `frontend/install.sh` installs once. Step 1 runs the gates through `deploy/test-commit.sh`, which looks for Node from nvm and from fnm, since a non-interactive shell has not read the rc file that puts it on `PATH`. `PYTHON` in `release.env` names the interpreter when it is not `.venv/bin/python`; `--no-tests` skips the gates.
Promotion is a fast-forward only. If `production` has commits `main` does not, that is a release made from somewhere else, and the script stops rather than discarding it.
Git publication is optional: run `git push origin production` yourself, or pass `--push-branch` to step 1 or `<folder>-0release`. `production` identifies the tested candidate; the server's running image identifies the deployed release.
Step 1 then exports the production commit: `git archive` unpacked into `release/<folder>/<sha>/` beside the checkout (the environment variable `RELEASE_DIR` names another folder for `release/`), with `deploy/Dockerfile.dockerignore` copied to `.dockerignore` at its root, and the short sha written to `release/<folder>/production`. Older trees are removed; a tree that already exists for this sha is kept, because `wslc` can build from a stale view of a folder deleted and recreated at the same path ([microsoft/WSL#41287](https://github.com/microsoft/WSL/issues/41287)). When the folder cannot be written, step 1 says so and fails after the promote; run it again.
**2: build.** Reads the sha from `production` and runs `wslc build -t <IMAGE>:<sha> -t <IMAGE>:latest -f <tree>\deploy\Dockerfile <tree>` with the tree's `\\wsl.localhost` path. `-f` is a path `wslc` opens on Windows and hands to the builder on stdin, which is why the ignore rules have to be at the root of the context. The image builds the React page itself, in a Node stage from `frontend/` and its lock file, so neither Node nor a built `app/web/dist/` is needed for this step.
`wslc` builds an **image**. No container starts or restarts on this machine.
**3: push.** Re-tags `latest` onto this candidate with `wslc tag`, then `wslc push`es both. It never rebuilds, so a push that failed on connectivity is just this step again. A refused push prints the `wslc login` line to use.
# Proposal - Operating it
[../deploy/server/backup.sh](../deploy/server/backup.sh), [../deploy/server/import.sh](../deploy/server/import.sh) and [../deploy/server/deploy.sh](../deploy/server/deploy.sh) name the compose service in `SERVICE=` at their top, and `NAME` in `deploy/release.env` names it for [../deploy/db.sh](../deploy/db.sh). Each has to be `<app>`. All of them read `APP_DATA` from `/srv/swag/.env`.
## The nightly backup
`backup.sh` beside the compose file takes one snapshot through the running container. Inside it, the app's own config names the database, which is opened read-only, so a path that names no file is an error rather than a new empty database. `VACUUM INTO` writes `/data/backups/<app>-<stamp>.db` as one consistent, compacted file without blocking a writer for long. The snapshot is reopened read-only and must hold every table in `TABLES` (`jobs ticks`; a test keeps it equal to `REQUIRED_TABLES` in [../deploy/server/import_database.py](../deploy/server/import_database.py)); otherwise it is deleted and the run fails. A project that replaces `ticks` with its own first table changes both. The snapshot is gzipped under `umask 077`, so every backup is owner-only, and everything past the newest `KEEP` (14 by default) is deleted.
Schedule it in a crontab that can run `docker compose`:
```
0 4 * * * bash /srv/swag/backup.sh >> /srv/swag/backup.log 2>&1
```
The container has to be running, since the snapshot is taken through `docker compose exec`. A backup on the same disk is not an off-server copy; `db.sh pull` below makes one.
## Moving the database between laptop and server
`deploy/db.sh` runs in the checkout, with the `ssh` and `scp` of that shell, and reads `SSH_HOST`, `REMOTE_DIR` (`/srv/swag`), `REMOTE_DATA` (`/srv/data/<app>`) and `NAME` from `deploy/release.env`.
`bash deploy/db.sh pull` runs the server's own `backup.sh` over ssh and downloads the newest `<app>-*.db.gz` into `backups/` in the checkout. Use it as a read-only copy. Pushing it back after the server has written discards those writes, and nothing merges two copies.
Pushing the laptop's database onto the server replaces the server's, including the password hash, sessions, passkeys and API tokens, because the `auth_*` tables live in the same file. Set the account up against the final database, not before the push.
**1. Stop the app.** The import is a file swap under SQLite and `import.sh` refuses a running service:
```bash
cd /srv/swag && docker compose stop <app>
```
**2. Push.** From the checkout:
```bash
bash deploy/db.sh push
```
It snapshots the local database (the one the checkout's config names, or `LOCAL_DB`) with `VACUUM INTO`, gzips it, asks before going on unless given `--yes`, uploads it as `$REMOTE_DATA/incoming.db.gz` and `import_database.py` into `$REMOTE_DIR`, and streams `import.sh` over ssh. On the server, `import.sh` runs `import_database.py` in a one-off container of the image already there (`--pull never`). The importer decompresses into a temporary file beside the database and requires `PRAGMA integrity_check` to answer `ok` and every table in `REQUIRED_TABLES` to exist. Only then does it snapshot the current database with `VACUUM INTO` into `/data/backups/<database name>-<stamp>.db.gz`, owner-only, refuse if `wal_checkpoint(TRUNCATE)` finds another connection, delete the `-wal` and `-shm` files and move the new file into place with one `os.replace`. It prints the snapshot's path. For `/data/app.db` the snapshot is `app-<stamp>.db.gz`, which `backup.sh` prunes with its own only while the service is also called `app`.
**3. Start the app.**
```bash
cd /srv/swag && docker compose up -d --wait <app>
```
## Restoring a backup on the server
The same import, by hand, with a file from `$APP_DATA/backups` in place of the pushed one. `import.sh` and `import_database.py` from `deploy/server/` have to be beside the compose file; `db.sh push` leaves the importer there but streams `import.sh`.
**1. Stop the app.** `cd /srv/swag && docker compose stop <app>`
**2. Stage the backup** as the incoming file:
```bash
sudo -u '#1000' cp /srv/data/<app>/backups/<app>-<stamp>.db.gz /srv/data/<app>/incoming.db.gz
```
The backups are owner-only and belong to uid 1000, which the importer runs as; a copy made as root is one it cannot read.
**3. Import.** `bash /srv/swag/import.sh /srv/swag` checks it, snapshots the current database and replaces it, as in step 2 above.
**4. Start the app.** `cd /srv/swag && docker compose up -d --wait <app>`
## Rolling back to a pinned image
Every release pushes `<IMAGE>:<sha>` and `<IMAGE>:latest`, where `<sha>` is the short hash of the promoted commit. `latest` moves on every release, so it cannot name an earlier one; the compose file runs `${APP_IMAGE}:${APP_TAG:-latest}`, and `APP_TAG` in `.env` pins a commit tag.
**1. Find the tag.** The image's package page in Gitea lists every pushed tag with its date. A tag removed by the package cleanup rule cannot be rolled back to.
**2. Pin it.** Set `APP_TAG=<sha>` in `/srv/swag/.env`.
**3. Deploy.** `bash /srv/swag/deploy.sh` pulls the pinned image, which the weekly prune removes from the box once no container runs it, and recreates `<app>` with it.
**4. Unpin** with `APP_TAG=latest` and deploy again once a fixed release is out.
A rollback replaces the code, not the database. The queue's tables only ever gain columns, each nullable or with a default; `pipeline/table.py` adds them and never drops one. A project whose migration changed the schema in a way the older code cannot read restores a backup taken before that release, with the app stopped, as above.
# Pitfalls
**Disk.** 37 GB, and it has been full. Four causes, all recurring. `:latest` orphans the previous image on every pull and nothing cleans them up; root's crontab runs `docker image prune -af` every Sunday at 04:00, and the `-a` also removes a pinned image that is not running, so a rollback tag must be pulled again. Gitea's package store under `/srv/data/gitea/gitea/packages` keeps every pushed `:sha` and `:latest` version of every image forever: it was 11 GB on 16 October 2026 after a week of newsdesk releases at 2.6 GB each, and the prune cron never touches it. A cleanup rule per owner at `/user/settings/packages/rules` (keep the most recent n, **container** type) is applied by Gitea's `cleanup_packages` cron at midnight, which also drops blobs unreferenced for 24 hours; `/-/admin/packages` lists every version with a delete button and a Cleanup expired data button runs the same job now. Journald defaults to 10% of the filesystem and is capped here at 200 MB. VS Code Remote installs a fresh server per version under `~/.vscode-server/cli/servers` forever. [disk_usage.sh](disk_usage.sh) prints all of it, read-only.
**SQLite is never safe to copy live.** Stop the container, or snapshot with `VACUUM INTO` through the running one. A `cp` of a WAL-mode database is a coin toss.
**Secrets in `.env`, never in the compose file.** It is easy to paste a compose file into a chat window; it is harder to do that by accident with a file you had to `cat` deliberately.
+408
View File
@@ -0,0 +1,408 @@
> A copy of the boilerplate's `documentation/Install WSL.md` at commit 09e144d, unedited below this line; the fix lands upstream first.
# Abstract
The sandbox is a WSL distro. Claude Code runs inside it from the desktop app with all permissions, because the distro cannot reach anything on Windows. Interops is turned off, no drives are mounted, and to keep it this way admin permissions are not granted.
Root is reached only from Windows, with `wsl -d claudebox -u root`. That is how you install and update packages later.
The repos live inside the distro in `/home/claude/cboxcode`. [Working from Windows](#working-from-windows) covers working with them from Windows.
# Install
## 1. Check that WSL 2 runs
```
PS> wsl --status
```
It should say `Default Version: 2` and nothing about virtualisation.
If it says virtualisation is not enabled, run `wsl.exe --install --no-distribution` once in an admin PowerShell and reboot. If it still complains, switch on Intel VT-x or AMD-V in the BIOS. This is the only step that can need admin rights.
If a `claudebox` from an earlier attempt exists, `wsl -l -v` lists it. `wsl --unregister claudebox` deletes it with everything inside.
## 2. Install Ubuntu as `claudebox`
```
PS> wsl --install Ubuntu-24.04 --name claudebox --no-launch --web-download
```
- `--name claudebox` names the distro, so it does not collide with an ordinary Ubuntu.
- `--no-launch` skips Ubuntu's first-start wizard, which would create a user with sudo. We create our own user in step 5.
- `--web-download` downloads it directly instead of through the Microsoft Store.
## 3. Open a root shell
```
PS> wsl -d claudebox -u root
```
The prompt changes to something like `root@PC:/mnt/c/...#`. You are now inside Linux as root. `C:` is still visible at this point; step 7 takes it away.
## 4. Update and install packages
```
# apt update
# apt upgrade -y
# apt install -y build-essential git curl ripgrep sqlite3 jq unzip
```
`apt update` fetches the package lists, `apt upgrade` installs updates, and `apt install` adds the tools: compilers for Python packages that build from source, git, curl, ripgrep for Claude's searches, sqlite3 to look into the databases, jq for JSON, unzip.
## 5. Create the user `claude`, without sudo
```
# useradd --create-home --shell /bin/bash claude
# groups claude
```
`useradd` creates the user with a home directory and bash. It gets no password and no groups. `groups claude` must print only `claude : claude`. If `sudo` appears there, the user can become root, and `gpasswd -d claude sudo` removes it.
```
# passwd -l root
```
This locks root's password, so `su` cannot become root from inside either. Ubuntu ships it locked, so this only makes sure.
## 6. Install uv and Claude Code as `claude`
Switch to the user. Root can do that without a password:
```
# su - claude
```
The prompt ends in `$` now. Install the two tools into `~/.local/bin`, which belongs to `claude`, so they update without root:
```
$ curl -LsSf https://astral.sh/uv/install.sh | sh
$ curl -fsSL https://claude.ai/install.sh | bash
$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
```
The third line puts `~/.local/bin` on the path for every new shell.
Set the git identity, with your own name and email, and create the folder for the repos:
```
$ git config --global user.name "Your Name"
$ git config --global user.email "you@example.com"
$ git config --global init.defaultBranch main
$ mkdir ~/cboxcode
```
Back to root:
```
$ exit
```
## 7. Cut the distro off from Windows
`/etc/wsl.conf` is the file WSL reads when the distro starts. Open it in the editor:
```
# nano /etc/wsl.conf
```
Delete what is there and type this:
```
[boot]
systemd=true
[user]
default=claude
[automount]
enabled=false
mountFsTab=false
[interop]
enabled=false
appendWindowsPath=false
```
- `[user] default=claude` makes every normal start log in as `claude`, not root.
- `[automount] enabled=false` stops WSL from mounting `C:` and the other drives under `/mnt`. `mountFsTab=false` also ignores `/etc/fstab`, so no other way of mounting them applies.
- `[interop] enabled=false` stops Linux from starting Windows programs. That includes `wsl.exe`, which would otherwise be a way back to root. `appendWindowsPath=false` keeps the Windows folders off the Linux path.
- `[boot] systemd=true` is Ubuntu's default and stays.
Save with `Ctrl+O` then `Enter`, quit with `Ctrl+X`.
The first start in step 3 left an empty folder per Windows drive under `/mnt`. Nothing will be mounted on them any more, so remove them to avoid confusion. `rmdir` only removes empty folders, so it cannot delete anything:
```
# rmdir /mnt/c /mnt/d
```
Name every drive letter that `ls /mnt` shows. `wsl` and `wslg` belong to WSL and stay. Leave the distro:
```
# exit
```
WSL only reads the file on a fresh start, so stop the distro:
```
PS> wsl --terminate claudebox
```
## 8. Check the isolation
Start it the normal way, which now logs in as `claude`:
```
PS> wsl -d claudebox
```
Each check and what it must answer:
```
$ whoami
claude
$ groups
claude
$ passwd -S claude
claude L ...
$ sudo -n true
sudo: a password is required
$ ls /mnt
wsl wslg
$ uv --version
$ claude --version
```
- `groups` showing only `claude` means no admin group, so sudo refuses even with a password.
- `L` in `passwd -S` means the password is locked: no password exists, so nothing can be typed at a sudo or su prompt that works. A plain `sudo` still asks for one, and any answer fails.
- `ls /mnt` showing no drive letters means no Windows drive is there. `mount | grep -i drvfs` printing nothing confirms it.
- The last two print version numbers.
The real test that Windows programs cannot start needs one. In a second PowerShell window, copy a harmless one in from Windows, run it inside, and delete it again:
```
PS> Copy-Item C:\Windows\System32\whoami.exe \\wsl.localhost\claudebox\tmp\
PS> wsl -d claudebox -- /tmp/whoami.exe
PS> Remove-Item \\wsl.localhost\claudebox\tmp\whoami.exe
```
With interop off it prints no Windows user name, only a WSL error after a few seconds. If it prints your Windows name, step 7 did not take.
If any check answers differently, the step it belongs to did not take: 5 for `groups` and `passwd`, 7 for `/mnt` and `whoami.exe`.
```
$ exit
```
## PowerShell 7
Our scripts are running on powershell 7, which needs to be installed.
Then, set it to the default profile in the terminal app.
```
PS> winget install Microsoft.PowerShell
```
## The execution policy
Running commands in the WSL need extra permissions:
```
PS> pwsh -ExecutionPolicy Bypass -File <command>
```
This is one reason why we add all important commands into the profile
## PyCharm
### Remote execution agent failure
PyCharm on Windows is not able to read folders on the cut-off WSL. Its bundled plugin Remote Execution Agent copies a helper called IJent into the distro from `/mnt/c/Program Files/JetBrains/...` and reads files and runs Git through it. With the drives unmounted the copy fails: opening a file reports "unable to locate", Git reports `execvpe(.../ttyfix) failed`, and `idea.log` shows `cp: cannot stat '/mnt/c/...ijent...'`.
So you need to disable the plugin: Settings > Plugins > Installed, search for `Remote Execution Agent`, untick it, restart PyCharm.
### Git
Set Settings > Version Control > Git > Path to Git executable to `C:\Program Files\Git\cmd\git.exe`, so it uses Windows Git instead of trying to start the distro's.
### Run configurations
We currently don't use them.
## Maintenance
### Update
Packages, as root from Windows:
```
PS> wsl -d claudebox -u root -- apt update
PS> wsl -d claudebox -u root -- apt upgrade -y
```
Claude Code updates itself. uv updates with the line below. The full path is needed because a command passed from PowerShell does not load `~/.bashrc`:
```
PS> wsl -d claudebox -- /home/claude/.local/bin/uv self update
```
### Remove WSL
Remove the distro with all repos in it, after pushing what matters:
```
PS> wsl --unregister claudebox
```
# Getting it ready for use
## Git
Tell Windows Git to trust that folder, since its files belong to the Linux user:
```
PS> git config --global --add safe.directory '%(prefix)///wsl.localhost/claudebox/home/claude/cboxcode/news'
```
`git config core.autocrlf false` stops Git from converting line endings in this repo. `git checkout -- .` rewrites the files Windows Git checked out with Windows line endings, so Linux Git does not report every line as changed.
## Working from Windows
The only direction that works is Windows into WSL: Windows reads and writes the checkout through `\\wsl.localhost\<distro>\...` and starts processes with `wsl.exe`. The distro never reaches back. Editing and the Git UI happen on Windows; the agents, the tests, `serve`, the jobs, `uv sync`, release step 1 and `db.sh` run in the distro, typed from Windows through the profile commands below; release steps 2 and 3 run on Windows with `wslc`, since the distro must not hold a container socket.
A `.venv` belongs to one OS: `bin/python` on Linux, `Scripts\python.exe` on Windows. The `.venv` in the checkout is the distro's. A Windows `uv sync` or `uv run` in that folder would replace it with a Windows one, and the distro's next `uv run` would swap it back, so run `uv` in the distro (`<folder>-sh` opens a shell there). A Windows interpreter for an editor's code insight belongs outside the checkout.
Run the app in the distro, not from Windows. Only one side may open a SQLite file at a time, since locking through the share is unreliable, and file change notifications through the share are unreliable too, so `--reload` started from Windows can miss edits.
## WSL Containers for the image
We don't need Docker Desktop but build images directly on WSL with `wslc`. It needs WSL 2.9.3 or later:
```
PS> wsl --update
PS> wsl --version
PS> wslc version
```
Log it in to the registry once, with a token that has package-write access, typed at its password prompt:
```
PS> wslc login git.rittershofer.org -u wolf
```
## Env files
Once per checkout, including every new repo, and again when `DISTRO` or `CHECKOUT` changes:
1. Copy `deploy/release.env.example` to `deploy/release.env` in the checkout (`cp` in the distro, or Explorer) and fill in `DISTRO` (`claudebox`), `CHECKOUT` (the path in the distro, `/home/claude/cboxcode/news`) and `IMAGE`. The file is gitignored.
## PS Profile commands
We want a simple way to run and deploy the app, `cd` into the repo. So we have script adding the relevant commands to the PS profile:
```
PS> pwsh -ExecutionPolicy Bypass -File \\wsl.localhost\claudebox\home\claude\cboxcode\news\deploy\wsl-setup.ps1
```
It creates commands into the profie
The prefix is the checkout folder's name, `news-` for `news` and `boilerplate-` for `boilerplate`; `-Name` on the `.ps1` sets another. The setup writes one block between `# >>> news (deploy/wsl-setup.ps1) >>>` and `# <<< news (deploy/wsl-setup.ps1) <<<` into `$PROFILE` and replaces it when run again. Blocks written by older versions of the setup, `newsdesk (deploy/wsl-setup.ps1)` and `wsl shared (deploy/wsl-setup.ps1)`, call scripts that no longer exist; delete them with `notepad $PROFILE`.
| Command | Runs | Does |
|---|---|---|
| `news-cd` | `Set-Location` | PowerShell into the checkout's folder |
| `news-sh` | `wsl.exe` | a shell in claudebox, in the checkout |
| `news-backend [-v]` | `bin/backend.sh` in claudebox | `uv run app serve` on :8787, until Ctrl-C |
| `news-frontend` | `bin/frontend.sh` in claudebox | Vite with hot reload on :5173, until Ctrl-C |
| `news-check` | `bin/check.sh` in claudebox | the gate on the working tree, without promoting |
| `news-1test [--push-branch] [--no-tests]` | `deploy/1-test-and-promote.sh` in claudebox | test local main, fast-forward `production`, export it |
| `news-2build` | `deploy\2-build-image.ps1` on Windows | build the exported tree with `wslc`, tags `<sha>` and `latest` |
| `news-3push` | `deploy\3-push-image.ps1` on Windows | push both tags with `wslc` |
| `news-0release [--push-branch] [--no-tests]` | the three above | `1test`, `2build`, `3push`, stopping at the first that fails |
| `<folder>-db pull\|push` | `deploy/db.sh` in claudebox | only where `deploy/db.sh` exists, as in the boilerplate |
If the setup does not run, paste the block by hand. `New-Item -Force $PROFILE` creates the file if `Test-Path $PROFILE` says it is missing; `notepad $PROFILE` opens it. For `news`, with `DISTRO=claudebox` and `CHECKOUT=/home/claude/cboxcode/news`, the setup writes exactly this:
```powershell
# >>> news (deploy/wsl-setup.ps1) >>>
function news-cd { Set-Location '\\wsl.localhost\claudebox\home\claude\cboxcode\news' }
function news-sh { wsl.exe -d claudebox --cd /home/claude/cboxcode/news }
function news-backend { wsl.exe -d claudebox --cd /home/claude/cboxcode/news -- bash bin/backend.sh @args }
function news-frontend { wsl.exe -d claudebox --cd /home/claude/cboxcode/news -- bash bin/frontend.sh @args }
function news-check { wsl.exe -d claudebox --cd /home/claude/cboxcode/news -- bash bin/check.sh @args }
function news-1test { wsl.exe -d claudebox --cd /home/claude/cboxcode/news -- bash deploy/1-test-and-promote.sh @args }
function news-2build { & (Get-Process -Id $PID).Path -NoProfile -ExecutionPolicy Bypass -File '\\wsl.localhost\claudebox\home\claude\cboxcode\news\deploy\2-build-image.ps1' }
function news-3push { & (Get-Process -Id $PID).Path -NoProfile -ExecutionPolicy Bypass -File '\\wsl.localhost\claudebox\home\claude\cboxcode\news\deploy\3-push-image.ps1' }
function news-0release { news-1test @args; if ($LASTEXITCODE -ne 0) { return }; news-2build; if ($LASTEXITCODE -ne 0) { return }; news-3push }
# <<< news (deploy/wsl-setup.ps1) <<<
```
For another checkout, replace `news` and the path.
### When a command fails
- `wsl -l -v` lists the distros. `DISTRO` must be one of the names exactly.
- `Test-Path \\wsl.localhost\claudebox\home\claude\cboxcode\news` must say `True`. `\\wsl$\claudebox\...` is the older spelling of the same share; if only that one answers, use it in the block.
- `$PROFILE` prints the profile file this PowerShell loads, and `Get-Command news-*` lists the commands it defined. Commands missing in one PowerShell and present in another means the setup ran in the other one.
- `Get-ExecutionPolicy -List` shows the policy per scope. `Restricted` or `AllSigned` where it applies means the profile does not load; a `MachinePolicy` or `UserPolicy` entry is group policy and overrides `CurrentUser`.
- `wslc version` failing means WSL is older than 2.9.3: `wsl --update`.
- `news-2build` saying there is no `production` file means step 1 has not exported yet: run `news-1test`.
- If `wslc build` refuses the `\\wsl.localhost` folder as its context, which Microsoft's documentation neither promises nor rules out, copy the tree to Windows and build there: `Copy-Item -Recurse \\wsl.localhost\claudebox\home\claude\cboxcode\release\news\<sha> $env:TEMP\release-<sha>`, then the `wslc build` line of `2-build-image.ps1` with that folder.
# Kill and restart
Run everything in an **admin PowerShell** on Windows.
### Normal shutdown
`wsl --shutdown`
### Killing it
If normal shutdown hangs:
```
taskkill /f /im wsl.exe
taskkill /f /im wslhost.exe
taskkill /f /im wslrelay.exe
```
### Check the service
`sc queryex WslService`
If it says `STOP_PENDING` or `START_PENDING`, take the PID from the output and run `taskkill /PID <PID> /F`.
If it says `STOPPED`, start it with `sc start WslService`, then run `wsl`.
### **Kill a lingering VM**
IF it still does not work check the VM
```
hcsdiag list
hcsdiag kill <ID>
```
If the old WSL VM is still listed, kill it with the ID shown. This was reported as the only fix short of a reboot in a complete-freeze case.
### **Restart the Hyper-V Host Compute Service**
```
Restart-Service vmcompute
```
A wedged vmcompute service is a common reason WSL stops working after it ran fine earlier in the same Windows session.
If it still hangs after all four, a reboot is the reliable fix.
Once it’s back up, running `wsl --update` is worth doing, since hangs like this are often fixed in newer versions.
# Pitfalls
- **Only open the distro from a normal PowerShell.** Opened from an admin one, WSL mounts the Windows drives despite `automount` being off ([microsoft/WSL#12443](https://github.com/microsoft/WSL/issues/12443)). `wsl --terminate claudebox` and a normal shell undo it.
- **Changes to `/etc/wsl.conf` apply only after `wsl --terminate claudebox`.**
-175
View File
@@ -1,175 +0,0 @@
services:
mealie:
image: ghcr.io/mealie-recipes/mealie:latest
container_name: mealie
restart: unless-stopped
# `deploy: resources: limits:` is Swarm-only and was silently ignored,
# so this container has never actually had a memory limit. This one works.
mem_limit: 1000m
# Was 9925:9000 on every interface — the internet, if the firewall is ever
# wrong. SWAG reaches it by container name on the shared network; this is
# only the fallback for when you are on the box itself.
ports:
- "127.0.0.1:9925:9000"
# Was a Docker-managed volume under /var/lib/docker, root-owned and
# invisible. A plain directory next to this file, so a backup is `tar`.
volumes:
- /srv/data/mealie:/app/data
environment:
ALLOW_SIGNUP: "false"
PUID: 1000
PGID: 1000
TZ: Europe/Brussels
MAX_WORKERS: 1
WEB_CONCURRENCY: 1
BASE_URL: https://mealie.rittershofer.org
# From .env, chmod 600. Never in this file.
OPENAI_API_KEY: ${OPENAI_API_KEY:?set OPENAI_API_KEY in .env}
# A base, not an endpoint. The old /v1/models made it request
# /v1/models/chat/completions.
OPENAI_BASE_URL: https://api.openai.com/v1
logging:
driver: json-file
options:
max-size: "20m"
max-file: "5"
gitea:
image: gitea/gitea:latest
container_name: gitea
restart: unless-stopped
environment:
- USER_UID=1000
- USER_GID=1000
volumes:
- /srv/data/gitea:/data
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
logging:
driver: json-file
options:
max-size: "20m"
max-file: "5"
newsdesk:
image: ${NEWSDESK_IMAGE:?set NEWSDESK_IMAGE in .env}:${NEWSDESK_TAG:-latest}
container_name: newsdesk
restart: unless-stopped
user: "${PUID:-1000}:${PGID:-1000}"
volumes:
- /srv/data/newsdesk:/data
environment:
TZ: ${TZ:-Europe/Berlin}
CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN:-}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
NEWSDESK_PASSWORD: ${NEWSDESK_PASSWORD:-}
mem_limit: ${NEWSDESK_MEM:-0}
logging:
driver: json-file
options:
max-size: "20m"
max-file: "5"
bookly:
image: git.rittershofer.org/wolf/bookly:${BOOKLY_TAG:-latest}
container_name: bookly
restart: unless-stopped
user: "1000:1000"
volumes:
- /srv/data/bookly:/data
environment:
TZ: Europe/Brussels
# Required: without it the demo would be open to anyone who finds the hostname, so Compose refuses to start.
BOOKLY_PASSWORD: ${BOOKLY_PASSWORD:?set BOOKLY_PASSWORD in .env}
# subscription: the Claude CLI with CLAUDE_CODE_OAUTH_TOKEN (same variable newsdesk reads).
# api: ANTHROPIC_API_KEY, billed per token, lower latency.
BOOKLY_BACKEND: ${BOOKLY_BACKEND:-subscription}
CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN:-}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
# No `ports:`. SWAG reaches :8000 by container name; nothing else can.
mem_limit: 0
logging:
driver: json-file
options:
max-size: "20m"
max-file: "5"
health:
image: ${HEALTH_IMAGE:?set HEALTH_IMAGE in .env}:${HEALTH_TAG:-latest}
container_name: health
restart: unless-stopped
# Warehouse, journal, job table, speech models, tiles, backups: all of it
# under one host directory, so a backup is one target.
volumes:
- ${HEALTH_DATA:-./data/health}:/data
user: "${PUID:-1000}:${PGID:-1000}"
environment:
TZ: ${TZ:-Europe/Berlin}
HEALTH_TIMEZONE: ${TZ:-Europe/Berlin}
# Which model writes the evening entry. The subscription token now; add
# ANTHROPIC_API_KEY and `docker compose up -d health` (not `restart`)
# to move to the API. Shared with the other services in this file.
CLAUDE_CODE_OAUTH_TOKEN: ${CLAUDE_CODE_OAUTH_TOKEN:-}
ANTHROPIC_API_KEY: ${HEALTH_ANTHROPIC_API_KEY:-}
# Normally empty. Set it to pin the password -- the way back in when a
# login is lost -- then empty it and recreate to hand the password back
# to the Settings page.
HEALTH_PASSWORD: ${HEALTH_PASSWORD:-}
# The speech model. `small` (0.7 GB) if turbo (1.9 GB while it runs)
# does not fit beside everything else on the box.
HEALTH_WHISPER_MODEL: ${HEALTH_WHISPER_MODEL:-large-v3-turbo}
# No `ports:` and no `networks:`: SWAG reaches it by name on this file's
# network, and nothing else reaches it at all.
# `deploy: resources: limits:` is Swarm-only and ignored here. 0 is
# unlimited until a transcription has been watched on the box.
mem_limit: ${HEALTH_MEM:-0}
logging:
driver: json-file
options:
max-size: "20m"
max-file: "5"
swag:
image: lscr.io/linuxserver/swag:latest
container_name: swag
cap_add:
- NET_ADMIN
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/Brussels
- URL=rittershofer.org
- VALIDATION=http
# Commas and nothing else — SWAG splits on the comma alone, so a space
# asks for a hostname with a leading space. `news` added for newsdesk;
# `collabora` dropped, since nothing serves it any more.
- SUBDOMAINS=www,mealie,git,news,play,body
- ONLY_SUBDOMAINS=false
volumes:
- /etc/config/swag:/config
ports:
- 443:443
- 80:80
logging:
driver: json-file
options:
max-size: "20m"
max-file: "5"
restart: unless-stopped
+10 -2
View File
@@ -16,7 +16,7 @@ What was decided for roledesk and the reason for each. The target design is the
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, and joining on one match merged both on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member. **A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, and joining on one match merged both on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member.
**One posting per place is folded too, on every host.** A host that lists one job once per office gives rows of the same company and title whose towns never meet; `dupes.fold` merges those of one host when their texts share at least `SAME_TEXT` (30%) of their words, and the kept role keeps each copy's place and link. It applies to every host, not only to readers that declare `places="copies"`, because most such rows came from hosts that do not declare it. A remote listing and an office listing stay two roles: the kept role takes a copy's places and not its remote, and the fetch gate reads a remote role by its area, so a remote listing folded behind an office elsewhere was rejected with it, and an office listing behind a remote role elsewhere likewise. **One posting per place is folded too, on every host.** A host that lists one job once per office gives rows of the same company and title whose towns never meet; `dupes.fold` merges those of one host when their texts share at least `SAME_TEXT` (30%) of their words, and the kept role keeps each copy's place and link. It applies to every host, not only to readers that declare `places="copies"`, because most such rows came from hosts that do not declare it. It keeps to one country, as ADR 1.4 has it, since the same rule across countries merged different openings whose texts happened to be alike (a federal sales role in the US behind one in Australia); a job posted once per country is then one role per country, which costs a few rows for a pan-European posting and hides nothing. A remote listing and an office listing stay two roles: the kept role takes a copy's places and not its remote, and the fetch gate reads a remote role by its area, so a remote listing folded behind an office elsewhere was rejected with it, and an office listing behind a remote role elsewhere likewise.
**A copy is stored as `roles.dup_of` and rebuilt from scratch after every pull.** Every reader of open roles adds `dup_of IS NULL`, which keeps `idx_roles_posted` usable where a window function over the filtered rows would scan them all. Where a group changed, the kept role takes the earliest `read_at` and each agent's latest vote from its copies; an unchanged group is left alone, so marking a role unread again sticks. The per-board counts on Sources still count copies, since they report what a board holds. **A copy is stored as `roles.dup_of` and rebuilt from scratch after every pull.** Every reader of open roles adds `dup_of IS NULL`, which keeps `idx_roles_posted` usable where a window function over the filtered rows would scan them all. Where a group changed, the kept role takes the earliest `read_at` and each agent's latest vote from its copies; an unchanged group is left alone, so marking a role unread again sticks. The per-board counts on Sources still count copies, since they report what a board holds.
@@ -70,6 +70,12 @@ What was decided for roledesk and the reason for each. The target design is the
**A read that stalls is incomplete, and a sweep's cursor may be a pair.** A reader that sees the host send a page again hands it over empty with a cursor, and the runner ends the read incomplete; a cursor of None there read as the last page, and on a host that states no total as a complete read that closed what the pages had not listed. With a stated total a page of ids already read is not a stall, since a resumed sweep starts on one when the list grew at the top. A cursor is kept across pulls when it survives JSON apart from a tuple coming back as a list, so Workday's and SuccessFactors' sweeps resume. **A read that stalls is incomplete, and a sweep's cursor may be a pair.** A reader that sees the host send a page again hands it over empty with a cursor, and the runner ends the read incomplete; a cursor of None there read as the last page, and on a host that states no total as a complete read that closed what the pages had not listed. With a stated total a page of ids already read is not a stall, since a resumed sweep starts on one when the list grew at the top. A cursor is kept across pulls when it survives JSON apart from a tuple coming back as a list, so Workday's and SuccessFactors' sweeps resume.
**A host that cannot filter on remote is asked for the agents' world where it takes the world in one read** (ADR Fetch prep and score 1.2). Asked for "DK, SE" alone such a host never listed a remote role filed under Germany, and 18 boards fetched no new remote role outside those two countries. Phenom and the Workday tenants without a remote facet take a list of countries, so Europe is one request and the gate sorts the rest out. SmartRecruiters, SuccessFactors and Eightfold take one country a request, where Europe is the whole board (Bosch 49 pages, a SuccessFactors site 10 roles a page), so there the location read stays and the row says remote was not asked.
**A role whose text says remote while no field does is possibly remote, read in prep with no model** (ADR 1.1 and 1.5). A role outside the agent's locations whose remote is unstated could never "turn out to be remote": extract reads only the cut, and the cut needs the gate. 5,016 open roles were rejected that way, 805 of them saying "remote" in their text. A phrase table ("fully remote", "work from home", the Danish and Swedish words) less its denials ("no remote", "remote sensing") marks `remote_text`; the gate's remote member takes the mark as a match while `remote` is empty, and once extract has answered, prep composes `remote` from the model and the mark no longer counts. A false mark costs at most a cut slot.
**Age is the floor under every board's closing** (ADR Fetch prep and score 1.3). An incomplete read (a ceiling, a page cap without a sweep) closed nothing ever, and a filtered read spared the roles acted on from its list and from age alike, so such roles stayed open for good. Now what no read has listed for 21 days closes under an incomplete read, and what a filtered read spares closes by age too; a role acted on is spared from the list close, which is guesswork, not from three weeks of absence. The alternative was to strike the sentence from the ADR, which would have left "may stay open for good" in the design.
**Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch in flight when the switch happens reopens them, so the settle closes every switched-off target once more. **Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch in flight when the switch happens reopens them, so the settle closes every switched-off target once more.
**A destructive switch asks first; a reopening one queues a pull.** Switching a board or search off, replacing a board and detaching one each close roles in a tap, so the page asks first when there are roles to lose, naming the count. Switching a target on, following or setting a board, setting All locations or All roles on and saving a search each queue `pull` at once instead of waiting up to six hours for the schedule. The pull is queued in the API route, not in the page's script, so a script calling the endpoint gets what a click gets; `claim_run`'s refusal is not an error, since the run in flight or the next one reads the change. **A destructive switch asks first; a reopening one queues a pull.** Switching a board or search off, replacing a board and detaching one each close roles in a tap, so the page asks first when there are roles to lose, naming the count. Switching a target on, following or setting a board, setting All locations or All roles on and saving a search each queue `pull` at once instead of waiting up to six hours for the schedule. The pull is queued in the API route, not in the page's script, so a script calling the endpoint gets what a click gets; `claim_run`'s refusal is not an error, since the run in flight or the next one reads the change.
@@ -182,6 +188,8 @@ What was decided for roledesk and the reason for each. The target design is the
**`extract_batch`, `judge_batch` and `summarise_batch` run isolated, with a timeout.** `context.check()` runs only between the calls `modeltask.one` makes, not inside one, so a call that hangs was unkillable short of restarting the server; a process boundary lets a Kill, or the timeout, end it. One call is bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for judge and summarise): a batch still running after that is stuck, not slow. The cost is that an isolated job's context has no working `stop`, `task` or `fan_out` (`context_for` in `pipeline/child.py`), so a refusal or an account-level failure inside a batch cannot spare the task's queued siblings. **`extract_batch`, `judge_batch` and `summarise_batch` run isolated, with a timeout.** `context.check()` runs only between the calls `modeltask.one` makes, not inside one, so a call that hangs was unkillable short of restarting the server; a process boundary lets a Kill, or the timeout, end it. One call is bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for judge and summarise): a batch still running after that is stuck, not slow. The cost is that an isolated job's context has no working `stop`, `task` or `fan_out` (`context_for` in `pipeline/child.py`), so a refusal or an account-level failure inside a batch cannot spare the task's queued siblings.
**The profile is the CV, the languages, the prompts, the likes and the dislikes; must-haves are gone** (ADR 4.2). The must-have text with a verdict per line and a career cap at 4 was the harder form of what likes and dislikes now say softly; it left the form on 2026-10-03 and its machinery followed on 2026-10-06 (migration 34). The judge reads the languages with their levels, since a role that asks for fluent Danish against conversational Danish is a skills gap the CV text alone did not show; a change to the languages re-judges the cut like a change to the CV.
**Judge reads a role against a written profile, after the `/rank` of github.com/MadsLorentzen/ai-job-search.** Prompt closeness compares topics; it cannot tell a step down, years of experience or a deal-breaker, which a model reading both texts can. The profile is the CV, its languages with a level, and two soft texts, "What I like" (`like_text`) and "What I don't like" (`avoid_text`), each covering the work, the workplace and situations with no split into tasks versus environment, since the prompts already say what the owner wants. A deal-breaker is a hard rule and belongs in the fetch gate, so the form has no must-haves box; `must_text` is set only through the API. Culture is left out because it is guessed from ad text, and location because the fetch gate does it. **Judge reads a role against a written profile, after the `/rank` of github.com/MadsLorentzen/ai-job-search.** Prompt closeness compares topics; it cannot tell a step down, years of experience or a deal-breaker, which a model reading both texts can. The profile is the CV, its languages with a level, and two soft texts, "What I like" (`like_text`) and "What I don't like" (`avoid_text`), each covering the work, the workplace and situations with no split into tasks versus environment, since the prompts already say what the owner wants. A deal-breaker is a hard rule and belongs in the fetch gate, so the form has no must-haves box; `must_text` is set only through the API. Culture is left out because it is guessed from ad text, and location because the fetch gate does it.
**The model reports facts and code applies the rules.** For each must-have line the model answers "met", "broken" only when the posting clearly contradicts it, or "unclear" when the text does not say, never inferred from silence; a must-have it leaves out counts as unclear. The cap is code's: a broken must-have makes career count as at most `BROKEN_CAP` (4) when a role is served (`app/models.py`), so the stored career stays the model's and the reader shows both. The CV match weights are applied when a role is served too, so changing them needs no model call. The verdicts store each line's text, so the reader still shows what a judgement was made against after the text is edited. The `input_key` carries `judge.FORMAT` (4) and the texts, so a change to the prompt's contract makes every judgement again. **The model reports facts and code applies the rules.** For each must-have line the model answers "met", "broken" only when the posting clearly contradicts it, or "unclear" when the text does not say, never inferred from silence; a must-have it leaves out counts as unclear. The cap is code's: a broken must-have makes career count as at most `BROKEN_CAP` (4) when a role is served (`app/models.py`), so the stored career stays the model's and the reader shows both. The CV match weights are applied when a role is served too, so changing them needs no model call. The verdicts store each line's text, so the reader still shows what a judgement was made against after the text is edited. The `input_key` carries `judge.FORMAT` (4) and the texts, so a change to the prompt's contract makes every judgement again.
@@ -236,7 +244,7 @@ What was decided for roledesk and the reason for each. The target design is the
**Search agents are cards, edited in place.** One person may run several agents: their own hunt and one that watches roles for someone else. An agent is defined by what it looks for, so its prompts are the card's body and the weights a small line under it. Edit turns that card into the form and New puts a blank one on top, because a form at the bottom of the page was out of sight of the card it changed. Job boards and Watchlist companies are edited the same way, one open at a time. **Search agents are cards, edited in place.** One person may run several agents: their own hunt and one that watches roles for someone else. An agent is defined by what it looks for, so its prompts are the card's body and the weights a small line under it. Edit turns that card into the form and New puts a blank one on top, because a form at the bottom of the page was out of sight of the card it changed. Job boards and Watchlist companies are edited the same way, one open at a time.
**A gate the form cannot say lands in a row under Advanced, so a load and a save lose nothing.** The form asks the fetch gate in its four parts by name rather than as the stored `require` and `any` sets with ISO codes, which the owner could not read; any other gate is a row under Other rules. Spellings of a town that differ by accents only share one chip, and both are stored. **The form's gates are the four parts of the fetch gate, and nothing else** (ADR Fetch prep and score 5.2 and Settings). Free rows of field, mode and values over twelve fields sat under Advanced, first to keep a rule an older form had saved; a row on seniority or pay was a permanent cut on fields half the roles leave empty, which the rail's filters do with a click and undo. The rows are gone; a stored gate the form cannot say is named under the fetch gate and kept as it is on save, so nothing is lost silently.
**Viewing as on Roles is a select, also with one agent.** The owner looked for the select to switch agent and found it gone after the second agent was switched off. It lists the enabled agents, since an agent that is off has no fresh scores. **Viewing as on Roles is a select, also with one agent.** The owner looked for the select to switch agent and found it gone after the second agent was switched off. It lists the enabled agents, since an agent that is off has no fresh scores.
-112
View File
@@ -1,112 +0,0 @@
# Deploying a service to rittershofer.org
The ground every service on this box shares: what the box is, the seven steps to add a service to it, the templates those steps need, and the things that have already gone wrong. [release.md](release.md) is how an image gets built and published; [server.md](server.md) is what happens once it has been.
`<app>` below is your *service's* name: the hostname, the data directory, the image tag, the compose service. It is the one thing you choose. The Python package stays `app` and the environment variables stay `APP_`, here and in every project started from this repository — nothing inside the checkout is renamed, and the files these steps need are already in place under `deploy/`.
## What the box is
A single Hetzner server running Ubuntu 24.04 on x86_64, with a 37 GB root filesystem and no separate data disk. One Docker Compose project holds everything.
/srv/swag/docker-compose.yml every service, one file, project name `swag`
/srv/swag/.env every secret, chmod 600, never in the compose file
/srv/data/<app>/ every service's data, owned 1000:1000
/etc/config/swag/ SWAG's /config: nginx confs, certificates, logs
A snapshot of that whole compose file, every neighbour included, is [box-docker-compose.yml](box-docker-compose.yml): what a new service block lands beside, and where the SWAG subdomain list lives.
`swag` is nginx, certbot and fail2ban in one container. It terminates TLS for every hostname and proxies to the others **by container name** over the project's default network. That shared network is the only reason all these services live in one compose file — nothing else about them is related.
Images are published to the Gitea instance on the same box, at `git.rittershofer.org/wolf/<app>`. Everything runs as uid/gid 1000 and `TZ=Europe/Brussels`.
Which compose file that is comes off the running container rather than out of memory:
```bash
docker inspect swag --format '{{index .Config.Labels "com.docker.compose.project.config_files"}}'
```
Declaring the network `external` from a second Compose project would work too, and would hardcode the first project's name into the second — a rename away from a confusing failure.
## The seven steps
**1. DNS.** `<app>.rittershofer.org` must resolve to the server *before* anything else. Confirm with `getent hosts <app>.rittershofer.org`.
**2. Certificate.** Add `<app>` to `SUBDOMAINS` in the swag service's environment. That list separates on commas and nothing else — a space after one asks for a hostname with a leading space. SWAG runs certbot on every container start and requests **one certificate covering every name in the list**, so a name that does not answer on port 80 fails the whole request and leaves the existing names unrenewed. This is why step 1 comes first. Restart swag, then check `docker logs swag` for the issuance. If SWAG already holds a wildcard certificate covering the name, reuse it and skip this step entirely.
**3. Registry login.** On the build machine a Gitea token with package **write**; on the server, package **read**. Same command both ends, token at the password prompt:
```bash
docker login git.rittershofer.org -u wolf
```
Git credentials and Docker registry credentials are separate. A successful Git push does not log Docker in.
**4. Data directory.** Root's `mkdir` produces a root-owned directory, and the container runs as 1000, so the chown is not optional:
```bash
sudo mkdir -p /srv/data/<app> && sudo chown -R 1000:1000 /srv/data/<app>
```
**5. Service block** from [../deploy/server/compose.yaml](../deploy/server/compose.yaml) into `/srv/swag/docker-compose.yml`, and the variables from [../deploy/server/env.example](../deploy/server/env.example) appended to `/srv/swag/.env`.
**6. Proxy conf** — [../deploy/server/swag/app.subdomain.conf](../deploy/server/swag/app.subdomain.conf) renamed to `<app>.subdomain.conf` in `/etc/config/swag/nginx/proxy-confs/`. SWAG's `/config` is a host directory bind-mounted into the container; read the host side off the running container rather than assuming where it is:
```bash
docker inspect swag --format '{{range .Mounts}}{{if eq .Destination "/config"}}{{.Source}}{{end}}{{end}}'
```
<config>/nginx/proxy-confs/ the per-site confs, where yours goes
<config>/nginx/proxy.conf the shared defaults every site includes
<config>/nginx/ssl.conf certificate paths and TLS settings
<config>/nginx/resolver.conf Docker's internal DNS
<config>/etc/letsencrypt/live/ the certificates themselves
Then validate, and reload only if validation passes:
```bash
docker exec swag nginx -t && docker exec swag nginx -s reload
```
Confirm the other sites still respond afterwards.
**7. Deploy.** Name the service, so nothing else in the file is recreated:
```bash
cd /srv/swag && docker compose up -d <app>
```
## What the templates already decide for you
Both templates are in [../deploy/server/](../deploy/server), commented in place. The decisions worth knowing before you read them:
**No `ports:`.** SWAG reaches the container by name on the project network. That is what keeps the app's port off the internet even if the firewall is wrong.
**No `networks:` either.** Being in that compose file is what puts it on the right one.
**`user: "1000:1000"`.** Container uid is host uid — there is no translation layer. Without this the process runs as whatever the image declares, usually root, and everything it writes into `/srv/data` lands root-owned.
**`mem_limit:`, not `deploy: resources: limits:`.** The latter is Swarm-only and silently ignored by `docker compose up`. 0 is unlimited, which is the honest default until the box has been watched under real load.
**`logging:` with a size cap.** Without it, json-file logs grow until the disk does not.
**`listen 443 ssl` without `http2`.** `http2 on;` is set once at `http` scope in `nginx.conf`, and the `listen ... http2` parameter is deprecated in nginx 1.25 and later.
**`proxy_read_timeout 86400s`** only if the app holds a WebSocket open. SWAG's `proxy.conf` already sets the Upgrade headers; what it also sets is a read timeout of a couple of minutes, which closes an idle socket on the minute. Delete those two lines if there is no socket.
## Things that have already bitten
**Disk.** 37 GB, and it has been full. Three causes, all recurring: `:latest` orphans the previous image on every pull and nothing cleans them up (a weekly `docker image prune -f` cron handles it — no `-a`, so pinned images survive); journald defaults to 10% of the filesystem and is capped here at 200 MB; and VS Code Remote installs a fresh server per version under `~/.vscode-server` forever.
**`cp -a "$SRC/."` with an unset `SRC`** copies the root filesystem. Use `"${SRC:?}"` and echo the value before running anything destructive with it.
**SWAG never touches your live confs**, only the `.sample` files beside them. That is deliberate and it means they drift for years. Diff them occasionally:
```bash
cd /etc/config/swag/nginx && for f in *.conf site-confs/*.conf proxy-confs/*.subdomain.conf; do echo "=== $f"; diff "$f" "$f.sample"; done
```
Take the samples wholesale and re-apply your own edits. `nginx.conf` and the site/proxy confs must be updated **together** — older ones include `ssl.conf` at `http` scope, newer ones per server block, and mixing the two produces `no "ssl_certificate" is defined`.
**SQLite is never safe to copy live.** Stop the container, or snapshot with `VACUUM INTO` through the running one. A `cp` of a WAL-mode database is a coin toss. See [server.md](server.md#the-database).
**Secrets in `.env`, never in the compose file.** It is easy to paste a compose file into a chat window; it is harder to do that by accident with a file you had to `cat` deliberately.
-241
View File
@@ -1,241 +0,0 @@
# Asking Claude, from a deployed app
One question — *ask Claude this, get JSON back* — and two ways to answer it, chosen by which credential is present in the environment. The caller asks without knowing which one ran.
api One call to the Claude API. Wants ANTHROPIC_API_KEY. No local
install, no subprocess, per-token billing. The unattended path.
claude_code The Claude Agent SDK drives the `claude` CLI, on the
subscription it is signed in with, authenticated by
CLAUDE_CODE_OAUTH_TOKEN. Nothing billed per token.
This is the reusable part of the deployment: the compose file passes both variables through, the app prefers the key when there is one, and switching from the subscription to the API is a line in `.env` and `docker compose up -d` — no rebuild, no code change.
## The two credentials
**`CLAUDE_CODE_OAUTH_TOKEN`** comes from `claude setup-token`, run interactively on a machine you are already signed in on. It lasts a year. Generating it needs browser authorization, so no deployment script can automate it — you produce it on your laptop and paste it into the server's `.env` with an editor. Then `docker compose up -d app`, because a `restart` will not pick it up.
**`ANTHROPIC_API_KEY`** is per-token billing from the Console. Nothing to renew, nothing to sign in to, and it is what production should run on once the thing is real.
Set one. If both are set, `auto` takes the key. Neither, and every call fails with a notice naming the reason, which is the correct outcome — better a batch that fails saying "no credentials" than one that waits for a human who is not there.
The API key and the OAuth token are not interchangeable and do not substitute for each other. `ANTHROPIC_API_KEY` in the environment does not make the CLI work, and an OAuth token does not authenticate an API call.
**Neither belongs in the image or in the compose file.** They live in the `.env` beside it, `chmod 600`, injected as environment variables at container creation. A compose file gets pasted into chat windows; a file you had to `cat` deliberately does not.
## Why there is a `backend` setting and not just detection
Detection answers "what can run here". The setting answers "what should". They differ in two cases worth naming: forcing `api` on a machine that also has the CLI, so a benchmark measures the thing you meant; and `off`, which makes everything needing a model fail on purpose, with a notice that says which switch did it. An app that silently degrades to no-model is worse than one that stops.
## What the `claude_code` path deliberately does not get
No tools (`tools=[]`, `allowed_tools=[]`), no filesystem settings (`setting_sources=None`, so no CLAUDE.md, no skills, no MCP servers leak into a classification call), no session to resume. One text-in, JSON-out turn — the same shape as the API call it stands in for. The CLI is built for agent work, and left to its defaults it will bring all of that with it.
## Thinking has to be stated, not defaulted
The CLI thinks unprompted; the Messages API thinks only when asked. So "say nothing about thinking" means opposite things on the two sides, and the two backends stop being interchangeable. State it explicitly to both. It is not a small difference: on a batch of twenty-five short classifications, the CLI's default cost 128 s against 17 s for the same answers.
Some models support manual extended thinking only and reject `adaptive`. Check before spending a call, not after.
## Both take the same JSON schema
The Agent SDK accepts the Messages API's `json_schema` output format and hands the parsed object back on `ResultMessage.structured_output`. That is what makes this two backends rather than two implementations: neither path is the degraded one, and moving between them is a config line.
## Reference implementation
Self-contained — no imports from your project, both SDKs optional. It is the shape, not the whole thing: the real one in newsdesk also streams partial answers out for a progress bar, retries, and classifies API errors by status code so a 401 and a 429 tell you different things.
```python
"""Ask Claude a question, constrained to a JSON schema, get a dict back.
Two backends, one seam. `ANTHROPIC_API_KEY` selects the API; otherwise the
Claude Agent SDK drives the `claude` CLI on a subscription, authenticated by
`CLAUDE_CODE_OAUTH_TOKEN`. Both are optional dependencies:
uv pip install anthropic # the api backend
uv pip install claude-agent-sdk # the claude_code backend
"""
from __future__ import annotations
import asyncio
import importlib.util
import json
import os
from dataclasses import dataclass
from typing import Any
class AgentError(Exception):
"""The backend was there and the call still did not produce an answer."""
@dataclass(slots=True)
class AgentConfig:
#: "auto", "api", "claude_code", or "off" to make every call fail.
backend: str = "auto"
model: str = "claude-haiku-4-5-20251001"
timeout: float = 300.0
max_tokens: int = 8192
#: "off" or "adaptive". Stated to both backends, never defaulted.
thinking: str = "off"
@dataclass(slots=True)
class Reply:
data: dict[str, Any]
backend: str
model: str
usage: dict[str, Any]
def backend(settings: AgentConfig) -> str:
"""Which backend will answer, or "" for nobody."""
if settings.backend == "off":
return ""
if settings.backend in ("api", "claude_code"):
return settings.backend
if any(os.environ.get(name) for name in
("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_PROFILE")):
return "api"
# The SDK bundles its own `claude`, so having the package is enough. Not
# proof the call will work — the CLI may be signed out — but that surfaces
# as an AgentError on the call, which the caller handles anyway.
if importlib.util.find_spec("claude_agent_sdk") is not None:
return "claude_code"
return ""
def _thinking(setting: str) -> dict[str, str]:
return {"type": "adaptive"} if setting == "adaptive" else {"type": "disabled"}
def ask_json(settings: AgentConfig, *, system: str, question: dict[str, Any],
schema: dict[str, Any]) -> Reply:
"""Ask `question`, constrained to `schema`, and return the parsed answer.
`system` should be byte-identical across the calls in a run — it is what
the API path caches. `question` is the part that changes.
"""
name = backend(settings)
if not name:
raise AgentError("no model backend is available: set ANTHROPIC_API_KEY "
"or install claude-agent-sdk with CLAUDE_CODE_OAUTH_TOKEN")
if name == "api":
return _ask_api(settings, system, question, schema)
return _ask_claude_code(settings, system, question, schema)
def _ask_api(settings: AgentConfig, system: str, question: dict[str, Any],
schema: dict[str, Any]) -> Reply:
try:
import anthropic
except ImportError as exc:
raise AgentError("the anthropic SDK isn't installed") from exc
try:
client = anthropic.Anthropic(timeout=settings.timeout)
response = client.messages.create(
model=settings.model,
max_tokens=settings.max_tokens,
# The instructions are byte-identical across every call; only the
# payload below changes. Caching them is most of the saving.
system=[{"type": "text", "text": system,
"cache_control": {"type": "ephemeral"}}],
messages=[{"role": "user", "content": json.dumps(question)}],
thinking=_thinking(settings.thinking),
output_config={"format": {"type": "json_schema", "schema": schema}},
)
except anthropic.APIError as exc:
# Carry the status through in the text: 401 ("your key is wrong") and
# 429 ("slow down") want different reactions, and the message body
# alone does not always separate them.
status = getattr(exc, "status_code", "")
raise AgentError(f"the API call failed: {status} {exc}".strip()) from exc
text = next((b.text for b in response.content if b.type == "text"), "")
return Reply(data=json.loads(text), backend="api",
model=response.model or settings.model,
usage=response.usage.model_dump() if response.usage else {})
def _ask_claude_code(settings: AgentConfig, system: str, question: dict[str, Any],
schema: dict[str, Any]) -> Reply:
try:
from claude_agent_sdk import (
ClaudeAgentOptions, ClaudeSDKError, ResultMessage, query,
)
except ImportError as exc:
raise AgentError("the Claude Agent SDK isn't installed") from exc
options = ClaudeAgentOptions(
system_prompt=system,
model=settings.model or None,
# A classification call, not an agent: nothing to run, nothing to read,
# and no project settings — so no CLAUDE.md, no skills and no MCP
# servers get in front of the model on the way to an answer.
tools=[],
allowed_tools=[],
permission_mode="dontAsk",
setting_sources=None,
thinking=_thinking(settings.thinking),
output_format={"type": "json_schema", "schema": schema},
# The API takes max_tokens as a parameter; the CLI takes it only as an
# environment variable. Without this the setting is quietly API-only.
# The SDK merges this into the inherited environment rather than
# replacing it, so naming one variable does not cost the CLI its PATH.
env={"CLAUDE_CODE_MAX_OUTPUT_TOKENS": str(settings.max_tokens)},
)
async def ask() -> Reply:
answer: Reply | None = None
async for message in query(prompt=json.dumps(question), options=options):
if not isinstance(message, ResultMessage):
continue
if message.is_error:
raise AgentError("; ".join(message.errors or []) or message.result
or f"the agent ended as {message.subtype}")
data = message.structured_output
if not isinstance(data, dict):
# Ended without structured output — a limit, an interruption.
# `result` is the raw text if there is any, worth one attempt.
data = json.loads(message.result or "")
answer = Reply(data=data, backend="claude_code",
model=settings.model, usage=message.usage or {})
if answer is None:
raise AgentError("the agent returned no result")
return answer
try:
# asyncio.run, so this is a synchronous call like the other backend.
# From inside a running loop, await `ask()` directly instead.
return asyncio.run(asyncio.wait_for(ask(), settings.timeout))
except TimeoutError as exc:
raise AgentError(f"no answer within {settings.timeout:.0f}s") from exc
except ClaudeSDKError as exc:
# The SDK has its own exception hierarchy — a CLI that exited, a line
# it could not parse, an answer that overran max_tokens — and none of
# it is an AgentError. Left alone, those escape a caller that catches
# AgentError per item and take down a whole run. Every failure of this
# backend is an AgentError or none of them are.
raise AgentError(str(exc)) from exc
```
Using it:
```python
settings = AgentConfig()
schema = {
"type": "object",
"properties": {"tags": {"type": "array", "items": {"type": "string"}}},
"required": ["tags"],
}
reply = ask_json(settings, system="Tag this article. Answer with the schema.",
question={"title": title, "body": body}, schema=schema)
print(reply.data["tags"], reply.backend)
```
## The two failures that will actually happen
**A changed token in `.env` appears to do nothing.** `docker compose restart` does not update environment variables. `docker compose up -d app` recreates the container, which does.
**The token expired and nothing said so.** The setup token lasts a year, and the failure looks like a model call failing rather than a credential problem. Read the error text — this is why the SDK's own message is passed through intact rather than summarised into "model call failed".
+6 -36
View File
@@ -1,48 +1,18 @@
# Open # Open
What is known and not yet fixed, checked against the code on 2026-10-05, the day of the architecture review. What was decided is in `decisions.md`, the target design in `documentation/`, the unbuilt readers in `plan-hosts-wave-2.md`. Counts are of the live database on that day. What is known and not yet fixed, checked against the code on 2026-10-06, after the architecture review of 2026-10-05. What was decided is in `decisions.md`, the target design in `documentation/`, the unbuilt readers in `plan-hosts-wave-2.md`. Counts are of the live database on that day.
## For the owner to decide ## For the owner to decide
Points where the code and `ADR Fetch prep and score` disagree, or where the ADR is silent and roles are lost. - **Remote on hosts that take one country a request.** SmartRecruiters, SuccessFactors and Eightfold (14 boards, about 15,000 roles) are still asked for DK, SE alone when the agents want remote, since Europe there is the whole board. Reading them whole is the only way to their remote roles elsewhere.
- **Remote listings across countries.** A job posted remote once per country is now one role per country (the fold keeps to one country), which lists a pan-European remote posting several times.
- **Remote roles on hosts that cannot filter on remote.** On 18 boards (SmartRecruiters, Phenom, SuccessFactors, Eightfold, and Workday tenants without a remote facet) the read for "DK, SE, remote" goes out as "DK, SE". The row says "remote not asked" and the remote roles already held are kept, but a new remote role elsewhere in the world gate is never fetched. ADR 1.2 ("a host that cannot filter on a part sends the board whole for that part") reads as reading such boards whole; Bosch is then about 50 requests a pull.
- **A role outside the agent's locations with remote unstated is never read by extract.** ADR 4.1 says a role can "turn out to be remote", but extract reads only the cut and the cut requires the gate. 5,016 open roles are rejected for location with remote unknown, and 805 of them say "remote" in their text. A word scan in prep costs no tokens and would let those through to extract.
- **Copies across countries.** ADR 1.4 requires the same country and overlapping towns. `dupes.fold` also merges one host's listings of one title across towns and countries when 30% of their words are shared: 2,125 open copies, about 1,550 in another country than the kept role, some of them different openings (Datadog's US federal role behind the Canberra one). The gate reads the kept role through its copies' places, so no office listing is hidden. Remote and office listings are kept apart since 2026-10-05; 176 remote listings still sit behind a remote kept role in another country, whose area alone the world gate reads.
- **Judge and languages.** ADR 4.2 names the agent's languages as an input. The judge does not read them (they only set the flag on a row), and editing them re-judges nothing. The ADR's list also omits the prompts and the must-haves, which the judge reads and which re-judge.
- **Must-haves.** A broken must-have caps career at 4, but `must_text` is set only through the API: it is not on the form and not in the ADR's Settings.
- **Show gated and Advanced rules.** ADR 5 says no order shows a role the gate rejects and that seniority and pay are filters. Show gated lists the rejected roles with the reason, on purpose; an Advanced row can gate on any of twelve fields.
- **Closing by age on boards.** ADR 1.3 closes a role no read has listed for 21 days. That holds for searches and for boards narrowed by terms. A board read that stays incomplete (a ceiling, the page cap without a sweep) closes nothing ever, and on a filtered board a role with any vote or an application never closes. Since closing is not valued, striking the rule from the ADR is the cheaper way to agree.
- **A search cut at its page cap leaves no record.** In the last pull 4 of 26 searches stopped at a cap (Jobbank and freehire at 1,000 rows, two Jobindex searches at 200). Nothing on a page says so, and Jobbank's feed is not sorted by date, so new ads can lie past the cap. - **A search cut at its page cap leaves no record.** In the last pull 4 of 26 searches stopped at a cap (Jobbank and freehire at 1,000 rows, two Jobindex searches at 200). Nothing on a page says so, and Jobbank's feed is not sorted by date, so new ads can lie past the cap.
- **A row without a title, link or id is dropped by its reader, unrecorded,** while its id still counts toward completeness. `ADR Plugins` says both "never drops rows" and "title and url are required"; the runner could validate, log and mark the read incomplete.
- **The world gate is not sent to a host when an agent sets no location.** Every big board is then read whole. Nothing is lost.
## What the ADRs do not say ## What the ADRs still do not say
What someone rebuilding from `ADR Fetch prep and score` would have to guess, by section. `ADR Fetch prep and score` was brought up to the code on 2026-10-06; left for the owner's read as a whole.
- **Prep has no section**, though the title names it: a host's words mapped through a table per host, a place text resolved against a gazetteer of towns, regions, countries and continents with regions across borders, remote and the remote area read from the place words, the seniority ladder, and that a text nothing maps is listed and passes the gate. `ADR Plugins` was rewritten on 2026-10-06 and names a target the code has not reached: a broken plugin left out instead of stopping the app; providers declaring their params with label, kind and choices, so form, CLI and API follow from the module; detail state kept for a search's roles as for a board's; a national provider declaring its implied country; page markers for hosts on a company's own domain; a contract test that names no reader.
- **1.1 Fetch gate.** Every role is stored whatever the gates say, and each agent applies its gate at scoring. An unknown location is no resolved place and no country, which includes an unresolved text and a wider region alone. A role in several places needs one inside. The world gate reads a remote role by its remote area. Exclusion matches whole words of the title.
- **1.2 Reading a board.** The union is per agent the location gate, or the world gate when no location is set, then over the agents. What happens when a host filters countries but not remote. A board's own saved params win over the agents'. With no enabled agent every board is read whole. A search reads 5 pages by default and 10 at most.
- **1.3 Closing.** "Filtered read" is not defined. Switching a board or search off closes its roles; a closed role listed again reopens.
- **1.4 Copies.** The second rule, one host's listings of one title merged when their texts are alike. Groups never chain. Votes and read state move to the kept copy. How titles and company names are normalised.
- **2.1 Closeness.** The embedding model (`intfloat/multilingual-e5-small` with its query and passage prefixes) and what is embedded (title and the first 2,000 characters of the text, else the teaser). The CV is cut into chunks of at most 1,200 characters. A level is a linear clip between two cosines measured per model (prompt 0.75 to 0.88, CV 0.77 to 0.90, job like -0.14 to 0.19). With several prompts each is scaled and the best counts. The bonus divides by all four weights also when one has left the sum.
- **2.2 Filter.** The bullets carry the "and" and the "or" by layout only. A liked or saved role must still pass the gate. A like is this agent's; a save under any agent counts.
- **3 Fetch text.** Only hosts with a role page are read, Jobindex never in a pull, and the caps (200 a board, 1,000 a run, retry after 7 days) are in `ADR Plugins` only.
- **4 Model analytics.** The model of each call and the batch sizes (20, 5, 10). Extract reads the first 2,500 and last 1,500 characters and also fills pay, currency, function and work permit; it gives up after 3 unanswered tries. The judge reads up to 8,000 characters, rates career against the prompts, likes and dislikes, and the CV match is 10 times the weighted mean, 0 to 100. Summarise writes about 60 words and 3 to 8 tags, once. A stage that is off, or has no model to ask, is skipped without a notice; only Settings › Sources › Runs says so. A cut role without text is neither judged nor summarised.
- **5 Feed.** Open roles and kept copies only. Opening or voting marks a role read. The list of filters, 60 rows a page, that ★ also likes, and what a role without an embedding scores. Newest is a third order, where the glossary and the flow say two.
- **Settings.** The agent's On/Off switch and the pull's schedule (every 6 h).
Wording to settle: 2.1 "over the sum of the want feed score weights" against 5.1.3 "its weight leaves the sum"; 4 "once extract has completed the ingestion" against 4.1 "a role that enters is extracted on the next pull"; 4.1's field list against "raw and processed fields for e.g. salary"; "model analytics score" is never defined (the number is the CV match); 1.3 against `ADR Plugins` 3.2, which lets a search declare `stated`, and 2.1, which says an incomplete read closes nothing; the title is "Architecture" while the other ADR calls it by its file name.
`ADR Plugins`, read lightly since it is not final:
- **The API block already differs from the code:** `empty` is "trusted" or "suspect", not None; `Page` is a plain tuple; `Reads` has an unused `title="provisional"`; a provider must define `allowed`; a reader may import `desk`.
- **Rules the code needed and the ADR lacks:** a cursor must survive JSON, where a tuple returns as a list; a reader that sees a page come again hands it over empty with a cursor; a wrong-shaped answer is an error, never an empty list.
- **Rules the code does not keep:** "at most 8 series" is enforced only with terms or more than 8 countries; the 10% guard starts at 1 role, not 10, for filtered, narrowed and suspect reads; a 429 after page one starts a sweep; a module that breaks the contract stops the import at the first problem, and with it the app.
- **"Never maps a word"** is broken by `Place.remote`, which makes every reader decide whether "hybrid" is office, and by the national providers that state their country (Jobindex DK, Platsbanken SE).
- **"Nothing else changes"** holds for a host. A provider with a new param or a key touches the Sources form, its script, the CLI, the API body and `sources.search_params`.
- Cases 4 and 5 are handled after the plugin, so "upstream" reads the wrong way round. The constants (0.5 s, 100 pages, 96 MiB, 21 and 7 days) will go stale faster than the rules.
## Structure ## Structure
-106
View File
@@ -1,106 +0,0 @@
# One password, on the open internet
The shape used by a single-user app that has to sit on a public hostname because that is where the browser is. No user management, no usernames, no registration: one password and the sessions it opens. Everything below is about making that enough.
This is the reasoning; `auth/` in this repository is the code, already written and already wired into `app/main.py`. `auth/README.md` is the ten-minute version of how to take the folder somewhere else.
## Locked by what it is bound to, not by a flag
The switch is the bind address. On `127.0.0.1` the door is already the machine's own login, and asking for a password again is theatre. On `0.0.0.0` it is reachable from elsewhere and locks itself.
Deriving it rather than configuring it removes the failure that looks fine: "deployed it, forgot to switch auth on". The production config then says `enabled: always` anyway — redundant with the bind address and worth saying out loud.
```yaml
auth:
enabled: always # auto | always | never
session_days: 30
# `auto` marks the cookie Secure when the request arrived over HTTPS. Behind
# a TLS-terminating proxy the app only ever sees plain HTTP and cannot work
# that out, so behind SWAG this must be `always`.
secure_cookie: always
```
## The password itself
Stored as an scrypt hash, never in the config file. The first locked start either takes `APP_PASSWORD` from the environment or generates one and prints it to the log once. After that it is the app's own Account screen that changes it.
`APP_PASSWORD` is the way back in when a login is lost, and it is authoritative while it is set — the Account screen refuses to change what the environment pins. Set it, get in, empty it, recreate the container, then change the password in the app.
Two consequences worth writing down: a generated password is printed to the container log, so the retained log holds it until it rotates — rotate the password once you are in. And a password the *user* chooses is the weak spot, not the lockout: a minimum length of 8 is a very different thing from the 16 random characters the app generates.
## The lockout
Five wrong answers are free — past a typo and a stale password manager entry, short of anything systematic. After that each attempt waits, starting at 30 s and doubling to a ceiling of 15 minutes. That is about a hundred guesses a day against a secret with far more than a hundred possibilities, which is what "guessing is pointless" means in practice.
Three details that matter more than the numbers:
**Read the lockout before checking the password, not after.** Otherwise a fast attacker extends their own lockout forever and never notices, and — more to the point — an honest person hammering the button digs a deeper hole with every press.
**A wrong answer still costs a full scrypt derivation.** The delay an attacker meets is the one the honest user meets, and it is what lets the free-attempt count be as generous as it is.
**Cap the doubling.** `30s * 2**failures` with an uncapped counter eventually overflows whatever holds it, and a bot that has sent ten thousand guesses turns every future login attempt into a 500 — a denial of service delivered by arithmetic.
## One counter, not one per address
The obvious improvement is to lock out the address that is guessing, so a stranger cannot keep the owner waiting at their own door. Behind a proxy it cannot be built honestly: the only address the app sees is the one the proxy puts in a header, the peer is a container on the same Docker network, and a header is set by whoever is talking. Keyed on that, an attacker gets a fresh, empty counter per request by editing one line of their own request, while the owner still gets locked out by their own typos. Strictly worse, and it would look like protection.
So the address is **logged, not decided from** — at the layer that knows who connected, the decision can be made properly. See the throttling section below.
The honest cost of one global counter: an attacker hammering the door keeps password login locked for the owner too. A passkey login that skips the lockout is what makes that survivable.
## Sessions
One kind of session whichever door opened it, so revoking and expiry have one implementation. A row per session with its user agent and last-seen time, a screen listing them, a button to revoke one and a button to close all the others. `purge()` on startup drops the expired ones — otherwise that table grows forever.
The cookie: `HttpOnly` so an XSS cannot read it, `Secure` per the config above, `SameSite=Lax`, `path=/`.
**No CSRF token, and that is a decision rather than an omission.** A CSRF token defends a cookie-authenticated app against another origin submitting a form on your behalf. `SameSite=Lax` already refuses to send the cookie on any cross-site request that is not a top-level navigation, and if no `GET` in the app changes anything, the attack the token would stop cannot be mounted. Adding one anyway means a second secret, a second thing to rotate, and a second way for a request to fail that the UI has to explain. Where that reasoning stops holding is a browser old enough to ignore `SameSite` entirely.
## Passkeys
Here already, behind the `passkeys` extra: `uv sync --all-extras`, and without it the routes answer 501 and the Account tab simply does not draw the section.
A passkey and a security key are one feature: a YubiKey registers through the same flow as a phone, because to WebAuthn they are the same thing.
A passkey is bound to an origin, and that binding is what makes it unphishable — the browser only hands the credential back to the origin that created it. So it needs HTTPS and a real hostname, with `localhost` the one exception, which is what lets it be tried before deploying. Behind a proxy, state both explicitly:
```yaml
auth:
rp_id: app.rittershofer.org
rp_name: app
origin: https://app.rittershofer.org
```
Left blank they are derived from the request, which behind a TLS-terminating proxy derives `http://` and rejects every ceremony as an origin mismatch. Changing `rp_id` later invalidates every key already registered against the old one.
**The password does not go away**, deliberately: it is the way back in if a key is lost, which is why revoking a passkey can never lock you out.
## Throttling a stranger who is guessing
The lockout already makes the secret safe. What is left is keeping the door usable for you, and keeping the traffic off the app. Two layers, in the order worth adding them:
**1. Rate limit in nginx.** Five lines in the proxy conf, included in [../deploy/server/swag/app.subdomain.conf](../deploy/server/swag/app.subdomain.conf). Per-address, no state to manage, no capabilities, and it cannot lock you out — it just makes requests wait. This is the part that earns its place.
**2. fail2ban, only if a real attack turns up.** SWAG ships it. It reads nginx's access log, where the address is the one that actually connected rather than a header, and bans in the firewall across every service on the box. A filter matching the login routes with a 401/403/429 status, and a jail pointing at `/config/log/nginx/access.log`, is about ten lines.
What fail2ban adds over the throttle is narrow and worth being honest about: your own password login stays usable during an attack, the ban covers the whole box rather than one site, and you get a list of who has been banned. It adds no protection of the password — that is already done. Against that it costs `NET_ADMIN` on the SWAG container, a regex that breaks silently if the log format changes, and an unban procedure for when it catches you.
Add it with a real log sample to test the regex against:
```bash
docker exec swag fail2ban-regex /config/log/nginx/access.log /config/fail2ban/filter.d/app.conf
```
If the domain is proxied through Cloudflare, nginx logs Cloudflare's addresses and a ban hits Cloudflare rather than the attacker. Check that before building anything on top of the log.
## What to log
One WARNING line per refusal, with the address first after `from`, so a filter is one regex:
```
WARNING auth.router: wrong password from 203.0.113.9
```
The log is the record. A second table of attempts in the database would only ever disagree with it. If the app is run at `log_level="warning"` there is no access log of its own, which makes this the whole record of who is knocking.
The forwarded address is the one piece of a stranger's own writing that ends up in that log, so strip it to the characters an address is made of before writing it. A line in this log is something a filter reads and a person believes, and neither should be forgeable by whoever is knocking.
+35
View File
@@ -0,0 +1,35 @@
# Plan: take the boilerplate and news of October 2026 into roledesk
Planned 2026-10-27. Folded into `decisions.md` and deleted when the packages land. The copies were last refreshed on 2026-09-29 (`pipeline/` at boilerplate a1c6f0d, `deploy/` at 1679e88, `auth/` older still, `desk/` at news a275c38). Since then the boilerplate has 88 commits and news 60 on `desk/`. None of our copies carries a local edit, so each refresh is `git archive` and then the app's adaptation.
## What changed upstream, and what it means here
- **pipeline/** (boilerplate 09e144d): the queue is backend only, so `POST /api/jobs` is gone and the page starts a kind through an app route; a task's `settle` and `callback` and what `context.task` names must be registered with `@callback`, since no row may name Python the app did not register; a task's step to the next is durable; a notice raised in an isolated child reaches the queue and its stderr the parent; an isolated child no longer outlives its worker; a claim landing while the queue stops hands its lease back; `/ws/jobs` frames are declared models; `FINAL`, `Tally.over`, `counters`. The table upgrades itself with `ALTER TABLE`, so the live database needs no migration of ours. Our Runs page still loads `<desk-jobs>` and `<desk-schedules>`, which the module still ships.
- **auth/** (same): the lockout is claimed before the scrypt, `APP_PASSWORD` recovery closes every session, Starlette's cookie parser, the rightmost forwarded hop in the log, `revoke_all` on a password change, a 421 for a non-loopback `Host` on an open app, API tokens as a third credential (`auth_tokens`, `Authorization: Bearer`), sockets closed with their credential. The app side adds the three hardening headers on every answer (`Hardening` in the boilerplate's `app/main.py`) and the SWAG rate limit on the passkey options route.
- **llm/** (new): the model seam, ported from news's agent tool; news's `desk/tools/agent.py` is now an adapter over it and `desk/modeltask.py` imports `llm`. Backends and names stay (`ask_json`, `backend`, `AgentError`, `Reply`); the per-model thinking rule knows Haiku and Sonnet 5.5.
- **desk/** (news 7727bb3): the generic modules moved on (`modeltask`: a reply the CLI cut off keeps whole items and counts what it lost, failures categorised; `notices`: every code the tree raises is in `BY_CODE`; `tools/agent` over `llm`; `tools/http` owns the one user agent and timeout; `config.AgentConfig.for_llm`). The source contract did not stay generic: `Item` is gone, a plugin yields an `Entry` or an `Article` with news's typed fields, media and recordings. Our two sources yield `Item` and our store reads it.
- **deploy/** (same): the release is split where the sandbox ends (the gate in bash in the distro, the image built and pushed on Windows with `wslc`), `wsl-setup`, `db.sh` reads the service from `release.env`, the image runs as `USER 1000`, pins `python:3.15-slim`, installs its extras, and the server gets a `config.example.yaml`; `backup.sh` asks the app inside the container. The boilerplate's `docs/` moved to `documentation/` (`Install Server.md`, `Install WSL.md`, the two proposals), so our five copied docs have no upstream any more.
- **Toolchain:** Python 3.15, fastapi 0.147, pydantic 2.14, starlette 1.7, uvicorn 0.54, anthropic 1.15, claude-agent-sdk 0.2.165; news runs 3.15 with torch from the CPU index.
- **The page:** the boilerplate and news both serve a React page now; the Jinja templates our Roles page was ported from are gone. Out of scope here; it is the owner's call whether roledesk follows.
## Decisions taken for this plan
- `desk/` stops being a whole copy of news. The generic modules (`modeltask`, `notices`, `config`, `tools/agent`, `tools/credentials`, `tools/embedding`, `tools/http`, `tools/failures`, `tools/checks`, `tools/scrape`, `tools/article`, `tools/browser`, `tools/framing`, `tools/marks`) keep news as their upstream and are refreshed file by file; `desk/sources/*` and `desk/models.py` are roledesk's own from now on, since the Plugins ADR defines our contract and news's `Entry` is a news thing. `desk/README.md` says so.
- `llm/` comes in whole from the boilerplate, under the coupling rule like `pipeline/` and `auth/`. The `agent` extra becomes `llm`.
- The model names stay as they are (Haiku 4.5); moving to 5.5 is a separate, measured change.
- The five boilerplate docs in `docs/` (`hosting`, `release`, `server`, `password-protection`, `llm-backends`) go; `docs/Install Server.md` and `docs/Install WSL.md` are copies of the boilerplate's `documentation/` files, and the two proposals are linked from the boilerplate checkout as news does.
- The toolchain step (3.15, the lock) is the planner's, last, on main, since it changes the environment every worktree tests with.
## Work packages
Each works in its own worktree made from main, owns the files named and nothing else, runs the full suite before committing, and reports. The planner merges in the order below.
| WP | Model | Owns | Builds |
|---|---|---|---|
| 1 pipeline | Opus | `pipeline/**`, `app/jobs/cascade.py`, `app/jobs/pull.py`, `app/jobs/fetch_text.py`, `app/web/api.py`, `app/static/jobs.js`, `app/static/app.js`, `app/static/live.js`, `app/templates/settings/sources.html`, `tests/test_web.py`, `tests/test_stages.py`, `tests/test_pull.py`, `tests/test_fetch_text.py`, `tests/test_cli.py` | `pipeline/` from boilerplate HEAD whole; `@callback` on `cascade.step`, `pull.close_boards`, `fetch_text.settle` and whatever else a row names; `POST /api/pull` in `api.py` and the Runs page and `app.js` using it; the `/ws/jobs` frames the page reads checked against `pipeline.web.schemas`; the jobs table of a copy of the live schema upgraded by the new `Jobs` in a test; `pipeline/tests` green here |
| 2 auth | Opus | `auth/**`, `app/main.py`, `app/templates/settings/account.html`, `app/templates/base.html`, `tests/test_app_hardening.py` (new), `tests/test_pages.py` | `auth/` from boilerplate HEAD whole; `connect` call checked against the new signature; `Hardening` headers in `app/main.py`; the Account page showing the token section; `auth/tests` green here |
| 3 llm and desk | Opus, after 1 and 2 merged | `llm/**`, `desk/**`, `app/jobs/stages.py`, `app/jobs/judge.py`, `app/jobs/extract.py`, `app/jobs/summarise.py`, `app/jobs/posting.py`, `app/jobs/embed.py`, `app/config.py`, `app/web/api.py` (the model routes only), `app/jobs/cascade.py` (the `agent.backend` lines only), `pyproject.toml` (packages, extras, testpaths, mypy modules), `config.example.yaml` (`agent:`), `tests/test_architecture.py`, `tests/test_coupling.py`, `tests/test_packaging.py`, `tests/test_stages.py`, `tests/test_judge.py`, `tests/test_extract.py`, `tests/test_summarise.py` | `llm/` from boilerplate HEAD; the generic desk modules from news HEAD; `desk/models.py` and `desk/sources/*` kept, with what `modeltask` and `notices` need from news's `models.py` (`ModelTaskResult`, `JsonDict`) checked; the app's model calls on the new adapter; the `llm` extra; `llm` in the layering tests; `desk/README.md` rewritten; `llm/tests` green here |
| 4 deploy | Sonnet | `deploy/**`, `bin/**`, `docs/hosting.md`, `docs/release.md`, `docs/server.md`, `docs/password-protection.md`, `docs/llm-backends.md`, `docs/Install Server.md`, `docs/Install WSL.md`, `docs/box-docker-compose.yml`, `README.md` (the Docs list and the deploy sentences), `tests/test_deploy.py` (new), `.gitattributes` | `deploy/` from boilerplate HEAD whole, then roledesk's own facts put back: the Dockerfile without the frontend stage and copying `desk`, `plugins`, `llm`, `seed`, with the extras `passkeys`, `embeddings`, `llm`; `release.env.example` with our image name; `server/config.example.yaml` with roledesk's keys; `bin/` without `frontend.sh`; the docs swap; `.cmd` files CRLF |
| 5 planner | Fable | `CLAUDE.md`, `docs/decisions.md`, `docs/open.md`, `.python-version`, `pyproject.toml` (`requires-python`, `[tool.uv.sources]`), `uv.lock` | the merges; Python 3.15 and `uv lock --upgrade` with torch from the CPU index as news has it; `CLAUDE.md`: `llm/` under the copies rule, `desk/` as above, the UI pointer at news's React page; the fold into `decisions.md`; this file deleted |
WP1, WP2 and WP4 run in parallel. WP3 starts when WP1 and WP2 are on main. The live server is restarted by the owner after the last merge.
Loaded 100 of 187 files, more files were not shown because too many files have changed in this diff. Show more