Docs: decisions on the boilerplate, tightened

One entry replaces "own repo, depending on news" and "desk and pipeline
are never edited here": built on the boilerplate with desk copied from
news, all four downstream copies, fixes upstream first, package app. The
superseded 8 MB cap entry is gone and the 96 MB one is unwrapped. Long
entries are cut to the decision and one reason, and the stale names
(roles.yaml, the one alpha page, app.js) follow the restructure. New: pull
as a job on a 6h schedule, and the coupling rule's place in the layers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-23 12:52:04 +02:00
1 parent 8b246c3035
commit 3c1b5c4309
1 file changed
+18 -25
+18 -25
View File
@@ -4,17 +4,15 @@ What was decided for roledesk and the one reason for each. The longer reasoning
## Shape
**Own repo, depending on news for `desk` and `pipeline`.** `pyproject.toml` takes `newsdesk` as an editable path dependency on `../news`. `desk` imports no app and news's architecture test enforces it, so it works as a library. Copying it would fork the plugin contract; `pipeline` alone may be copied later if depending on a repo named news becomes the problem.
**`desk` and `pipeline` are never edited here.** A change roles needs lands in news, and roledesk picks it up through the editable install.
**Built on the boilerplate, with `desk` copied from news.** `pipeline/` (queue and scheduler, pipeline 2.0), `auth/` and `deploy/` come from `../boilerplate` and `desk/` from `../news`, all four as downstream copies refreshed with `git archive`. A fix lands upstream first, or the next refresh reverts it. The package is `app`, per the boilerplate rule, since the deploy kit and the `APP_*` variables assume it.
**The domain noun is `role`.** `jobs/` keeps meaning the work the app queues, so "job" never has two senses in one sentence.
**Own database, queue and port.** `roledesk.db`, `roles.yaml`, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently now that `Jobs` takes a path.
**Own database, queue and port.** `roledesk.db`, `config.yaml`, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently.
**Storage is written, not imported.** `desk` has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`.
**Layers are `models, config, storage, jobs, web, cli`.** `tests/test_architecture.py` holds the direction, the doors, and that a plugin imports `desk` only, since a plugin is app-agnostic.
**Layers are `models, config, storage, jobs, web, main, cli`.** `tests/test_architecture.py` holds the direction and the doors within `app`; `tests/test_coupling.py` holds that `pipeline`, `auth` and `desk` import nothing from `app` or each other, except `desk` from `pipeline`, and that a plugin imports `desk` only.
## Data
@@ -38,36 +36,31 @@ What was decided for roledesk and the one reason for each. The longer reasoning
## Ingestion
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** The `ats` plugin stamps `ats:ok:<target>` with the run's own marker and `ats:ids:<target>` with the ids it saw on every board it parses without raising. The `pull` task's settle then closes a board only when its stamp matches the marker the finder wrote before any batch ran. A batch job that raises part way through a ten-board chunk still leaves the boards it had already finished stamped, so one bad board does not cost its batch-mates their closure — and a board a batch never got to (the whole job crashed, or was never claimed) stays open rather than being closed on an empty id list.
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** The `ats` plugin stamps each board it parses without raising with the run's marker and the ids it saw, and the pull's settle closes a board only when its stamp matches the marker written before any batch ran. So a batch that raises part way still closes the boards it finished, and a board no batch reached stays open instead of being closed on an empty id list.
**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department but not the job text; that is only on the per-posting detail endpoint, and fetching one extra request per role would multiply a board's request count by however many roles it lists. `description` stays empty for this host; title, location, department and the derived `function` are still filled.
**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department; the job text is only on the per-posting endpoint, and one extra request per role would multiply a board's request count by its role count. `description` stays empty for this host.
**The 8 MB cap and `?content=true` fight each other on the largest boards, and the cap won.** Both are named in the plan — the cap from openings.go, `content=true` so Greenhouse ships the description inline — but jobhunt-hq's own feeds never asked for content, so it never hit the size this combination produces. The real run (2026-09-23) refused seven boards outright for it: Anduril at 41 MB, and Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex between 9 and 19 MB, each because the biggest employers publish the most roles with the longest descriptions. Left as is, because raising the cap only moves the line and a company that big is exactly the case `openings.go`'s cap was written against; a real fix — retry without `content=true` when the capped response is refused, trading the description for at least the listing — is a follow-up, not a guess made here.
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
**The cap is raised to 96 MB, with one retry for Greenhouse.** The follow-up
the entry above deferred. Every URL `ats` fetches is on a fixed allow-list of
seven ATS API hosts, not an arbitrary one, so the cap openings.go wrote
against arbitrary URLs was the wrong number to port unchanged. 96 MB fits
every board seen so far, including Anduril's 41 MB; a board that still
exceeds it retries once without `?content=true` (Greenhouse only, since that
is the one host whose feed URL carries the flag), trading the description for
at least the listing.
**Workable's shape is confirmed, not a populated example.** Every slug tried during the build returned the documented empty `{"total": 0, "results": []}` before Cloudflare rate-limited the session, so the parser follows Workable's published job object. Recheck it against a live board with open roles the next time the plugin is touched.
**Workable's shape is confirmed, not a populated example.** Every slug tried against `apply.workable.com/api/v3/accounts/{slug}/jobs` during this build returned the documented `{"total": 0, "results": []}` before Cloudflare rate-limited further requests from this session, so the parser is written from Workable's long-published job object rather than from a fetched row. Worth a fixture-level recheck against a live board with open roles the next time this plugin is touched.
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit on the Schedules tab overrides for good. `pull` is not also a task, which would draw two buttons on the Jobs tab.
## Ranking
**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the gates and floors the score, `-1` mutes.
**`fit` is the cosine between a role and the agent's intent, min-max scaled across the roles that agent scores in the run.** `(cos + 1) / 2` spanned only about 0.55 to 0.79 in the real run, so a prefer gate at 0.2 outweighed fit; scaled, the closest role is 1 and the farthest 0. A single role gets 1, gated roles stay out of the range, and the starter agents' prefer weights are 0.05. The intent is a CV or a paragraph, so the embedding backend is effectively required: TF-IDF cannot relate "Kubernetes" to "container orchestration".
**`fit` is the cosine between a role and the agent's intent, min-max scaled across the roles that agent scores in the run.** `(cos + 1) / 2` spanned only about 0.55 to 0.79 in the real run, so a prefer gate at 0.2 outweighed all of fit. A single role gets 1, gated roles stay out of the range, and the starter agents' prefer weights are 0.05.
**The embedding backend is effectively required.** The intent is a CV or a paragraph per agent, and TF-IDF cannot relate "Kubernetes" to "container orchestration".
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty.
**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 and the 0.9 floor for a favourite means the same thing whatever the weights are. A prefer gate normalised by the prefer weights alone, then weighted as one term, comes to the same number.
**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 and the 0.9 floor for a favourite means the same thing whatever the weights are.
**Gates match loosely where the column is free text.** `place` is a case-insensitive substring, like the place filter, so `Berlin` matches "Berlin, Germany". `company` compares company keys. `languages` passes when every language the role requires is one of the values. `salary_min` is at least the smallest value. The rest are case-insensitive equality. The gates are evaluated in SQL as one `CASE` column each, so a `require` gate is the `WHERE` it reads as while the gated row and its reason survive. The reason lists every failed require gate, not just the first.
**Gates match loosely where the column is free text.** `place` is a case-insensitive substring, so `Berlin` matches "Berlin, Germany"; `company` compares company keys, `languages` passes when every required language is a value, `salary_min` is at least the smallest value, and the rest are case-insensitive equality. Each gate is one SQL `CASE` column, so a require gate filters while the gated row keeps every failed gate as its reason.
**A role with no vector is scored without `fit`.** The fit weight leaves the denominator, so a role `embed` has not reached yet ranks on recency, affinity and prefer gates instead of sinking to the bottom. It is gated as "no vector" only when fit is the only weighted term. An agent whose intent could not be embedded (empty, or no sentence-transformers) says "no intent vector" instead.
**A role with no vector is scored without `fit`.** The fit weight leaves the denominator, so a role `embed` has not reached yet ranks on recency, affinity and prefer gates instead of sinking to the bottom. It is gated as "no vector" only when fit is the only weighted term, and as "no intent vector" when the intent could not be embedded.
**Each run replaces an agent's scores.** `replace_scores` deletes and inserts in one transaction, so a role closed or muted since the last run drops out rather than keeping a stale score, and a reader sees one list or the other.
@@ -77,13 +70,13 @@ at least the listing.
**The pull cascade is `pull → extract → embed → score`.** Every stage is always in the chain and a stage that is off finds nothing, as in newsdesk. Extract precedes score so the gates read what it filled, and reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly.
## The alpha page
## The pages
**The template renders an empty shell; `static/app.js` fills it and drives every action.** `index.html` carries only the filter bar's `<select>` options (from `roledesk.models` and `plugins.ats.FUNCTIONS`, so they can't drift from what the store accepts) and the section containers. Every row, count and button state is a fetch from the JS, including the initial load — one code path for "show this" rather than a server-rendered first page plus a client re-render for every filter change after.
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models` and `plugins.ats.FUNCTIONS`, so they cannot drift from what the store accepts). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change.
**`roles(company=, place=)` moved from a key match to a case-insensitive substring on the column a board wrote.** The API's company and place fields are free text a person types, not a slug; a search for "Trade Republic" should not have to know it folds to `trade-republic-bank`. Nothing outside `tests/test_storage.py` called either parameter, so the change is free.
**`roles(company=, place=)` match a case-insensitive substring of what the board wrote.** The API's company and place fields are free text a person types, not a slug, so "Trade Republic" finds `trade-republic-bank` without knowing the key.
**A muted company is excluded in SQL, before `LIMIT`, not filtered out of the page after.** Filtering after `LIMIT` shipped first and broke on real data within the same session: Databricks' 884 roles, all freshly pulled at once by the cap fix above, were the newest 200-plus rows in the table, so muting it — with the default query taking the newest 200 — left nothing to show. `Store.roles(hide_muted=True)` adds `company_key NOT IN (SELECT key FROM companies WHERE affinity = -1)` to the `WHERE` clause instead, so the limit is applied to what is actually shown.
**A muted company is excluded in SQL, before `LIMIT`.** Filtering after `LIMIT` shipped first and showed nothing once Databricks' 884 freshly pulled roles filled the newest 200 rows and Databricks was muted.
## Outside the app