Files
jobboard/docs/decisions.md
T
WolfandClaude Opus 5.5 4dfb8831aa Extract gate columns from posting text with a Haiku ModelTask
roledesk job extract reads open, unanswered roles 20 to a call and fills
place, country, remote, seniority, salary, currency, function, permit and
languages, only into columns the board left empty. Schema v2 adds
roles.permit, roles.languages and roles.extract_tries. Off by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-23 12:05:30 +02:00

79 lines
9.8 KiB
Markdown

# Key decisions
What was decided for roledesk and the one reason for each. The longer reasoning is in `../news/docs/JobBoardPlan/roles-build-plan.md` and `jobhunt-hq-assessment.md`.
## Shape
**Own repo, depending on news for `desk` and `pipeline`.** `pyproject.toml` takes `newsdesk` as an editable path dependency on `../news`. `desk` imports no app and news's architecture test enforces it, so it works as a library. Copying it would fork the plugin contract; `pipeline` alone may be copied later if depending on a repo named news becomes the problem.
**`desk` and `pipeline` are never edited here.** A change roles needs lands in news, and roledesk picks it up through the editable install.
**The domain noun is `role`.** `jobs/` keeps meaning the work the app queues, so "job" never has two senses in one sentence.
**Own database, queue and port.** `roledesk.db`, `roles.yaml`, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently now that `Jobs` takes a path.
**Storage is written, not imported.** `desk` has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared `Connection` moves into `desk` below `pipeline`.
**Layers are `models, config, storage, jobs, web, cli`.** `tests/test_architecture.py` holds the direction, the doors, and that a plugin imports `desk` only, since a plugin is app-agnostic.
## Data
**Clustering key is `(company_key, title_key, place_key)`.** newsdesk's `location_key OR title_key` collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.
**`place`, `country` and `remote` are separate columns.** `location` already means where an `Item`'s original lives. `remote` alone over-matches, because boards write "UK | Remote", so a gate on remote also reads `country`.
**`raw` is transport, columns are storage.** No `Item.facets`. A plugin puts structured fields in `raw` under the column's name and `upsert_role` unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what `extract` found.
**`extract` fills, the board wins.** `set_extracted` writes a model's value only into an empty column. A board's structured field is first-party, and `upsert_role` would put it back on the next pull anyway, so letting the model replace it would make a column flip between two answers. `place` brings `place_key` with it, which only ever moves a role from the empty place key to a real one.
**`extract` reads the board's location too.** The model gets title, the board's `place` text, and the first 3000 characters of the description. Greenhouse's location line ("Berlin, Germany") is often the only place a country is stated, and it is part of the posting, so the "only what the posting states" contract still holds.
**`permit` and `languages` are columns, and so is `extract_tries`.** `permit` answers the plan's open item 1 (visa sponsorship as a gate field) while nothing depends on it yet. newsdesk keeps give-up counts in `raw`; here `upsert_role` rewrites `raw` on every pull, which would reset the count, so the counter is its own column.
**Postings are persisted.** Storing every posting is what gives `first_seen`, which is the only honest posted date since Greenhouse resets `updated_at` on every edit, and `closed_at`, set by `mark_closed` when a complete fetch of a board no longer lists a role. A role listed again is open again.
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with status in `saved, applied, screen, technical, onsite, offer, rejected, withdrawn`, `next_action`, `next_date`, and an append-only `application_events` written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.
**`subscriptions` names its labels `company` and `board`.** newsdesk's `outlet` and `section` are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.
## Ingestion
**Closure is decided from `self.state`, not from what a `pull_batch` job returns.** The `ats` plugin stamps `ats:ok:<target>` with the run's own marker and `ats:ids:<target>` with the ids it saw on every board it parses without raising. The `pull` task's settle then closes a board only when its stamp matches the marker the finder wrote before any batch ran. A batch job that raises part way through a ten-board chunk still leaves the boards it had already finished stamped, so one bad board does not cost its batch-mates their closure — and a board a batch never got to (the whole job crashed, or was never claimed) stays open rather than being closed on an empty id list.
**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department but not the job text; that is only on the per-posting detail endpoint, and fetching one extra request per role would multiply a board's request count by however many roles it lists. `description` stays empty for this host; title, location, department and the derived `function` are still filled.
**The 8 MB cap and `?content=true` fight each other on the largest boards, and the cap won.** Both are named in the plan — the cap from openings.go, `content=true` so Greenhouse ships the description inline — but jobhunt-hq's own feeds never asked for content, so it never hit the size this combination produces. The real run (2026-09-23) refused seven boards outright for it: Anduril at 41 MB, and Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex between 9 and 19 MB, each because the biggest employers publish the most roles with the longest descriptions. Left as is, because raising the cap only moves the line and a company that big is exactly the case `openings.go`'s cap was written against; a real fix — retry without `content=true` when the capped response is refused, trading the description for at least the listing — is a follow-up, not a guess made here.
**The cap is raised to 96 MB, with one retry for Greenhouse.** The follow-up
the entry above deferred. Every URL `ats` fetches is on a fixed allow-list of
seven ATS API hosts, not an arbitrary one, so the cap openings.go wrote
against arbitrary URLs was the wrong number to port unchanged. 96 MB fits
every board seen so far, including Anduril's 41 MB; a board that still
exceeds it retries once without `?content=true` (Greenhouse only, since that
is the one host whose feed URL carries the flag), trading the description for
at least the listing.
**Workable's shape is confirmed, not a populated example.** Every slug tried against `apply.workable.com/api/v3/accounts/{slug}/jobs` during this build returned the documented `{"total": 0, "results": []}` before Cloudflare rate-limited further requests from this session, so the parser is written from Workable's long-published job object rather than from a fetched row. Worth a fixture-level recheck against a live board with open roles the next time this plugin is touched.
## Ranking
**Scoring is per search agent.** A role is scored as `(uid, agent) -> score`, because a role is only good for a given search. Each gate has a mode: `require` becomes a SQL `WHERE`, `prefer` becomes a weighted term in the score. Gated roles keep their row with a reason. `companies.affinity = 2` skips the gates and floors the score, `-1` mutes.
**`fit` is cosine between the role and the agent's intent text.** The intent is a CV or a paragraph per agent. This is why the embedding backend is effectively required: TF-IDF cannot relate "Kubernetes" to "container orchestration".
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty.
## The alpha page
**The template renders an empty shell; `static/app.js` fills it and drives every action.** `index.html` carries only the filter bar's `<select>` options (from `roledesk.models` and `plugins.ats.FUNCTIONS`, so they can't drift from what the store accepts) and the section containers. Every row, count and button state is a fetch from the JS, including the initial load — one code path for "show this" rather than a server-rendered first page plus a client re-render for every filter change after.
**`roles(company=, place=)` moved from a key match to a case-insensitive substring on the column a board wrote.** The API's company and place fields are free text a person types, not a slug; a search for "Trade Republic" should not have to know it folds to `trade-republic-bank`. Nothing outside `tests/test_storage.py` called either parameter, so the change is free.
**A muted company is excluded in SQL, before `LIMIT`, not filtered out of the page after.** Filtering after `LIMIT` shipped first and broke on real data within the same session: Databricks' 884 roles, all freshly pulled at once by the cap fix above, were the newest 200-plus rows in the table, so muting it — with the default query taking the newest 200 — left nothing to show. `Store.roles(hide_muted=True)` adds `company_key NOT IN (SELECT key FROM companies WHERE affinity = -1)` to the `WHERE` clause instead, so the limit is applied to what is actually shown.
## Outside the app
**Unposted roles and the network stay outside.** Their state is inside: an application row with no `role_uid`, a `contact`, `next_action` and `next_date`, and a company `status` such as researching or contacted.
**LinkedIn enters only as alert mail over IMAP.** Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.