Files
jobboard/docs/decisions.md
T
WolfandClaude Sonnet 5 0f59452b2a Seed 186 companies, add roledesk companies, and pull and close every board
seed/companies.json: 179 companies from jobhunt-hq's seed.json (MIT),
reduced to key/name/board/careers_url, plus a hand-added DACH set.
`roledesk companies seed|add|check` import it, add one company from its
board with a live check first, and report dead boards.

roledesk/jobs/pull.py is now a pipeline task: ten boards per pull_batch job
on the http resource, upserting every role the ats plugin yields. Its final
job's settle closes every board this run answered in full and no longer
lists a role from, reading the plugin's own self.state stamps rather than
each batch job's return value, so one bad board in a batch does not cost the
other nine their closure. `roledesk job pull` runs it, and it registers
itself in pipeline.TASKS so <desk-jobs> can start it too.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 11:27:30 +02:00

5.9 KiB

Key decisions

What was decided for roledesk and the one reason for each. The longer reasoning is in ../news/docs/JobBoardPlan/roles-build-plan.md and jobhunt-hq-assessment.md.

Shape

Own repo, depending on news for desk and pipeline. pyproject.toml takes newsdesk as an editable path dependency on ../news. desk imports no app and news's architecture test enforces it, so it works as a library. Copying it would fork the plugin contract; pipeline alone may be copied later if depending on a repo named news becomes the problem.

desk and pipeline are never edited here. A change roles needs lands in news, and roledesk picks it up through the editable install.

The domain noun is role. jobs/ keeps meaning the work the app queues, so "job" never has two senses in one sentence.

Own database, queue and port. roledesk.db, roles.yaml, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently now that Jobs takes a path.

Storage is written, not imported. desk has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared Connection moves into desk below pipeline.

Layers are models, config, storage, jobs, web, cli. tests/test_architecture.py holds the direction, the doors, and that a plugin imports desk only, since a plugin is app-agnostic.

Data

Clustering key is (company_key, title_key, place_key). newsdesk's location_key OR title_key collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.

place, country and remote are separate columns. location already means where an Item's original lives. remote alone over-matches, because boards write "UK | Remote", so a gate on remote also reads country.

raw is transport, columns are storage. No Item.facets. A plugin puts structured fields in raw under the column's name and upsert_role unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what extract found.

Postings are persisted. Storing every posting is what gives first_seen, which is the only honest posted date since Greenhouse resets updated_at on every edit, and closed_at, set by mark_closed when a complete fetch of a board no longer lists a role. A role listed again is open again.

The tracker schema is taken from jobhunt-hq (MIT). applications with status in saved, applied, screen, technical, onsite, offer, rejected, withdrawn, next_action, next_date, and an append-only application_events written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.

subscriptions names its labels company and board. newsdesk's outlet and section are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.

Ingestion

Closure is decided from self.state, not from what a pull_batch job returns. The ats plugin stamps ats:ok:<target> with the run's own marker and ats:ids:<target> with the ids it saw on every board it parses without raising. The pull task's settle then closes a board only when its stamp matches the marker the finder wrote before any batch ran. A batch job that raises part way through a ten-board chunk still leaves the boards it had already finished stamped, so one bad board does not cost its batch-mates their closure — and a board a batch never got to (the whole job crashed, or was never claimed) stays open rather than being closed on an empty id list.

SmartRecruiters' board list has no description. Its /postings endpoint gives title, location and department but not the job text; that is only on the per-posting detail endpoint, and fetching one extra request per role would multiply a board's request count by however many roles it lists. description stays empty for this host; title, location, department and the derived function are still filled.

Workable's shape is confirmed, not a populated example. Every slug tried against apply.workable.com/api/v3/accounts/{slug}/jobs during this build returned the documented {"total": 0, "results": []} before Cloudflare rate-limited further requests from this session, so the parser is written from Workable's long-published job object rather than from a fetched row. Worth a fixture-level recheck against a live board with open roles the next time this plugin is touched.

Ranking

Scoring is per search agent. A role is scored as (uid, agent) -> score, because a role is only good for a given search. Each gate has a mode: require becomes a SQL WHERE, prefer becomes a weighted term in the score. Gated roles keep their row with a reason. companies.affinity = 2 skips the gates and floors the score, -1 mutes.

fit is cosine between the role and the agent's intent text. The intent is a CV or a paragraph per agent. This is why the embedding backend is effectively required: TF-IDF cannot relate "Kubernetes" to "container orchestration".

Summary and tagging are off; extract replaces them. Every profile declares SummaryPolicy(enabled=False) and TaggingPolicy(enabled=False). extract is a roledesk ModelTask on Haiku that turns title and description into the gate columns, and skips rows a plugin already filled from structured fields.

Outside the app

Unposted roles and the network stay outside. Their state is inside: an application row with no role_uid, a contact, next_action and next_date, and a company status such as researching or contacted.

LinkedIn enters only as alert mail over IMAP. Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.