Files
jobboard/docs/decisions.md
T
WolfandClaude Sonnet 5 b44f78ca7e Serve one alpha page: filters, roles, the tracker, and Jobs
/, GET /api/roles|companies|applications, POST /api/companies/{key}/affinity
and /api/applications[/{id}/status]. The Jinja2 template renders an empty
shell; static/app.js fills it and drives Save, affinity and status changes.
<desk-jobs> from pipeline.web mounts at the bottom so Pull can be started
from the page.

Store.roles gains q, function, remote, days and hide_muted, and company/
place move from a key match to a substring on the text a board wrote.
hide_muted filters in SQL before LIMIT: filtering after it shipped first and
broke immediately on real data, since Databricks' freshly pulled 884 roles
were the newest 200-plus rows in the table and muting it left nothing to
show.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 11:56:10 +02:00

8.7 KiB

Key decisions

What was decided for roledesk and the one reason for each. The longer reasoning is in ../news/docs/JobBoardPlan/roles-build-plan.md and jobhunt-hq-assessment.md.

Shape

Own repo, depending on news for desk and pipeline. pyproject.toml takes newsdesk as an editable path dependency on ../news. desk imports no app and news's architecture test enforces it, so it works as a library. Copying it would fork the plugin contract; pipeline alone may be copied later if depending on a repo named news becomes the problem.

desk and pipeline are never edited here. A change roles needs lands in news, and roledesk picks it up through the editable install.

The domain noun is role. jobs/ keeps meaning the work the app queues, so "job" never has two senses in one sentence.

Own database, queue and port. roledesk.db, roles.yaml, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently now that Jobs takes a path.

Storage is written, not imported. desk has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared Connection moves into desk below pipeline.

Layers are models, config, storage, jobs, web, cli. tests/test_architecture.py holds the direction, the doors, and that a plugin imports desk only, since a plugin is app-agnostic.

Data

Clustering key is (company_key, title_key, place_key). newsdesk's location_key OR title_key collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.

place, country and remote are separate columns. location already means where an Item's original lives. remote alone over-matches, because boards write "UK | Remote", so a gate on remote also reads country.

raw is transport, columns are storage. No Item.facets. A plugin puts structured fields in raw under the column's name and upsert_role unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what extract found.

Postings are persisted. Storing every posting is what gives first_seen, which is the only honest posted date since Greenhouse resets updated_at on every edit, and closed_at, set by mark_closed when a complete fetch of a board no longer lists a role. A role listed again is open again.

The tracker schema is taken from jobhunt-hq (MIT). applications with status in saved, applied, screen, technical, onsite, offer, rejected, withdrawn, next_action, next_date, and an append-only application_events written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.

subscriptions names its labels company and board. newsdesk's outlet and section are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.

Ingestion

Closure is decided from self.state, not from what a pull_batch job returns. The ats plugin stamps ats:ok:<target> with the run's own marker and ats:ids:<target> with the ids it saw on every board it parses without raising. The pull task's settle then closes a board only when its stamp matches the marker the finder wrote before any batch ran. A batch job that raises part way through a ten-board chunk still leaves the boards it had already finished stamped, so one bad board does not cost its batch-mates their closure — and a board a batch never got to (the whole job crashed, or was never claimed) stays open rather than being closed on an empty id list.

SmartRecruiters' board list has no description. Its /postings endpoint gives title, location and department but not the job text; that is only on the per-posting detail endpoint, and fetching one extra request per role would multiply a board's request count by however many roles it lists. description stays empty for this host; title, location, department and the derived function are still filled.

The 8 MB cap and ?content=true fight each other on the largest boards, and the cap won. Both are named in the plan — the cap from openings.go, content=true so Greenhouse ships the description inline — but jobhunt-hq's own feeds never asked for content, so it never hit the size this combination produces. The real run (2026-09-23) refused seven boards outright for it: Anduril at 41 MB, and Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex between 9 and 19 MB, each because the biggest employers publish the most roles with the longest descriptions. Left as is, because raising the cap only moves the line and a company that big is exactly the case openings.go's cap was written against; a real fix — retry without content=true when the capped response is refused, trading the description for at least the listing — is a follow-up, not a guess made here.

The cap is raised to 96 MB, with one retry for Greenhouse. The follow-up the entry above deferred. Every URL ats fetches is on a fixed allow-list of seven ATS API hosts, not an arbitrary one, so the cap openings.go wrote against arbitrary URLs was the wrong number to port unchanged. 96 MB fits every board seen so far, including Anduril's 41 MB; a board that still exceeds it retries once without ?content=true (Greenhouse only, since that is the one host whose feed URL carries the flag), trading the description for at least the listing.

Workable's shape is confirmed, not a populated example. Every slug tried against apply.workable.com/api/v3/accounts/{slug}/jobs during this build returned the documented {"total": 0, "results": []} before Cloudflare rate-limited further requests from this session, so the parser is written from Workable's long-published job object rather than from a fetched row. Worth a fixture-level recheck against a live board with open roles the next time this plugin is touched.

Ranking

Scoring is per search agent. A role is scored as (uid, agent) -> score, because a role is only good for a given search. Each gate has a mode: require becomes a SQL WHERE, prefer becomes a weighted term in the score. Gated roles keep their row with a reason. companies.affinity = 2 skips the gates and floors the score, -1 mutes.

fit is cosine between the role and the agent's intent text. The intent is a CV or a paragraph per agent. This is why the embedding backend is effectively required: TF-IDF cannot relate "Kubernetes" to "container orchestration".

Summary and tagging are off; extract replaces them. Every profile declares SummaryPolicy(enabled=False) and TaggingPolicy(enabled=False). extract is a roledesk ModelTask on Haiku that turns title and description into the gate columns, and skips rows a plugin already filled from structured fields.

The alpha page

The template renders an empty shell; static/app.js fills it and drives every action. index.html carries only the filter bar's <select> options (from roledesk.models and plugins.ats.FUNCTIONS, so they can't drift from what the store accepts) and the section containers. Every row, count and button state is a fetch from the JS, including the initial load — one code path for "show this" rather than a server-rendered first page plus a client re-render for every filter change after.

roles(company=, place=) moved from a key match to a case-insensitive substring on the column a board wrote. The API's company and place fields are free text a person types, not a slug; a search for "Trade Republic" should not have to know it folds to trade-republic-bank. Nothing outside tests/test_storage.py called either parameter, so the change is free.

A muted company is excluded in SQL, before LIMIT, not filtered out of the page after. Filtering after LIMIT shipped first and broke on real data within the same session: Databricks' 884 roles, all freshly pulled at once by the cap fix above, were the newest 200-plus rows in the table, so muting it — with the default query taking the newest 200 — left nothing to show. Store.roles(hide_muted=True) adds company_key NOT IN (SELECT key FROM companies WHERE affinity = -1) to the WHERE clause instead, so the limit is applied to what is actually shown.

Outside the app

Unposted roles and the network stay outside. Their state is inside: an application row with no role_uid, a contact, next_action and next_date, and a company status such as researching or contacted.

LinkedIn enters only as alert mail over IMAP. Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.