Files
jobboard/docs/decisions.md
T
WolfandClaude Sonnet 5 867ebe6405 Raise the ats body cap to 96 MB, retry Greenhouse without content=true
openings.go's 8 MB cap was written against arbitrary URLs; every URL here is
on a fixed allow-list of seven ATS hosts, and the real pull of 2026-09-23
refused Anduril, Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex
for it. A board that still exceeds 96 MB now retries once without
content=true (Greenhouse only), trading the description for the listing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-23 11:39:39 +02:00

7.3 KiB

Key decisions

What was decided for roledesk and the one reason for each. The longer reasoning is in ../news/docs/JobBoardPlan/roles-build-plan.md and jobhunt-hq-assessment.md.

Shape

Own repo, depending on news for desk and pipeline. pyproject.toml takes newsdesk as an editable path dependency on ../news. desk imports no app and news's architecture test enforces it, so it works as a library. Copying it would fork the plugin contract; pipeline alone may be copied later if depending on a repo named news becomes the problem.

desk and pipeline are never edited here. A change roles needs lands in news, and roledesk picks it up through the editable install.

The domain noun is role. jobs/ keeps meaning the work the app queues, so "job" never has two senses in one sentence.

Own database, queue and port. roledesk.db, roles.yaml, port 8790. A shared file with newsdesk would buy "articles about this company" by cosine, which is worth less than two apps that can migrate independently now that Jobs takes a path.

Storage is written, not imported. desk has no storage package on purpose, so ~120 lines of connection plumbing are copied from newsdesk. If the same edit lands in both a second time, a shared Connection moves into desk below pipeline.

Layers are models, config, storage, jobs, web, cli. tests/test_architecture.py holds the direction, the doors, and that a plugin imports desk only, since a plugin is app-agnostic.

Data

Clustering key is (company_key, title_key, place_key). newsdesk's location_key OR title_key collapses two outlets running one story, which is right for wire copy and wrong for ads, where the same title at two companies is two jobs.

place, country and remote are separate columns. location already means where an Item's original lives. remote alone over-matches, because boards write "UK | Remote", so a gate on remote also reads country.

raw is transport, columns are storage. No Item.facets. A plugin puts structured fields in raw under the column's name and upsert_role unpacks them into indexed columns, which is where gates query them. An empty value never overwrites a filled column, so a terse board does not erase what extract found.

Postings are persisted. Storing every posting is what gives first_seen, which is the only honest posted date since Greenhouse resets updated_at on every edit, and closed_at, set by mark_closed when a complete fetch of a board no longer lists a role. A role listed again is open again.

The tracker schema is taken from jobhunt-hq (MIT). applications with status in saved, applied, screen, technical, onsite, offer, rejected, withdrawn, next_action, next_date, and an append-only application_events written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.

subscriptions names its labels company and board. newsdesk's outlet and section are news words; roledesk has no rows to migrate, so it uses its own names from the first migration.

Ingestion

Closure is decided from self.state, not from what a pull_batch job returns. The ats plugin stamps ats:ok:<target> with the run's own marker and ats:ids:<target> with the ids it saw on every board it parses without raising. The pull task's settle then closes a board only when its stamp matches the marker the finder wrote before any batch ran. A batch job that raises part way through a ten-board chunk still leaves the boards it had already finished stamped, so one bad board does not cost its batch-mates their closure — and a board a batch never got to (the whole job crashed, or was never claimed) stays open rather than being closed on an empty id list.

SmartRecruiters' board list has no description. Its /postings endpoint gives title, location and department but not the job text; that is only on the per-posting detail endpoint, and fetching one extra request per role would multiply a board's request count by however many roles it lists. description stays empty for this host; title, location, department and the derived function are still filled.

The 8 MB cap and ?content=true fight each other on the largest boards, and the cap won. Both are named in the plan — the cap from openings.go, content=true so Greenhouse ships the description inline — but jobhunt-hq's own feeds never asked for content, so it never hit the size this combination produces. The real run (2026-09-23) refused seven boards outright for it: Anduril at 41 MB, and Anthropic, OpenAI, Databricks, Carvana, Axon and Airwallex between 9 and 19 MB, each because the biggest employers publish the most roles with the longest descriptions. Left as is, because raising the cap only moves the line and a company that big is exactly the case openings.go's cap was written against; a real fix — retry without content=true when the capped response is refused, trading the description for at least the listing — is a follow-up, not a guess made here.

The cap is raised to 96 MB, with one retry for Greenhouse. The follow-up the entry above deferred. Every URL ats fetches is on a fixed allow-list of seven ATS API hosts, not an arbitrary one, so the cap openings.go wrote against arbitrary URLs was the wrong number to port unchanged. 96 MB fits every board seen so far, including Anduril's 41 MB; a board that still exceeds it retries once without ?content=true (Greenhouse only, since that is the one host whose feed URL carries the flag), trading the description for at least the listing.

Workable's shape is confirmed, not a populated example. Every slug tried against apply.workable.com/api/v3/accounts/{slug}/jobs during this build returned the documented {"total": 0, "results": []} before Cloudflare rate-limited further requests from this session, so the parser is written from Workable's long-published job object rather than from a fetched row. Worth a fixture-level recheck against a live board with open roles the next time this plugin is touched.

Ranking

Scoring is per search agent. A role is scored as (uid, agent) -> score, because a role is only good for a given search. Each gate has a mode: require becomes a SQL WHERE, prefer becomes a weighted term in the score. Gated roles keep their row with a reason. companies.affinity = 2 skips the gates and floors the score, -1 mutes.

fit is cosine between the role and the agent's intent text. The intent is a CV or a paragraph per agent. This is why the embedding backend is effectively required: TF-IDF cannot relate "Kubernetes" to "container orchestration".

Summary and tagging are off; extract replaces them. Every profile declares SummaryPolicy(enabled=False) and TaggingPolicy(enabled=False). extract is a roledesk ModelTask on Haiku that turns title and description into the gate columns, and skips rows a plugin already filled from structured fields.

Outside the app

Unposted roles and the network stay outside. Their state is inside: an application row with no role_uid, a contact, next_action and next_date, and a company status such as researching or contacted.

LinkedIn enters only as alert mail over IMAP. Scraping it is banned by contract and enforced. The alert mails also carry personalised recommendations no public API exposes.