8.4 KiB
Plan: board size, two strikes, text for SmartRecruiters and Workable, Jobindex link-out
From the architecture walkthrough of 2026-09-30. The owner's decisions:
- A board's size is decided once, when the board is added, and changed only by a Recheck. It is no longer rewritten by every unfiltered read. A board with no size yet is sized on its first pull before its reach is chosen, so a new big board is not read whole worldwide first (Infineon's first pull read 999 roles across the world and cut at page 100).
- A host that states no total on page 1 (Teamtailor, HR Manager,
complete="end") is read to the end once to size it. - A board swept across pulls closes a role only when two finished sweeps in a row missed it. Offset paging drifts when roles close between pulls, so one sweep can miss a role that is still open.
- No head-then-tail read and no sweep display: a new role on page 1 of a swept board waits one sweep.
- Big boards on hosts that cannot filter (Greenhouse, Ashby, Personio) are still stored whole. Gates keep their out-of-reach roles away from enrich, extract and summarise, so they cost prep and embedding only.
- SmartRecruiters and Workable get per-role text through
detail, read byenrichfor the top roles. Both per-role endpoints were checked live on 2026-09-30:api.smartrecruiters.com/v1/companies/<id>/postings/<id>hasjobAd.sections(about 4,500 characters on a Delivery Hero role),apply.workable.com/api/v2/accounts/<slug>/jobs/<shortcode>hasdescription,requirementsandbenefits(about 7,000 on a Hugging Face role). 209 and 145 open roles have no text today. - Jobindex's
vis-jobpage carries a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's ownjobannoncepage). When the owner fetches a Jobindex role's text, that link is resolved, the text is read from it, and the reader's link-out goes there. On demand only, since following it is a click in Jobindex's statistics. - The Companies line says when a Workday or Oracle site does not know a country or has no remote option, instead of naming what was meant to be asked.
Work packages
Each package owns its files; no two packages edit the same file. The planner rewires fetch_text in app/web/api.py to WP D's function after both merge, and folds this plan into docs/decisions.md.
WP A: board size and the Companies line (Opus)
Owns app/jobs/reach.py, app/jobs/sources.py, plugins/ats.py, app/web/api.py, app/static/companies.js, app/cli.py, and the tests tests/test_reach.py, tests/test_web_reach.py, tests/test_sources.py, tests/test_companies.py, tests/test_ats.py, tests/test_web.py, plus new test files of its own.
- Size lives in
stateas today,ats:size:<target>, withats:sizechecked:<target>(ISO date). No migration. plugins/ats.pygetsboard_size(target) -> int: a whole host's listing length; a paged host withcomplete="stated"the total on page 1 (call the module'spageonce with an unfilteredAsk()); a host withcomplete="end"or no stated total, read to the end through_runner.read_listingwithmax_pageshigh enough for the host (a Teamtailor board is rarely over 1,000). No filter, no terms.set_boardwithcheck=Truesizes the board withboard_sizeinstead offetch_board's whole read, which read Infineon to page 100. It still refuses a board that does not answer, and still returns a role count for the CLI and the route.- An ordinary pull stops writing
ats:size. A board with no size gets one in its batch before it is read:Plan.for_subscriptionhands such a board the agents' reach with"unsized": trueand"small": SMALL;ATSSource._boardseesunsized, callsboard_size, writes both state keys, and reads whole when the size is undersmall, else with the reach it was given.reach.sizesdrops its fallback to open-role counts once every existing board hasats:size(all 287 live boards have one today except thosewhycallsunknown). - Recheck:
POST /api/sources/{id}/sizerunsboard_size, writes both keys, queues a pull like the other routes that change what is read, and answers the new size and the Companies line.app companies size <key>does the same from the CLI, andapp companies size --missingsizes every board without one./api/companiesreturnssizeandsize_checked. The Companies row shows "999 roles · checked 30 Sep" beside the reach and a Recheck button, in the pattern of the row's existing actions. reach.linereads the facets the runner keeps (ats:facets:<target>,{"checked", "facets"}) for a reader whose country or remote filter isids, and names what the site does not know: "follows agents: DK · SE not on this site · no remote option · all roles". When every value is unknown: "site lists none of DK, SE: nothing asked". With no facets kept yet, the line is as today.
WP B: two strikes for swept boards (Sonnet)
Owns plugins/readers/_runner.py, app/jobs/pull.py, tests/test_runner.py, tests/test_pull.py, plus new test files of its own.
Listinggetsswept: bool, true when a complete listing finished a sweep carried over from an earlier pull (read_listingresumed from a stored sweep).stampwrites<prefix>:swept:<target>as "1" or "".close_boards: for a complete, unnarrowed board read withsweptset, the roles to close are the board's open roles missing from the ids now and also inats:missing:<target>, the ids the previous finished sweep missed. It writes the ids missed now toats:missing:<target>. A filtered swept read keeps its exceptions (votes, applications). A complete read that was not swept closes as today and clearsats:missing:<target>.- Nothing else in closure changes: failing for
STALE_DAYS, switched off, searches.
WP C: per-role text for SmartRecruiters and Workable (Sonnet)
Owns plugins/readers/smartrecruiters.py, plugins/readers/workable.py, tests/test_ats_smartrecruiters.py, tests/test_ats_workable.py, and new fixtures under tests/fixtures/ats/ for the two detail answers.
- Each gets
detail(get, slug, row)andREADS.text="detail". SmartRecruiters joinsjobDescription,qualificationsandadditionalInformation(HTML to text as the other readers do; leavecompanyDescriptionout). Workable joinsdescription,requirementsandbenefits. Return onlyDETAIL_FIELDS; add places, remote or employment only where the answer states them more fully than the list. - The row needs what the detail URL needs: SmartRecruiters' posting id is the
source_id; Workable'sshortcodemust be kept in the row if it is not already. tests/test_readers_contract.pymust pass with the new declarations; its fixtures are owned here for these two readers only.- Check both against the live endpoints once, with the fetch the runner hands over (
ats._getter), and record the date in the module docstring as the other readers do.
WP D: Jobindex link-out (Sonnet)
Owns plugins/readers/jobindex.py, tests/test_readers_jobindex.py, a new app/jobs/posting.py, app/storage/schema.py, app/storage/store.py, app/models.py, app/static/roles.js, tests/test_storage.py, and new test files of its own.
- Migration 26:
roles.posting_url TEXT NOT NULL DEFAULT ''.upsert_rolenever writes it, so a pull keeps it. plugins/readers/jobindex.pygetsposting_link(html: str) -> str: the "Se jobbet"hrefof avis-jobpage, absolute,''when there is none. Fixtures from a savedvis-jobpage of each kind (employer site, The Hub,jobannonce).app/jobs/posting.py:fetch_posting(store, uid) -> Role, whatfetch_textinapp/web/api.pydoes today, plus: for a Jobindex role, fetchoriginal_url, takeposting_link, store it asposting_url, and scrape the text from it, falling back to thevis-jobpage's own text when there is no link or it gives none. Same errors asfetch_textraises today, as exceptions the route maps. Do not editapi.py; the planner rewires the route.roles.js: the reader's link-out and the "Also posted" links preferposting_urloveroriginal_url, throughsafeUrl.
After the merge
The planner rewires fetch_text, runs app companies size --missing against the live database, runs the full test suite, restarts the server on 8790 if it was running, and folds this plan into docs/decisions.md: the SmartRecruiters and Workable "no description" entries, the size entry under "The agents set the reach", closure, and the Jobindex on-demand text entry.