Files
jobboard/docs/plan-boards-and-text.md
T

8.4 KiB

Plan: board size, two strikes, text for SmartRecruiters and Workable, Jobindex link-out

From the architecture walkthrough of 2026-09-30. The owner's decisions:

  • A board's size is decided once, when the board is added, and changed only by a Recheck. It is no longer rewritten by every unfiltered read. A board with no size yet is sized on its first pull before its reach is chosen, so a new big board is not read whole worldwide first (Infineon's first pull read 999 roles across the world and cut at page 100).
  • A host that states no total on page 1 (Teamtailor, HR Manager, complete="end") is read to the end once to size it.
  • A board swept across pulls closes a role only when two finished sweeps in a row missed it. Offset paging drifts when roles close between pulls, so one sweep can miss a role that is still open.
  • No head-then-tail read and no sweep display: a new role on page 1 of a swept board waits one sweep.
  • Big boards on hosts that cannot filter (Greenhouse, Ashby, Personio) are still stored whole. Gates keep their out-of-reach roles away from enrich, extract and summarise, so they cost prep and embedding only.
  • SmartRecruiters and Workable get per-role text through detail, read by enrich for the top roles. Both per-role endpoints were checked live on 2026-09-30: api.smartrecruiters.com/v1/companies/<id>/postings/<id> has jobAd.sections (about 4,500 characters on a Delivery Hero role), apply.workable.com/api/v2/accounts/<slug>/jobs/<shortcode> has description, requirements and benefits (about 7,000 on a Hugging Face role). 209 and 145 open roles have no text today.
  • Jobindex's vis-job page carries a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's own jobannonce page). When the owner fetches a Jobindex role's text, that link is resolved, the text is read from it, and the reader's link-out goes there. On demand only, since following it is a click in Jobindex's statistics.
  • The Companies line says when a Workday or Oracle site does not know a country or has no remote option, instead of naming what was meant to be asked.

Work packages

Each package owns its files; no two packages edit the same file. The planner rewires fetch_text in app/web/api.py to WP D's function after both merge, and folds this plan into docs/decisions.md.

WP A: board size and the Companies line (Opus)

Owns app/jobs/reach.py, app/jobs/sources.py, plugins/ats.py, app/web/api.py, app/static/companies.js, app/cli.py, and the tests tests/test_reach.py, tests/test_web_reach.py, tests/test_sources.py, tests/test_companies.py, tests/test_ats.py, tests/test_web.py, plus new test files of its own.

  1. Size lives in state as today, ats:size:<target>, with ats:sizechecked:<target> (ISO date). No migration.
  2. plugins/ats.py gets board_size(target) -> int: a whole host's listing length; a paged host with complete="stated" the total on page 1 (call the module's page once with an unfiltered Ask()); a host with complete="end" or no stated total, read to the end through _runner.read_listing with max_pages high enough for the host (a Teamtailor board is rarely over 1,000). No filter, no terms.
  3. set_board with check=True sizes the board with board_size instead of fetch_board's whole read, which read Infineon to page 100. It still refuses a board that does not answer, and still returns a role count for the CLI and the route.
  4. An ordinary pull stops writing ats:size. A board with no size gets one in its batch before it is read: Plan.for_subscription hands such a board the agents' reach with "unsized": true and "small": SMALL; ATSSource._board sees unsized, calls board_size, writes both state keys, and reads whole when the size is under small, else with the reach it was given. reach.sizes drops its fallback to open-role counts once every existing board has ats:size (all 287 live boards have one today except those why calls unknown).
  5. Recheck: POST /api/sources/{id}/size runs board_size, writes both keys, queues a pull like the other routes that change what is read, and answers the new size and the Companies line. app companies size <key> does the same from the CLI, and app companies size --missing sizes every board without one. /api/companies returns size and size_checked. The Companies row shows "999 roles · checked 30 Sep" beside the reach and a Recheck button, in the pattern of the row's existing actions.
  6. reach.line reads the facets the runner keeps (ats:facets:<target>, {"checked", "facets"}) for a reader whose country or remote filter is ids, and names what the site does not know: "follows agents: DK · SE not on this site · no remote option · all roles". When every value is unknown: "site lists none of DK, SE: nothing asked". With no facets kept yet, the line is as today.

WP B: two strikes for swept boards (Sonnet)

Owns plugins/readers/_runner.py, app/jobs/pull.py, tests/test_runner.py, tests/test_pull.py, plus new test files of its own.

  1. Listing gets swept: bool, true when a complete listing finished a sweep carried over from an earlier pull (read_listing resumed from a stored sweep). stamp writes <prefix>:swept:<target> as "1" or "".
  2. close_boards: for a complete, unnarrowed board read with swept set, the roles to close are the board's open roles missing from the ids now and also in ats:missing:<target>, the ids the previous finished sweep missed. It writes the ids missed now to ats:missing:<target>. A filtered swept read keeps its exceptions (votes, applications). A complete read that was not swept closes as today and clears ats:missing:<target>.
  3. Nothing else in closure changes: failing for STALE_DAYS, switched off, searches.

WP C: per-role text for SmartRecruiters and Workable (Sonnet)

Owns plugins/readers/smartrecruiters.py, plugins/readers/workable.py, tests/test_ats_smartrecruiters.py, tests/test_ats_workable.py, and new fixtures under tests/fixtures/ats/ for the two detail answers.

  1. Each gets detail(get, slug, row) and READS.text="detail". SmartRecruiters joins jobDescription, qualifications and additionalInformation (HTML to text as the other readers do; leave companyDescription out). Workable joins description, requirements and benefits. Return only DETAIL_FIELDS; add places, remote or employment only where the answer states them more fully than the list.
  2. The row needs what the detail URL needs: SmartRecruiters' posting id is the source_id; Workable's shortcode must be kept in the row if it is not already.
  3. tests/test_readers_contract.py must pass with the new declarations; its fixtures are owned here for these two readers only.
  4. Check both against the live endpoints once, with the fetch the runner hands over (ats._getter), and record the date in the module docstring as the other readers do.

Owns plugins/readers/jobindex.py, tests/test_readers_jobindex.py, a new app/jobs/posting.py, app/storage/schema.py, app/storage/store.py, app/models.py, app/static/roles.js, tests/test_storage.py, and new test files of its own.

  1. Migration 26: roles.posting_url TEXT NOT NULL DEFAULT ''. upsert_role never writes it, so a pull keeps it.
  2. plugins/readers/jobindex.py gets posting_link(html: str) -> str: the "Se jobbet" href of a vis-job page, absolute, '' when there is none. Fixtures from a saved vis-job page of each kind (employer site, The Hub, jobannonce).
  3. app/jobs/posting.py: fetch_posting(store, uid) -> Role, what fetch_text in app/web/api.py does today, plus: for a Jobindex role, fetch original_url, take posting_link, store it as posting_url, and scrape the text from it, falling back to the vis-job page's own text when there is no link or it gives none. Same errors as fetch_text raises today, as exceptions the route maps. Do not edit api.py; the planner rewires the route.
  4. roles.js: the reader's link-out and the "Also posted" links prefer posting_url over original_url, through safeUrl.

After the merge

The planner rewires fetch_text, runs app companies size --missing against the live database, runs the full test suite, restarts the server on 8790 if it was running, and folds this plan into docs/decisions.md: the SmartRecruiters and Workable "no description" entries, the size entry under "The agents set the reach", closure, and the Jobindex on-demand text entry.