From a19c528e3f94c18a49421642b02419ac0e61edee Mon Sep 17 00:00:00 2001 From: Wolf Date: Wed, 30 Sep 2026 13:19:07 +0200 Subject: [PATCH] Plan board size set once, two strikes for swept boards, SmartRecruiters and Workable text, Jobindex link-out Co-Authored-By: Claude Opus 5.5 --- docs/plan-boards-and-text.md | 57 ++++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 docs/plan-boards-and-text.md diff --git a/docs/plan-boards-and-text.md b/docs/plan-boards-and-text.md new file mode 100644 index 0000000..3568b8a --- /dev/null +++ b/docs/plan-boards-and-text.md @@ -0,0 +1,57 @@ +# Plan: board size, two strikes, text for SmartRecruiters and Workable, Jobindex link-out + +From the architecture walkthrough of 2026-09-30. The owner's decisions: + +- A board's size is decided once, when the board is added, and changed only by a Recheck. It is no longer rewritten by every unfiltered read. A board with no size yet is sized on its first pull before its reach is chosen, so a new big board is not read whole worldwide first (Infineon's first pull read 999 roles across the world and cut at page 100). +- A host that states no total on page 1 (Teamtailor, HR Manager, `complete="end"`) is read to the end once to size it. +- A board swept across pulls closes a role only when two finished sweeps in a row missed it. Offset paging drifts when roles close between pulls, so one sweep can miss a role that is still open. +- No head-then-tail read and no sweep display: a new role on page 1 of a swept board waits one sweep. +- Big boards on hosts that cannot filter (Greenhouse, Ashby, Personio) are still stored whole. Gates keep their out-of-reach roles away from enrich, extract and summarise, so they cost prep and embedding only. +- SmartRecruiters and Workable get per-role text through `detail`, read by `enrich` for the top roles. Both per-role endpoints were checked live on 2026-09-30: `api.smartrecruiters.com/v1/companies//postings/` has `jobAd.sections` (about 4,500 characters on a Delivery Hero role), `apply.workable.com/api/v2/accounts//jobs/` has `description`, `requirements` and `benefits` (about 7,000 on a Hugging Face role). 209 and 145 open roles have no text today. +- Jobindex's `vis-job` page carries a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's own `jobannonce` page). When the owner fetches a Jobindex role's text, that link is resolved, the text is read from it, and the reader's link-out goes there. On demand only, since following it is a click in Jobindex's statistics. +- The Companies line says when a Workday or Oracle site does not know a country or has no remote option, instead of naming what was meant to be asked. + +## Work packages + +Each package owns its files; no two packages edit the same file. The planner rewires `fetch_text` in `app/web/api.py` to WP D's function after both merge, and folds this plan into `docs/decisions.md`. + +### WP A: board size and the Companies line (Opus) + +Owns `app/jobs/reach.py`, `app/jobs/sources.py`, `plugins/ats.py`, `app/web/api.py`, `app/static/companies.js`, `app/cli.py`, and the tests `tests/test_reach.py`, `tests/test_web_reach.py`, `tests/test_sources.py`, `tests/test_companies.py`, `tests/test_ats.py`, `tests/test_web.py`, plus new test files of its own. + +1. Size lives in `state` as today, `ats:size:`, with `ats:sizechecked:` (ISO date). No migration. +2. `plugins/ats.py` gets `board_size(target) -> int`: a whole host's listing length; a paged host with `complete="stated"` the total on page 1 (call the module's `page` once with an unfiltered `Ask()`); a host with `complete="end"` or no stated total, read to the end through `_runner.read_listing` with `max_pages` high enough for the host (a Teamtailor board is rarely over 1,000). No filter, no terms. +3. `set_board` with `check=True` sizes the board with `board_size` instead of `fetch_board`'s whole read, which read Infineon to page 100. It still refuses a board that does not answer, and still returns a role count for the CLI and the route. +4. An ordinary pull stops writing `ats:size`. A board with no size gets one in its batch before it is read: `Plan.for_subscription` hands such a board the agents' reach with `"unsized": true` and `"small": SMALL`; `ATSSource._board` sees `unsized`, calls `board_size`, writes both state keys, and reads whole when the size is under `small`, else with the reach it was given. `reach.sizes` drops its fallback to open-role counts once every existing board has `ats:size` (all 287 live boards have one today except those `why` calls `unknown`). +5. Recheck: `POST /api/sources/{id}/size` runs `board_size`, writes both keys, queues a pull like the other routes that change what is read, and answers the new size and the Companies line. `app companies size ` does the same from the CLI, and `app companies size --missing` sizes every board without one. `/api/companies` returns `size` and `size_checked`. The Companies row shows "999 roles · checked 30 Sep" beside the reach and a Recheck button, in the pattern of the row's existing actions. +6. `reach.line` reads the facets the runner keeps (`ats:facets:`, `{"checked", "facets"}`) for a reader whose country or remote filter is `ids`, and names what the site does not know: "follows agents: DK · SE not on this site · no remote option · all roles". When every value is unknown: "site lists none of DK, SE: nothing asked". With no facets kept yet, the line is as today. + +### WP B: two strikes for swept boards (Sonnet) + +Owns `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_runner.py`, `tests/test_pull.py`, plus new test files of its own. + +1. `Listing` gets `swept: bool`, true when a complete listing finished a sweep carried over from an earlier pull (`read_listing` resumed from a stored sweep). `stamp` writes `:swept:` as "1" or "". +2. `close_boards`: for a complete, unnarrowed board read with `swept` set, the roles to close are the board's open roles missing from the ids now and also in `ats:missing:`, the ids the previous finished sweep missed. It writes the ids missed now to `ats:missing:`. A filtered swept read keeps its exceptions (votes, applications). A complete read that was not swept closes as today and clears `ats:missing:`. +3. Nothing else in closure changes: failing for `STALE_DAYS`, switched off, searches. + +### WP C: per-role text for SmartRecruiters and Workable (Sonnet) + +Owns `plugins/readers/smartrecruiters.py`, `plugins/readers/workable.py`, `tests/test_ats_smartrecruiters.py`, `tests/test_ats_workable.py`, and new fixtures under `tests/fixtures/ats/` for the two detail answers. + +1. Each gets `detail(get, slug, row)` and `READS.text="detail"`. SmartRecruiters joins `jobDescription`, `qualifications` and `additionalInformation` (HTML to text as the other readers do; leave `companyDescription` out). Workable joins `description`, `requirements` and `benefits`. Return only `DETAIL_FIELDS`; add places, remote or employment only where the answer states them more fully than the list. +2. The row needs what the detail URL needs: SmartRecruiters' posting id is the `source_id`; Workable's `shortcode` must be kept in the row if it is not already. +3. `tests/test_readers_contract.py` must pass with the new declarations; its fixtures are owned here for these two readers only. +4. Check both against the live endpoints once, with the fetch the runner hands over (`ats._getter`), and record the date in the module docstring as the other readers do. + +### WP D: Jobindex link-out (Sonnet) + +Owns `plugins/readers/jobindex.py`, `tests/test_readers_jobindex.py`, a new `app/jobs/posting.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/models.py`, `app/static/roles.js`, `tests/test_storage.py`, and new test files of its own. + +1. Migration 26: `roles.posting_url TEXT NOT NULL DEFAULT ''`. `upsert_role` never writes it, so a pull keeps it. +2. `plugins/readers/jobindex.py` gets `posting_link(html: str) -> str`: the "Se jobbet" `href` of a `vis-job` page, absolute, `''` when there is none. Fixtures from a saved `vis-job` page of each kind (employer site, The Hub, `jobannonce`). +3. `app/jobs/posting.py`: `fetch_posting(store, uid) -> Role`, what `fetch_text` in `app/web/api.py` does today, plus: for a Jobindex role, fetch `original_url`, take `posting_link`, store it as `posting_url`, and scrape the text from it, falling back to the `vis-job` page's own text when there is no link or it gives none. Same errors as `fetch_text` raises today, as exceptions the route maps. Do not edit `api.py`; the planner rewires the route. +4. `roles.js`: the reader's link-out and the "Also posted" links prefer `posting_url` over `original_url`, through `safeUrl`. + +## After the merge + +The planner rewires `fetch_text`, runs `app companies size --missing` against the live database, runs the full test suite, restarts the server on 8790 if it was running, and folds this plan into `docs/decisions.md`: the SmartRecruiters and Workable "no description" entries, the size entry under "The agents set the reach", closure, and the Jobindex on-demand text entry.