From 4cf81437a27164ee6e628a7e59d2de0e683549df Mon Sep 17 00:00:00 2001 From: Wolf Date: Wed, 30 Sep 2026 13:49:35 +0200 Subject: [PATCH] Fold the board size, two strikes, per-role text and Jobindex link-out plan into the decisions Co-Authored-By: Claude Opus 5.5 --- docs/decisions.md | 12 ++++---- docs/plan-boards-and-text.md | 57 ------------------------------------ plugins/readers/_contract.py | 2 +- plugins/readers/_runner.py | 2 +- 4 files changed, 9 insertions(+), 64 deletions(-) delete mode 100644 docs/plan-boards-and-text.md diff --git a/docs/decisions.md b/docs/decisions.md index dac09b1..fd7ede0 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -86,13 +86,13 @@ What was decided for roledesk and the one reason for each. The longer reasoning **Discovery that finds nothing readable does not detach a board.** `discover --add` on a company that is read from a board and whose site now names only, say, Workday changes nothing and says so. A board that answers is stronger evidence than a page that names no board; `set-board` or Detach is the explicit step. -**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department; the job text is only on the per-posting endpoint, and one extra request per role would multiply a board's request count by its role count. `description` stays empty for this host. +**SmartRecruiters' board list has no description, so its text is read per role by `enrich`.** Its `/postings` endpoint gives title, location and department; the text is only on `/postings/{id}`, whose `jobAd.sections` hold `jobDescription`, `qualifications` and `additionalInformation` (joined) and `companyDescription` (left out, the same blurb on every role). Reading it for every role would multiply a board's requests by its roles, so the reader declares `text="detail"` and `enrich` reads only the ranked roles (2026-09-30; 209 open roles had no text before). **The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing. **A board request is streamed, with a 30 s read timeout and a 60 s deadline.** `_get` counts the body as it arrives and refuses it at `MAX_BYTES`, where reading `response.content` first held the whole body in memory before the check. httpx's timeouts are per phase, so a board that keeps sending a byte now and then never timed out; the deadline covers the whole request, redirects and body included, and is checked between chunks. openings.go's 10 s read timeout failed `greenhouse:mongodb` and `greenhouse:feedzai` in a pull of 2026-09-28. Measured on 2026-09-29, three runs each of those two and the five Greenhouse boards with the most open roles, with and without `pay_transparency=true` (84 requests, every one a CloudFront miss): a body of up to 10.4 MB arrives in under a second once it starts, and the first byte waited up to 16.6 s, with the three slowest at 16.6, 14.4 and 10.9 s. The flag made no measurable difference; the slow first byte is the origin's, with or without it. A read may wait 30 s, about twice the worst seen, and the deadline is 60 s, which leaves a large board room on a slower link. -**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`, and `description` stays empty as for SmartRecruiters rather than costing one request per role. +**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by `enrich` for the ranked roles as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id. **Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse. @@ -110,9 +110,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning **Every source is a reader module that declares what it can do.** The 22 hosts and the 3 search providers are modules in `plugins/readers/`, each with `READS = Reads(...)`: board or search, whole or paged listing, where the text is (listing, excerpt, detail, none), how completeness is known (stated, end, whole, never), page size and the host's ceiling, the filters it takes per dimension (none, one value a request, many, host ids through `facets`), whether it states one place, several, or one posting per place, which fields it states and whether its date is trusted, a key, a provisional title, whether an unknown slug answers an empty board, and the robots decision. Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `problems()` checks the declaration against the functions at import, and `tests/test_readers_contract.py` runs every module against fixtures, so a new reader that drops a place or a teaser fails there. The contract is the docstring of `plugins/readers/__init__.py`; the types are `_contract.py`. -**One runner calls every reader.** `plugins/readers/_runner.py` owns the guarded fetch, paging, one request series per filter value unioned by id, completeness, the fallback feed, 429s and failures. `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` saves its cursor and continues on the next pull; the sweep that reaches the end closes what it did not list. A host ceiling (Workday 2,000) is not swept; a filter has to get under it. +**One runner calls every reader.** `plugins/readers/_runner.py` owns the guarded fetch, paging, one request series per filter value unioned by id, completeness, the fallback feed, 429s and failures. `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` saves its cursor and continues on the next pull, without reading page 1 again, so a new role on page 1 waits one sweep; the owner accepted that over reading the head every pull. A host ceiling (Workday 2,000) is not swept; a filter has to get under it. -**The agents set the reach of a pull.** `app/jobs/reach.py` computes it once per pull: the union of the enabled agents' `any` gates on country, town (as its country) and remote, or everything when one agent has none. Fetching has two filters, countries and role types, and a company has an override for each. A company set to **All countries** (`companies.all_countries`, migration 24) is read with the wide reach: the union of the enabled agents' `require` reach, Europe today, and the union of the `any` gates, so the override never reads less than the board would without it. An agent with no `require` gate on place makes the wide reach everywhere. For a company the owner cares about, a role in Freiburg may be open to remote, or worth asking about for Denmark, while Microsoft Germany is not wanted. The overrides are separate from favourite and liked, and global rather than per agent, because they decide what is fetched and fetching is shared. They matter only for a board of 300 roles or more on a host that filters, so a small company is affected once it grows past that. The Companies list states both halves per board, e.g. "follows agents: DK, SE, remote · all roles" or "all countries (Europe) · follows agents' terms: engineer, product, data", or why the board is read whole. Favourite (affinity 2) affects scoring only, where it skips the `any` gates; it used to widen the fetch to its agent's `require` reach, and migration 21 tagged every company that was a favourite of any agent so nothing read before is lost. That tag, first named All roles in `companies.all_roles`, also replaced a board's `{"reach": "all"}` param, which read everywhere: migration 21 tagged those companies and logged each such board, since no everywhere level is left. Migration 24 renamed the tag to All countries and gave its companies (Mistral AI and OpenAI on live) All roles as well, since the owner wanted their every role. A board's `params` override the agents' reach and All countries. Boards under 300 roles, and hosts with a one-country filter over more than 8 countries, are read whole. SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday and Oracle filter at the source; countries and remote go as separate series, because the hosts intersect two facets. A complete filtered read closes roles outside the filter, except roles with a vote or an application; a filtered read, or one from a host whose unknown slug answers empty, that lists under 10% of the last read with the same filter counts as failed. +**A board swept across pulls closes a role only when two finished sweeps in a row missed it.** Roles closing above the saved offset between pulls shift later roles back past it, so a finished sweep misses roles that are still open, and the stated-total check can still pass because `seen` holds ids that closed meanwhile. A complete read that finished a sweep is stamped `ats:swept:`; the settle closes the open roles missing now that are also in `ats:missing:`, the ids the previous finished sweep missed, and keeps the rest there. A complete read that was not swept closes as before and clears the record. A role the host closes takes up to two sweeps to close. + +**The agents set the reach of a pull.** `app/jobs/reach.py` computes it once per pull: the union of the enabled agents' `any` gates on country, town (as its country) and remote, or everything when one agent has none. Fetching has two filters, countries and role types, and a company has an override for each. A company set to **All countries** (`companies.all_countries`, migration 24) is read with the wide reach: the union of the enabled agents' `require` reach, Europe today, and the union of the `any` gates, so the override never reads less than the board would without it. An agent with no `require` gate on place makes the wide reach everywhere. For a company the owner cares about, a role in Freiburg may be open to remote, or worth asking about for Denmark, while Microsoft Germany is not wanted. The overrides are separate from favourite and liked, and global rather than per agent, because they decide what is fetched and fetching is shared. They matter only for a board of 300 roles or more on a host that filters, so a small company is affected once it grows past that. The Companies list states both halves per board, e.g. "follows agents: DK, SE, remote · all roles" or "all countries (Europe) · follows agents' terms: engineer, product, data", or why the board is read whole. Favourite (affinity 2) affects scoring only, where it skips the `any` gates; it used to widen the fetch to its agent's `require` reach, and migration 21 tagged every company that was a favourite of any agent so nothing read before is lost. That tag, first named All roles in `companies.all_roles`, also replaced a board's `{"reach": "all"}` param, which read everywhere: migration 21 tagged those companies and logged each such board, since no everywhere level is left. Migration 24 renamed the tag to All countries and gave its companies (Mistral AI and OpenAI on live) All roles as well, since the owner wanted their every role. A board's `params` override the agents' reach and All countries. Boards under 300 roles, and hosts with a one-country filter over more than 8 countries, are read whole. A board's size (`ats:size:`, with `ats:sizechecked:`) is set once, not by every read: `set_board` sizes a new board with `ats.board_size` (a stated total from one unfiltered page 1, else a whole read), a board with no size is sized in its batch before its reach is chosen, and Recheck on the Companies row, `app companies size ` and `--missing` size it again. Before, a new board had no size and was read whole on its first pull: Infineon read 999 roles worldwide and cut at page 100. A board whose reader cannot filter is sized by its whole read instead of a second request. On 2026-09-30 `--missing` sized 38 boards the old open-role fallback had sized, among them Bosch 4,830, Verisure 3,447, Microsoft 2,345 and NVIDIA at Workday's 2,000. For a Workday or Oracle site, which filters by its own location ids, the Companies line names a country the site does not list and a missing remote option ("follows agents: DK, remote · SE not on this site"), and "nothing asked" when it lists none, where it used to name what was meant to be asked. SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday and Oracle filter at the source; countries and remote go as separate series, because the hosts intersect two facets. A complete filtered read closes roles outside the filter, except roles with a vote or an application; a filtered read, or one from a host whose unknown slug answers empty, that lists under 10% of the last read with the same filter counts as failed. **A role has a list of places, resolved against a gazetteer.** Readers kept one place of several, joined them, or the host posted a copy per place. A reader now returns every place and never multiplies a row. `places` (city, region, country, continent, with `place_includes` for Øresund across two countries) and `place_aliases` are seeded from the dictionaries that were in code; `role_places` links each place of a role, and an unknown text is listed by `unresolved_places` until an alias links it. The Place filter and the gates join over the tree, which halved their time and dropped the `EU` fallback. Copies of one posting per place, same host, company and title and texts at least 30% alike, fold into one role that keeps each copy's place and link; the rule applies to every host, since the 5,024 such roles came mostly from hosts that do not declare copies. @@ -254,7 +256,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning **A company affinity change or an agent save queues a `score` that will see it.** `POST /api/companies/{key}/affinity` and `POST /api/score`, which the Search agents page calls after a save, answer with the id of that job, and the page waits on it. `score` is `solo`, so the queue dedupes a plain request into the live job keyed `score`; a queued one has not read anything yet and sees the change, but a running one read the old state, and deduping into it reported "Saved and scored" for a score that did not include the save. So a request during a run queues one follow-up with `depends_on` the running job, `after="done"` and the key `score:after:`, which makes ten clicks during one run one extra pass, and a queued follow-up is the answer until it starts. Without a score after it, an affinity change is invisible in Best match until the next 6-hourly pull, and an unmuted company's roles have no score rows until then. -**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` scrapes `original_url` only when the read page asks for it, because following Jobindex's redirect for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair. +**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` (`app/jobs/posting.py`) fetches it only when the read page asks, because following Jobindex's link for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair. A Jobindex role's `vis-job` page is a short ad with a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's `jobannonce`); the fetch keeps that link as `roles.posting_url` (migration 26, never written by a pull), reads the text from it, and falls back to the `vis-job` page's own text. The reader's link-out prefers `posting_url`, so it leads to the full posting once the text was fetched. **Summarising is bounded to the top roles of each agent, by that agent's own `limits`, and off unless switched on.** The pull's `summarise` stage only considers roles with no summary yet that are in the top `summarise_top` (default 30) of an enabled agent's scores and at `summarise_min` (default 0.65, about 35 roles for the starter agent, stricter than extract's since a summary is the long answer), so the cost stays flat whatever the backlog of open roles is. The bounds live on the agent, beside its gates and weights, for the same reason as extract's: a wide monitoring search and a narrow hunt want different depths, and the agent is where the difference is already expressed. Whether the stage runs is the switch under Runs, or `summarise.enabled` in config.yaml until the switch is used, and both default to off like `extract`; the read page's Summarise button asks for one role's summary either way. diff --git a/docs/plan-boards-and-text.md b/docs/plan-boards-and-text.md deleted file mode 100644 index 3568b8a..0000000 --- a/docs/plan-boards-and-text.md +++ /dev/null @@ -1,57 +0,0 @@ -# Plan: board size, two strikes, text for SmartRecruiters and Workable, Jobindex link-out - -From the architecture walkthrough of 2026-09-30. The owner's decisions: - -- A board's size is decided once, when the board is added, and changed only by a Recheck. It is no longer rewritten by every unfiltered read. A board with no size yet is sized on its first pull before its reach is chosen, so a new big board is not read whole worldwide first (Infineon's first pull read 999 roles across the world and cut at page 100). -- A host that states no total on page 1 (Teamtailor, HR Manager, `complete="end"`) is read to the end once to size it. -- A board swept across pulls closes a role only when two finished sweeps in a row missed it. Offset paging drifts when roles close between pulls, so one sweep can miss a role that is still open. -- No head-then-tail read and no sweep display: a new role on page 1 of a swept board waits one sweep. -- Big boards on hosts that cannot filter (Greenhouse, Ashby, Personio) are still stored whole. Gates keep their out-of-reach roles away from enrich, extract and summarise, so they cost prep and embedding only. -- SmartRecruiters and Workable get per-role text through `detail`, read by `enrich` for the top roles. Both per-role endpoints were checked live on 2026-09-30: `api.smartrecruiters.com/v1/companies//postings/` has `jobAd.sections` (about 4,500 characters on a Delivery Hero role), `apply.workable.com/api/v2/accounts//jobs/` has `description`, `requirements` and `benefits` (about 7,000 on a Hugging Face role). 209 and 145 open roles have no text today. -- Jobindex's `vis-job` page carries a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's own `jobannonce` page). When the owner fetches a Jobindex role's text, that link is resolved, the text is read from it, and the reader's link-out goes there. On demand only, since following it is a click in Jobindex's statistics. -- The Companies line says when a Workday or Oracle site does not know a country or has no remote option, instead of naming what was meant to be asked. - -## Work packages - -Each package owns its files; no two packages edit the same file. The planner rewires `fetch_text` in `app/web/api.py` to WP D's function after both merge, and folds this plan into `docs/decisions.md`. - -### WP A: board size and the Companies line (Opus) - -Owns `app/jobs/reach.py`, `app/jobs/sources.py`, `plugins/ats.py`, `app/web/api.py`, `app/static/companies.js`, `app/cli.py`, and the tests `tests/test_reach.py`, `tests/test_web_reach.py`, `tests/test_sources.py`, `tests/test_companies.py`, `tests/test_ats.py`, `tests/test_web.py`, plus new test files of its own. - -1. Size lives in `state` as today, `ats:size:`, with `ats:sizechecked:` (ISO date). No migration. -2. `plugins/ats.py` gets `board_size(target) -> int`: a whole host's listing length; a paged host with `complete="stated"` the total on page 1 (call the module's `page` once with an unfiltered `Ask()`); a host with `complete="end"` or no stated total, read to the end through `_runner.read_listing` with `max_pages` high enough for the host (a Teamtailor board is rarely over 1,000). No filter, no terms. -3. `set_board` with `check=True` sizes the board with `board_size` instead of `fetch_board`'s whole read, which read Infineon to page 100. It still refuses a board that does not answer, and still returns a role count for the CLI and the route. -4. An ordinary pull stops writing `ats:size`. A board with no size gets one in its batch before it is read: `Plan.for_subscription` hands such a board the agents' reach with `"unsized": true` and `"small": SMALL`; `ATSSource._board` sees `unsized`, calls `board_size`, writes both state keys, and reads whole when the size is under `small`, else with the reach it was given. `reach.sizes` drops its fallback to open-role counts once every existing board has `ats:size` (all 287 live boards have one today except those `why` calls `unknown`). -5. Recheck: `POST /api/sources/{id}/size` runs `board_size`, writes both keys, queues a pull like the other routes that change what is read, and answers the new size and the Companies line. `app companies size ` does the same from the CLI, and `app companies size --missing` sizes every board without one. `/api/companies` returns `size` and `size_checked`. The Companies row shows "999 roles · checked 30 Sep" beside the reach and a Recheck button, in the pattern of the row's existing actions. -6. `reach.line` reads the facets the runner keeps (`ats:facets:`, `{"checked", "facets"}`) for a reader whose country or remote filter is `ids`, and names what the site does not know: "follows agents: DK · SE not on this site · no remote option · all roles". When every value is unknown: "site lists none of DK, SE: nothing asked". With no facets kept yet, the line is as today. - -### WP B: two strikes for swept boards (Sonnet) - -Owns `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_runner.py`, `tests/test_pull.py`, plus new test files of its own. - -1. `Listing` gets `swept: bool`, true when a complete listing finished a sweep carried over from an earlier pull (`read_listing` resumed from a stored sweep). `stamp` writes `:swept:` as "1" or "". -2. `close_boards`: for a complete, unnarrowed board read with `swept` set, the roles to close are the board's open roles missing from the ids now and also in `ats:missing:`, the ids the previous finished sweep missed. It writes the ids missed now to `ats:missing:`. A filtered swept read keeps its exceptions (votes, applications). A complete read that was not swept closes as today and clears `ats:missing:`. -3. Nothing else in closure changes: failing for `STALE_DAYS`, switched off, searches. - -### WP C: per-role text for SmartRecruiters and Workable (Sonnet) - -Owns `plugins/readers/smartrecruiters.py`, `plugins/readers/workable.py`, `tests/test_ats_smartrecruiters.py`, `tests/test_ats_workable.py`, and new fixtures under `tests/fixtures/ats/` for the two detail answers. - -1. Each gets `detail(get, slug, row)` and `READS.text="detail"`. SmartRecruiters joins `jobDescription`, `qualifications` and `additionalInformation` (HTML to text as the other readers do; leave `companyDescription` out). Workable joins `description`, `requirements` and `benefits`. Return only `DETAIL_FIELDS`; add places, remote or employment only where the answer states them more fully than the list. -2. The row needs what the detail URL needs: SmartRecruiters' posting id is the `source_id`; Workable's `shortcode` must be kept in the row if it is not already. -3. `tests/test_readers_contract.py` must pass with the new declarations; its fixtures are owned here for these two readers only. -4. Check both against the live endpoints once, with the fetch the runner hands over (`ats._getter`), and record the date in the module docstring as the other readers do. - -### WP D: Jobindex link-out (Sonnet) - -Owns `plugins/readers/jobindex.py`, `tests/test_readers_jobindex.py`, a new `app/jobs/posting.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/models.py`, `app/static/roles.js`, `tests/test_storage.py`, and new test files of its own. - -1. Migration 26: `roles.posting_url TEXT NOT NULL DEFAULT ''`. `upsert_role` never writes it, so a pull keeps it. -2. `plugins/readers/jobindex.py` gets `posting_link(html: str) -> str`: the "Se jobbet" `href` of a `vis-job` page, absolute, `''` when there is none. Fixtures from a saved `vis-job` page of each kind (employer site, The Hub, `jobannonce`). -3. `app/jobs/posting.py`: `fetch_posting(store, uid) -> Role`, what `fetch_text` in `app/web/api.py` does today, plus: for a Jobindex role, fetch `original_url`, take `posting_link`, store it as `posting_url`, and scrape the text from it, falling back to the `vis-job` page's own text when there is no link or it gives none. Same errors as `fetch_text` raises today, as exceptions the route maps. Do not edit `api.py`; the planner rewires the route. -4. `roles.js`: the reader's link-out and the "Also posted" links prefer `posting_url` over `original_url`, through `safeUrl`. - -## After the merge - -The planner rewires `fetch_text`, runs `app companies size --missing` against the live database, runs the full test suite, restarts the server on 8790 if it was running, and folds this plan into `docs/decisions.md`: the SmartRecruiters and Workable "no description" entries, the size entry under "The agents set the reach", closure, and the Jobindex on-demand text entry. diff --git a/plugins/readers/_contract.py b/plugins/readers/_contract.py index 8192cb5..4dfc671 100644 --- a/plugins/readers/_contract.py +++ b/plugins/readers/_contract.py @@ -165,7 +165,7 @@ class Ask(_Frozen): `terms` are the agents' fetch terms a board is narrowed by (see `app/jobs/reach.py`). The runner makes one series per term, each with that term as `query`, so a reader only ever reads `query`. A read with - terms is never the whole board: it sets no size and closes nothing.""" + terms is never the whole board: it closes nothing.""" countries: tuple[str, ...] = () places: tuple[str, ...] = () diff --git a/plugins/readers/_runner.py b/plugins/readers/_runner.py index a4272c6..ebf070c 100644 --- a/plugins/readers/_runner.py +++ b/plugins/readers/_runner.py @@ -398,7 +398,7 @@ def filtered(ask: Ask) -> bool: def narrowed(module: ModuleType, ask: Ask) -> bool: """Whether `ask` reads a board by words: fetch terms, or a board's own query. Such a read is never the whole board, nor the board within its - places, so it sets no size and closes nothing. A search's query is what + places, so it closes nothing. A search's query is what the search is, and does not count.""" board = not declared(module) or module.READS.scope == "board" return bool(ask.terms) or (board and bool(ask.query))