Fold the board size, two strikes, per-role text and Jobindex link-out plan into the decisions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 13:49:35 +02:00
1 parent 699bf81e3b
commit 4cf81437a2
4 files changed
+9 -64

No files matched your search

+7 -5
View File
@@ -86,13 +86,13 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Discovery that finds nothing readable does not detach a board.** `discover --add` on a company that is read from a board and whose site now names only, say, Workday changes nothing and says so. A board that answers is stronger evidence than a page that names no board; `set-board` or Detach is the explicit step.
**SmartRecruiters' board list has no description.** Its `/postings` endpoint gives title, location and department; the job text is only on the per-posting endpoint, and one extra request per role would multiply a board's request count by its role count. `description` stays empty for this host.
**SmartRecruiters' board list has no description, so its text is read per role by `enrich`.** Its `/postings` endpoint gives title, location and department; the text is only on `/postings/{id}`, whose `jobAd.sections` hold `jobDescription`, `qualifications` and `additionalInformation` (joined) and `companyDescription` (left out, the same blurb on every role). Reading it for every role would multiply a board's requests by its roles, so the reader declares `text="detail"` and `enrich` reads only the ranked roles (2026-09-30; 209 open roles had no text before).
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
**A board request is streamed, with a 30 s read timeout and a 60 s deadline.** `_get` counts the body as it arrives and refuses it at `MAX_BYTES`, where reading `response.content` first held the whole body in memory before the check. httpx's timeouts are per phase, so a board that keeps sending a byte now and then never timed out; the deadline covers the whole request, redirects and body included, and is checked between chunks. openings.go's 10 s read timeout failed `greenhouse:mongodb` and `greenhouse:feedzai` in a pull of 2026-09-28. Measured on 2026-09-29, three runs each of those two and the five Greenhouse boards with the most open roles, with and without `pay_transparency=true` (84 requests, every one a CloudFront miss): a body of up to 10.4 MB arrives in under a second once it starts, and the first byte waited up to 16.6 s, with the three slowest at 16.6, 14.4 and 10.9 s. The flag made no measurable difference; the slow first byte is the origin's, with or without it. A read may wait 30 s, about twice the worst seen, and the deadline is 60 s, which leaves a large board room on a slower link.
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`, and `description` stays empty as for SmartRecruiters rather than costing one request per role.
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by `enrich` for the ranked roles as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id.
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse.
@@ -110,9 +110,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Every source is a reader module that declares what it can do.** The 22 hosts and the 3 search providers are modules in `plugins/readers/`, each with `READS = Reads(...)`: board or search, whole or paged listing, where the text is (listing, excerpt, detail, none), how completeness is known (stated, end, whole, never), page size and the host's ceiling, the filters it takes per dimension (none, one value a request, many, host ids through `facets`), whether it states one place, several, or one posting per place, which fields it states and whether its date is trusted, a key, a provisional title, whether an unknown slug answers an empty board, and the robots decision. Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `problems()` checks the declaration against the functions at import, and `tests/test_readers_contract.py` runs every module against fixtures, so a new reader that drops a place or a teaser fails there. The contract is the docstring of `plugins/readers/__init__.py`; the types are `_contract.py`.
**One runner calls every reader.** `plugins/readers/_runner.py` owns the guarded fetch, paging, one request series per filter value unioned by id, completeness, the fallback feed, 429s and failures. `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` saves its cursor and continues on the next pull; the sweep that reaches the end closes what it did not list. A host ceiling (Workday 2,000) is not swept; a filter has to get under it.
**One runner calls every reader.** `plugins/readers/_runner.py` owns the guarded fetch, paging, one request series per filter value unioned by id, completeness, the fallback feed, 429s and failures. `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` saves its cursor and continues on the next pull, without reading page 1 again, so a new role on page 1 waits one sweep; the owner accepted that over reading the head every pull. A host ceiling (Workday 2,000) is not swept; a filter has to get under it.
**The agents set the reach of a pull.** `app/jobs/reach.py` computes it once per pull: the union of the enabled agents' `any` gates on country, town (as its country) and remote, or everything when one agent has none. Fetching has two filters, countries and role types, and a company has an override for each. A company set to **All countries** (`companies.all_countries`, migration 24) is read with the wide reach: the union of the enabled agents' `require` reach, Europe today, and the union of the `any` gates, so the override never reads less than the board would without it. An agent with no `require` gate on place makes the wide reach everywhere. For a company the owner cares about, a role in Freiburg may be open to remote, or worth asking about for Denmark, while Microsoft Germany is not wanted. The overrides are separate from favourite and liked, and global rather than per agent, because they decide what is fetched and fetching is shared. They matter only for a board of 300 roles or more on a host that filters, so a small company is affected once it grows past that. The Companies list states both halves per board, e.g. "follows agents: DK, SE, remote · all roles" or "all countries (Europe) · follows agents' terms: engineer, product, data", or why the board is read whole. Favourite (affinity 2) affects scoring only, where it skips the `any` gates; it used to widen the fetch to its agent's `require` reach, and migration 21 tagged every company that was a favourite of any agent so nothing read before is lost. That tag, first named All roles in `companies.all_roles`, also replaced a board's `{"reach": "all"}` param, which read everywhere: migration 21 tagged those companies and logged each such board, since no everywhere level is left. Migration 24 renamed the tag to All countries and gave its companies (Mistral AI and OpenAI on live) All roles as well, since the owner wanted their every role. A board's `params` override the agents' reach and All countries. Boards under 300 roles, and hosts with a one-country filter over more than 8 countries, are read whole. SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday and Oracle filter at the source; countries and remote go as separate series, because the hosts intersect two facets. A complete filtered read closes roles outside the filter, except roles with a vote or an application; a filtered read, or one from a host whose unknown slug answers empty, that lists under 10% of the last read with the same filter counts as failed.
**A board swept across pulls closes a role only when two finished sweeps in a row missed it.** Roles closing above the saved offset between pulls shift later roles back past it, so a finished sweep misses roles that are still open, and the stated-total check can still pass because `seen` holds ids that closed meanwhile. A complete read that finished a sweep is stamped `ats:swept:<target>`; the settle closes the open roles missing now that are also in `ats:missing:<target>`, the ids the previous finished sweep missed, and keeps the rest there. A complete read that was not swept closes as before and clears the record. A role the host closes takes up to two sweeps to close.
**The agents set the reach of a pull.** `app/jobs/reach.py` computes it once per pull: the union of the enabled agents' `any` gates on country, town (as its country) and remote, or everything when one agent has none. Fetching has two filters, countries and role types, and a company has an override for each. A company set to **All countries** (`companies.all_countries`, migration 24) is read with the wide reach: the union of the enabled agents' `require` reach, Europe today, and the union of the `any` gates, so the override never reads less than the board would without it. An agent with no `require` gate on place makes the wide reach everywhere. For a company the owner cares about, a role in Freiburg may be open to remote, or worth asking about for Denmark, while Microsoft Germany is not wanted. The overrides are separate from favourite and liked, and global rather than per agent, because they decide what is fetched and fetching is shared. They matter only for a board of 300 roles or more on a host that filters, so a small company is affected once it grows past that. The Companies list states both halves per board, e.g. "follows agents: DK, SE, remote · all roles" or "all countries (Europe) · follows agents' terms: engineer, product, data", or why the board is read whole. Favourite (affinity 2) affects scoring only, where it skips the `any` gates; it used to widen the fetch to its agent's `require` reach, and migration 21 tagged every company that was a favourite of any agent so nothing read before is lost. That tag, first named All roles in `companies.all_roles`, also replaced a board's `{"reach": "all"}` param, which read everywhere: migration 21 tagged those companies and logged each such board, since no everywhere level is left. Migration 24 renamed the tag to All countries and gave its companies (Mistral AI and OpenAI on live) All roles as well, since the owner wanted their every role. A board's `params` override the agents' reach and All countries. Boards under 300 roles, and hosts with a one-country filter over more than 8 countries, are read whole. A board's size (`ats:size:<target>`, with `ats:sizechecked:<target>`) is set once, not by every read: `set_board` sizes a new board with `ats.board_size` (a stated total from one unfiltered page 1, else a whole read), a board with no size is sized in its batch before its reach is chosen, and Recheck on the Companies row, `app companies size <key>` and `--missing` size it again. Before, a new board had no size and was read whole on its first pull: Infineon read 999 roles worldwide and cut at page 100. A board whose reader cannot filter is sized by its whole read instead of a second request. On 2026-09-30 `--missing` sized 38 boards the old open-role fallback had sized, among them Bosch 4,830, Verisure 3,447, Microsoft 2,345 and NVIDIA at Workday's 2,000. For a Workday or Oracle site, which filters by its own location ids, the Companies line names a country the site does not list and a missing remote option ("follows agents: DK, remote · SE not on this site"), and "nothing asked" when it lists none, where it used to name what was meant to be asked. SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday and Oracle filter at the source; countries and remote go as separate series, because the hosts intersect two facets. A complete filtered read closes roles outside the filter, except roles with a vote or an application; a filtered read, or one from a host whose unknown slug answers empty, that lists under 10% of the last read with the same filter counts as failed.
**A role has a list of places, resolved against a gazetteer.** Readers kept one place of several, joined them, or the host posted a copy per place. A reader now returns every place and never multiplies a row. `places` (city, region, country, continent, with `place_includes` for Øresund across two countries) and `place_aliases` are seeded from the dictionaries that were in code; `role_places` links each place of a role, and an unknown text is listed by `unresolved_places` until an alias links it. The Place filter and the gates join over the tree, which halved their time and dropped the `EU` fallback. Copies of one posting per place, same host, company and title and texts at least 30% alike, fold into one role that keeps each copy's place and link; the rule applies to every host, since the 5,024 such roles came mostly from hosts that do not declare copies.
@@ -254,7 +256,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**A company affinity change or an agent save queues a `score` that will see it.** `POST /api/companies/{key}/affinity` and `POST /api/score`, which the Search agents page calls after a save, answer with the id of that job, and the page waits on it. `score` is `solo`, so the queue dedupes a plain request into the live job keyed `score`; a queued one has not read anything yet and sees the change, but a running one read the old state, and deduping into it reported "Saved and scored" for a score that did not include the save. So a request during a run queues one follow-up with `depends_on` the running job, `after="done"` and the key `score:after:<id>`, which makes ten clicks during one run one extra pass, and a queued follow-up is the answer until it starts. Without a score after it, an affinity change is invisible in Best match until the next 6-hourly pull, and an unmuted company's roles have no score rows until then.
**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` scrapes `original_url` only when the read page asks for it, because following Jobindex's redirect for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair.
**A search role's posting text is fetched on demand, not in the pull.** `POST /api/roles/{uid}/text` (`app/jobs/posting.py`) fetches it only when the read page asks, because following Jobindex's link for every role during a pull would count as a click in Jobindex's own statistics; one click for a posting somebody actually opened is fair. A Jobindex role's `vis-job` page is a short ad with a "Se jobbet" link to the posting itself (the employer's site, The Hub, or Jobindex's `jobannonce`); the fetch keeps that link as `roles.posting_url` (migration 26, never written by a pull), reads the text from it, and falls back to the `vis-job` page's own text. The reader's link-out prefers `posting_url`, so it leads to the full posting once the text was fetched.
**Summarising is bounded to the top roles of each agent, by that agent's own `limits`, and off unless switched on.** The pull's `summarise` stage only considers roles with no summary yet that are in the top `summarise_top` (default 30) of an enabled agent's scores and at `summarise_min` (default 0.65, about 35 roles for the starter agent, stricter than extract's since a summary is the long answer), so the cost stays flat whatever the backlog of open roles is. The bounds live on the agent, beside its gates and weights, for the same reason as extract's: a wide monitoring search and a narrow hunt want different depths, and the agent is where the difference is already expressed. Whether the stage runs is the switch under Runs, or `summarise.enabled` in config.yaml until the switch is used, and both default to off like `extract`; the read page's Summarise button asks for one role's summary either way.