Merge the ranking core: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text, the feed kept to the fetch gate, saved filters backend, migration 32 (wp-core)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-10-04 15:18:35 +02:00
commit 558189d747
37 files changed
+1314 -788

No files matched your search

+20 -20
View File
@@ -4,7 +4,7 @@ This file lists every source roledesk reads roles from, how each is read and wha
The code has two plugins: `ats` (`plugins/ats.py`) reads the board of a company someone listed, with one module per host in `plugins/readers/`; `search` (`plugins/search.py`) runs saved queries on sites that list roles across companies. Below, a module of `ats` is a "host" and a module of `search` is a "provider". A company's list of roles on a host is its "board"; a board is attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `enrich:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
## Flow
@@ -26,13 +26,13 @@ The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section say
| `sweep` | `stamp`, the cursor of a read cut at `MAX_PAGES` | `_runner.read` on the next pull |
| `count` | `stamp`, the roles the last read listed; reset by Recheck | `_runner.read`, the 10% guard |
| `missing` | `close_boards` (`_close_swept`), the ids the last finished sweep missed | the next finished sweep |
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `enrich`, the guard's escape after 21 days |
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `fetch_text`, the guard's escape after 21 days |
| `facets` | `_runner.facets_for`, kept one day | `reach.site_facets`, the Companies line |
| `size`, `sizechecked` | `ats.keep_size`: `set_board`, Recheck, `app companies size`, or the first read of an unsized board | `reach.sizes`, which decides whether the board is filtered |
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`enrich`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `enrich.plan` through `detail_due` |
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`fetch_text`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `fetch_text.plan` through `detail_due` |
4. `close_boards`, the settle of the pull, closes what a complete read did not list, closes a board or search that failed for `STALE_DAYS`, releases the run, and raises one `pull.settled` notice in the Problems log when a target did not answer or was read incomplete, or a board closed more than 50 roles or more than half of its open ones.
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score`, `enrich` (reads the role pages of the best-ranked roles without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `summarise`. Extract and summarise are switches under Runs; enrich is on by default.
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score` (which writes each agent's cut), `fetch_text` (reads the role pages of the roles in the cut without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `judge`, `summarise`. Extract, judge and summarise are switches under Runs; fetch text is on by default.
## The three ways a source is read
@@ -40,23 +40,23 @@ The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section say
**Paged.** The list arrives in pages of a fixed size (`READS.page_size`), read one after another `PAGE_DELAY` (0.5 s) apart, at most `MAX_PAGES` (100) pages a pull. The read is complete when the number of distinct roles read equals the total the host states (`complete="stated"`), or, for a host that states none, when the read reaches an empty page (`complete="end"`); a Jobindex search is never complete (`complete="never"`). A board of more than 100 pages is swept: each pull stores what it read and the next continues from the saved cursor, and the read is complete when a sweep reaches the end. Roles closing between pulls shift an offset-paged host's later roles back past the cursor, so a role of a swept board closes only when two finished sweeps in a row missed it. A host with a `ceiling` (Workday 2,000) answers nothing past it: a read that reaches it is incomplete, is not swept and closes nothing, so a filter must bring the board under it.
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The `enrich` stage reads those pages after scoring (`app/jobs/enrich.py`):
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The fetch text stage reads those pages after scoring (`app/jobs/fetch_text.py`):
- It takes open roles without text that an enabled agent with a prompt ranks ungated within its best `enrich.top` (500) at `enrich.min_score` (0.5) or above, best score first.
- It reads at most `enrich.per_board` (200) roles per board and `enrich.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
- A read that gave no text is tried again after `enrich.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
- It runs after every pull, after a saved agent and after a star change, not after a vote, and it can be switched off under Runs.
- It takes open roles without text in the cut of an enabled agent (the best by fit and by CV closeness, and the liked and saved, see `app/jobs/score.py`), best final score first.
- It reads at most `fetch_text.per_board` (200) roles per board and `fetch_text.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
- A read that gave no text is tried again after `fetch_text.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
- It runs after every pull, after a saved agent, an affinity change or an override switched, not after a vote, and it can be switched off under Runs.
A role below the bar for every agent is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles that rank do. Four consequences follow from the code:
A role in no agent's cut is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles in the cut do. Four consequences follow from the code:
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow `enrich` encode it again.
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow fetch text encode it again.
- A role page may fill only `_runner.DETAIL_FIELDS`: description, location and places, teaser, remote, country, employment, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
- Completeness, and so closing, is decided by the list alone. A board of 800 roles closes its roles from the first pull.
- SmartRecruiters and Workable list their roles in pages and belong here too, since their list has no text.
## Overview
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles `enrich` has read.
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles fetch text has read.
| Source | Boards | Open roles | How it is read | Roles per request | Description | Pay | Limit that matters most |
|---|---|---|---|---|---|---|---|
@@ -95,20 +95,20 @@ Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open
| Limit | Value | What it means |
|---|---|---|
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); enrich is on by default. |
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); fetch text is on by default. |
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. Enrich reads one batch per board on the same pool. One pull runs at a time. |
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request | The same for boards, role pages and searches. A slow board fails for this pull and is asked again next pull. |
| Pages per board | 100 a pull | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
| Wait | 0.5 s before every page after the first, for boards and searches alike, and before every role page | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
| Per-role reads | `enrich`, after scoring: 200 per board and 1,000 per run, ranked roles first | See "List, then each role". Roles the agents do not rank are not read. |
| Per-role reads | `fetch_text`, after scoring: 200 per board and 1,000 per run, the roles in the agents' cut | See "List, then each role". Roles in no cut are not read. |
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing, and so does a read that asked nothing. A role listed again reopens. A board switched off closes its roles at once. A filtered read keeps roles with a vote or an application, and remote roles when the host could not be asked for remote; a read by fetch terms closes only roles not listed for 21 days. |
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. A Jobindex search is never complete, so its roles close by the 21 days only. A search failing for 21 days closes its roles. |
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, a row that would not store, and a read that lists under 10% of what the same read listed last time (`SUSPECT_SHARE`) once that was at least `SUSPECT_MIN`, 10. A search that fails is recorded the same way and its row says so. |
| A board that fails for 21 days | all its open roles close at the next pull, and a suspect answer that has stood that long is accepted | The next answer reopens what it lists. `STALE_DAYS` is the one 21. |
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next enrich run | Nothing is closed on a list cut short by a 429. |
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next fetch text run | Nothing is closed on a list cut short by a 429. |
How far a board is read: a pull asks each board of 300 roles or more for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of once per country (33 codes today), unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application. A board's size is set once, when it is added, on Recheck or by `app companies size`, and a pull does not rewrite it; a board with no size yet is sized first in its batch and then read as its size says, and a board whose reader cannot narrow the reach is read whole and sized by that read. A host that cannot filter a dimension ignores it: SmartRecruiters, Phenom, SuccessFactors and Eightfold have no remote filter, so their reach asks for countries only.
@@ -313,7 +313,7 @@ What no source does:
## Hosts read as list, then each role
For every host below the pull reads the list only; the description arrives when `enrich` reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
### SmartRecruiters
@@ -323,7 +323,7 @@ For every host below the pull reads the list only; the description arrives when
| Companies | 11 boards: Bosch (about 4,800 roles on the site, 13 open in reach), Delivery Hero, Wise |
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
| Requests per pull | one series per country of the reach (two for Bosch: DK and SE), each roles / 100, rounded up; a board read whole is roles / 100 |
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by `enrich` for ranked roles only: 1 of 343 open roles has text |
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only: 1 of 343 open roles has text |
| Date | `releasedDate`; trusted |
| Place and country | `fullLocation`, else city and country, with empty parts removed; the host's `country` is handed on |
| Remote | `location.remote`, true or false, as the host's word |
@@ -333,7 +333,7 @@ For every host below the pull reads the list only; the description arrives when
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
| Limitations | 100 pages a pull; a larger board is swept across pulls. Whether the identifier is case-sensitive is unsettled: the module docstring says it is, `docs/decisions.md` says it is not (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until `enrich` has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until fetch text has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
### Workable
@@ -343,7 +343,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
| Companies | 8 boards: Hugging Face, Kahoot!, Starling Bank |
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
| Requests per pull | roles / 10, rounded up, 0.5 s apart; the largest board has 50 roles, so 5 |
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by `enrich` for ranked roles only: 1 of 157 open roles has text |
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only: 1 of 157 open roles has text |
| Date | `published`; trusted |
| Place and country | every `locations` entry as "city, region, country", each remote when the role is; `location` when there are none |
| Remote | `workplace` (remote, hybrid, on_site) as the host's word, else the `remote` flag |
@@ -592,7 +592,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
| Saved search | `q` and optionally `region` (an `amt` id from the site's own search form: `REGIONS` in the module, 2 is Storkøbenhavn) and `remote` (`helt` fully remote, `delvist` partly) |
| List | the RSS feed `jobbank.dk/job/rss/`, 100 items a page; an empty page ends it. It states no total and is not sorted by date |
| Pages read | at most the module's `ceiling`, which keeps a run under about 20 pages |
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by `enrich` for the roles an agent ranks high. The listing carries none |
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by fetch text for the roles in the cut. The listing carries none |
| Date | `datePosted` from the role's page; trusted |
| Place | the region in the item's description line, and `jobLocation.addressLocality` from the page; country Denmark |
| Remote | not stated by a row; the search's `remote` filter narrows the list |
+3 -3
View File
@@ -37,8 +37,8 @@ whose reader cannot narrow that reach is read whole anyway, and sized by
that read once it is complete, rather than by a second request.
A pull reads listings only. A host that states a role's text on the role's
own page (`text="detail"`) is read there by `app/jobs/enrich.py` after
scoring, best roles first, through `read_board_details`. A search provider
own page (`text="detail"`) is read there by `app/jobs/fetch_text.py` after
scoring, for the roles in the agents' cut, through `read_board_details`. A search provider
whose reader has `detail` (Jobbank) is read the same way, under its search
target `provider:name`; `read_detail` is the one read the Fetch button makes.
@@ -435,7 +435,7 @@ class ATSSource(Source):
return f"{host}:{slug}:{source_id}"
# A host with `detail` states a role's text on the role's own page,
# which the `enrich` stage reads after scoring (`read_board_details`),
# which the `fetch_text` stage reads after scoring (`read_board_details`),
# never the pull. Its rows go out without a description, so the text
# a role's page gave stays, and a role read on its own page goes out
# without the listing's place and remote as well, so what its page
+6 -6
View File
@@ -215,11 +215,11 @@ is shown and embedded until `description` arrives. A reader that sees one
keeps it.
**Detail.** `detail(get, slug, row)` reads one role's own page and returns a
dict of the fields it found. The pull reads listings only; the `enrich`
stage (`app/jobs/enrich.py`) calls `detail` after scoring, so the requests
go where the ranking is: the open roles without text that an agent ranks
among its best `enrich.top` at `enrich.min_score` or above, best first, at
most `enrich.per_board` from one board and `enrich.per_run` in all, one
dict of the fields it found. The pull reads listings only; the `fetch_text`
stage (`app/jobs/fetch_text.py`) calls `detail` after scoring, so the
requests go where the model pass will read: the open roles without text in
the cut of an enabled agent, best first, at most `fetch_text.per_board`
from one board and `fetch_text.per_run` in all, one
batch per board through `ats.read_board_details` and
`_runner.read_details`, `_runner.PAGE_DELAY` apart, the roles never tried
before those whose read failed. Only `_runner.DETAIL_FIELDS` are taken:
@@ -230,7 +230,7 @@ any other key is left out with a debug log line, `seniority` among them.
`title`, `url`, `department` and `salary_text` cannot be filled from
`detail`: the listing rewrites them every pull. A 404 or a 410 counts as read; any
other failure is logged and retried on a later run; a read that gave no text
is retried after `enrich.retry_days`; a 429 ends the reads of that board for
is retried after `fetch_text.retry_days`; a 429 ends the reads of that board for
the run. A role read in detail is sent by later pulls without the listing's
place and remote, so what its own page said about them stays. Completeness
is decided by the listing alone.