Merge the ranking core: fetch gate overrides in scoring, the cut per agent, stages read the cut, fetch text, the feed kept to the fetch gate, saved filters backend, migration 32 (wp-core)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
558189d747
37 files changed
+1314
-788
No files matched your search
+20
-20
@@ -4,7 +4,7 @@ This file lists every source roledesk reads roles from, how each is read and wha
|
||||
|
||||
The code has two plugins: `ats` (`plugins/ats.py`) reads the board of a company someone listed, with one module per host in `plugins/readers/`; `search` (`plugins/search.py`) runs saved queries on sites that list roles across companies. Below, a module of `ats` is a "host" and a module of `search` is a "provider". A company's list of roles on a host is its "board"; a board is attached as a target `host:slug`, e.g. `workday:lego.wd103/LEGO_External`.
|
||||
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `enrich:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
|
||||
What a reader does is declared in its `READS` (`plugins/readers/_contract.py`) and listed per reader on the Sources page. The per-host sections below keep what a declaration cannot say: endpoints, quirks, examples and limits. The shared limits are constants in `plugins/readers/_runner.py` (`MAX_BYTES`, `TIMEOUTS`, `DEADLINE`, `MAX_PAGES`, `PAGE_DELAY`, `DETAIL_CAP`, `SUSPECT_SHARE`, `MAX_SERIES`), `app/jobs/pull.py` (`BATCH_SIZE`, `STALE_DAYS`), `app/jobs/reach.py` (`SMALL`) and the `fetch_text:` block of the config; the table "Rules every source shares" states their effect and is not a second place to change them.
|
||||
|
||||
## Flow
|
||||
|
||||
@@ -26,13 +26,13 @@ The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section say
|
||||
| `sweep` | `stamp`, the cursor of a read cut at `MAX_PAGES` | `_runner.read` on the next pull |
|
||||
| `count` | `stamp`, the roles the last read listed; reset by Recheck | `_runner.read`, the 10% guard |
|
||||
| `missing` | `close_boards` (`_close_swept`), the ids the last finished sweep missed | the next finished sweep |
|
||||
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `enrich`, the guard's escape after 21 days |
|
||||
| `failing` | `ats._board` and `run_batch` through `record_failure`, `{since, reason}`; removed after the next stamp. `search:failing:<target>` is the same for a search | `close_boards` (21 days), the Companies list and the search row, `fetch_text`, the guard's escape after 21 days |
|
||||
| `facets` | `_runner.facets_for`, kept one day | `reach.site_facets`, the Companies line |
|
||||
| `size`, `sizechecked` | `ats.keep_size`: `set_board`, Recheck, `app companies size`, or the first read of an unsized board | `reach.sizes`, which decides whether the board is filtered |
|
||||
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`enrich`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `enrich.plan` through `detail_due` |
|
||||
| `detailed`, `detailfailed`, `detailempty` | `ats.read_board_details` (`fetch_text`): ids read, ids whose read failed, `{id: date}` of reads that gave no text; the pull drops the ids a complete listing no longer lists | `fetch_text.plan` through `detail_due` |
|
||||
|
||||
4. `close_boards`, the settle of the pull, closes what a complete read did not list, closes a board or search that failed for `STALE_DAYS`, releases the run, and raises one `pull.settled` notice in the Problems log when a target did not answer or was read incomplete, or a board closed more than 50 roles or more than half of its open ones.
|
||||
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score`, `enrich` (reads the role pages of the best-ranked roles without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `summarise`. Extract and summarise are switches under Runs; enrich is on by default.
|
||||
5. Then, in order: `prep` (maps the hosts' words, resolves places and company aliases, folds copies with `app/dupes.py`), `embed`, `score` (which writes each agent's cut), `fetch_text` (reads the role pages of the roles in the cut without text), `prep`, `embed` and `score` again for the roles it filled (skipped when it filled none), `extract`, `score`, `judge`, `summarise`. Extract, judge and summarise are switches under Runs; fetch text is on by default.
|
||||
|
||||
## The three ways a source is read
|
||||
|
||||
@@ -40,23 +40,23 @@ The order of the stages is `CASCADES` in `app/jobs/cascade.py`; this section say
|
||||
|
||||
**Paged.** The list arrives in pages of a fixed size (`READS.page_size`), read one after another `PAGE_DELAY` (0.5 s) apart, at most `MAX_PAGES` (100) pages a pull. The read is complete when the number of distinct roles read equals the total the host states (`complete="stated"`), or, for a host that states none, when the read reaches an empty page (`complete="end"`); a Jobindex search is never complete (`complete="never"`). A board of more than 100 pages is swept: each pull stores what it read and the next continues from the saved cursor, and the read is complete when a sweep reaches the end. Roles closing between pulls shift an offset-paged host's later roles back past the cursor, so a role of a swept board closes only when two finished sweeps in a row missed it. A host with a `ceiling` (Workday 2,000) answers nothing past it: a read that reaches it is incomplete, is not swept and closes nothing, so a filter must bring the board under it.
|
||||
|
||||
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The `enrich` stage reads those pages after scoring (`app/jobs/enrich.py`):
|
||||
**List, then each role.** The list, in one request or in pages, carries title, place and mostly a date, but no text (Oracle, Phenom and Jobylon add a teaser). The text is on each role's own page (`READS.text="detail"`), which the pull does not read. The fetch text stage reads those pages after scoring (`app/jobs/fetch_text.py`):
|
||||
|
||||
- It takes open roles without text that an enabled agent with a prompt ranks ungated within its best `enrich.top` (500) at `enrich.min_score` (0.5) or above, best score first.
|
||||
- It reads at most `enrich.per_board` (200) roles per board and `enrich.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
|
||||
- A read that gave no text is tried again after `enrich.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
|
||||
- It runs after every pull, after a saved agent and after a star change, not after a vote, and it can be switched off under Runs.
|
||||
- It takes open roles without text in the cut of an enabled agent (the best by fit and by CV closeness, and the liked and saved, see `app/jobs/score.py`), best final score first.
|
||||
- It reads at most `fetch_text.per_board` (200) roles per board and `fetch_text.per_run` (1,000) per run, roles never tried before roles whose read failed, `PAGE_DELAY` apart, one batch per board on the `http` pool.
|
||||
- A read that gave no text is tried again after `fetch_text.retry_days` (7). A 404 or 410 counts as read; another failure is retried on a later run; a 429 ends the reads of that board for the run.
|
||||
- It runs after every pull, after a saved agent, an affinity change or an override switched, not after a vote, and it can be switched off under Runs.
|
||||
|
||||
A role below the bar for every agent is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles that rank do. Four consequences follow from the code:
|
||||
A role in no agent's cut is never read, so it stays ranked on its teaser or its title. A board of 800 roles does not get all its text; the roles in the cut do. Four consequences follow from the code:
|
||||
|
||||
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow `enrich` encode it again.
|
||||
- `embed` encodes a role once, from its description, else its teaser, else its title. When better text arrives its vector is dropped and the `prep`, `embed` and `score` that follow fetch text encode it again.
|
||||
- A role page may fill only `_runner.DETAIL_FIELDS`: description, location and places, teaser, remote, country, employment, pay figures and date. Title, link, department and the pay line come from the list, which states them on every pull. A role whose page was read is sent without the list's place and remote on later pulls, so what its page said stays; a place the host changes after that is not picked up.
|
||||
- Completeness, and so closing, is decided by the list alone. A board of 800 roles closes its roles from the first pull.
|
||||
- SmartRecruiters and Workable list their roles in pages and belong here too, since their list has no text.
|
||||
|
||||
## Overview
|
||||
|
||||
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles `enrich` has read.
|
||||
Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open roles" counts every open role of the host's boards, copies included; the feed's 20,578 leaves copies out, the 24,003 below do not. Since the agents' reach filters big boards at the source (`app/jobs/reach.py`), a host's count is what is in reach, not what its boards hold: SmartRecruiters read 7,238 roles before the filter and 343 after. "Description" is where the text comes from; for the "list, then each role" hosts it is filled only for roles fetch text has read.
|
||||
|
||||
| Source | Boards | Open roles | How it is read | Roles per request | Description | Pay | Limit that matters most |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -95,20 +95,20 @@ Numbers are from the live database on 2026-09-30, after the pull of 12:52. "Open
|
||||
|
||||
| Limit | Value | What it means |
|
||||
|---|---|---|
|
||||
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); enrich is on by default. |
|
||||
| Schedule | `pull` every 6 h by default, editable under Runs | Each pull starts the cascade in "Flow". Extract and summarise are off in the config and switched under Runs (both are on in the live database); fetch text is on by default. |
|
||||
| Batches | 10 boards or searches per batch, 8 batches at once (`workers.http: 8`) | Within a batch boards are read one after another. Enrich reads one batch per board on the same pool. One pull runs at a time. |
|
||||
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
|
||||
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
|
||||
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request | The same for boards, role pages and searches. A slow board fails for this pull and is asked again next pull. |
|
||||
| Pages per board | 100 a pull | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
|
||||
| Wait | 0.5 s before every page after the first, for boards and searches alike, and before every role page | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
|
||||
| Per-role reads | `enrich`, after scoring: 200 per board and 1,000 per run, ranked roles first | See "List, then each role". Roles the agents do not rank are not read. |
|
||||
| Per-role reads | `fetch_text`, after scoring: 200 per board and 1,000 per run, the roles in the agents' cut | See "List, then each role". Roles in no cut are not read. |
|
||||
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing, and so does a read that asked nothing. A role listed again reopens. A board switched off closes its roles at once. A filtered read keeps roles with a vote or an application, and remote roles when the host could not be asked for remote; a read by fetch terms closes only roles not listed for 21 days. |
|
||||
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. A Jobindex search is never complete, so its roles close by the 21 days only. A search failing for 21 days closes its roles. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, a row that would not store, and a read that lists under 10% of what the same read listed last time (`SUSPECT_SHARE`) once that was at least `SUSPECT_MIN`, 10. A search that fails is recorded the same way and its row says so. |
|
||||
| A board that fails for 21 days | all its open roles close at the next pull, and a suspect answer that has stood that long is accepted | The next answer reopens what it lists. `STALE_DAYS` is the one 21. |
|
||||
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next enrich run | Nothing is closed on a list cut short by a 429. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a role page: the reads of that board stop until the next fetch text run | Nothing is closed on a list cut short by a 429. |
|
||||
|
||||
How far a board is read: a pull asks each board of 300 roles or more for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of once per country (33 codes today), unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application. A board's size is set once, when it is added, on Recheck or by `app companies size`, and a pull does not rewrite it; a board with no size yet is sized first in its batch and then read as its size says, and a board whose reader cannot narrow the reach is read whole and sized by that read. A host that cannot filter a dimension ignores it: SmartRecruiters, Phenom, SuccessFactors and Eightfold have no remote filter, so their reach asks for countries only.
|
||||
|
||||
@@ -313,7 +313,7 @@ What no source does:
|
||||
|
||||
## Hosts read as list, then each role
|
||||
|
||||
For every host below the pull reads the list only; the description arrives when `enrich` reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
For every host below the pull reads the list only; the description arrives when fetch text reads the role's page, so "Requests per pull" counts list requests. "Places" are a list of the places the host states; only the `location` line joins them.
|
||||
|
||||
### SmartRecruiters
|
||||
|
||||
@@ -323,7 +323,7 @@ For every host below the pull reads the list only; the description arrives when
|
||||
| Companies | 11 boards: Bosch (about 4,800 roles on the site, 13 open in reach), Delivery Hero, Wise |
|
||||
| List | `api.smartrecruiters.com/v1/companies/<id>/postings`, 100 per request by offset; states `totalFound` |
|
||||
| Requests per pull | one series per country of the reach (two for Bosch: DK and SE), each roles / 100, rounded up; a board read whole is roles / 100 |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by `enrich` for ranked roles only: 1 of 343 open roles has text |
|
||||
| Description | per role, `GET /v1/companies/<id>/postings/<id>`: the `jobAd.sections` `jobDescription`, `qualifications` and `additionalInformation`, joined; `companyDescription` is the same blurb on every role and is left out. Read by fetch text for the roles in the cut only: 1 of 343 open roles has text |
|
||||
| Date | `releasedDate`; trusted |
|
||||
| Place and country | `fullLocation`, else city and country, with empty parts removed; the host's `country` is handed on |
|
||||
| Remote | `location.remote`, true or false, as the host's word |
|
||||
@@ -333,7 +333,7 @@ For every host below the pull reads the list only; the description arrives when
|
||||
| Filter at the source | `?country=dk`, one country a request; remote is ignored by the API, so a remote role outside the reach is not read. Bosch DK: 2 of 4,830 (2026-09-30). Which countries is the agents' reach, see above. Fetch terms as `?q=engineer`, one a request, AND with the country; it matches title and text (Bosch: 1,713 of 4,830) |
|
||||
| Limitations | 100 pages a pull; a larger board is swept across pulls. Whether the identifier is case-sensitive is unsettled: the module docstring says it is, `docs/decisions.md` says it is not (`docs/open.md`). An unknown identifier answers an empty list, not an error: `READS.empty="suspect"` keeps a known board from closing its roles on it, but a mistyped slug attached as a new board reads as an empty, complete board. |
|
||||
|
||||
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until `enrich` has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
The reader's "Fetch the posting" button shows for a role with no `article_text` and a description under 500 characters, so for these roles until fetch text has read their page. It reads one posting page on demand into `article_text`, which `summarise` reads and `embed` and `extract` do not. Whether it reads a SmartRecruiters page cleanly was not checked.
|
||||
|
||||
### Workable
|
||||
|
||||
@@ -343,7 +343,7 @@ The reader's "Fetch the posting" button shows for a role with no `article_text`
|
||||
| Companies | 8 boards: Hugging Face, Kahoot!, Starling Bank |
|
||||
| List | a POST to `apply.workable.com/api/v3/accounts/<account>/jobs`, 10 per request with a `nextPage` token; states `total` |
|
||||
| Requests per pull | roles / 10, rounded up, 0.5 s apart; the largest board has 50 roles, so 5 |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by `enrich` for ranked roles only: 1 of 157 open roles has text |
|
||||
| Description | per role, `GET /api/v2/accounts/<account>/jobs/<shortcode>`: `description`, `requirements` and `benefits`, joined. The row keeps the `shortcode`, since `source_id` is the numeric id. Read by fetch text for the roles in the cut only: 1 of 157 open roles has text |
|
||||
| Date | `published`; trusted |
|
||||
| Place and country | every `locations` entry as "city, region, country", each remote when the role is; `location` when there are none |
|
||||
| Remote | `workplace` (remote, hybrid, on_site) as the host's word, else the `remote` flag |
|
||||
@@ -592,7 +592,7 @@ A saved search is a query with its parameters, stored as a `search` subscription
|
||||
| Saved search | `q` and optionally `region` (an `amt` id from the site's own search form: `REGIONS` in the module, 2 is Storkøbenhavn) and `remote` (`helt` fully remote, `delvist` partly) |
|
||||
| List | the RSS feed `jobbank.dk/job/rss/`, 100 items a page; an empty page ends it. It states no total and is not sorted by date |
|
||||
| Pages read | at most the module's `ceiling`, which keeps a run under about 20 pages |
|
||||
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by `enrich` for the roles an agent ranks high. The listing carries none |
|
||||
| Text | one request a role: the JSON-LD `JobPosting` on `/job/<id>/...`, read by fetch text for the roles in the cut. The listing carries none |
|
||||
| Date | `datePosted` from the role's page; trusted |
|
||||
| Place | the region in the item's description line, and `jobLocation.addressLocality` from the page; country Denmark |
|
||||
| Remote | not stated by a row; the search's `remote` filter narrows the list |
|
||||
|
||||
+3
-3
@@ -37,8 +37,8 @@ whose reader cannot narrow that reach is read whole anyway, and sized by
|
||||
that read once it is complete, rather than by a second request.
|
||||
|
||||
A pull reads listings only. A host that states a role's text on the role's
|
||||
own page (`text="detail"`) is read there by `app/jobs/enrich.py` after
|
||||
scoring, best roles first, through `read_board_details`. A search provider
|
||||
own page (`text="detail"`) is read there by `app/jobs/fetch_text.py` after
|
||||
scoring, for the roles in the agents' cut, through `read_board_details`. A search provider
|
||||
whose reader has `detail` (Jobbank) is read the same way, under its search
|
||||
target `provider:name`; `read_detail` is the one read the Fetch button makes.
|
||||
|
||||
@@ -435,7 +435,7 @@ class ATSSource(Source):
|
||||
return f"{host}:{slug}:{source_id}"
|
||||
|
||||
# A host with `detail` states a role's text on the role's own page,
|
||||
# which the `enrich` stage reads after scoring (`read_board_details`),
|
||||
# which the `fetch_text` stage reads after scoring (`read_board_details`),
|
||||
# never the pull. Its rows go out without a description, so the text
|
||||
# a role's page gave stays, and a role read on its own page goes out
|
||||
# without the listing's place and remote as well, so what its page
|
||||
|
||||
@@ -215,11 +215,11 @@ is shown and embedded until `description` arrives. A reader that sees one
|
||||
keeps it.
|
||||
|
||||
**Detail.** `detail(get, slug, row)` reads one role's own page and returns a
|
||||
dict of the fields it found. The pull reads listings only; the `enrich`
|
||||
stage (`app/jobs/enrich.py`) calls `detail` after scoring, so the requests
|
||||
go where the ranking is: the open roles without text that an agent ranks
|
||||
among its best `enrich.top` at `enrich.min_score` or above, best first, at
|
||||
most `enrich.per_board` from one board and `enrich.per_run` in all, one
|
||||
dict of the fields it found. The pull reads listings only; the `fetch_text`
|
||||
stage (`app/jobs/fetch_text.py`) calls `detail` after scoring, so the
|
||||
requests go where the model pass will read: the open roles without text in
|
||||
the cut of an enabled agent, best first, at most `fetch_text.per_board`
|
||||
from one board and `fetch_text.per_run` in all, one
|
||||
batch per board through `ats.read_board_details` and
|
||||
`_runner.read_details`, `_runner.PAGE_DELAY` apart, the roles never tried
|
||||
before those whose read failed. Only `_runner.DETAIL_FIELDS` are taken:
|
||||
@@ -230,7 +230,7 @@ any other key is left out with a debug log line, `seniority` among them.
|
||||
`title`, `url`, `department` and `salary_text` cannot be filled from
|
||||
`detail`: the listing rewrites them every pull. A 404 or a 410 counts as read; any
|
||||
other failure is logged and retried on a later run; a read that gave no text
|
||||
is retried after `enrich.retry_days`; a 429 ends the reads of that board for
|
||||
is retried after `fetch_text.retry_days`; a 429 ends the reads of that board for
|
||||
the run. A role read in detail is sent by later pulls without the listing's
|
||||
place and remote, so what its own page said about them stays. Completeness
|
||||
is decided by the listing alone.
|
||||
|
||||
Reference in new issue
Block a user