Runner: readers read from their declaration, ats and search thin over it

plugins/readers/_runner.py takes the guarded fetch, the listing (whole,
paged, one series per filter value unioned by id), completeness from READS,
the sweep of a board over the page cap across pulls, the 10% guard on a
filtered or empty="suspect" read, facets kept a day, the detail reads, the
raw an Item carries and the failure bookkeeping out of plugins/ats.py.
Ask comes from a subscription's params (ask_for, open for the reach), with
a declared key resolved into Ask.params["key"] and refused when missing.

The runtime robots.txt check goes (decided at build time, READS.robots).
plugins/search.py reads the providers of readers.PROVIDERS through the
runner and applies the saved pages param. A filtered board's close keeps
roles with a vote or an application. DETAIL_FIELDS gain places, teaser,
country and employment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 09:28:33 +02:00
1 parent 779b136757
commit 902089a5e2
14 files changed
+1608 -1492

No files matched your search

+14 -14
View File
@@ -129,7 +129,8 @@ exists.
**Fetching.** `allowed(slug)` returns a predicate over a lowercased host
name; `_shared.only(host)` is the usual one. Every request goes through
`get`, which is `ats._get` bound to the board:
`get`, which is `_runner.fetch` bound to the target (`_runner.getter`; the
`ats` Source binds it through `ats._get`):
get(url, *, method="GET", payload=None, headers=None) -> bytes
@@ -138,7 +139,7 @@ default User-Agent and `Accept: application/json, text/xml`, and replace
either where they name it. `get` is https only on port 443, never to an IP
address or a local name (`localhost`, `.local`, `.internal`, ...), follows at
most three redirects each checked against `allowed(slug)`, cuts the body at
`ats.MAX_BYTES` and ends the whole request within `ats.DEADLINE`. It returns
`_runner.MAX_BYTES` and ends the whole request within `_runner.DEADLINE`. It returns
the body and raises `httpx.HTTPStatusError` on a non-2xx answer (read
`exc.response.status_code`), `httpx.HTTPError` on a timeout or a connection
failure, and another exception (`ValueError` among them) for a refused host,
@@ -151,8 +152,9 @@ id, to decide completeness), the total the host states or None, and the
cursor of the next page or None after the last. The first call gets cursor
None. A board with a total is complete when as many distinct ids were read;
one without, when the read ended on an empty page. A 429 after the first page
ends the read, incomplete; `ats.MAX_PAGES` pages at most, `ats.PAGE_DELAY`
apart. A paged module may also have `parse(body, slug)` for one page's body.
ends the read, incomplete; `_runner.MAX_PAGES` pages a pull at most,
`_runner.PAGE_DELAY` apart, and a board cut there is swept across pulls
(`_runner.read_listing`). A paged module may also have `parse(body, slug)` for one page's body.
**Rows.** A `Row` (`_contract.Row`) is a dict with `source_id` (the host's
own id for the role, unique on the board), `title`, `url` (http or https,
@@ -183,11 +185,11 @@ keeps it.
**Detail.** `detail(get, slug, row)` reads one role's own page and returns a
dict of the fields it found. The shared code calls it for roles not yet read
(`ats:detailed:<target>`), at most `ats.DETAIL_CAP` (200) a board a pull,
`ats.PAGE_DELAY` apart, the roles never tried before those that failed
(`ats:detailfailed:<target>`). Only `ats.DETAIL_FIELDS` are taken:
`description`, `location`, `remote`, `currency`, `salary_min`, `salary_max`
and `posted`, the keys whose columns survive a later pull that sends the
(`ats:detailed:<target>`), at most `_runner.DETAIL_CAP` (200) a board a
pull, `_runner.PAGE_DELAY` apart, the roles never tried before those that
failed (`ats:detailfailed:<target>`). Only `_runner.DETAIL_FIELDS` are taken:
`description`, `location`, `places`, `teaser`, `remote`, `country`,
`employment`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the
listing's empty value; any other key is left out with a debug log line.
`title`, `url`, `department` and `salary_text` cannot be filled from
`detail`: the listing rewrites them every pull. A role not read in detail
@@ -203,11 +205,9 @@ Whoever builds a reader reads the host's robots.txt for the paths the reader
requests and reports what it disallows, and the owner decides whether the
host is read anyway. `robots` records that decision with its date, e.g.
`"read: /wday/cxs/ allowed (checked 2026-09-29)"` or `"read though /api/ is
disallowed: owner, 2026-09-30"`. The runtime check in `ats.py`, which
`robots_page` and `ROBOTS_ANY_CASE` feed, predates this rule and goes with
the runner; a new reader defines neither. Until then it reads
`https://<host>/robots.txt` at most once a day per board and fails the board
when the page is disallowed for `*` or the app's token.
disallowed: owner, 2026-09-30"`. Nothing reads robots.txt at run time;
`robots_page` and `ROBOTS_ANY_CASE`, which fed the former runtime check, are
read by nothing and a new reader defines neither.
**Discovery.** `probe_url(slug)` is a cheap URL that answers 2xx only when
the slug is a board; `probe_ok(body)` narrows that when a host answers an