Runner: readers read from their declaration, ats and search thin over it
plugins/readers/_runner.py takes the guarded fetch, the listing (whole, paged, one series per filter value unioned by id), completeness from READS, the sweep of a board over the page cap across pulls, the 10% guard on a filtered or empty="suspect" read, facets kept a day, the detail reads, the raw an Item carries and the failure bookkeeping out of plugins/ats.py. Ask comes from a subscription's params (ask_for, open for the reach), with a declared key resolved into Ask.params["key"] and refused when missing. The runtime robots.txt check goes (decided at build time, READS.robots). plugins/search.py reads the providers of readers.PROVIDERS through the runner and applies the saved pages param. A filtered board's close keeps roles with a vote or an application. DETAIL_FIELDS gain places, teaser, country and employment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
779b136757
commit
902089a5e2
14 files changed
+1608
-1492
No files matched your search
+14
-14
@@ -129,7 +129,8 @@ exists.
|
||||
|
||||
**Fetching.** `allowed(slug)` returns a predicate over a lowercased host
|
||||
name; `_shared.only(host)` is the usual one. Every request goes through
|
||||
`get`, which is `ats._get` bound to the board:
|
||||
`get`, which is `_runner.fetch` bound to the target (`_runner.getter`; the
|
||||
`ats` Source binds it through `ats._get`):
|
||||
|
||||
get(url, *, method="GET", payload=None, headers=None) -> bytes
|
||||
|
||||
@@ -138,7 +139,7 @@ default User-Agent and `Accept: application/json, text/xml`, and replace
|
||||
either where they name it. `get` is https only on port 443, never to an IP
|
||||
address or a local name (`localhost`, `.local`, `.internal`, ...), follows at
|
||||
most three redirects each checked against `allowed(slug)`, cuts the body at
|
||||
`ats.MAX_BYTES` and ends the whole request within `ats.DEADLINE`. It returns
|
||||
`_runner.MAX_BYTES` and ends the whole request within `_runner.DEADLINE`. It returns
|
||||
the body and raises `httpx.HTTPStatusError` on a non-2xx answer (read
|
||||
`exc.response.status_code`), `httpx.HTTPError` on a timeout or a connection
|
||||
failure, and another exception (`ValueError` among them) for a refused host,
|
||||
@@ -151,8 +152,9 @@ id, to decide completeness), the total the host states or None, and the
|
||||
cursor of the next page or None after the last. The first call gets cursor
|
||||
None. A board with a total is complete when as many distinct ids were read;
|
||||
one without, when the read ended on an empty page. A 429 after the first page
|
||||
ends the read, incomplete; `ats.MAX_PAGES` pages at most, `ats.PAGE_DELAY`
|
||||
apart. A paged module may also have `parse(body, slug)` for one page's body.
|
||||
ends the read, incomplete; `_runner.MAX_PAGES` pages a pull at most,
|
||||
`_runner.PAGE_DELAY` apart, and a board cut there is swept across pulls
|
||||
(`_runner.read_listing`). A paged module may also have `parse(body, slug)` for one page's body.
|
||||
|
||||
**Rows.** A `Row` (`_contract.Row`) is a dict with `source_id` (the host's
|
||||
own id for the role, unique on the board), `title`, `url` (http or https,
|
||||
@@ -183,11 +185,11 @@ keeps it.
|
||||
|
||||
**Detail.** `detail(get, slug, row)` reads one role's own page and returns a
|
||||
dict of the fields it found. The shared code calls it for roles not yet read
|
||||
(`ats:detailed:<target>`), at most `ats.DETAIL_CAP` (200) a board a pull,
|
||||
`ats.PAGE_DELAY` apart, the roles never tried before those that failed
|
||||
(`ats:detailfailed:<target>`). Only `ats.DETAIL_FIELDS` are taken:
|
||||
`description`, `location`, `remote`, `currency`, `salary_min`, `salary_max`
|
||||
and `posted`, the keys whose columns survive a later pull that sends the
|
||||
(`ats:detailed:<target>`), at most `_runner.DETAIL_CAP` (200) a board a
|
||||
pull, `_runner.PAGE_DELAY` apart, the roles never tried before those that
|
||||
failed (`ats:detailfailed:<target>`). Only `_runner.DETAIL_FIELDS` are taken:
|
||||
`description`, `location`, `places`, `teaser`, `remote`, `country`,
|
||||
`employment`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the
|
||||
listing's empty value; any other key is left out with a debug log line.
|
||||
`title`, `url`, `department` and `salary_text` cannot be filled from
|
||||
`detail`: the listing rewrites them every pull. A role not read in detail
|
||||
@@ -203,11 +205,9 @@ Whoever builds a reader reads the host's robots.txt for the paths the reader
|
||||
requests and reports what it disallows, and the owner decides whether the
|
||||
host is read anyway. `robots` records that decision with its date, e.g.
|
||||
`"read: /wday/cxs/ allowed (checked 2026-09-29)"` or `"read though /api/ is
|
||||
disallowed: owner, 2026-09-30"`. The runtime check in `ats.py`, which
|
||||
`robots_page` and `ROBOTS_ANY_CASE` feed, predates this rule and goes with
|
||||
the runner; a new reader defines neither. Until then it reads
|
||||
`https://<host>/robots.txt` at most once a day per board and fails the board
|
||||
when the page is disallowed for `*` or the app's token.
|
||||
disallowed: owner, 2026-09-30"`. Nothing reads robots.txt at run time;
|
||||
`robots_page` and `ROBOTS_ANY_CASE`, which fed the former runtime check, are
|
||||
read by nothing and a new reader defines neither.
|
||||
|
||||
**Discovery.** `probe_url(slug)` is a cheap URL that answers 2xx only when
|
||||
the slug is a board; `probe_ok(body)` narrows that when a host answers an
|
||||
|
||||
Reference in new issue
Block a user