State the final host contract in the package docstring and the plan, with ranks, case, detail fields and robots.txt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-29 19:47:09 +02:00
1 parent 4fb06191cf
commit 2ff5046993
3 files changed
+154 -46

No files matched your search

+34 -14
View File
@@ -46,32 +46,52 @@ The 49 were missed because the slug is not the company's name (`BoschGroup`, `ji
## Contracts between packages
Host module, `plugins/atshosts/<host>.py`:
Host module, `plugins/atshosts/<host>.py`, as built by WP0. The docstring of `plugins/atshosts/__init__.py` holds the same contract and is the one to keep current. A module imports `desk` and `._shared` only, never `plugins.ats`.
```python
NAME = "workday" # the host of a target `workday:<slug>`
LABEL = "Workday" # the Sources page's select
PATTERNS = (r"https?://...",) # discover: over a URL, group 1 is the slug
NAME = "workday" # the host of a target `workday:<slug>`; equals the module's file name
LABEL = "Workday" # the Sources page's select
PATTERNS = (r"https?://...",) # discovery: regexes over a URL, compiled with IGNORECASE
RANK = 100 # optional int, default 100; lower comes first
KEEP_CASE = True # optional, default False; keep a discovered slug's case
def slug_ok(slug: str) -> bool: ... # strict; a slug that fails is no target
def allowed(slug: str) -> Callable[[str], bool]: ... # hostnames a fetch may reach
def careers_url(slug: str) -> str: ... # the page a person opens
def robots_page(slug: str) -> str: ... # optional; the URL robots.txt must allow
def slug_ok(slug: str) -> bool: ... # strict, re.fullmatch
def allowed(slug: str) -> Callable[[str], bool]: ... # host names a fetch may reach
def careers_url(slug: str) -> str: ... # the page a person opens
# one of the two
def feed_url(slug: str) -> str: ...
def feed_url(slug: str) -> str: ... # the whole board in one answer
def parse(body: bytes, slug: str) -> list[Row]: ...
def page(get: Get, slug: str, cursor: Any) -> Page: ...
def page(get: Get, slug: str, cursor: Any) -> Page: ... # a paged board, one page a call
def rows(decoded: Any, slug: str) -> list[Row]: ...
# optional
def detail(get: Get, slug: str, row: Row) -> Row: ...
def probe_url(slug: str) -> str: ...
def probe_ok(body: bytes) -> bool: ...
def fallback_url(slug: str) -> str: ... # feed hosts: a smaller feed, read once when feed_url is over the byte cap
def robots_page(slug: str) -> str: ... # the URL robots.txt must allow; on a host allowed(slug) admits
def detail(get: Get, slug: str, row: Row) -> dict[str, Any]: ... # fields from the role's own page
def probe_url(slug: str) -> str: ... # discovery: answers 2xx only for a board
def probe_ok(body: bytes) -> bool: ... # discovery: when 2xx is not enough
def probe_slugs(guess: str) -> tuple[str, ...]: ... # the slugs a guessed name is probed as; default (guess,)
```
`Get` is `get(url, *, method="GET", payload=None, headers=None) -> bytes`. `Row` and `Page` are as in `plugins/ats.py` today. `plugins/atshosts/__init__.py` lists the modules with `pkgutil.iter_modules`, sorted by name, so a new host registers by existing.
`Get`, `Row` and `Page` are in `plugins/atshosts/_shared.py`, with the helpers several hosts share: `only(host)` for `allowed`, `path_slug_ok` and `label_slug_ok` for `slug_ok`, `parse_dt`, `is_web_url`, `num`, `yearly_amount` and `pay_line`. `plugins/atshosts/__init__.py` lists the modules with `pkgutil.iter_modules`, checks each against the contract at import (an ImportError names the module and what it lacks), and orders `HOSTS` by `(RANK, NAME)`. The eight first hosts hold ranks 10 to 80, which keeps the order they had before the modules: the select opens on Greenhouse, and a guessed slug on two hosts goes to the one probed first.
Order and names. `RANK` orders the Sources page's select, the plugin's description and discovery's probes. `NAME` equals the file name. A host that `discover.py` names in its own `_UNREADABLE` table is taken from its module as soon as the module exists, so a wave 1 package does not edit `discover.py`.
Patterns. The slug is group 1, or, when a pattern has more groups, the groups that matched joined by "/" (`https?://([a-z0-9-]+\.wd\d+)\.myworkdayjobs\.com/(?:[a-z]{2}-[A-Z]{2}/)?([A-Za-z0-9_-]+)` gives `lego.wd103/LEGO_External`). Only group 1 is checked against discovery's words that name no board (`api`, `jobs`, `embed`, ...). The slug is lowercased unless the module sets `KEEP_CASE = True`, which a host with case-sensitive paths needs, since robots.txt paths are case-sensitive too. A pattern without a group names the host with no slug.
Slugs. `slug_ok` uses `re.fullmatch` (a `$` lets a trailing newline through) and refuses anything that could add a path, query, fragment, port or user to the URLs built from it. A slug of several parts joins them with "/": Lever's EU boards are `lever:eu/<site>`.
`get(url, *, method="GET", payload=None, headers=None) -> bytes` is `ats._get` bound to the board. A POST sends `payload` as JSON; `headers` are added to the default User-Agent and Accept and replace either where they name it. It fetches https on port 443 only, never an IP address or a local name (`localhost`, `.local`, `.internal`, `.lan`, `.home.arpa`), follows at most three redirects each checked against `allowed(slug)`, cuts the body at `MAX_BYTES` and ends within `DEADLINE`. It returns the body and raises `httpx.HTTPStatusError` on a non-2xx answer, `httpx.HTTPError` on a timeout or a connection failure, and another exception, `ValueError` among them, for a refused host, an oversized body or a broken redirect. A module lets these through.
`Page` is `(decoded, results, total, cursor)`: the decoded page handed to `rows`, its raw results as dicts with an `id` (counted by distinct id for completeness), the total the host states or None, and the next cursor or None after the last page. The first call gets cursor None. A board with a total is complete when as many distinct ids were read, one without when the read ended on an empty page; a 429 after the first page ends the read, incomplete. A paged module may also have `parse(body, slug)` for one page's body, for its tests.
A `Row` has `source_id`, `title`, `location`, `url`, `posted` (aware datetime or None), `department`, `description` (plain text), `remote` (True, False, a word or None) and `salary_text`, and, where the host states pay, `salary_min`, `salary_max` (yearly integers) and `currency`.
`detail` returns a dict of what the role's own page states. The shared code calls it for roles not yet read (`ats:detailed:<target>`), at most `DETAIL_CAP` (200) per board per pull, `PAGE_DELAY` apart, the roles never tried before those whose detail failed (`ats:detailfailed:<target>`). Only `DETAIL_FIELDS` are taken: `description`, `location`, `remote`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the listing's empty value (`upsert_role`'s keep-filled columns, the salary figures, and `posted_at`, which keeps the earliest date). Any other key is dropped with a debug log line: `title`, `url`, `department` and `salary_text` cannot be filled from `detail`, since the listing rewrites them every pull. A role not read in detail on a pull is sent with an empty description. A 404 or 410 counts as read, any other failure is retried on a later pull, and a 429 ends the detail reads for that board and pull. Completeness is decided by the listing alone.
robots.txt. For a module with `robots_page`, the shared code reads `https://<its host>/robots.txt` before the listing, at most once a day per board (`ats:robots:<target>`), following a redirect to the host's `www.` twin or back and no other. It fails the board with "robots.txt disallows it" when the rules for `*` or for the app's own token disallow the page, matched per RFC 9309: the longest matching rule wins, Allow wins a tie, `*` and a final `$` in paths, paths case-sensitive, an empty Disallow allows. A 4xx other than 429 means there are no rules, which allows. A 5xx, a 429, a timeout, a connection error or a redirect elsewhere fails the board with "robots.txt could not be read", and the next pull asks again.
Slugs: Workday `lego.wd103/LEGO_External`; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (`jobs.teradyne.com/MiR`); Oracle `<host>/<siteNumber>`.