Runner: readers read from their declaration, ats and search thin over it

plugins/readers/_runner.py takes the guarded fetch, the listing (whole,
paged, one series per filter value unioned by id), completeness from READS,
the sweep of a board over the page cap across pulls, the 10% guard on a
filtered or empty="suspect" read, facets kept a day, the detail reads, the
raw an Item carries and the failure bookkeeping out of plugins/ats.py.
Ask comes from a subscription's params (ask_for, open for the reach), with
a declared key resolved into Ask.params["key"] and refused when missing.

The runtime robots.txt check goes (decided at build time, READS.robots).
plugins/search.py reads the providers of readers.PROVIDERS through the
runner and applies the saved pages param. A filtered board's close keeps
roles with a vote or an application. DETAIL_FIELDS gain places, teaser,
country and employment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 09:28:33 +02:00
1 parent 779b136757
commit 902089a5e2
14 files changed
+1608 -1492

No files matched your search

+131 -589
View File
@@ -2,50 +2,44 @@
`plugins/readers/`, one module per host.
Every one of these hosts serves a company's own list of roles as public,
unauthenticated JSON, XML or RSS — first-party, no scraping, and a role that
drops off the board is *closed*, which is the cheapest staleness check there
is. What a host says and how its feed reads is its module's; this file holds
what every host shares: the fetch and its allow-list, paging and
completeness, the per-role detail reads, robots.txt, the classifier, and the
plugin itself.
unauthenticated JSON, XML or RSS, and a role that drops off the board is
*closed*, which is the cheapest staleness check there is. What a host says
and how its feed reads is its module's, what every reader shares is
`plugins/readers/_runner.py`'s; this file is the `ats` Source over the runner,
the classifier, and the names the app and the tests reach for on `ats`.
The allow-list and the classifier are ported from `openings.go` and
`cmd/seedgen/main.go` in
[jobhunt-hq](https://github.com/rembrandtreyes/jobhunt-hq) (MIT), including
`classify`'s nine functions and their keywords. The body cap is not a
straight port — see `MAX_BYTES` below.
`classify`'s nine functions and their keywords.
A target is `host:slug`, e.g. `greenhouse:databricks`, `ashby:deepl`,
`lever:eu/tomtom`, `teamtailor:careers.acme.com`; each host module says what
its slug is. Every board is fetched independently; one that fails or times
out is logged, recorded as failing under `ats:failing:<target>` (see
`ATSSource.fetch`), and skipped, and the rest continue.
its slug is. Every board is read independently; one that fails is logged,
recorded as failing under `ats:failing:<target>`, and skipped. A board's
subscription `params` (`{"countries": ["DK"], "remote": true}`) are its
filter, narrowed to what its reader declares (`_runner.ask_for`).
A host that pages states a total or does not. A board with a total counts as
complete only when it read as many distinct roles as the total says, one
without when it read up to an empty page (see `_paged`); a host that answers
with the whole board at once is complete whenever it parses.
The limits below are read when a board is read, so a test that replaces one
on this module reaches the runner.
"""
from __future__ import annotations
import ipaddress
import json
import logging
import re
import sys
import time
from collections.abc import Callable, Iterator, MutableMapping
from datetime import UTC, datetime, timedelta
from typing import Any
from collections.abc import Callable, Iterator
from datetime import UTC, datetime
from pathlib import Path
from urllib.parse import unquote
from typing import Any
import httpx
from desk.sources import (
API_VERSION, ARTICLE_READER, Item, PluginSpec, Profile, Source,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks, USER_AGENT,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks, USER_AGENT, # noqa: F401
)
try:
@@ -58,6 +52,7 @@ except ModuleNotFoundError:
# file. So the root goes on the path then, and only then.
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from plugins import readers
from plugins.readers import _runner
from plugins.readers._shared import Get, Row
# The names the parsers had here before they moved to their host modules,
@@ -70,156 +65,57 @@ from plugins.readers.teamtailor import parse as _parse_teamtailor # noqa: F401
log = logging.getLogger(__name__)
#: openings.go's 8 MB body cap is not kept: it was written against arbitrary URLs, and every
#: URL fetched here is a feed on a fixed allow-list of ATS hosts, Greenhouse's
#: with `?content=true`. The real run of 2026-09-23 hit it on the biggest employers —
#: Anduril at 41 MB, Anthropic, OpenAI, Databricks, Carvana, Axon and
#: Airwallex between 9 and 19 MB — and skipped their boards outright. Raised
#: to 96 MB so those boards fit; a board that still exceeds it gets one retry
#: at its host's `fallback_url` where it has one (Greenhouse, without
#: `?content=true`, see `_fetch_rows`), so at least the titles land.
MAX_BYTES = 96 << 20
#: The runner's limits, kept here as the `ats` Source's own: see `_runner`.
MAX_BYTES = _runner.MAX_BYTES
TIMEOUTS = _runner.TIMEOUTS
DEADLINE = _runner.DEADLINE
MAX_PAGES = _runner.MAX_PAGES
PAGE_DELAY = _runner.PAGE_DELAY
DETAIL_CAP = _runner.DETAIL_CAP
DETAIL_FIELDS = _runner.DETAIL_FIELDS
#: openings.go's 10 s did not hold: `greenhouse:mongodb` and `greenhouse:feedzai`
#: failed a pull of 2026-09-28 with a read timeout. Measured live on
#: 2026-09-29, three runs each of mongodb, feedzai and the five Greenhouse
#: boards with the most open roles (databricks, stripe, anthropic, feverup,
#: datadog), with and without `pay_transparency=true`, cached and with a
#: cache-busting parameter (84 requests; CloudFront answered every one
#: "Miss", so all reached the origin): the body arrives in under 1 s once it
#: starts (at most 0.09 s between two chunks, 10.4 MB the largest), but the
#: first byte sometimes waits: 16.6 s (feverup, no flag), 14.4 s (anthropic,
#: flag), 10.9 s (feverup, flag), 5.2 s and 3.6 s, and under 2.6 s otherwise.
#: The flag made no difference that three runs could tell apart from that
#: noise. So a read may wait 30 s, about twice the worst first byte seen,
#: and a whole request, redirects and body included, 60 s (see `_get`),
#: about four times the slowest seen, which still leaves a 96 MB body room
#: on a link far slower than the one measured.
TIMEOUTS = httpx.Timeout(10.0, read=30.0)
DEADLINE = 60.0
#: The runner's errors, under the names they had here.
_Refused = _runner.Refused
_TooLarge = _runner.TooLarge
_local_host = _runner.local_host
#: host -> the name the host gives itself: the hosts this plugin reads, in
#: the order a select offers them. What the Sources page's host select is
#: rendered from, so it cannot offer a host `is_known_target` refuses.
HOSTS: dict[str, str] = {name: module.LABEL for name, module in readers.HOSTS.items()}
#: host -> parser, for the hosts whose module parses a whole body.
_PARSERS: dict[str, Callable[[bytes, str], list[Row]]] = {
name: module.parse for name, module in readers.HOSTS.items() if hasattr(module, "parse")}
def _limits() -> _runner.Limits:
return _runner.Limits(max_pages=MAX_PAGES, page_delay=PAGE_DELAY, detail_cap=DETAIL_CAP)
def _allowed_host(host: str, slug: str) -> Callable[[str], bool]:
"""The hosts a request or a redirect may land on for one board: what its
host module's `allowed` says, which is one fixed API host or the
company's own site. Anything else is refused, at the first hop and at
every redirect: a feed is external content, and a board row must never
be able to point this fetch at an internal address."""
host module's `allowed` says. Anything else is refused, at the first hop
and at every redirect."""
module = readers.HOSTS.get(host)
return module.allowed(slug) if module is not None else lambda h: False
class _Refused(Exception):
"""A board answered somewhere its allow-list does not cover."""
class _TooLarge(ValueError):
"""A response exceeded `MAX_BYTES`. A `ValueError` so a caller that only
knows the old shape still catches it; `_fetch_rows` catches it by type to
retry at the host's `fallback_url`."""
class _Disallowed(Exception):
"""The site's robots.txt disallows the page a board is read from (see
`_check_robots`)."""
class _RobotsUnreachable(Exception):
"""The site's robots.txt could not be read: a 5xx, a 429, a timeout, a
connection error or a redirect off the site. RFC 9309 says to assume
the site disallows everything then, so the board is not read this pull."""
#: Host name suffixes that name a machine on a private network, never a
#: company's careers site.
_LOCAL_SUFFIXES = (".localhost", ".local", ".internal", ".lan", ".home.arpa")
def _local_host(name: str) -> bool:
"""Whether a host name is an address or a local name rather than a public
DNS name: an IP literal (v4 or v6), anything whose last label is a number
(`2130706433`, `127.1` and `0x7f000001` all reach 127.0.0.1 through the
resolver), `localhost`, or a private suffix. A Teamtailor slug can be any
host name, so the allow-list alone would let one point the fetch here."""
name = name.strip("[]").rstrip(".").lower()
try:
ipaddress.ip_address(name)
return True
except ValueError:
pass
last = name.rsplit(".", 1)[-1]
return (last.isdigit() or bool(re.fullmatch(r"0x[0-9a-f]+", last))
or name == "localhost" or name.endswith(_LOCAL_SUFFIXES))
def _get(client: httpx.Client, url: str, host: str, slug: str, *,
method: str = "GET", payload: dict[str, Any] | None = None,
headers: dict[str, str] | None = None, www_twin: bool = False) -> bytes:
"""Fetch one URL for a board: https only on the default port, never an
address or a local name (`_local_host`), the host's own allow-list, at
most three redirect hops each checked the same way, and the body cut off
at `MAX_BYTES`. Port of openings.go's `fetchOne`/`feedCache.allow`. A POST
sends `payload` as JSON, `{}` when there is none. `headers` are sent on
top of the User-Agent and Accept every request carries, and replace
either where they name it. With `www_twin`, a redirect may also land on
the first URL's host with `www.` added or taken off, which only the
robots.txt read asks for (see `_check_robots`).
The body is streamed and counted as it arrives, so an oversized one is
refused at `MAX_BYTES` rather than after it has been read whole, and
the whole request, redirects included, must end within `DEADLINE`: the
per-read `TIMEOUTS` alone let a board that trickles bytes run on for
good. The deadline is checked between chunks, so it can be overrun by
at most one read timeout."""
allow = _allowed_host(host, slug)
if www_twin:
bare = httpx.URL(url).host.removeprefix("www.")
board_allow = allow
allow = lambda h: board_allow(h) or h.removeprefix("www.") == bare # noqa: E731
sent = {"User-Agent": USER_AGENT, "Accept": "application/json, text/xml"} | (headers or {})
deadline = time.monotonic() + DEADLINE
current = url
for _hop in range(4):
parsed = httpx.URL(current)
if (parsed.scheme != "https" or parsed.port not in (None, 443)
or _local_host(parsed.host) or not allow(parsed.host)):
raise _Refused(f"{current} is not an allowed host for {host}")
with client.stream(
method, current, timeout=TIMEOUTS, follow_redirects=False, headers=sent,
json=(payload or {}) if method == "POST" else None,
) as response:
if response.is_redirect:
location = response.headers.get("location") or ""
if not location:
raise ValueError("redirect with no location")
current = str(httpx.URL(current).join(location))
continue
response.raise_for_status()
chunks: list[bytes] = []
size = 0
for chunk in response.iter_bytes():
size += len(chunk)
if size > MAX_BYTES:
raise _TooLarge(f"response too large (over {MAX_BYTES} bytes)")
if time.monotonic() > deadline:
raise httpx.ReadTimeout(f"no whole answer within {DEADLINE:.0f} s",
request=response.request)
chunks.append(chunk)
return b"".join(chunks)
raise ValueError("too many redirects")
headers: dict[str, str] | None = None) -> bytes:
"""`_runner.fetch` for one board, with this module's `MAX_BYTES` and
`DEADLINE`."""
return _runner.fetch(client, url, _allowed_host(host, slug), method=method,
payload=payload, headers=headers, max_bytes=MAX_BYTES,
deadline=DEADLINE, name=host)
def _getter(client: httpx.Client, host: str, slug: str) -> Get:
"""The `get` a host module is handed: `_get` bound to one board. `_get`
is looked up when a request is made, not when this is built, so a test
that replaces `ats._get` reaches every host; `payload` and `headers` are
passed on only when given, which keeps a replacement that predates them
working."""
is looked up when a request is made, so a test that replaces `ats._get`
reaches every host; `payload` and `headers` are passed on only when
given, which keeps a replacement that predates them working."""
def get(url: str, *, method: str = "GET", payload: dict[str, Any] | None = None,
headers: dict[str, str] | None = None) -> bytes:
extra: dict[str, Any] = {}
@@ -231,12 +127,6 @@ def _getter(client: httpx.Client, host: str, slug: str) -> Get:
return get
#: host -> parser, for the hosts whose module parses a whole body. Every one
#: takes the raw response body and the slug.
_PARSERS: dict[str, Callable[[bytes, str], list[Row]]] = {
name: module.parse for name, module in readers.HOSTS.items() if hasattr(module, "parse")}
# --------------------------------------------------------- the classifier
#: The fixed set of functions, and the keyword rules that map a title
@@ -381,362 +271,40 @@ def careers_url(target: str) -> str:
return module.careers_url(slug) if module is not None else ""
#: Pages read from one paged board at most, and the pause between two of
#: them. Workable answers ten roles a page, SmartRecruiters a hundred and
#: Teamtailor fifty, so the cap is 1,000, 10,000 and 5,000 roles; the pause
#: keeps Workable, which answers a burst with 429, from seeing one. The same
#: pause separates two detail reads.
MAX_PAGES = 100
PAGE_DELAY = 0.5
def _paged(client: httpx.Client, target: str) -> tuple[list[Row], bool]:
"""Every page of a board whose host pages, `PAGE_DELAY` apart, until the
host has no next page, the stated total is reached, or `MAX_PAGES`.
Returns the rows and whether the board is complete: as many distinct
results read as the host's stated total, or, for a host that states none
(Teamtailor), the read ended on an empty page (a page with nothing new in
it ends the read short of that, incomplete, so an offset the host ignores
cannot loop). Counted on the raw results, so a row the parser drops does
not make a board incomplete, and on distinct ids, so a page that shifted
under an offset and repeated a role does not make it complete.
A 429 after the first page stops the read with what it has, incomplete;
one on the first page raises like any other failure."""
host, _, slug = target.partition(":")
module = readers.HOSTS[host]
get = _getter(client, host, slug)
rows: dict[str, Row] = {}
seen: set[str] = set()
total: Any = None
cursor: Any = None
for page in range(MAX_PAGES):
if page:
time.sleep(PAGE_DELAY)
try:
data, results, total, cursor = module.page(get, slug, cursor)
except httpx.HTTPStatusError as exc:
if not page or exc.response.status_code != 429:
raise
log.warning("ats: %s: 429 on page %d; kept %d of %s, not complete",
target, page + 1, len(seen), total)
return list(rows.values()), False
before = len(seen)
seen |= {str(r.get("id") or r.get("shortcode") or "") for r in results}
for row in module.rows(data, slug):
rows.setdefault(row["source_id"], row)
if (cursor is None or (isinstance(total, int) and len(seen) >= total)
or (total is None and len(seen) == before)):
break
complete = len(seen) == total if isinstance(total, int) else total is None and cursor is None
if not complete:
log.warning("ats: %s: read %d of %s stated; not complete", target, len(seen),
"an unstated total" if total is None else total)
return list(rows.values()), complete
def _fetch_rows(client: httpx.Client, target: str) -> tuple[list[Row], bool]:
"""One board's listing, fetched and parsed: its rows and whether they are
the whole board. Raises on any failure — refused host, timeout, a feed
shape the host's parser does not understand. A host that does not page
answers with the whole board, so it is complete whenever it parses; a
paged one is complete when `_paged` says so.
A body that exceeds `MAX_BYTES` gets one retry at the host's
`fallback_url` where it has one (Greenhouse, without `?content=true`):
the description is lost, but the titles land rather than the whole board
being skipped."""
def _fetch_rows(client: httpx.Client, target: str,
ask: readers.Ask | None = None) -> tuple[list[Row], bool]:
"""One board's listing, read and parsed, unfiltered unless `ask` says:
its rows and whether they are the whole board. Raises on any failure.
No sweep and no state: what `fetch_board` and the host tests read."""
if not is_known_target(target):
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
host, _, slug = target.partition(":")
module = readers.HOSTS[host]
if hasattr(module, "page"):
return _paged(client, target)
get = _getter(client, host, slug)
try:
body = get(module.feed_url(slug))
except _TooLarge:
if not hasattr(module, "fallback_url"):
raise
log.warning("ats: %s: body exceeded %d bytes; retrying at the smaller feed",
target, MAX_BYTES)
body = get(module.fallback_url(slug))
return module.parse(body, slug), True
# -------------------------------------------------------------- robots.txt
#: How long a robots.txt answer holds for one board before it is read again.
ROBOTS_TTL = timedelta(days=1)
#: This app's product token, the name a robots.txt group addresses it by:
#: the `desk` of `desk/<version> (personal aggregator)`.
ROBOTS_TOKEN = USER_AGENT.split("/", 1)[0].strip().lower()
#: One rule of a group: `(allow, path pattern)`.
_Rule = tuple[bool, str]
def _robots_groups(text: str) -> list[tuple[set[str], list[_Rule]]]:
"""The groups of a robots.txt, per RFC 9309: one or more `user-agent`
lines (their tokens lowercased) and the `allow` and `disallow` rules that
follow them, up to the next `user-agent` line after a rule. Comments,
other keys and rules before any `user-agent` line are left out, and so is
an empty rule, which allows everything and so says nothing."""
groups: list[tuple[set[str], list[_Rule]]] = []
in_agents = False
for line in text.splitlines():
key, sep, value = line.split("#", 1)[0].partition(":")
key, value = key.strip().lower(), value.strip()
if not sep:
continue
if key == "user-agent":
if not in_agents:
groups.append((set(), []))
in_agents = True
groups[-1][0].add(value.lower())
elif key in ("allow", "disallow") and groups:
in_agents = False
if value:
groups[-1][1].append((key == "allow", value))
return groups
def _robots_rules(groups: list[tuple[set[str], list[_Rule]]], token: str) -> list[_Rule]:
"""The rules for `token`: every group that names it (matched whole and
ignoring case, as RFC 9309 says, not as a substring), merged; else every
`*` group, merged; else none, which allows everything."""
for wanted in (token, "*"):
rules = [rule for agents, group in groups if wanted in agents for rule in group]
if rules or any(wanted in agents for agents, _group in groups):
return rules
return []
def _robots_pattern(path: str) -> re.Pattern[str]:
"""A rule's path as a regex anchored at the start: `*` is any run of
characters, and a `$` at the end anchors the end."""
anchored = path.endswith("$")
body = ".*".join(re.escape(part) for part in path.rstrip("$").split("*"))
return re.compile(body + (r"\Z" if anchored else ""), re.DOTALL)
def _robots_verdict(rules: list[_Rule], path: str) -> bool:
"""Whether `path` is allowed by `rules`: the longest matching rule wins,
an allow wins a tie, and no match allows. Both sides are compared
percent-decoded, so `%7E` and `~` are one character."""
path = unquote(path)
best: tuple[int, bool] | None = None
for allow, pattern in rules:
if _robots_pattern(unquote(pattern)).match(path):
weight = (len(pattern), allow)
best = weight if best is None or weight > best else best
return best is None or best[1]
def _robots_allows(body: bytes, page: str) -> bool:
"""Whether a robots.txt lets `page` be fetched, by RFC 9309: by its rules
for every agent (`*`), and by those it has for this app's own token
(`ROBOTS_TOKEN`, which fall back to `*`), so a site that shuts out this
app by name is honoured as well. Paths are case-sensitive; a byte order
mark is dropped."""
groups = _robots_groups(body.decode("utf-8-sig", "replace"))
url = httpx.URL(page)
path = (url.raw_path or b"/").decode("ascii", "replace")
return all(_robots_verdict(_robots_rules(groups, token), path)
for token in ("*", ROBOTS_TOKEN))
def _check_robots(client: httpx.Client, target: str,
state: MutableMapping[str, str] | None = None) -> None:
"""Raise `_Disallowed` when the site of a host with a `robots_page` has a
robots.txt that disallows that page, and `_RobotsUnreachable` when it
cannot be read. Read through `_get`, so from the board's own allowed
hosts only, except that a redirect between the host and its `www.` twin
is followed (at most three hops, https only); any other redirect off the
site is unreachable. With `state`, the verdict and when it was reached
are kept under `ats:robots:<target>` and held for `ROBOTS_TTL`.
By RFC 9309: a 4xx other than 429 means there is no robots.txt to obey,
which allows, and is kept like any other verdict. A 5xx, a 429, a
timeout or a connection error means the file is unreachable, which the
RFC says to treat as disallowing everything: the board is not read on
this pull, fails with "robots.txt could not be read", and nothing is
kept, so the next pull asks again. A 429 counts as unreachable although
it is a 4xx, since it says "not now" rather than "there is none"."""
host, _, slug = target.partition(":")
module = readers.HOSTS.get(host)
if module is None or not hasattr(module, "robots_page") or not is_known_target(target):
return
page = module.robots_page(slug)
key = f"ats:robots:{target}"
now = datetime.now(UTC)
if state is not None:
try:
kept = json.loads(state.get(key) or "{}")
fresh = now - datetime.fromisoformat(kept["checked"]) < ROBOTS_TTL
except (ValueError, KeyError, TypeError):
kept, fresh = {}, False
if fresh:
if not kept.get("allowed", True):
raise _Disallowed(f"robots.txt disallows {page}")
return
url = f"https://{httpx.URL(page).host}/robots.txt"
try:
body = _get(client, url, host, slug, headers={"Accept": "text/plain"}, www_twin=True)
except httpx.HTTPStatusError as exc:
code = exc.response.status_code
if not 400 <= code < 500 or code == 429:
raise _RobotsUnreachable(f"robots.txt answered {code}") from exc
allowed = True
except Exception as exc:
raise _RobotsUnreachable(f"robots.txt not read: {exc}") from exc
else:
# A host that serves a path whatever its case is matched without
# case, so a rule written `/Jobs/` forbids a site named `jobs` too.
if getattr(module, "ROBOTS_ANY_CASE", False):
allowed = _robots_allows(body.lower(), page.lower())
else:
allowed = _robots_allows(body, page)
if state is not None:
state[key] = json.dumps({"checked": now.isoformat(timespec="seconds"),
"allowed": allowed})
if not allowed:
raise _Disallowed(f"robots.txt disallows {page}")
# ---------------------------------------------------------------- details
#: Roles read in detail from one board in one pull at most. The rest of a
#: large new board is read over the next pulls, 200 at a time.
DETAIL_CAP = 200
#: The row keys a host's `detail` may fill: those whose column survives a
#: later pull that sends the listing's empty value for them, so what a role's
#: own page said is not wiped the pull after it was read. `description`,
#: `location` (`place`, `place_key`, `country`), `remote` and `currency` are
#: `upsert_role`'s keep-filled columns, `salary_min` and `salary_max` are only
#: overwritten by a figure, and `posted` feeds `posted_at`, which keeps the
#: earliest date a role ever had. A filled column is still overwritten by
#: another filled value, so `_read_details` withholds the listing's place and
#: remote for a role already read. `title` and `url` are not here, since the
#: listing always states them and would overwrite them; nor are `department`
#: (no column of its own, only `raw` and the classifier's guess) and
#: `salary_text`, both rewritten from the listing on every pull.
DETAIL_FIELDS = ("description", "location", "remote", "currency", "salary_min",
"salary_max", "posted")
listing = _runner.read_listing(readers.HOSTS[host], slug, _getter(client, host, slug), ask,
limits=_limits())
return listing.rows, listing.complete
def _read_details(client: httpx.Client, target: str, rows: list[Row], complete: bool,
detailed: set[str], failed: set[str]
) -> tuple[list[Row], set[str], set[str]]:
"""The rows of a board whose host has `detail`, with the roles not yet
read in detail filled from their own pages: at most `DETAIL_CAP` of them,
`PAGE_DELAY` apart, first the roles never tried and then those whose
detail failed before (`failed`), so a few roles that always fail cannot
take the whole cap pull after pull. Returns the rows and the compound ids
read (for `ats:detailed:<target>`) and failed (for
`ats:detailfailed:<target>`).
Only `DETAIL_FIELDS` are taken from what `detail` returns; any other key
is left out with a debug line. Every row not read in detail in this call
goes out with an empty description: a role read on an earlier pull keeps
the one it has, since `upsert_role` never overwrites a filled column with
an empty value, and a teaser from the listing must not replace it. A role
read on an earlier pull goes out without the listing's place and remote
as well, for the same reason: the listing states them every pull
("2 Locations"), and would replace what the role's own page said.
A detail that answers 404 or 410 counts as read: the role is gone or has
no page of its own, and asking again would not change that. One that
raises otherwise is logged and the role kept in `failed`; a 429 stops the
reads for this board until the next pull, and does not mark the role as
failing. None of this touches completeness, which is the listing's alone.
The ids of roles no longer listed are dropped from both sets, but only
after a complete listing, so one cut short does not make the next pull
read its missing roles again."""
"""`_runner.read_details` for one board, with `rows` as the whole listing
when `complete`."""
host, _, slug = target.partition(":")
module = readers.HOSTS[host]
get = _getter(client, host, slug)
done, failing = set(detailed), set(failed) - set(detailed)
def compound(row: Row) -> str:
return f"{host}:{slug}:{row['source_id']}"
unread = [row for row in rows if compound(row) not in done]
queue = ([row for row in unread if compound(row) not in failing]
+ [row for row in unread if compound(row) in failing])[:DETAIL_CAP]
filled: dict[str, Row] = {}
for row in queue:
key = compound(row)
time.sleep(PAGE_DELAY)
try:
answer = module.detail(get, slug, row)
except httpx.HTTPStatusError as exc:
code = exc.response.status_code
if code == 429:
log.warning("ats: %s: 429 on a detail read; the rest wait for the next pull",
target)
break
if code in (404, 410):
done.add(key)
failing.discard(key)
else:
log.warning("ats: %s: detail of %s answered %s", target, row["source_id"], code)
failing.add(key)
continue
except Exception as exc:
log.warning("ats: %s: detail of %s failed (%s)", target, row["source_id"], exc)
failing.add(key)
continue
if ignored := sorted(set(answer) - set(DETAIL_FIELDS)):
log.debug("ats: %s: detail of %s: %s left out, not in DETAIL_FIELDS",
target, row["source_id"], ", ".join(ignored))
filled[key] = {k: v for k, v in answer.items() if k in DETAIL_FIELDS}
done.add(key)
failing.discard(key)
out = []
for row in rows:
key = compound(row)
if key in filled:
out.append(row | {"description": ""} | filled[key])
elif key in detailed:
out.append(row | {"description": "", "location": "", "remote": None})
else:
out.append(row | {"description": ""})
if complete:
listed = {compound(row) for row in rows}
done &= listed
failing &= listed
return out, done, failing
listed = {f"{host}:{slug}:{row['source_id']}" for row in rows} if complete else None
return _runner.read_details(readers.HOSTS[host], host, slug, _getter(client, host, slug),
rows, detailed, failed, listed=listed, limits=_limits())
def fetch_board(target: str) -> tuple[list[Row], bool]:
"""Fetch and parse one board directly: no `Source`, no `self.state`, just
"does this board answer, what does it list, and is that all of it". For
`app companies add|check`, which ask that question about one board at a
time outside any pull. The listing only, with no detail reads, and
robots.txt read afresh. Raises the same way `_fetch_rows` does, and
`_Disallowed` or `_RobotsUnreachable` for a board robots.txt shuts out or
that cannot be checked."""
"""Fetch and parse one board directly: no `Source`, no state, just "does
this board answer, what does it list, and is that all of it". For `app
companies add|check`. The listing only, unfiltered, no detail reads.
Raises the same way `_fetch_rows` does."""
with httpx.Client() as client:
_check_robots(client, target)
return _fetch_rows(client, target)
def failure_reason(exc: BaseException) -> str:
"""One short line for why a board did not answer: the status of an HTTP
error, else the exception's class and the first line of its message."""
if isinstance(exc, _Disallowed):
return "robots.txt disallows it"
if isinstance(exc, _RobotsUnreachable):
return "robots.txt could not be read"
if isinstance(exc, httpx.HTTPStatusError):
return f"{exc.response.status_code} {exc.response.reason_phrase}".strip()
lines = str(exc).strip().splitlines()
text = f"{type(exc).__name__}: {lines[0]}" if lines else type(exc).__name__
return text[:200]
failure_reason = _runner.failure_reason
# -------------------------------------------------------------- the plugin
@@ -745,9 +313,8 @@ class ATSSource(Source):
name = "ats"
description = f"Company job boards: {', '.join(HOSTS.values())}."
#: The raw entries under `sources.ats.boards` are `{"url": target, "label":
#: company}` — `url` carries the `host:slug` target and `label` the
#: company name, which is how `Source.targets()`'s generic dict reading
#: hands both back without a plugin-specific override.
#: company, "params": {...}}`: `url` carries the `host:slug` target,
#: `label` the company name and `params` the board's filter.
subscription_field = "boards"
spec = PluginSpec(
@@ -763,99 +330,74 @@ class ATSSource(Source):
)},
)
def _id_set(self, key: str) -> set[str]:
"""A JSON list of ids kept in state, as a set; empty when there is
none or it does not parse."""
try:
return {str(i) for i in json.loads(self.state.get(key) or "[]")}
except (ValueError, TypeError):
return set()
def fetch(self) -> Iterator[Item]:
with httpx.Client() as client:
# Handed in by the batch that runs this plugin, as the marker of
# the pull run that queued it (see `app/jobs/pull.py`), and never
# read from state, where a later run may have replaced it. A
# plugin run on its own — a test, a dry run — has no run to stamp
# itself against, so it makes one up.
# the pull run that queued it (see `app/jobs/pull.py`). A plugin
# run on its own (a test, a dry run) makes one up.
run_marker = str(self.options.get("run") or "") or datetime.now(UTC).isoformat(
timespec="seconds")
for target, label in self.targets():
host, _, slug = target.partition(":")
failing = f"ats:failing:{target}"
try:
_check_robots(client, target, self.state)
rows, complete = _fetch_rows(client, target)
except Exception as exc:
log.warning("ats: %s failed (%s)", target, exc)
# The first failure in a row sets `since`, later ones only
# the reason; `app.jobs.pull` closes the board's roles
# once `since` is `STALE_DAYS` old.
try:
since = json.loads(self.state.get(failing, "") or "{}").get("since")
except ValueError:
since = None
self.state[failing] = json.dumps({
"since": since or datetime.now(UTC).isoformat(timespec="seconds"),
"reason": failure_reason(exc)})
continue
self.state.pop(failing, None)
yield from self._board(client, target, label, run_marker)
detail_keys = (f"ats:detailed:{target}", f"ats:detailfailed:{target}")
detail_sets: tuple[set[str], set[str]] | None = None
if hasattr(readers.HOSTS.get(host), "detail"):
detailed, failed = (self._id_set(key) for key in detail_keys)
rows, detailed, failed = _read_details(client, target, rows, complete,
detailed, failed)
detail_sets = (detailed, failed)
def _board(self, client: httpx.Client, target: str, label: str,
run_marker: str) -> Iterator[Item]:
host, _, slug = target.partition(":")
failing = f"ats:failing:{target}"
try:
if not is_known_target(target):
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
module = readers.HOSTS[host]
get = _getter(client, host, slug)
ask = _runner.ask_for(module, self.params_for(target),
keys=self.options.get("keys"))
listing = _runner.read(self.state, "ats", target, module, slug, get, ask,
limits=_limits())
except Exception as exc:
log.warning("ats: %s failed (%s)", target, exc)
_runner.record_failure(self.state, failing, exc)
return
self.state.pop(failing, None)
seen_ids = []
for row in rows:
# The compound id, matching `roles.source_id` — what
# `Store.mark_closed` compares against, not the board's own
# bare job id.
compound_id = f"{host}:{slug}:{row['source_id']}"
seen_ids.append(compound_id)
yield Item(
source_id=compound_id,
location=row["url"],
title=row["title"],
excerpt=(row["description"] or "")[:300],
body=row["description"],
published_at=row["posted"],
subscription=label,
kind="role",
raw={
"company": label,
"place": row["location"],
"department": row["department"],
"function": classify(row["department"], row["title"]),
"remote": row["remote"],
# Every parser sets salary_text, empty or not (see
# "salary_text" in `readers/_shared.py`); only
# the hosts that state structured pay also set the
# normalised figures, so the rest leave them at
# row.get's default of no figure.
"salary_text": row.get("salary_text", ""),
"salary_min": row.get("salary_min"),
"salary_max": row.get("salary_max"),
"currency": row.get("currency", ""),
"board": host,
"slug": slug,
},
)
# Stamped only once the board has parsed without raising, so a
# board that blew up half way through is neither marked ok nor
# has a partial id list treated as the whole of it. `complete`
# says whether the ids are the whole board, which the pull
# needs before it closes what they leave out. The ids read in
# detail are kept here too, after every row went out, so a
# pull stopped half way reads the rest again rather than
# losing their descriptions.
self.state[f"ats:ok:{target}"] = run_marker
self.state[f"ats:complete:{target}"] = "1" if complete else ""
self.state[f"ats:ids:{target}"] = json.dumps(seen_ids)
if detail_sets is not None:
for key, ids in zip(detail_keys, detail_sets, strict=True):
self.state[key] = json.dumps(sorted(ids))
def compound(source_id: str) -> str:
# Matches `roles.source_id`, which `Store.mark_closed` compares.
return f"{host}:{slug}:{source_id}"
rows = listing.rows
detail_keys = (f"ats:detailed:{target}", f"ats:detailfailed:{target}")
detail_sets: tuple[set[str], set[str]] | None = None
if hasattr(module, "detail"):
detailed, failed = (_runner.id_set(self.state, key) for key in detail_keys)
listed = {compound(i) for i in listing.ids} if listing.complete else None
rows, detailed, failed = _runner.read_details(
module, host, slug, get, rows, detailed, failed, listed=listed,
limits=_limits())
detail_sets = (detailed, failed)
for row in rows:
description = row.get("description") or ""
yield Item(
source_id=compound(row["source_id"]),
location=row["url"],
title=row["title"],
excerpt=description[:300],
body=description,
published_at=row.get("posted"),
subscription=label,
kind="role",
raw=_runner.raw_of(row) | {
"company": label,
"function": classify(row.get("department") or "", row["title"]),
"board": host,
"slug": slug,
},
)
# Stamped only once every row went out, so a board that blew up half
# way through is neither marked ok nor has a partial id list treated
# as the whole of it, and a pull stopped half way reads the rest of
# its details again.
_runner.stamp(self.state, "ats", target, run_marker, listing,
[compound(i) for i in listing.ids])
if detail_sets is not None:
for key, ids in zip(detail_keys, detail_sets, strict=True):
self.state[key] = json.dumps(sorted(ids))
+14 -14
View File
@@ -129,7 +129,8 @@ exists.
**Fetching.** `allowed(slug)` returns a predicate over a lowercased host
name; `_shared.only(host)` is the usual one. Every request goes through
`get`, which is `ats._get` bound to the board:
`get`, which is `_runner.fetch` bound to the target (`_runner.getter`; the
`ats` Source binds it through `ats._get`):
get(url, *, method="GET", payload=None, headers=None) -> bytes
@@ -138,7 +139,7 @@ default User-Agent and `Accept: application/json, text/xml`, and replace
either where they name it. `get` is https only on port 443, never to an IP
address or a local name (`localhost`, `.local`, `.internal`, ...), follows at
most three redirects each checked against `allowed(slug)`, cuts the body at
`ats.MAX_BYTES` and ends the whole request within `ats.DEADLINE`. It returns
`_runner.MAX_BYTES` and ends the whole request within `_runner.DEADLINE`. It returns
the body and raises `httpx.HTTPStatusError` on a non-2xx answer (read
`exc.response.status_code`), `httpx.HTTPError` on a timeout or a connection
failure, and another exception (`ValueError` among them) for a refused host,
@@ -151,8 +152,9 @@ id, to decide completeness), the total the host states or None, and the
cursor of the next page or None after the last. The first call gets cursor
None. A board with a total is complete when as many distinct ids were read;
one without, when the read ended on an empty page. A 429 after the first page
ends the read, incomplete; `ats.MAX_PAGES` pages at most, `ats.PAGE_DELAY`
apart. A paged module may also have `parse(body, slug)` for one page's body.
ends the read, incomplete; `_runner.MAX_PAGES` pages a pull at most,
`_runner.PAGE_DELAY` apart, and a board cut there is swept across pulls
(`_runner.read_listing`). A paged module may also have `parse(body, slug)` for one page's body.
**Rows.** A `Row` (`_contract.Row`) is a dict with `source_id` (the host's
own id for the role, unique on the board), `title`, `url` (http or https,
@@ -183,11 +185,11 @@ keeps it.
**Detail.** `detail(get, slug, row)` reads one role's own page and returns a
dict of the fields it found. The shared code calls it for roles not yet read
(`ats:detailed:<target>`), at most `ats.DETAIL_CAP` (200) a board a pull,
`ats.PAGE_DELAY` apart, the roles never tried before those that failed
(`ats:detailfailed:<target>`). Only `ats.DETAIL_FIELDS` are taken:
`description`, `location`, `remote`, `currency`, `salary_min`, `salary_max`
and `posted`, the keys whose columns survive a later pull that sends the
(`ats:detailed:<target>`), at most `_runner.DETAIL_CAP` (200) a board a
pull, `_runner.PAGE_DELAY` apart, the roles never tried before those that
failed (`ats:detailfailed:<target>`). Only `_runner.DETAIL_FIELDS` are taken:
`description`, `location`, `places`, `teaser`, `remote`, `country`,
`employment`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the
listing's empty value; any other key is left out with a debug log line.
`title`, `url`, `department` and `salary_text` cannot be filled from
`detail`: the listing rewrites them every pull. A role not read in detail
@@ -203,11 +205,9 @@ Whoever builds a reader reads the host's robots.txt for the paths the reader
requests and reports what it disallows, and the owner decides whether the
host is read anyway. `robots` records that decision with its date, e.g.
`"read: /wday/cxs/ allowed (checked 2026-09-29)"` or `"read though /api/ is
disallowed: owner, 2026-09-30"`. The runtime check in `ats.py`, which
`robots_page` and `ROBOTS_ANY_CASE` feed, predates this rule and goes with
the runner; a new reader defines neither. Until then it reads
`https://<host>/robots.txt` at most once a day per board and fails the board
when the page is disallowed for `*` or the app's token.
disallowed: owner, 2026-09-30"`. Nothing reads robots.txt at run time;
`robots_page` and `ROBOTS_ANY_CASE`, which fed the former runtime check, are
read by nothing and a new reader defines neither.
**Discovery.** `probe_url(slug)` is a cheap URL that answers 2xx only when
the slug is a board; `probe_ok(body)` narrows that when a host answers an
+712
View File
@@ -0,0 +1,712 @@
"""The runner: what the app does for every reader, board or search, from what
the reader declares in `READS`.
A reader says how its host answers; this module reads it: the guarded fetch
(`fetch`, bound to one target by `getter`), the `Ask` a subscription resolves
to (`ask_for`) and its series (`series`), the listing whole or paged, one
series per `one`-filter value unioned by id (`read_listing`), completeness,
the sweep of an oversized paged board across pulls, the 10% guard on a
filtered or `empty="suspect"` read (`read`), the per-role detail reads
(`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
bookkeeping (`record_failure`, `failure_reason`). `plugins/ats.py` and
`plugins/search.py` are thin `Source` classes over it: they pass their own
limits (`Limits`) and state prefix (`ats`, `search`), so their state keys stay
as they were.
A module without `READS` is read with the older signatures, `feed_url(slug)`
and `page(get, slug, cursor)`, and never gets an `Ask`.
State keys, under the Source's prefix and the target (`ats:sweep:<target>`):
sweep a paged read cut at `max_pages`: `{"asks", "series", "cursor",
"seen", "rows", "whole"}`, continued by the next pull
count `{"asks", "count"}`, the roles the last whole read listed, for
the 10% guard
facets `{"checked", "facets"}`, a board's `facets`, kept `FACETS_TTL`
"""
from __future__ import annotations
import ipaddress
import itertools
import json
import logging
import os
import re
import time
from collections.abc import Callable, Iterable, Mapping, MutableMapping
from dataclasses import dataclass, field
from datetime import UTC, datetime, timedelta
from types import ModuleType
from typing import Any
import httpx
from desk.sources import USER_AGENT
from ._contract import Ask, Key, Place, Reads
from ._shared import Get, Row
log = logging.getLogger(__name__)
#: openings.go's 8 MB body cap is not kept: every URL fetched here is a feed
#: on a fixed allow-list, and the real run of 2026-09-23 hit 8 MB on the
#: biggest employers (Anduril at 41 MB, Anthropic, OpenAI, Databricks between
#: 9 and 19 MB). 96 MB fits them; a board still over it gets one retry at its
#: reader's `fallback_url` where it has one.
MAX_BYTES = 96 << 20
#: The first byte of a Greenhouse board waited up to 16.6 s in 84 requests
#: measured on 2026-09-29, and the body then arrived in under 1 s. So a read
#: may wait 30 s and a whole request, redirects and body included, 60 s.
TIMEOUTS = httpx.Timeout(10.0, read=30.0)
DEADLINE = 60.0
#: Pages read from one target in one pull at most, and the pause between two
#: requests (pages and detail reads), which keeps Workable from answering 429.
MAX_PAGES = 100
PAGE_DELAY = 0.5
#: Roles read in detail from one board in one pull at most.
DETAIL_CAP = 200
#: The row keys a reader's `detail` may fill: those whose column survives a
#: later pull that sends the listing's empty value, so what a role's own page
#: said is not wiped the pull after it was read. `title`, `url`, `department`
#: and `salary_text` are left out, since the listing rewrites them every pull.
#: Workday, SuccessFactors and Rippling state `country` and `employment` on a
#: role's own page only.
DETAIL_FIELDS = ("description", "location", "places", "teaser", "remote", "country",
"employment", "currency", "salary_min", "salary_max", "posted")
#: How long a board's `facets` are kept before they are asked again.
FACETS_TTL = timedelta(days=1)
#: A filtered or `empty="suspect"` read listing under this share of what the
#: same read listed last time counts as a failure.
SUSPECT_SHARE = 0.1
#: The keys of a row that an `Item` carries in its own fields, not in `raw`.
ITEM_KEYS = ("source_id", "title", "url", "description", "posted")
class Refused(Exception):
"""A request or a redirect aimed at a host its allow-list does not cover."""
class TooLarge(ValueError):
"""A response over the byte cap. A `ValueError`, as it was before it had a
type of its own."""
class Suspect(Exception):
"""A filtered or `empty="suspect"` read that listed under `SUSPECT_SHARE`
of what it listed last time."""
class MissingKey(ValueError):
"""A reader's `READS.key` has no value in the config or the environment."""
@dataclass(frozen=True, slots=True)
class Limits:
"""What one Source allows a read: pages, the pause between requests, and
the detail reads a pull."""
max_pages: int = MAX_PAGES
page_delay: float = PAGE_DELAY
detail_cap: int = DETAIL_CAP
@dataclass(slots=True)
class Listing:
"""One read of a target's listing. `rows` are this pull's rows, `ids` the
`source_id`s the read listed: this pull's, or the whole sweep's once it
reached the end. `complete` says `ids` are the whole list. `sweep` is the
state to keep for the next pull, None when no sweep is under way.
`filtered` says the read asked the host for a part of the list."""
rows: list[Row]
complete: bool
ids: list[str] = field(default_factory=list)
filtered: bool = False
sweep: dict[str, Any] | None = None
asks: str = ""
# --------------------------------------------------------------- modules
def declared(module: ModuleType) -> bool:
"""Whether `module` declares `READS`, and so takes `ask`."""
return isinstance(getattr(module, "READS", None), Reads)
def module_of(name: str) -> ModuleType | None:
"""The reader named `name`, a board host first, else a search provider."""
from plugins import readers
return readers.HOSTS.get(name) or readers.PROVIDERS.get(name)
# ----------------------------------------------------------------- fetch
#: Host name suffixes that name a machine on a private network.
_LOCAL_SUFFIXES = (".localhost", ".local", ".internal", ".lan", ".home.arpa")
def local_host(name: str) -> bool:
"""Whether a host name is an address or a local name rather than a public
DNS name: an IP literal, anything whose last label is a number
(`2130706433`, `127.1` and `0x7f000001` all reach 127.0.0.1), `localhost`
or a private suffix."""
name = name.strip("[]").rstrip(".").lower()
try:
ipaddress.ip_address(name)
return True
except ValueError:
pass
last = name.rsplit(".", 1)[-1]
return (last.isdigit() or bool(re.fullmatch(r"0x[0-9a-f]+", last))
or name == "localhost" or name.endswith(_LOCAL_SUFFIXES))
def fetch(client: httpx.Client, url: str, allow: Callable[[str], bool], *,
method: str = "GET", payload: dict[str, Any] | None = None,
headers: dict[str, str] | None = None, max_bytes: int | None = None,
deadline: float | None = None, name: str = "") -> bytes:
"""Fetch one URL for one target: https on port 443 only, never an address
or a local name, only hosts `allow` accepts, at most three redirects each
checked the same way, the body cut at `max_bytes` (`MAX_BYTES`) and the
whole request ended within `deadline` (`DEADLINE`) seconds. A POST sends
`payload` as JSON, `{}` when None; `headers` are added to the
User-Agent and Accept and replace either where they name it.
The body is streamed and counted as it arrives, and the deadline checked
between chunks, so a trickling answer cannot run on. Raises `Refused`,
`TooLarge`, `httpx.HTTPStatusError` on a non-2xx answer, `httpx.HTTPError`
on a timeout or a connection failure, and ValueError on a broken
redirect."""
cap = MAX_BYTES if max_bytes is None else max_bytes
limit = DEADLINE if deadline is None else deadline
sent = {"User-Agent": USER_AGENT, "Accept": "application/json, text/xml"} | (headers or {})
ends = time.monotonic() + limit
current = url
for _hop in range(4):
parsed = httpx.URL(current)
if (parsed.scheme != "https" or parsed.port not in (None, 443)
or local_host(parsed.host) or not allow(parsed.host)):
raise Refused(f"{current} is not an allowed host for {name or 'this reader'}")
with client.stream(
method, current, timeout=TIMEOUTS, follow_redirects=False, headers=sent,
json=(payload or {}) if method == "POST" else None,
) as response:
if response.is_redirect:
location = response.headers.get("location") or ""
if not location:
raise ValueError("redirect with no location")
current = str(httpx.URL(current).join(location))
continue
response.raise_for_status()
chunks: list[bytes] = []
size = 0
for chunk in response.iter_bytes():
size += len(chunk)
if size > cap:
raise TooLarge(f"response too large (over {cap} bytes)")
if time.monotonic() > ends:
raise httpx.ReadTimeout(f"no whole answer within {limit:.0f} s",
request=response.request)
chunks.append(chunk)
return b"".join(chunks)
raise ValueError("too many redirects")
def getter(client: httpx.Client, allow: Callable[[str], bool], name: str = "") -> Get:
"""The `get` a reader is handed: `fetch` bound to one target's allow-list."""
def get(url: str, *, method: str = "GET", payload: dict[str, Any] | None = None,
headers: dict[str, str] | None = None) -> bytes:
return fetch(client, url, allow, method=method, payload=payload, headers=headers,
name=name)
return get
# ------------------------------------------------------------------- ask
#: A filter dimension of `Filters` -> the field of `Ask` it fills.
_ASK_FIELD = {"country": "countries", "place": "places", "remote": "remote", "query": "query"}
#: Keys a subscription's params or a reach may use, -> the dimension.
_PARAM_DIMENSION = {"countries": "country", "country": "country", "places": "place",
"place": "place", "remote": "remote", "query": "query", "q": "query"}
def _values(value: Any) -> tuple[str, ...]:
"""A param's value as a tuple of non-empty strings."""
items = value if isinstance(value, list | tuple | set) else [value]
return tuple(str(v).strip() for v in items if v not in (None, "", 0) and str(v).strip())
def _dimensions(module: ModuleType, given: Mapping[str, Any]) -> dict[str, Any]:
"""`dimension -> value` from a params or reach mapping: the generic keys
(`countries`, `places`, `remote`, `q`) and the reader's own `PARAMS`
(saved param -> dimension, e.g. Jobindex's `area` -> `place`)."""
names = _PARAM_DIMENSION | dict(getattr(module, "PARAMS", {}) or {})
out: dict[str, Any] = {}
for key, value in given.items():
dimension = names.get(key)
if dimension == "remote":
if isinstance(value, bool):
out["remote"] = value
elif dimension == "query":
if str(value or "").strip():
out["query"] = str(value).strip()
elif dimension in ("country", "place"):
values = _values(value)
if dimension == "country":
values = tuple(v.upper() for v in values)
if values:
out[dimension] = values
return out
def resolve_key(key: Key, keys: Mapping[str, Any] | None) -> dict[str, str]:
"""A reader's key: each of `key.fields` from the config entry
`sources.search.<key.name>` (`keys` is `sources.search`), else from the
environment variable at the same position in `key.env`. Raises
`MissingKey` naming both places when a field has neither."""
entry = (keys or {}).get(key.name) or {}
out: dict[str, str] = {}
for index, name in enumerate(key.fields):
env = key.env[index] if index < len(key.env) else ""
value = str((entry.get(name) if isinstance(entry, Mapping) else "") or "") \
or (os.environ.get(env, "") if env else "")
if not value:
where = f"sources.search.{key.name}: {', '.join(key.fields)}"
if key.env:
where += f", or {' and '.join(key.env)}"
raise MissingKey(f"{key.name} needs a key: {where}")
out[name] = value
return out
def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None = None,
reach: Mapping[str, Any] | None = None, *,
keys: Mapping[str, Any] | None = None) -> Ask:
"""The `Ask` one target resolves to. `reach` (from the agents, package F)
and then the subscription's own `params` (a board's override, e.g.
`{"countries": ["DK"], "remote": true}`, or a search's saved params) set
the dimensions, params winning per dimension, narrowed to the ones the
reader declares in `READS.filters`. A search's params go whole into
`Ask.params`; a reader with `READS.key` gets the resolved key under
`Ask.params["key"]` (see `resolve_key`, which raises when it is missing).
A module without `READS` gets `Ask()`."""
if not declared(module):
return Ask()
reads: Reads = module.READS
params = dict(subscription_params or {})
wanted = _dimensions(module, reach or {}) | _dimensions(module, params)
fields: dict[str, Any] = {_ASK_FIELD[name]: value for name, value in wanted.items()
if name in reads.filters.declared()}
extra: dict[str, Any] = dict(params) if reads.scope == "search" else {}
if reads.key is not None:
extra["key"] = resolve_key(reads.key, keys)
return Ask(**fields, params=extra)
def filtered(ask: Ask) -> bool:
"""Whether `ask` reads a part of a board: a country, a place or remote."""
return bool(ask.countries or ask.places or ask.remote is not None)
def series(module: ModuleType, ask: Ask,
facets: Mapping[str, Mapping[str, str]] | None = None) -> list[Ask]:
"""The series of requests `ask` takes on this reader: one per value of a
dimension declared `one` (the product when two are), one with every
value for `many`, and host ids from `facets` for `ids` (values `facets`
does not know are dropped; an empty list when none is left, since the
host then has nothing in reach). `[ask]` for a module without `READS`."""
if not declared(module):
return [ask]
reach = module.READS.filters
countries, places = ask.countries, ask.places
for dimension in ("country", "place"):
values = countries if dimension == "country" else places
if getattr(reach, dimension) != "ids" or not values:
continue
known = (facets or {}).get(dimension) or {}
mapped = tuple(known[v] for v in values if v in known)
if not mapped:
log.info("readers: %s knows none of %s %s", module.NAME, dimension,
", ".join(values))
return []
if dimension == "country":
countries = mapped
else:
places = mapped
by_country = [(c,) for c in countries] if reach.country == "one" and countries \
else [countries]
by_place = [(p,) for p in places] if reach.place == "one" and places else [places]
return [ask.model_copy(update={"countries": c, "places": p})
for c, p in itertools.product(by_country, by_place)]
def fingerprint(asks: Iterable[Ask]) -> str:
"""What a read asked, without its params (which may hold a key): the same
string for the same series, so a sweep or a count is only compared with
a read that asked the same."""
return json.dumps([ask.model_dump(exclude={"params"}, mode="json") for ask in asks],
sort_keys=True)
# --------------------------------------------------------------- listing
def _result_ids(results: Iterable[dict[str, Any]]) -> set[str]:
return {str(r.get("id") or r.get("shortcode") or "") for r in results}
def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
limits: Limits) -> dict[str, Row]:
"""A whole listing, one request a series, unioned by `source_id`. A body
over the cap gets one retry at the reader's `fallback_url`."""
rows: dict[str, Row] = {}
for index, ask in enumerate(asks):
if index:
time.sleep(limits.page_delay)
url = module.feed_url(slug, ask) if declared(module) else module.feed_url(slug)
try:
body = get(url)
except TooLarge:
if not hasattr(module, "fallback_url"):
raise
log.warning("readers: %s:%s: body over the cap; reading the smaller feed",
module.NAME, slug)
body = get(module.fallback_url(slug))
for row in module.parse(body, slug):
rows.setdefault(row["source_id"], row)
return rows
def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None, *,
limits: Limits | None = None, sweep: Mapping[str, Any] | None = None,
can_sweep: bool = False,
facets: Mapping[str, Mapping[str, str]] | None = None) -> Listing:
"""One target's listing, read and parsed. Raises on any failure of the
first request, and on a reader that does not understand an answer.
A whole listing is complete whenever it parses. A paged one reads each
series' pages, `limits.page_delay` apart, until the host has no next
page, the stated total is reached, a page brings nothing new (a host
that ignores the cursor, incomplete) or `limits.max_pages` pages in all.
A series is complete when it read as many distinct result ids as its
stated total, or, without a total, when it ended on the host's last
page; never for `complete="never"`. The read is complete when every
series is. A 429 after the first page ends the read, incomplete.
With `can_sweep` (boards), a read cut at `max_pages` or by a 429 returns
the `sweep` to keep, and the next pull, handed it back, continues from
its series and cursor; the sweep is complete when it reaches the end,
and `ids` then hold every id it listed. A read at the reader's `ceiling`
is not swept: the host answers nothing past it."""
limits = limits or Limits()
ask = ask or Ask()
asks = series(module, ask, facets)
is_filtered = filtered(ask)
key = fingerprint(asks)
if not asks:
return Listing([], True, [], is_filtered, None, key)
if not hasattr(module, "page"):
rows = _whole(module, slug, get, asks, limits)
return Listing(list(rows.values()), True, list(rows), is_filtered, None, key)
reads: Reads | None = module.READS if declared(module) else None
never = reads is not None and reads.complete == "never"
ceiling = reads.ceiling if reads is not None else None
start, cursor, seen, earlier, whole = 0, None, set(), [], True
if sweep and sweep.get("asks") == key:
start, cursor = int(sweep.get("series") or 0), sweep.get("cursor")
seen, earlier = set(sweep.get("seen") or []), list(sweep.get("rows") or [])
whole = bool(sweep.get("whole", True))
rows: dict[str, Row] = {}
pages = 0
stop: str = ""
index = start
for index in range(start, len(asks)):
total: Any = None
ended = stalled = False
while True:
if pages >= limits.max_pages:
stop = "cap"
break
if pages:
time.sleep(limits.page_delay)
try:
if reads is not None:
data, results, total, cursor = module.page(get, slug, cursor, asks[index])
else:
data, results, total, cursor = module.page(get, slug, cursor)
except httpx.HTTPStatusError as exc:
if not pages or exc.response.status_code != 429:
raise
log.warning("readers: %s:%s: 429 on page %d; kept %d, not complete",
module.NAME, slug, pages + 1, len(rows))
stop = "429"
break
pages += 1
before = len(seen)
seen |= _result_ids(results)
for row in module.rows(data, slug):
rows.setdefault(row["source_id"], row)
if cursor is None or (isinstance(total, int) and len(seen) >= total):
ended = True
break
if total is None and len(seen) == before:
stalled = True
break
if stop:
break
done = (not never and not stalled
and (len(seen) >= total if isinstance(total, int) else ended))
if not done:
log.warning("readers: %s:%s: read %d of %s; not complete", module.NAME, slug,
len(seen), "an unstated total" if total is None else total)
whole = whole and done
cursor, seen = None, set()
listed = list(dict.fromkeys(earlier + list(rows)))
if not stop:
return Listing(list(rows.values()), whole, listed, is_filtered, None, key)
log.warning("readers: %s:%s: stopped after %d pages (%s); not complete",
module.NAME, slug, pages, stop)
at_ceiling = ceiling is not None and len(seen) >= ceiling
if not can_sweep or never or at_ceiling or not _storable(cursor):
return Listing(list(rows.values()), False, list(rows), is_filtered, None, key)
keep = {"asks": key, "series": index, "cursor": cursor, "seen": sorted(seen),
"rows": listed, "whole": whole}
return Listing(list(rows.values()), False, list(rows), is_filtered, keep, key)
def _storable(cursor: Any) -> bool:
"""Whether a cursor survives a trip through JSON in state."""
try:
return json.loads(json.dumps(cursor)) == cursor
except (TypeError, ValueError):
return False
# ------------------------------------------------------------ with state
def _load(state: Mapping[str, str], key: str) -> dict[str, Any]:
try:
value = json.loads(state.get(key) or "{}")
except (TypeError, ValueError):
return {}
return value if isinstance(value, dict) else {}
def facets_for(module: ModuleType, slug: str, get: Get, ask: Ask,
state: MutableMapping[str, str] | None = None, key: str = "") -> dict[str, Any]:
"""The reader's `facets` when `ask` sets a dimension it declares `ids`,
kept in `state[key]` for `FACETS_TTL`; `{}` otherwise."""
if not declared(module) or not hasattr(module, "facets"):
return {}
ids = [name for name, reach in module.READS.filters.declared().items() if reach == "ids"]
if not any(getattr(ask, _ASK_FIELD[name]) for name in ids if name in ("country", "place")):
return {}
now = datetime.now(UTC)
if state is not None and key:
kept = _load(state, key)
try:
if now - datetime.fromisoformat(kept["checked"]) < FACETS_TTL:
return dict(kept.get("facets") or {})
except (KeyError, TypeError, ValueError):
pass
found = dict(module.facets(get, slug) or {})
if state is not None and key:
state[key] = json.dumps({"checked": now.isoformat(timespec="seconds"),
"facets": found})
return found
def read(state: MutableMapping[str, str], prefix: str, target: str, module: ModuleType,
slug: str, get: Get, ask: Ask | None = None, *,
limits: Limits | None = None, can_sweep: bool = True) -> Listing:
"""`read_listing` for one target with its state under `<prefix>:...:<target>`:
the facets it needs, the sweep it continues, and the 10% guard. A read
that is filtered or whose reader declares `empty="suspect"`, and that
listed under `SUSPECT_SHARE` of what the same read listed last time,
raises `Suspect`. Nothing is written here but the facets; `stamp` keeps
the rest once the rows are stored."""
ask = ask or Ask()
facets = facets_for(module, slug, get, ask, state, f"{prefix}:facets:{target}")
listing = read_listing(module, slug, get, ask, limits=limits, can_sweep=can_sweep,
sweep=_load(state, f"{prefix}:sweep:{target}"), facets=facets)
suspect = declared(module) and module.READS.empty == "suspect"
last = _load(state, f"{prefix}:count:{target}")
count = int(last.get("count") or 0) if last.get("asks") == listing.asks else 0
if ((listing.filtered or suspect) and listing.sweep is None and count
and len(listing.ids) < SUSPECT_SHARE * count):
raise Suspect(f"listed {len(listing.ids)} roles, under "
f"{SUSPECT_SHARE:.0%} of the {count} it listed last time")
return listing
def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
listing: Listing, ids: list[str]) -> None:
"""Record a read whose rows are stored: `ok` (the run's marker),
`complete`, `ids` (the compound ids, what the pull closes against),
`filtered`, the sweep to continue or none, and the count for the 10%
guard once a read is whole."""
state[f"{prefix}:ok:{target}"] = run
state[f"{prefix}:complete:{target}"] = "1" if listing.complete else ""
state[f"{prefix}:ids:{target}"] = json.dumps(ids)
state[f"{prefix}:filtered:{target}"] = "1" if listing.filtered else ""
if listing.sweep is not None:
state[f"{prefix}:sweep:{target}"] = json.dumps(listing.sweep)
else:
state.pop(f"{prefix}:sweep:{target}", None)
state[f"{prefix}:count:{target}"] = json.dumps({"asks": listing.asks,
"count": len(listing.ids)})
# ---------------------------------------------------------------- details
def id_set(state: Mapping[str, str], key: str) -> set[str]:
"""A JSON list of ids kept in state, as a set; empty when there is none
or it does not parse."""
try:
return {str(i) for i in json.loads(state.get(key) or "[]")}
except (TypeError, ValueError):
return set()
def read_details(module: ModuleType, name: str, slug: str, get: Get, rows: list[Row],
detailed: set[str], failed: set[str], *, listed: set[str] | None = None,
limits: Limits | None = None) -> tuple[list[Row], set[str], set[str]]:
"""The rows with the roles not yet read in detail filled from their own
pages, at most `limits.detail_cap`, `limits.page_delay` apart: first the
roles never tried, then those in `failed`. Ids are compound,
`<name>:<slug>:<source_id>`. Returns the rows and the ids read and
failed, to keep as the caller's `detailed` and `failed` sets.
Only `DETAIL_FIELDS` are taken from what `detail` returns. A `location`
without `places` becomes the one place. Every row not read now goes out
with an empty description, so one read earlier stays, and a row read on
an earlier pull without the listing's place and remote, so what its page
said stays. A 404 or 410 counts as read, a 429 stops the reads for this
pull, anything else keeps the role in `failed`. `listed` is every id of a
complete listing: ids outside it leave both sets."""
limits = limits or Limits()
done, failing = set(detailed), set(failed) - set(detailed)
def compound(row: Row) -> str:
return f"{name}:{slug}:{row['source_id']}"
unread = [row for row in rows if compound(row) not in done]
queue = ([row for row in unread if compound(row) not in failing]
+ [row for row in unread if compound(row) in failing])[:limits.detail_cap]
filled: dict[str, Row] = {}
for row in queue:
key = compound(row)
time.sleep(limits.page_delay)
try:
answer = module.detail(get, slug, row)
except httpx.HTTPStatusError as exc:
code = exc.response.status_code
if code == 429:
log.warning("readers: %s:%s: 429 on a detail read; the rest wait", name, slug)
break
if code in (404, 410):
done.add(key)
failing.discard(key)
else:
log.warning("readers: %s:%s: detail of %s answered %s", name, slug,
row["source_id"], code)
failing.add(key)
continue
except Exception as exc:
log.warning("readers: %s:%s: detail of %s failed (%s)", name, slug,
row["source_id"], exc)
failing.add(key)
continue
if ignored := sorted(set(answer) - set(DETAIL_FIELDS)):
log.debug("readers: %s:%s: detail of %s: %s left out, not in DETAIL_FIELDS",
name, slug, row["source_id"], ", ".join(ignored))
taken: Row = {k: v for k, v in answer.items() if k in DETAIL_FIELDS} # type: ignore[assignment]
if taken.get("location") and "places" not in taken:
taken["places"] = [{"text": taken["location"], "remote": None}]
filled[key] = taken
done.add(key)
failing.discard(key)
out: list[Row] = []
for row in rows:
key = compound(row)
if key in filled:
out.append(row | {"description": ""} | filled[key]) # type: ignore[operator]
elif key in detailed:
out.append(row | {"description": "", "location": "", "places": [], # type: ignore[operator]
"remote": None})
else:
out.append(row | {"description": ""}) # type: ignore[operator]
if listed is not None:
done &= listed
failing &= listed
return out, done, failing
# -------------------------------------------------------------------- raw
def places_of(row: Mapping[str, Any]) -> list[Place]:
"""A row's places: its `places`, or, for a row without (an undeclared
reader's), its `location` as the one place, split nothing."""
places = row.get("places")
if not isinstance(places, list):
location = str(row.get("location") or "").strip()
return [{"text": location, "remote": None}] if location else []
out: list[Place] = []
for place in places:
if isinstance(place, Mapping) and str(place.get("text") or "").strip():
out.append({"text": str(place["text"]).strip(), "remote": place.get("remote")})
return out
def raw_of(row: Mapping[str, Any]) -> dict[str, Any]:
"""The `raw` of an `Item` for one row: every key the row states except
those the `Item` carries itself (`ITEM_KEYS`), with `places` always a
list, `place` the display line of the places joined with "; ", and the
keys storage reads always present (`department`, `remote`,
`salary_text`, `salary_min`, `salary_max`, `currency`, `teaser`)."""
places = places_of(row)
raw = {k: v for k, v in row.items() if k not in ITEM_KEYS}
return raw | {
"places": places,
"place": "; ".join(place["text"] for place in places),
"department": row.get("department") or "",
"remote": row.get("remote"),
"salary_text": row.get("salary_text") or "",
"salary_min": row.get("salary_min"),
"salary_max": row.get("salary_max"),
"currency": row.get("currency") or "",
"teaser": row.get("teaser") or "",
}
# --------------------------------------------------------------- failures
def failure_reason(exc: BaseException) -> str:
"""One short line for why a target did not answer: the status of an HTTP
error, else the exception's class and the first line of its message."""
if isinstance(exc, httpx.HTTPStatusError):
return f"{exc.response.status_code} {exc.response.reason_phrase}".strip()
lines = str(exc).strip().splitlines()
text = f"{type(exc).__name__}: {lines[0]}" if lines else type(exc).__name__
return text[:200]
def record_failure(state: MutableMapping[str, str], key: str, exc: BaseException) -> None:
"""Keep a failure under `key` as `{"since", "reason"}`: the first failure
in a row sets `since`, later ones only the reason. The pull closes a
target's roles once `since` is old enough; its next answer removes the
key."""
since = _load(state, key).get("since")
state[key] = json.dumps({"since": since or datetime.now(UTC).isoformat(timespec="seconds"),
"reason": failure_reason(exc)})
+64 -381
View File
@@ -5,421 +5,109 @@ roles at companies nobody has listed yet, by asking a cross-company board a
saved question: "backend developer" in Region Hovedstaden, "python" in Berlin.
A subscription is one saved search; its `params` are the question.
Three boards, checked live on 2026-09-23 and, for Platsbanken, 2026-09-29:
jobindex Denmark's largest job board. Every search result page has an RSS
twin at /jobsoegning.rss, no key needed. `q` takes Jobindex's
own syntax: words are OR, `"a b"` a phrase, `+word` required.
`area` is its `geoareaid`: 1 Region Hovedstaden, 4 Region
Sjælland, 2 Midtjylland, 5 Syddanmark, 3 Nordjylland, 8 Aarhus.
20 results a page.
adzuna An aggregator with a documented API for most of Europe, the UK
and the US, but not Denmark (adzuna.dk does not exist). Needs a
free key from developer.adzuna.com, as `sources.search.adzuna:
{app_id, app_key}` or `ADZUNA_APP_ID`/`ADZUNA_APP_KEY`. Built
against Adzuna's documented response and not yet run live.
platsbanken Sweden's public employment service, through the JobTech search
API at jobsearch.api.jobtechdev.se, no key needed. `region` is
the id of a län in the JobTech taxonomy (Skåne län is
`CaRE_1nn_cSU`), empty for all of Sweden. 100 results a page,
and an `offset` above 2000 is refused. It states `total`. It has
no remote flag worth reading (`workplace_model` is mostly
empty), so `remote` comes from the headline.
Considered and left out: Jobnet (its search endpoint answers 401 outside the
browser session of its own site) and Jooble (a key needs a manual
application, and its API host answered a Cloudflare challenge).
The providers are the readers of `scope="search"` in `plugins/readers/`
(`readers.PROVIDERS`), read by `plugins/readers/_runner.py` like any other
reader: the search's params go in `Ask.params`, a declared key is resolved
from `sources.search.<name>` or the environment and refused when missing,
and the saved `pages` param bounds the pages read (`DEFAULT_PAGES`, at most
`MAX_PAGES`).
A search result list is never a board's complete list, so a role missing from
one run is not closed here. A search that read everything that matches was
complete, and the pull closes what it stopped listing; the rest close once no
search has seen them for `STALE_DAYS` (see `app/jobs/pull.py`). Adzuna and
Platsbanken state how many results match, so their searches are complete when
they read that many; Jobindex states no count, so a Jobindex search is
complete when a page within the page limit came back short.
complete (`complete="stated"`, as many distinct results as the stated total),
and the pull closes what it stopped listing; the rest close once no search
has seen them for `STALE_DAYS` (see `app/jobs/pull.py`).
"""
from __future__ import annotations
import html
import json
import logging
import os
import re
import xml.etree.ElementTree as ET
from collections.abc import Callable, Iterator
import sys
from collections.abc import Iterator, Mapping
from datetime import UTC, datetime
from email.utils import parsedate_to_datetime
from pathlib import Path
from types import ModuleType
from typing import Any
import httpx
from desk.sources import (
API_VERSION, ARTICLE_READER, Item, PluginSpec, Profile, Source,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks, USER_AGENT, strip_html,
SummaryPolicy, TaggingPolicy, TargetSpec, Tasks,
)
try:
from plugins import readers
except ModuleNotFoundError:
# Loaded by path by `desk.sources.registry`; see the same lines in `ats.py`.
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from plugins import readers
from plugins.readers import _runner
from plugins.readers._shared import Row
log = logging.getLogger(__name__)
TIMEOUT = 15.0
#: Pages fetched per search unless its params say otherwise, and the most a
#: Pages read per search unless its params say otherwise, and the most a
#: search may ask for. A search that needs more than ten pages is too broad to
#: read anyway.
DEFAULT_PAGES = 5
MAX_PAGES = 10
PROVIDERS = ("jobindex", "adzuna", "platsbanken")
#: What every provider turns a result into.
Row = dict[str, Any]
#: The pause between two pages of one search.
PAGE_DELAY = _runner.PAGE_DELAY
def _web_url(url: str) -> bool:
parsed = httpx.URL((url or "").strip())
return parsed.scheme in ("http", "https") and bool(parsed.host)
def _pages(params: dict[str, Any]) -> int:
def _pages(params: Mapping[str, Any]) -> int:
try:
return max(1, min(int(params.get("pages") or DEFAULT_PAGES), MAX_PAGES))
except (TypeError, ValueError):
return DEFAULT_PAGES
def _get(client: httpx.Client, url: str, params: dict[str, Any]) -> bytes:
response = client.get(url, params=params, timeout=TIMEOUT, follow_redirects=True,
headers={"User-Agent": USER_AGENT})
response.raise_for_status()
return response.content
# ---------------------------------------------------------------- jobindex
JOBINDEX_RSS = "https://www.jobindex.dk/jobsoegning.rss"
JOBINDEX_PAGE = 20
#: Jobindex's `geoareaid` for the regions and the biggest towns, as checked
#: against the titles its feeds answer with. 0 is all of Denmark.
JOBINDEX_AREAS: dict[int, str] = {
0: "All of Denmark", 1: "Region Hovedstaden", 4: "Region Sjælland",
2: "Region Midtjylland", 5: "Region Syddanmark", 3: "Region Nordjylland",
8: "Aarhus", 7: "Aalborg",
}
_ALT = re.compile(r'\balt="([^"]+)"', re.IGNORECASE)
_H4 = re.compile(r"<h4>(.*?)</h4>", re.IGNORECASE | re.DOTALL)
_AREA = re.compile(r'class="jix_robotjob--area">(.*?)</span>', re.IGNORECASE | re.DOTALL)
_PARAGRAPHS = re.compile(r"<p>(.*?)</p>", re.IGNORECASE | re.DOTALL)
#: "Ledige job i Region Hovedstaden indeholdende ..." -> "Region Hovedstaden".
_SEARCH_AREA = re.compile(r"Ledige job i (.+?) indeholdende")
#: What a Danish ad's town line says about where the work is done:
#: "Birkerød, Hybrid position", "Nordhavn og mulighed for hjemmearbejde".
_HYBRID = re.compile(r"\bhybrid|hjemmearbejde", re.IGNORECASE)
_REMOTE = re.compile(r"\bremote\b|fjernarbejde|fuldt hjemmearbejde|distansarbete|på distans", re.IGNORECASE)
def _remote(text: str) -> str | None:
if _REMOTE.search(text):
return "remote"
return "hybrid" if _HYBRID.search(text) else None
def _text(fragment: str) -> str:
"""HTML to plain text. Jobindex breaks titles after a slash with a
zero-width space, which would otherwise end up in `title_key`."""
return html.unescape(strip_html(fragment)).replace("\u200b", "").strip()
def parse_jobindex(body: bytes) -> list[Row]:
return _jobindex_page(body)[0]
def _jobindex_page(body: bytes) -> tuple[list[Row], int]:
"""One page of a Jobindex RSS search, and how many items it had before
any was dropped. An item is `<title>` "Role, Company",
a `<link>` to Jobindex's own page for it, and a `<description>` of HTML
whose `<h4>` is the title alone, whose logo's `alt` is the company on a
paid ad, and whose `jix_robotjob--area` span is the town. A crawled ad
("robot job") has no logo, so its company is what follows the title's
last comma, or '' when there is none. A body with no `<channel>` is not
a search result and raises, since an empty list would read as a search
that ran out and close everything it listed before."""
root = ET.fromstring(body)
channel = root.find("channel")
if channel is None:
raise ValueError("jobindex answered without an RSS channel")
area_match = _SEARCH_AREA.search(channel.findtext("title") or "")
search_area = area_match.group(1).strip() if area_match else ""
rows = []
items = channel.findall("item")
for item in items:
link = (item.findtext("link") or "").strip()
full_title = (item.findtext("title") or "").strip()
if not full_title or not _web_url(link):
continue
description = item.findtext("description") or ""
h4 = _H4.search(description)
title = _text(h4.group(1)) if h4 else full_title
alt = _ALT.search(description)
company = html.unescape(alt.group(1)).strip() if alt else ""
if not company and full_title.startswith(title + ", "):
company = full_title[len(title) + 2:].strip()
area = _AREA.search(description)
posted = None
if pub := item.findtext("pubDate"):
try:
posted = parsedate_to_datetime(pub)
except (TypeError, ValueError):
posted = None
place = _text(area.group(1)) if area else search_area
rows.append({
"id": link.rstrip("/").rsplit("/", 1)[-1],
"title": title or full_title,
"company": company,
"place": place,
"country": "DK",
"remote": _remote(f"{place} {title}"),
"url": link,
"posted": posted,
"description": " ".join(_text(p) for p in _PARAGRAPHS.findall(description)),
"salary_min": None, "salary_max": None, "currency": "",
})
return rows, len(items)
def search_jobindex(client: httpx.Client, params: dict[str, Any],
keys: dict[str, Any]) -> tuple[list[Row], bool]:
"""Every page of one search, newest first, until a short page or the page
limit. Returns the rows and whether the results ran out: a page with
fewer than `JOBINDEX_PAGE` items, counted before the parser drops any, so
a dropped item does not pass for the end. Jobindex states no total, so a
search that read the page limit full is never complete."""
query: dict[str, Any] = {"q": str(params.get("q") or ""), "sort": "date"}
if params.get("area"):
query["geoareaid"] = params["area"]
rows: list[Row] = []
for page in range(1, _pages(params) + 1):
got, items = _jobindex_page(_get(client, JOBINDEX_RSS, query | {"page": page}))
rows += got
if items < JOBINDEX_PAGE:
return rows, True
return rows, False
# ------------------------------------------------------------------ adzuna
ADZUNA_SEARCH = "https://api.adzuna.com/v1/api/jobs/{country}/search/{page}"
ADZUNA_PAGE = 50
#: Adzuna's countries in Europe, and the currency its salaries are in.
ADZUNA_COUNTRIES = {"gb": "GBP", "de": "EUR", "nl": "EUR", "fr": "EUR", "at": "EUR",
"be": "EUR", "es": "EUR", "it": "EUR", "pl": "PLN", "ch": "CHF"}
def parse_adzuna(body: bytes, country: str) -> list[Row]:
"""One page of Adzuna's `/search`: `{"count", "results": [...]}`. Titles
and snippets carry `<strong>` highlights. A salary Adzuna estimated
(`salary_is_predicted`) is not the posting's, so it is dropped."""
return _adzuna_rows(json.loads(body), country)
def _adzuna_rows(data: dict[str, Any], country: str) -> list[Row]:
rows = []
for job in data.get("results") or []:
title = _text(job.get("title") or "")
url = (job.get("redirect_url") or "").strip()
if not title or not _web_url(url):
continue
predicted = str(job.get("salary_is_predicted") or "0") == "1"
rows.append({
"id": str(job.get("id") or ""),
"title": title,
"company": ((job.get("company") or {}).get("display_name") or "").strip(),
"place": ((job.get("location") or {}).get("display_name") or "").strip(),
"country": country.upper(),
"remote": _remote(title),
"url": url,
"posted": _parse_iso(job.get("created")),
"description": _text(job.get("description") or ""),
"salary_min": None if predicted else job.get("salary_min"),
"salary_max": None if predicted else job.get("salary_max"),
"currency": "" if predicted else ADZUNA_COUNTRIES.get(country, ""),
})
return rows
def _parse_iso(value: Any) -> datetime | None:
try:
parsed = datetime.fromisoformat(str(value).replace("Z", "+00:00"))
except ValueError:
return None
return parsed if parsed.utcoffset() is not None else parsed.replace(tzinfo=UTC)
def search_adzuna(client: httpx.Client, params: dict[str, Any],
keys: dict[str, Any]) -> tuple[list[Row], bool]:
"""Every page of one search, until a short page, the stated `count` or
the page limit. Returns the rows and whether they are all `count` says
match, or, on an answer with no `count`, whether a page came back short."""
app_id = keys.get("app_id") or os.environ.get("ADZUNA_APP_ID", "")
app_key = keys.get("app_key") or os.environ.get("ADZUNA_APP_KEY", "")
if not (app_id and app_key):
raise ValueError("adzuna needs sources.search.adzuna.app_id and app_key, "
"or ADZUNA_APP_ID and ADZUNA_APP_KEY")
country = str(params.get("country") or "").lower()
if country not in ADZUNA_COUNTRIES:
raise ValueError(f"adzuna country {country!r} is not one of "
f"{', '.join(sorted(ADZUNA_COUNTRIES))}")
query: dict[str, Any] = {"app_id": app_id, "app_key": app_key,
"results_per_page": ADZUNA_PAGE, "sort_by": "date",
"what": str(params.get("q") or "")}
if params.get("where"):
query["where"] = params["where"]
rows: list[Row] = []
seen: set[str] = set()
count: Any = None
ran_out = False
for page in range(1, _pages(params) + 1):
url = ADZUNA_SEARCH.format(country=country, page=page)
data = json.loads(_get(client, url, query))
results = data.get("results") or []
count = data.get("count", count)
seen |= {str(job.get("id") or "") for job in results}
rows += _adzuna_rows(data, country)
if len(results) < ADZUNA_PAGE:
ran_out = True
break
if isinstance(count, int) and len(seen) >= count:
break
# Adzuna states how many results match; the search is complete when it
# read that many distinct ones. Without a count, a short page is the end.
return rows, (len(seen) == count) if isinstance(count, int) else ran_out
PLATSBANKEN_SEARCH = "https://jobsearch.api.jobtechdev.se/search"
PLATSBANKEN_PAGE = 100
#: The API refuses an `offset` above 2000, which no page limit reaches.
PLATSBANKEN_MAX_OFFSET = 2000
#: The 21 Swedish län as concept ids of the JobTech taxonomy, from its `region`
#: concepts (which also list regions outside Sweden). "" is all of Sweden and
#: is not a key here.
PLATSBANKEN_REGIONS: dict[str, str] = {
"DQZd_uYs_oKb": "Blekinge län", "oDpK_oZ2_WYt": "Dalarnas län",
"K8iD_VQv_2BA": "Gotlands län", "zupA_8Nt_xcD": "Gävleborgs län",
"wjee_qH2_yb6": "Hallands län", "65Ms_7r1_RTG": "Jämtlands län",
"MtbE_xWT_eMi": "Jönköpings län", "9QUH_2bb_6Np": "Kalmar län",
"tF3y_MF9_h5G": "Kronobergs län", "9hXe_F4g_eTG": "Norrbottens län",
"CaRE_1nn_cSU": "Skåne län", "CifL_Rzy_Mku": "Stockholms län",
"s93u_BEb_sx2": "Södermanlands län", "zBon_eET_fFU": "Uppsala län",
"EVVp_h6U_GSZ": "Värmlands län", "g5Tt_CAV_zBd": "Västerbottens län",
"NvUF_SP1_1zo": "Västernorrlands län", "G6DV_fKE_Viz": "Västmanlands län",
"zdoY_6u5_Krt": "Västra Götalands län", "xTCk_nT5_Zjm": "Örebro län",
"oLT3_Q9p_3nn": "Östergötlands län",
}
def parse_platsbanken(body: bytes) -> list[Row]:
"""One page of JobTech's `/search`: `{"total": {"value"}, "hits": [...]}`.
A hit's `webpage_url` is Platsbanken's own page for the ad. `remote` is
read from the headline and from `workplace_model`, which the ads mostly
leave empty."""
return _platsbanken_rows(json.loads(body))
def _platsbanken_rows(data: dict[str, Any]) -> list[Row]:
rows = []
for hit in data.get("hits") or []:
title = _text(hit.get("headline") or "")
url = (hit.get("webpage_url") or "").strip()
if not title or not _web_url(url):
continue
address = hit.get("workplace_address") or {}
model = (hit.get("workplace_model") or {}).get("label") or ""
rows.append({
"id": str(hit.get("id") or ""),
"title": title,
"company": ((hit.get("employer") or {}).get("name") or "").strip(),
"place": (address.get("municipality") or address.get("region") or "").strip(),
"country": "SE",
"remote": _remote(f"{title} {model}"),
"url": url,
"posted": _parse_iso(hit.get("publication_date")),
"description": _text((hit.get("description") or {}).get("text") or ""),
"salary_min": None, "salary_max": None, "currency": "",
})
return rows
def search_platsbanken(client: httpx.Client, params: dict[str, Any],
keys: dict[str, Any]) -> tuple[list[Row], bool]:
"""Every page of one search, newest first, until a short page, the stated
total or the page limit. Returns the rows and whether they are all the
`total` says match, as for Adzuna. `region` is a key of
`PLATSBANKEN_REGIONS`, or empty for all of Sweden."""
region = str(params.get("region") or "")
if region and region not in PLATSBANKEN_REGIONS:
raise ValueError(f"platsbanken region {region!r} is not one of the ids in "
"PLATSBANKEN_REGIONS")
query: dict[str, Any] = {"q": str(params.get("q") or ""), "sort": "pubdate-desc",
"limit": PLATSBANKEN_PAGE}
if region:
query["region"] = region
rows: list[Row] = []
seen: set[str] = set()
total: Any = None
ran_out = False
for page in range(_pages(params)):
offset = page * PLATSBANKEN_PAGE
if offset > PLATSBANKEN_MAX_OFFSET:
break
data = json.loads(_get(client, PLATSBANKEN_SEARCH, query | {"offset": offset}))
hits = data.get("hits") or []
total = (data.get("total") or {}).get("value", total)
seen |= {str(hit.get("id") or "") for hit in hits}
rows += _platsbanken_rows(data)
if len(hits) < PLATSBANKEN_PAGE:
ran_out = True
break
if isinstance(total, int) and len(seen) >= total:
break
return rows, (len(seen) == total) if isinstance(total, int) else ran_out
_SEARCHES: dict[str, Callable[[httpx.Client, dict[str, Any], dict[str, Any]],
tuple[list[Row], bool]]] = {
"jobindex": search_jobindex,
"adzuna": search_adzuna,
"platsbanken": search_platsbanken,
}
def _limits(params: Mapping[str, Any]) -> _runner.Limits:
return _runner.Limits(max_pages=_pages(params), page_delay=PAGE_DELAY)
def provider_of(target: str) -> str:
"""`jobindex:copenhagen-backend` -> `jobindex`, or '' for an unknown one."""
provider, sep, name = target.partition(":")
return provider if sep and name.strip() and provider in _SEARCHES else ""
return provider if sep and name.strip() and provider in readers.PROVIDERS else ""
def _read(client: httpx.Client, target: str, params: Mapping[str, Any],
keys: Mapping[str, Any] | None) -> tuple[ModuleType, _runner.Listing]:
"""One search read through the runner: its provider and its listing.
Raises ValueError for an unknown provider or a missing key, before any
request."""
provider = provider_of(target)
if not provider:
raise ValueError(f"{target!r} is not provider:name with provider one of "
f"{', '.join(readers.PROVIDERS)}")
module = readers.PROVIDERS[provider]
slug = target.partition(":")[2]
ask = _runner.ask_for(module, params, keys=keys)
get = _runner.getter(client, module.allowed(slug), provider)
return module, _runner.read_listing(module, slug, get, ask, limits=_limits(params))
def run_search(target: str, params: dict[str, Any],
keys: dict[str, Any] | None = None) -> tuple[list[Row], bool]:
"""One search, outside any pull: for `app searches add`, which runs a
search once before saving it."""
provider = provider_of(target)
if not provider:
raise ValueError(f"{target!r} is not provider:name with provider one of "
f"{', '.join(PROVIDERS)}")
search once before saving it. `keys` is the config's `sources.search`.
Each row also carries `place`, the display line of its places."""
with httpx.Client() as client:
return _SEARCHES[provider](client, params, (keys or {}).get(provider) or {})
_module, listing = _read(client, target, params, keys)
rows = [row | {"place": _runner.raw_of(row)["place"]} for row in listing.rows]
return rows, listing.complete
# -------------------------------------------------------------- the plugin
class SearchSource(Source):
name = "search"
description = "Saved searches on cross-company job boards: Jobindex, Adzuna, Platsbanken."
description = "Saved searches on cross-company job boards: " + ", ".join(
module.LABEL for module in readers.PROVIDERS.values()) + "."
#: Entries are `{"url": target, "label": name, "params": {...}}`.
subscription_field = "searches"
@@ -447,37 +135,32 @@ class SearchSource(Source):
log.warning("search: %r is not provider:name; skipped", target)
continue
try:
rows, complete = _SEARCHES[provider](
client, self.params_for(target), self.options.get(provider) or {})
_module, listing = _read(client, target, self.params_for(target),
self.options)
except Exception as exc:
log.warning("search: %s failed (%s)", target, exc)
continue
seen_ids = []
for row in rows:
source_id = f"{provider}:{row['id']}"
for row in listing.rows:
source_id = f"{provider}:{row['source_id']}"
seen_ids.append(source_id)
description = row.get("description") or ""
yield Item(
source_id=source_id,
location=row["url"],
title=row["title"],
excerpt=(row["description"] or "")[:300],
body=row["description"],
published_at=row["posted"],
excerpt=description[:300],
body=description,
published_at=row.get("posted"),
subscription=label,
kind="role",
raw={
"company": row["company"],
"place": row["place"],
"country": row["country"],
"remote": row["remote"],
"salary_min": row["salary_min"],
"salary_max": row["salary_max"],
"currency": row["currency"],
raw=_runner.raw_of(row) | {
"company": row.get("company") or "",
"provider": provider,
"search": target,
},
)
self.state[f"search:ok:{target}"] = run_marker
self.state[f"search:complete:{target}"] = "1" if complete else ""
self.state[f"search:complete:{target}"] = "1" if listing.complete else ""
self.state[f"search:ids:{target}"] = json.dumps(seen_ids)