Let the runner drop and count a row without an id, a title or a link, as ADR Plugins 3.8 has it, in one place for whole, paged and search reads: the count is stamped per target, logged, named on the board's row, and over 10% of the rows fails the read; the 28 readers hand every listing over instead of skipping it

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-10-06 14:04:33 +02:00
1 parent 38c31c830b
commit 911dfe8584
58 files changed
+479 -203

No files matched your search

+14 -6
View File
@@ -17,8 +17,9 @@ A target is `host:slug`, e.g. `greenhouse:databricks`, `ashby:deepl`,
`lever:eu/tomtom`, `teamtailor:careers.acme.com`; each host module says what
its slug is. Every board is read independently; one that fails is logged,
recorded as failing under `ats:failing:<target>`, and skipped, and so is one
whose read lists under a tenth of what it listed last time (`_runner.read`'s
guard), until it has failed that way for the pull's `stale_days`. A board's
whose read lists under a tenth of what it listed last time, or drops more
than a tenth of the rows its reader handed over (`_runner.read`'s guards),
until it has failed that way for the pull's `stale_days`. A board's
filter is the `reach` option the pull hands it (the agents', see
`app/jobs/reach.py`), over which its subscription `params` (`{"countries":
["DK"], "remote": true}`) win, narrowed to what its reader declares
@@ -343,8 +344,10 @@ SIZE_PAGES = 100
def board_size(target: str, *, client: httpx.Client | None = None) -> int:
"""How many roles board `target` lists, unfiltered and without terms: a
whole host's listing, the total a `complete="stated"` host gives on page
1, else the listing read to the end, at most `SIZE_PAGES` pages. Raises
as `_fetch_rows` does, so a board that does not answer is refused."""
1, else the listing read to the end, at most `SIZE_PAGES` pages. A read
counts the rows the runner dropped as well (`_listed`), as a stated total
does. Raises as `_fetch_rows` does, so a board that does not answer is
refused."""
if not is_known_target(target):
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
if client is None:
@@ -358,7 +361,12 @@ def board_size(target: str, *, client: httpx.Client | None = None) -> int:
if isinstance(total, int):
return total
limits = _runner.Limits(max_pages=SIZE_PAGES, page_delay=PAGE_DELAY)
return len(_runner.read_listing(module, slug, get, limits=limits).ids)
return _listed(_runner.read_listing(module, slug, get, limits=limits))
def _listed(listing: _runner.Listing) -> int:
"""The listings a read found: the ids it kept and the rows it dropped."""
return len(listing.ids) + listing.dropped
def keep_size(state: Any, target: str, size: int, *, today: str = "",
@@ -462,7 +470,7 @@ class ATSSource(Source):
[compound(i) for i in listing.ids])
self.state.pop(failing, None)
if unsized and listing.complete and not (listing.filtered or listing.narrowed):
keep_size(self.state, target, len(listing.ids), reset=False)
keep_size(self.state, target, _listed(listing), reset=False)
if detail_sets is not None:
# A complete listing drops the ids it no longer lists.
for key, ids in zip(detail_keys(target), detail_sets, strict=False):
+7 -2
View File
@@ -179,7 +179,7 @@ ends the read, incomplete; `_runner.MAX_PAGES` pages a pull at most,
**Rows.** A `Row` (`_contract.Row`) is a dict with `source_id` (the host's
own id for the role, unique on the board), `title`, `url` (http or https,
checked with `_shared.is_web_url`) and `places`; `location` (the host's
which the runner checks with `_shared.is_web_url`) and `places`; `location` (the host's
one-line place, kept while storage reads it), `posted` (an aware datetime
or None, see `_shared.parse_dt`), `department`, `description` (plain text,
see `_shared.html_text`), `remote` (True, False, a word or None) and
@@ -187,7 +187,12 @@ see `_shared.html_text`), `remote` (True, False, a word or None) and
(yearly integers) and `currency` (ISO 4217), see `_shared.yearly_amount`;
where it states them, `country` (ISO 3166 alpha-2), `seniority` and
`employment`; `teaser`; and, on a search row, `company`. The runner keeps
every stated key in `raw`, so a new field needs no change there.
every stated key in `raw`, so a new field needs no change there. A reader
never drops a listing: one it cannot make a whole row of (no id, no title or
no link) it hands over with what it has, an empty string for what is
missing, and the runner drops it and counts it (`_runner.valid`). A reader
leaves out only what is not a listing of the board: an entry that is not an
object, a second copy of an id, a role of another board.
**Places.** A role has a list of places: `places` is `[Place(text=...,
remote=...)]`, one entry per place the host states, in the host's order, in
+2
View File
@@ -205,6 +205,8 @@ class Row(TypedDict, total=False):
`company` is the employer's name on a search row, which lists many
(`plugins/search.py` stores it); a board row carries none, the board is
the company. Any other stated key is kept in `raw` by the runner.
`source_id`, `title` and `url` come empty from a listing that lacks
them: the runner drops such a row and counts it (`_runner.valid`).
"""
source_id: Required[str]
+98 -20
View File
@@ -4,10 +4,12 @@ the reader declares in `READS`.
A reader says how its host answers; this module reads it: the guarded fetch
(`fetch`, bound to one target by `getter`), the `Ask` a subscription resolves
to (`ask_for`) and its series (`series`), the listing whole or paged, one
series per `one`-filter value unioned by id (`read_listing`), completeness,
the sweep of an oversized paged board across pulls, the 10% guard on every
read that is not a sweep (`read`), the per-role detail reads
(`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
series per `one`-filter value unioned by id (`read_listing`), the rows it
keeps (`valid`: a row without an id, a title or a link is dropped and
counted here, never by the reader), completeness, the sweep of an oversized
paged board across pulls, the two 10% guards (`read`: on what a read that
is not a sweep listed, and on what any read dropped), the per-role detail
reads (`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
bookkeeping (`record_failure`, `failure_reason`). `plugins/ats.py` and
`plugins/search.py` are thin `Source` classes over it: they pass their own
limits (`Limits`) and state prefix (`ats`, `search`), so their state keys stay
@@ -30,6 +32,8 @@ State keys, under the Source's prefix and the target (`ats:sweep:<target>`):
drifts when roles close between pulls, so `app.jobs.pull` closes
a role of such a board only when two finished sweeps in a row
missed it (it keeps the ids the last one missed in `missing`)
dropped the rows the last stamped read dropped (`Listing.dropped`), as a
number, "" for none; `app.jobs.limits` names it on the board's row
"""
from __future__ import annotations
@@ -52,7 +56,7 @@ import httpx
from desk.sources import USER_AGENT
from ._contract import Ask, Key, Place, Reads
from ._shared import Get, Row
from ._shared import Get, Row, is_web_url
log = logging.getLogger(__name__)
@@ -96,7 +100,8 @@ SUSPECT_SHARE = 0.1
#: The last count from which the guard holds every read. A read that is
#: filtered, narrowed by terms or of an `empty="suspect"` reader is held from
#: a count of 1, as it was before every read was; a whole read of a board
#: that listed a handful of roles can drop to none by itself.
#: that listed a handful of roles can drop to none by itself. It is also the
#: rows a read must hand over before the share it dropped can fail it.
SUSPECT_MIN = 10
#: The keys of a row that an `Item` carries in its own fields, not in `raw`.
@@ -113,8 +118,8 @@ class TooLarge(ValueError):
class Suspect(Exception):
"""A read that listed under `SUSPECT_SHARE` of what it listed last time
(see `read`)."""
"""A read that listed under `SUSPECT_SHARE` of what it listed last time,
or dropped more than that share of its rows (see `read`)."""
class MissingKey(ValueError):
@@ -147,7 +152,9 @@ class Listing:
and roles closing meanwhile may have shifted an offset-paged host's later
roles back past the cursor, so `ids` can miss a role that is still open.
`remote_unasked` says a filtered read was meant to ask for remote roles
as well and could not, so its `ids` say nothing of the remote roles."""
as well and could not, so its `ids` say nothing of the remote roles.
`handed` counts the rows the reader handed over this pull, `dropped`
those of them `valid` refused, which are in neither `rows` nor `ids`."""
rows: list[Row]
complete: bool
@@ -158,6 +165,8 @@ class Listing:
narrowed: bool = False
swept: bool = False
remote_unasked: bool = False
handed: int = 0
dropped: int = 0
# --------------------------------------------------------------- modules
@@ -524,8 +533,42 @@ def _result_ids(results: Iterable[dict[str, Any]]) -> set[str]:
return {str(r.get("id") or r.get("shortcode") or "") for r in results}
def valid(row: Mapping[str, Any]) -> bool:
"""Whether the runner keeps a row: a non-empty `source_id`, a non-empty
`title` and a web URL (`is_web_url`). A reader hands over every listing
as far as it can make it out; a row without one of the three cannot be
stored or opened, and the runner drops it and counts it."""
try:
link = is_web_url(str(row.get("url") or ""))
except (httpx.InvalidURL, ValueError): # a string httpx cannot read as a URL
link = False
return (link and bool(str(row.get("source_id") or "").strip())
and bool(str(row.get("title") or "").strip()))
@dataclass(slots=True)
class _Tally:
"""The rows a read was handed and dropped, across its series and pages."""
handed: int = 0
dropped: int = 0
def _keep(found: Iterable[Row], rows: dict[str, Row], tally: _Tally) -> None:
"""The rows a reader's `parse` or `rows` returned, into `rows` by
`source_id`, the first of an id kept; those `valid` refuses are counted
in `tally` and left out. The one place a row is checked, for the whole
and the paged listing alike."""
for row in found:
tally.handed += 1
if not valid(row):
tally.dropped += 1
continue
rows.setdefault(row["source_id"], row)
def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
limits: Limits) -> dict[str, Row]:
limits: Limits, tally: _Tally) -> dict[str, Row]:
"""A whole listing, one request a series, unioned by `source_id`. A body
over the cap gets one retry at the reader's `fallback_url`."""
rows: dict[str, Row] = {}
@@ -541,8 +584,7 @@ def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
log.warning("readers: %s:%s: body over the cap; reading the smaller feed",
module.NAME, slug)
body = get(module.fallback_url(slug))
for row in module.parse(body, slug):
rows.setdefault(row["source_id"], row)
_keep(module.parse(body, slug), rows, tally)
return rows
@@ -580,7 +622,29 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
only: a search's places are what the search is, as its query is.
`remote` says the reach or the params wanted remote roles; when no series
asked for them (a reader without a remote filter, or one whose facets
state none beside places) the listing is `remote_unasked`."""
state none beside places) the listing is `remote_unasked`.
Every row `rows` or `parse` returns goes through `valid`: one without a
`source_id`, a title or a web URL is dropped here, counted in
`Listing.dropped` and logged once for the read. Completeness counts the
distinct result ids `page` returns beside its rows, not the rows kept, so
a dropped row still counts as read; a whole listing is complete whenever
it parses, whatever it dropped."""
tally = _Tally()
listing = _listing(module, slug, get, ask, limits=limits, sweep=sweep,
can_sweep=can_sweep, facets=facets, remote=remote, tally=tally)
listing.handed, listing.dropped = tally.handed, tally.dropped
if tally.dropped:
log.warning("readers: %s:%s: dropped %d of %d listings: no id, title or link",
module.NAME, slug, tally.dropped, tally.handed)
return listing
def _listing(module: ModuleType, slug: str, get: Get, ask: Ask | None, *,
limits: Limits | None, sweep: Mapping[str, Any] | None, can_sweep: bool,
facets: Mapping[str, Mapping[str, str]] | None, remote: bool,
tally: _Tally) -> Listing:
"""`read_listing` before its count of dropped rows, which `tally` keeps."""
limits = limits or Limits()
ask = ask or Ask()
reads: Reads = module.READS
@@ -595,7 +659,7 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
"nothing asked, not closing", module.NAME, slug)
return Listing([], False, [], True, None, key, by_words, remote_unasked=unasked)
if not hasattr(module, "page"):
whole_rows = _whole(module, slug, get, asks, limits)
whole_rows = _whole(module, slug, get, asks, limits, tally)
return Listing(list(whole_rows.values()), True, list(whole_rows), is_filtered, None,
key, by_words, remote_unasked=unasked)
@@ -632,8 +696,7 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
pages += 1
before = len(seen)
seen |= _result_ids(results)
for row in module.rows(data, slug):
rows.setdefault(row["source_id"], row)
_keep(module.rows(data, slug), rows, tally)
if cursor is None or (isinstance(total, int) and len(seen) >= total):
ended = True
break
@@ -747,13 +810,26 @@ def read(state: MutableMapping[str, str], prefix: str, target: str, module: Modu
that really shrank is accepted once the target has been failing for
`limits.stale_days` (its `<prefix>:failing:<target>` record), and its
stamp then resets the count; a Recheck resets it at once
(`plugins.ats.keep_size`)."""
(`plugins.ats.keep_size`).
A read, a sweep's included, that dropped more than `SUSPECT_SHARE` of
the rows its reader handed over (`valid`), once it was handed at least
`SUSPECT_MIN`, raises `Suspect` the same way: a page whose shape changed
parses to rows without a title or a link, and the roles it hid would
close. It gives way after `limits.stale_days` of failing as well."""
limits = limits or Limits()
ask = ask or Ask()
facets = facets_for(module, slug, get, ask, state, f"{prefix}:facets:{target}")
listing = read_listing(module, slug, get, ask, limits=limits, can_sweep=can_sweep,
sweep=_load(state, f"{prefix}:sweep:{target}"), facets=facets,
remote=remote)
if listing.handed >= SUSPECT_MIN and listing.dropped > SUSPECT_SHARE * listing.handed:
why = (f"dropped {listing.dropped} of {listing.handed} listings, over "
f"{SUSPECT_SHARE:.0%}: no id, title or link")
if not _failing_for(state, f"{prefix}:failing:{target}", limits.stale_days):
raise Suspect(why)
log.warning("readers: %s:%s: %s, accepted after %d days of failing",
module.NAME, slug, why, limits.stale_days)
held = listing.filtered or listing.narrowed or module.READS.empty == "suspect"
last = _load(state, f"{prefix}:count:{target}")
count = int(last.get("count") or 0) if last.get("asks") == listing.asks else 0
@@ -788,9 +864,10 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
`complete`, `ids` (the compound ids, what the pull closes against),
`filtered`, `narrowed` (a read by terms, which the pull never closes
against), whether it finished a sweep from earlier pulls (`swept`),
whether a filtered read left remote unasked (`remote_unasked`), the
sweep to continue or none, and the count for the 10% guard once a read
is not a sweep. A read that asked nothing keeps the count it had."""
whether a filtered read left remote unasked (`remote_unasked`), the rows
it dropped (`dropped`, "" for none), the sweep to continue or none, and
the count for the 10% guard once a read is not a sweep. A read that
asked nothing keeps the count it had."""
state[f"{prefix}:ok:{target}"] = run
state[f"{prefix}:complete:{target}"] = "1" if listing.complete else ""
state[f"{prefix}:swept:{target}"] = "1" if listing.swept else ""
@@ -798,6 +875,7 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
state[f"{prefix}:filtered:{target}"] = "1" if listing.filtered else ""
state[f"{prefix}:narrowed:{target}"] = "1" if listing.narrowed else ""
state[f"{prefix}:remote_unasked:{target}"] = "1" if listing.remote_unasked else ""
state[f"{prefix}:dropped:{target}"] = str(listing.dropped) if listing.dropped else ""
if listing.sweep is not None:
state[f"{prefix}:sweep:{target}"] = json.dumps(listing.sweep)
return
+1 -3
View File
@@ -24,7 +24,7 @@ from urllib.parse import urlencode
from desk.sources import strip_html
from ._contract import Ask, Fields, Filters, Key, Place, Reads
from ._shared import Get, Page, Row, is_web_url, only, parse_dt
from ._shared import Get, Page, Row, only, parse_dt
NAME = "adzuna"
LABEL = "Adzuna"
@@ -111,8 +111,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
for job in decoded["results"]:
title = _text(job.get("title") or "")
url = (job.get("redirect_url") or "").strip()
if not title or not is_web_url(url):
continue
predicted = str(job.get("salary_is_predicted") or "0") == "1"
town = ((job.get("location") or {}).get("display_name") or "").strip()
places: list[Place] = [{"text": town, "remote": None}] if town else []
+1 -3
View File
@@ -14,7 +14,7 @@ from typing import Any
from ._contract import Ask, Fields, Place, Reads, Row
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
log = logging.getLogger(__name__)
@@ -126,8 +126,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for j in data.get("jobs") or []:
title = (j.get("title") or "").strip()
url = (j.get("jobUrl") or j.get("applyUrl") or "").strip()
if not title or not is_web_url(url):
continue
description = j.get("descriptionHtml") or j.get("descriptionPlain") or ""
compensation = j.get("compensation") or {}
salary_min, salary_max, currency = _salary(compensation.get("summaryComponents") or [])
-2
View File
@@ -106,8 +106,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for j in listed:
title = (j.get("jobOpeningName") or "").strip()
source_id = str(j.get("id") or "")
if not title or not source_id:
continue
places = _places(j)
row: Row = {
"source_id": source_id,
-2
View File
@@ -132,8 +132,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for p in postings:
title = (p.get("title") or "").strip()
post_id = str(p.get("id") or "")
if not title or not post_id:
continue
job = p.get("job") or {}
places = [name for loc in job.get("jobLocations") or []
if (name := ((loc.get("location") or {}).get("name") or "").strip())]
+1 -3
View File
@@ -47,7 +47,7 @@ from urllib.parse import quote
from ._contract import Ask, Fields, Filters, Place, Reads, Row
from ._countries import name_of
from ._shared import Get, Page, add_place, html_text, is_web_url, only, parse_dt, place
from ._shared import Get, Page, add_place, html_text, only, parse_dt, place
READS = Reads(
scope="board", listing="paged", page_size=10, text="detail", complete="stated",
@@ -151,8 +151,6 @@ def rows(positions: list[dict[str, Any]], slug: str) -> list[Row]:
source_id = str(p.get("id") or "")
path = str(p.get("positionUrl") or "").strip()
url = f"https://{host}{path}" if path.startswith("/") and path[1:2] != "/" else ""
if not title or not source_id or not is_web_url(url):
continue
places = _places(p.get("locations") or p.get("standardizedLocations"))
out.append({
"source_id": source_id,
+1 -3
View File
@@ -37,7 +37,7 @@ from urllib.parse import urlencode, urlsplit, urlunsplit
from desk.sources import USER_AGENT
from ._contract import Ask, Fields, Filters, Place, Reads
from ._shared import (Get, Page, Row, html_text, is_web_url, num, only, parse_dt, pay_line,
from ._shared import (Get, Page, Row, html_text, num, only, parse_dt, pay_line,
yearly_amount)
NAME = "freehire"
@@ -170,8 +170,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
title = " ".join(html.unescape(str(hit.get("title") or "")).split())
source_id = str(hit.get("public_slug") or "").strip()
url = _url(str(hit.get("url") or ""))
if not title or not source_id or not is_web_url(url):
continue
mode = str(hit.get("work_mode") or "").strip()
places = _places(hit, mode)
countries = [str(c) for c in hit.get("countries") or [] if c]
-2
View File
@@ -116,8 +116,6 @@ def rows(postings: list[dict[str, Any]], slug: str) -> list[Row]:
for p in postings:
title = (p.get("title") or "").strip()
ext_id = str(p.get("extId") or "")
if not title or not ext_id:
continue
locations = p.get("locations") or []
places: list[Place] = []
for loc in locations:
+1 -3
View File
@@ -13,7 +13,7 @@ from typing import Any
from ._contract import Ask, Fields, Place, Reads, Row
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place
NAME = "greenhouse"
LABEL = "Greenhouse"
@@ -149,8 +149,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for j in data.get("jobs") or []:
title = (j.get("title") or "").strip()
url = (j.get("absolute_url") or "").strip()
if not title or not is_web_url(url):
continue
departments = j.get("departments") or []
ranges = j.get("pay_input_ranges") or []
salary_min, salary_max, currency = _salary(ranges)
-2
View File
@@ -90,8 +90,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
for ad in data.get("jobAdDetails") or []:
title = (ad.get("title") or "").strip()
ad_id = str(ad.get("id") or "").strip()
if not title or not ad_id:
continue
site = str(ad.get("site") or "").strip()
country = str(ad.get("country") or "").strip()
location = f"{site}, {country}" if site and country and country.lower() not in site.lower() \
+1 -3
View File
@@ -30,7 +30,7 @@ from typing import Any
from ._contract import Ask, Fields, Reads, Row
from ._shared import Get, Page, html_text, is_web_url, only, parse_dt
from ._shared import Get, Page, html_text, only, parse_dt
READS = Reads(
scope="board", listing="paged", page_size=100, text="listing", complete="end",
@@ -106,8 +106,6 @@ def rows(items: list[ET.Element], slug: str) -> list[Row]:
for item in items:
title = (item.findtext("title") or "").strip()
url = (item.findtext("link") or "").strip()
if not title or not is_web_url(url):
continue
out.append({
"source_id": (item.findtext("guid") or url).strip(),
"title": title,
-2
View File
@@ -127,8 +127,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
j = job.get("data") or {}
title = (j.get("title") or "").strip()
role = str(j.get("slug") or j.get("req_id") or "")
if not title or not role:
continue
categories = j.get("categories") or []
text = "\n".join(p for p in (html_text(j.get("description")),
html_text(j.get("qualifications"))) if p)
+3 -5
View File
@@ -36,7 +36,7 @@ from typing import Any
from urllib.parse import urlencode, urlsplit
from ._contract import Ask, Fields, Filters, Place, Reads
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt, place
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt, place
NAME = "jobbank"
LABEL = "Akademikernes Jobbank (Denmark)"
@@ -132,8 +132,8 @@ def _query(ask: Ask, number: int) -> tuple[str, str]:
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
"""One page of a search; the cursor is the page number. The next page
follows a full page (counted before any item is dropped, so a dropped item
does not pass for the end) while the ceiling is not reached."""
follows a full page (counted before the runner drops an item, so a
dropped item does not pass for the end) while the ceiling is not reached."""
number = int(cursor or 1)
url, remote = _query(ask, number)
try:
@@ -184,8 +184,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
out: list[Row] = []
for item in decoded["items"]:
link, title = item["link"], item["title"]
if not title or not item["id"] or not is_web_url(link):
continue
line = _LINE.fullmatch(item["description"])
kind = line.group("kind").strip() if line else ""
company, regions, cut = _regions(line.group("rest")) if line else ("", [], False)
+3 -5
View File
@@ -117,9 +117,9 @@ def _query(ask: Ask, number: int) -> str:
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
"""One page of a search, newest first; the cursor is the page number.
The next page follows a full page (counted before any item is dropped, so
a dropped item does not pass for the end) while the 200-result ceiling is
not reached."""
The next page follows a full page (counted before the runner drops an
item, so a dropped item does not pass for the end) while the 200-result
ceiling is not reached."""
number = int(cursor or 1)
root = ET.fromstring(get(_query(ask, number)))
channel = root.find("channel")
@@ -143,8 +143,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
out: list[Row] = []
for item in decoded["items"]:
link, full_title = item["link"], item["title"]
if not full_title or not is_web_url(link):
continue
description = item["description"]
h4 = _H4.search(description)
title = _text(h4.group(1)) if h4 else full_title
-2
View File
@@ -138,8 +138,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
for ad in decoded.get("jobAds") or []:
ident = str(ad.get("jobAdId") or "").strip()
title = _text(ad.get("title"))
if not ident or not title:
continue
link = str(ad.get("jobAdUrl") or "").strip()
town = _text(ad.get("municipality") or ad.get("postalDistrictName"))
places: list[Place] = [place(town)] if town else []
+1 -3
View File
@@ -176,8 +176,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
title = (j.get("title") or "").strip()
job_id = str(j.get("id") or "").strip()
path = str(j.get("url") or "").strip()
if not title or not job_id or not _ROLE_PATH.fullmatch(path):
continue
function = str(j.get("function") or "").strip()
places = [place(name) for name in dict.fromkeys(
str(name).strip() for name in j.get("locations") or [] if str(name).strip())]
@@ -186,7 +184,7 @@ def parse(body: bytes, slug: str) -> list[Row]:
"title": title,
"places": places,
"location": "; ".join(pl["text"] for pl in places),
"url": f"https://{_EMP}{path}",
"url": f"https://{_EMP}{path}" if _ROLE_PATH.fullmatch(path) else "",
"posted": _published(j.get("published_date")),
"department": "" if function == "All functions" else function,
"description": "",
+1 -3
View File
@@ -22,7 +22,7 @@ from typing import Any
from ._contract import Ask, Fields, Place, Reads, Row
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
log = logging.getLogger(__name__)
@@ -123,8 +123,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for j in data:
title = (j.get("text") or "").strip()
url = (j.get("hostedUrl") or "").strip()
if not title or not is_web_url(url):
continue
categories = j.get("categories") or {}
description = j.get("description") or j.get("descriptionPlain") or ""
salary_range = j.get("salaryRange") or {}
+4 -4
View File
@@ -198,9 +198,9 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
for role in search.get("requisitionList") or []:
role_id = str(role.get("Id") or "").strip()
title = _text(role.get("Title"))
# The id goes into a URL, and a listing is external content.
if not title or not re.fullmatch(r"[A-Za-z0-9_-]+", role_id):
continue
# The id goes into a URL, and a listing is external content: an id
# of other characters gets no link, and the runner drops the row.
safe = bool(re.fullmatch(r"[A-Za-z0-9_-]+", role_id))
remote = _text(role.get("WorkplaceTypeCode")) or None
at_office = (remote or "").lower() in ("ora_hybrid", "ora_on_site")
places: list[Place] = []
@@ -214,7 +214,7 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
"title": title,
"places": places,
"location": "; ".join(p["text"] for p in places),
"url": f"{_base(slug)}/job/{role_id}",
"url": f"{_base(slug)}/job/{role_id}" if safe else "",
"posted": parse_dt(role.get("PostedDate")),
"department": (_text(role.get("JobFamily")) or _text(role.get("JobFunction"))
or _text(role.get("Department")) or _text(role.get("Organization"))),
-2
View File
@@ -104,8 +104,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for position in root.findall("position"):
job_id = (position.findtext("id") or "").strip()
title = (position.findtext("name") or "").strip()
if not job_id or not title:
continue
description = " ".join(
(value.text or "") for value in position.findall("jobDescriptions/jobDescription/value"))
salary_info = position.find("salaryInformation")
-2
View File
@@ -168,8 +168,6 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
for job in search.get("data", {}).get("jobs") or []:
key = _key(job)
title = _text(job.get("title"))
if not title or not key:
continue
places: list[Place] = []
for text in (job.get("location"), *(job.get("multi_location") or [])):
if isinstance(text, str):
+1 -3
View File
@@ -22,7 +22,7 @@ from typing import Any
from ._contract import Ask, Fields, Reads
from ._shared import HEADING, Row, html_text, is_web_url, label_slug_ok, num, only, pay_line, yearly_amount
from ._shared import HEADING, Row, html_text, label_slug_ok, num, only, pay_line, yearly_amount
NAME = "pinpoint"
LABEL = "Pinpoint"
@@ -121,8 +121,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
title = (j.get("title") or "").strip()
url = (j.get("url") or "").strip()
source_id = str(j.get("id") or "")
if not title or not source_id or not is_web_url(url):
continue
job = j.get("job") or {}
low, high, currency, salary_text = _pay(j)
workplace = str(j.get("workplace_type") or "").strip()
+1 -3
View File
@@ -22,7 +22,7 @@ from urllib.parse import urlencode
from desk.sources import strip_html
from ._contract import Ask, Fields, Filters, Place, Reads
from ._shared import Get, Page, Row, html_text, is_web_url, only, parse_dt
from ._shared import Get, Page, Row, html_text, only, parse_dt
NAME = "platsbanken"
LABEL = "Platsbanken"
@@ -108,8 +108,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
for hit in decoded.get("hits") or []:
title = _text(hit.get("headline") or "")
url = (hit.get("webpage_url") or "").strip()
if not title or not is_web_url(url):
continue
addresses = hit.get("workplace_addresses") or [hit.get("workplace_address") or {}]
places = [place for address in addresses if isinstance(address, dict)
if (place := _place(address))]
+1 -3
View File
@@ -13,7 +13,7 @@ from typing import Any
from ._contract import Ask, Fields, Place, Reads
from ._shared import Row, add_place, html_text, is_web_url, label_slug_ok, num, only, parse_dt, pay_line, yearly_amount
from ._shared import Row, add_place, html_text, label_slug_ok, num, only, parse_dt, pay_line, yearly_amount
NAME = "recruitee"
LABEL = "Recruitee"
@@ -105,8 +105,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
for j in data.get("offers") or []:
title = (j.get("title") or "").strip()
url = (j.get("careers_url") or "").strip()
if not title or not is_web_url(url):
continue
salary = j.get("salary") if isinstance(j.get("salary"), dict) else {}
salary_min, salary_max, currency = yearly_amount(
salary.get("min"), salary.get("max"), salary.get("currency"),
+14 -10
View File
@@ -82,27 +82,31 @@ def parse(body: bytes, slug: str) -> list[Row]:
raise ValueError("not a Rippling board: the answer is not a list")
roles: dict[str, Row] = {}
places: dict[str, list[Place]] = {}
for entry in data:
for index, entry in enumerate(data):
job_id = str(entry.get("uuid") or "").strip()
title = (entry.get("name") or "").strip()
if not title or not _UUID.fullmatch(job_id):
continue
# The id goes into a URL: one of other characters gets no link, and
# the runner drops the row. An entry without an id is a role of its
# own, not merged with the others that lack one.
key = job_id or f"#{index}"
place = str(((entry.get("workLocation") or {}).get("label")) or "").strip()
places.setdefault(job_id, [])
places.setdefault(key, [])
if place:
add_place(places[job_id], _place(place))
roles.setdefault(job_id, {
add_place(places[key], _place(place))
roles.setdefault(key, {
"source_id": job_id,
"title": title,
"url": f"https://{_SITE}/{slug}/jobs/{job_id}",
"url": f"https://{_SITE}/{slug}/jobs/{job_id}" if _UUID.fullmatch(job_id) else "",
"posted": None,
"department": str(((entry.get("department") or {}).get("label")) or "").strip(),
"description": "",
"salary_text": "", # the board's list states no pay
})
for job_id, row in roles.items():
row["places"] = places[job_id]
row["location"] = "; ".join(p["text"] for p in places[job_id])
if title and not roles[key]["title"]:
roles[key]["title"] = title # an entry of the role that has one
for key, row in roles.items():
row["places"] = places[key]
row["location"] = "; ".join(p["text"] for p in places[key])
return list(roles.values())
-2
View File
@@ -115,8 +115,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
for j in data.get("content") or []:
title = (j.get("name") or "").strip()
job_id = str(j.get("id") or "")
if not title or not job_id:
continue
location = j.get("location") or {}
loc_text = (location.get("fullLocation")
or ", ".join(p for p in (location.get("city"), location.get("country")) if p))
+25 -16
View File
@@ -63,7 +63,7 @@ from desk.sources import strip_html
from ._contract import Ask, Fields, Filters, Place, Reads
from ._countries import NAMES
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt
NAME = "successfactors"
LABEL = "SuccessFactors"
@@ -247,8 +247,9 @@ def _tile_fields(chunk: str) -> tuple[str, str, str]:
def _entries(text: str) -> list[dict[str, Any]]:
"""The roles of one search page, in the order shown, whichever layout it is
in and before the prefix filter: `id`, `href` (as written, relative),
`title`, `place`, `date` and `department`. A role is listed once, and a row
without a link to a role is not one."""
`title`, `place`, `date` and `department`. A role is listed once. The id
and the title are in the role's link, so a row without one is handed over
with an empty `id`, `href` and `title`, for the runner to drop and count."""
table = _ROW.split(text)[1:]
chunks = table or _TILE.split(text)[1:]
out: list[dict[str, Any]] = []
@@ -256,10 +257,12 @@ def _entries(text: str) -> list[dict[str, Any]]:
for chunk in chunks:
if table:
chunk = chunk.split("</tr>", 1)[0]
if not (link := _LINK.search(chunk)):
continue
href, job_id, title = html.unescape(link.group(1)), link.group(2), _text(link.group(3))
if job_id in seen:
if link := _LINK.search(chunk):
href, job_id, title = (html.unescape(link.group(1)), link.group(2),
_text(link.group(3)))
else:
href = job_id = title = ""
if job_id and job_id in seen:
continue
seen.add(job_id)
place, date, department = (_table_fields if table else _tile_fields)(chunk)
@@ -283,12 +286,14 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
if not _SEARCH_PAGE.search(text):
raise ValueError("not a SuccessFactors search page")
found = _entries(text)
ids = [entry["id"] for entry in found if entry["id"]]
total = _total(text)
if found and found[0]["id"] == previous:
first = ids[0] if ids else None
if first is not None and first == previous:
return text, [], total, (start, previous)
more = bool(found) and (total is None or start + len(found) < total)
return (text, [{"id": entry["id"]} for entry in found], total,
(start + len(found), found[0]["id"]) if more else None)
return (text, [{"id": job_id} for job_id in ids], total,
(start + len(found), first) if more else None)
def rows(decoded: str, slug: str) -> list[Row]:
@@ -296,17 +301,21 @@ def rows(decoded: str, slug: str) -> list[Row]:
slug's prefix when it has one; the id in it is the requisition's `source_id`.
The table and the tile layout give a title, a location, a date and, where
the site has the column, a department; nothing else about a role is on
the list, so its description and pay are empty."""
the list, so its description and pay are empty.
A role linked on another host, or outside the prefix, is another board's
(another site, or another brand of a shared site), and is left out: it is
no role of this company. A row with no link or no title is handed over,
and the runner drops it."""
host, prefix = _parts(slug)
base = f"https://{host}/"
out = []
for entry in _entries(decoded):
url = urljoin(base, entry["href"])
if urlsplit(url).hostname != host or not is_web_url(url):
url = urljoin(base, entry["href"]) if entry["href"] else ""
if url and urlsplit(url).hostname != host:
continue
if prefix and not urlsplit(url).path.lower().startswith(f"{prefix.lower()}/job/"):
continue
if not entry["title"]:
if url and prefix and not urlsplit(url).path.lower().startswith(
f"{prefix.lower()}/job/"):
continue
out.append({
"source_id": entry["id"],
+1 -3
View File
@@ -15,7 +15,7 @@ from typing import Any
from ._contract import Ask, Fields, Place, Reads
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt
NAME = "teamtailor"
LABEL = "Teamtailor"
@@ -115,8 +115,6 @@ def rows(items: list[ET.Element], slug: str) -> list[Row]:
for item in items:
title = (item.findtext("title") or "").strip()
url = (item.findtext("link") or "").strip()
if not title or not is_web_url(url):
continue
status = (item.findtext("remoteStatus") or "").strip()
remote = {"fully": True, "hybrid": False, "onsite": False}.get(status)
places: list[Place] = []
-2
View File
@@ -102,8 +102,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
for j in data.get("results") or []:
title = (j.get("title") or "").strip()
shortcode = (j.get("shortcode") or "").strip()
if not title or not shortcode:
continue
workplace = str(j.get("workplace") or "").strip()
remote = bool(j.get("remote")) or workplace == "remote"
places = _places(j, remote)
+6 -6
View File
@@ -124,8 +124,8 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
total = stated if isinstance(stated, int) and not isinstance(stated, bool) else None
postings = data.get("jobPostings") or []
# Every posting is counted, a path-less one (LEGO lists one that has only
# `bulletFields`) by its position, so the parser dropping it does not
# leave the board short of its total.
# `bulletFields`) by its position, so the runner dropping its row does
# not leave the board short of its total.
results = [{"id": _posting_id(p, offset + i)} for i, p in enumerate(postings)]
following = offset + len(postings)
more = bool(postings) and (total is None or following < total)
@@ -274,13 +274,13 @@ def rows(decoded: dict[str, Any], slug: str) -> list[Row]:
`bulletFields`, which lead with the requisition id, and on some tenants
`remoteType`. No description, no date and no country: `detail` has
those, and the places of a role that "2 Locations" hides. The id is the requisition id where the path carries it, and the
path where not; a posting with no title or no path is skipped."""
path where not. A posting with no path has neither an id nor a link, and
the runner drops it."""
out = []
for p in decoded.get("jobPostings") or []:
title = (p.get("title") or "").strip()
path = (p.get("externalPath") or "").strip()
if not title or not _ROLE_PATH.fullmatch(path):
continue
path = path if _ROLE_PATH.fullmatch(path) else ""
remote = _remote(p.get("remoteType"))
place = _place(p.get("locationsText"), remote)
out.append({
@@ -288,7 +288,7 @@ def rows(decoded: dict[str, Any], slug: str) -> list[Row]:
"title": title,
"places": [place] if place else [],
"location": place["text"] if place else "",
"url": f"https://{_host(slug)}/{_parts(slug)[2]}{path}", # type: ignore[index]
"url": f"https://{_host(slug)}/{_parts(slug)[2]}{path}" if path else "", # type: ignore[index]
"posted": _posted(p.get("postedOn") or ""),
"department": "",
"description": "",
+2 -1
View File
@@ -19,7 +19,8 @@ roles close once no search has seen them for `STALE_DAYS` (see
`app/jobs/pull.py`).
In a pull a search is read by `_runner.read` under the prefix `search`, as a
board is under `ats`: the 10% guard holds it, `_runner.stamp` records it,
board is under `ats`: the runner drops a row without an id, a title or a
link, the 10% guards hold it, `_runner.stamp` records it,
and a search that fails or is suspect is kept as failing under
`search:failing:<target>` (`_runner.record_failure`) until it answers, which
the settle reads as it reads a failing board's.