Let the runner drop and count a row without an id, a title or a link, as ADR Plugins 3.8 has it, in one place for whole, paged and search reads: the count is stamped per target, logged, named on the board's row, and over 10% of the rows fails the read; the 28 readers hand every listing over instead of skipping it
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
38c31c830b
commit
911dfe8584
58 files changed
+479
-203
No files matched your search
+14
-6
@@ -17,8 +17,9 @@ A target is `host:slug`, e.g. `greenhouse:databricks`, `ashby:deepl`,
|
||||
`lever:eu/tomtom`, `teamtailor:careers.acme.com`; each host module says what
|
||||
its slug is. Every board is read independently; one that fails is logged,
|
||||
recorded as failing under `ats:failing:<target>`, and skipped, and so is one
|
||||
whose read lists under a tenth of what it listed last time (`_runner.read`'s
|
||||
guard), until it has failed that way for the pull's `stale_days`. A board's
|
||||
whose read lists under a tenth of what it listed last time, or drops more
|
||||
than a tenth of the rows its reader handed over (`_runner.read`'s guards),
|
||||
until it has failed that way for the pull's `stale_days`. A board's
|
||||
filter is the `reach` option the pull hands it (the agents', see
|
||||
`app/jobs/reach.py`), over which its subscription `params` (`{"countries":
|
||||
["DK"], "remote": true}`) win, narrowed to what its reader declares
|
||||
@@ -343,8 +344,10 @@ SIZE_PAGES = 100
|
||||
def board_size(target: str, *, client: httpx.Client | None = None) -> int:
|
||||
"""How many roles board `target` lists, unfiltered and without terms: a
|
||||
whole host's listing, the total a `complete="stated"` host gives on page
|
||||
1, else the listing read to the end, at most `SIZE_PAGES` pages. Raises
|
||||
as `_fetch_rows` does, so a board that does not answer is refused."""
|
||||
1, else the listing read to the end, at most `SIZE_PAGES` pages. A read
|
||||
counts the rows the runner dropped as well (`_listed`), as a stated total
|
||||
does. Raises as `_fetch_rows` does, so a board that does not answer is
|
||||
refused."""
|
||||
if not is_known_target(target):
|
||||
raise ValueError(f"{target!r} is not host:slug for a known ATS host")
|
||||
if client is None:
|
||||
@@ -358,7 +361,12 @@ def board_size(target: str, *, client: httpx.Client | None = None) -> int:
|
||||
if isinstance(total, int):
|
||||
return total
|
||||
limits = _runner.Limits(max_pages=SIZE_PAGES, page_delay=PAGE_DELAY)
|
||||
return len(_runner.read_listing(module, slug, get, limits=limits).ids)
|
||||
return _listed(_runner.read_listing(module, slug, get, limits=limits))
|
||||
|
||||
|
||||
def _listed(listing: _runner.Listing) -> int:
|
||||
"""The listings a read found: the ids it kept and the rows it dropped."""
|
||||
return len(listing.ids) + listing.dropped
|
||||
|
||||
|
||||
def keep_size(state: Any, target: str, size: int, *, today: str = "",
|
||||
@@ -462,7 +470,7 @@ class ATSSource(Source):
|
||||
[compound(i) for i in listing.ids])
|
||||
self.state.pop(failing, None)
|
||||
if unsized and listing.complete and not (listing.filtered or listing.narrowed):
|
||||
keep_size(self.state, target, len(listing.ids), reset=False)
|
||||
keep_size(self.state, target, _listed(listing), reset=False)
|
||||
if detail_sets is not None:
|
||||
# A complete listing drops the ids it no longer lists.
|
||||
for key, ids in zip(detail_keys(target), detail_sets, strict=False):
|
||||
|
||||
@@ -179,7 +179,7 @@ ends the read, incomplete; `_runner.MAX_PAGES` pages a pull at most,
|
||||
|
||||
**Rows.** A `Row` (`_contract.Row`) is a dict with `source_id` (the host's
|
||||
own id for the role, unique on the board), `title`, `url` (http or https,
|
||||
checked with `_shared.is_web_url`) and `places`; `location` (the host's
|
||||
which the runner checks with `_shared.is_web_url`) and `places`; `location` (the host's
|
||||
one-line place, kept while storage reads it), `posted` (an aware datetime
|
||||
or None, see `_shared.parse_dt`), `department`, `description` (plain text,
|
||||
see `_shared.html_text`), `remote` (True, False, a word or None) and
|
||||
@@ -187,7 +187,12 @@ see `_shared.html_text`), `remote` (True, False, a word or None) and
|
||||
(yearly integers) and `currency` (ISO 4217), see `_shared.yearly_amount`;
|
||||
where it states them, `country` (ISO 3166 alpha-2), `seniority` and
|
||||
`employment`; `teaser`; and, on a search row, `company`. The runner keeps
|
||||
every stated key in `raw`, so a new field needs no change there.
|
||||
every stated key in `raw`, so a new field needs no change there. A reader
|
||||
never drops a listing: one it cannot make a whole row of (no id, no title or
|
||||
no link) it hands over with what it has, an empty string for what is
|
||||
missing, and the runner drops it and counts it (`_runner.valid`). A reader
|
||||
leaves out only what is not a listing of the board: an entry that is not an
|
||||
object, a second copy of an id, a role of another board.
|
||||
|
||||
**Places.** A role has a list of places: `places` is `[Place(text=...,
|
||||
remote=...)]`, one entry per place the host states, in the host's order, in
|
||||
|
||||
@@ -205,6 +205,8 @@ class Row(TypedDict, total=False):
|
||||
`company` is the employer's name on a search row, which lists many
|
||||
(`plugins/search.py` stores it); a board row carries none, the board is
|
||||
the company. Any other stated key is kept in `raw` by the runner.
|
||||
`source_id`, `title` and `url` come empty from a listing that lacks
|
||||
them: the runner drops such a row and counts it (`_runner.valid`).
|
||||
"""
|
||||
|
||||
source_id: Required[str]
|
||||
|
||||
+98
-20
@@ -4,10 +4,12 @@ the reader declares in `READS`.
|
||||
A reader says how its host answers; this module reads it: the guarded fetch
|
||||
(`fetch`, bound to one target by `getter`), the `Ask` a subscription resolves
|
||||
to (`ask_for`) and its series (`series`), the listing whole or paged, one
|
||||
series per `one`-filter value unioned by id (`read_listing`), completeness,
|
||||
the sweep of an oversized paged board across pulls, the 10% guard on every
|
||||
read that is not a sweep (`read`), the per-role detail reads
|
||||
(`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
|
||||
series per `one`-filter value unioned by id (`read_listing`), the rows it
|
||||
keeps (`valid`: a row without an id, a title or a link is dropped and
|
||||
counted here, never by the reader), completeness, the sweep of an oversized
|
||||
paged board across pulls, the two 10% guards (`read`: on what a read that
|
||||
is not a sweep listed, and on what any read dropped), the per-role detail
|
||||
reads (`read_details`), the `raw` an `Item` carries (`raw_of`), and the failure
|
||||
bookkeeping (`record_failure`, `failure_reason`). `plugins/ats.py` and
|
||||
`plugins/search.py` are thin `Source` classes over it: they pass their own
|
||||
limits (`Limits`) and state prefix (`ats`, `search`), so their state keys stay
|
||||
@@ -30,6 +32,8 @@ State keys, under the Source's prefix and the target (`ats:sweep:<target>`):
|
||||
drifts when roles close between pulls, so `app.jobs.pull` closes
|
||||
a role of such a board only when two finished sweeps in a row
|
||||
missed it (it keeps the ids the last one missed in `missing`)
|
||||
dropped the rows the last stamped read dropped (`Listing.dropped`), as a
|
||||
number, "" for none; `app.jobs.limits` names it on the board's row
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -52,7 +56,7 @@ import httpx
|
||||
from desk.sources import USER_AGENT
|
||||
|
||||
from ._contract import Ask, Key, Place, Reads
|
||||
from ._shared import Get, Row
|
||||
from ._shared import Get, Row, is_web_url
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -96,7 +100,8 @@ SUSPECT_SHARE = 0.1
|
||||
#: The last count from which the guard holds every read. A read that is
|
||||
#: filtered, narrowed by terms or of an `empty="suspect"` reader is held from
|
||||
#: a count of 1, as it was before every read was; a whole read of a board
|
||||
#: that listed a handful of roles can drop to none by itself.
|
||||
#: that listed a handful of roles can drop to none by itself. It is also the
|
||||
#: rows a read must hand over before the share it dropped can fail it.
|
||||
SUSPECT_MIN = 10
|
||||
|
||||
#: The keys of a row that an `Item` carries in its own fields, not in `raw`.
|
||||
@@ -113,8 +118,8 @@ class TooLarge(ValueError):
|
||||
|
||||
|
||||
class Suspect(Exception):
|
||||
"""A read that listed under `SUSPECT_SHARE` of what it listed last time
|
||||
(see `read`)."""
|
||||
"""A read that listed under `SUSPECT_SHARE` of what it listed last time,
|
||||
or dropped more than that share of its rows (see `read`)."""
|
||||
|
||||
|
||||
class MissingKey(ValueError):
|
||||
@@ -147,7 +152,9 @@ class Listing:
|
||||
and roles closing meanwhile may have shifted an offset-paged host's later
|
||||
roles back past the cursor, so `ids` can miss a role that is still open.
|
||||
`remote_unasked` says a filtered read was meant to ask for remote roles
|
||||
as well and could not, so its `ids` say nothing of the remote roles."""
|
||||
as well and could not, so its `ids` say nothing of the remote roles.
|
||||
`handed` counts the rows the reader handed over this pull, `dropped`
|
||||
those of them `valid` refused, which are in neither `rows` nor `ids`."""
|
||||
|
||||
rows: list[Row]
|
||||
complete: bool
|
||||
@@ -158,6 +165,8 @@ class Listing:
|
||||
narrowed: bool = False
|
||||
swept: bool = False
|
||||
remote_unasked: bool = False
|
||||
handed: int = 0
|
||||
dropped: int = 0
|
||||
|
||||
|
||||
# --------------------------------------------------------------- modules
|
||||
@@ -524,8 +533,42 @@ def _result_ids(results: Iterable[dict[str, Any]]) -> set[str]:
|
||||
return {str(r.get("id") or r.get("shortcode") or "") for r in results}
|
||||
|
||||
|
||||
def valid(row: Mapping[str, Any]) -> bool:
|
||||
"""Whether the runner keeps a row: a non-empty `source_id`, a non-empty
|
||||
`title` and a web URL (`is_web_url`). A reader hands over every listing
|
||||
as far as it can make it out; a row without one of the three cannot be
|
||||
stored or opened, and the runner drops it and counts it."""
|
||||
try:
|
||||
link = is_web_url(str(row.get("url") or ""))
|
||||
except (httpx.InvalidURL, ValueError): # a string httpx cannot read as a URL
|
||||
link = False
|
||||
return (link and bool(str(row.get("source_id") or "").strip())
|
||||
and bool(str(row.get("title") or "").strip()))
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class _Tally:
|
||||
"""The rows a read was handed and dropped, across its series and pages."""
|
||||
|
||||
handed: int = 0
|
||||
dropped: int = 0
|
||||
|
||||
|
||||
def _keep(found: Iterable[Row], rows: dict[str, Row], tally: _Tally) -> None:
|
||||
"""The rows a reader's `parse` or `rows` returned, into `rows` by
|
||||
`source_id`, the first of an id kept; those `valid` refuses are counted
|
||||
in `tally` and left out. The one place a row is checked, for the whole
|
||||
and the paged listing alike."""
|
||||
for row in found:
|
||||
tally.handed += 1
|
||||
if not valid(row):
|
||||
tally.dropped += 1
|
||||
continue
|
||||
rows.setdefault(row["source_id"], row)
|
||||
|
||||
|
||||
def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
|
||||
limits: Limits) -> dict[str, Row]:
|
||||
limits: Limits, tally: _Tally) -> dict[str, Row]:
|
||||
"""A whole listing, one request a series, unioned by `source_id`. A body
|
||||
over the cap gets one retry at the reader's `fallback_url`."""
|
||||
rows: dict[str, Row] = {}
|
||||
@@ -541,8 +584,7 @@ def _whole(module: ModuleType, slug: str, get: Get, asks: list[Ask],
|
||||
log.warning("readers: %s:%s: body over the cap; reading the smaller feed",
|
||||
module.NAME, slug)
|
||||
body = get(module.fallback_url(slug))
|
||||
for row in module.parse(body, slug):
|
||||
rows.setdefault(row["source_id"], row)
|
||||
_keep(module.parse(body, slug), rows, tally)
|
||||
return rows
|
||||
|
||||
|
||||
@@ -580,7 +622,29 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
only: a search's places are what the search is, as its query is.
|
||||
`remote` says the reach or the params wanted remote roles; when no series
|
||||
asked for them (a reader without a remote filter, or one whose facets
|
||||
state none beside places) the listing is `remote_unasked`."""
|
||||
state none beside places) the listing is `remote_unasked`.
|
||||
|
||||
Every row `rows` or `parse` returns goes through `valid`: one without a
|
||||
`source_id`, a title or a web URL is dropped here, counted in
|
||||
`Listing.dropped` and logged once for the read. Completeness counts the
|
||||
distinct result ids `page` returns beside its rows, not the rows kept, so
|
||||
a dropped row still counts as read; a whole listing is complete whenever
|
||||
it parses, whatever it dropped."""
|
||||
tally = _Tally()
|
||||
listing = _listing(module, slug, get, ask, limits=limits, sweep=sweep,
|
||||
can_sweep=can_sweep, facets=facets, remote=remote, tally=tally)
|
||||
listing.handed, listing.dropped = tally.handed, tally.dropped
|
||||
if tally.dropped:
|
||||
log.warning("readers: %s:%s: dropped %d of %d listings: no id, title or link",
|
||||
module.NAME, slug, tally.dropped, tally.handed)
|
||||
return listing
|
||||
|
||||
|
||||
def _listing(module: ModuleType, slug: str, get: Get, ask: Ask | None, *,
|
||||
limits: Limits | None, sweep: Mapping[str, Any] | None, can_sweep: bool,
|
||||
facets: Mapping[str, Mapping[str, str]] | None, remote: bool,
|
||||
tally: _Tally) -> Listing:
|
||||
"""`read_listing` before its count of dropped rows, which `tally` keeps."""
|
||||
limits = limits or Limits()
|
||||
ask = ask or Ask()
|
||||
reads: Reads = module.READS
|
||||
@@ -595,7 +659,7 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
"nothing asked, not closing", module.NAME, slug)
|
||||
return Listing([], False, [], True, None, key, by_words, remote_unasked=unasked)
|
||||
if not hasattr(module, "page"):
|
||||
whole_rows = _whole(module, slug, get, asks, limits)
|
||||
whole_rows = _whole(module, slug, get, asks, limits, tally)
|
||||
return Listing(list(whole_rows.values()), True, list(whole_rows), is_filtered, None,
|
||||
key, by_words, remote_unasked=unasked)
|
||||
|
||||
@@ -632,8 +696,7 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
pages += 1
|
||||
before = len(seen)
|
||||
seen |= _result_ids(results)
|
||||
for row in module.rows(data, slug):
|
||||
rows.setdefault(row["source_id"], row)
|
||||
_keep(module.rows(data, slug), rows, tally)
|
||||
if cursor is None or (isinstance(total, int) and len(seen) >= total):
|
||||
ended = True
|
||||
break
|
||||
@@ -747,13 +810,26 @@ def read(state: MutableMapping[str, str], prefix: str, target: str, module: Modu
|
||||
that really shrank is accepted once the target has been failing for
|
||||
`limits.stale_days` (its `<prefix>:failing:<target>` record), and its
|
||||
stamp then resets the count; a Recheck resets it at once
|
||||
(`plugins.ats.keep_size`)."""
|
||||
(`plugins.ats.keep_size`).
|
||||
|
||||
A read, a sweep's included, that dropped more than `SUSPECT_SHARE` of
|
||||
the rows its reader handed over (`valid`), once it was handed at least
|
||||
`SUSPECT_MIN`, raises `Suspect` the same way: a page whose shape changed
|
||||
parses to rows without a title or a link, and the roles it hid would
|
||||
close. It gives way after `limits.stale_days` of failing as well."""
|
||||
limits = limits or Limits()
|
||||
ask = ask or Ask()
|
||||
facets = facets_for(module, slug, get, ask, state, f"{prefix}:facets:{target}")
|
||||
listing = read_listing(module, slug, get, ask, limits=limits, can_sweep=can_sweep,
|
||||
sweep=_load(state, f"{prefix}:sweep:{target}"), facets=facets,
|
||||
remote=remote)
|
||||
if listing.handed >= SUSPECT_MIN and listing.dropped > SUSPECT_SHARE * listing.handed:
|
||||
why = (f"dropped {listing.dropped} of {listing.handed} listings, over "
|
||||
f"{SUSPECT_SHARE:.0%}: no id, title or link")
|
||||
if not _failing_for(state, f"{prefix}:failing:{target}", limits.stale_days):
|
||||
raise Suspect(why)
|
||||
log.warning("readers: %s:%s: %s, accepted after %d days of failing",
|
||||
module.NAME, slug, why, limits.stale_days)
|
||||
held = listing.filtered or listing.narrowed or module.READS.empty == "suspect"
|
||||
last = _load(state, f"{prefix}:count:{target}")
|
||||
count = int(last.get("count") or 0) if last.get("asks") == listing.asks else 0
|
||||
@@ -788,9 +864,10 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
|
||||
`complete`, `ids` (the compound ids, what the pull closes against),
|
||||
`filtered`, `narrowed` (a read by terms, which the pull never closes
|
||||
against), whether it finished a sweep from earlier pulls (`swept`),
|
||||
whether a filtered read left remote unasked (`remote_unasked`), the
|
||||
sweep to continue or none, and the count for the 10% guard once a read
|
||||
is not a sweep. A read that asked nothing keeps the count it had."""
|
||||
whether a filtered read left remote unasked (`remote_unasked`), the rows
|
||||
it dropped (`dropped`, "" for none), the sweep to continue or none, and
|
||||
the count for the 10% guard once a read is not a sweep. A read that
|
||||
asked nothing keeps the count it had."""
|
||||
state[f"{prefix}:ok:{target}"] = run
|
||||
state[f"{prefix}:complete:{target}"] = "1" if listing.complete else ""
|
||||
state[f"{prefix}:swept:{target}"] = "1" if listing.swept else ""
|
||||
@@ -798,6 +875,7 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
|
||||
state[f"{prefix}:filtered:{target}"] = "1" if listing.filtered else ""
|
||||
state[f"{prefix}:narrowed:{target}"] = "1" if listing.narrowed else ""
|
||||
state[f"{prefix}:remote_unasked:{target}"] = "1" if listing.remote_unasked else ""
|
||||
state[f"{prefix}:dropped:{target}"] = str(listing.dropped) if listing.dropped else ""
|
||||
if listing.sweep is not None:
|
||||
state[f"{prefix}:sweep:{target}"] = json.dumps(listing.sweep)
|
||||
return
|
||||
|
||||
@@ -24,7 +24,7 @@ from urllib.parse import urlencode
|
||||
from desk.sources import strip_html
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Key, Place, Reads
|
||||
from ._shared import Get, Page, Row, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, only, parse_dt
|
||||
|
||||
NAME = "adzuna"
|
||||
LABEL = "Adzuna"
|
||||
@@ -111,8 +111,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
for job in decoded["results"]:
|
||||
title = _text(job.get("title") or "")
|
||||
url = (job.get("redirect_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
predicted = str(job.get("salary_is_predicted") or "0") == "1"
|
||||
town = ((job.get("location") or {}).get("display_name") or "").strip()
|
||||
places: list[Place] = [{"text": town, "remote": None}] if town else []
|
||||
|
||||
@@ -14,7 +14,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads, Row
|
||||
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -126,8 +126,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data.get("jobs") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("jobUrl") or j.get("applyUrl") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
description = j.get("descriptionHtml") or j.get("descriptionPlain") or ""
|
||||
compensation = j.get("compensation") or {}
|
||||
salary_min, salary_max, currency = _salary(compensation.get("summaryComponents") or [])
|
||||
|
||||
@@ -106,8 +106,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in listed:
|
||||
title = (j.get("jobOpeningName") or "").strip()
|
||||
source_id = str(j.get("id") or "")
|
||||
if not title or not source_id:
|
||||
continue
|
||||
places = _places(j)
|
||||
row: Row = {
|
||||
"source_id": source_id,
|
||||
|
||||
@@ -132,8 +132,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for p in postings:
|
||||
title = (p.get("title") or "").strip()
|
||||
post_id = str(p.get("id") or "")
|
||||
if not title or not post_id:
|
||||
continue
|
||||
job = p.get("job") or {}
|
||||
places = [name for loc in job.get("jobLocations") or []
|
||||
if (name := ((loc.get("location") or {}).get("name") or "").strip())]
|
||||
|
||||
@@ -47,7 +47,7 @@ from urllib.parse import quote
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads, Row
|
||||
from ._countries import name_of
|
||||
from ._shared import Get, Page, add_place, html_text, is_web_url, only, parse_dt, place
|
||||
from ._shared import Get, Page, add_place, html_text, only, parse_dt, place
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=10, text="detail", complete="stated",
|
||||
@@ -151,8 +151,6 @@ def rows(positions: list[dict[str, Any]], slug: str) -> list[Row]:
|
||||
source_id = str(p.get("id") or "")
|
||||
path = str(p.get("positionUrl") or "").strip()
|
||||
url = f"https://{host}{path}" if path.startswith("/") and path[1:2] != "/" else ""
|
||||
if not title or not source_id or not is_web_url(url):
|
||||
continue
|
||||
places = _places(p.get("locations") or p.get("standardizedLocations"))
|
||||
out.append({
|
||||
"source_id": source_id,
|
||||
|
||||
@@ -37,7 +37,7 @@ from urllib.parse import urlencode, urlsplit, urlunsplit
|
||||
from desk.sources import USER_AGENT
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import (Get, Page, Row, html_text, is_web_url, num, only, parse_dt, pay_line,
|
||||
from ._shared import (Get, Page, Row, html_text, num, only, parse_dt, pay_line,
|
||||
yearly_amount)
|
||||
|
||||
NAME = "freehire"
|
||||
@@ -170,8 +170,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
title = " ".join(html.unescape(str(hit.get("title") or "")).split())
|
||||
source_id = str(hit.get("public_slug") or "").strip()
|
||||
url = _url(str(hit.get("url") or ""))
|
||||
if not title or not source_id or not is_web_url(url):
|
||||
continue
|
||||
mode = str(hit.get("work_mode") or "").strip()
|
||||
places = _places(hit, mode)
|
||||
countries = [str(c) for c in hit.get("countries") or [] if c]
|
||||
|
||||
@@ -116,8 +116,6 @@ def rows(postings: list[dict[str, Any]], slug: str) -> list[Row]:
|
||||
for p in postings:
|
||||
title = (p.get("title") or "").strip()
|
||||
ext_id = str(p.get("extId") or "")
|
||||
if not title or not ext_id:
|
||||
continue
|
||||
locations = p.get("locations") or []
|
||||
places: list[Place] = []
|
||||
for loc in locations:
|
||||
|
||||
@@ -13,7 +13,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads, Row
|
||||
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place
|
||||
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place
|
||||
|
||||
NAME = "greenhouse"
|
||||
LABEL = "Greenhouse"
|
||||
@@ -149,8 +149,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data.get("jobs") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("absolute_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
departments = j.get("departments") or []
|
||||
ranges = j.get("pay_input_ranges") or []
|
||||
salary_min, salary_max, currency = _salary(ranges)
|
||||
|
||||
@@ -90,8 +90,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
for ad in data.get("jobAdDetails") or []:
|
||||
title = (ad.get("title") or "").strip()
|
||||
ad_id = str(ad.get("id") or "").strip()
|
||||
if not title or not ad_id:
|
||||
continue
|
||||
site = str(ad.get("site") or "").strip()
|
||||
country = str(ad.get("country") or "").strip()
|
||||
location = f"{site}, {country}" if site and country and country.lower() not in site.lower() \
|
||||
|
||||
@@ -30,7 +30,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Reads, Row
|
||||
from ._shared import Get, Page, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, html_text, only, parse_dt
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=100, text="listing", complete="end",
|
||||
@@ -106,8 +106,6 @@ def rows(items: list[ET.Element], slug: str) -> list[Row]:
|
||||
for item in items:
|
||||
title = (item.findtext("title") or "").strip()
|
||||
url = (item.findtext("link") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
out.append({
|
||||
"source_id": (item.findtext("guid") or url).strip(),
|
||||
"title": title,
|
||||
|
||||
@@ -127,8 +127,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
j = job.get("data") or {}
|
||||
title = (j.get("title") or "").strip()
|
||||
role = str(j.get("slug") or j.get("req_id") or "")
|
||||
if not title or not role:
|
||||
continue
|
||||
categories = j.get("categories") or []
|
||||
text = "\n".join(p for p in (html_text(j.get("description")),
|
||||
html_text(j.get("qualifications"))) if p)
|
||||
|
||||
@@ -36,7 +36,7 @@ from typing import Any
|
||||
from urllib.parse import urlencode, urlsplit
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt, place
|
||||
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt, place
|
||||
|
||||
NAME = "jobbank"
|
||||
LABEL = "Akademikernes Jobbank (Denmark)"
|
||||
@@ -132,8 +132,8 @@ def _query(ask: Ask, number: int) -> tuple[str, str]:
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
|
||||
"""One page of a search; the cursor is the page number. The next page
|
||||
follows a full page (counted before any item is dropped, so a dropped item
|
||||
does not pass for the end) while the ceiling is not reached."""
|
||||
follows a full page (counted before the runner drops an item, so a
|
||||
dropped item does not pass for the end) while the ceiling is not reached."""
|
||||
number = int(cursor or 1)
|
||||
url, remote = _query(ask, number)
|
||||
try:
|
||||
@@ -184,8 +184,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
out: list[Row] = []
|
||||
for item in decoded["items"]:
|
||||
link, title = item["link"], item["title"]
|
||||
if not title or not item["id"] or not is_web_url(link):
|
||||
continue
|
||||
line = _LINE.fullmatch(item["description"])
|
||||
kind = line.group("kind").strip() if line else ""
|
||||
company, regions, cut = _regions(line.group("rest")) if line else ("", [], False)
|
||||
|
||||
@@ -117,9 +117,9 @@ def _query(ask: Ask, number: int) -> str:
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
|
||||
"""One page of a search, newest first; the cursor is the page number.
|
||||
The next page follows a full page (counted before any item is dropped, so
|
||||
a dropped item does not pass for the end) while the 200-result ceiling is
|
||||
not reached."""
|
||||
The next page follows a full page (counted before the runner drops an
|
||||
item, so a dropped item does not pass for the end) while the 200-result
|
||||
ceiling is not reached."""
|
||||
number = int(cursor or 1)
|
||||
root = ET.fromstring(get(_query(ask, number)))
|
||||
channel = root.find("channel")
|
||||
@@ -143,8 +143,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
out: list[Row] = []
|
||||
for item in decoded["items"]:
|
||||
link, full_title = item["link"], item["title"]
|
||||
if not full_title or not is_web_url(link):
|
||||
continue
|
||||
description = item["description"]
|
||||
h4 = _H4.search(description)
|
||||
title = _text(h4.group(1)) if h4 else full_title
|
||||
|
||||
@@ -138,8 +138,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
for ad in decoded.get("jobAds") or []:
|
||||
ident = str(ad.get("jobAdId") or "").strip()
|
||||
title = _text(ad.get("title"))
|
||||
if not ident or not title:
|
||||
continue
|
||||
link = str(ad.get("jobAdUrl") or "").strip()
|
||||
town = _text(ad.get("municipality") or ad.get("postalDistrictName"))
|
||||
places: list[Place] = [place(town)] if town else []
|
||||
|
||||
@@ -176,8 +176,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
title = (j.get("title") or "").strip()
|
||||
job_id = str(j.get("id") or "").strip()
|
||||
path = str(j.get("url") or "").strip()
|
||||
if not title or not job_id or not _ROLE_PATH.fullmatch(path):
|
||||
continue
|
||||
function = str(j.get("function") or "").strip()
|
||||
places = [place(name) for name in dict.fromkeys(
|
||||
str(name).strip() for name in j.get("locations") or [] if str(name).strip())]
|
||||
@@ -186,7 +184,7 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
"title": title,
|
||||
"places": places,
|
||||
"location": "; ".join(pl["text"] for pl in places),
|
||||
"url": f"https://{_EMP}{path}",
|
||||
"url": f"https://{_EMP}{path}" if _ROLE_PATH.fullmatch(path) else "",
|
||||
"posted": _published(j.get("published_date")),
|
||||
"department": "" if function == "All functions" else function,
|
||||
"description": "",
|
||||
|
||||
@@ -22,7 +22,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads, Row
|
||||
from ._shared import add_place, html_text, is_web_url, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
from ._shared import add_place, html_text, num, only, parse_dt, path_slug_ok, pay_line, place, yearly_amount
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
@@ -123,8 +123,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data:
|
||||
title = (j.get("text") or "").strip()
|
||||
url = (j.get("hostedUrl") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
categories = j.get("categories") or {}
|
||||
description = j.get("description") or j.get("descriptionPlain") or ""
|
||||
salary_range = j.get("salaryRange") or {}
|
||||
|
||||
@@ -198,9 +198,9 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
|
||||
for role in search.get("requisitionList") or []:
|
||||
role_id = str(role.get("Id") or "").strip()
|
||||
title = _text(role.get("Title"))
|
||||
# The id goes into a URL, and a listing is external content.
|
||||
if not title or not re.fullmatch(r"[A-Za-z0-9_-]+", role_id):
|
||||
continue
|
||||
# The id goes into a URL, and a listing is external content: an id
|
||||
# of other characters gets no link, and the runner drops the row.
|
||||
safe = bool(re.fullmatch(r"[A-Za-z0-9_-]+", role_id))
|
||||
remote = _text(role.get("WorkplaceTypeCode")) or None
|
||||
at_office = (remote or "").lower() in ("ora_hybrid", "ora_on_site")
|
||||
places: list[Place] = []
|
||||
@@ -214,7 +214,7 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
|
||||
"title": title,
|
||||
"places": places,
|
||||
"location": "; ".join(p["text"] for p in places),
|
||||
"url": f"{_base(slug)}/job/{role_id}",
|
||||
"url": f"{_base(slug)}/job/{role_id}" if safe else "",
|
||||
"posted": parse_dt(role.get("PostedDate")),
|
||||
"department": (_text(role.get("JobFamily")) or _text(role.get("JobFunction"))
|
||||
or _text(role.get("Department")) or _text(role.get("Organization"))),
|
||||
|
||||
@@ -104,8 +104,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for position in root.findall("position"):
|
||||
job_id = (position.findtext("id") or "").strip()
|
||||
title = (position.findtext("name") or "").strip()
|
||||
if not job_id or not title:
|
||||
continue
|
||||
description = " ".join(
|
||||
(value.text or "") for value in position.findall("jobDescriptions/jobDescription/value"))
|
||||
salary_info = position.find("salaryInformation")
|
||||
|
||||
@@ -168,8 +168,6 @@ def rows(search: dict[str, Any], slug: str) -> list[Row]:
|
||||
for job in search.get("data", {}).get("jobs") or []:
|
||||
key = _key(job)
|
||||
title = _text(job.get("title"))
|
||||
if not title or not key:
|
||||
continue
|
||||
places: list[Place] = []
|
||||
for text in (job.get("location"), *(job.get("multi_location") or [])):
|
||||
if isinstance(text, str):
|
||||
|
||||
@@ -22,7 +22,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Reads
|
||||
from ._shared import HEADING, Row, html_text, is_web_url, label_slug_ok, num, only, pay_line, yearly_amount
|
||||
from ._shared import HEADING, Row, html_text, label_slug_ok, num, only, pay_line, yearly_amount
|
||||
|
||||
NAME = "pinpoint"
|
||||
LABEL = "Pinpoint"
|
||||
@@ -121,8 +121,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("url") or "").strip()
|
||||
source_id = str(j.get("id") or "")
|
||||
if not title or not source_id or not is_web_url(url):
|
||||
continue
|
||||
job = j.get("job") or {}
|
||||
low, high, currency, salary_text = _pay(j)
|
||||
workplace = str(j.get("workplace_type") or "").strip()
|
||||
|
||||
@@ -22,7 +22,7 @@ from urllib.parse import urlencode
|
||||
from desk.sources import strip_html
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import Get, Page, Row, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, html_text, only, parse_dt
|
||||
|
||||
NAME = "platsbanken"
|
||||
LABEL = "Platsbanken"
|
||||
@@ -108,8 +108,6 @@ def rows(decoded: Any, slug: str) -> list[Row]:
|
||||
for hit in decoded.get("hits") or []:
|
||||
title = _text(hit.get("headline") or "")
|
||||
url = (hit.get("webpage_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
addresses = hit.get("workplace_addresses") or [hit.get("workplace_address") or {}]
|
||||
places = [place for address in addresses if isinstance(address, dict)
|
||||
if (place := _place(address))]
|
||||
|
||||
@@ -13,7 +13,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads
|
||||
from ._shared import Row, add_place, html_text, is_web_url, label_slug_ok, num, only, parse_dt, pay_line, yearly_amount
|
||||
from ._shared import Row, add_place, html_text, label_slug_ok, num, only, parse_dt, pay_line, yearly_amount
|
||||
|
||||
NAME = "recruitee"
|
||||
LABEL = "Recruitee"
|
||||
@@ -105,8 +105,6 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
for j in data.get("offers") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
url = (j.get("careers_url") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
salary = j.get("salary") if isinstance(j.get("salary"), dict) else {}
|
||||
salary_min, salary_max, currency = yearly_amount(
|
||||
salary.get("min"), salary.get("max"), salary.get("currency"),
|
||||
|
||||
+14
-10
@@ -82,27 +82,31 @@ def parse(body: bytes, slug: str) -> list[Row]:
|
||||
raise ValueError("not a Rippling board: the answer is not a list")
|
||||
roles: dict[str, Row] = {}
|
||||
places: dict[str, list[Place]] = {}
|
||||
for entry in data:
|
||||
for index, entry in enumerate(data):
|
||||
job_id = str(entry.get("uuid") or "").strip()
|
||||
title = (entry.get("name") or "").strip()
|
||||
if not title or not _UUID.fullmatch(job_id):
|
||||
continue
|
||||
# The id goes into a URL: one of other characters gets no link, and
|
||||
# the runner drops the row. An entry without an id is a role of its
|
||||
# own, not merged with the others that lack one.
|
||||
key = job_id or f"#{index}"
|
||||
place = str(((entry.get("workLocation") or {}).get("label")) or "").strip()
|
||||
places.setdefault(job_id, [])
|
||||
places.setdefault(key, [])
|
||||
if place:
|
||||
add_place(places[job_id], _place(place))
|
||||
roles.setdefault(job_id, {
|
||||
add_place(places[key], _place(place))
|
||||
roles.setdefault(key, {
|
||||
"source_id": job_id,
|
||||
"title": title,
|
||||
"url": f"https://{_SITE}/{slug}/jobs/{job_id}",
|
||||
"url": f"https://{_SITE}/{slug}/jobs/{job_id}" if _UUID.fullmatch(job_id) else "",
|
||||
"posted": None,
|
||||
"department": str(((entry.get("department") or {}).get("label")) or "").strip(),
|
||||
"description": "",
|
||||
"salary_text": "", # the board's list states no pay
|
||||
})
|
||||
for job_id, row in roles.items():
|
||||
row["places"] = places[job_id]
|
||||
row["location"] = "; ".join(p["text"] for p in places[job_id])
|
||||
if title and not roles[key]["title"]:
|
||||
roles[key]["title"] = title # an entry of the role that has one
|
||||
for key, row in roles.items():
|
||||
row["places"] = places[key]
|
||||
row["location"] = "; ".join(p["text"] for p in places[key])
|
||||
return list(roles.values())
|
||||
|
||||
|
||||
|
||||
@@ -115,8 +115,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
for j in data.get("content") or []:
|
||||
title = (j.get("name") or "").strip()
|
||||
job_id = str(j.get("id") or "")
|
||||
if not title or not job_id:
|
||||
continue
|
||||
location = j.get("location") or {}
|
||||
loc_text = (location.get("fullLocation")
|
||||
or ", ".join(p for p in (location.get("city"), location.get("country")) if p))
|
||||
|
||||
@@ -63,7 +63,7 @@ from desk.sources import strip_html
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._countries import NAMES
|
||||
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt
|
||||
|
||||
NAME = "successfactors"
|
||||
LABEL = "SuccessFactors"
|
||||
@@ -247,8 +247,9 @@ def _tile_fields(chunk: str) -> tuple[str, str, str]:
|
||||
def _entries(text: str) -> list[dict[str, Any]]:
|
||||
"""The roles of one search page, in the order shown, whichever layout it is
|
||||
in and before the prefix filter: `id`, `href` (as written, relative),
|
||||
`title`, `place`, `date` and `department`. A role is listed once, and a row
|
||||
without a link to a role is not one."""
|
||||
`title`, `place`, `date` and `department`. A role is listed once. The id
|
||||
and the title are in the role's link, so a row without one is handed over
|
||||
with an empty `id`, `href` and `title`, for the runner to drop and count."""
|
||||
table = _ROW.split(text)[1:]
|
||||
chunks = table or _TILE.split(text)[1:]
|
||||
out: list[dict[str, Any]] = []
|
||||
@@ -256,10 +257,12 @@ def _entries(text: str) -> list[dict[str, Any]]:
|
||||
for chunk in chunks:
|
||||
if table:
|
||||
chunk = chunk.split("</tr>", 1)[0]
|
||||
if not (link := _LINK.search(chunk)):
|
||||
continue
|
||||
href, job_id, title = html.unescape(link.group(1)), link.group(2), _text(link.group(3))
|
||||
if job_id in seen:
|
||||
if link := _LINK.search(chunk):
|
||||
href, job_id, title = (html.unescape(link.group(1)), link.group(2),
|
||||
_text(link.group(3)))
|
||||
else:
|
||||
href = job_id = title = ""
|
||||
if job_id and job_id in seen:
|
||||
continue
|
||||
seen.add(job_id)
|
||||
place, date, department = (_table_fields if table else _tile_fields)(chunk)
|
||||
@@ -283,12 +286,14 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
if not _SEARCH_PAGE.search(text):
|
||||
raise ValueError("not a SuccessFactors search page")
|
||||
found = _entries(text)
|
||||
ids = [entry["id"] for entry in found if entry["id"]]
|
||||
total = _total(text)
|
||||
if found and found[0]["id"] == previous:
|
||||
first = ids[0] if ids else None
|
||||
if first is not None and first == previous:
|
||||
return text, [], total, (start, previous)
|
||||
more = bool(found) and (total is None or start + len(found) < total)
|
||||
return (text, [{"id": entry["id"]} for entry in found], total,
|
||||
(start + len(found), found[0]["id"]) if more else None)
|
||||
return (text, [{"id": job_id} for job_id in ids], total,
|
||||
(start + len(found), first) if more else None)
|
||||
|
||||
|
||||
def rows(decoded: str, slug: str) -> list[Row]:
|
||||
@@ -296,17 +301,21 @@ def rows(decoded: str, slug: str) -> list[Row]:
|
||||
slug's prefix when it has one; the id in it is the requisition's `source_id`.
|
||||
The table and the tile layout give a title, a location, a date and, where
|
||||
the site has the column, a department; nothing else about a role is on
|
||||
the list, so its description and pay are empty."""
|
||||
the list, so its description and pay are empty.
|
||||
|
||||
A role linked on another host, or outside the prefix, is another board's
|
||||
(another site, or another brand of a shared site), and is left out: it is
|
||||
no role of this company. A row with no link or no title is handed over,
|
||||
and the runner drops it."""
|
||||
host, prefix = _parts(slug)
|
||||
base = f"https://{host}/"
|
||||
out = []
|
||||
for entry in _entries(decoded):
|
||||
url = urljoin(base, entry["href"])
|
||||
if urlsplit(url).hostname != host or not is_web_url(url):
|
||||
url = urljoin(base, entry["href"]) if entry["href"] else ""
|
||||
if url and urlsplit(url).hostname != host:
|
||||
continue
|
||||
if prefix and not urlsplit(url).path.lower().startswith(f"{prefix.lower()}/job/"):
|
||||
continue
|
||||
if not entry["title"]:
|
||||
if url and prefix and not urlsplit(url).path.lower().startswith(
|
||||
f"{prefix.lower()}/job/"):
|
||||
continue
|
||||
out.append({
|
||||
"source_id": entry["id"],
|
||||
|
||||
@@ -15,7 +15,7 @@ from typing import Any
|
||||
|
||||
|
||||
from ._contract import Ask, Fields, Place, Reads
|
||||
from ._shared import Get, Page, Row, add_place, html_text, is_web_url, only, parse_dt
|
||||
from ._shared import Get, Page, Row, add_place, html_text, only, parse_dt
|
||||
|
||||
NAME = "teamtailor"
|
||||
LABEL = "Teamtailor"
|
||||
@@ -115,8 +115,6 @@ def rows(items: list[ET.Element], slug: str) -> list[Row]:
|
||||
for item in items:
|
||||
title = (item.findtext("title") or "").strip()
|
||||
url = (item.findtext("link") or "").strip()
|
||||
if not title or not is_web_url(url):
|
||||
continue
|
||||
status = (item.findtext("remoteStatus") or "").strip()
|
||||
remote = {"fully": True, "hybrid": False, "onsite": False}.get(status)
|
||||
places: list[Place] = []
|
||||
|
||||
@@ -102,8 +102,6 @@ def rows(data: dict[str, Any], slug: str) -> list[Row]:
|
||||
for j in data.get("results") or []:
|
||||
title = (j.get("title") or "").strip()
|
||||
shortcode = (j.get("shortcode") or "").strip()
|
||||
if not title or not shortcode:
|
||||
continue
|
||||
workplace = str(j.get("workplace") or "").strip()
|
||||
remote = bool(j.get("remote")) or workplace == "remote"
|
||||
places = _places(j, remote)
|
||||
|
||||
@@ -124,8 +124,8 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
total = stated if isinstance(stated, int) and not isinstance(stated, bool) else None
|
||||
postings = data.get("jobPostings") or []
|
||||
# Every posting is counted, a path-less one (LEGO lists one that has only
|
||||
# `bulletFields`) by its position, so the parser dropping it does not
|
||||
# leave the board short of its total.
|
||||
# `bulletFields`) by its position, so the runner dropping its row does
|
||||
# not leave the board short of its total.
|
||||
results = [{"id": _posting_id(p, offset + i)} for i, p in enumerate(postings)]
|
||||
following = offset + len(postings)
|
||||
more = bool(postings) and (total is None or following < total)
|
||||
@@ -274,13 +274,13 @@ def rows(decoded: dict[str, Any], slug: str) -> list[Row]:
|
||||
`bulletFields`, which lead with the requisition id, and on some tenants
|
||||
`remoteType`. No description, no date and no country: `detail` has
|
||||
those, and the places of a role that "2 Locations" hides. The id is the requisition id where the path carries it, and the
|
||||
path where not; a posting with no title or no path is skipped."""
|
||||
path where not. A posting with no path has neither an id nor a link, and
|
||||
the runner drops it."""
|
||||
out = []
|
||||
for p in decoded.get("jobPostings") or []:
|
||||
title = (p.get("title") or "").strip()
|
||||
path = (p.get("externalPath") or "").strip()
|
||||
if not title or not _ROLE_PATH.fullmatch(path):
|
||||
continue
|
||||
path = path if _ROLE_PATH.fullmatch(path) else ""
|
||||
remote = _remote(p.get("remoteType"))
|
||||
place = _place(p.get("locationsText"), remote)
|
||||
out.append({
|
||||
@@ -288,7 +288,7 @@ def rows(decoded: dict[str, Any], slug: str) -> list[Row]:
|
||||
"title": title,
|
||||
"places": [place] if place else [],
|
||||
"location": place["text"] if place else "",
|
||||
"url": f"https://{_host(slug)}/{_parts(slug)[2]}{path}", # type: ignore[index]
|
||||
"url": f"https://{_host(slug)}/{_parts(slug)[2]}{path}" if path else "", # type: ignore[index]
|
||||
"posted": _posted(p.get("postedOn") or ""),
|
||||
"department": "",
|
||||
"description": "",
|
||||
|
||||
+2
-1
@@ -19,7 +19,8 @@ roles close once no search has seen them for `STALE_DAYS` (see
|
||||
`app/jobs/pull.py`).
|
||||
|
||||
In a pull a search is read by `_runner.read` under the prefix `search`, as a
|
||||
board is under `ats`: the 10% guard holds it, `_runner.stamp` records it,
|
||||
board is under `ats`: the runner drops a row without an id, a title or a
|
||||
link, the 10% guards hold it, `_runner.stamp` records it,
|
||||
and a search that fails or is suspect is kept as failing under
|
||||
`search:failing:<target>` (`_runner.record_failure`) until it answers, which
|
||||
the settle reads as it reads a failing board's.
|
||||
|
||||
Reference in new issue
Block a user