Fetch terms narrow what big boards read with the wide reach send (WP F2)
An agent has fetch_terms (migration 23, API, a chip field in the agent
editor). The terms of a pull are the union over the enabled agents, none
when one of them has none. They go only to wide reads: a company tagged All
roles, or a reach ask_for drops as too many countries for a one host; a
board's {"terms": false} switches them off. SmartRecruiters q,
SuccessFactors q, Workday searchText, Phenom keywords, Eightfold query and
Oracle keyword declare query="one"; the runner makes one series per term
times the country series, drops the places first over MAX_SERIES, then the
terms. A term read is narrowed: it writes no ats:size, closes nothing and
prunes no per-role state, and the 10% guard holds it to its own
fingerprint. The Companies reach line counts the terms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3917b31209
commit
0fd45b333a
28 files changed
+839
-113
No files matched your search
@@ -68,7 +68,8 @@ class Filters(_Frozen):
|
||||
"""What the host filters at the source, per dimension. `country` takes
|
||||
ISO 3166 alpha-2 codes, `place` place names, `remote` a bool, `query`
|
||||
words. The runner never hands a reader a value on a dimension it
|
||||
declared `none`."""
|
||||
declared `none`. A board host with `query="one"` also takes the agents'
|
||||
fetch terms, one term a series (`Ask.terms`, see `_runner.ask_for`)."""
|
||||
|
||||
country: Filter = "none"
|
||||
place: Filter = "none"
|
||||
@@ -159,13 +160,19 @@ class Ask(_Frozen):
|
||||
series of their own: a reader never gets both `remote=True` and a
|
||||
country or a place. With `remote="ids"`, `remote_ids` holds the host's
|
||||
ids of its remote values (`facets(...)["remote"]`, any key), and
|
||||
`remote` stays True."""
|
||||
`remote` stays True.
|
||||
|
||||
`terms` are the agents' fetch terms a board is narrowed by (see
|
||||
`app/jobs/reach.py`). The runner makes one series per term, each with
|
||||
that term as `query`, so a reader only ever reads `query`. A read with
|
||||
terms is never the whole board: it sets no size and closes nothing."""
|
||||
|
||||
countries: tuple[str, ...] = ()
|
||||
places: tuple[str, ...] = ()
|
||||
remote: bool | None = None
|
||||
remote_ids: tuple[str, ...] = ()
|
||||
query: str = ""
|
||||
terms: tuple[str, ...] = ()
|
||||
params: dict[str, Any] = {}
|
||||
|
||||
|
||||
|
||||
+100
-25
@@ -124,7 +124,9 @@ class Listing:
|
||||
`source_id`s the read listed: this pull's, or the whole sweep's once it
|
||||
reached the end. `complete` says `ids` are the whole list. `sweep` is the
|
||||
state to keep for the next pull, None when no sweep is under way.
|
||||
`filtered` says the read asked the host for a part of the list."""
|
||||
`filtered` says the read asked the host for the roles in some places,
|
||||
`narrowed` for the roles matching words (`narrowed()`): the ids of
|
||||
neither are the whole board."""
|
||||
|
||||
rows: list[Row]
|
||||
complete: bool
|
||||
@@ -132,6 +134,7 @@ class Listing:
|
||||
filtered: bool = False
|
||||
sweep: dict[str, Any] | None = None
|
||||
asks: str = ""
|
||||
narrowed: bool = False
|
||||
|
||||
|
||||
# --------------------------------------------------------------- modules
|
||||
@@ -289,11 +292,39 @@ def resolve_key(key: Key, keys: Mapping[str, Any] | None) -> dict[str, str]:
|
||||
return out
|
||||
|
||||
|
||||
#: The most series a reach may cost on a `one` filter: a reach of more
|
||||
#: countries (the wide reach, Europe) reads such a host unfiltered instead.
|
||||
#: The most series a reach may cost: a reach of more countries on a `one`
|
||||
#: filter (the wide reach, Europe) reads such a host unfiltered instead, and
|
||||
#: fetch terms times the reach's series may not cost more either.
|
||||
MAX_SERIES = 8
|
||||
|
||||
|
||||
def _terms(value: Any) -> tuple[str, ...]:
|
||||
"""Fetch terms as given in a reach: stripped, empty ones dropped, one of
|
||||
each regardless of case, in the order given."""
|
||||
seen: set[str] = set()
|
||||
out: list[str] = []
|
||||
for term in _values(value):
|
||||
if term.casefold() not in seen:
|
||||
seen.add(term.casefold())
|
||||
out.append(term)
|
||||
return tuple(out)
|
||||
|
||||
|
||||
def _cost(reads: Reads, wanted: Mapping[str, Any]) -> int:
|
||||
"""How many series the location part of `wanted` takes on this reader:
|
||||
one per value of a `one` dimension (the product of two), one for a
|
||||
`many` or `ids` dimension, and one more for remote beside places, which
|
||||
the runner asks as a series of its own (see `series`)."""
|
||||
count = 1
|
||||
for dimension in ("country", "place"):
|
||||
values = wanted.get(dimension, ())
|
||||
if values and getattr(reads.filters, dimension) == "one":
|
||||
count *= len(values)
|
||||
if wanted.get("remote") and (wanted.get("country") or wanted.get("place")):
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None = None,
|
||||
reach: Mapping[str, Any] | None = None, *,
|
||||
keys: Mapping[str, Any] | None = None) -> Ask:
|
||||
@@ -306,30 +337,63 @@ def ask_for(module: ModuleType, subscription_params: Mapping[str, Any] | None =
|
||||
dropped. A `remote` of False asks nothing. A search's params go whole into
|
||||
`Ask.params`; a reader with `READS.key` gets the resolved key under
|
||||
`Ask.params["key"]` (see `resolve_key`, which raises when it is missing).
|
||||
A module without `READS` gets `Ask()`."""
|
||||
A module without `READS` gets `Ask()`.
|
||||
|
||||
Fetch terms (`reach["terms"]`) go only to a board reader that declares
|
||||
`query="one"`, and only on a wide read: `reach["wide"]` (a company
|
||||
tagged All roles), or a reach this rule just dropped as too many
|
||||
countries. A board whose params say `"terms": false` or name a query of
|
||||
their own takes none. Terms times the reach's series (`_cost`) may cost
|
||||
at most `MAX_SERIES`: over it, the place part of the reach goes first,
|
||||
whole, since remote without its countries would read only the remote
|
||||
roles; over it with terms alone, the terms go too and the board is read
|
||||
unfiltered."""
|
||||
if not declared(module):
|
||||
return Ask()
|
||||
reads: Reads = module.READS
|
||||
params = dict(subscription_params or {})
|
||||
reached = _dimensions(module, reach or {})
|
||||
given = reach or {}
|
||||
reached = _dimensions(module, given)
|
||||
wide = bool(given.get("wide"))
|
||||
if reads.filters.country == "one" and len(reached.get("country", ())) > MAX_SERIES:
|
||||
reached = {}
|
||||
wide = True
|
||||
wanted = reached | _dimensions(module, params)
|
||||
if wanted.get("remote") is False:
|
||||
del wanted["remote"]
|
||||
fields: dict[str, Any] = {_ASK_FIELD[name]: value for name, value in wanted.items()
|
||||
if name in reads.filters.declared()}
|
||||
declared_now = reads.filters.declared()
|
||||
wanted = {name: value for name, value in wanted.items() if name in declared_now}
|
||||
terms = _terms(given.get("terms"))
|
||||
if (not wide or reads.scope != "board" or reads.filters.query != "one"
|
||||
or params.get("terms") is False or "query" in wanted):
|
||||
terms = ()
|
||||
if terms and len(terms) * _cost(reads, wanted) > MAX_SERIES:
|
||||
wanted = {name: value for name, value in wanted.items()
|
||||
if name not in ("country", "place", "remote")}
|
||||
if len(terms) > MAX_SERIES:
|
||||
terms = ()
|
||||
fields: dict[str, Any] = {_ASK_FIELD[name]: value for name, value in wanted.items()}
|
||||
extra: dict[str, Any] = dict(params) if reads.scope == "search" else {}
|
||||
if reads.key is not None:
|
||||
extra["key"] = resolve_key(reads.key, keys)
|
||||
return Ask(**fields, params=extra)
|
||||
return Ask(**fields, terms=terms, params=extra)
|
||||
|
||||
|
||||
def filtered(ask: Ask) -> bool:
|
||||
"""Whether `ask` reads a part of a board: a country, a place or remote."""
|
||||
"""Whether `ask` reads a part of a board by place: a country, a place or
|
||||
remote. Terms are `narrowed`."""
|
||||
return bool(ask.countries or ask.places or ask.remote is not None)
|
||||
|
||||
|
||||
def narrowed(module: ModuleType, ask: Ask) -> bool:
|
||||
"""Whether `ask` reads a board by words: fetch terms, or a board's own
|
||||
query. Such a read is never the whole board, nor the board within its
|
||||
places, so it sets no size and closes nothing. A search's query is what
|
||||
the search is, and does not count."""
|
||||
board = not declared(module) or module.READS.scope == "board"
|
||||
return bool(ask.terms) or (board and bool(ask.query))
|
||||
|
||||
|
||||
def series(module: ModuleType, ask: Ask,
|
||||
facets: Mapping[str, Mapping[str, str]] | None = None) -> list[Ask]:
|
||||
"""The series of requests `ask` takes on this reader. Remote with a
|
||||
@@ -340,15 +404,22 @@ def series(module: ModuleType, ask: Ask,
|
||||
values `facets` does not know are dropped, and a part with none left is
|
||||
no series, since the host has nothing of it; a dimension `facets` does
|
||||
not state at all is read unfiltered, and so is remote alone, while the
|
||||
remote part beside places is then left out. `[ask]` for a module
|
||||
without `READS`."""
|
||||
remote part beside places is then left out. Each of those is then asked
|
||||
once per fetch term (`ask.terms`), with the term as `query`: the host
|
||||
intersects a keyword with its facets, as the survey found on every
|
||||
host, so the union by id is "one of the terms, in these places". `[ask]`
|
||||
for a module without `READS`."""
|
||||
if not declared(module):
|
||||
return [ask]
|
||||
if ask.remote and (ask.countries or ask.places):
|
||||
return [*_series(module, ask.model_copy(update={"remote": None}), facets),
|
||||
*_series(module, ask.model_copy(update={"countries": (), "places": ()}),
|
||||
facets, beside=True)]
|
||||
return _series(module, ask, facets)
|
||||
parts = [*_series(module, ask.model_copy(update={"remote": None}), facets),
|
||||
*_series(module, ask.model_copy(update={"countries": (), "places": ()}),
|
||||
facets, beside=True)]
|
||||
else:
|
||||
parts = _series(module, ask, facets)
|
||||
if not ask.terms:
|
||||
return parts
|
||||
return [part.model_copy(update={"query": term}) for term in ask.terms for part in parts]
|
||||
|
||||
|
||||
def _series(module: ModuleType, ask: Ask, facets: Mapping[str, Mapping[str, str]] | None,
|
||||
@@ -455,12 +526,13 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
ask = ask or Ask()
|
||||
asks = series(module, ask, facets)
|
||||
is_filtered = filtered(ask)
|
||||
by_words = narrowed(module, ask)
|
||||
key = fingerprint(asks)
|
||||
if not asks:
|
||||
return Listing([], True, [], is_filtered, None, key)
|
||||
return Listing([], True, [], is_filtered, None, key, by_words)
|
||||
if not hasattr(module, "page"):
|
||||
rows = _whole(module, slug, get, asks, limits)
|
||||
return Listing(list(rows.values()), True, list(rows), is_filtered, None, key)
|
||||
return Listing(list(rows.values()), True, list(rows), is_filtered, None, key, by_words)
|
||||
|
||||
reads: Reads | None = module.READS if declared(module) else None
|
||||
never = reads is not None and reads.complete == "never"
|
||||
@@ -518,15 +590,15 @@ def read_listing(module: ModuleType, slug: str, get: Get, ask: Ask | None = None
|
||||
|
||||
listed = list(dict.fromkeys(earlier + list(rows)))
|
||||
if not stop:
|
||||
return Listing(list(rows.values()), whole, listed, is_filtered, None, key)
|
||||
return Listing(list(rows.values()), whole, listed, is_filtered, None, key, by_words)
|
||||
log.warning("readers: %s:%s: stopped after %d pages (%s); not complete",
|
||||
module.NAME, slug, pages, stop)
|
||||
at_ceiling = ceiling is not None and len(seen) >= ceiling
|
||||
if not can_sweep or never or at_ceiling or not _storable(cursor):
|
||||
return Listing(list(rows.values()), False, list(rows), is_filtered, None, key)
|
||||
return Listing(list(rows.values()), False, list(rows), is_filtered, None, key, by_words)
|
||||
keep = {"asks": key, "series": index, "cursor": cursor, "seen": sorted(seen),
|
||||
"rows": listed, "whole": whole}
|
||||
return Listing(list(rows.values()), False, list(rows), is_filtered, keep, key)
|
||||
return Listing(list(rows.values()), False, list(rows), is_filtered, keep, key, by_words)
|
||||
|
||||
|
||||
def _storable(cursor: Any) -> bool:
|
||||
@@ -577,9 +649,10 @@ def read(state: MutableMapping[str, str], prefix: str, target: str, module: Modu
|
||||
limits: Limits | None = None, can_sweep: bool = True) -> Listing:
|
||||
"""`read_listing` for one target with its state under `<prefix>:...:<target>`:
|
||||
the facets it needs, the sweep it continues, and the 10% guard. A read
|
||||
that is filtered or whose reader declares `empty="suspect"`, and that
|
||||
listed under `SUSPECT_SHARE` of what the same read listed last time,
|
||||
raises `Suspect`. Nothing is written here but the facets; `stamp` keeps
|
||||
that is filtered, narrowed by terms, or whose reader declares
|
||||
`empty="suspect"`, and that listed under `SUSPECT_SHARE` of what the same
|
||||
read listed last time (the same `fingerprint`, so a term read is only
|
||||
held to a term read with the same terms and places), raises `Suspect`. Nothing is written here but the facets; `stamp` keeps
|
||||
the rest once the rows are stored."""
|
||||
ask = ask or Ask()
|
||||
facets = facets_for(module, slug, get, ask, state, f"{prefix}:facets:{target}")
|
||||
@@ -588,7 +661,7 @@ def read(state: MutableMapping[str, str], prefix: str, target: str, module: Modu
|
||||
suspect = declared(module) and module.READS.empty == "suspect"
|
||||
last = _load(state, f"{prefix}:count:{target}")
|
||||
count = int(last.get("count") or 0) if last.get("asks") == listing.asks else 0
|
||||
if ((listing.filtered or suspect) and listing.sweep is None and count
|
||||
if ((listing.filtered or listing.narrowed or suspect) and listing.sweep is None and count
|
||||
and len(listing.ids) < SUSPECT_SHARE * count):
|
||||
raise Suspect(f"listed {len(listing.ids)} roles, under "
|
||||
f"{SUSPECT_SHARE:.0%} of the {count} it listed last time")
|
||||
@@ -599,12 +672,14 @@ def stamp(state: MutableMapping[str, str], prefix: str, target: str, run: str,
|
||||
listing: Listing, ids: list[str]) -> None:
|
||||
"""Record a read whose rows are stored: `ok` (the run's marker),
|
||||
`complete`, `ids` (the compound ids, what the pull closes against),
|
||||
`filtered`, the sweep to continue or none, and the count for the 10%
|
||||
`filtered`, `narrowed` (a read by terms, which the pull never closes
|
||||
against), the sweep to continue or none, and the count for the 10%
|
||||
guard once a read is whole."""
|
||||
state[f"{prefix}:ok:{target}"] = run
|
||||
state[f"{prefix}:complete:{target}"] = "1" if listing.complete else ""
|
||||
state[f"{prefix}:ids:{target}"] = json.dumps(ids)
|
||||
state[f"{prefix}:filtered:{target}"] = "1" if listing.filtered else ""
|
||||
state[f"{prefix}:narrowed:{target}"] = "1" if listing.narrowed else ""
|
||||
if listing.sweep is not None:
|
||||
state[f"{prefix}:sweep:{target}"] = json.dumps(listing.sweep)
|
||||
else:
|
||||
|
||||
@@ -29,6 +29,11 @@ name in `_countries` is read unfiltered.
|
||||
|
||||
Discovery names Eightfold from an `eightfold.ai` link, and the host and the
|
||||
domain from one that carries `domain=`.
|
||||
|
||||
A fetch term goes as `query=`, one a request (`Filters(query="one")`);
|
||||
it matches title, text and skills, and is an AND with `location` (survey,
|
||||
2026-09-30: `software` with Sweden answered 102 of Ericsson's 154 Swedish
|
||||
roles).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -48,7 +53,7 @@ from ._shared import Get, Page, add_place, is_web_url, only, parse_dt, place
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=10, text="detail", complete="stated",
|
||||
places="many", filters=Filters(country="one"),
|
||||
places="many", filters=Filters(country="one", query="one"),
|
||||
fields=Fields(posted="listed", remote="stated"),
|
||||
robots="read: robots.txt allows /careers (checked 2026-09-29)",
|
||||
)
|
||||
@@ -97,13 +102,15 @@ def careers_url(slug: str) -> str:
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask) -> Page:
|
||||
"""One page of `/api/pcsx/search` -> `{"status": 200, "data": {"positions":
|
||||
[...], "count": n}}`, in the one country `ask` names; the next starts at
|
||||
the offset after this page's positions, and an empty page ends the read."""
|
||||
the offset after this page's positions, and an empty page ends the read.
|
||||
`query` goes as `query=`."""
|
||||
host, domain = parts(slug)
|
||||
start = int(cursor or 0)
|
||||
country = name_of(ask.countries[0]) if ask.countries else ""
|
||||
where = f"&location={quote(country)}" if country else ""
|
||||
words = f"&query={quote(ask.query, safe='')}" if ask.query else ""
|
||||
answer = json.loads(get(f"https://{host}/api/pcsx/search?domain={domain}"
|
||||
f"&sort_by=timestamp&start={start}{where}"))
|
||||
f"&sort_by=timestamp&start={start}{where}{words}"))
|
||||
data = answer.get("data") if isinstance(answer, dict) else None
|
||||
positions = data.get("positions") if isinstance(data, dict) else None
|
||||
if not isinstance(positions, list):
|
||||
|
||||
@@ -20,6 +20,10 @@ the pod does not know answers the whole board (review). Remote is
|
||||
`selectedWorkplaceTypesFacet=ORA_REMOTE` (`remote="one"`); two facets are an
|
||||
intersection (Finland and remote answered 0 on 2026-09-30), which is why the
|
||||
runner asks for remote in a series of its own.
|
||||
|
||||
A fetch term goes into the finder as `keyword=engineer`, one a request
|
||||
(`Filters(query="one")`); it matches the text (survey, 2026-09-30: 468 of
|
||||
Nokia's 554).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -54,7 +58,7 @@ PATTERNS = (
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=200, text="detail", teaser=True,
|
||||
complete="stated", places="many", filters=Filters(country="ids", remote="one"),
|
||||
complete="stated", places="many", filters=Filters(country="ids", remote="one", query="one"),
|
||||
fields=Fields(posted="trusted", remote="stated"),
|
||||
robots=("read: robots.txt allows /hcmUI/CandidateExperience/en/sites/<site>/jobs "
|
||||
"(checked 2026-09-29)"),
|
||||
@@ -126,9 +130,10 @@ FACET_LIMIT = 10
|
||||
|
||||
def _selected(ask: Ask) -> str:
|
||||
"""The finder's facet parameters for `ask`, each ending in a comma: the
|
||||
location ids (none when one is `*`, a country the pod did not list) and
|
||||
remote."""
|
||||
out = ""
|
||||
keyword, the location ids (none when one is `*`, a country the pod did
|
||||
not list) and remote. The keyword is escaped whole, so a comma or a
|
||||
semicolon in it cannot start another finder parameter."""
|
||||
out = f"keyword={quote(ask.query, safe='')}," if ask.query else ""
|
||||
ids = [i for i in ask.countries if re.fullmatch(r"\d+", i)]
|
||||
if ids and "*" not in ask.countries:
|
||||
out += f"selectedLocationsFacet={'%3B'.join(ids)},"
|
||||
|
||||
@@ -17,6 +17,10 @@ matches. Philips answered 111 of 873 for Netherlands and Germany on
|
||||
2026-09-30. There is no remote field to filter on. A code with no English
|
||||
name in `_countries` is left out of the list, and a list left empty reads the
|
||||
whole board.
|
||||
|
||||
A fetch term goes as `keywords` in the body, one a request
|
||||
(`Filters(query="one")`); it matches the text, not only the title, and is an
|
||||
AND with `selected_fields` (survey, 2026-09-30, Philips).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -46,7 +50,7 @@ PATTERNS = (
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=100, text="detail", teaser=True,
|
||||
complete="stated", places="many", filters=Filters(country="many"),
|
||||
complete="stated", places="many", filters=Filters(country="many", query="one"),
|
||||
fields=Fields(posted="trusted", remote="stated"),
|
||||
robots="read: robots.txt allows /<country>/<lang>/search-results (checked 2026-09-29)",
|
||||
)
|
||||
@@ -87,10 +91,11 @@ def careers_url(slug: str) -> str:
|
||||
return f"{_root(slug)}/search-results"
|
||||
|
||||
|
||||
def _body(slug: str, offset: int, countries: tuple[str, ...] = ()) -> dict[str, Any]:
|
||||
def _body(slug: str, offset: int, countries: tuple[str, ...] = (),
|
||||
keywords: str = "") -> dict[str, Any]:
|
||||
"""What the site's search page sends for a listing of every role, newest
|
||||
first, less the parts that only tell the page what to draw, in the
|
||||
`countries` (ISO codes) given. `lang` is `en_global` for `/global/en` and
|
||||
`countries` (ISO codes) given and matching `keywords` when given. `lang` is `en_global` for `/global/en` and
|
||||
`en_us` for `/us/en`, the site's own spelling on the three sites read."""
|
||||
_host, country, lang = _parts(slug)
|
||||
names = [name for code in countries if (name := name_of(code))]
|
||||
@@ -100,7 +105,7 @@ def _body(slug: str, offset: int, countries: tuple[str, ...] = ()) -> dict[str,
|
||||
"subsearch": "", "from": offset, "jobs": True, "counts": True, "all_fetch": False,
|
||||
"pageType": "landingPage", "size": PAGE_SIZE, "clearAll": False,
|
||||
"jdsource": "facets", "isSliderEnable": False, "siteType": "external",
|
||||
"location": "", "keywords": "", "global": True,
|
||||
"location": "", "keywords": keywords, "global": True,
|
||||
"selected_fields": {"country": names} if names else {},
|
||||
"sort": {"order": "desc", "field": "postedDate"}, "locationData": {},
|
||||
}
|
||||
@@ -118,7 +123,7 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
offset = int(cursor or 0)
|
||||
host = _parts(slug)[0]
|
||||
data = json.loads(get(f"https://{host}/widgets", method="POST",
|
||||
payload=_body(slug, offset, ask.countries)))
|
||||
payload=_body(slug, offset, ask.countries, ask.query)))
|
||||
search = data.get("refineSearch")
|
||||
if not isinstance(search, dict) or search.get("status", 200) != 200:
|
||||
raise ValueError("not a Phenom search answer: no refineSearch")
|
||||
|
||||
@@ -8,6 +8,11 @@ A country is filtered on the server, `?country=dk`, one a request
|
||||
(`Filters(country="one")`): Bosch's DE, SE and DK summed to what the app held
|
||||
(review, 2026-09-29), and DK answered 2 of 4,830 on 2026-09-30. The API
|
||||
ignores a remote parameter, so remote is not filtered.
|
||||
|
||||
A fetch term goes as `?q=engineer`, one a request (`Filters(query="one")`):
|
||||
it matches title and text, and with `country=` it is an AND (survey,
|
||||
2026-09-30: `q=engineer` answered 1,713 of Bosch's 4,830, with `country=dk`
|
||||
none). How several words in one `q` combine is untested.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -16,6 +21,7 @@ import json
|
||||
import re
|
||||
from collections.abc import Callable
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
from ._contract import Ask, Fields, Filters, Place, Reads
|
||||
from ._shared import Get, Page, Row, only, parse_dt, path_slug_ok
|
||||
@@ -30,7 +36,7 @@ PATTERNS = (
|
||||
#: An unknown identifier answers an empty list, which is `empty="suspect"`.
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=100, text="none", complete="stated",
|
||||
places="one", empty="suspect", filters=Filters(country="one"),
|
||||
places="one", empty="suspect", filters=Filters(country="one", query="one"),
|
||||
fields=Fields(posted="trusted", remote="stated", country="stated", seniority="stated",
|
||||
employment="stated"),
|
||||
robots="api: no robots.txt check needed (public board API), 2026-09-30",
|
||||
@@ -66,12 +72,13 @@ def probe_ok(body: bytes) -> bool:
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
"""One page, in the one country `ask` names, which the API takes in lower
|
||||
case."""
|
||||
case, and matching its `query` (`q=`)."""
|
||||
offset = int(cursor or 0)
|
||||
country = ask.countries[0].lower() if ask.countries else ""
|
||||
where = f"&country={country}" if re.fullmatch(r"[a-z]{2}", country) else ""
|
||||
words = f"&q={quote(ask.query, safe='')}" if ask.query else ""
|
||||
data = json.loads(get(f"https://{_API}/v1/companies/{slug}/postings"
|
||||
f"?offset={offset}&limit={PAGE_SIZE}{where}"))
|
||||
f"?offset={offset}&limit={PAGE_SIZE}{where}{words}"))
|
||||
content = data.get("content") or []
|
||||
return data, content, data.get("totalFound"), offset + len(content) if content else None
|
||||
|
||||
|
||||
@@ -44,6 +44,10 @@ places with their country from it; a role at several places repeats the
|
||||
`jobLocation` block, once per place. Novo Nordisk's pages have none: the
|
||||
description is still in `itemprop="description"`, and the listing's date and
|
||||
location stand.
|
||||
|
||||
A fetch term goes as the search form's `q=engineer`, one a request
|
||||
(`Filters(query="one")`); with `optionsFacetsDD_country=DK` it is an AND
|
||||
(survey, 2026-09-30: 40 of Vestas' 64 Danish roles).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -53,7 +57,7 @@ import re
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
from urllib.parse import quote, urljoin, urlsplit
|
||||
|
||||
from desk.sources import strip_html
|
||||
|
||||
@@ -85,7 +89,7 @@ KEEP_CASE = True
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", text="detail", complete="stated", places="many",
|
||||
filters=Filters(country="one"),
|
||||
filters=Filters(country="one", query="one"),
|
||||
fields=Fields(posted="trusted"),
|
||||
robots="read: robots.txt allows /search/ and /job/ (checked 2026-09-29)",
|
||||
)
|
||||
@@ -155,12 +159,13 @@ def careers_url(slug: str) -> str:
|
||||
return f"https://{host}{prefix}/search/"
|
||||
|
||||
|
||||
def _search_url(slug: str, start: int, country: str = "") -> str:
|
||||
def _search_url(slug: str, start: int, country: str = "", query: str = "") -> str:
|
||||
"""The search page from `start`, in one country when `country` (an ISO
|
||||
code) is given."""
|
||||
code) is given, matching `query` (the form's `q`) when given."""
|
||||
host, prefix = _parts(slug)
|
||||
facet = f"&optionsFacetsDD_country={country}" if re.fullmatch(r"[A-Z]{2}", country) else ""
|
||||
return f"https://{host}{prefix}/search/?q=&locationsearch={facet}&startrow={start}"
|
||||
return (f"https://{host}{prefix}/search/?q={quote(query, safe='')}&locationsearch="
|
||||
f"{facet}&startrow={start}")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ listing
|
||||
@@ -264,13 +269,13 @@ def _entries(text: str) -> list[dict[str, Any]]:
|
||||
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
"""One search page, in the one country `ask` names. The cursor is `(start row, first role of the page
|
||||
"""One search page, in the one country `ask` names, matching its `query`. The cursor is `(start row, first role of the page
|
||||
before)`: the next start row is what this page returned, and a page that
|
||||
starts with the same role as the last one is the site ignoring `startrow`,
|
||||
which ends the read (incomplete) instead of looping."""
|
||||
start, previous = cursor or (0, None)
|
||||
country = ask.countries[0] if ask.countries else ""
|
||||
text = get(_search_url(slug, start, country)).decode("utf-8", "replace")
|
||||
text = get(_search_url(slug, start, country, ask.query)).decode("utf-8", "replace")
|
||||
if not _SEARCH_PAGE.search(text):
|
||||
raise ValueError("not a SuccessFactors search page")
|
||||
found = _entries(text)
|
||||
|
||||
@@ -23,6 +23,10 @@ series of its own. Each tenant names its facets its own way (LEGO's
|
||||
one whose values are countries. A host id is `<facetParameter>=<id>`. LEGO
|
||||
answered 97 of 455 for Denmark on 2026-09-30; a role at several places is
|
||||
kept when one of them is in the country.
|
||||
|
||||
A fetch term goes as `searchText` in the body, one a request
|
||||
(`Filters(query="one")`); it is an AND with `appliedFacets` (survey,
|
||||
2026-09-30: LEGO's `engineer` answered 106, 45 of them in Denmark).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -46,7 +50,7 @@ KEEP_CASE = True
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", page_size=20, ceiling=2000, text="detail",
|
||||
complete="stated", places="many", filters=Filters(country="ids", remote="ids"),
|
||||
complete="stated", places="many", filters=Filters(country="ids", remote="ids", query="one"),
|
||||
fields=Fields(posted="trusted", remote="stated", employment="stated"),
|
||||
robots="read: robots.txt allows /<Site>/ (checked 2026-09-29)",
|
||||
)
|
||||
@@ -114,7 +118,7 @@ def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
data = json.loads(get(
|
||||
f"https://{_host(slug)}/wday/cxs/{tenant}/{site}/jobs", method="POST",
|
||||
payload={"appliedFacets": _applied(ask), "limit": PAGE_SIZE, "offset": offset,
|
||||
"searchText": ""}))
|
||||
"searchText": ask.query}))
|
||||
if cursor is None:
|
||||
stated = data.get("total")
|
||||
total = stated if isinstance(stated, int) and not isinstance(stated, bool) else None
|
||||
|
||||
Reference in new issue
Block a user