Fetch terms narrow what big boards read with the wide reach send (WP F2)
An agent has fetch_terms (migration 23, API, a chip field in the agent
editor). The terms of a pull are the union over the enabled agents, none
when one of them has none. They go only to wide reads: a company tagged All
roles, or a reach ask_for drops as too many countries for a one host; a
board's {"terms": false} switches them off. SmartRecruiters q,
SuccessFactors q, Workday searchText, Phenom keywords, Eightfold query and
Oracle keyword declare query="one"; the runner makes one series per term
times the country series, drops the places first over MAX_SERIES, then the
terms. A term read is narrowed: it writes no ats:size, closes nothing and
prunes no per-role state, and the 10% guard holds it to its own
fingerprint. The Companies reach line counts the terms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
3917b31209
commit
0fd45b333a
28 files changed
+839
-113
No files matched your search
@@ -44,6 +44,10 @@ places with their country from it; a role at several places repeats the
|
||||
`jobLocation` block, once per place. Novo Nordisk's pages have none: the
|
||||
description is still in `itemprop="description"`, and the listing's date and
|
||||
location stand.
|
||||
|
||||
A fetch term goes as the search form's `q=engineer`, one a request
|
||||
(`Filters(query="one")`); with `optionsFacetsDD_country=DK` it is an AND
|
||||
(survey, 2026-09-30: 40 of Vestas' 64 Danish roles).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -53,7 +57,7 @@ import re
|
||||
from collections.abc import Callable
|
||||
from datetime import UTC, datetime
|
||||
from typing import Any
|
||||
from urllib.parse import urljoin, urlsplit
|
||||
from urllib.parse import quote, urljoin, urlsplit
|
||||
|
||||
from desk.sources import strip_html
|
||||
|
||||
@@ -85,7 +89,7 @@ KEEP_CASE = True
|
||||
|
||||
READS = Reads(
|
||||
scope="board", listing="paged", text="detail", complete="stated", places="many",
|
||||
filters=Filters(country="one"),
|
||||
filters=Filters(country="one", query="one"),
|
||||
fields=Fields(posted="trusted"),
|
||||
robots="read: robots.txt allows /search/ and /job/ (checked 2026-09-29)",
|
||||
)
|
||||
@@ -155,12 +159,13 @@ def careers_url(slug: str) -> str:
|
||||
return f"https://{host}{prefix}/search/"
|
||||
|
||||
|
||||
def _search_url(slug: str, start: int, country: str = "") -> str:
|
||||
def _search_url(slug: str, start: int, country: str = "", query: str = "") -> str:
|
||||
"""The search page from `start`, in one country when `country` (an ISO
|
||||
code) is given."""
|
||||
code) is given, matching `query` (the form's `q`) when given."""
|
||||
host, prefix = _parts(slug)
|
||||
facet = f"&optionsFacetsDD_country={country}" if re.fullmatch(r"[A-Z]{2}", country) else ""
|
||||
return f"https://{host}{prefix}/search/?q=&locationsearch={facet}&startrow={start}"
|
||||
return (f"https://{host}{prefix}/search/?q={quote(query, safe='')}&locationsearch="
|
||||
f"{facet}&startrow={start}")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ listing
|
||||
@@ -264,13 +269,13 @@ def _entries(text: str) -> list[dict[str, Any]]:
|
||||
|
||||
|
||||
def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
|
||||
"""One search page, in the one country `ask` names. The cursor is `(start row, first role of the page
|
||||
"""One search page, in the one country `ask` names, matching its `query`. The cursor is `(start row, first role of the page
|
||||
before)`: the next start row is what this page returned, and a page that
|
||||
starts with the same role as the last one is the site ignoring `startrow`,
|
||||
which ends the read (incomplete) instead of looping."""
|
||||
start, previous = cursor or (0, None)
|
||||
country = ask.countries[0] if ask.countries else ""
|
||||
text = get(_search_url(slug, start, country)).decode("utf-8", "replace")
|
||||
text = get(_search_url(slug, start, country, ask.query)).decode("utf-8", "replace")
|
||||
if not _SEARCH_PAGE.search(text):
|
||||
raise ValueError("not a SuccessFactors search page")
|
||||
found = _entries(text)
|
||||
|
||||
Reference in new issue
Block a user