Fetch terms narrow what big boards read with the wide reach send (WP F2)

An agent has fetch_terms (migration 23, API, a chip field in the agent
editor). The terms of a pull are the union over the enabled agents, none
when one of them has none. They go only to wide reads: a company tagged All
roles, or a reach ask_for drops as too many countries for a one host; a
board's {"terms": false} switches them off. SmartRecruiters q,
SuccessFactors q, Workday searchText, Phenom keywords, Eightfold query and
Oracle keyword declare query="one"; the runner makes one series per term
times the country series, drops the places first over MAX_SERIES, then the
terms. A term read is narrowed: it writes no ats:size, closes nothing and
prunes no per-role state, and the 10% guard holds it to its own
fingerprint. The Companies reach line counts the terms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 11:59:27 +02:00
1 parent 3917b31209
commit 0fd45b333a
28 files changed
+839 -113

No files matched your search

+12 -7
View File
@@ -44,6 +44,10 @@ places with their country from it; a role at several places repeats the
`jobLocation` block, once per place. Novo Nordisk's pages have none: the
description is still in `itemprop="description"`, and the listing's date and
location stand.
A fetch term goes as the search form's `q=engineer`, one a request
(`Filters(query="one")`); with `optionsFacetsDD_country=DK` it is an AND
(survey, 2026-09-30: 40 of Vestas' 64 Danish roles).
"""
from __future__ import annotations
@@ -53,7 +57,7 @@ import re
from collections.abc import Callable
from datetime import UTC, datetime
from typing import Any
from urllib.parse import urljoin, urlsplit
from urllib.parse import quote, urljoin, urlsplit
from desk.sources import strip_html
@@ -85,7 +89,7 @@ KEEP_CASE = True
READS = Reads(
scope="board", listing="paged", text="detail", complete="stated", places="many",
filters=Filters(country="one"),
filters=Filters(country="one", query="one"),
fields=Fields(posted="trusted"),
robots="read: robots.txt allows /search/ and /job/ (checked 2026-09-29)",
)
@@ -155,12 +159,13 @@ def careers_url(slug: str) -> str:
return f"https://{host}{prefix}/search/"
def _search_url(slug: str, start: int, country: str = "") -> str:
def _search_url(slug: str, start: int, country: str = "", query: str = "") -> str:
"""The search page from `start`, in one country when `country` (an ISO
code) is given."""
code) is given, matching `query` (the form's `q`) when given."""
host, prefix = _parts(slug)
facet = f"&optionsFacetsDD_country={country}" if re.fullmatch(r"[A-Z]{2}", country) else ""
return f"https://{host}{prefix}/search/?q=&locationsearch={facet}&startrow={start}"
return (f"https://{host}{prefix}/search/?q={quote(query, safe='')}&locationsearch="
f"{facet}&startrow={start}")
# ------------------------------------------------------------------ listing
@@ -264,13 +269,13 @@ def _entries(text: str) -> list[dict[str, Any]]:
def page(get: Get, slug: str, cursor: Any, ask: Ask = Ask()) -> Page:
"""One search page, in the one country `ask` names. The cursor is `(start row, first role of the page
"""One search page, in the one country `ask` names, matching its `query`. The cursor is `(start row, first role of the page
before)`: the next start row is what this page returned, and a page that
starts with the same role as the last one is the site ignoring `startrow`,
which ends the read (incomplete) instead of looping."""
start, previous = cursor or (0, None)
country = ask.countries[0] if ask.countries else ""
text = get(_search_url(slug, start, country)).decode("utf-8", "replace")
text = get(_search_url(slug, start, country, ask.query)).decode("utf-8", "replace")
if not _SEARCH_PAGE.search(text):
raise ValueError("not a SuccessFactors search page")
found = _entries(text)