Fold the readers plan into the decisions and delete it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 10:06:50 +02:00
1 parent ca80219bbb
commit b567144612
3 files changed
+18 -183

No files matched your search

+17 -3
View File
@@ -68,7 +68,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Jobindex and Adzuna, not Jobnet or Jooble.** Jobindex, Denmark's largest board, has an RSS twin for every search page, with phrase and required-word syntax and a region id; no key. Adzuna has a documented API with a free key for the UK and most of the EU, but no Denmark. Jobnet's search endpoint answers 401 outside its own site's session, and Jooble gives a key only on a manual application and answered a Cloudflare challenge. LinkedIn and Indeed have no public search API.
**Platsbanken is the third search provider, for Sweden.** The JobTech API (`jobsearch.api.jobtechdev.se/search`) needs no key, states a total, so completeness follows Adzuna's rule, and filters by the taxonomy id of a län (`PLATSBANKEN_REGIONS`; Skåne is `CaRE_1nn_cSU`). It refuses an offset above 2000, which the page limit never reaches. The ads carry no usable remote field, so `remote` is read off the headline. Free text ranks by all words rather than requiring them; a quoted phrase narrows it.
**Platsbanken is the third search provider, for Sweden.** The JobTech API (`jobsearch.api.jobtechdev.se/search`) needs no key, states a total, so completeness follows Adzuna's rule, and filters by the taxonomy id of a län (`platsbanken.REGIONS`; Skåne is `CaRE_1nn_cSU`). It refuses an offset above 2000, which the page limit never reaches. The ads carry no usable remote field, so `remote` is read off the headline. Free text ranks by all words rather than requiring them; a quoted phrase narrows it.
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. An Adzuna search listed everything when it read as many distinct results as the `count` it states; a Jobindex search, whose feed states no count, when a page within its page limit came back with fewer than 20 items, counted before the parser drops any, so one malformed item does not pass for the end. Such a search closes like a board; the rest close once no search has listed them for 21 days. A Jobindex answer with no RSS channel raises instead of reading as an empty, complete search. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists.
@@ -104,9 +104,23 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
**A description stated only on the role's own page is read once, 200 roles per board per pull.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. A module's `detail` is called for roles not yet read, those never tried before those that failed, and the ids read are kept in `ats:detailed:<target>`; a known role is sent without a description on later pulls, which `upsert_role` ignores. `detail` may fill `DETAIL_FIELDS` only (description, location, remote, currency, salary_min, salary_max, posted), the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing, since the listing rewrites them every pull. Completeness is decided by the listing alone, so a board of 800 roles closes its roles from the first pull and has all its text after four.
**Per-role text is read by the enrich run, ranked roles first.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. The pull reads listings only. After scoring, the `enrich` stage reads the role pages of roles without text that an enabled agent ranks in its top `enrich.top` (500) at `enrich.min_score` (0.5) or more, best score first, 200 per board and 1,000 per run, and only the roles it filled are embedded and scored again. It follows every pull, a saved agent and a star change, not a vote. A read that brought no text is tried again after `retry_days` (7). `detail` may fill `DETAIL_FIELDS` only, the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing. Completeness is decided by the listing alone. The pull used to read 200 pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered.
**robots.txt is honoured, by RFC 9309.** A module names the page a person opens, and a board whose site disallows that page for `*` or for the app's own token fails with "robots.txt disallows it" and closes its roles after 21 days like any failing board. The matcher takes the longest rule, an Allow on a tie, `*` and `$`, and case-sensitive paths, or paths without case for a host that serves any case, as Workday does. A 4xx means no rules; a 5xx, a 429 or a timeout fails the board for that pull. `urllib.robotparser` was replaced because it takes the first rule and knows no wildcard, which refused Infineon and let `/*company` through. This keeps Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy and Umbraco off the automatic route. The SuccessFactors reader uses `/search/` and `/job/` only, never `/services/` or a `career*.successfactors.eu` back end.
**robots.txt is decided when a reader is built, not checked at run time.** The builder reads the host's robots.txt, the owner decides per reader whether to read it anyway, and `READS.robots` records the decision and the date. The runtime matcher went with the readers rework. Jobindex is read in full although its robots.txt disallows `page=` and `geoareaid=`; Adzuna's API host disallows `/` for every agent, and whether a keyed API is read regardless is open with the owner; Adzuna has no key configured and has never run. HR Manager's portal disallows everything but the application form; no attached company uses it. The companies once kept off the automatic route by robots.txt (Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy, Umbraco) stay on the Watchlist with reason `robots` until each is decided. The SuccessFactors reader uses `/search/` and `/job/` only.
**Every source is a reader module that declares what it can do.** The 22 hosts and the 3 search providers are modules in `plugins/readers/`, each with `READS = Reads(...)`: board or search, whole or paged listing, where the text is (listing, excerpt, detail, none), how completeness is known (stated, end, whole, never), page size and the host's ceiling, the filters it takes per dimension (none, one value a request, many, host ids through `facets`), whether it states one place, several, or one posting per place, which fields it states and whether its date is trusted, a key, a provisional title, whether an unknown slug answers an empty board, and the robots decision. Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `problems()` checks the declaration against the functions at import, and `tests/test_readers_contract.py` runs every module against fixtures, so a new reader that drops a place or a teaser fails there. The contract is the docstring of `plugins/readers/__init__.py`; the types are `_contract.py`.
**One runner calls every reader.** `plugins/readers/_runner.py` owns the guarded fetch, paging, one request series per filter value unioned by id, completeness, the fallback feed, 429s and failures. `plugins/ats.py` and `plugins/search.py` are thin `Source` classes over it and keep their state prefixes, so no stored key moved. A paged read cut by `MAX_PAGES` saves its cursor and continues on the next pull; the sweep that reaches the end closes what it did not list. A host ceiling (Workday 2,000) is not swept; a filter has to get under it.
**The agents set the reach of a pull.** `app/jobs/reach.py` computes it once per pull: the union of the enabled agents' `any` gates on country, town (as its country) and remote, or everything when one agent has none. A favourite company is read with its agent's `require` reach, Europe today, so a company worth asking is read beyond Denmark and Sweden. A board's `params` override both. Boards under 300 roles, and hosts with a one-country filter over more than 8 countries, are read whole. SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday and Oracle filter at the source; countries and remote go as separate series, because the hosts intersect two facets. A complete filtered read closes roles outside the filter, except roles with a vote or an application; a filtered read, or one from a host whose unknown slug answers empty, that lists under 10% of the last read with the same filter counts as failed.
**A role has a list of places, resolved against a gazetteer.** Readers kept one place of several, joined them, or the host posted a copy per place. A reader now returns every place and never multiplies a row. `places` (city, region, country, continent, with `place_includes` for Øresund across two countries) and `place_aliases` are seeded from the dictionaries that were in code; `role_places` links each place of a role, and an unknown text is listed by `unresolved_places` until an alias links it. The Place filter and the gates join over the tree, which halved their time and dropped the `EU` fallback. Copies of one posting per place, same host, company and title and texts at least 30% alike, fold into one role that keeps each copy's place and link; the rule applies to every host, since the 5,024 such roles came mostly from hosts that do not declare copies.
**A company has aliases.** `company_aliases` maps a folded name (marks stripped, ø to o, legal form dropped) to a company, so "Orsted" and "Ørsted" are one company without changing `company_key`. `merge_companies` moves roles, subscriptions and ratings. Empty enrichment columns (sector, stage, size band, founded, last funding, source and date) wait for the startupmap.one and The Hub readers.
**A list teaser is kept.** A short description in a listing (Phenom, Oracle, Jobylon, Jobindex, Adzuna) is `roles.teaser`, shown and embedded until the full text arrives, when the vector is dropped and the text replaces it.
**Seniority has three sources.** `seniority_stated` is the host's word mapped by a table in its reader, `seniority_title` is `seniority.from_title`, and `seniority_model` is `extract`'s verdict; `roles.seniority` is the first of model, stated, title. `extract` reads roles without a model value, every agent's first pick before anyone's second, with the other two as hints it may overrule, since host vocabularies differ and 57% of roles state nothing. `app seniority` shows per host how often the stated level agrees with the model.
**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key the page's JavaScript carries, which can change without notice and marks an endpoint that is not open. They are on the Watchlist with reason `key`.