From 795ca3628c19fafc1d4a4f43c7c3307511688967 Mon Sep 17 00:00:00 2001 From: Wolf Date: Wed, 30 Sep 2026 12:01:35 +0200 Subject: [PATCH] Fold the prep plan into the decisions, keep the filter survey as reference Co-Authored-By: Claude Opus 5.5 --- docs/decisions.md | 6 +- .../{plan-prep-survey.md => filter-survey.md} | 2 +- docs/plan-prep.md | 75 ------------------- plugins/README.md | 2 +- tests/test_fetch_terms.py | 4 +- 5 files changed, 9 insertions(+), 80 deletions(-) rename docs/{plan-prep-survey.md => filter-survey.md} (98%) delete mode 100644 docs/plan-prep.md diff --git a/docs/decisions.md b/docs/decisions.md index 22ca83a..0408018 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -120,7 +120,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning **A list teaser is kept.** A short description in a listing (Phenom, Oracle, Jobylon, Jobindex, Adzuna) is `roles.teaser`, shown and embedded until the full text arrives, when the vector is dropped and the text replaces it. -**Seniority has three sources.** `seniority_stated` is the host's word mapped by a table in its reader, `seniority_title` is `seniority.from_title`, and `seniority_model` is `extract`'s verdict; `roles.seniority` is the first of model, stated, title. `extract` reads roles without a model value, every agent's first pick before anyone's second, with the other two as hints it may overrule, since host vocabularies differ and 57% of roles state nothing. `app seniority` shows per host how often the stated level agrees with the model. +**Readers scrape; `prep` decides.** A reader hands on the host's own words (`seniority`, `employment`, `remote`, and `country` only when the host states one per role) and does light cleanup only: text from HTML, dates, the places of a role as a list, pay figures. Mapping lived in 25 readers, each spelling its tables its own way, so changing what "Experienced" means touched a plugin. The `prep` stage (`app/prep/`) maps centrally after every pull and every enrich run. Seniority, employment and remote go through the host's table in `app/prep/hosts/.py` (whole words, or `MATCH` fragments by substring or token), then a shared word map. Country comes from the resolved places (several European countries give `EU`), and the host's country is only a fallback. Places and company aliases are resolved, and copies and duplicates are folded over the whole database. `upsert_role` stores the host words (`*_host` columns, migration 22) and marks the role `prep_due`. When a table changes, its fingerprint makes the next prep map every open role again from the stored words, with no pull. A word no table knows is listed in `unmapped` (`app prep unmapped`, the Sources page). A host's `remote: False` stays unknown rather than onsite, because SmartRecruiters, Ashby, Recruitee and Workable send it for hybrid roles too. + +**Fetch terms narrow what big boards send across Europe.** An agent's `fetch_terms` (migration 23) are words a big board must match when it is read with the wide reach: a company tagged All roles, or a one-country host whose reach was dropped for needing over 8 requests. The terms of a pull are the union over the enabled agents; one enabled agent without terms switches them off. The narrow reach takes no terms, because it already leaves at most about 150 roles on any big board. The survey of 2026-09-30 found that every host's keyword matches the whole text, so a term cuts less than a title filter would ("engineer" left 1,713 of Bosch's 4,830). SmartRecruiters, SuccessFactors, Workday, Phenom, Eightfold and Oracle take one term a request, with one series per term unioned by id; over 8 series the place filter goes first, then the terms. A term-filtered read is never the whole board: it writes no size, prunes no per-role state and closes nothing. Category and level filters wait until something maps agent terms to host ids. A town in a gate is still fetched as its country. + +**Seniority has three sources.** `seniority_stated` is the host's word mapped by that host's table in `app/prep/hosts/`, `seniority_title` is `seniority.from_title`, and `seniority_model` is `extract`'s verdict; `roles.seniority` is the first of model, stated, title. `extract` reads roles without a model value, every agent's first pick before anyone's second, with the other two as hints it may overrule, since host vocabularies differ and 57% of roles state nothing. `app seniority` shows per host how often the stated level agrees with the model. **No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key the page's JavaScript carries, which can change without notice and marks an endpoint that is not open. They are on the Watchlist with reason `key`. diff --git a/docs/plan-prep-survey.md b/docs/filter-survey.md similarity index 98% rename from docs/plan-prep-survey.md rename to docs/filter-survey.md index c532b5e..4584096 100644 --- a/docs/plan-prep-survey.md +++ b/docs/filter-survey.md @@ -1,6 +1,6 @@ # Survey: which filters the hosts take at the source -Work package S of `plan-prep.md`, 2026-09-30. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. +Surveyed 2026-09-30 for the fetch terms (see `decisions.md`, Ingestion); kept as the reference for each host's filter parameters. Each host got at most two live requests, one small page each, against boards that are already attached (`roledesk.db` opened read-only for the targets). Nothing else was read except one page of HTML or JS where noted. Marks used in the table: diff --git a/docs/plan-prep.md b/docs/plan-prep.md deleted file mode 100644 index 599c3d5..0000000 --- a/docs/plan-prep.md +++ /dev/null @@ -1,75 +0,0 @@ -# Plan: readers only scrape, a prep stage decides, and the agents narrow the fetch - -Planned 2026-09-30 after the readers rework landed. It is folded into `decisions.md` and deleted when it lands. - -## What the owner asked for - -1. **A reader is a scraper.** It gets the data in and makes light touches only: HTML to text, a date parsed, the places of a role as a list. It hands on the host's own words, and it maps nothing to the app's vocabulary. -2. **A prep stage makes every decision that needs a rule or data across roles.** That covers seniority, employment type, remote, country from places, places against the gazetteer, company aliases, and folding copies and duplicates across the whole database, not only the run. It runs centrally, once, between the pull and embedding. -3. **Two levels of reach, set by the agents.** - - The wide reach is an agent's `require` gates, Europe today, and it is what "all roles" means. - - The narrow reach is the `any` gates: Denmark, Sweden, or towns. - - Big boards (300 roles or more) are read with the narrow reach where the reader can filter. A company tagged **All roles** is read with the wide reach. A reader that cannot filter reads the whole board. - - Being a favourite affects scoring only. -4. **More filters at the source.** If a host can filter by keyword, title, category or level, the agents' own terms go to it, so a board with 1,000 roles or more (Bosch, Delivery Hero) is narrowed before it is read. - -## Where it stands - -| Part | Today | Gap | -|---|---|---| -| Places, companies | mapped centrally in `upsert_role` (gazetteer, `company_aliases`) | none, but it runs inline in the upsert and not as a stage | -| Seniority, employment | a word table in each reader, `_shared.table_word` | per reader; an unknown word is only a debug log line | -| Remote | each reader turns the host value into True, False or a word | per reader, spelt differently | -| Country | a reader sets it when a role has one country | should come from the places | -| Copies, duplicates | `rebuild_dupes` over the whole database after a pull | already central; to become part of prep | -| Reach | narrow = union of `any` gates; favourites = `require` reach | the All roles tag is being built (WP K) with the wide reach; favourite stops widening the fetch | -| Filters at the source | country and remote on six hosts | no keyword, category or level filters; towns are fetched as their country | - -## Design - -**Raw row keys.** A reader sets: -- `seniority` and `employment` to the host's own word, unchanged (e.g. `"Experienced"`, `"Regular Full Time"`) -- `remote` to the host's own value: a bool, or the host's word -- no `country` unless the host states one per role. When it does, the key carries the host's own spelling, or its ISO code as given. - -The `READS.fields` declaration stays, and says which of these a host states. `_shared.table_word` and every per-host table leave the readers. - -**Storage.** `roles` keeps the host words in `seniority_host`, `employment_host` and `remote_host`, and `raw` keeps them too. The mapped columns (`seniority_stated`, `employment`, `remote`, `country`) are written by prep only. Prep can map again from the stored host words when a table changes, with no new pull. - -**The prep stage** is `app/prep/`: one module per concern, and one table file per field with a section per host. -- **Seniority:** the host word through that host's table, then the title rules (`seniority.from_title`). The model value from `extract` still wins over both. -- **Employment:** one table per host into `EMPLOYMENTS`. -- **Remote:** the host value through one shared word map, per host where a host needs its own. -- **Country:** from the role's resolved places through the gazetteer. A host-stated country is only a fallback. -- **Places and companies:** the gazetteer and alias resolution move from inline `upsert_role` calls into prep. `upsert_role` stores the raw values; prep links them. -- **Copies and duplicates:** prep runs `rebuild_dupes`, which already works over the whole database. -- **Unknown words:** kept in `unmapped(field, host, word, roles, first_seen)` and listed by `app prep unmapped` and on the Sources page, beside the unresolved places. Adding a word to a table and running `app prep --remap` fixes the stored roles. - -The cascade becomes `pull, prep, embed, score, enrich, prep (filled roles), embed, score, extract, ...`. Prep is cheap: a pass over the changed roles, plus the dupes rebuild, which takes 0.8 s on the live data. - -**Filters beyond location.** An agent gains `fetch_terms`: words a role must match for a big board to list it, such as `engineer`, `product`, `data`. The pull's query term is the union over the enabled agents. If any enabled agent has no terms, nothing is filtered by term, because that agent wants every role. A reader declares `filters.query`, and where a host has them, a category or level filter. The runner sends the terms as one series per term (unioned by id) or as one query, as the host allows. - -Terms apply only to boards of 300 roles or more that are not tagged All roles. The closing guards of the location filter hold: roles with a vote or an application stay open, and a read under 10% of the last one fails. Terms also go to the search providers that already take a query. - -**Towns.** Today a town gate is fetched as its country. A host that filters by place (`filters.place`) is asked for the towns. Every other host keeps the country and leaves the towns to the gates. - -## Work packages - -| WP | Model | Builds | Owns | Needs | -|---|---|---|---|---| -| S Survey | Sonnet | Per host that can filter (SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday, Oracle, Workable, Teamtailor, Jibe, Lever) and the search providers, which keyword, title, category, level and place filters exist and how they combine. At most 2 live requests per host. Also: how a town gate reaches the fetch today, and what it should become. Output `docs/plan-prep-survey.md`. | that file | | -| R1 Raw rows a to l | Sonnet | Readers Ashby to Lever hand on host words; tables move out into a file prep owns (`app/prep/tables/*.py`, one section per host, copied verbatim) | those modules, their tests, `app/prep/tables/` sections for them | K merged | -| R2 Raw rows o to z | Sonnet | The same for Oracle to Workday and the three providers | those modules and tests, their table sections | K merged | -| P Prep stage | Opus | `app/prep/` stage, host-word columns and the `unmapped` table (migration 22), `upsert_role` storing raw values, gazetteer and alias resolution moved into prep, the cascade, `app prep` CLI, the Sources page list of unmapped words | `app/prep/*` except the table sections, `app/storage/*`, `app/jobs/cascade.py`, `app/jobs/stages.py`, `app/cli.py`, `app/static/sources.js`, their tests | K merged; R1 and R2 merge with it | -| F2 Terms and towns | Opus | `fetch_terms` on agents (model, API, agent editor), terms and towns in the reach, runner series for `query` and `place`, reader filters per the survey | `app/jobs/reach.py`, `plugins/readers/_runner.py`, `_contract.py`, the filtering readers, `app/static/agents.js` | S, and R1/R2/P merged | -| F2 done 2026-09-30 | Opus | `agents.fetch_terms` (migration 23, API, a chip field in the agent editor); terms in the reach for wide reads only; one series per term on SmartRecruiters `q`, SuccessFactors `q`, Workday `searchText`, Phenom `keywords`, Eightfold `query`, Oracle `keyword`; places dropped first, then terms, over `MAX_SERIES`; a term read (`ats:narrowed`) sets no size and closes nothing; the reach line counts the terms. Towns stay fetched as their country | branch `wp-f2-terms` | | - -The survey (`docs/plan-prep-survey.md`) narrowed F2: the narrow reach already leaves at most 149 roles on any big board, and keywords match the whole text, so terms go only to reads with the wide reach (All roles, or a one-country host over `MAX_SERIES`). `query` is built in this order: SmartRecruiters, SuccessFactors, then Workday, Phenom, Eightfold and Oracle. Category and level filters wait until something maps agent terms to host ids. Towns stay fetched as their country. A term-filtered read never sets `ats:size` and never closes roles as gone. - -Order: S, and K finishing. Then R1, R2 and P in parallel, merged together, because the raw keys and prep have to land at once. Then F2. - -## Decisions taken by the owner, 2026-09-30 - -1. Readers scrape and hand on host words. All mapping is central, in prep. -2. The wide reach is each agent's `require` reach (Europe). The All roles tag reads a company with it. The narrow reach applies to big boards otherwise. A favourite no longer widens the fetch. -3. Keyword, category and level filters at the source are used where hosts offer them, driven by the agents' fetch terms. diff --git a/plugins/README.md b/plugins/README.md index a4ca952..ec8eaed 100644 --- a/plugins/README.md +++ b/plugins/README.md @@ -76,7 +76,7 @@ Fetch terms narrow a wide read by words. Each agent has a list of them (`fetch_t What no source does: - Filter on the host's side for a host other than SmartRecruiters, SuccessFactors, Workday, Oracle, Phenom and Eightfold. Greenhouse, Ashby, Recruitee and the rest are read whole. -- Filter by category or level at the source. Workday, Oracle, Jibe and Eightfold state category ids, but nothing maps an agent's terms to them yet (`docs/plan-prep-survey.md`). +- Filter by category or level at the source. Workday, Oracle, Jibe and Eightfold state category ids, but nothing maps an agent's terms to them yet (`docs/filter-survey.md`). - Read more than one board per company. DFDS is read from its Danish Teamtailor board only. - Use a key taken from a site's script. Maersk, New Work and Welcome to the Jungle are on the Watchlist for that reason. - Read a page that needs a browser to run its script. diff --git a/tests/test_fetch_terms.py b/tests/test_fetch_terms.py index 44cbfc6..7267607 100644 --- a/tests/test_fetch_terms.py +++ b/tests/test_fetch_terms.py @@ -1,7 +1,7 @@ -"""Fetch terms (work package F2 of `docs/plan-prep.md`): an agent's words that +"""Fetch terms (see "Fetch terms" in `docs/decisions.md`): an agent's words that narrow what a big board read with the wide reach sends. The terms of a pull, where they apply, how the runner makes series of them, the request each host -sends (the parameters `docs/plan-prep-survey.md` verified), the guards on a +sends (the parameters `docs/filter-survey.md` verified), the guards on a term read, and the agent editor's API. No network.""" from __future__ import annotations