An agent has fetch_terms (migration 23, API, a chip field in the agent
editor). The terms of a pull are the union over the enabled agents, none
when one of them has none. They go only to wide reads: a company tagged All
roles, or a reach ask_for drops as too many countries for a one host; a
board's {"terms": false} switches them off. SmartRecruiters q,
SuccessFactors q, Workday searchText, Phenom keywords, Eightfold query and
Oracle keyword declare query="one"; the runner makes one series per term
times the country series, drops the places first over MAX_SERIES, then the
terms. A term read is narrowed: it writes no ats:size, closes nothing and
prunes no per-role state, and the 10% guard holds it to its own
fingerprint. The Companies reach line counts the terms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
76 lines
8.6 KiB
Markdown
76 lines
8.6 KiB
Markdown
# Plan: readers only scrape, a prep stage decides, and the agents narrow the fetch
|
|
|
|
Planned 2026-09-30 after the readers rework landed. It is folded into `decisions.md` and deleted when it lands.
|
|
|
|
## What the owner asked for
|
|
|
|
1. **A reader is a scraper.** It gets the data in and makes light touches only: HTML to text, a date parsed, the places of a role as a list. It hands on the host's own words, and it maps nothing to the app's vocabulary.
|
|
2. **A prep stage makes every decision that needs a rule or data across roles.** That covers seniority, employment type, remote, country from places, places against the gazetteer, company aliases, and folding copies and duplicates across the whole database, not only the run. It runs centrally, once, between the pull and embedding.
|
|
3. **Two levels of reach, set by the agents.**
|
|
- The wide reach is an agent's `require` gates, Europe today, and it is what "all roles" means.
|
|
- The narrow reach is the `any` gates: Denmark, Sweden, or towns.
|
|
- Big boards (300 roles or more) are read with the narrow reach where the reader can filter. A company tagged **All roles** is read with the wide reach. A reader that cannot filter reads the whole board.
|
|
- Being a favourite affects scoring only.
|
|
4. **More filters at the source.** If a host can filter by keyword, title, category or level, the agents' own terms go to it, so a board with 1,000 roles or more (Bosch, Delivery Hero) is narrowed before it is read.
|
|
|
|
## Where it stands
|
|
|
|
| Part | Today | Gap |
|
|
|---|---|---|
|
|
| Places, companies | mapped centrally in `upsert_role` (gazetteer, `company_aliases`) | none, but it runs inline in the upsert and not as a stage |
|
|
| Seniority, employment | a word table in each reader, `_shared.table_word` | per reader; an unknown word is only a debug log line |
|
|
| Remote | each reader turns the host value into True, False or a word | per reader, spelt differently |
|
|
| Country | a reader sets it when a role has one country | should come from the places |
|
|
| Copies, duplicates | `rebuild_dupes` over the whole database after a pull | already central; to become part of prep |
|
|
| Reach | narrow = union of `any` gates; favourites = `require` reach | the All roles tag is being built (WP K) with the wide reach; favourite stops widening the fetch |
|
|
| Filters at the source | country and remote on six hosts | no keyword, category or level filters; towns are fetched as their country |
|
|
|
|
## Design
|
|
|
|
**Raw row keys.** A reader sets:
|
|
- `seniority` and `employment` to the host's own word, unchanged (e.g. `"Experienced"`, `"Regular Full Time"`)
|
|
- `remote` to the host's own value: a bool, or the host's word
|
|
- no `country` unless the host states one per role. When it does, the key carries the host's own spelling, or its ISO code as given.
|
|
|
|
The `READS.fields` declaration stays, and says which of these a host states. `_shared.table_word` and every per-host table leave the readers.
|
|
|
|
**Storage.** `roles` keeps the host words in `seniority_host`, `employment_host` and `remote_host`, and `raw` keeps them too. The mapped columns (`seniority_stated`, `employment`, `remote`, `country`) are written by prep only. Prep can map again from the stored host words when a table changes, with no new pull.
|
|
|
|
**The prep stage** is `app/prep/`: one module per concern, and one table file per field with a section per host.
|
|
- **Seniority:** the host word through that host's table, then the title rules (`seniority.from_title`). The model value from `extract` still wins over both.
|
|
- **Employment:** one table per host into `EMPLOYMENTS`.
|
|
- **Remote:** the host value through one shared word map, per host where a host needs its own.
|
|
- **Country:** from the role's resolved places through the gazetteer. A host-stated country is only a fallback.
|
|
- **Places and companies:** the gazetteer and alias resolution move from inline `upsert_role` calls into prep. `upsert_role` stores the raw values; prep links them.
|
|
- **Copies and duplicates:** prep runs `rebuild_dupes`, which already works over the whole database.
|
|
- **Unknown words:** kept in `unmapped(field, host, word, roles, first_seen)` and listed by `app prep unmapped` and on the Sources page, beside the unresolved places. Adding a word to a table and running `app prep --remap` fixes the stored roles.
|
|
|
|
The cascade becomes `pull, prep, embed, score, enrich, prep (filled roles), embed, score, extract, ...`. Prep is cheap: a pass over the changed roles, plus the dupes rebuild, which takes 0.8 s on the live data.
|
|
|
|
**Filters beyond location.** An agent gains `fetch_terms`: words a role must match for a big board to list it, such as `engineer`, `product`, `data`. The pull's query term is the union over the enabled agents. If any enabled agent has no terms, nothing is filtered by term, because that agent wants every role. A reader declares `filters.query`, and where a host has them, a category or level filter. The runner sends the terms as one series per term (unioned by id) or as one query, as the host allows.
|
|
|
|
Terms apply only to boards of 300 roles or more that are not tagged All roles. The closing guards of the location filter hold: roles with a vote or an application stay open, and a read under 10% of the last one fails. Terms also go to the search providers that already take a query.
|
|
|
|
**Towns.** Today a town gate is fetched as its country. A host that filters by place (`filters.place`) is asked for the towns. Every other host keeps the country and leaves the towns to the gates.
|
|
|
|
## Work packages
|
|
|
|
| WP | Model | Builds | Owns | Needs |
|
|
|---|---|---|---|---|
|
|
| S Survey | Sonnet | Per host that can filter (SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday, Oracle, Workable, Teamtailor, Jibe, Lever) and the search providers, which keyword, title, category, level and place filters exist and how they combine. At most 2 live requests per host. Also: how a town gate reaches the fetch today, and what it should become. Output `docs/plan-prep-survey.md`. | that file | |
|
|
| R1 Raw rows a to l | Sonnet | Readers Ashby to Lever hand on host words; tables move out into a file prep owns (`app/prep/tables/*.py`, one section per host, copied verbatim) | those modules, their tests, `app/prep/tables/` sections for them | K merged |
|
|
| R2 Raw rows o to z | Sonnet | The same for Oracle to Workday and the three providers | those modules and tests, their table sections | K merged |
|
|
| P Prep stage | Opus | `app/prep/` stage, host-word columns and the `unmapped` table (migration 22), `upsert_role` storing raw values, gazetteer and alias resolution moved into prep, the cascade, `app prep` CLI, the Sources page list of unmapped words | `app/prep/*` except the table sections, `app/storage/*`, `app/jobs/cascade.py`, `app/jobs/stages.py`, `app/cli.py`, `app/static/sources.js`, their tests | K merged; R1 and R2 merge with it |
|
|
| F2 Terms and towns | Opus | `fetch_terms` on agents (model, API, agent editor), terms and towns in the reach, runner series for `query` and `place`, reader filters per the survey | `app/jobs/reach.py`, `plugins/readers/_runner.py`, `_contract.py`, the filtering readers, `app/static/agents.js` | S, and R1/R2/P merged |
|
|
| F2 done 2026-09-30 | Opus | `agents.fetch_terms` (migration 23, API, a chip field in the agent editor); terms in the reach for wide reads only; one series per term on SmartRecruiters `q`, SuccessFactors `q`, Workday `searchText`, Phenom `keywords`, Eightfold `query`, Oracle `keyword`; places dropped first, then terms, over `MAX_SERIES`; a term read (`ats:narrowed`) sets no size and closes nothing; the reach line counts the terms. Towns stay fetched as their country | branch `wp-f2-terms` | |
|
|
|
|
The survey (`docs/plan-prep-survey.md`) narrowed F2: the narrow reach already leaves at most 149 roles on any big board, and keywords match the whole text, so terms go only to reads with the wide reach (All roles, or a one-country host over `MAX_SERIES`). `query` is built in this order: SmartRecruiters, SuccessFactors, then Workday, Phenom, Eightfold and Oracle. Category and level filters wait until something maps agent terms to host ids. Towns stay fetched as their country. A term-filtered read never sets `ats:size` and never closes roles as gone.
|
|
|
|
Order: S, and K finishing. Then R1, R2 and P in parallel, merged together, because the raw keys and prep have to land at once. Then F2.
|
|
|
|
## Decisions taken by the owner, 2026-09-30
|
|
|
|
1. Readers scrape and hand on host words. All mapping is central, in prep.
|
|
2. The wide reach is each agent's `require` reach (Europe). The All roles tag reads a company with it. The narrow reach applies to big boards otherwise. A favourite no longer widens the fetch.
|
|
3. Keyword, category and level filters at the source are used where hosts offer them, driven by the agents' fetch terms.
|