Files
jobboard/docs/plan-prep.md
T

8.1 KiB

Plan: readers only scrape, a prep stage decides, and the agents narrow the fetch

Planned 2026-09-30 after the readers rework landed. It is folded into decisions.md and deleted when it lands.

What the owner asked for

  1. A reader is a scraper. It gets the data in and makes light touches only: HTML to text, a date parsed, the places of a role as a list. It hands on the host's own words, and it maps nothing to the app's vocabulary.
  2. A prep stage makes every decision that needs a rule or data across roles. That covers seniority, employment type, remote, country from places, places against the gazetteer, company aliases, and folding copies and duplicates across the whole database, not only the run. It runs centrally, once, between the pull and embedding.
  3. Two levels of reach, set by the agents.
    • The wide reach is an agent's require gates, Europe today, and it is what "all roles" means.
    • The narrow reach is the any gates: Denmark, Sweden, or towns.
    • Big boards (300 roles or more) are read with the narrow reach where the reader can filter. A company tagged All roles is read with the wide reach. A reader that cannot filter reads the whole board.
    • Being a favourite affects scoring only.
  4. More filters at the source. If a host can filter by keyword, title, category or level, the agents' own terms go to it, so a board with 1,000 roles or more (Bosch, Delivery Hero) is narrowed before it is read.

Where it stands

Part Today Gap
Places, companies mapped centrally in upsert_role (gazetteer, company_aliases) none, but it runs inline in the upsert and not as a stage
Seniority, employment a word table in each reader, _shared.table_word per reader; an unknown word is only a debug log line
Remote each reader turns the host value into True, False or a word per reader, spelt differently
Country a reader sets it when a role has one country should come from the places
Copies, duplicates rebuild_dupes over the whole database after a pull already central; to become part of prep
Reach narrow = union of any gates; favourites = require reach the All roles tag is being built (WP K) with the wide reach; favourite stops widening the fetch
Filters at the source country and remote on six hosts no keyword, category or level filters; towns are fetched as their country

Design

Raw row keys. A reader sets:

  • seniority and employment to the host's own word, unchanged (e.g. "Experienced", "Regular Full Time")
  • remote to the host's own value: a bool, or the host's word
  • no country unless the host states one per role. When it does, the key carries the host's own spelling, or its ISO code as given.

The READS.fields declaration stays, and says which of these a host states. _shared.table_word and every per-host table leave the readers.

Storage. roles keeps the host words in seniority_host, employment_host and remote_host, and raw keeps them too. The mapped columns (seniority_stated, employment, remote, country) are written by prep only. Prep can map again from the stored host words when a table changes, with no new pull.

The prep stage is app/prep/: one module per concern, and one table file per field with a section per host.

  • Seniority: the host word through that host's table, then the title rules (seniority.from_title). The model value from extract still wins over both.
  • Employment: one table per host into EMPLOYMENTS.
  • Remote: the host value through one shared word map, per host where a host needs its own.
  • Country: from the role's resolved places through the gazetteer. A host-stated country is only a fallback.
  • Places and companies: the gazetteer and alias resolution move from inline upsert_role calls into prep. upsert_role stores the raw values; prep links them.
  • Copies and duplicates: prep runs rebuild_dupes, which already works over the whole database.
  • Unknown words: kept in unmapped(field, host, word, roles, first_seen) and listed by app prep unmapped and on the Sources page, beside the unresolved places. Adding a word to a table and running app prep --remap fixes the stored roles.

The cascade becomes pull, prep, embed, score, enrich, prep (filled roles), embed, score, extract, .... Prep is cheap: a pass over the changed roles, plus the dupes rebuild, which takes 0.8 s on the live data.

Filters beyond location. An agent gains fetch_terms: words a role must match for a big board to list it, such as engineer, product, data. The pull's query term is the union over the enabled agents. If any enabled agent has no terms, nothing is filtered by term, because that agent wants every role. A reader declares filters.query, and where a host has them, a category or level filter. The runner sends the terms as one series per term (unioned by id) or as one query, as the host allows.

Terms apply only to boards of 300 roles or more that are not tagged All roles. The closing guards of the location filter hold: roles with a vote or an application stay open, and a read under 10% of the last one fails. Terms also go to the search providers that already take a query.

Towns. Today a town gate is fetched as its country. A host that filters by place (filters.place) is asked for the towns. Every other host keeps the country and leaves the towns to the gates.

Work packages

WP Model Builds Owns Needs
S Survey Sonnet Per host that can filter (SmartRecruiters, Eightfold, SuccessFactors, Phenom, Workday, Oracle, Workable, Teamtailor, Jibe, Lever) and the search providers, which keyword, title, category, level and place filters exist and how they combine. At most 2 live requests per host. Also: how a town gate reaches the fetch today, and what it should become. Output docs/plan-prep-survey.md. that file
R1 Raw rows a to l Sonnet Readers Ashby to Lever hand on host words; tables move out into a file prep owns (app/prep/tables/*.py, one section per host, copied verbatim) those modules, their tests, app/prep/tables/ sections for them K merged
R2 Raw rows o to z Sonnet The same for Oracle to Workday and the three providers those modules and tests, their table sections K merged
P Prep stage Opus app/prep/ stage, host-word columns and the unmapped table (migration 22), upsert_role storing raw values, gazetteer and alias resolution moved into prep, the cascade, app prep CLI, the Sources page list of unmapped words app/prep/* except the table sections, app/storage/*, app/jobs/cascade.py, app/jobs/stages.py, app/cli.py, app/static/sources.js, their tests K merged; R1 and R2 merge with it
F2 Terms and towns Opus fetch_terms on agents (model, API, agent editor), terms and towns in the reach, runner series for query and place, reader filters per the survey app/jobs/reach.py, plugins/readers/_runner.py, _contract.py, the filtering readers, app/static/agents.js S, and R1/R2/P merged

The survey (docs/plan-prep-survey.md) narrowed F2: the narrow reach already leaves at most 149 roles on any big board, and keywords match the whole text, so terms go only to reads with the wide reach (All roles, or a one-country host over MAX_SERIES). query is built in this order: SmartRecruiters, SuccessFactors, then Workday, Phenom, Eightfold and Oracle. Category and level filters wait until something maps agent terms to host ids. Towns stay fetched as their country. A term-filtered read never sets ats:size and never closes roles as gone.

Order: S, and K finishing. Then R1, R2 and P in parallel, merged together, because the raw keys and prep have to land at once. Then F2.

Decisions taken by the owner, 2026-09-30

  1. Readers scrape and hand on host words. All mapping is central, in prep.
  2. The wide reach is each agent's require reach (Europe). The All roles tag reads a company with it. The narrow reach applies to big boards otherwise. A favourite no longer widens the fetch.
  3. Keyword, category and level filters at the source are used where hosts offer them, driven by the agents' fetch terms.