19 KiB
Plan: readers as plugins with a declared contract
Planned 2026-09-30. Turns every source of roles (the 22 ATS hosts, the 3 search providers, the company-specific readers of plan-hosts-wave-2.md) into a reader module with one documented API, and one runner that reads what each reader declares and handles paging, filters, per-role text, limits and failures for all of them. Pattern taken from ../news (desk/sources/spec.py, docs/plugin-rework.md): the plugin declares, the application does. Folded into decisions.md and deleted when the packages land.
This plan supersedes WP9 (provisional title, as far as the contract goes) and WP11 (board filters) of plan-hosts-wave-2.md, and covers items 2, 7, 17 and the "Plugin contract" and "Filtering at the source" sections of review-ingest.md.
How far we are
About two thirds of the way for boards, none of the way for searches.
| Part | Today | Gap |
|---|---|---|
| One module per host | yes, 22 in plugins/atshosts/, checked at import by problems() |
none |
| Documented contract | docstring of plugins/atshosts/__init__.py |
covers hosts only |
| Capabilities | inferred from which functions exist (page means paged, detail means per-role text) |
not declared; nothing about filters, caps, fields, date trust, places |
| Runner | plugins/ats.py: guarded get, robots.txt, paging, per-role reads, failure state |
fused with the ats Source; search does not use it |
| Search providers | three functions in plugins/search.py with their own getter |
no robots.txt, no size cap, no host allow-list, not modules |
| Row | a bare dict with fixed keys; ats.py builds raw from a fixed list |
a host cannot hand over country, seniority, employment type, several places |
| Date trust | TRUSTED_POSTED_HOSTS in app/models.py |
a host fact kept outside the host |
| Filters at the source | none for boards; searches have per-provider params built in app/jobs/sources.py:search_params |
no way for a reader to say what it filters, nor for the runner to pass it |
| Oversized boards | read incomplete on every pull (MAX_PAGES 100) |
no reading across pulls |
| Per-role text | 200 per board per pull, never-tried first, inside the pull | not ordered by score |
What a reader must announce
The list below is what the 25 readers do today, plus what the wave-2 readers need. The user's list (whole board, paged, list then text, location filter) is complete for the listing. It misses the traits under "also".
| Trait | Values | Who, today |
|---|---|---|
| Scope | board (one company, its whole list) or search (many companies, never a whole list) |
22 hosts, 3 providers |
| Listing | whole (one request) or paged |
whole: Greenhouse, Lever, Ashby, Recruitee, Personio, Deel, Pinpoint, BambooHR, Jobylon, Rippling. Paged: the other 12 and the 3 providers |
| Text | listing (full text in the list), excerpt (a teaser in the list), detail (one request per role), none |
detail: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Gem, Jobylon, BambooHR, Rippling. Excerpt: Jobindex, Adzuna. None: SmartRecruiters, Workable |
| Completeness | stated (the host gives a total), end (an empty page ends it), whole (one answer is all), never (a search result is never a whole list) |
|
| Page size and host ceiling | integers; the ceiling is where the host stops answering whatever we ask | Workday 2,000; Platsbanken offset 2,000; Jobindex 200; Eightfold 10 a page |
| Filters | per dimension (country, place, remote, query): none, one (one value per request), many (a list in one request), ids (values must be looked up on the host first) |
country one: SmartRecruiters, Eightfold. Many: Phenom. Ids: Workday, Oracle, SuccessFactors. Query: the 3 providers. Greenhouse, Ashby, Recruitee: none |
| Places | one, many, copies (see below) |
|
| Also: fields stated | pay (figures, line, none), posted (trusted, listed, none), remote, country, seniority, employment |
seniority: Personio, SmartRecruiters, Recruitee, Jobylon, none mapped |
| Access | key name (Adzuna), a smaller fallback feed (Greenhouse), the robots.txt decision taken when the reader was built | |
| Title | final or provisional (a sitemap reader knows a URL before a title) |
none yet; WP10 of wave 2 needs it |
| Empty answer | trusted or suspect: whether an unknown slug answers an empty board |
suspect: Ashby, SmartRecruiters, Gem |
Rate limits (429) and the per-request limits stay runner rules, the same for every reader. No reader has a need that differs.
Several places for one role
A role at "Berlin, Munich, Remote (DE)" arrives in three shapes today:
- One place of several kept. Ashby, Lever, Personio, Greenhouse, Workable, Recruitee, Platsbanken read the first and drop the rest. SuccessFactors keeps the last.
- All places joined in one line. Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold, Jibe join them with
"; ".countrybecomesEUover several countries, which the country gate does not expand. - One posting per place. The host itself lists copies with their own ids (1,741 groups, 5,021 roles).
dupes.pyleaves them apart.
The rule (owner, 2026-09-30): a role has a list of places. A reader never multiplies a row per place and never drops a place. Row.places is a list of Place(text, remote). Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares places="copies", and dupes.py folds them by company and title into one role, keeping each copy's place with its own outbound link.
The contract test (WP A) feeds every reader that declares places="many" a fixture with at least two places and fails the reader if fewer come out.
Places and companies as symbols
Places and companies become rows that other rows link to, not text matched at query time. Today the place knowledge sits in code (_CITIES, _REGIONS, _COUNTRY_NAMES in app/models.py, regions and aliases in app/places.py) and is matched word by word against roles.place_fold for every query.
places(id, name, kind, parent_id)withkindone ofcity,region,country,continent: Copenhagen belongs to Hovedstaden, which belongs to DK, which belongs to Europe. Seeded from the dictionaries in code.place_aliases(fold, place_id):kobenhavn,kopenhamn,copenhagenpoint to Copenhagen, andskaneandskåneto Skåne. Typing either spelling finds the place.role_places(uid, place_id, text, url): one row per place of a role.place_idis empty for a text the gazetteer does not know. That text is added to anunresolvedlist, which a person or a later job resolves, and it is linked once resolved.urlis the copy's own link when copies were folded.- A country filter or gate is a join over the tree: DK expands to every place under DK, and Europe to every country under Europe. The
EUfallback and the per-request word matching go. This also enables counts per place. roles.placestays the displayed line: the first place, then "+2" when there are more.company_aliases(fold, company_key): "Orsted" and "Ørsted" become one company without changing the primary key. Companies gain empty columns for enrichment (sector, stage, size band, founded, last funding, source and date of each fact), which the startupmap.one and The Hub readers fill later.
The API
The package plugins/atshosts/ becomes plugins/readers/, and the search providers move into it. The contract stays the package docstring, extended, with plugins/readers/_contract.py holding the types and problems(). plugins/README.md stays the list of sources and their measured limits.
# plugins/readers/workday.py
NAME = "workday"
LABEL = "Workday"
PATTERNS = (r"...",)
READS = Reads(
scope="board",
listing="paged", page_size=20, ceiling=2000,
text="detail",
complete="stated",
filters=Filters(country="ids", remote="ids"),
places="many",
fields=Fields(posted="listed", pay="none", remote="stated"),
robots="read: robots.txt allows /wday/cxs/ (checked 2026-09-29)",
empty="trusted",
)
def slug_ok(slug) -> bool
def allowed(slug) -> Callable[[str], bool]
def careers_url(slug) -> str
# listing="whole"
def feed_url(slug, ask: Ask) -> str
def parse(body, slug) -> list[Row]
# listing="paged"
def page(get, slug, cursor, ask: Ask) -> Page
def rows(decoded, slug) -> list[Row]
# text="detail"
def detail(get, slug, row) -> dict
# filters with "ids"
def facets(get, slug) -> dict[str, dict[str, str]] # {"country": {"DE": "<host id>"}}
# optional, as today
def fallback_url, robots_page, probe_url, probe_ok, probe_slugs
Ask is what the runner wants from one series of requests: countries, places, remote, query, params (a search's saved params). A reader reads only the dimensions it declared, and the runner never hands it a value it did not declare. With one, the runner makes one series per value and unions them by id. With ids, it calls facets once a day per board, keeps the result in state and puts host ids into Ask. Row becomes a TypedDict with places, and optional country, seniority, employment. The runner puts every stated key into raw, so a new field needs no runner change.
problems() checks the declaration against the functions: paged needs page and rows, text="detail" needs detail, an ids filter needs facets, scope="search" needs complete="never" or stated, key needs a config name. A module that fails refuses to import, as today.
TRUSTED_POSTED_HOSTS goes: fields.posted="trusted" says the same thing in the module.
robots.txt is decided when a reader is built, not checked at run time (owner, 2026-09-30). Whoever builds a reader reads the host's robots.txt, reports what it disallows, and the owner decides per reader whether to read the host anyway. robots is a required string that records the decision and the date. The runtime check, its state keys and its tests go. Jobindex is read with paging and area. HR Manager, left out because its robots.txt disallows reading, goes back to the owner for a decision.
The runner
plugins/readers/_runner.py, taken out of plugins/ats.py. plugins/ats.py and plugins/search.py stay as thin Source classes, so subscriptions and state keys (ats:*, search:*) keep their names and no migration is needed.
For one target, the runner:
-
Loads the reader and its subscription.
-
Resolves
Ask. The reach is set in the front end by the agents that are switched on:- Base reach: the union of their
anygates on country, town and remote, or everything when one agent has none. - Favourite companies (favourite = 2, which already skips the
anygates): only therequiregates, which today means Europe. Mistral or Databricks are read across Europe, so a role outside Denmark still shows up and can prompt a question to them. - Board override:
paramson the board's subscription.
The reach is then narrowed to the filters the reader declares. A search's own params go in
Ask.params. Boards under 300 roles get no filter. - Base reach: the union of their
-
Applies the key.
-
Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded
getfor every reader: host allow-list, https only, size cap, deadline, fallback feed. A teaser in the list is kept (Row.teaser). -
Sweeps across pulls. A paged read that hits
MAX_PAGESsaves its cursor and the ids seen, and the next pull continues from there. A sweep that reaches the end is complete and closes what it did not list. A host ceiling (Workday 2,000) is not swept: the reader must filter below it or stays incomplete. -
Decides completeness from the declaration, and applies the guards on closing: a filtered read closes roles outside the filter, except roles with a vote, an application or a note. A filtered or
empty="suspect"read that returns under 10% of the last count counts as failed. -
Records failures (
failingsince and reason, 21 days to close), 429 handling and waits, as today.
The runner makes two runs (owner, 2026-09-30):
- Listing run (the pull). The agents hand the reach to the runner. The runner calls each reader with its filters and handles paging and sweeps. Readers return standard rows, with every place of a role. Then
dupes.pyfolds copies, andembedandscorerun. - Enrich run (new stage
enrich). After scoring, the runner is handed the ranked roles. For each role that still needs a call (text throughdetail, and later other per-role facts), it fetches the highest-scored first, within a budget per board and per run. It drops the vector of each role whose text improved, andembedandscorerun again for those roles only.
The enrich run follows every rescore that changes the ranking materially: new roles from a pull, an agent created or edited, a company starred or unstarred. A vote on a role does not trigger it. A role never scored above the agents' thresholds gets no extra call. That is the budget which keeps the enrich run from reading every board in full.
Seniority: stated, then the model
43% of open roles have a level, all taken from title words. A sample of 59 without a level was 85% individual contributors. Personio, SmartRecruiters, Recruitee and Jobylon state a level that nothing reads. extract has never run.
A role keeps three sources of a level, and the displayed roles.seniority is the first that is filled, in this order:
seniority_model: the model's verdict, fromextractrun as an afterburner. It reads roles whose model value is empty. The prompt includes the stated and title values as hints, and the model may overrule them.seniority_stated: the host's own word, mapped to the app's vocabulary (app.seniority.SENIORITIES) by a table per reader in_shared.py. The raw word stays inraw. A word the table does not know maps to empty and is logged, so the table can grow.seniority_title:seniority.from_title, as today.
The filters read roles.seniority. A later comparison of stated and model values measures how far each host can be trusted.
Decisions taken by the owner, 2026-09-30
- Filters come from the agents that are switched on. Favourite companies are read across the
requirereach (Europe), and a board can override its filter. - robots.txt is decided per reader at build time. Nothing checks it at run time. Jobindex is read in full.
- Places are a list per role. Copies per place are folded, each keeping its own link. Places and companies become symbols with aliases.
- A teaser is kept. A short description in the list (Phenom and Oracle: 1,066 roles, plus Jobindex and Adzuna excerpts) goes into
Row.teaserandroles.teaser. Until the full text arrives, it is shown and embedded, and a reader that sees one must hand it over. When the full text arrives, it replaces the teaser for embedding and the vector is dropped, as today. - Seniority has a stated value from the plugin and a model value from an afterburner that fills what is empty and may overrule it.
Work packages
Each package owns the files listed and their tests. The planner makes each worktree from local main (see memory: isolation worktrees branch from stale origin), merges, and removes the xfail marks a package fixed.
| WP | Model | Builds | Owns | Needs |
|---|---|---|---|---|
| A Contract | Opus | rename atshosts to readers; _contract.py with Reads, Filters, Fields, Ask, Place, Row (with places, teaser), problems(); the extended docstring with the robots.txt rule; a conformance test over every module (declaration against functions, places="many" and teaser fixtures) marking today's gaps xfail per module |
plugins/readers/__init__.py, _contract.py, _shared.py, every import of atshosts, tests/test_readers_contract.py |
|
| B Runner | Opus | _runner.py out of ats.py; runtime robots.txt check removed; Ask from params; series union; sweep; closing guards; ats.py and search.py thin; raw built from every stated Row key |
plugins/ats.py, plugins/search.py, plugins/readers/_runner.py, app/jobs/pull.py, tests/test_ats.py, tests/test_search.py, tests/test_pull.py |
A |
| C Providers | Sonnet | jobindex.py, adzuna.py, platsbanken.py as reader modules with teaser and places; search_params built from the declarations |
the three modules, app/jobs/sources.py, their tests |
A |
| D1 Hosts a to l | Sonnet | READS, all places, teaser, host-stated seniority mapped to seniority_stated, date trust in fields, robots decision string for Ashby to Lever |
those modules and tests | A |
| D2 Hosts o to w | Sonnet | the same for Oracle to Workday | those modules and tests | A |
| E1 Places and teaser storage | Opus | places, place_aliases, role_places, roles.teaser, the three seniority columns, migration seeded from the dictionaries in code; upsert_role writes them; gates and the Place filter as joins over the tree; TRUSTED_POSTED_HOSTS removed; embed falls back to the teaser |
app/storage/*, app/models.py, app/places.py, app/jobs/embed.py, their tests |
A |
| F Reach and filters | Opus | reach from the agents and favourites into Ask; facets and Ask for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; each reader's declaration shown on the Sources page |
app/jobs/reach.py, those six modules, app/web/api.py, app/static/companies.js, app/static/sources.js |
B, D1, D2, E1 |
| G Enrich run | Opus | enrich as its own stage over the ranked roles, highest score first; triggered after a pull's scoring, after an agent is saved and after a star changes, not after a vote; re-embed and rescore only the roles it filled |
app/jobs/detail.py, app/jobs/cascade.py, app/jobs/stages.py, their tests |
B |
| E2 Copies and place list | Sonnet | dupes.py folds copies per place into one role with per-place links; the reader shows the place list with links; the teaser is shown when there is no text |
app/dupes.py, app/static/roles.js, app/templates/*, their tests |
E1 |
| I Seniority | Sonnet | seniority_stated, seniority_model, seniority_title and the precedence; extract as an afterburner over roles without a model value, with the hints |
app/seniority.py, app/jobs/extract.py, their tests (the columns come from E1) |
E1 |
| H Companies as symbols | Sonnet | company_aliases, alias folding on insert and lookup, empty enrichment columns |
app/storage/* after E1, app/jobs/sources.py after C, their tests |
E1, C |
Order: A first. Then B, C, D1, D2 and E1 in parallel. Then F, G, E2, H and I in parallel; H adds a migration after E1's. After the merges, the planner attaches Infineon and Microsoft on Eightfold with a country filter, updates plugins/README.md, and folds this plan into decisions.md.