13 KiB
Plan: readers as plugins with a declared contract
Planned 2026-09-30. Turns every source of roles (the 22 ATS hosts, the 3 search providers, the company-specific readers of plan-hosts-wave-2.md) into a reader module with one documented API, and one runner that reads what each reader declares and handles paging, filters, per-role text, limits and failures for all of them. Pattern taken from ../news (desk/sources/spec.py, docs/plugin-rework.md): the plugin declares, the application does. Folded into decisions.md and deleted when the packages land.
This plan supersedes WP9 (provisional title, as far as the contract goes) and WP11 (board filters) of plan-hosts-wave-2.md, and covers items 2, 7, 17 and the "Plugin contract" and "Filtering at the source" sections of review-ingest.md.
How far we are
About two thirds of the way for boards, none of the way for searches.
| Part | Today | Gap |
|---|---|---|
| One module per host | yes, 22 in plugins/atshosts/, checked at import by problems() |
none |
| Documented contract | docstring of plugins/atshosts/__init__.py |
covers hosts only |
| Capabilities | inferred from which functions exist (page means paged, detail means per-role text) |
not declared; nothing about filters, caps, fields, date trust, places |
| Runner | plugins/ats.py: guarded get, robots.txt, paging, per-role reads, failure state |
fused with the ats Source; search does not use it |
| Search providers | three functions in plugins/search.py with their own getter |
no robots.txt, no size cap, no host allow-list, not modules |
| Row | a bare dict with fixed keys; ats.py builds raw from a fixed list |
a host cannot hand over country, seniority, employment type, several places |
| Date trust | TRUSTED_POSTED_HOSTS in app/models.py |
a host fact kept outside the host |
| Filters at the source | none for boards; searches have per-provider params built in app/jobs/sources.py:search_params |
no way for a reader to say what it filters, nor for the runner to pass it |
| Oversized boards | read incomplete on every pull (MAX_PAGES 100) |
no reading across pulls |
| Per-role text | 200 per board per pull, never-tried first, inside the pull | not ordered by score |
What a reader must announce
The list below is what the 25 readers do today, plus what the wave-2 readers need. The user's list (whole board, paged, list then text, location filter) is complete for the listing. It misses the traits under "also".
| Trait | Values | Who, today |
|---|---|---|
| Scope | board (one company, its whole list) or search (many companies, never a whole list) |
22 hosts, 3 providers |
| Listing | whole (one request) or paged |
whole: Greenhouse, Lever, Ashby, Recruitee, Personio, Deel, Pinpoint, BambooHR, Jobylon, Rippling. Paged: the other 12 and the 3 providers |
| Text | listing (full text in the list), excerpt (a teaser in the list), detail (one request per role), none |
detail: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Gem, Jobylon, BambooHR, Rippling. Excerpt: Jobindex, Adzuna. None: SmartRecruiters, Workable |
| Completeness | stated (the host gives a total), end (an empty page ends it), whole (one answer is all), never (a search result is never a whole list) |
|
| Page size and host ceiling | integers; the ceiling is where the host stops answering whatever we ask | Workday 2,000; Platsbanken offset 2,000; Jobindex 200; Eightfold 10 a page |
| Filters | per dimension (country, place, remote, query): none, one (one value per request), many (a list in one request), ids (values must be looked up on the host first) |
country one: SmartRecruiters, Eightfold. Many: Phenom. Ids: Workday, Oracle, SuccessFactors. Query: the 3 providers. Greenhouse, Ashby, Recruitee: none |
| Places | one, many, copies (see below) |
|
| Also: fields stated | pay (figures, line, none), posted (trusted, listed, none), remote, country, seniority, employment |
seniority: Personio, SmartRecruiters, Recruitee, Jobylon, none mapped |
| Access | key name (Adzuna), robots.txt page, a smaller fallback feed (Greenhouse) | |
| Title | final or provisional (a sitemap reader knows a URL before a title) |
none yet; WP10 of wave 2 needs it |
| Empty answer | trusted or suspect: whether an unknown slug answers an empty board |
suspect: Ashby, SmartRecruiters, Gem |
Rate limits (429) and the per-request limits stay runner rules, the same for every reader. No reader has a need that differs.
Several places for one role
A role at "Berlin, Munich, Remote (DE)" arrives in three shapes today:
- One place of several kept. Ashby, Lever, Personio, Greenhouse, Workable, Recruitee, Platsbanken read the first and drop the rest. SuccessFactors keeps the last.
- All places joined in one line. Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold, Jibe join them with
"; ".countrybecomesEUover several countries, which the country gate does not expand. - One posting per place. The host itself lists copies with their own ids (1,741 groups, 5,021 roles).
dupes.pyleaves them apart.
The rule: one posting on the host is one row, and a row carries every place the host states. A reader never multiplies a row per place and never drops a place. Row.places is a list of Place(text, city, country, remote). The runner fills city and country from text via app/places.py if the reader leaves them empty. Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares places="copies", and dupes.py folds them by company and title, which answers owner decision 10 of the review with yes.
Storage: roles.place stays the displayed line (the first place, "+2" when there are more), and a new table role_places(role_id, place_fold, country) holds all of them. The place and country gates and the Place filter match any row of it, so the EU fallback goes. The contract test (WP A) feeds every reader that declares places="many" a fixture with at least two places and fails the reader if fewer come out.
The API
The package plugins/atshosts/ becomes plugins/readers/, and the search providers move into it. The contract stays the package docstring, extended, with plugins/readers/_contract.py holding the types and problems(). plugins/README.md stays the list of sources and their measured limits.
# plugins/readers/workday.py
NAME = "workday"
LABEL = "Workday"
PATTERNS = (r"...",)
READS = Reads(
scope="board",
listing="paged", page_size=20, ceiling=2000,
text="detail",
complete="stated",
filters=Filters(country="ids", remote="ids"),
places="many",
fields=Fields(posted="listed", pay="none", remote="stated"),
robots=True,
empty="trusted",
)
def slug_ok(slug) -> bool
def allowed(slug) -> Callable[[str], bool]
def careers_url(slug) -> str
# listing="whole"
def feed_url(slug, ask: Ask) -> str
def parse(body, slug) -> list[Row]
# listing="paged"
def page(get, slug, cursor, ask: Ask) -> Page
def rows(decoded, slug) -> list[Row]
# text="detail"
def detail(get, slug, row) -> dict
# filters with "ids"
def facets(get, slug) -> dict[str, dict[str, str]] # {"country": {"DE": "<host id>"}}
# optional, as today
def fallback_url, robots_page, probe_url, probe_ok, probe_slugs
Ask is what the runner wants from one series of requests: countries, places, remote, query, params (a search's saved params). A reader reads only the dimensions it declared, and the runner never hands it a value it did not declare. With one, the runner makes one series per value and unions them by id. With ids, it calls facets once a day per board, keeps the result in state and puts host ids into Ask. Row becomes a TypedDict with places, and optional country, seniority, employment. The runner puts every stated key into raw, so a new field needs no runner change.
problems() checks the declaration against the functions: paged needs page and rows, text="detail" needs detail, an ids filter needs facets, scope="search" needs complete="never" or stated, key needs a config name. A module that fails refuses to import, as today.
TRUSTED_POSTED_HOSTS goes: fields.posted="trusted" says the same thing in the module.
The runner
plugins/readers/_runner.py, taken out of plugins/ats.py. plugins/ats.py and plugins/search.py stay as thin Source classes, so subscriptions and state keys (ats:*, search:*) keep their names and no migration is needed.
For one target, the runner:
- Loads the reader and its subscription.
- Resolves
Ask: the reach of the enabled agents (union of theiranygates on country, town and remote;allif one agent has none), narrowed to what the reader declares, overridden byparamson the board's subscription. A search's own params go inAsk.params. Boards under 300 roles get no filter. - Checks robots.txt, for searches too, and applies the key.
- Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded
getfor every reader: host allow-list, https only, size cap, deadline, fallback feed. - Sweeps across pulls. A paged read that hits
MAX_PAGESsaves its cursor and the ids seen, and the next pull continues from there. A sweep that reaches the end is complete and closes what it did not list. A host ceiling (Workday 2,000) is not swept: the reader must filter below it or stays incomplete. - Decides completeness from the declaration, and applies the guards on closing: a filtered read closes roles outside the filter, except roles with a vote, an application or a note. A filtered or
empty="suspect"read that returns under 10% of the last count counts as failed. - Records failures (
failingsince and reason, 21 days to close), 429 handling and waits, as today.
Per-role text leaves the pull. A new stage detail runs after score in the cascade (pull, embed, score, detail, embed, score, ...). It reads the text of the highest-scored roles without text first, 200 per board per run, and drops the vector of each role it fills so the next embed encodes it with the text. The feed ranks titles first and gets the text for the top roles.
Decisions (defaults taken, change before WP B starts)
- Where filters come from. Default: derived at pull time from the enabled agents' gates, as
review-ingest.mddesigned, with a per-board override on the Companies page. The alternative is a filter set by hand per board. - Jobindex under robots.txt. Once searches go through the runner, the robots check fails Jobindex paging and area (review finding 21). Default: Jobindex declares
robots=Trueand reads the first page of each search, 20 results. The alternatives are to exempt it or to drop it. - Copies per place folded. Default: yes, as above.
Work packages
Each package owns the files listed and their tests. The planner makes each worktree from local main (see memory: isolation worktrees branch from stale origin).
| WP | Model | Builds | Owns | Needs |
|---|---|---|---|---|
| A Contract | Opus | rename atshosts to readers; _contract.py with Reads, Filters, Fields, Ask, Place, Row, problems(); the extended docstring; a conformance test over every module (declaration against functions; places="many" fixtures) that marks today's gaps xfail per module |
plugins/readers/__init__.py, _contract.py, _shared.py, every import of atshosts, tests/test_readers_contract.py |
|
| B Runner | Opus | _runner.py out of ats.py; Ask resolution without reach (params only); series union; sweep; closing guards; ats.py and search.py thin |
plugins/ats.py, plugins/search.py, plugins/readers/_runner.py, app/jobs/pull.py, tests/test_ats.py, tests/test_search.py, tests/test_pull.py |
A |
| C Providers | Sonnet | jobindex.py, adzuna.py, platsbanken.py as reader modules; search_params built from the declarations |
the three modules, app/jobs/sources.py, their tests |
A |
| D1 Hosts a to l | Sonnet | READS and places for Ashby to Lever; host-stated seniority; date trust into fields |
those modules and tests | A |
| D2 Hosts o to w | Sonnet | same for Oracle to Workday | those modules and tests | A |
| E Places | Opus | role_places, migration, gates and Place filter over it, dupes.py fold of copies, app/models.py without TRUSTED_POSTED_HOSTS |
app/storage/*, app/models.py, app/dupes.py, app/places.py, their tests |
A |
| F Reach and filters | Opus | reach from agents into Ask; facets and Ask for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; readers' declarations shown on the Sources page |
app/jobs/reach.py, those six modules, app/web/api.py, app/static/companies.js, app/static/sources.js |
B, D1, D2 |
| G Detail stage | Sonnet | detail as its own stage by score; cascade order |
app/jobs/detail.py, app/jobs/cascade.py, app/jobs/stages.py, tests |
B |
A goes first. B, C, D1, D2 and E run in parallel. F and G follow B. The planner merges, removes each xfail a package fixed, attaches Infineon and Microsoft on Eightfold with a country filter, and updates plugins/README.md.