Plan readers as plugins: declared traits, one API, one runner, one row per posting with all its places

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 08:34:49 +02:00
1 parent 667e1d35f4
commit 10fdd60988
2 files changed
+141

No files matched your search

+2
View File
@@ -2,6 +2,8 @@
Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped.
WP9 (the contract part) and WP11 are superseded by `docs/plan-readers.md`; WP10 and WP12 to WP15 build against its contract once its WP A has landed.
## Where it stands
286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
+139
View File
@@ -0,0 +1,139 @@
# Plan: readers as plugins with a declared contract
Planned 2026-09-30. Turns every source of roles (the 22 ATS hosts, the 3 search providers, the company-specific readers of `plan-hosts-wave-2.md`) into a reader module with one documented API, and one runner that reads what each reader declares and handles paging, filters, per-role text, limits and failures for all of them. Pattern taken from `../news` (`desk/sources/spec.py`, `docs/plugin-rework.md`): the plugin declares, the application does. Folded into `decisions.md` and deleted when the packages land.
This plan supersedes WP9 (provisional title, as far as the contract goes) and WP11 (board filters) of `plan-hosts-wave-2.md`, and covers items 2, 7, 17 and the "Plugin contract" and "Filtering at the source" sections of `review-ingest.md`.
## How far we are
About two thirds of the way for boards, none of the way for searches.
| Part | Today | Gap |
|---|---|---|
| One module per host | yes, 22 in `plugins/atshosts/`, checked at import by `problems()` | none |
| Documented contract | docstring of `plugins/atshosts/__init__.py` | covers hosts only |
| Capabilities | inferred from which functions exist (`page` means paged, `detail` means per-role text) | not declared; nothing about filters, caps, fields, date trust, places |
| Runner | `plugins/ats.py`: guarded `get`, robots.txt, paging, per-role reads, failure state | fused with the `ats` Source; search does not use it |
| Search providers | three functions in `plugins/search.py` with their own getter | no robots.txt, no size cap, no host allow-list, not modules |
| Row | a bare dict with fixed keys; `ats.py` builds `raw` from a fixed list | a host cannot hand over country, seniority, employment type, several places |
| Date trust | `TRUSTED_POSTED_HOSTS` in `app/models.py` | a host fact kept outside the host |
| Filters at the source | none for boards; searches have per-provider params built in `app/jobs/sources.py:search_params` | no way for a reader to say what it filters, nor for the runner to pass it |
| Oversized boards | read incomplete on every pull (`MAX_PAGES` 100) | no reading across pulls |
| Per-role text | 200 per board per pull, never-tried first, inside the pull | not ordered by score |
## What a reader must announce
The list below is what the 25 readers do today, plus what the wave-2 readers need. The user's list (whole board, paged, list then text, location filter) is complete for the listing. It misses the traits under "also".
| Trait | Values | Who, today |
|---|---|---|
| Scope | `board` (one company, its whole list) or `search` (many companies, never a whole list) | 22 hosts, 3 providers |
| Listing | `whole` (one request) or `paged` | whole: Greenhouse, Lever, Ashby, Recruitee, Personio, Deel, Pinpoint, BambooHR, Jobylon, Rippling. Paged: the other 12 and the 3 providers |
| Text | `listing` (full text in the list), `excerpt` (a teaser in the list), `detail` (one request per role), `none` | detail: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Gem, Jobylon, BambooHR, Rippling. Excerpt: Jobindex, Adzuna. None: SmartRecruiters, Workable |
| Completeness | `stated` (the host gives a total), `end` (an empty page ends it), `whole` (one answer is all), `never` (a search result is never a whole list) | |
| Page size and host ceiling | integers; the ceiling is where the host stops answering whatever we ask | Workday 2,000; Platsbanken offset 2,000; Jobindex 200; Eightfold 10 a page |
| Filters | per dimension (country, place, remote, query): `none`, `one` (one value per request), `many` (a list in one request), `ids` (values must be looked up on the host first) | country one: SmartRecruiters, Eightfold. Many: Phenom. Ids: Workday, Oracle, SuccessFactors. Query: the 3 providers. Greenhouse, Ashby, Recruitee: none |
| Places | `one`, `many`, `copies` (see below) | |
| **Also:** fields stated | pay (`figures`, `line`, `none`), posted (`trusted`, `listed`, `none`), remote, country, seniority, employment | seniority: Personio, SmartRecruiters, Recruitee, Jobylon, none mapped |
| Access | key name (Adzuna), robots.txt page, a smaller fallback feed (Greenhouse) | |
| Title | `final` or `provisional` (a sitemap reader knows a URL before a title) | none yet; WP10 of wave 2 needs it |
| Empty answer | `trusted` or `suspect`: whether an unknown slug answers an empty board | suspect: Ashby, SmartRecruiters, Gem |
Rate limits (429) and the per-request limits stay runner rules, the same for every reader. No reader has a need that differs.
## Several places for one role
A role at "Berlin, Munich, Remote (DE)" arrives in three shapes today:
- **One place of several kept.** Ashby, Lever, Personio, Greenhouse, Workable, Recruitee, Platsbanken read the first and drop the rest. SuccessFactors keeps the last.
- **All places joined in one line.** Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold, Jibe join them with `"; "`. `country` becomes `EU` over several countries, which the country gate does not expand.
- **One posting per place.** The host itself lists copies with their own ids (1,741 groups, 5,021 roles). `dupes.py` leaves them apart.
The rule: **one posting on the host is one row, and a row carries every place the host states.** A reader never multiplies a row per place and never drops a place. `Row.places` is a list of `Place(text, city, country, remote)`. The runner fills `city` and `country` from `text` via `app/places.py` if the reader leaves them empty. Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares `places="copies"`, and `dupes.py` folds them by company and title, which answers owner decision 10 of the review with yes.
Storage: `roles.place` stays the displayed line (the first place, "+2" when there are more), and a new table `role_places(role_id, place_fold, country)` holds all of them. The place and country gates and the Place filter match any row of it, so the `EU` fallback goes. The contract test (WP A) feeds every reader that declares `places="many"` a fixture with at least two places and fails the reader if fewer come out.
## The API
The package `plugins/atshosts/` becomes `plugins/readers/`, and the search providers move into it. The contract stays the package docstring, extended, with `plugins/readers/_contract.py` holding the types and `problems()`. `plugins/README.md` stays the list of sources and their measured limits.
```python
# plugins/readers/workday.py
NAME = "workday"
LABEL = "Workday"
PATTERNS = (r"...",)
READS = Reads(
scope="board",
listing="paged", page_size=20, ceiling=2000,
text="detail",
complete="stated",
filters=Filters(country="ids", remote="ids"),
places="many",
fields=Fields(posted="listed", pay="none", remote="stated"),
robots=True,
empty="trusted",
)
def slug_ok(slug) -> bool
def allowed(slug) -> Callable[[str], bool]
def careers_url(slug) -> str
# listing="whole"
def feed_url(slug, ask: Ask) -> str
def parse(body, slug) -> list[Row]
# listing="paged"
def page(get, slug, cursor, ask: Ask) -> Page
def rows(decoded, slug) -> list[Row]
# text="detail"
def detail(get, slug, row) -> dict
# filters with "ids"
def facets(get, slug) -> dict[str, dict[str, str]] # {"country": {"DE": "<host id>"}}
# optional, as today
def fallback_url, robots_page, probe_url, probe_ok, probe_slugs
```
`Ask` is what the runner wants from one series of requests: `countries`, `places`, `remote`, `query`, `params` (a search's saved params). A reader reads only the dimensions it declared, and the runner never hands it a value it did not declare. With `one`, the runner makes one series per value and unions them by id. With `ids`, it calls `facets` once a day per board, keeps the result in state and puts host ids into `Ask`. `Row` becomes a `TypedDict` with `places`, and optional `country`, `seniority`, `employment`. The runner puts every stated key into `raw`, so a new field needs no runner change.
`problems()` checks the declaration against the functions: `paged` needs `page` and `rows`, `text="detail"` needs `detail`, an `ids` filter needs `facets`, `scope="search"` needs `complete="never"` or `stated`, `key` needs a config name. A module that fails refuses to import, as today.
`TRUSTED_POSTED_HOSTS` goes: `fields.posted="trusted"` says the same thing in the module.
## The runner
`plugins/readers/_runner.py`, taken out of `plugins/ats.py`. `plugins/ats.py` and `plugins/search.py` stay as thin `Source` classes, so subscriptions and state keys (`ats:*`, `search:*`) keep their names and no migration is needed.
For one target, the runner:
1. Loads the reader and its subscription.
2. Resolves `Ask`: the reach of the enabled agents (union of their `any` gates on country, town and remote; `all` if one agent has none), narrowed to what the reader declares, overridden by `params` on the board's subscription. A search's own params go in `Ask.params`. Boards under 300 roles get no filter.
3. Checks robots.txt, for searches too, and applies the key.
4. Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded `get` for every reader: host allow-list, https only, size cap, deadline, fallback feed.
5. **Sweeps across pulls.** A paged read that hits `MAX_PAGES` saves its cursor and the ids seen, and the next pull continues from there. A sweep that reaches the end is complete and closes what it did not list. A host ceiling (Workday 2,000) is not swept: the reader must filter below it or stays incomplete.
6. Decides completeness from the declaration, and applies the guards on closing: a filtered read closes roles outside the filter, except roles with a vote, an application or a note. A filtered or `empty="suspect"` read that returns under 10% of the last count counts as failed.
7. Records failures (`failing` since and reason, 21 days to close), 429 handling and waits, as today.
Per-role text leaves the pull. A new stage `detail` runs after `score` in the cascade (`pull, embed, score, detail, embed, score, ...`). It reads the text of the highest-scored roles without text first, 200 per board per run, and drops the vector of each role it fills so the next `embed` encodes it with the text. The feed ranks titles first and gets the text for the top roles.
## Decisions (defaults taken, change before WP B starts)
1. **Where filters come from.** Default: derived at pull time from the enabled agents' gates, as `review-ingest.md` designed, with a per-board override on the Companies page. The alternative is a filter set by hand per board.
2. **Jobindex under robots.txt.** Once searches go through the runner, the robots check fails Jobindex paging and area (review finding 21). Default: Jobindex declares `robots=True` and reads the first page of each search, 20 results. The alternatives are to exempt it or to drop it.
3. **Copies per place folded.** Default: yes, as above.
## Work packages
Each package owns the files listed and their tests. The planner makes each worktree from local `main` (see memory: isolation worktrees branch from stale origin).
| WP | Model | Builds | Owns | Needs |
|---|---|---|---|---|
| A Contract | Opus | rename `atshosts` to `readers`; `_contract.py` with `Reads`, `Filters`, `Fields`, `Ask`, `Place`, `Row`, `problems()`; the extended docstring; a conformance test over every module (declaration against functions; `places="many"` fixtures) that marks today's gaps `xfail` per module | `plugins/readers/__init__.py`, `_contract.py`, `_shared.py`, every `import` of `atshosts`, `tests/test_readers_contract.py` | |
| B Runner | Opus | `_runner.py` out of `ats.py`; `Ask` resolution without reach (params only); series union; sweep; closing guards; `ats.py` and `search.py` thin | `plugins/ats.py`, `plugins/search.py`, `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_ats.py`, `tests/test_search.py`, `tests/test_pull.py` | A |
| C Providers | Sonnet | `jobindex.py`, `adzuna.py`, `platsbanken.py` as reader modules; `search_params` built from the declarations | the three modules, `app/jobs/sources.py`, their tests | A |
| D1 Hosts a to l | Sonnet | `READS` and `places` for Ashby to Lever; host-stated seniority; date trust into `fields` | those modules and tests | A |
| D2 Hosts o to w | Sonnet | same for Oracle to Workday | those modules and tests | A |
| E Places | Opus | `role_places`, migration, gates and Place filter over it, `dupes.py` fold of copies, `app/models.py` without `TRUSTED_POSTED_HOSTS` | `app/storage/*`, `app/models.py`, `app/dupes.py`, `app/places.py`, their tests | A |
| F Reach and filters | Opus | reach from agents into `Ask`; `facets` and `Ask` for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; readers' declarations shown on the Sources page | `app/jobs/reach.py`, those six modules, `app/web/api.py`, `app/static/companies.js`, `app/static/sources.js` | B, D1, D2 |
| G Detail stage | Sonnet | `detail` as its own stage by score; cascade order | `app/jobs/detail.py`, `app/jobs/cascade.py`, `app/jobs/stages.py`, tests | B |
A goes first. B, C, D1, D2 and E run in parallel. F and G follow B. The planner merges, removes each `xfail` a package fixed, attaches Infineon and Microsoft on Eightfold with a country filter, and updates `plugins/README.md`.