Plan readers as plugins: declared traits, one API, one runner, one row per posting with all its places
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
667e1d35f4
commit
10fdd60988
2 files changed
+141
No files matched your search
@@ -2,6 +2,8 @@
|
||||
|
||||
Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped.
|
||||
|
||||
WP9 (the contract part) and WP11 are superseded by `docs/plan-readers.md`; WP10 and WP12 to WP15 build against its contract once its WP A has landed.
|
||||
|
||||
## Where it stands
|
||||
|
||||
286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
# Plan: readers as plugins with a declared contract
|
||||
|
||||
Planned 2026-09-30. Turns every source of roles (the 22 ATS hosts, the 3 search providers, the company-specific readers of `plan-hosts-wave-2.md`) into a reader module with one documented API, and one runner that reads what each reader declares and handles paging, filters, per-role text, limits and failures for all of them. Pattern taken from `../news` (`desk/sources/spec.py`, `docs/plugin-rework.md`): the plugin declares, the application does. Folded into `decisions.md` and deleted when the packages land.
|
||||
|
||||
This plan supersedes WP9 (provisional title, as far as the contract goes) and WP11 (board filters) of `plan-hosts-wave-2.md`, and covers items 2, 7, 17 and the "Plugin contract" and "Filtering at the source" sections of `review-ingest.md`.
|
||||
|
||||
## How far we are
|
||||
|
||||
About two thirds of the way for boards, none of the way for searches.
|
||||
|
||||
| Part | Today | Gap |
|
||||
|---|---|---|
|
||||
| One module per host | yes, 22 in `plugins/atshosts/`, checked at import by `problems()` | none |
|
||||
| Documented contract | docstring of `plugins/atshosts/__init__.py` | covers hosts only |
|
||||
| Capabilities | inferred from which functions exist (`page` means paged, `detail` means per-role text) | not declared; nothing about filters, caps, fields, date trust, places |
|
||||
| Runner | `plugins/ats.py`: guarded `get`, robots.txt, paging, per-role reads, failure state | fused with the `ats` Source; search does not use it |
|
||||
| Search providers | three functions in `plugins/search.py` with their own getter | no robots.txt, no size cap, no host allow-list, not modules |
|
||||
| Row | a bare dict with fixed keys; `ats.py` builds `raw` from a fixed list | a host cannot hand over country, seniority, employment type, several places |
|
||||
| Date trust | `TRUSTED_POSTED_HOSTS` in `app/models.py` | a host fact kept outside the host |
|
||||
| Filters at the source | none for boards; searches have per-provider params built in `app/jobs/sources.py:search_params` | no way for a reader to say what it filters, nor for the runner to pass it |
|
||||
| Oversized boards | read incomplete on every pull (`MAX_PAGES` 100) | no reading across pulls |
|
||||
| Per-role text | 200 per board per pull, never-tried first, inside the pull | not ordered by score |
|
||||
|
||||
## What a reader must announce
|
||||
|
||||
The list below is what the 25 readers do today, plus what the wave-2 readers need. The user's list (whole board, paged, list then text, location filter) is complete for the listing. It misses the traits under "also".
|
||||
|
||||
| Trait | Values | Who, today |
|
||||
|---|---|---|
|
||||
| Scope | `board` (one company, its whole list) or `search` (many companies, never a whole list) | 22 hosts, 3 providers |
|
||||
| Listing | `whole` (one request) or `paged` | whole: Greenhouse, Lever, Ashby, Recruitee, Personio, Deel, Pinpoint, BambooHR, Jobylon, Rippling. Paged: the other 12 and the 3 providers |
|
||||
| Text | `listing` (full text in the list), `excerpt` (a teaser in the list), `detail` (one request per role), `none` | detail: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Gem, Jobylon, BambooHR, Rippling. Excerpt: Jobindex, Adzuna. None: SmartRecruiters, Workable |
|
||||
| Completeness | `stated` (the host gives a total), `end` (an empty page ends it), `whole` (one answer is all), `never` (a search result is never a whole list) | |
|
||||
| Page size and host ceiling | integers; the ceiling is where the host stops answering whatever we ask | Workday 2,000; Platsbanken offset 2,000; Jobindex 200; Eightfold 10 a page |
|
||||
| Filters | per dimension (country, place, remote, query): `none`, `one` (one value per request), `many` (a list in one request), `ids` (values must be looked up on the host first) | country one: SmartRecruiters, Eightfold. Many: Phenom. Ids: Workday, Oracle, SuccessFactors. Query: the 3 providers. Greenhouse, Ashby, Recruitee: none |
|
||||
| Places | `one`, `many`, `copies` (see below) | |
|
||||
| **Also:** fields stated | pay (`figures`, `line`, `none`), posted (`trusted`, `listed`, `none`), remote, country, seniority, employment | seniority: Personio, SmartRecruiters, Recruitee, Jobylon, none mapped |
|
||||
| Access | key name (Adzuna), robots.txt page, a smaller fallback feed (Greenhouse) | |
|
||||
| Title | `final` or `provisional` (a sitemap reader knows a URL before a title) | none yet; WP10 of wave 2 needs it |
|
||||
| Empty answer | `trusted` or `suspect`: whether an unknown slug answers an empty board | suspect: Ashby, SmartRecruiters, Gem |
|
||||
|
||||
Rate limits (429) and the per-request limits stay runner rules, the same for every reader. No reader has a need that differs.
|
||||
|
||||
## Several places for one role
|
||||
|
||||
A role at "Berlin, Munich, Remote (DE)" arrives in three shapes today:
|
||||
|
||||
- **One place of several kept.** Ashby, Lever, Personio, Greenhouse, Workable, Recruitee, Platsbanken read the first and drop the rest. SuccessFactors keeps the last.
|
||||
- **All places joined in one line.** Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold, Jibe join them with `"; "`. `country` becomes `EU` over several countries, which the country gate does not expand.
|
||||
- **One posting per place.** The host itself lists copies with their own ids (1,741 groups, 5,021 roles). `dupes.py` leaves them apart.
|
||||
|
||||
The rule: **one posting on the host is one row, and a row carries every place the host states.** A reader never multiplies a row per place and never drops a place. `Row.places` is a list of `Place(text, city, country, remote)`. The runner fills `city` and `country` from `text` via `app/places.py` if the reader leaves them empty. Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares `places="copies"`, and `dupes.py` folds them by company and title, which answers owner decision 10 of the review with yes.
|
||||
|
||||
Storage: `roles.place` stays the displayed line (the first place, "+2" when there are more), and a new table `role_places(role_id, place_fold, country)` holds all of them. The place and country gates and the Place filter match any row of it, so the `EU` fallback goes. The contract test (WP A) feeds every reader that declares `places="many"` a fixture with at least two places and fails the reader if fewer come out.
|
||||
|
||||
## The API
|
||||
|
||||
The package `plugins/atshosts/` becomes `plugins/readers/`, and the search providers move into it. The contract stays the package docstring, extended, with `plugins/readers/_contract.py` holding the types and `problems()`. `plugins/README.md` stays the list of sources and their measured limits.
|
||||
|
||||
```python
|
||||
# plugins/readers/workday.py
|
||||
NAME = "workday"
|
||||
LABEL = "Workday"
|
||||
PATTERNS = (r"...",)
|
||||
|
||||
READS = Reads(
|
||||
scope="board",
|
||||
listing="paged", page_size=20, ceiling=2000,
|
||||
text="detail",
|
||||
complete="stated",
|
||||
filters=Filters(country="ids", remote="ids"),
|
||||
places="many",
|
||||
fields=Fields(posted="listed", pay="none", remote="stated"),
|
||||
robots=True,
|
||||
empty="trusted",
|
||||
)
|
||||
|
||||
def slug_ok(slug) -> bool
|
||||
def allowed(slug) -> Callable[[str], bool]
|
||||
def careers_url(slug) -> str
|
||||
|
||||
# listing="whole"
|
||||
def feed_url(slug, ask: Ask) -> str
|
||||
def parse(body, slug) -> list[Row]
|
||||
# listing="paged"
|
||||
def page(get, slug, cursor, ask: Ask) -> Page
|
||||
def rows(decoded, slug) -> list[Row]
|
||||
# text="detail"
|
||||
def detail(get, slug, row) -> dict
|
||||
# filters with "ids"
|
||||
def facets(get, slug) -> dict[str, dict[str, str]] # {"country": {"DE": "<host id>"}}
|
||||
# optional, as today
|
||||
def fallback_url, robots_page, probe_url, probe_ok, probe_slugs
|
||||
```
|
||||
|
||||
`Ask` is what the runner wants from one series of requests: `countries`, `places`, `remote`, `query`, `params` (a search's saved params). A reader reads only the dimensions it declared, and the runner never hands it a value it did not declare. With `one`, the runner makes one series per value and unions them by id. With `ids`, it calls `facets` once a day per board, keeps the result in state and puts host ids into `Ask`. `Row` becomes a `TypedDict` with `places`, and optional `country`, `seniority`, `employment`. The runner puts every stated key into `raw`, so a new field needs no runner change.
|
||||
|
||||
`problems()` checks the declaration against the functions: `paged` needs `page` and `rows`, `text="detail"` needs `detail`, an `ids` filter needs `facets`, `scope="search"` needs `complete="never"` or `stated`, `key` needs a config name. A module that fails refuses to import, as today.
|
||||
|
||||
`TRUSTED_POSTED_HOSTS` goes: `fields.posted="trusted"` says the same thing in the module.
|
||||
|
||||
## The runner
|
||||
|
||||
`plugins/readers/_runner.py`, taken out of `plugins/ats.py`. `plugins/ats.py` and `plugins/search.py` stay as thin `Source` classes, so subscriptions and state keys (`ats:*`, `search:*`) keep their names and no migration is needed.
|
||||
|
||||
For one target, the runner:
|
||||
|
||||
1. Loads the reader and its subscription.
|
||||
2. Resolves `Ask`: the reach of the enabled agents (union of their `any` gates on country, town and remote; `all` if one agent has none), narrowed to what the reader declares, overridden by `params` on the board's subscription. A search's own params go in `Ask.params`. Boards under 300 roles get no filter.
|
||||
3. Checks robots.txt, for searches too, and applies the key.
|
||||
4. Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded `get` for every reader: host allow-list, https only, size cap, deadline, fallback feed.
|
||||
5. **Sweeps across pulls.** A paged read that hits `MAX_PAGES` saves its cursor and the ids seen, and the next pull continues from there. A sweep that reaches the end is complete and closes what it did not list. A host ceiling (Workday 2,000) is not swept: the reader must filter below it or stays incomplete.
|
||||
6. Decides completeness from the declaration, and applies the guards on closing: a filtered read closes roles outside the filter, except roles with a vote, an application or a note. A filtered or `empty="suspect"` read that returns under 10% of the last count counts as failed.
|
||||
7. Records failures (`failing` since and reason, 21 days to close), 429 handling and waits, as today.
|
||||
|
||||
Per-role text leaves the pull. A new stage `detail` runs after `score` in the cascade (`pull, embed, score, detail, embed, score, ...`). It reads the text of the highest-scored roles without text first, 200 per board per run, and drops the vector of each role it fills so the next `embed` encodes it with the text. The feed ranks titles first and gets the text for the top roles.
|
||||
|
||||
## Decisions (defaults taken, change before WP B starts)
|
||||
|
||||
1. **Where filters come from.** Default: derived at pull time from the enabled agents' gates, as `review-ingest.md` designed, with a per-board override on the Companies page. The alternative is a filter set by hand per board.
|
||||
2. **Jobindex under robots.txt.** Once searches go through the runner, the robots check fails Jobindex paging and area (review finding 21). Default: Jobindex declares `robots=True` and reads the first page of each search, 20 results. The alternatives are to exempt it or to drop it.
|
||||
3. **Copies per place folded.** Default: yes, as above.
|
||||
|
||||
## Work packages
|
||||
|
||||
Each package owns the files listed and their tests. The planner makes each worktree from local `main` (see memory: isolation worktrees branch from stale origin).
|
||||
|
||||
| WP | Model | Builds | Owns | Needs |
|
||||
|---|---|---|---|---|
|
||||
| A Contract | Opus | rename `atshosts` to `readers`; `_contract.py` with `Reads`, `Filters`, `Fields`, `Ask`, `Place`, `Row`, `problems()`; the extended docstring; a conformance test over every module (declaration against functions; `places="many"` fixtures) that marks today's gaps `xfail` per module | `plugins/readers/__init__.py`, `_contract.py`, `_shared.py`, every `import` of `atshosts`, `tests/test_readers_contract.py` | |
|
||||
| B Runner | Opus | `_runner.py` out of `ats.py`; `Ask` resolution without reach (params only); series union; sweep; closing guards; `ats.py` and `search.py` thin | `plugins/ats.py`, `plugins/search.py`, `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_ats.py`, `tests/test_search.py`, `tests/test_pull.py` | A |
|
||||
| C Providers | Sonnet | `jobindex.py`, `adzuna.py`, `platsbanken.py` as reader modules; `search_params` built from the declarations | the three modules, `app/jobs/sources.py`, their tests | A |
|
||||
| D1 Hosts a to l | Sonnet | `READS` and `places` for Ashby to Lever; host-stated seniority; date trust into `fields` | those modules and tests | A |
|
||||
| D2 Hosts o to w | Sonnet | same for Oracle to Workday | those modules and tests | A |
|
||||
| E Places | Opus | `role_places`, migration, gates and Place filter over it, `dupes.py` fold of copies, `app/models.py` without `TRUSTED_POSTED_HOSTS` | `app/storage/*`, `app/models.py`, `app/dupes.py`, `app/places.py`, their tests | A |
|
||||
| F Reach and filters | Opus | reach from agents into `Ask`; `facets` and `Ask` for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; readers' declarations shown on the Sources page | `app/jobs/reach.py`, those six modules, `app/web/api.py`, `app/static/companies.js`, `app/static/sources.js` | B, D1, D2 |
|
||||
| G Detail stage | Sonnet | `detail` as its own stage by score; cascade order | `app/jobs/detail.py`, `app/jobs/cascade.py`, `app/jobs/stages.py`, tests | B |
|
||||
|
||||
A goes first. B, C, D1, D2 and E run in parallel. F and G follow B. The planner merges, removes each `xfail` a package fixed, attaches Infineon and Microsoft on Eightfold with a country filter, and updates `plugins/README.md`.
|
||||
Reference in new issue
Block a user