From 10fdd60988f7c177f6125ce6bc1f0301c93c85f8 Mon Sep 17 00:00:00 2001 From: Wolf Date: Wed, 30 Sep 2026 08:34:49 +0200 Subject: [PATCH] Plan readers as plugins: declared traits, one API, one runner, one row per posting with all its places Co-Authored-By: Claude Opus 5.5 --- docs/plan-hosts-wave-2.md | 2 + docs/plan-readers.md | 139 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) create mode 100644 docs/plan-readers.md diff --git a/docs/plan-hosts-wave-2.md b/docs/plan-hosts-wave-2.md index eff84e9..49f0c2c 100644 --- a/docs/plan-hosts-wave-2.md +++ b/docs/plan-hosts-wave-2.md @@ -2,6 +2,8 @@ Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped. +WP9 (the contract part) and WP11 are superseded by `docs/plan-readers.md`; WP10 and WP12 to WP15 build against its contract once its WP A has landed. + ## Where it stands 286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet. diff --git a/docs/plan-readers.md b/docs/plan-readers.md new file mode 100644 index 0000000..21fe48e --- /dev/null +++ b/docs/plan-readers.md @@ -0,0 +1,139 @@ +# Plan: readers as plugins with a declared contract + +Planned 2026-09-30. Turns every source of roles (the 22 ATS hosts, the 3 search providers, the company-specific readers of `plan-hosts-wave-2.md`) into a reader module with one documented API, and one runner that reads what each reader declares and handles paging, filters, per-role text, limits and failures for all of them. Pattern taken from `../news` (`desk/sources/spec.py`, `docs/plugin-rework.md`): the plugin declares, the application does. Folded into `decisions.md` and deleted when the packages land. + +This plan supersedes WP9 (provisional title, as far as the contract goes) and WP11 (board filters) of `plan-hosts-wave-2.md`, and covers items 2, 7, 17 and the "Plugin contract" and "Filtering at the source" sections of `review-ingest.md`. + +## How far we are + +About two thirds of the way for boards, none of the way for searches. + +| Part | Today | Gap | +|---|---|---| +| One module per host | yes, 22 in `plugins/atshosts/`, checked at import by `problems()` | none | +| Documented contract | docstring of `plugins/atshosts/__init__.py` | covers hosts only | +| Capabilities | inferred from which functions exist (`page` means paged, `detail` means per-role text) | not declared; nothing about filters, caps, fields, date trust, places | +| Runner | `plugins/ats.py`: guarded `get`, robots.txt, paging, per-role reads, failure state | fused with the `ats` Source; search does not use it | +| Search providers | three functions in `plugins/search.py` with their own getter | no robots.txt, no size cap, no host allow-list, not modules | +| Row | a bare dict with fixed keys; `ats.py` builds `raw` from a fixed list | a host cannot hand over country, seniority, employment type, several places | +| Date trust | `TRUSTED_POSTED_HOSTS` in `app/models.py` | a host fact kept outside the host | +| Filters at the source | none for boards; searches have per-provider params built in `app/jobs/sources.py:search_params` | no way for a reader to say what it filters, nor for the runner to pass it | +| Oversized boards | read incomplete on every pull (`MAX_PAGES` 100) | no reading across pulls | +| Per-role text | 200 per board per pull, never-tried first, inside the pull | not ordered by score | + +## What a reader must announce + +The list below is what the 25 readers do today, plus what the wave-2 readers need. The user's list (whole board, paged, list then text, location filter) is complete for the listing. It misses the traits under "also". + +| Trait | Values | Who, today | +|---|---|---| +| Scope | `board` (one company, its whole list) or `search` (many companies, never a whole list) | 22 hosts, 3 providers | +| Listing | `whole` (one request) or `paged` | whole: Greenhouse, Lever, Ashby, Recruitee, Personio, Deel, Pinpoint, BambooHR, Jobylon, Rippling. Paged: the other 12 and the 3 providers | +| Text | `listing` (full text in the list), `excerpt` (a teaser in the list), `detail` (one request per role), `none` | detail: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Gem, Jobylon, BambooHR, Rippling. Excerpt: Jobindex, Adzuna. None: SmartRecruiters, Workable | +| Completeness | `stated` (the host gives a total), `end` (an empty page ends it), `whole` (one answer is all), `never` (a search result is never a whole list) | | +| Page size and host ceiling | integers; the ceiling is where the host stops answering whatever we ask | Workday 2,000; Platsbanken offset 2,000; Jobindex 200; Eightfold 10 a page | +| Filters | per dimension (country, place, remote, query): `none`, `one` (one value per request), `many` (a list in one request), `ids` (values must be looked up on the host first) | country one: SmartRecruiters, Eightfold. Many: Phenom. Ids: Workday, Oracle, SuccessFactors. Query: the 3 providers. Greenhouse, Ashby, Recruitee: none | +| Places | `one`, `many`, `copies` (see below) | | +| **Also:** fields stated | pay (`figures`, `line`, `none`), posted (`trusted`, `listed`, `none`), remote, country, seniority, employment | seniority: Personio, SmartRecruiters, Recruitee, Jobylon, none mapped | +| Access | key name (Adzuna), robots.txt page, a smaller fallback feed (Greenhouse) | | +| Title | `final` or `provisional` (a sitemap reader knows a URL before a title) | none yet; WP10 of wave 2 needs it | +| Empty answer | `trusted` or `suspect`: whether an unknown slug answers an empty board | suspect: Ashby, SmartRecruiters, Gem | + +Rate limits (429) and the per-request limits stay runner rules, the same for every reader. No reader has a need that differs. + +## Several places for one role + +A role at "Berlin, Munich, Remote (DE)" arrives in three shapes today: + +- **One place of several kept.** Ashby, Lever, Personio, Greenhouse, Workable, Recruitee, Platsbanken read the first and drop the rest. SuccessFactors keeps the last. +- **All places joined in one line.** Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold, Jibe join them with `"; "`. `country` becomes `EU` over several countries, which the country gate does not expand. +- **One posting per place.** The host itself lists copies with their own ids (1,741 groups, 5,021 roles). `dupes.py` leaves them apart. + +The rule: **one posting on the host is one row, and a row carries every place the host states.** A reader never multiplies a row per place and never drops a place. `Row.places` is a list of `Place(text, city, country, remote)`. The runner fills `city` and `country` from `text` via `app/places.py` if the reader leaves them empty. Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares `places="copies"`, and `dupes.py` folds them by company and title, which answers owner decision 10 of the review with yes. + +Storage: `roles.place` stays the displayed line (the first place, "+2" when there are more), and a new table `role_places(role_id, place_fold, country)` holds all of them. The place and country gates and the Place filter match any row of it, so the `EU` fallback goes. The contract test (WP A) feeds every reader that declares `places="many"` a fixture with at least two places and fails the reader if fewer come out. + +## The API + +The package `plugins/atshosts/` becomes `plugins/readers/`, and the search providers move into it. The contract stays the package docstring, extended, with `plugins/readers/_contract.py` holding the types and `problems()`. `plugins/README.md` stays the list of sources and their measured limits. + +```python +# plugins/readers/workday.py +NAME = "workday" +LABEL = "Workday" +PATTERNS = (r"...",) + +READS = Reads( + scope="board", + listing="paged", page_size=20, ceiling=2000, + text="detail", + complete="stated", + filters=Filters(country="ids", remote="ids"), + places="many", + fields=Fields(posted="listed", pay="none", remote="stated"), + robots=True, + empty="trusted", +) + +def slug_ok(slug) -> bool +def allowed(slug) -> Callable[[str], bool] +def careers_url(slug) -> str + +# listing="whole" +def feed_url(slug, ask: Ask) -> str +def parse(body, slug) -> list[Row] +# listing="paged" +def page(get, slug, cursor, ask: Ask) -> Page +def rows(decoded, slug) -> list[Row] +# text="detail" +def detail(get, slug, row) -> dict +# filters with "ids" +def facets(get, slug) -> dict[str, dict[str, str]] # {"country": {"DE": ""}} +# optional, as today +def fallback_url, robots_page, probe_url, probe_ok, probe_slugs +``` + +`Ask` is what the runner wants from one series of requests: `countries`, `places`, `remote`, `query`, `params` (a search's saved params). A reader reads only the dimensions it declared, and the runner never hands it a value it did not declare. With `one`, the runner makes one series per value and unions them by id. With `ids`, it calls `facets` once a day per board, keeps the result in state and puts host ids into `Ask`. `Row` becomes a `TypedDict` with `places`, and optional `country`, `seniority`, `employment`. The runner puts every stated key into `raw`, so a new field needs no runner change. + +`problems()` checks the declaration against the functions: `paged` needs `page` and `rows`, `text="detail"` needs `detail`, an `ids` filter needs `facets`, `scope="search"` needs `complete="never"` or `stated`, `key` needs a config name. A module that fails refuses to import, as today. + +`TRUSTED_POSTED_HOSTS` goes: `fields.posted="trusted"` says the same thing in the module. + +## The runner + +`plugins/readers/_runner.py`, taken out of `plugins/ats.py`. `plugins/ats.py` and `plugins/search.py` stay as thin `Source` classes, so subscriptions and state keys (`ats:*`, `search:*`) keep their names and no migration is needed. + +For one target, the runner: + +1. Loads the reader and its subscription. +2. Resolves `Ask`: the reach of the enabled agents (union of their `any` gates on country, town and remote; `all` if one agent has none), narrowed to what the reader declares, overridden by `params` on the board's subscription. A search's own params go in `Ask.params`. Boards under 300 roles get no filter. +3. Checks robots.txt, for searches too, and applies the key. +4. Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded `get` for every reader: host allow-list, https only, size cap, deadline, fallback feed. +5. **Sweeps across pulls.** A paged read that hits `MAX_PAGES` saves its cursor and the ids seen, and the next pull continues from there. A sweep that reaches the end is complete and closes what it did not list. A host ceiling (Workday 2,000) is not swept: the reader must filter below it or stays incomplete. +6. Decides completeness from the declaration, and applies the guards on closing: a filtered read closes roles outside the filter, except roles with a vote, an application or a note. A filtered or `empty="suspect"` read that returns under 10% of the last count counts as failed. +7. Records failures (`failing` since and reason, 21 days to close), 429 handling and waits, as today. + +Per-role text leaves the pull. A new stage `detail` runs after `score` in the cascade (`pull, embed, score, detail, embed, score, ...`). It reads the text of the highest-scored roles without text first, 200 per board per run, and drops the vector of each role it fills so the next `embed` encodes it with the text. The feed ranks titles first and gets the text for the top roles. + +## Decisions (defaults taken, change before WP B starts) + +1. **Where filters come from.** Default: derived at pull time from the enabled agents' gates, as `review-ingest.md` designed, with a per-board override on the Companies page. The alternative is a filter set by hand per board. +2. **Jobindex under robots.txt.** Once searches go through the runner, the robots check fails Jobindex paging and area (review finding 21). Default: Jobindex declares `robots=True` and reads the first page of each search, 20 results. The alternatives are to exempt it or to drop it. +3. **Copies per place folded.** Default: yes, as above. + +## Work packages + +Each package owns the files listed and their tests. The planner makes each worktree from local `main` (see memory: isolation worktrees branch from stale origin). + +| WP | Model | Builds | Owns | Needs | +|---|---|---|---|---| +| A Contract | Opus | rename `atshosts` to `readers`; `_contract.py` with `Reads`, `Filters`, `Fields`, `Ask`, `Place`, `Row`, `problems()`; the extended docstring; a conformance test over every module (declaration against functions; `places="many"` fixtures) that marks today's gaps `xfail` per module | `plugins/readers/__init__.py`, `_contract.py`, `_shared.py`, every `import` of `atshosts`, `tests/test_readers_contract.py` | | +| B Runner | Opus | `_runner.py` out of `ats.py`; `Ask` resolution without reach (params only); series union; sweep; closing guards; `ats.py` and `search.py` thin | `plugins/ats.py`, `plugins/search.py`, `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_ats.py`, `tests/test_search.py`, `tests/test_pull.py` | A | +| C Providers | Sonnet | `jobindex.py`, `adzuna.py`, `platsbanken.py` as reader modules; `search_params` built from the declarations | the three modules, `app/jobs/sources.py`, their tests | A | +| D1 Hosts a to l | Sonnet | `READS` and `places` for Ashby to Lever; host-stated seniority; date trust into `fields` | those modules and tests | A | +| D2 Hosts o to w | Sonnet | same for Oracle to Workday | those modules and tests | A | +| E Places | Opus | `role_places`, migration, gates and Place filter over it, `dupes.py` fold of copies, `app/models.py` without `TRUSTED_POSTED_HOSTS` | `app/storage/*`, `app/models.py`, `app/dupes.py`, `app/places.py`, their tests | A | +| F Reach and filters | Opus | reach from agents into `Ask`; `facets` and `Ask` for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; readers' declarations shown on the Sources page | `app/jobs/reach.py`, those six modules, `app/web/api.py`, `app/static/companies.js`, `app/static/sources.js` | B, D1, D2 | +| G Detail stage | Sonnet | `detail` as its own stage by score; cascade order | `app/jobs/detail.py`, `app/jobs/cascade.py`, `app/jobs/stages.py`, tests | B | + +A goes first. B, C, D1, D2 and E run in parallel. F and G follow B. The planner merges, removes each `xfail` a package fixed, attaches Infineon and Microsoft on Eightfold with a country filter, and updates `plugins/README.md`.