Plan readers: owner decisions on reach, robots.txt at build time, places and companies as symbols, teaser kept
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
10fdd60988
commit
af337ef558
1 file changed
+42
-21
+42
-21
@@ -35,7 +35,7 @@ The list below is what the 25 readers do today, plus what the wave-2 readers nee
|
||||
| Filters | per dimension (country, place, remote, query): `none`, `one` (one value per request), `many` (a list in one request), `ids` (values must be looked up on the host first) | country one: SmartRecruiters, Eightfold. Many: Phenom. Ids: Workday, Oracle, SuccessFactors. Query: the 3 providers. Greenhouse, Ashby, Recruitee: none |
|
||||
| Places | `one`, `many`, `copies` (see below) | |
|
||||
| **Also:** fields stated | pay (`figures`, `line`, `none`), posted (`trusted`, `listed`, `none`), remote, country, seniority, employment | seniority: Personio, SmartRecruiters, Recruitee, Jobylon, none mapped |
|
||||
| Access | key name (Adzuna), robots.txt page, a smaller fallback feed (Greenhouse) | |
|
||||
| Access | key name (Adzuna), a smaller fallback feed (Greenhouse), the robots.txt decision taken when the reader was built | |
|
||||
| Title | `final` or `provisional` (a sitemap reader knows a URL before a title) | none yet; WP10 of wave 2 needs it |
|
||||
| Empty answer | `trusted` or `suspect`: whether an unknown slug answers an empty board | suspect: Ashby, SmartRecruiters, Gem |
|
||||
|
||||
@@ -49,9 +49,20 @@ A role at "Berlin, Munich, Remote (DE)" arrives in three shapes today:
|
||||
- **All places joined in one line.** Teamtailor, Deel, Workday, Phenom, Oracle, Eightfold, Jibe join them with `"; "`. `country` becomes `EU` over several countries, which the country gate does not expand.
|
||||
- **One posting per place.** The host itself lists copies with their own ids (1,741 groups, 5,021 roles). `dupes.py` leaves them apart.
|
||||
|
||||
The rule: **one posting on the host is one row, and a row carries every place the host states.** A reader never multiplies a row per place and never drops a place. `Row.places` is a list of `Place(text, city, country, remote)`. The runner fills `city` and `country` from `text` via `app/places.py` if the reader leaves them empty. Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares `places="copies"`, and `dupes.py` folds them by company and title, which answers owner decision 10 of the review with yes.
|
||||
The rule (owner, 2026-09-30): **a role has a list of places.** A reader never multiplies a row per place and never drops a place. `Row.places` is a list of `Place(text, remote)`. Copies the host publishes per place are a host fact that a reader cannot undo. The reader declares `places="copies"`, and `dupes.py` folds them by company and title into one role, keeping each copy's place with its own outbound link.
|
||||
|
||||
Storage: `roles.place` stays the displayed line (the first place, "+2" when there are more), and a new table `role_places(role_id, place_fold, country)` holds all of them. The place and country gates and the Place filter match any row of it, so the `EU` fallback goes. The contract test (WP A) feeds every reader that declares `places="many"` a fixture with at least two places and fails the reader if fewer come out.
|
||||
The contract test (WP A) feeds every reader that declares `places="many"` a fixture with at least two places and fails the reader if fewer come out.
|
||||
|
||||
### Places and companies as symbols
|
||||
|
||||
Places and companies become rows that other rows link to, not text matched at query time. Today the place knowledge sits in code (`_CITIES`, `_REGIONS`, `_COUNTRY_NAMES` in `app/models.py`, regions and aliases in `app/places.py`) and is matched word by word against `roles.place_fold` for every query.
|
||||
|
||||
- `places(id, name, kind, parent_id)` with `kind` one of `city`, `region`, `country`, `continent`: Copenhagen belongs to Hovedstaden, which belongs to DK, which belongs to Europe. Seeded from the dictionaries in code.
|
||||
- `place_aliases(fold, place_id)`: `kobenhavn`, `kopenhamn`, `copenhagen` point to Copenhagen, and `skane` and `skåne` to Skåne. Typing either spelling finds the place.
|
||||
- `role_places(uid, place_id, text, url)`: one row per place of a role. `place_id` is empty for a text the gazetteer does not know. That text is added to an `unresolved` list, which a person or a later job resolves, and it is linked once resolved. `url` is the copy's own link when copies were folded.
|
||||
- A country filter or gate is a join over the tree: DK expands to every place under DK, and Europe to every country under Europe. The `EU` fallback and the per-request word matching go. This also enables counts per place.
|
||||
- `roles.place` stays the displayed line: the first place, then "+2" when there are more.
|
||||
- `company_aliases(fold, company_key)`: "Orsted" and "Ørsted" become one company without changing the primary key. Companies gain empty columns for enrichment (sector, stage, size band, founded, last funding, source and date of each fact), which the startupmap.one and The Hub readers fill later.
|
||||
|
||||
## The API
|
||||
|
||||
@@ -71,7 +82,7 @@ READS = Reads(
|
||||
filters=Filters(country="ids", remote="ids"),
|
||||
places="many",
|
||||
fields=Fields(posted="listed", pay="none", remote="stated"),
|
||||
robots=True,
|
||||
robots="read: robots.txt allows /wday/cxs/ (checked 2026-09-29)",
|
||||
empty="trusted",
|
||||
)
|
||||
|
||||
@@ -99,6 +110,8 @@ def fallback_url, robots_page, probe_url, probe_ok, probe_slugs
|
||||
|
||||
`TRUSTED_POSTED_HOSTS` goes: `fields.posted="trusted"` says the same thing in the module.
|
||||
|
||||
**robots.txt is decided when a reader is built, not checked at run time** (owner, 2026-09-30). Whoever builds a reader reads the host's robots.txt, reports what it disallows, and the owner decides per reader whether to read the host anyway. `robots` is a required string that records the decision and the date. The runtime check, its state keys and its tests go. Jobindex is read with paging and area. HR Manager, left out because its robots.txt disallows reading, goes back to the owner for a decision.
|
||||
|
||||
## The runner
|
||||
|
||||
`plugins/readers/_runner.py`, taken out of `plugins/ats.py`. `plugins/ats.py` and `plugins/search.py` stay as thin `Source` classes, so subscriptions and state keys (`ats:*`, `search:*`) keep their names and no migration is needed.
|
||||
@@ -106,34 +119,42 @@ def fallback_url, robots_page, probe_url, probe_ok, probe_slugs
|
||||
For one target, the runner:
|
||||
|
||||
1. Loads the reader and its subscription.
|
||||
2. Resolves `Ask`: the reach of the enabled agents (union of their `any` gates on country, town and remote; `all` if one agent has none), narrowed to what the reader declares, overridden by `params` on the board's subscription. A search's own params go in `Ask.params`. Boards under 300 roles get no filter.
|
||||
3. Checks robots.txt, for searches too, and applies the key.
|
||||
4. Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded `get` for every reader: host allow-list, https only, size cap, deadline, fallback feed.
|
||||
2. Resolves `Ask`. The reach is set in the front end by the agents that are switched on:
|
||||
- Base reach: the union of their `any` gates on country, town and remote, or everything when one agent has none.
|
||||
- Favourite companies (favourite = 2, which already skips the `any` gates): only the `require` gates, which today means Europe. Mistral or Databricks are read across Europe, so a role outside Denmark still shows up and can prompt a question to them.
|
||||
- Board override: `params` on the board's subscription.
|
||||
|
||||
The reach is then narrowed to the filters the reader declares. A search's own params go in `Ask.params`. Boards under 300 roles get no filter.
|
||||
3. Applies the key.
|
||||
4. Reads the listing: whole, paged, or one series per filter value, unioned by id. Through one guarded `get` for every reader: host allow-list, https only, size cap, deadline, fallback feed. A teaser in the list is kept (`Row.teaser`).
|
||||
5. **Sweeps across pulls.** A paged read that hits `MAX_PAGES` saves its cursor and the ids seen, and the next pull continues from there. A sweep that reaches the end is complete and closes what it did not list. A host ceiling (Workday 2,000) is not swept: the reader must filter below it or stays incomplete.
|
||||
6. Decides completeness from the declaration, and applies the guards on closing: a filtered read closes roles outside the filter, except roles with a vote, an application or a note. A filtered or `empty="suspect"` read that returns under 10% of the last count counts as failed.
|
||||
7. Records failures (`failing` since and reason, 21 days to close), 429 handling and waits, as today.
|
||||
|
||||
Per-role text leaves the pull. A new stage `detail` runs after `score` in the cascade (`pull, embed, score, detail, embed, score, ...`). It reads the text of the highest-scored roles without text first, 200 per board per run, and drops the vector of each role it fills so the next `embed` encodes it with the text. The feed ranks titles first and gets the text for the top roles.
|
||||
|
||||
## Decisions (defaults taken, change before WP B starts)
|
||||
## Decisions taken by the owner, 2026-09-30
|
||||
|
||||
1. **Where filters come from.** Default: derived at pull time from the enabled agents' gates, as `review-ingest.md` designed, with a per-board override on the Companies page. The alternative is a filter set by hand per board.
|
||||
2. **Jobindex under robots.txt.** Once searches go through the runner, the robots check fails Jobindex paging and area (review finding 21). Default: Jobindex declares `robots=True` and reads the first page of each search, 20 results. The alternatives are to exempt it or to drop it.
|
||||
3. **Copies per place folded.** Default: yes, as above.
|
||||
1. **Filters** come from the agents that are switched on. Favourite companies are read across the `require` reach (Europe), and a board can override its filter.
|
||||
2. **robots.txt** is decided per reader at build time. Nothing checks it at run time. Jobindex is read in full.
|
||||
3. **Places** are a list per role. Copies per place are folded, each keeping its own link. Places and companies become symbols with aliases.
|
||||
4. **A teaser is kept.** A short description in the list (Phenom and Oracle: 1,066 roles, plus Jobindex and Adzuna excerpts) goes into `Row.teaser` and `roles.teaser`. Until the full text arrives, it is shown and embedded, and a reader that sees one must hand it over. When the full text arrives, it replaces the teaser for embedding and the vector is dropped, as today.
|
||||
|
||||
## Work packages
|
||||
|
||||
Each package owns the files listed and their tests. The planner makes each worktree from local `main` (see memory: isolation worktrees branch from stale origin).
|
||||
Each package owns the files listed and their tests. The planner makes each worktree from local `main` (see memory: isolation worktrees branch from stale origin), merges, and removes the `xfail` marks a package fixed.
|
||||
|
||||
| WP | Model | Builds | Owns | Needs |
|
||||
|---|---|---|---|---|
|
||||
| A Contract | Opus | rename `atshosts` to `readers`; `_contract.py` with `Reads`, `Filters`, `Fields`, `Ask`, `Place`, `Row`, `problems()`; the extended docstring; a conformance test over every module (declaration against functions; `places="many"` fixtures) that marks today's gaps `xfail` per module | `plugins/readers/__init__.py`, `_contract.py`, `_shared.py`, every `import` of `atshosts`, `tests/test_readers_contract.py` | |
|
||||
| B Runner | Opus | `_runner.py` out of `ats.py`; `Ask` resolution without reach (params only); series union; sweep; closing guards; `ats.py` and `search.py` thin | `plugins/ats.py`, `plugins/search.py`, `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_ats.py`, `tests/test_search.py`, `tests/test_pull.py` | A |
|
||||
| C Providers | Sonnet | `jobindex.py`, `adzuna.py`, `platsbanken.py` as reader modules; `search_params` built from the declarations | the three modules, `app/jobs/sources.py`, their tests | A |
|
||||
| D1 Hosts a to l | Sonnet | `READS` and `places` for Ashby to Lever; host-stated seniority; date trust into `fields` | those modules and tests | A |
|
||||
| D2 Hosts o to w | Sonnet | same for Oracle to Workday | those modules and tests | A |
|
||||
| E Places | Opus | `role_places`, migration, gates and Place filter over it, `dupes.py` fold of copies, `app/models.py` without `TRUSTED_POSTED_HOSTS` | `app/storage/*`, `app/models.py`, `app/dupes.py`, `app/places.py`, their tests | A |
|
||||
| F Reach and filters | Opus | reach from agents into `Ask`; `facets` and `Ask` for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; readers' declarations shown on the Sources page | `app/jobs/reach.py`, those six modules, `app/web/api.py`, `app/static/companies.js`, `app/static/sources.js` | B, D1, D2 |
|
||||
| G Detail stage | Sonnet | `detail` as its own stage by score; cascade order | `app/jobs/detail.py`, `app/jobs/cascade.py`, `app/jobs/stages.py`, tests | B |
|
||||
| A Contract | Opus | rename `atshosts` to `readers`; `_contract.py` with `Reads`, `Filters`, `Fields`, `Ask`, `Place`, `Row` (with `places`, `teaser`), `problems()`; the extended docstring with the robots.txt rule; a conformance test over every module (declaration against functions, `places="many"` and teaser fixtures) marking today's gaps `xfail` per module | `plugins/readers/__init__.py`, `_contract.py`, `_shared.py`, every `import` of `atshosts`, `tests/test_readers_contract.py` | |
|
||||
| B Runner | Opus | `_runner.py` out of `ats.py`; runtime robots.txt check removed; `Ask` from params; series union; sweep; closing guards; `ats.py` and `search.py` thin; `raw` built from every stated `Row` key | `plugins/ats.py`, `plugins/search.py`, `plugins/readers/_runner.py`, `app/jobs/pull.py`, `tests/test_ats.py`, `tests/test_search.py`, `tests/test_pull.py` | A |
|
||||
| C Providers | Sonnet | `jobindex.py`, `adzuna.py`, `platsbanken.py` as reader modules with teaser and places; `search_params` built from the declarations | the three modules, `app/jobs/sources.py`, their tests | A |
|
||||
| D1 Hosts a to l | Sonnet | `READS`, all places, teaser, host-stated seniority, date trust in `fields`, robots decision string for Ashby to Lever | those modules and tests | A |
|
||||
| D2 Hosts o to w | Sonnet | the same for Oracle to Workday | those modules and tests | A |
|
||||
| E1 Places and teaser storage | Opus | `places`, `place_aliases`, `role_places`, `roles.teaser`, migration seeded from the dictionaries in code; `upsert_role` writes them; gates and the Place filter as joins over the tree; `TRUSTED_POSTED_HOSTS` removed; embed falls back to the teaser | `app/storage/*`, `app/models.py`, `app/places.py`, `app/jobs/embed.py`, their tests | A |
|
||||
| F Reach and filters | Opus | reach from the agents and favourites into `Ask`; `facets` and `Ask` for SmartRecruiters, Eightfold, Workday, SuccessFactors, Phenom, Oracle; board override on the Companies page; each reader's declaration shown on the Sources page | `app/jobs/reach.py`, those six modules, `app/web/api.py`, `app/static/companies.js`, `app/static/sources.js` | B, D1, D2, E1 |
|
||||
| G Detail stage | Sonnet | `detail` as its own stage, ordered by score; cascade order | `app/jobs/detail.py`, `app/jobs/cascade.py`, `app/jobs/stages.py`, their tests | B |
|
||||
| E2 Copies and place list | Sonnet | `dupes.py` folds copies per place into one role with per-place links; the reader shows the place list with links; the teaser is shown when there is no text | `app/dupes.py`, `app/static/roles.js`, `app/templates/*`, their tests | E1 |
|
||||
| H Companies as symbols | Sonnet | `company_aliases`, alias folding on insert and lookup, empty enrichment columns | `app/storage/*` after E1, `app/jobs/sources.py` after C, their tests | E1, C |
|
||||
|
||||
A goes first. B, C, D1, D2 and E run in parallel. F and G follow B. The planner merges, removes each `xfail` a package fixed, attaches Infineon and Microsoft on Eightfold with a country filter, and updates `plugins/README.md`.
|
||||
Order: A first. Then B, C, D1, D2 and E1 in parallel. Then F, G, E2 and H in parallel. After the merges, the planner attaches Infineon and Microsoft on Eightfold with a country filter, updates `plugins/README.md`, and folds this plan into `decisions.md`.
|
||||
Reference in new issue
Block a user