78 lines
17 KiB
Markdown
78 lines
17 KiB
Markdown
# Plan: the review of 2026-09-30, before new features
|
|
|
|
Written 2026-09-30 against main at `d47f4f5`, from five read-only reviews of the code, the docs and the live database: silent failures in ingestion, the reader contract, docs against code, settings against the pages, and a bug hunt. What is built here is folded into `decisions.md`, the rest goes to `open.md`, and this file is deleted.
|
|
|
|
`docs/plan-remote.md` is being built at the same time in `../jobboard-wt-remote-a` and `-b`, and its packages own `app/jobs/`, `app/storage/`, `app/web/api.py`, `plugins/` and `app/static/`. The packages below own single files inside those directories and keep their hunks small; the planner merges whichever lands second and resolves. The shared checkout also carries uncommitted edits to `score.py`, `models.py`, `store.py`, `agents.js`, `app.css`, `agents.html` and `test_score.py`; no package here touches those.
|
|
|
|
## What the reviews found
|
|
|
|
The test suite passes (2,923 tests, 20 skipped for missing fixtures), ruff is clean, mypy reports 15 typing errors, git is clean of stray branches. The pipeline as designed holds: a reader that raises is recorded per board and shown on its row from day one, a crashed batch closes only the boards it stamped, one pull runs at a time, two strikes protect swept boards, the reach is never empty. The problems are at the edges of that design.
|
|
|
|
**Roles are lost without a trace in four ways.**
|
|
|
|
1. **A host that cannot filter on remote drops the remote dimension silently.** `ask_for` keeps only the dimensions the reader declares (`_runner.py:374`), so on SmartRecruiters, Phenom, SuccessFactors and Eightfold the agents' reach "DK, SE, remote" becomes "DK, SE", the read counts as filtered, and `_close_unlisted` closes every remote role outside DK and SE. The Companies line says "follows agents: DK, SE" and nothing about remote. On 2026-09-29 and 30 this closed 72 remote roles inside the agent's Europe rule (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3), and new remote roles on those 21 boards are never fetched.
|
|
2. **206 of 309 boards have no guard against an empty answer.** The 10% guard runs only for filtered, narrowed or `empty="suspect"` reads (`_runner.py:674`). A Greenhouse or Lever answer whose shape changed parses to zero rows with HTTP 200, is stamped complete, and the settle closes the whole board: 15,635 open roles sit on unguarded boards.
|
|
3. **A crash after the read starves the rest of the batch, every pull.** `ats._board` removes the failing key before the rows are stored (`ats.py:410`), and `item_of`, `upsert_role` and `stamp` run outside the try, with no per-target guard in `run_batch` (`pull.py:198`). A bad row fails the whole batch job: that board has neither a stamp nor a failing record, the boards after it in the batch are not read, and batches are stable, so the same neighbours are skipped on every pull, never close, never age, never show "not answering".
|
|
4. **A search that fails leaves no trace, and searches have no guard.** `search.py:140` logs and continues; there is no `search:failing` key and the search row has no failure field. Searches call `read_listing`, not `read`, so a Platsbanken answer with `hits` renamed reads as complete and empty and closes every role of the search at once.
|
|
|
|
**What the pull does is not shown.** The settle's counts (boards, unanswered, incomplete, closed) go into the job result, which nothing renders; the Problems log is `/api/notices` only. The settle of 2026-09-30 10:06 closed 12,588 roles with one target unanswered and one incomplete, and the notices table stayed empty. Detail reads that fail, a 429 that stops them, `ats:complete` empty, `ats:swept`, `ats:narrowed` and the ceiling are state only. A board's size is set once and never again, so a size of 0 (a mistyped SmartRecruiters, Gem or Ashby slug) means "read whole" for good, and `set_board` keeps a board that lists nothing with a toast. A filtered read whose facets know none of the wanted values asks nothing and closes the whole board (`_runner.py:452`), and the line reads "site lists none of DK, SE: nothing asked" as if it were a fact about the site.
|
|
|
|
**The reader contract is declared but not enforced.** Of the `READS` traits only `posted`, `filters`, `complete`, `ceiling`, `empty`, `key` and `scope` are read at run time; `pay`, `remote`, `country`, `seniority`, `employment`, `title`, `teaser`, `page_size`, `places` and `text` are read by nothing and checked by nothing, so Platsbanken declares `remote="none"` while its rows set it, HR Manager declares `places="one"` and returns none, and seven readers record a robots.txt path they do not request (Oracle, Workday, Eightfold, Gem, Jibe, Phenom, Jobylon). Eight readers record "no robots.txt check needed". `config.py:171` still lists `jobindex` and `adzuna` as the only providers, so a Platsbanken starter search in config.yaml fails validation. A search provider does not plug in by existing: `PARAMS`, `clean_params` and the row key `company` are needed and not in the contract, and provider names are hard-coded in nine places. `discover._UNREADABLE` holds ten hosts that have modules. `undeclared()` returns nothing and its code paths are dead, as are `robots_page` and `ROBOTS_ANY_CASE`.
|
|
|
|
**The docs lag the code by a day of 224 commits.** `plugins/README.md` says SmartRecruiters and Workable have no text, that the pull reads 200 role pages per board (the enrich stage reads them, ranked roles first, under its own budgets), that Lever, Ashby, Greenhouse and Recruitee read one place, that a board over 100 pages is never complete, and its counts are of 2026-09-29. README.md says 175 readable boards (286 in the seed, 310 live) and that `config.example.yaml` documents every key: 21 keys it reads are missing, the whole `enrich:` block among them. `decisions.md` contradicts itself on the cascade (L175 lacks prep and enrich), Jobindex completeness (L73 against `complete="never"`), HR Manager (L109 says none attached; `hrmanager:teknologisk` has 7 open roles) and Novonesis, Darktrace and Criteo (L109 says Watchlist; all three are Workday boards). `open.md` is right on every item but the boilerplate one: `pipeline/` equals boilerplate HEAD, `auth/` lacks the phone commit, `deploy/` lacks the WSL scripts.
|
|
|
|
**Settings.** What a person changes while using the app is on the pages: agents, gates, weights, limits, fetch terms, stage switches, boards, reach, overrides, searches, the schedule, the account. What is config only is wiring (server, database, auth, workers, models, keys) plus the enrich depth, which the Runs page wrongly says is per agent. The model badge on Runs shows `extract.model` even when `agent.model` overrides it. Constants that decide coverage stay in code: `SMALL` 300, `STALE_DAYS` 21, the guard's 10%, `DETAIL_CAP` 200; the pages hard-code 300 in tooltips.
|
|
|
|
**Bugs outside ingestion, confirmed.** A board's own reach stores "UK" or "Germany" as a country code and, on a host that filters by ids, reads nothing and closes the board. A seniority range with no level in it (`level_from=senior&level_to=junior`) lists every level. Fetch the posting returns the role without the agent, so the reader's vote goes blank, and the row's excerpt goes blank for teaser-only roles. The toast's "all messages" link is a 404. Replace board always warns that roles will close. `_sync_copy_places` rewrites every copy-place row on every prep. A refused application status change leaves the refused status showing.
|
|
|
|
## Decisions taken for this plan
|
|
|
|
1. **Every whole read is guarded.** A read that lists under 10% of what the same read listed last time raises `Suspect` on every host, not only the suspect ones, when the last count was at least `SUSPECT_MIN` (10). The board is then failing with that reason, its roles stay open, and after `STALE_DAYS` of the same answer the read is accepted and the count reset, as Recheck resets it at once. A board that really shrank costs one Recheck or three weeks.
|
|
2. **A filtered read that could not ask remote does not close remote roles.** When the reach wants remote and the reader declares no remote filter, `stamp` records `remote_unasked`, `_close_unlisted` leaves `remote = 'remote'` roles open, and those close when the board has not listed them for `STALE_DAYS`. The line says "remote not asked on SmartRecruiters". Reading such boards whole instead is the owner's call (Bosch is about 70 requests a pull) and stays open in `open.md`; the 72 roles already closed reopen only if such a read lists them.
|
|
3. **A read that asks nothing is not a read.** When an `ids` dimension maps to no host id, the listing is incomplete and closes nothing, logged at warning, and the line says "nothing asked: not closing".
|
|
4. **Every target is guarded in the batch.** `run_batch` wraps each target: an exception after the read records the board as failing with its reason and moves on to the next target. The failing key is removed after `stamp`, not before the rows.
|
|
5. **Searches share the runner's bookkeeping.** `search.py` reads through `_runner.read` (guard included), records `search:failing:<target>` through `record_failure`, and the search row shows "not answering since" as the board row does. The settle reads failures for both sources.
|
|
6. **The settle speaks.** It raises one notice per pull naming what needs a look: every board or search unanswered or incomplete, every board that closed more than `CLOSE_NOTICE` (50) roles or more than half its open roles, and the totals. Its result key `failed` becomes `unanswered`. A finder that raises releases the pull claim.
|
|
7. **Narrowed boards close by age.** A role of a board read with terms closes when the board has not listed it for `STALE_DAYS`, the rule searches already have.
|
|
8. **A board's limits are words on its row, and a filter.** Each company row carries `limits`, computed from the reader's declaration and the board's state, and the Companies filter offers them: not readable yet (exists), failing, read incomplete, filtered by reach, remote not asked, narrowed by terms, size 0, text per role, no text, no pay, date not trusted, without text. The reader list on Sources shows every trait.
|
|
9. **Declarations are checked against fixtures.** The conformance test parses each reader's listing and detail fixtures and asserts every declared field; a missing fixture fails instead of skipping. `places` gains `none`. The search contract gains `PARAMS`, `clean_params` and the row key `company`. Each robots record names the paths the reader requests. Dead paths for undeclared modules and `robots_page` go.
|
|
10. **Docs say what the code does, once.** The shared limits live in `_runner.py`'s constants and the README table points at them; `READS` is the source for what a reader does and the per-host sections keep observations, quirks and examples; the cascade is `CASCADES`. The counts in `plugins/README.md` are refreshed and dated.
|
|
|
|
## Work packages
|
|
|
|
Each owns the files it names and nothing else; tests for a package live in the test files it names. Builders do not edit `docs/` or `plugins/README.md`: the planner folds the landed behaviour into the docs after the merges. `uv run pytest` and `uv run ruff check app plugins tests` must pass in the worktree before a package is done, and no package starts, stops or kills a server.
|
|
|
|
| WP | Model | Owns | Builds |
|
|
|---|---|---|---|
|
|
| R1 Losses | Opus | `plugins/readers/_runner.py`, `plugins/ats.py`, `plugins/search.py`, `app/jobs/pull.py`, `app/jobs/cascade.py`, `tests/test_ats.py`, `tests/test_readers.py`, `tests/test_pull.py` (new), `tests/test_search.py` (new or existing) | decisions 1 to 7, plus the mypy fixes in those files (`published_at` may be None, the `rows` redefinition, unused ignores) and the removal of the `declared()` branches now that every reader declares |
|
|
| R3 Bugs | Sonnet | `app/web/api.py`, `app/config.py`, `app/cli.py`, `app/jobs/stages.py`, `app/static/roles.js`, `app/static/toasts.js`, `app/static/companies.js`, `app/static/applications.js`, `app/static/jobs.js`, `config.example.yaml`, `tests/test_pages.py`, `tests/test_api_reach.py` (new) | the confirmed bugs: reach countries stored as ISO codes through `places.parse`, an empty seniority range answers 422, Fetch returns the role with the agent and `excerptOf` falls back to the teaser, the messages link, the Replace board compare, the refused status restored; `SearchSeed.provider` validated against `readers.PROVIDERS`; the Runs page's enrich sentence and model badge (`agent.model` wins); the cli and stages docstrings; `config.example.yaml` gains every key `app/config.py` and `desk/config.py` read. `_sync_copy_places` waits: `store.py` is being edited in the shared checkout |
|
|
| R4 Contract | Opus | `plugins/readers/__init__.py`, `plugins/readers/_contract.py`, `plugins/readers/<host>.py` (declarations and docstrings only), `app/jobs/discover.py`, `tests/test_readers_contract.py`, `tests/test_ats_<host>.py`, `tests/test_discover.py`, `tests/test_architecture.py`, `tests/fixtures/ats/` | decision 9: the fixture checks per declared field, `places="none"`, Platsbanken `remote="stated"`, HR Manager `places="none"`, the search contract, robots records naming the requested paths after reading each host's robots.txt again (the decision and date stay the owner's), the `_UNREADABLE` entries with modules deleted, `robots_page` and `ROBOTS_ANY_CASE` deleted with their test lines, the architecture test extended to `plugins/readers/*.py` with `._countries` allowed, a test that every `app/prep/hosts/<name>.py` names a reader and every reader stating words has a table, and the contract docstring corrected (Detail is read by `enrich`; the example `READS` matches `workday.py`; the touch points for a new reader listed: the module, the prep table, fixtures and tests, the README section, the `_UNREADABLE` entry to delete) |
|
|
| R2 Limits | Sonnet, after R3 merges | `app/jobs/limits.py` (new), `app/web/api.py` (`CompanyOut`, `ReaderOut`, `_company_outs` only), `app/static/companies.js`, `app/static/sources.js`, `app/templates/settings/sources.html`, `tests/test_limits.py` (new), `tests/test_pages.py` | decision 8: `limits.for_board(module, state, target, sub)` returns the words; `CompanyOut.limits` and `CompanyOut.board_open` (the board's own open count, beside the company's); the row shows the words after the reach line; the filter select gains one option per word; `ReaderOut` gains `complete`, `page_size`, `ceiling`, `teaser`, `fields`, `empty`; `sources.js` renders them |
|
|
| R5 Docs | Sonnet, now | `README.md`, `plugins/README.md`, `docs/open.md`, `docs/decisions.md`, `docs/filter-survey.md`, `docs/plan-hosts-wave-2.md` | decision 10 against main as it is: every correction the docs review listed, the counts of 2026-09-30, the boilerplate drift stated as it is, the contradictions in `decisions.md` resolved by marking the later decision, `open.md` gaining what this plan does not build (below) |
|
|
|
|
R1, R3, R4 and R5 start at once in worktrees made from main (`git worktree add ../jobboard-wt-<name> -b wp-<name> main`, then `uv sync --all-extras` there). R2 starts from main after R3 is merged, since both edit `api.py`.
|
|
|
|
## Not built here, to `open.md`
|
|
|
|
- Reading remote from the hosts that cannot filter on it (decision 2), and the 72 closed roles.
|
|
- Provider names hard-coded in nine places (`cli.py`, `dupes.py`, `sources.py`, `pages.py`, `sources.html`, `sources.js`, `api.py`, `posting.py`) instead of `readers.PROVIDERS`; `has_key` re-implements `resolve_key`.
|
|
- The compatibility names on `ats.py` (`_PARSERS`, `_parse_dt`, `_greenhouse_salary`, `_parse_teamtailor`, `_Refused`, `DETAIL_FIELDS`) that only tests reach for.
|
|
- `title="provisional"` cannot work while `DETAIL_FIELDS` has no `title`; wave 2's WP9 must own `_runner.py`.
|
|
- Some `posted="trusted"` declarations rest on the detail read (Workday, BambooHR, Rippling) or on a fallback the host moves (Greenhouse `updated_at`, Phenom `postedDate`).
|
|
- Dupes fold same-host roles with the same title and town without comparing text: 99 open copies share under 30% of their words with the kept role.
|
|
- Enrich never reads a role below the bar, so 349 textless roles are ranked on their title for good.
|
|
- A board's size is never re-measured by a pull; Recheck is the only way.
|
|
- Fetch terms are configured on no agent, so decision 7 is latent.
|
|
- A starred company on a big filtered board is scored on roles the fetch closes; All countries is the workaround.
|
|
- `SMALL`, `STALE_DAYS`, the guard share and `DETAIL_CAP` are code constants and the pages hard-code 300.
|
|
- `auth/` and `deploy/` lag the boilerplate; `desk/` lags news by four commits.
|
|
- The Jobindex robots record names `sort=` too, which `decisions.md` does not.
|
|
- `The todo list.md` holds items the code has done (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.
|
|
|
|
## Planner, after the merges
|
|
|
|
- Merge R1, R3, R4, R5, then start R2 and merge it; merge main into each branch first when the remote packages landed in between.
|
|
- Fold decisions 1 to 9 into `decisions.md` (Ingestion, The pages) and the limits words into `plugins/README.md`.
|
|
- Run one pull on the deployed server, read its settle notice, and check the Companies filter against the live boards.
|
|
- Delete this file.
|