17 KiB
Plan: the review of 2026-09-30, before new features
Written 2026-09-30 against main at d47f4f5, from five read-only reviews of the code, the docs and the live database: silent failures in ingestion, the reader contract, docs against code, settings against the pages, and a bug hunt. What is built here is folded into decisions.md, the rest goes to open.md, and this file is deleted.
docs/plan-remote.md is being built at the same time in ../jobboard-wt-remote-a and -b, and its packages own app/jobs/, app/storage/, app/web/api.py, plugins/ and app/static/. The packages below own single files inside those directories and keep their hunks small; the planner merges whichever lands second and resolves. The shared checkout also carries uncommitted edits to score.py, models.py, store.py, agents.js, app.css, agents.html and test_score.py; no package here touches those.
What the reviews found
The test suite passes (2,923 tests, 20 skipped for missing fixtures), ruff is clean, mypy reports 15 typing errors, git is clean of stray branches. The pipeline as designed holds: a reader that raises is recorded per board and shown on its row from day one, a crashed batch closes only the boards it stamped, one pull runs at a time, two strikes protect swept boards, the reach is never empty. The problems are at the edges of that design.
Roles are lost without a trace in four ways.
- A host that cannot filter on remote drops the remote dimension silently.
ask_forkeeps only the dimensions the reader declares (_runner.py:374), so on SmartRecruiters, Phenom, SuccessFactors and Eightfold the agents' reach "DK, SE, remote" becomes "DK, SE", the read counts as filtered, and_close_unlistedcloses every remote role outside DK and SE. The Companies line says "follows agents: DK, SE" and nothing about remote. On 2026-09-29 and 30 this closed 72 remote roles inside the agent's Europe rule (Philips 34, Bosch 25, Coolblue 10, Delivery Hero 3), and new remote roles on those 21 boards are never fetched. - 206 of 309 boards have no guard against an empty answer. The 10% guard runs only for filtered, narrowed or
empty="suspect"reads (_runner.py:674). A Greenhouse or Lever answer whose shape changed parses to zero rows with HTTP 200, is stamped complete, and the settle closes the whole board: 15,635 open roles sit on unguarded boards. - A crash after the read starves the rest of the batch, every pull.
ats._boardremoves the failing key before the rows are stored (ats.py:410), anditem_of,upsert_roleandstamprun outside the try, with no per-target guard inrun_batch(pull.py:198). A bad row fails the whole batch job: that board has neither a stamp nor a failing record, the boards after it in the batch are not read, and batches are stable, so the same neighbours are skipped on every pull, never close, never age, never show "not answering". - A search that fails leaves no trace, and searches have no guard.
search.py:140logs and continues; there is nosearch:failingkey and the search row has no failure field. Searches callread_listing, notread, so a Platsbanken answer withhitsrenamed reads as complete and empty and closes every role of the search at once.
What the pull does is not shown. The settle's counts (boards, unanswered, incomplete, closed) go into the job result, which nothing renders; the Problems log is /api/notices only. The settle of 2026-09-30 10:06 closed 12,588 roles with one target unanswered and one incomplete, and the notices table stayed empty. Detail reads that fail, a 429 that stops them, ats:complete empty, ats:swept, ats:narrowed and the ceiling are state only. A board's size is set once and never again, so a size of 0 (a mistyped SmartRecruiters, Gem or Ashby slug) means "read whole" for good, and set_board keeps a board that lists nothing with a toast. A filtered read whose facets know none of the wanted values asks nothing and closes the whole board (_runner.py:452), and the line reads "site lists none of DK, SE: nothing asked" as if it were a fact about the site.
The reader contract is declared but not enforced. Of the READS traits only posted, filters, complete, ceiling, empty, key and scope are read at run time; pay, remote, country, seniority, employment, title, teaser, page_size, places and text are read by nothing and checked by nothing, so Platsbanken declares remote="none" while its rows set it, HR Manager declares places="one" and returns none, and seven readers record a robots.txt path they do not request (Oracle, Workday, Eightfold, Gem, Jibe, Phenom, Jobylon). Eight readers record "no robots.txt check needed". config.py:171 still lists jobindex and adzuna as the only providers, so a Platsbanken starter search in config.yaml fails validation. A search provider does not plug in by existing: PARAMS, clean_params and the row key company are needed and not in the contract, and provider names are hard-coded in nine places. discover._UNREADABLE holds ten hosts that have modules. undeclared() returns nothing and its code paths are dead, as are robots_page and ROBOTS_ANY_CASE.
The docs lag the code by a day of 224 commits. plugins/README.md says SmartRecruiters and Workable have no text, that the pull reads 200 role pages per board (the enrich stage reads them, ranked roles first, under its own budgets), that Lever, Ashby, Greenhouse and Recruitee read one place, that a board over 100 pages is never complete, and its counts are of 2026-09-29. README.md says 175 readable boards (286 in the seed, 310 live) and that config.example.yaml documents every key: 21 keys it reads are missing, the whole enrich: block among them. decisions.md contradicts itself on the cascade (L175 lacks prep and enrich), Jobindex completeness (L73 against complete="never"), HR Manager (L109 says none attached; hrmanager:teknologisk has 7 open roles) and Novonesis, Darktrace and Criteo (L109 says Watchlist; all three are Workday boards). open.md is right on every item but the boilerplate one: pipeline/ equals boilerplate HEAD, auth/ lacks the phone commit, deploy/ lacks the WSL scripts.
Settings. What a person changes while using the app is on the pages: agents, gates, weights, limits, fetch terms, stage switches, boards, reach, overrides, searches, the schedule, the account. What is config only is wiring (server, database, auth, workers, models, keys) plus the enrich depth, which the Runs page wrongly says is per agent. The model badge on Runs shows extract.model even when agent.model overrides it. Constants that decide coverage stay in code: SMALL 300, STALE_DAYS 21, the guard's 10%, DETAIL_CAP 200; the pages hard-code 300 in tooltips.
Bugs outside ingestion, confirmed. A board's own reach stores "UK" or "Germany" as a country code and, on a host that filters by ids, reads nothing and closes the board. A seniority range with no level in it (level_from=senior&level_to=junior) lists every level. Fetch the posting returns the role without the agent, so the reader's vote goes blank, and the row's excerpt goes blank for teaser-only roles. The toast's "all messages" link is a 404. Replace board always warns that roles will close. _sync_copy_places rewrites every copy-place row on every prep. A refused application status change leaves the refused status showing.
Decisions taken for this plan
- Every whole read is guarded. A read that lists under 10% of what the same read listed last time raises
Suspecton every host, not only the suspect ones, when the last count was at leastSUSPECT_MIN(10). The board is then failing with that reason, its roles stay open, and afterSTALE_DAYSof the same answer the read is accepted and the count reset, as Recheck resets it at once. A board that really shrank costs one Recheck or three weeks. - A filtered read that could not ask remote does not close remote roles. When the reach wants remote and the reader declares no remote filter,
stamprecordsremote_unasked,_close_unlistedleavesremote = 'remote'roles open, and those close when the board has not listed them forSTALE_DAYS. The line says "remote not asked on SmartRecruiters". Reading such boards whole instead is the owner's call (Bosch is about 70 requests a pull) and stays open inopen.md; the 72 roles already closed reopen only if such a read lists them. - A read that asks nothing is not a read. When an
idsdimension maps to no host id, the listing is incomplete and closes nothing, logged at warning, and the line says "nothing asked: not closing". - Every target is guarded in the batch.
run_batchwraps each target: an exception after the read records the board as failing with its reason and moves on to the next target. The failing key is removed afterstamp, not before the rows. - Searches share the runner's bookkeeping.
search.pyreads through_runner.read(guard included), recordssearch:failing:<target>throughrecord_failure, and the search row shows "not answering since" as the board row does. The settle reads failures for both sources. - The settle speaks. It raises one notice per pull naming what needs a look: every board or search unanswered or incomplete, every board that closed more than
CLOSE_NOTICE(50) roles or more than half its open roles, and the totals. Its result keyfailedbecomesunanswered. A finder that raises releases the pull claim. - Narrowed boards close by age. A role of a board read with terms closes when the board has not listed it for
STALE_DAYS, the rule searches already have. - A board's limits are words on its row, and a filter. Each company row carries
limits, computed from the reader's declaration and the board's state, and the Companies filter offers them: not readable yet (exists), failing, read incomplete, filtered by reach, remote not asked, narrowed by terms, size 0, text per role, no text, no pay, date not trusted, without text. The reader list on Sources shows every trait. - Declarations are checked against fixtures. The conformance test parses each reader's listing and detail fixtures and asserts every declared field; a missing fixture fails instead of skipping.
placesgainsnone. The search contract gainsPARAMS,clean_paramsand the row keycompany. Each robots record names the paths the reader requests. Dead paths for undeclared modules androbots_pagego. - Docs say what the code does, once. The shared limits live in
_runner.py's constants and the README table points at them;READSis the source for what a reader does and the per-host sections keep observations, quirks and examples; the cascade isCASCADES. The counts inplugins/README.mdare refreshed and dated.
Work packages
Each owns the files it names and nothing else; tests for a package live in the test files it names. Builders do not edit docs/ or plugins/README.md: the planner folds the landed behaviour into the docs after the merges. uv run pytest and uv run ruff check app plugins tests must pass in the worktree before a package is done, and no package starts, stops or kills a server.
| WP | Model | Owns | Builds |
|---|---|---|---|
| R1 Losses | Opus | plugins/readers/_runner.py, plugins/ats.py, plugins/search.py, app/jobs/pull.py, app/jobs/cascade.py, tests/test_ats.py, tests/test_readers.py, tests/test_pull.py (new), tests/test_search.py (new or existing) |
decisions 1 to 7, plus the mypy fixes in those files (published_at may be None, the rows redefinition, unused ignores) and the removal of the declared() branches now that every reader declares |
| R3 Bugs | Sonnet | app/web/api.py, app/config.py, app/cli.py, app/jobs/stages.py, app/static/roles.js, app/static/toasts.js, app/static/companies.js, app/static/applications.js, app/static/jobs.js, config.example.yaml, tests/test_pages.py, tests/test_api_reach.py (new) |
the confirmed bugs: reach countries stored as ISO codes through places.parse, an empty seniority range answers 422, Fetch returns the role with the agent and excerptOf falls back to the teaser, the messages link, the Replace board compare, the refused status restored; SearchSeed.provider validated against readers.PROVIDERS; the Runs page's enrich sentence and model badge (agent.model wins); the cli and stages docstrings; config.example.yaml gains every key app/config.py and desk/config.py read. _sync_copy_places waits: store.py is being edited in the shared checkout |
| R4 Contract | Opus | plugins/readers/__init__.py, plugins/readers/_contract.py, plugins/readers/<host>.py (declarations and docstrings only), app/jobs/discover.py, tests/test_readers_contract.py, tests/test_ats_<host>.py, tests/test_discover.py, tests/test_architecture.py, tests/fixtures/ats/ |
decision 9: the fixture checks per declared field, places="none", Platsbanken remote="stated", HR Manager places="none", the search contract, robots records naming the requested paths after reading each host's robots.txt again (the decision and date stay the owner's), the _UNREADABLE entries with modules deleted, robots_page and ROBOTS_ANY_CASE deleted with their test lines, the architecture test extended to plugins/readers/*.py with ._countries allowed, a test that every app/prep/hosts/<name>.py names a reader and every reader stating words has a table, and the contract docstring corrected (Detail is read by enrich; the example READS matches workday.py; the touch points for a new reader listed: the module, the prep table, fixtures and tests, the README section, the _UNREADABLE entry to delete) |
| R2 Limits | Sonnet, after R3 merges | app/jobs/limits.py (new), app/web/api.py (CompanyOut, ReaderOut, _company_outs only), app/static/companies.js, app/static/sources.js, app/templates/settings/sources.html, tests/test_limits.py (new), tests/test_pages.py |
decision 8: limits.for_board(module, state, target, sub) returns the words; CompanyOut.limits and CompanyOut.board_open (the board's own open count, beside the company's); the row shows the words after the reach line; the filter select gains one option per word; ReaderOut gains complete, page_size, ceiling, teaser, fields, empty; sources.js renders them |
| R5 Docs | Sonnet, now | README.md, plugins/README.md, docs/open.md, docs/decisions.md, docs/filter-survey.md, docs/plan-hosts-wave-2.md |
decision 10 against main as it is: every correction the docs review listed, the counts of 2026-09-30, the boilerplate drift stated as it is, the contradictions in decisions.md resolved by marking the later decision, open.md gaining what this plan does not build (below) |
R1, R3, R4 and R5 start at once in worktrees made from main (git worktree add ../jobboard-wt-<name> -b wp-<name> main, then uv sync --all-extras there). R2 starts from main after R3 is merged, since both edit api.py.
Not built here, to open.md
- Reading remote from the hosts that cannot filter on it (decision 2), and the 72 closed roles.
- Provider names hard-coded in nine places (
cli.py,dupes.py,sources.py,pages.py,sources.html,sources.js,api.py,posting.py) instead ofreaders.PROVIDERS;has_keyre-implementsresolve_key. - The compatibility names on
ats.py(_PARSERS,_parse_dt,_greenhouse_salary,_parse_teamtailor,_Refused,DETAIL_FIELDS) that only tests reach for. title="provisional"cannot work whileDETAIL_FIELDShas notitle; wave 2's WP9 must own_runner.py.- Some
posted="trusted"declarations rest on the detail read (Workday, BambooHR, Rippling) or on a fallback the host moves (Greenhouseupdated_at, PhenompostedDate). - Dupes fold same-host roles with the same title and town without comparing text: 99 open copies share under 30% of their words with the kept role.
- Enrich never reads a role below the bar, so 349 textless roles are ranked on their title for good.
- A board's size is never re-measured by a pull; Recheck is the only way.
- Fetch terms are configured on no agent, so decision 7 is latent.
- A starred company on a big filtered board is scored on roles the fetch closes; All countries is the workaround.
SMALL,STALE_DAYS, the guard share andDETAIL_CAPare code constants and the pages hard-code 300.auth/anddeploy/lag the boilerplate;desk/lags news by four commits.- The Jobindex robots record names
sort=too, whichdecisions.mddoes not. The todo list.mdholds items the code has done (Jobindex robots, Eightfold cap, places, aliases, teaser, ladder, hybrid on Ashby); the owner prunes it.
Planner, after the merges
- Merge R1, R3, R4, R5, then start R2 and merge it; merge main into each branch first when the remote packages landed in between.
- Fold decisions 1 to 9 into
decisions.md(Ingestion, The pages) and the limits words intoplugins/README.md. - Run one pull on the deployed server, read its settle notice, and check the Companies filter against the live boards.
- Delete this file.