Prune done work from the docs: fold the ingest review's open findings into open.md and delete it, cut the hosts wave-2 plan to what is unbuilt, drop the smaller-hosts survey

The plugins README no longer describes a run-time robots.txt check or boards that never close past the page cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Opus 5.5 committed 2026-09-30 13:40:43 +02:00
1 parent a44bf5dacb
commit 802656aca4
8 files changed
+70 -472

No files matched your search

+19 -15
View File
@@ -1,51 +1,55 @@
# Plan: the readers left after the hosts of 2026-09-29
Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped.
Planned 2026-09-29. None of the packages below is built. Each builds against the reader contract in `plugins/readers/__init__.py`. What is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages land or are dropped.
WP9 (the contract part) and WP11 were superseded by the readers rework (see `decisions.md`, Ingestion); WP10 and WP12 to WP15 build against the reader contract in `plugins/readers/__init__.py`.
The board filter this plan had as WP11 landed as the agents' reach (`decisions.md`, Ingestion), which also brought Infineon and Microsoft under Eightfold's page cap.
## Where it stands
286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
76 companies are on the Watchlist, each with the careers page opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
| Route | Companies | Roles seen |
|---|---|---|
| Sitemap and structured data | Arm, Mambu, Odoo, Aiven, SOPHiA GENETICS, AUTO1, Apple, ASML | about 2,000 |
| Own JSON API | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | about 570 |
| Own HTML list | Zalando, Netcompany, Bolt, CHECK24, Google with DeepMind, Lucca, Shopify, Framer, Bird, Ableton | about 950 |
| A host that is read, with a gap | Infineon, Microsoft (Eightfold's page cap), Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
| A host that is read, with a gap | Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md` and the smaller hosts in `otherhosts.md`. The raw responses were not kept; a builder fetches its own fixtures.
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures.
## Decisions to take first
**A listing that knows no title.** A sitemap lists URLs. `detail` may not set a title, since the listing rewrites it on every pull, so a sitemap reader would show a title made from the URL. Two ways out: the module reads the role pages inside `page`, which costs one request per role on every pull, or the contract gains a row key that marks the title as provisional, and `upsert_role` keeps a stored title over a provisional one. The second is the smaller cost per pull and touches `app/storage/store.py`.
**A listing that knows no title.** A sitemap lists URLs, and the listing rewrites the title on every pull, so a sitemap reader would show a title made from the URL. The contract already has `Reads(title="provisional")` (`plugins/readers/_contract.py`), but nothing honours it: `upsert_role` still overwrites a title that `detail` filled. WP9 adds that rule.
**A filter in a target.** Eightfold takes a location, Workday and SuccessFactors a country facet. A target is `host:slug` and has nowhere to state one. `subscriptions.params` exists for searches; a board could use it, and `page` would receive it. This brings Infineon and Microsoft under the page cap and cuts a pull of Bosch (4,812 roles), Philips or SAP to the countries the agents accept.
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. Fourteen hosts landed as fourteen files without a shared edit, so the package takes a module per company as it is; `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
## Work packages
Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The common rules of wave 1 apply: the live request limits, robots.txt read first, no key from a script, a final report with the targets to attach.
Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The live request limits apply, robots.txt is read first, no key is taken from a script, and each package ends with a report of the targets to attach.
| WP | Model | Builds | Needs |
|---|---|---|---|
| 9 Provisional title | Opus | the row key, `upsert_role`'s rule and its tests; owns `plugins/ats.py`, `plugins/readers/__init__.py`, `app/storage/store.py`, `tests/test_readers.py`, `tests/test_storage.py` | the first decision |
| 9 Provisional title | Opus | `upsert_role` keeps a stored title over a provisional one, and its tests; owns `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | |
| 10 Sitemap reader | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
| 11 Board filters | Opus | `params` on a board subscription, handed to `page`; owns `plugins/ats.py`, `app/jobs/pull.py`, `app/jobs/sources.py`, `app/web/api.py`, `app/static/companies.js` and their tests | the second decision |
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9, and WP9 and WP11 both own `plugins/ats.py`, so they run one after the other.
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9.
### WP10 notes: Mambu and Arm
Both were read live on 2026-09-29 without a browser or a block.
- **Mambu, classic iCIMS.** `https://careers-mambu.icims.com/sitemap.xml` has one `<loc>` per job (`/jobs/{id}/{slug}/job`) with `lastmod`, plus `/jobs/intro`; robots.txt allows it and lists the sitemap. The detail `.../jobs/{id}/x/job?in_iframe=1` carries a JSON-LD `JobPosting` with `datePosted`, `description`, `jobLocation[].address.addressCountry`, `validThrough` and `employmentType`. Tenants use other subdomain prefixes, so the target stores the whole host. 8 roles.
- **Arm, TalentBrew (Radancy).** The `*-arm.icims.com` hosts only redirect to `careers.arm.com`. `https://careers.arm.com/sitemap.xml` lists all 399 `/job/...` URLs in one request, without titles. The detail's JSON-LD has `datePosted` (unpadded, "2026-6-25"), `description`, `identifier` and `jobLocation[].address`. The paged list at `/search-jobs/results` is HTML inside JSON and only answers the full parameter set; its `data-total-results` counts content pages, `data-total-job-results` counts roles.
## Planner, after the merges
- Attach each board with `app companies set-board`, which reads it first and clears the reason.
- Bring `seed/companies.json` in line with the live database: the board of every company that has one shown, the reason of every company that has none.
- Add a host to `TRUSTED_POSTED_HOSTS` where its date is a posting date, with the evidence in the comment.
- Bring `seed/companies.json` in line with the live database: the board of every company that has one, the reason of every company that has none.
- Declare `Fields(posted="trusted")` on a reader whose date is a posting date, with the evidence in the docstring.
- Restart the server by its PID after a backup in `../jobboard-backups/`.
- Find the career host of the nine Teamtailor companies by opening each careers page.