Files
jobboard/docs/plan-hosts-wave-2.md
T
2026-09-30 10:06:50 +02:00

52 lines
5.1 KiB
Markdown

# Plan: the readers left after the hosts of 2026-09-29
Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped.
WP9 (the contract part) and WP11 were superseded by the readers rework (see `decisions.md`, Ingestion); WP10 and WP12 to WP15 build against the reader contract in `plugins/readers/__init__.py`.
## Where it stands
286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
| Route | Companies | Roles seen |
|---|---|---|
| Sitemap and structured data | Arm, Mambu, Odoo, Aiven, SOPHiA GENETICS, AUTO1, Apple, ASML | about 2,000 |
| Own JSON API | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | about 570 |
| Own HTML list | Zalando, Netcompany, Bolt, CHECK24, Google with DeepMind, Lucca, Shopify, Framer, Bird, Ableton | about 950 |
| A host that is read, with a gap | Infineon, Microsoft (Eightfold's page cap), Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md` and the smaller hosts in `otherhosts.md`. The raw responses were not kept; a builder fetches its own fixtures.
## Decisions to take first
**A listing that knows no title.** A sitemap lists URLs. `detail` may not set a title, since the listing rewrites it on every pull, so a sitemap reader would show a title made from the URL. Two ways out: the module reads the role pages inside `page`, which costs one request per role on every pull, or the contract gains a row key that marks the title as provisional, and `upsert_role` keeps a stored title over a provisional one. The second is the smaller cost per pull and touches `app/storage/store.py`.
**A filter in a target.** Eightfold takes a location, Workday and SuccessFactors a country facet. A target is `host:slug` and has nowhere to state one. `subscriptions.params` exists for searches; a board could use it, and `page` would receive it. This brings Infineon and Microsoft under the page cap and cuts a pull of Bosch (4,812 roles), Philips or SAP to the countries the agents accept.
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. Fourteen hosts landed as fourteen files without a shared edit, so the package takes a module per company as it is; `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
## Work packages
Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The common rules of wave 1 apply: the live request limits, robots.txt read first, no key from a script, a final report with the targets to attach.
| WP | Model | Builds | Needs |
|---|---|---|---|
| 9 Provisional title | Opus | the row key, `upsert_role`'s rule and its tests; owns `plugins/ats.py`, `plugins/readers/__init__.py`, `app/storage/store.py`, `tests/test_readers.py`, `tests/test_storage.py` | the first decision |
| 10 Sitemap reader | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
| 11 Board filters | Opus | `params` on a board subscription, handed to `page`; owns `plugins/ats.py`, `app/jobs/pull.py`, `app/jobs/sources.py`, `app/web/api.py`, `app/static/companies.js` and their tests | the second decision |
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9, and WP9 and WP11 both own `plugins/ats.py`, so they run one after the other.
## Planner, after the merges
- Attach each board with `app companies set-board`, which reads it first and clears the reason.
- Bring `seed/companies.json` in line with the live database: the board of every company that has one shown, the reason of every company that has none.
- Add a host to `TRUSTED_POSTED_HOSTS` where its date is a posting date, with the evidence in the comment.
- Restart the server by its PID after a backup in `../jobboard-backups/`.
- Find the career host of the nine Teamtailor companies by opening each careers page.