The plugins README no longer describes a run-time robots.txt check or boards that never close past the page cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
56 lines
5.0 KiB
Markdown
56 lines
5.0 KiB
Markdown
# Plan: the readers left after the hosts of 2026-09-29
|
|
|
|
Planned 2026-09-29. None of the packages below is built. Each builds against the reader contract in `plugins/readers/__init__.py`. What is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages land or are dropped.
|
|
|
|
The board filter this plan had as WP11 landed as the agents' reach (`decisions.md`, Ingestion), which also brought Infineon and Microsoft under Eightfold's page cap.
|
|
|
|
## Where it stands
|
|
|
|
76 companies are on the Watchlist, each with the careers page opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
|
|
|
|
| Route | Companies | Roles seen |
|
|
|---|---|---|
|
|
| Sitemap and structured data | Arm, Mambu, Odoo, Aiven, SOPHiA GENETICS, AUTO1, Apple, ASML | about 2,000 |
|
|
| Own JSON API | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | about 570 |
|
|
| Own HTML list | Zalando, Netcompany, Bolt, CHECK24, Google with DeepMind, Lucca, Shopify, Framer, Bird, Ableton | about 950 |
|
|
| A host that is read, with a gap | Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
|
|
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
|
|
|
|
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md`. Arm and Mambu are not in it; their notes are under WP10. The raw responses were not kept, so a builder fetches its own fixtures.
|
|
|
|
## Decisions to take first
|
|
|
|
**A listing that knows no title.** A sitemap lists URLs, and the listing rewrites the title on every pull, so a sitemap reader would show a title made from the URL. The contract already has `Reads(title="provisional")` (`plugins/readers/_contract.py`), but nothing honours it: `upsert_role` still overwrites a title that `detail` filled. WP9 adds that rule.
|
|
|
|
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
|
|
|
|
## Work packages
|
|
|
|
Each owns its modules in `plugins/readers/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The live request limits apply, robots.txt is read first, no key is taken from a script, and each package ends with a report of the targets to attach.
|
|
|
|
| WP | Model | Builds | Needs |
|
|
|---|---|---|---|
|
|
| 9 Provisional title | Opus | `upsert_role` keeps a stored title over a provisional one, and its tests; owns `app/storage/store.py`, `plugins/ats.py`, `tests/test_storage.py` | |
|
|
| 10 Sitemap reader | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
|
|
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
|
|
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
|
|
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
|
|
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
|
|
|
|
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9.
|
|
|
|
### WP10 notes: Mambu and Arm
|
|
|
|
Both were read live on 2026-09-29 without a browser or a block.
|
|
|
|
- **Mambu, classic iCIMS.** `https://careers-mambu.icims.com/sitemap.xml` has one `<loc>` per job (`/jobs/{id}/{slug}/job`) with `lastmod`, plus `/jobs/intro`; robots.txt allows it and lists the sitemap. The detail `.../jobs/{id}/x/job?in_iframe=1` carries a JSON-LD `JobPosting` with `datePosted`, `description`, `jobLocation[].address.addressCountry`, `validThrough` and `employmentType`. Tenants use other subdomain prefixes, so the target stores the whole host. 8 roles.
|
|
- **Arm, TalentBrew (Radancy).** The `*-arm.icims.com` hosts only redirect to `careers.arm.com`. `https://careers.arm.com/sitemap.xml` lists all 399 `/job/...` URLs in one request, without titles. The detail's JSON-LD has `datePosted` (unpadded, "2026-6-25"), `description`, `identifier` and `jobLocation[].address`. The paged list at `/search-jobs/results` is HTML inside JSON and only answers the full parameter set; its `data-total-results` counts content pages, `data-total-job-results` counts roles.
|
|
|
|
## Planner, after the merges
|
|
|
|
- Attach each board with `app companies set-board`, which reads it first and clears the reason.
|
|
- Bring `seed/companies.json` in line with the live database: the board of every company that has one, the reason of every company that has none.
|
|
- Declare `Fields(posted="trusted")` on a reader whose date is a posting date, with the evidence in the docstring.
|
|
- Restart the server by its PID after a backup in `../jobboard-backups/`.
|
|
- Find the career host of the nine Teamtailor companies by opening each careers page.
|