Files
jobboard/docs/plan-hosts-and-by-hand.md
T

152 lines
19 KiB
Markdown

# Plan: a module per host, new hosts, and the list of companies checked by hand
Planned 2026-09-29. Three waves of work packages with disjoint files, built in parallel worktrees made from `main`, merged by the planner. This file and `docs/plan-hosts/` are folded into `decisions.md` and deleted when the work lands.
## Why
164 of the 362 companies have no board attached: 9 on Teamtailor whose career host is not found yet (the other 23 were attached today), and 155 on a host the app cannot read or on no host it could name. They include LEGO, Novo Nordisk, Vestas, Danske Bank, SimCorp, Ørsted, Klarna and Zalando. A company the app cannot read is shown as "not readable yet" with a link that nobody can edit, so the owner has no list to work through by hand.
## What the survey found
Nine Sonnet agents read every one of the 155 career sites on 2026-09-29. Their notes are in `docs/plan-hosts/`: `workday.md`, `successfactors.md`, `otherhosts.md`, `own-1.md` to `own-4.md`, and `survey-own.json` with one record per company (verified `jobs_url`, class, host, identifier, listing URL, roles seen, evidence). Raw responses for fixtures are in the session scratchpad, `/tmp/claude-1000/-home-claude-cboxcode-jobboard/1734d897-7509-48fc-844c-efe768acbf7a/scratchpad/`, and a builder copies the ones it needs, trimmed, into `tests/fixtures/ats/`.
| Route | Companies | Examples |
|---|---|---|
| Board on a host the app reads already | 49 + 9 Teamtailor | Bosch, Wise, Ubisoft, Checkout.com, Lunar, DFDS, Voi, Remote |
| SuccessFactors | 20 | Novo Nordisk, Vestas, Carlsberg, Pandora, Coloplast, Scania, SAP, Tryg |
| Workday | 19 | LEGO, Zeiss, SimCorp, Zendesk, Unity, Saxo Bank, Red Hat |
| Oracle | 4 | Danske Bank, Nokia, Sinch, Milestone |
| Phenom | 3 | Philips, Just Eat Takeaway, King |
| Eightfold | 3 | Microsoft, Ericsson, Infineon |
| Eleven small hosts | 15 | Klarna (Deel), Booking.com (Jibe), sennder (Gem), Einride (Jobylon) |
| Sitemap and structured data | 8 | Arm, Mambu, Odoo, Aiven, AUTO1, ASML |
| Own JSON API | 8 | Ørsted, Nordea, Amazon, Vinted, Dynatrace |
| Own HTML list | 10 | Zalando, Netcompany, Google, Bolt, CHECK24 |
| By hand | about 25 | Revolut, Meta, Maersk, Bang & Olufsen, Visma, HashiCorp |
The 49 were missed because the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain, the board is on Lever's EU host, or the careers page refuses a plain request.
## Decisions
**A host is a module.** `plugins/ats.py` holds eight hosts in six parallel tables, and `app/jobs/discover.py` holds three more, so two builders adding a host edit the same lines. Each host moves into `plugins/atshosts/<host>.py`, and `ats.py` and `discover.py` build their tables from the package. The package is a sibling of `ats.py` and not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module.
**A host is handed the fetch.** A host module imports nothing from `plugins.ats`. Its functions receive `get`, which `ats.py` binds to its own `_get` at call time, so the allow-list, the redirect rule, the byte cap and the deadline stay in one place and a test that patches `ats._get` reaches every host.
**A description read per role is read once.** Workday, SuccessFactors and BambooHR state the description only on the role's own page. A host with a `detail` function has it called for roles not yet read in detail, at most `DETAIL_CAP` (200) per board per pull; the ids read are kept in `ats:detailed:<target>`. A known role is sent without a description on later pulls, which `upsert_role` ignores since an empty value never overwrites a filled column. Completeness is decided by the listing alone.
**robots.txt is honoured.** A host module names the page whose path must be allowed. The shared code reads the site's robots.txt at most once a day per board and fails a board that disallows it for `*`, with the reason "robots.txt disallows it". This takes Novonesis, Darktrace, Criteo and Bang & Olufsen off the automatic route. The SuccessFactors reader uses `/sitemap.xml`, `/search/` and `/job/` only, never `/services/` or the `career*.successfactors.eu` back end.
**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key that the page's JavaScript carries. They go on the by-hand list.
**The by-hand list is the `companies` table, and the Watchlist tab is the list.** A company is on it when it has no shown board and is not known from a search only. `careers_url`, `notes`, `watch` (the rating) and `careers_opened_at` exist already, the last being the date a person last looked at the careers page; migration 16 adds `hand_reason`. There is no `checked_at`, since opening the page from the Watchlist stamps the date already. No second table, since a company has one careers link and the list needs no history of checks.
**A link typed by hand is kept.** `add_discovered` and the seed fill `careers_url` only when it is empty.
**A company has one board.** DFDS has eleven Teamtailor boards by country, and Onfido, Tink and Glovo sit inside their owners' boards. DFDS gets its Danish board; the other three go on the by-hand list with reason `group`.
## Contracts between packages
Host module, `plugins/atshosts/<host>.py`, as built by WP0. The docstring of `plugins/atshosts/__init__.py` holds the same contract and is the one to keep current. A module imports `desk` and `._shared` only, never `plugins.ats`.
```python
NAME = "workday" # the host of a target `workday:<slug>`; equals the module's file name
LABEL = "Workday" # the Sources page's select
PATTERNS = (r"https?://...",) # discovery: regexes over a URL, compiled with IGNORECASE
RANK = 100 # optional int, default 100; lower comes first
KEEP_CASE = True # optional, default False; keep a discovered slug's case
ROBOTS_ANY_CASE = True # optional, default False; read robots.txt paths without case
def slug_ok(slug: str) -> bool: ... # strict, re.fullmatch
def allowed(slug: str) -> Callable[[str], bool]: ... # host names a fetch may reach
def careers_url(slug: str) -> str: ... # the page a person opens
# one of the two
def feed_url(slug: str) -> str: ... # the whole board in one answer
def parse(body: bytes, slug: str) -> list[Row]: ...
def page(get: Get, slug: str, cursor: Any) -> Page: ... # a paged board, one page a call
def rows(decoded: Any, slug: str) -> list[Row]: ...
# optional
def fallback_url(slug: str) -> str: ... # feed hosts: a smaller feed, read once when feed_url is over the byte cap
def robots_page(slug: str) -> str: ... # the URL robots.txt must allow; on a host allowed(slug) admits
def detail(get: Get, slug: str, row: Row) -> dict[str, Any]: ... # fields from the role's own page
def probe_url(slug: str) -> str: ... # discovery: answers 2xx only for a board
def probe_ok(body: bytes) -> bool: ... # discovery: when 2xx is not enough
def probe_slugs(guess: str) -> tuple[str, ...]: ... # the slugs a guessed name is probed as; default (guess,)
```
`Get`, `Row` and `Page` are in `plugins/atshosts/_shared.py`, with the helpers several hosts share: `only(host)` for `allowed`, `path_slug_ok` and `label_slug_ok` for `slug_ok`, `parse_dt`, `is_web_url`, `num`, `yearly_amount` and `pay_line`. `plugins/atshosts/__init__.py` lists the modules with `pkgutil.iter_modules`, checks each against the contract at import (an ImportError names the module and what it lacks), and orders `HOSTS` by `(RANK, NAME)`. The eight first hosts hold ranks 10 to 80, which keeps the order they had before the modules: the select opens on Greenhouse, and a guessed slug on two hosts goes to the one probed first.
Order and names. `RANK` orders the Sources page's select, the plugin's description and discovery's probes. `NAME` equals the file name. A host that `discover.py` names in its own `_UNREADABLE` table is taken from its module as soon as the module exists, so a wave 1 package does not edit `discover.py`.
Patterns. The slug is group 1, or, when a pattern has more groups, the groups that matched joined by "/" (`https?://([a-z0-9-]+\.wd\d+)\.myworkdayjobs\.com/(?:[a-z]{2}-[A-Z]{2}/)?([A-Za-z0-9_-]+)` gives `lego.wd103/LEGO_External`). Only group 1 is checked against discovery's words that name no board (`api`, `jobs`, `embed`, ...). The slug is lowercased unless the module sets `KEEP_CASE = True`, which a host with case-sensitive paths needs, since robots.txt paths are case-sensitive too. A pattern without a group names the host with no slug.
Slugs. `slug_ok` uses `re.fullmatch` (a `$` lets a trailing newline through) and refuses anything that could add a path, query, fragment, port or user to the URLs built from it. A slug of several parts joins them with "/": Lever's EU boards are `lever:eu/<site>`.
`get(url, *, method="GET", payload=None, headers=None) -> bytes` is `ats._get` bound to the board. A POST sends `payload` as JSON; `headers` are added to the default User-Agent and Accept and replace either where they name it. It fetches https on port 443 only, never an IP address or a local name (`localhost`, `.local`, `.internal`, `.lan`, `.home.arpa`), follows at most three redirects each checked against `allowed(slug)`, cuts the body at `MAX_BYTES` and ends within `DEADLINE`. It returns the body and raises `httpx.HTTPStatusError` on a non-2xx answer, `httpx.HTTPError` on a timeout or a connection failure, and another exception, `ValueError` among them, for a refused host, an oversized body or a broken redirect. A module lets these through.
`Page` is `(decoded, results, total, cursor)`: the decoded page handed to `rows`, its raw results as dicts with an `id` (counted by distinct id for completeness), the total the host states or None, and the next cursor or None after the last page. The first call gets cursor None. A board with a total is complete when as many distinct ids were read, one without when the read ended on an empty page; a 429 after the first page ends the read, incomplete. A paged module may also have `parse(body, slug)` for one page's body, for its tests.
A `Row` has `source_id`, `title`, `location`, `url`, `posted` (aware datetime or None), `department`, `description` (plain text), `remote` (True, False, a word or None) and `salary_text`, and, where the host states pay, `salary_min`, `salary_max` (yearly integers) and `currency`.
`detail` returns a dict of what the role's own page states. The shared code calls it for roles not yet read (`ats:detailed:<target>`), at most `DETAIL_CAP` (200) per board per pull, `PAGE_DELAY` apart, the roles never tried before those whose detail failed (`ats:detailfailed:<target>`). Only `DETAIL_FIELDS` are taken: `description`, `location`, `remote`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the listing's empty value (`upsert_role`'s keep-filled columns, the salary figures, and `posted_at`, which keeps the earliest date). Any other key is dropped with a debug log line: `title`, `url`, `department` and `salary_text` cannot be filled from `detail`, since the listing rewrites them every pull. A role not read in detail on a pull is sent with an empty description. A 404 or 410 counts as read, any other failure is retried on a later pull, and a 429 ends the detail reads for that board and pull. Completeness is decided by the listing alone.
robots.txt. For a module with `robots_page`, the shared code reads `https://<its host>/robots.txt` before the listing, at most once a day per board (`ats:robots:<target>`), following a redirect to the host's `www.` twin or back and no other. It fails the board with "robots.txt disallows it" when the rules for `*` or for the app's own token disallow the page, matched per RFC 9309: the longest matching rule wins, Allow wins a tie, `*` and a final `$` in paths, paths case-sensitive, an empty Disallow allows. A 4xx other than 429 means there are no rules, which allows. A 5xx, a 429, a timeout, a connection error or a redirect elsewhere fails the board with "robots.txt could not be read", and the next pull asks again.
Slugs: Workday `lego.wd103/LEGO_External`; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (`jobs.teradyne.com/MiR`); Oracle `<host>/<siteNumber>`.
Company JSON, in `GET /api/companies`: `CompanyOut` has `domain`, `city`, `watch` and `careers_opened_at` from the Watchlist, and gains `notes`, `hand_reason` and `by_hand` (bool, no board shown and not `from_search`). `hand_reason` is one of `''`, `unread` (readable, no parser yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone`, `added` (by hand or by an import, not looked at). `from_search` is false for a company with a `hand_reason`.
| Endpoint | Body | Answer |
|---|---|---|
| `POST /api/companies` | `name`, `careers_url`, `notes`, `hand_reason` (default `added`) | 201 `CompanyOut`; 409 when the key is taken |
| `PATCH /api/companies/{key}` | any of `name`, `careers_url`, `notes`, `hand_reason` | `CompanyOut`; the key stays; 409 on `name` for a company with a board |
| `DELETE /api/companies/{key}` | none | 204; 409 when it has a board or open roles; its `agent_companies` rows go with it |
`POST /api/companies/{key}/careers` and `POST /api/companies/{key}/opened` of the Watchlist stay and answer a `Company`; `POST .../careers` and `PATCH` check the link with the same function (`check_careers_url` in `app/jobs/sources.py`). `careers_url` is http or https, as `safeUrl` demands on the page. A person looking at the page is `POST .../opened`, which stamps `careers_opened_at`.
## Work packages
Wave 0, in parallel, from today's `main`:
| WP | Branch | Model | Owns |
|---|---|---|---|
| 0 Host modules | `wp0-hosts` | Opus | `plugins/ats.py`, `plugins/atshosts/`, `app/jobs/discover.py`, `tests/test_ats.py`, `tests/test_discover.py`, `tests/test_atshosts.py` |
| 1 By-hand backend | `wp1-byhand-api` | Sonnet | `app/models.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/web/api.py`, `app/jobs/sources.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_sources.py` |
| 2b By-hand editing | `wp2b-byhand-ui` | Opus | `app/static/watchlist.js`, `app/templates/watchlist.html`, `app/static/app.css`, `tests/test_pages.py` |
Wave 1, in parallel, from `main` after WP0 is merged. Each owns its modules in `plugins/atshosts/`, `tests/test_ats_<host>.py` and `tests/fixtures/ats/<host>*`, and nothing else:
| WP | Branch | Model | Hosts |
|---|---|---|---|
| 3 | `wp3-workday` | Sonnet | Workday |
| 4 | `wp4-successfactors` | Sonnet | SuccessFactors |
| 5 | `wp5-oracle-phenom` | Sonnet | Oracle, Phenom |
| 6 | `wp6-json-hosts` | Sonnet | Eightfold, Jibe, Pinpoint, BambooHR, Gem |
| 7 | `wp7-small-hosts` | Sonnet | Deel, Jobylon, HR-ON, HR Manager, Rippling, HiBob |
The sitemap reader (job URLs from a sitemap, the role from the page's JSON-LD or microdata; Arm, Mambu, Odoo, Aiven, AUTO1, ASML) moved from wave 1 to wave 2. A sitemap lists URLs only, so the title would come from the role's own page, and `detail` may not set a title, since the listing rewrites it on every pull. The contract needs a rule for a listing that knows no title before that reader is built.
Wave 2 is decided after wave 1 by the roles each company would add: a module per company for the own JSON APIs (Ørsted, Nordea, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons) and the own HTML lists (Zalando, Netcompany, Google with DeepMind, Bolt, CHECK24). Until then these companies are on the by-hand list with reason `unread`.
**WP0.** The eight hosts move into modules with no change in behaviour: the existing tests pass with edits to imports only. `careers_url`, `is_known_target`, `_allowed_host` and `_paged` lose their Teamtailor branches to the module. `_get` takes `headers`. `detail` and the robots check are added to the shared code with tests against a fake host. `discover.py` takes `READABLE`, `HOSTS`, `_PROBES` and the body checks of `probe` from the modules; the hosts it names but cannot read stay in a table of its own. Discovery gains Lever's EU host. The tests that use `workday:` or `successfactors` as the unreadable host use `taleo` instead.
**WP1.** Migration 16 with `hand_reason` only, the endpoints `POST /api/companies`, `PATCH` and `DELETE`, `CompanyOut`'s new fields, `remove_company` taking the `agent_companies` rows, `careers_url` filled only when empty in `add_discovered` and `seed_companies`, the seed reading `hand_reason`, `check_careers_url` as the one rule for a typed link, and `app companies add`, `edit`, `checked`, `by-hand` and `remove` in the CLI, where `checked` stamps `careers_opened_at` and `by-hand` lists the never opened first. The `workday:` and `successfactors` fixtures in its three test files become `taleo`.
**WP2b.** New, Edit and Remove on the Watchlist tab, which is the by-hand list. New puts a form on top of the list: name, careers link, note and reason, sent to `POST /api/companies`. A row gains Edit, which turns it into the same form in place (`PATCH`), and Remove, which asks first (`DELETE`, and a 409 is shown in the row). A row shows the reason and the note. New and Edit use the form pattern of the job boards section (`sources.js`: a `<template>`, the row becomes the form in place). The page lists a company by the rule it has (no shown board and not `from_search`, or a failing board), so a company added by hand appears at once. The Sources rail is left as it is. Until WP1 is merged the page is built against the contract above.
**WP3 to WP8.** One module per host as the contract says, with parser tests from a saved response, paging and completeness tests against a fake `get`, and the allow-list and slug tests. Each host's notes name the listing, the paging quirks and the fields.
## Planner, after the merges
- After WP1: write the survey into the live database and `seed/companies.json`: the verified `jobs_url` as `careers_url` for the 155, and the `hand_reason` of each company that stays by hand.
- Attach the 49 boards the survey found with `set_board`, which fetches each before it lands, and find the career host of the 9 Teamtailor companies left. Not attached: Wallbox and Metacore (empty boards, possibly stale), Glovo, Cognigy.
- After each host of wave 1: attach its boards, add it to `TRUSTED_POSTED_HOSTS` where its date is a posting date, and name it in `README.md`.
- Run a full pull on a copy of the live database before the live one.
- Read one Workday and one SuccessFactors board from the deployed server, since every Workday host is behind Cloudflare and the survey ran from this machine.
- Correct the seed: Yousign is Youtrust, Philips's link is the global site, Novonesis's tenant is `novonesis`, Booking.com and Arm are not on iCIMS.
- Done on 2026-09-29: the survey's links and reasons are in the live database and the seed, and 43 boards are attached (241 of 362 companies have one). Left on the Watchlist from the 49: Cognigy, Glovo and Contractbook (`group`), Metacore, Wallbox and Quantum Systems (`unread`).
- An unknown SmartRecruiters id answers an empty listing and not 404, so a mistyped slug reads as a complete board with no roles and would close a company's roles. `set_board` should refuse a SmartRecruiters board that has never listed a role, or the module should check the company resource first.
- Fold this file into `decisions.md` and delete it with `docs/plan-hosts/`.