diff --git a/docs/plan-hosts-and-by-hand.md b/docs/plan-hosts-and-by-hand.md new file mode 100644 index 0000000..de17121 --- /dev/null +++ b/docs/plan-hosts-and-by-hand.md @@ -0,0 +1,128 @@ +# Plan: a module per host, new hosts, and the list of companies checked by hand + +Planned 2026-09-29. Three waves of work packages with disjoint files, built in parallel worktrees made from `main`, merged by the planner. This file and `docs/plan-hosts/` are folded into `decisions.md` and deleted when the work lands. + +## Why + +187 of the 362 companies have no board attached: 32 on Teamtailor, whose parser landed today without the boards being attached, and 155 on a host the app cannot read or on no host it could name. They include LEGO, Novo Nordisk, Vestas, Danske Bank, SimCorp, Ørsted, Klarna and Zalando. A company the app cannot read is shown as "not readable yet" with a link that nobody can edit, so the owner has no list to work through by hand. + +## What the survey found + +Nine Sonnet agents read every one of the 155 career sites on 2026-09-29. Their notes are in `docs/plan-hosts/`: `workday.md`, `successfactors.md`, `otherhosts.md`, `own-1.md` to `own-4.md`, and `survey-own.json` with one record per company (verified `jobs_url`, class, host, identifier, listing URL, roles seen, evidence). Raw responses for fixtures are in the session scratchpad, `/tmp/claude-1000/-home-claude-cboxcode-jobboard/1734d897-7509-48fc-844c-efe768acbf7a/scratchpad/`, and a builder copies the ones it needs, trimmed, into `tests/fixtures/ats/`. + +| Route | Companies | Examples | +|---|---|---| +| Board on a host the app reads already | 49 + 32 Teamtailor | Bosch, Wise, Ubisoft, Checkout.com, Lunar, DFDS, Voi, Remote | +| SuccessFactors | 20 | Novo Nordisk, Vestas, Carlsberg, Pandora, Coloplast, Scania, SAP, Tryg | +| Workday | 19 | LEGO, Zeiss, SimCorp, Zendesk, Unity, Saxo Bank, Red Hat | +| Oracle | 4 | Danske Bank, Nokia, Sinch, Milestone | +| Phenom | 3 | Philips, Just Eat Takeaway, King | +| Eightfold | 3 | Microsoft, Ericsson, Infineon | +| Eleven small hosts | 15 | Klarna (Deel), Booking.com (Jibe), sennder (Gem), Einride (Jobylon) | +| Sitemap and structured data | 8 | Arm, Mambu, Odoo, Aiven, AUTO1, ASML | +| Own JSON API | 8 | Ørsted, Nordea, Amazon, Vinted, Dynatrace | +| Own HTML list | 10 | Zalando, Netcompany, Google, Bolt, CHECK24 | +| By hand | about 25 | Revolut, Meta, Maersk, Bang & Olufsen, Visma, HashiCorp | + +The 49 were missed because the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain, the board is on Lever's EU host, or the careers page refuses a plain request. + +## Decisions + +**A host is a module.** `plugins/ats.py` holds eight hosts in six parallel tables, and `app/jobs/discover.py` holds three more, so two builders adding a host edit the same lines. Each host moves into `plugins/atshosts/.py`, and `ats.py` and `discover.py` build their tables from the package. The package is a sibling of `ats.py` and not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module. + +**A host is handed the fetch.** A host module imports nothing from `plugins.ats`. Its functions receive `get`, which `ats.py` binds to its own `_get` at call time, so the allow-list, the redirect rule, the byte cap and the deadline stay in one place and a test that patches `ats._get` reaches every host. + +**A description read per role is read once.** Workday, SuccessFactors and BambooHR state the description only on the role's own page. A host with a `detail` function has it called for roles not yet read in detail, at most `DETAIL_CAP` (200) per board per pull; the ids read are kept in `ats:detailed:`. A known role is sent without a description on later pulls, which `upsert_role` ignores since an empty value never overwrites a filled column. Completeness is decided by the listing alone. + +**robots.txt is honoured.** A host module names the page whose path must be allowed. The shared code reads the site's robots.txt at most once a day per board and fails a board that disallows it for `*`, with the reason "robots.txt disallows it". This takes Novonesis, Darktrace, Criteo and Bang & Olufsen off the automatic route. The SuccessFactors reader uses `/sitemap.xml`, `/search/` and `/job/` only, never `/services/` or the `career*.successfactors.eu` back end. + +**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key that the page's JavaScript carries. They go on the by-hand list. + +**The by-hand list is the `companies` table.** A company is on it when it has no shown board and is not known from a search only. `careers_url` and `notes` exist already; migration 15 adds `hand_reason` and `checked_at`. No second table, since a company has one careers link and the list needs no history of checks. + +**A link typed by hand is kept.** `add_discovered` and the seed fill `careers_url` only when it is empty. + +**A company has one board.** DFDS has eleven Teamtailor boards by country, and Onfido, Tink and Glovo sit inside their owners' boards. DFDS gets its Danish board; the other three go on the by-hand list with reason `group`. + +## Contracts between packages + +Host module, `plugins/atshosts/.py`: + +```python +NAME = "workday" # the host of a target `workday:` +LABEL = "Workday" # the Sources page's select +PATTERNS = (r"https?://...",) # discover: over a URL, group 1 is the slug + +def slug_ok(slug: str) -> bool: ... # strict; a slug that fails is no target +def allowed(slug: str) -> Callable[[str], bool]: ... # hostnames a fetch may reach +def careers_url(slug: str) -> str: ... # the page a person opens +def robots_page(slug: str) -> str: ... # optional; the URL robots.txt must allow + +# one of the two +def feed_url(slug: str) -> str: ... +def parse(body: bytes, slug: str) -> list[Row]: ... + +def page(get: Get, slug: str, cursor: Any) -> Page: ... +def rows(decoded: Any, slug: str) -> list[Row]: ... + +# optional +def detail(get: Get, slug: str, row: Row) -> Row: ... +def probe_url(slug: str) -> str: ... +def probe_ok(body: bytes) -> bool: ... +``` + +`Get` is `get(url, *, method="GET", payload=None, headers=None) -> bytes`. `Row` and `Page` are as in `plugins/ats.py` today. `plugins/atshosts/__init__.py` lists the modules with `pkgutil.iter_modules`, sorted by name, so a new host registers by existing. + +Slugs: Workday `lego.wd103/LEGO_External`; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (`jobs.teradyne.com/MiR`); Oracle `/`. + +Company JSON, in `GET /api/companies`: `CompanyOut` gains `domain`, `notes`, `hand_reason`, `checked_at` and `by_hand` (bool). `hand_reason` is one of `''`, `unread` (readable, no parser yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone`, `added` (by hand or by an import, not looked at). `from_search` is false for a company with a `hand_reason`. + +| Endpoint | Body | Answer | +|---|---|---| +| `POST /api/companies` | `name`, `careers_url`, `notes`, `hand_reason` (default `added`) | 201 `CompanyOut`; 409 when the key is taken | +| `PATCH /api/companies/{key}` | any of `name`, `careers_url`, `notes`, `hand_reason` | `CompanyOut`; the key stays; 409 on `name` for a company with a board | +| `POST /api/companies/{key}/checked` | none | `CompanyOut` with `checked_at` now | +| `DELETE /api/companies/{key}` | none | 204; 409 when it has a board or open roles; its `agent_companies` rows go with it | + +`careers_url` is http or https, as `safeUrl` demands on the page. + +## Work packages + +Wave 0, in parallel, from today's `main`: + +| WP | Branch | Model | Owns | +|---|---|---|---| +| 0 Host modules | `wp0-hosts` | Opus | `plugins/ats.py`, `plugins/atshosts/`, `app/jobs/discover.py`, `tests/test_ats.py`, `tests/test_discover.py`, `tests/test_atshosts.py` | +| 1 By-hand backend | `wp1-byhand-api` | Sonnet | `app/models.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/web/api.py`, `app/jobs/sources.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_sources.py` | +| 2 By-hand page | `wp2-byhand-ui` | Opus | `app/templates/settings/sources.html`, `app/static/byhand.js`, `app/static/companies.js`, `app/static/app.css`, `app/web/pages.py` | + +Wave 1, in parallel, from `main` after WP0 is merged. Each owns its modules in `plugins/atshosts/`, `tests/test_ats_.py` and `tests/fixtures/ats/*`, and nothing else: + +| WP | Branch | Model | Hosts | +|---|---|---|---| +| 3 | `wp3-workday` | Sonnet | Workday | +| 4 | `wp4-successfactors` | Sonnet | SuccessFactors | +| 5 | `wp5-oracle-phenom` | Sonnet | Oracle, Phenom | +| 6 | `wp6-json-hosts` | Sonnet | Eightfold, Jibe, Pinpoint, BambooHR, Gem | +| 7 | `wp7-small-hosts` | Sonnet | Deel, Jobylon, HR-ON, HR Manager, Rippling, HiBob | +| 8 | `wp8-sitemap` | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | + +Wave 2 is decided after wave 1 by the roles each company would add: a module per company for the own JSON APIs (Ørsted, Nordea, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons) and the own HTML lists (Zalando, Netcompany, Google with DeepMind, Bolt, CHECK24). Until then these companies are on the by-hand list with reason `unread`. + +**WP0.** The eight hosts move into modules with no change in behaviour: the existing tests pass with edits to imports only. `careers_url`, `is_known_target`, `_allowed_host` and `_paged` lose their Teamtailor branches to the module. `_get` takes `headers`. `detail` and the robots check are added to the shared code with tests against a fake host. `discover.py` takes `READABLE`, `HOSTS`, `_PROBES` and the body checks of `probe` from the modules; the hosts it names but cannot read stay in a table of its own. Discovery gains Lever's EU host. The tests that use `workday:` or `successfactors` as the unreadable host use `taleo` instead. + +**WP1.** Migration 15, the four endpoints, `CompanyOut`'s new fields, `remove_company` taking the `agent_companies` rows, `careers_url` filled only when empty in `add_discovered` and `seed_companies`, the seed reading `hand_reason`, and `app companies add`, `edit`, `checked` and `remove` in the CLI. The `workday:` and `successfactors` fixtures in its three test files become `taleo`. + +**WP2.** A section "Check by hand" in the Sources rail, after Company boards: a count, a New company button, and rows sorted by `checked_at`, never checked first. A row shows the name, the affinity control, the reason, the careers link, the note, "checked 3 days ago", and Checked, Edit and Remove. New and Edit use the form pattern of the job boards section (`sources.js`: a `