Files
jobboard/docs/plan-hosts-and-by-hand.md
T

13 KiB

Plan: a module per host, new hosts, and the list of companies checked by hand

Planned 2026-09-29. Three waves of work packages with disjoint files, built in parallel worktrees made from main, merged by the planner. This file and docs/plan-hosts/ are folded into decisions.md and deleted when the work lands.

Why

164 of the 362 companies have no board attached: 9 on Teamtailor whose career host is not found yet (the other 23 were attached today), and 155 on a host the app cannot read or on no host it could name. They include LEGO, Novo Nordisk, Vestas, Danske Bank, SimCorp, Ørsted, Klarna and Zalando. A company the app cannot read is shown as "not readable yet" with a link that nobody can edit, so the owner has no list to work through by hand.

What the survey found

Nine Sonnet agents read every one of the 155 career sites on 2026-09-29. Their notes are in docs/plan-hosts/: workday.md, successfactors.md, otherhosts.md, own-1.md to own-4.md, and survey-own.json with one record per company (verified jobs_url, class, host, identifier, listing URL, roles seen, evidence). Raw responses for fixtures are in the session scratchpad, /tmp/claude-1000/-home-claude-cboxcode-jobboard/1734d897-7509-48fc-844c-efe768acbf7a/scratchpad/, and a builder copies the ones it needs, trimmed, into tests/fixtures/ats/.

Route Companies Examples
Board on a host the app reads already 49 + 9 Teamtailor Bosch, Wise, Ubisoft, Checkout.com, Lunar, DFDS, Voi, Remote
SuccessFactors 20 Novo Nordisk, Vestas, Carlsberg, Pandora, Coloplast, Scania, SAP, Tryg
Workday 19 LEGO, Zeiss, SimCorp, Zendesk, Unity, Saxo Bank, Red Hat
Oracle 4 Danske Bank, Nokia, Sinch, Milestone
Phenom 3 Philips, Just Eat Takeaway, King
Eightfold 3 Microsoft, Ericsson, Infineon
Eleven small hosts 15 Klarna (Deel), Booking.com (Jibe), sennder (Gem), Einride (Jobylon)
Sitemap and structured data 8 Arm, Mambu, Odoo, Aiven, AUTO1, ASML
Own JSON API 8 Ørsted, Nordea, Amazon, Vinted, Dynatrace
Own HTML list 10 Zalando, Netcompany, Google, Bolt, CHECK24
By hand about 25 Revolut, Meta, Maersk, Bang & Olufsen, Visma, HashiCorp

The 49 were missed because the slug is not the company's name (BoschGroup, jimdo.com, workatbackbase), the board is on a custom domain, the board is on Lever's EU host, or the careers page refuses a plain request.

Decisions

A host is a module. plugins/ats.py holds eight hosts in six parallel tables, and app/jobs/discover.py holds three more, so two builders adding a host edit the same lines. Each host moves into plugins/atshosts/<host>.py, and ats.py and discover.py build their tables from the package. The package is a sibling of ats.py and not plugins/ats/, since desk.sources.registry globs plugins/*.py and the tests patch names on the ats module.

A host is handed the fetch. A host module imports nothing from plugins.ats. Its functions receive get, which ats.py binds to its own _get at call time, so the allow-list, the redirect rule, the byte cap and the deadline stay in one place and a test that patches ats._get reaches every host.

A description read per role is read once. Workday, SuccessFactors and BambooHR state the description only on the role's own page. A host with a detail function has it called for roles not yet read in detail, at most DETAIL_CAP (200) per board per pull; the ids read are kept in ats:detailed:<target>. A known role is sent without a description on later pulls, which upsert_role ignores since an empty value never overwrites a filled column. Completeness is decided by the listing alone.

robots.txt is honoured. A host module names the page whose path must be allowed. The shared code reads the site's robots.txt at most once a day per board and fails a board that disallows it for *, with the reason "robots.txt disallows it". This takes Novonesis, Darktrace, Criteo and Bang & Olufsen off the automatic route. The SuccessFactors reader uses /sitemap.xml, /search/ and /job/ only, never /services/ or the career*.successfactors.eu back end.

No reader is built on a key lifted from a site's script. Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key that the page's JavaScript carries. They go on the by-hand list.

The by-hand list is the companies table, and the Watchlist tab is the list. A company is on it when it has no shown board and is not known from a search only. careers_url, notes, watch (the rating) and careers_opened_at exist already, the last being the date a person last looked at the careers page; migration 16 adds hand_reason. There is no checked_at, since opening the page from the Watchlist stamps the date already. No second table, since a company has one careers link and the list needs no history of checks.

A link typed by hand is kept. add_discovered and the seed fill careers_url only when it is empty.

A company has one board. DFDS has eleven Teamtailor boards by country, and Onfido, Tink and Glovo sit inside their owners' boards. DFDS gets its Danish board; the other three go on the by-hand list with reason group.

Contracts between packages

Host module, plugins/atshosts/<host>.py:

NAME = "workday"                       # the host of a target `workday:<slug>`
LABEL = "Workday"                      # the Sources page's select
PATTERNS = (r"https?://...",)          # discover: over a URL, group 1 is the slug

def slug_ok(slug: str) -> bool: ...    # strict; a slug that fails is no target
def allowed(slug: str) -> Callable[[str], bool]: ...   # hostnames a fetch may reach
def careers_url(slug: str) -> str: ... # the page a person opens
def robots_page(slug: str) -> str: ... # optional; the URL robots.txt must allow

# one of the two
def feed_url(slug: str) -> str: ...
def parse(body: bytes, slug: str) -> list[Row]: ...

def page(get: Get, slug: str, cursor: Any) -> Page: ...
def rows(decoded: Any, slug: str) -> list[Row]: ...

# optional
def detail(get: Get, slug: str, row: Row) -> Row: ...
def probe_url(slug: str) -> str: ...
def probe_ok(body: bytes) -> bool: ...

Get is get(url, *, method="GET", payload=None, headers=None) -> bytes. Row and Page are as in plugins/ats.py today. plugins/atshosts/__init__.py lists the modules with pkgutil.iter_modules, sorted by name, so a new host registers by existing.

Slugs: Workday lego.wd103/LEGO_External; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (jobs.teradyne.com/MiR); Oracle <host>/<siteNumber>.

Company JSON, in GET /api/companies: CompanyOut has domain, city, watch and careers_opened_at from the Watchlist, and gains notes, hand_reason and by_hand (bool, no board shown and not from_search). hand_reason is one of '', unread (readable, no parser yet), robots, blocked (the site answers a program with a challenge), key, group, gone, added (by hand or by an import, not looked at). from_search is false for a company with a hand_reason.

Endpoint Body Answer
POST /api/companies name, careers_url, notes, hand_reason (default added) 201 CompanyOut; 409 when the key is taken
PATCH /api/companies/{key} any of name, careers_url, notes, hand_reason CompanyOut; the key stays; 409 on name for a company with a board
DELETE /api/companies/{key} none 204; 409 when it has a board or open roles; its agent_companies rows go with it

POST /api/companies/{key}/careers and POST /api/companies/{key}/opened of the Watchlist stay and answer a Company; POST .../careers and PATCH check the link with the same function (check_careers_url in app/jobs/sources.py). careers_url is http or https, as safeUrl demands on the page. A person looking at the page is POST .../opened, which stamps careers_opened_at.

Work packages

Wave 0, in parallel, from today's main:

WP Branch Model Owns
0 Host modules wp0-hosts Opus plugins/ats.py, plugins/atshosts/, app/jobs/discover.py, tests/test_ats.py, tests/test_discover.py, tests/test_atshosts.py
1 By-hand backend wp1-byhand-api Sonnet app/models.py, app/storage/schema.py, app/storage/store.py, app/web/api.py, app/jobs/sources.py, app/cli.py, tests/test_storage.py, tests/test_web.py, tests/test_sources.py
2b By-hand editing wp2b-byhand-ui Opus app/static/watchlist.js, app/templates/watchlist.html, app/static/app.css, tests/test_pages.py

Wave 1, in parallel, from main after WP0 is merged. Each owns its modules in plugins/atshosts/, tests/test_ats_<host>.py and tests/fixtures/ats/<host>*, and nothing else:

WP Branch Model Hosts
3 wp3-workday Sonnet Workday
4 wp4-successfactors Sonnet SuccessFactors
5 wp5-oracle-phenom Sonnet Oracle, Phenom
6 wp6-json-hosts Sonnet Eightfold, Jibe, Pinpoint, BambooHR, Gem
7 wp7-small-hosts Sonnet Deel, Jobylon, HR-ON, HR Manager, Rippling, HiBob
8 wp8-sitemap Sonnet sitemap: job URLs from a sitemap, the role from the page's JSON-LD or microdata

Wave 2 is decided after wave 1 by the roles each company would add: a module per company for the own JSON APIs (Ørsted, Nordea, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons) and the own HTML lists (Zalando, Netcompany, Google with DeepMind, Bolt, CHECK24). Until then these companies are on the by-hand list with reason unread.

WP0. The eight hosts move into modules with no change in behaviour: the existing tests pass with edits to imports only. careers_url, is_known_target, _allowed_host and _paged lose their Teamtailor branches to the module. _get takes headers. detail and the robots check are added to the shared code with tests against a fake host. discover.py takes READABLE, HOSTS, _PROBES and the body checks of probe from the modules; the hosts it names but cannot read stay in a table of its own. Discovery gains Lever's EU host. The tests that use workday: or successfactors as the unreadable host use taleo instead.

WP1. Migration 16 with hand_reason only, the endpoints POST /api/companies, PATCH and DELETE, CompanyOut's new fields, remove_company taking the agent_companies rows, careers_url filled only when empty in add_discovered and seed_companies, the seed reading hand_reason, check_careers_url as the one rule for a typed link, and app companies add, edit, checked, by-hand and remove in the CLI, where checked stamps careers_opened_at and by-hand lists the never opened first. The workday: and successfactors fixtures in its three test files become taleo.

WP2b. New, Edit and Remove on the Watchlist tab, which is the by-hand list. New puts a form on top of the list: name, careers link, note and reason, sent to POST /api/companies. A row gains Edit, which turns it into the same form in place (PATCH), and Remove, which asks first (DELETE, and a 409 is shown in the row). A row shows the reason and the note. New and Edit use the form pattern of the job boards section (sources.js: a <template>, the row becomes the form in place). The page lists a company by the rule it has (no shown board and not from_search, or a failing board), so a company added by hand appears at once. The Sources rail is left as it is. Until WP1 is merged the page is built against the contract above.

WP3 to WP8. One module per host as the contract says, with parser tests from a saved response, paging and completeness tests against a fake get, and the allow-list and slug tests. Each host's notes name the listing, the paging quirks and the fields.

Planner, after the merges

  • After WP1: write the survey into the live database and seed/companies.json: the verified jobs_url as careers_url for the 155, and the hand_reason of each company that stays by hand.
  • Attach the 49 boards the survey found with set_board, which fetches each before it lands, and find the career host of the 9 Teamtailor companies left. Not attached: Wallbox and Metacore (empty boards, possibly stale), Glovo, Cognigy.
  • After each host of wave 1: attach its boards, add it to TRUSTED_POSTED_HOSTS where its date is a posting date, and name it in README.md.
  • Run a full pull on a copy of the live database before the live one.
  • Read one Workday and one SuccessFactors board from the deployed server, since every Workday host is behind Cloudflare and the survey ran from this machine.
  • Correct the seed: Yousign is Youtrust, Philips's link is the global site, Novonesis's tenant is novonesis, Booking.com and Arm are not on iCIMS.
  • Fold this file into decisions.md and delete it with docs/plan-hosts/.