Fold the hosts plan into the decisions, and plan the readers that are left

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
WolfandClaude Fable 5.1 committed 2026-09-29 20:24:01 +02:00
1 parent b3f9bfcd9d
commit 7785e62b0d
9 files changed
+74 -949

No files matched your search

+16
View File
@@ -96,6 +96,20 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**A board that fails on every fetch for 21 days closes its roles.** A board whose fetch raised was only logged, so a company that moved ATS kept its roles open for good behind a 404. The `ats` plugin now records `ats:failing:<target>` in `state` with the time of the first failure in a row and the latest reason, and a fetch that answers removes it. Once the first failure is `STALE_DAYS` old, the same window a search role gets, the settle closes the board's roles; the next answer reopens what it lists through `upsert_role`, as for a board switched back on. The record is plugin state rather than a column because the plugin only sees `self.state`. `/api/companies` returns it as `failing_since` and `failing_reason`, the Companies list shows "not answering since", and `app companies check` prints it beside its live result.
**A host is a module in `plugins/atshosts/`, and registers by existing.** `plugins/ats.py` held eight hosts in six parallel tables and `app/jobs/discover.py` three more, so two people adding a host edited the same lines. Each host is now one file that states its name, label, URL patterns, slug check, allowed host names and either a feed and a parser or a pager; `ats.py` and `discover.py` build their tables from the package, ordered by the module's `RANK` so the select and the probes kept their order. The package sits beside `ats.py` and is not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module. The contract is the docstring of `plugins/atshosts/__init__.py`, and a module that breaks it fails the import by name.
**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
**A description stated only on the role's own page is read once, 200 roles per board per pull.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. A module's `detail` is called for roles not yet read, those never tried before those that failed, and the ids read are kept in `ats:detailed:<target>`; a known role is sent without a description on later pulls, which `upsert_role` ignores. `detail` may fill `DETAIL_FIELDS` only (description, location, remote, currency, salary_min, salary_max, posted), the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing, since the listing rewrites them every pull. Completeness is decided by the listing alone, so a board of 800 roles closes its roles from the first pull and has all its text after four.
**robots.txt is honoured, by RFC 9309.** A module names the page a person opens, and a board whose site disallows that page for `*` or for the app's own token fails with "robots.txt disallows it" and closes its roles after 21 days like any failing board. The matcher takes the longest rule, an Allow on a tie, `*` and `$`, and case-sensitive paths, or paths without case for a host that serves any case, as Workday does. A 4xx means no rules; a 5xx, a 429 or a timeout fails the board for that pull. `urllib.robotparser` was replaced because it takes the first rule and knows no wildcard, which refused Infineon and let `/*company` through. This keeps Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy and Umbraco off the automatic route. The SuccessFactors reader uses `/search/` and `/job/` only, never `/services/` or a `career*.successfactors.eu` back end.
**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key the page's JavaScript carries, which can change without notice and marks an endpoint that is not open. They are on the Watchlist with reason `key`.
**A company has one board, and a company inside its owner's board is read by hand.** DFDS has eleven Teamtailor boards by country and is read from the Danish one. Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio and Native Instruments list their roles in Entrust's, Visa's, Delivery Hero's, NICE's, Scrive's, SAP's and inMusic's boards, where no field separates them reliably; they carry reason `group`.
**The seed's gap was mostly discovery, not parsers.** A survey of the 155 companies without a board (2026-09-29, every careers site opened) found 49 on a host the app already read: the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain or on Lever's EU instance (`lever:eu/<site>`), or the careers page refuses a plain request. 43 were attached by hand from the survey, 46 more with the fourteen hosts added that day, and 286 of 362 companies have a board. Discovery keeps a slug's case for a module with `KEEP_CASE`, since Workday's site names, Gem's, Deel's and Rippling's slugs are case-sensitive; SmartRecruiters and Ashby are not, checked live.
**`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs.
## Ranking
@@ -154,6 +168,8 @@ What was decided for roledesk and the one reason for each. The longer reasoning
**Watchlist is a third tab: the companies the app cannot read, rated yes or no.** The owner goes through the careers pages of such companies by hand and wanted that list one click from Roles rather than behind the cog, so `TABS` has three entries and the phone bar's tabs are tighter. A company is on it when it has no board subscription and is not known from a search only, or when its board is failing: the rule of the Company boards filter "Not readable yet", read from `GET /api/companies`. The rating is `companies.watch` (migration 15: NULL unrated, 1 on the list, 0 not), global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. A row rated under Unrated leaves the list at once. On a phone the search field of Roles is left a few letters wide by the third tab, so while it has the focus the tabs and the queue chip step aside.
**A company on the Watchlist says why it is there, and can be added, edited and removed on the tab.** `companies.hand_reason` (migration 16) is one of `unread` (readable, no module yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone` and `added` (by hand or by an import, not looked at); attaching a board clears it. `POST`, `PATCH` and `DELETE /api/companies` and `app companies add`, `edit`, `checked`, `remove` and `by-hand` are what the LinkedIn import will write through. A company with a reason is never "known from a search only", so one added with a name alone is listed. No second table and no `checked_at`: a company has one careers link, and `careers_opened_at` already says when somebody last looked. A careers link somebody typed is kept: discovery and the seed fill it only when empty. A name is changed only while the company has no board, since the subscription is found by the key the name folds to.
**Settings are three sub-tabs: Search agents, Sources, Account.** Six sub-tabs were three setup pages and three system pages for two things, what the app ranks by and where roles come from, so each is one page now and Account stays its own.
**The old paths' 301s are gone.** `MOVED` answered 301 for the twelve paths above, kept for the few days between the move and this cut, on the assumption that a bookmark should still land. A browser caches a 301 indefinitely, so a path a future page reused would be unreachable through no fault of that page; a one-person app with no outstanding bookmarks does not need to carry that risk.
+9
View File
@@ -10,3 +10,12 @@ What is left after the phone rework of 2026-09-25, for whoever picks the repo up
- **Fit does not read seniority.** With three prompts on the live data of 2026-09-29 an individual contributor's posting ranks beside a head's on the same subject. Switch extract on so `seniority` fills, then add a prefer gate on it, and vote.
- **Nine Teamtailor seed companies have no board**, because their `careers_url` is the marketing site: Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial.
- **Votes do not train fit.** A vote only moves the rank at read time. Feeding votes into the prompt vectors is worth doing once there are a few hundred.
- **76 companies are read by hand**, on the Watchlist with a careers link opened on 2026-09-29: 35 `unread`, 10 `robots`, 9 Teamtailor ones without a career host, 8 `group`, 7 `blocked`, 4 `gone`, 3 `key`. The remaining readers are planned in `docs/plan-hosts-wave-2.md`.
- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them.
- **An unknown SmartRecruiters or Gem id answers an empty listing, not 404**, so a mistyped slug reads as a complete board with no roles and would close a company's roles. `set_board` reads the board first, which catches a typo only when the person notices "0 roles".
- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the shared reader waits `PAGE_DELAY`, 0.5. A read of its board is two requests.
- **Eightfold returns 10 roles a page and `MAX_PAGES` is 100**, so Infineon (1,289 roles) and Microsoft (about 2,300) would never be read complete and are not attached. The endpoint takes a location filter; a target has nowhere to state one.
- **A SuccessFactors site on a company's own domain is not discovered from its own pages**, which link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand.
- **HR Manager's portal disallows reading in robots.txt, its feed host has no robots.txt.** The module checks the feed's host, so a board would be read. Seaborg, the one company on it, is not attached until the owner decides whether the feed counts as allowed.
- **Six new hosts' dates are not trusted** (Eightfold, Jibe, Gem, Deel, HiBob, HR Manager; Pinpoint states none), so their backlog counts as posted when first seen. Check each date on a live board and add the host to `TRUSTED_POSTED_HOSTS`.
-151
View File
@@ -1,151 +0,0 @@
# Plan: a module per host, new hosts, and the list of companies checked by hand
Planned 2026-09-29. Three waves of work packages with disjoint files, built in parallel worktrees made from `main`, merged by the planner. This file and `docs/plan-hosts/` are folded into `decisions.md` and deleted when the work lands.
## Why
164 of the 362 companies have no board attached: 9 on Teamtailor whose career host is not found yet (the other 23 were attached today), and 155 on a host the app cannot read or on no host it could name. They include LEGO, Novo Nordisk, Vestas, Danske Bank, SimCorp, Ørsted, Klarna and Zalando. A company the app cannot read is shown as "not readable yet" with a link that nobody can edit, so the owner has no list to work through by hand.
## What the survey found
Nine Sonnet agents read every one of the 155 career sites on 2026-09-29. Their notes are in `docs/plan-hosts/`: `workday.md`, `successfactors.md`, `otherhosts.md`, `own-1.md` to `own-4.md`, and `survey-own.json` with one record per company (verified `jobs_url`, class, host, identifier, listing URL, roles seen, evidence). Raw responses for fixtures are in the session scratchpad, `/tmp/claude-1000/-home-claude-cboxcode-jobboard/1734d897-7509-48fc-844c-efe768acbf7a/scratchpad/`, and a builder copies the ones it needs, trimmed, into `tests/fixtures/ats/`.
| Route | Companies | Examples |
|---|---|---|
| Board on a host the app reads already | 49 + 9 Teamtailor | Bosch, Wise, Ubisoft, Checkout.com, Lunar, DFDS, Voi, Remote |
| SuccessFactors | 20 | Novo Nordisk, Vestas, Carlsberg, Pandora, Coloplast, Scania, SAP, Tryg |
| Workday | 19 | LEGO, Zeiss, SimCorp, Zendesk, Unity, Saxo Bank, Red Hat |
| Oracle | 4 | Danske Bank, Nokia, Sinch, Milestone |
| Phenom | 3 | Philips, Just Eat Takeaway, King |
| Eightfold | 3 | Microsoft, Ericsson, Infineon |
| Eleven small hosts | 15 | Klarna (Deel), Booking.com (Jibe), sennder (Gem), Einride (Jobylon) |
| Sitemap and structured data | 8 | Arm, Mambu, Odoo, Aiven, AUTO1, ASML |
| Own JSON API | 8 | Ørsted, Nordea, Amazon, Vinted, Dynatrace |
| Own HTML list | 10 | Zalando, Netcompany, Google, Bolt, CHECK24 |
| By hand | about 25 | Revolut, Meta, Maersk, Bang & Olufsen, Visma, HashiCorp |
The 49 were missed because the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain, the board is on Lever's EU host, or the careers page refuses a plain request.
## Decisions
**A host is a module.** `plugins/ats.py` holds eight hosts in six parallel tables, and `app/jobs/discover.py` holds three more, so two builders adding a host edit the same lines. Each host moves into `plugins/atshosts/<host>.py`, and `ats.py` and `discover.py` build their tables from the package. The package is a sibling of `ats.py` and not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module.
**A host is handed the fetch.** A host module imports nothing from `plugins.ats`. Its functions receive `get`, which `ats.py` binds to its own `_get` at call time, so the allow-list, the redirect rule, the byte cap and the deadline stay in one place and a test that patches `ats._get` reaches every host.
**A description read per role is read once.** Workday, SuccessFactors and BambooHR state the description only on the role's own page. A host with a `detail` function has it called for roles not yet read in detail, at most `DETAIL_CAP` (200) per board per pull; the ids read are kept in `ats:detailed:<target>`. A known role is sent without a description on later pulls, which `upsert_role` ignores since an empty value never overwrites a filled column. Completeness is decided by the listing alone.
**robots.txt is honoured.** A host module names the page whose path must be allowed. The shared code reads the site's robots.txt at most once a day per board and fails a board that disallows it for `*`, with the reason "robots.txt disallows it". This takes Novonesis, Darktrace, Criteo and Bang & Olufsen off the automatic route. The SuccessFactors reader uses `/sitemap.xml`, `/search/` and `/job/` only, never `/services/` or the `career*.successfactors.eu` back end.
**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key that the page's JavaScript carries. They go on the by-hand list.
**The by-hand list is the `companies` table, and the Watchlist tab is the list.** A company is on it when it has no shown board and is not known from a search only. `careers_url`, `notes`, `watch` (the rating) and `careers_opened_at` exist already, the last being the date a person last looked at the careers page; migration 16 adds `hand_reason`. There is no `checked_at`, since opening the page from the Watchlist stamps the date already. No second table, since a company has one careers link and the list needs no history of checks.
**A link typed by hand is kept.** `add_discovered` and the seed fill `careers_url` only when it is empty.
**A company has one board.** DFDS has eleven Teamtailor boards by country, and Onfido, Tink and Glovo sit inside their owners' boards. DFDS gets its Danish board; the other three go on the by-hand list with reason `group`.
## Contracts between packages
Host module, `plugins/atshosts/<host>.py`, as built by WP0. The docstring of `plugins/atshosts/__init__.py` holds the same contract and is the one to keep current. A module imports `desk` and `._shared` only, never `plugins.ats`.
```python
NAME = "workday" # the host of a target `workday:<slug>`; equals the module's file name
LABEL = "Workday" # the Sources page's select
PATTERNS = (r"https?://...",) # discovery: regexes over a URL, compiled with IGNORECASE
RANK = 100 # optional int, default 100; lower comes first
KEEP_CASE = True # optional, default False; keep a discovered slug's case
ROBOTS_ANY_CASE = True # optional, default False; read robots.txt paths without case
def slug_ok(slug: str) -> bool: ... # strict, re.fullmatch
def allowed(slug: str) -> Callable[[str], bool]: ... # host names a fetch may reach
def careers_url(slug: str) -> str: ... # the page a person opens
# one of the two
def feed_url(slug: str) -> str: ... # the whole board in one answer
def parse(body: bytes, slug: str) -> list[Row]: ...
def page(get: Get, slug: str, cursor: Any) -> Page: ... # a paged board, one page a call
def rows(decoded: Any, slug: str) -> list[Row]: ...
# optional
def fallback_url(slug: str) -> str: ... # feed hosts: a smaller feed, read once when feed_url is over the byte cap
def robots_page(slug: str) -> str: ... # the URL robots.txt must allow; on a host allowed(slug) admits
def detail(get: Get, slug: str, row: Row) -> dict[str, Any]: ... # fields from the role's own page
def probe_url(slug: str) -> str: ... # discovery: answers 2xx only for a board
def probe_ok(body: bytes) -> bool: ... # discovery: when 2xx is not enough
def probe_slugs(guess: str) -> tuple[str, ...]: ... # the slugs a guessed name is probed as; default (guess,)
```
`Get`, `Row` and `Page` are in `plugins/atshosts/_shared.py`, with the helpers several hosts share: `only(host)` for `allowed`, `path_slug_ok` and `label_slug_ok` for `slug_ok`, `parse_dt`, `is_web_url`, `num`, `yearly_amount` and `pay_line`. `plugins/atshosts/__init__.py` lists the modules with `pkgutil.iter_modules`, checks each against the contract at import (an ImportError names the module and what it lacks), and orders `HOSTS` by `(RANK, NAME)`. The eight first hosts hold ranks 10 to 80, which keeps the order they had before the modules: the select opens on Greenhouse, and a guessed slug on two hosts goes to the one probed first.
Order and names. `RANK` orders the Sources page's select, the plugin's description and discovery's probes. `NAME` equals the file name. A host that `discover.py` names in its own `_UNREADABLE` table is taken from its module as soon as the module exists, so a wave 1 package does not edit `discover.py`.
Patterns. The slug is group 1, or, when a pattern has more groups, the groups that matched joined by "/" (`https?://([a-z0-9-]+\.wd\d+)\.myworkdayjobs\.com/(?:[a-z]{2}-[A-Z]{2}/)?([A-Za-z0-9_-]+)` gives `lego.wd103/LEGO_External`). Only group 1 is checked against discovery's words that name no board (`api`, `jobs`, `embed`, ...). The slug is lowercased unless the module sets `KEEP_CASE = True`, which a host with case-sensitive paths needs, since robots.txt paths are case-sensitive too. A pattern without a group names the host with no slug.
Slugs. `slug_ok` uses `re.fullmatch` (a `$` lets a trailing newline through) and refuses anything that could add a path, query, fragment, port or user to the URLs built from it. A slug of several parts joins them with "/": Lever's EU boards are `lever:eu/<site>`.
`get(url, *, method="GET", payload=None, headers=None) -> bytes` is `ats._get` bound to the board. A POST sends `payload` as JSON; `headers` are added to the default User-Agent and Accept and replace either where they name it. It fetches https on port 443 only, never an IP address or a local name (`localhost`, `.local`, `.internal`, `.lan`, `.home.arpa`), follows at most three redirects each checked against `allowed(slug)`, cuts the body at `MAX_BYTES` and ends within `DEADLINE`. It returns the body and raises `httpx.HTTPStatusError` on a non-2xx answer, `httpx.HTTPError` on a timeout or a connection failure, and another exception, `ValueError` among them, for a refused host, an oversized body or a broken redirect. A module lets these through.
`Page` is `(decoded, results, total, cursor)`: the decoded page handed to `rows`, its raw results as dicts with an `id` (counted by distinct id for completeness), the total the host states or None, and the next cursor or None after the last page. The first call gets cursor None. A board with a total is complete when as many distinct ids were read, one without when the read ended on an empty page; a 429 after the first page ends the read, incomplete. A paged module may also have `parse(body, slug)` for one page's body, for its tests.
A `Row` has `source_id`, `title`, `location`, `url`, `posted` (aware datetime or None), `department`, `description` (plain text), `remote` (True, False, a word or None) and `salary_text`, and, where the host states pay, `salary_min`, `salary_max` (yearly integers) and `currency`.
`detail` returns a dict of what the role's own page states. The shared code calls it for roles not yet read (`ats:detailed:<target>`), at most `DETAIL_CAP` (200) per board per pull, `PAGE_DELAY` apart, the roles never tried before those whose detail failed (`ats:detailfailed:<target>`). Only `DETAIL_FIELDS` are taken: `description`, `location`, `remote`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the listing's empty value (`upsert_role`'s keep-filled columns, the salary figures, and `posted_at`, which keeps the earliest date). Any other key is dropped with a debug log line: `title`, `url`, `department` and `salary_text` cannot be filled from `detail`, since the listing rewrites them every pull. A role not read in detail on a pull is sent with an empty description. A 404 or 410 counts as read, any other failure is retried on a later pull, and a 429 ends the detail reads for that board and pull. Completeness is decided by the listing alone.
robots.txt. For a module with `robots_page`, the shared code reads `https://<its host>/robots.txt` before the listing, at most once a day per board (`ats:robots:<target>`), following a redirect to the host's `www.` twin or back and no other. It fails the board with "robots.txt disallows it" when the rules for `*` or for the app's own token disallow the page, matched per RFC 9309: the longest matching rule wins, Allow wins a tie, `*` and a final `$` in paths, paths case-sensitive, an empty Disallow allows. A 4xx other than 429 means there are no rules, which allows. A 5xx, a 429, a timeout, a connection error or a redirect elsewhere fails the board with "robots.txt could not be read", and the next pull asks again.
Slugs: Workday `lego.wd103/LEGO_External`; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (`jobs.teradyne.com/MiR`); Oracle `<host>/<siteNumber>`.
Company JSON, in `GET /api/companies`: `CompanyOut` has `domain`, `city`, `watch` and `careers_opened_at` from the Watchlist, and gains `notes`, `hand_reason` and `by_hand` (bool, no board shown and not `from_search`). `hand_reason` is one of `''`, `unread` (readable, no parser yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone`, `added` (by hand or by an import, not looked at). `from_search` is false for a company with a `hand_reason`.
| Endpoint | Body | Answer |
|---|---|---|
| `POST /api/companies` | `name`, `careers_url`, `notes`, `hand_reason` (default `added`) | 201 `CompanyOut`; 409 when the key is taken |
| `PATCH /api/companies/{key}` | any of `name`, `careers_url`, `notes`, `hand_reason` | `CompanyOut`; the key stays; 409 on `name` for a company with a board |
| `DELETE /api/companies/{key}` | none | 204; 409 when it has a board or open roles; its `agent_companies` rows go with it |
`POST /api/companies/{key}/careers` and `POST /api/companies/{key}/opened` of the Watchlist stay and answer a `Company`; `POST .../careers` and `PATCH` check the link with the same function (`check_careers_url` in `app/jobs/sources.py`). `careers_url` is http or https, as `safeUrl` demands on the page. A person looking at the page is `POST .../opened`, which stamps `careers_opened_at`.
## Work packages
Wave 0, in parallel, from today's `main`:
| WP | Branch | Model | Owns |
|---|---|---|---|
| 0 Host modules | `wp0-hosts` | Opus | `plugins/ats.py`, `plugins/atshosts/`, `app/jobs/discover.py`, `tests/test_ats.py`, `tests/test_discover.py`, `tests/test_atshosts.py` |
| 1 By-hand backend | `wp1-byhand-api` | Sonnet | `app/models.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/web/api.py`, `app/jobs/sources.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_sources.py` |
| 2b By-hand editing | `wp2b-byhand-ui` | Opus | `app/static/watchlist.js`, `app/templates/watchlist.html`, `app/static/app.css`, `tests/test_pages.py` |
Wave 1, in parallel, from `main` after WP0 is merged. Each owns its modules in `plugins/atshosts/`, `tests/test_ats_<host>.py` and `tests/fixtures/ats/<host>*`, and nothing else:
| WP | Branch | Model | Hosts |
|---|---|---|---|
| 3 | `wp3-workday` | Sonnet | Workday |
| 4 | `wp4-successfactors` | Sonnet | SuccessFactors |
| 5 | `wp5-oracle-phenom` | Sonnet | Oracle, Phenom |
| 6 | `wp6-json-hosts` | Sonnet | Eightfold, Jibe, Pinpoint, BambooHR, Gem |
| 7 | `wp7-small-hosts` | Sonnet | Deel, Jobylon, HR-ON, HR Manager, Rippling, HiBob |
The sitemap reader (job URLs from a sitemap, the role from the page's JSON-LD or microdata; Arm, Mambu, Odoo, Aiven, AUTO1, ASML) moved from wave 1 to wave 2. A sitemap lists URLs only, so the title would come from the role's own page, and `detail` may not set a title, since the listing rewrites it on every pull. The contract needs a rule for a listing that knows no title before that reader is built.
Wave 2 is decided after wave 1 by the roles each company would add: a module per company for the own JSON APIs (Ørsted, Nordea, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons) and the own HTML lists (Zalando, Netcompany, Google with DeepMind, Bolt, CHECK24). Until then these companies are on the by-hand list with reason `unread`.
**WP0.** The eight hosts move into modules with no change in behaviour: the existing tests pass with edits to imports only. `careers_url`, `is_known_target`, `_allowed_host` and `_paged` lose their Teamtailor branches to the module. `_get` takes `headers`. `detail` and the robots check are added to the shared code with tests against a fake host. `discover.py` takes `READABLE`, `HOSTS`, `_PROBES` and the body checks of `probe` from the modules; the hosts it names but cannot read stay in a table of its own. Discovery gains Lever's EU host. The tests that use `workday:` or `successfactors` as the unreadable host use `taleo` instead.
**WP1.** Migration 16 with `hand_reason` only, the endpoints `POST /api/companies`, `PATCH` and `DELETE`, `CompanyOut`'s new fields, `remove_company` taking the `agent_companies` rows, `careers_url` filled only when empty in `add_discovered` and `seed_companies`, the seed reading `hand_reason`, `check_careers_url` as the one rule for a typed link, and `app companies add`, `edit`, `checked`, `by-hand` and `remove` in the CLI, where `checked` stamps `careers_opened_at` and `by-hand` lists the never opened first. The `workday:` and `successfactors` fixtures in its three test files become `taleo`.
**WP2b.** New, Edit and Remove on the Watchlist tab, which is the by-hand list. New puts a form on top of the list: name, careers link, note and reason, sent to `POST /api/companies`. A row gains Edit, which turns it into the same form in place (`PATCH`), and Remove, which asks first (`DELETE`, and a 409 is shown in the row). A row shows the reason and the note. New and Edit use the form pattern of the job boards section (`sources.js`: a `<template>`, the row becomes the form in place). The page lists a company by the rule it has (no shown board and not `from_search`, or a failing board), so a company added by hand appears at once. The Sources rail is left as it is. Until WP1 is merged the page is built against the contract above.
**WP3 to WP8.** One module per host as the contract says, with parser tests from a saved response, paging and completeness tests against a fake `get`, and the allow-list and slug tests. Each host's notes name the listing, the paging quirks and the fields.
## Planner, after the merges
- After WP1: write the survey into the live database and `seed/companies.json`: the verified `jobs_url` as `careers_url` for the 155, and the `hand_reason` of each company that stays by hand.
- Attach the 49 boards the survey found with `set_board`, which fetches each before it lands, and find the career host of the 9 Teamtailor companies left. Not attached: Wallbox and Metacore (empty boards, possibly stale), Glovo, Cognigy.
- After each host of wave 1: attach its boards, add it to `TRUSTED_POSTED_HOSTS` where its date is a posting date, and name it in `README.md`.
- Run a full pull on a copy of the live database before the live one.
- Read one Workday and one SuccessFactors board from the deployed server, since every Workday host is behind Cloudflare and the survey ran from this machine.
- Correct the seed: Yousign is Youtrust, Philips's link is the global site, Novonesis's tenant is `novonesis`, Booking.com and Arm are not on iCIMS.
- Done on 2026-09-29: the survey's links and reasons are in the live database and the seed, and 43 boards are attached (241 of 362 companies have one). Left on the Watchlist from the 49: Cognigy, Glovo and Contractbook (`group`), Metacore, Wallbox and Quantum Systems (`unread`).
- An unknown SmartRecruiters id answers an empty listing and not 404, so a mistyped slug reads as a complete board with no roles and would close a company's roles. `set_board` should refuse a SmartRecruiters board that has never listed a role, or the module should check the company resource first.
- Fold this file into `decisions.md` and delete it with `docs/plan-hosts/`.
+49
View File
@@ -0,0 +1,49 @@
# Plan: the readers left after the hosts of 2026-09-29
Planned 2026-09-29, after wave 0 and wave 1 of the hosts plan landed. What landed is in `decisions.md` under Ingestion and The phone; what is open without a plan is in `open.md`. This file and `docs/plan-hosts/` are deleted when the packages below land or are dropped.
## Where it stands
286 of 362 companies have a board. 76 are on the Watchlist, each with the careers page that was opened on 2026-09-29 and, but for nine, a reason. 35 of them carry `unread`: a program can read them and nothing does yet.
| Route | Companies | Roles seen |
|---|---|---|
| Sitemap and structured data | Arm, Mambu, Odoo, Aiven, SOPHiA GENETICS, AUTO1, Apple, ASML | about 2,000 |
| Own JSON API | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | about 570 |
| Own HTML list | Zalando, Netcompany, Bolt, CHECK24, Google with DeepMind, Lucca, Shopify, Framer, Bird, Ableton | about 950 |
| A host that is read, with a gap | Infineon, Microsoft (Eightfold's page cap), Tryg, DNB, Allegro (SuccessFactors), Quantum Systems (Recruitee on a custom domain), Siemens (Avature), Metacore, Wallbox (empty boards) | |
| Teamtailor, career host not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial | |
The survey's record of each company is in `docs/plan-hosts/survey-own.json` (class, host, identifier, listing URL, roles seen, the request that showed it), with a paragraph per host in `own-1.md` to `own-4.md` and the smaller hosts in `otherhosts.md`. The raw responses were not kept; a builder fetches its own fixtures.
## Decisions to take first
**A listing that knows no title.** A sitemap lists URLs. `detail` may not set a title, since the listing rewrites it on every pull, so a sitemap reader would show a title made from the URL. Two ways out: the module reads the role pages inside `page`, which costs one request per role on every pull, or the contract gains a row key that marks the title as provisional, and `upsert_role` keeps a stored title over a provisional one. The second is the smaller cost per pull and touches `app/storage/store.py`.
**A filter in a target.** Eightfold takes a location, Workday and SuccessFactors a country facet. A target is `host:slug` and has nowhere to state one. `subscriptions.params` exists for searches; a board could use it, and `page` would receive it. This brings Infineon and Microsoft under the page cap and cuts a pull of Bosch (4,812 roles), Philips or SAP to the countries the agents accept.
**One module per company.** An own API or an own HTML list is read by a module named after the company, with a fixed slug. Fourteen hosts landed as fourteen files without a shared edit, so the package takes a module per company as it is; `RANK` puts them after the hosts, and they have no `PATTERNS`, since discovery finds nothing a slug could be made from.
## Work packages
Each owns its modules in `plugins/atshosts/`, `tests/test_ats_<name>.py` and `tests/fixtures/ats/<name>*`, and nothing else. The common rules of wave 1 apply: the live request limits, robots.txt read first, no key from a script, a final report with the targets to attach.
| WP | Model | Builds | Needs |
|---|---|---|---|
| 9 Provisional title | Opus | the row key, `upsert_role`'s rule and its tests; owns `plugins/ats.py`, `plugins/atshosts/__init__.py`, `app/storage/store.py`, `tests/test_atshosts.py`, `tests/test_storage.py` | the first decision |
| 10 Sitemap reader | Sonnet | `sitemap`: job URLs from a sitemap, the role from the page's JSON-LD or microdata | WP9 |
| 11 Board filters | Opus | `params` on a board subscription, handed to `page`; owns `plugins/ats.py`, `app/jobs/pull.py`, `app/jobs/sources.py`, `app/web/api.py`, `app/static/companies.js` and their tests | the second decision |
| 12 Own APIs | Sonnet | Ørsted, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons | |
| 13 Own lists, Nordic and German | Sonnet | Zalando, Netcompany, CHECK24, Bolt | |
| 14 Own lists, the rest | Sonnet | Google with DeepMind, Lucca, Shopify; Framer, Bird and Ableton only if an afternoon is left | |
| 15 Gaps in hosts that are read | Sonnet | Recruitee on a custom domain; the SuccessFactors pages of Tryg, DNB and Allegro; Avature for Siemens | |
WP12 to WP15 can start at once and in parallel. WP10 waits for WP9, and WP9 and WP11 both own `plugins/ats.py`, so they run one after the other.
## Planner, after the merges
- Attach each board with `app companies set-board`, which reads it first and clears the reason.
- Run `scratchpad/sync_seed.py`'s rule by hand or again: every board shown in the live database into `seed/companies.json`, every reason of a company without one.
- Add a host to `TRUSTED_POSTED_HOSTS` where its date is a posting date, with the evidence in the comment.
- Restart the server by its PID after a backup in `../jobboard-backups/`.
- Find the career host of the nine Teamtailor companies by opening each careers page.
-145
View File
@@ -1,145 +0,0 @@
[
{
"key": "milestone-systems",
"name": "Milestone Systems",
"domain": "milestonesys.com",
"careers_url": "https://www.milestonesys.com/careers/",
"ats": "oracle",
"ats_slug": "",
"country": "DK",
"city": "Brøndby",
"status": "researching"
},
{
"key": "configit",
"name": "Configit",
"domain": "configit.com",
"careers_url": "https://configit.com/about/careers/",
"ats": "bamboohr",
"ats_slug": "",
"country": "DK",
"city": "Copenhagen",
"status": "researching"
},
{
"key": "xeneta",
"name": "Xeneta",
"domain": "xeneta.com",
"careers_url": "https://www.xeneta.com/",
"ats": "pinpoint",
"ats_slug": "",
"country": "NO",
"city": "Oslo",
"status": "researching"
},
{
"key": "native-instruments",
"name": "Native Instruments",
"domain": "native-instruments.com",
"careers_url": "https://www.native-instruments.com/pages/careers",
"ats": "pinpoint",
"ats_slug": "",
"country": "DE",
"city": "Berlin",
"status": "researching"
},
{
"key": "mambu",
"name": "Mambu",
"domain": "mambu.com",
"careers_url": "https://mambu.com/en/careers",
"ats": "icims",
"ats_slug": "",
"country": "DE",
"city": "Berlin",
"status": "researching"
},
{
"key": "booking-com",
"name": "Booking.com",
"domain": "booking.com",
"careers_url": "https://careers.booking.com",
"ats": "icims",
"ats_slug": "",
"country": "NL",
"city": "Amsterdam",
"status": "researching"
},
{
"key": "arm",
"name": "Arm",
"domain": "arm.com",
"careers_url": "https://careers.arm.com/",
"ats": "icims",
"ats_slug": "",
"country": "GB",
"city": "Cambridge",
"status": "researching"
},
{
"key": "sennder",
"name": "sennder",
"domain": "sennder.com",
"careers_url": "https://www.sennder.com/de/open-positions",
"ats": "gem",
"ats_slug": "",
"country": "DE",
"city": "Berlin",
"status": "researching"
},
{
"key": "productboard",
"name": "Productboard",
"domain": "productboard.com",
"careers_url": "https://www.productboard.com/careers/open-positions/",
"ats": "gem",
"ats_slug": "",
"country": "CZ",
"city": "Prague",
"status": "researching"
},
{
"key": "infineon",
"name": "Infineon",
"domain": "infineon.com",
"careers_url": "https://jobs.infineon.com/careers",
"ats": "eightfold",
"ats_slug": "",
"country": "DE",
"city": "Munich",
"status": "researching"
},
{
"key": "just-eat-takeaway-com",
"name": "Just Eat Takeaway.com",
"domain": "justeattakeaway.com",
"careers_url": "https://careers.justeattakeaway.com/global/en",
"ats": "phenom",
"ats_slug": "",
"country": "NL",
"city": "Amsterdam",
"status": "researching"
},
{
"key": "philips",
"name": "Philips",
"domain": "philips.com",
"careers_url": "https://www.careers.philips.com/na/en/",
"ats": "phenom",
"ats_slug": "",
"country": "NL",
"city": "Amsterdam",
"status": "researching"
},
{
"key": "yousign",
"name": "Yousign",
"domain": "yousign.com",
"careers_url": "https://youtrust.com/",
"ats": "welcometothejungle",
"ats_slug": "",
"country": "FR",
"city": "Paris",
"status": "researching"
}
]
-101
View File
@@ -1,101 +0,0 @@
[
{
"key": "novo-nordisk",
"name": "Novo Nordisk",
"domain": "novonordisk.com",
"careers_url": "https://www.novonordisk.com/careers/find-a-job.html",
"ats": "successfactors",
"ats_slug": "",
"country": "DK",
"city": "Bagsværd",
"status": "researching"
},
{
"key": "vestas",
"name": "Vestas",
"domain": "vestas.com",
"careers_url": "https://careers.vestas.com/search/?q=&locationsearch=",
"ats": "successfactors",
"ats_slug": "",
"country": "DK",
"city": "Aarhus",
"status": "researching"
},
{
"key": "demant",
"name": "Demant",
"domain": "demant.com",
"careers_url": "https://careers.demant.com",
"ats": "successfactors",
"ats_slug": "",
"country": "DK",
"city": "Smørum",
"status": "researching"
},
{
"key": "carlsberg",
"name": "Carlsberg",
"domain": "carlsberggroup.com",
"careers_url": "https://careers.carlsberg.com/",
"ats": "successfactors",
"ats_slug": "",
"country": "DK",
"city": "Copenhagen",
"status": "researching"
},
{
"key": "bang-olufsen",
"name": "Bang & Olufsen",
"domain": "bang-olufsen.com",
"careers_url": "https://www.bang-olufsen.com/de/de/story/careers",
"ats": "successfactors",
"ats_slug": "",
"country": "DK",
"city": "Struer",
"status": "researching"
},
{
"key": "systematic",
"name": "Systematic",
"domain": "systematic.com",
"careers_url": "https://jobs.systematic.com/search/?createNewAlert=false&amp;q=&amp;locationsearch=DE+%2B+AT+%2B+CH&amp;optionsFacetsDD_country=&amp;optionsFacetsDD_customfield2=",
"ats": "successfactors",
"ats_slug": "",
"country": "DK",
"city": "Aarhus",
"status": "researching"
},
{
"key": "scania",
"name": "Scania",
"domain": "scania.com",
"careers_url": "https://jobs.scania.com/go/Jobs-at-Scania/9096601/",
"ats": "successfactors",
"ats_slug": "",
"country": "SE",
"city": "Södertälje",
"status": "researching"
},
{
"key": "sap",
"name": "SAP",
"domain": "sap.com",
"careers_url": "https://careers.sap.com/",
"ats": "successfactors",
"ats_slug": "",
"country": "DE",
"city": "Walldorf",
"status": "researching"
},
{
"key": "signavio",
"name": "Signavio",
"domain": "signavio.com",
"careers_url": "https://careers.sap.com/",
"ats": "successfactors",
"ats_slug": "",
"country": "DE",
"city": "Berlin",
"status": "researching"
}
]
-134
View File
@@ -1,134 +0,0 @@
[
{
"key": "zendesk",
"name": "Zendesk",
"domain": "zendesk.com",
"careers_url": "https://www.zendesk.de/company/careers/",
"ats": "workday",
"ats_slug": "",
"country": "DK",
"city": "Copenhagen",
"status": "researching"
},
{
"key": "unity",
"name": "Unity",
"domain": "unity.com",
"careers_url": "https://unity.com/careers/positions",
"ats": "workday",
"ats_slug": "",
"country": "DK",
"city": "Copenhagen",
"status": "researching"
},
{
"key": "lego-group",
"name": "LEGO Group",
"domain": "lego.com",
"careers_url": "https://www.lego.com/de-de/careers",
"ats": "workday",
"ats_slug": "",
"country": "DK",
"city": "Billund",
"status": "researching"
},
{
"key": "saxo-bank",
"name": "Saxo Bank",
"domain": "home.saxo",
"careers_url": "https://www.home.saxo/about-us/careers",
"ats": "workday",
"ats_slug": "",
"country": "DK",
"city": "Copenhagen",
"status": "researching"
},
{
"key": "novonesis",
"name": "Novonesis",
"domain": "novonesis.com",
"careers_url": "https://www.novonesis.com/en/careers/jobs",
"ats": "workday",
"ats_slug": "",
"country": "DK",
"city": "Lyngby",
"status": "researching"
},
{
"key": "gn-group",
"name": "GN Group",
"domain": "gn.com",
"careers_url": "https://www.gn.com/",
"ats": "workday",
"ats_slug": "",
"country": "DK",
"city": "Ballerup",
"status": "researching"
},
{
"key": "equinor",
"name": "Equinor",
"domain": "equinor.com",
"careers_url": "https://www.equinor.com/careers",
"ats": "workday",
"ats_slug": "",
"country": "NO",
"city": "Stavanger",
"status": "researching"
},
{
"key": "axis-communications",
"name": "Axis Communications",
"domain": "axis.com",
"careers_url": "https://www.axis.com/de-de",
"ats": "workday",
"ats_slug": "",
"country": "SE",
"city": "Lund",
"status": "researching"
},
{
"key": "mister-spex",
"name": "Mister Spex",
"domain": "misterspex.com",
"careers_url": "https://corporate.misterspex.com/de/karriere/",
"ats": "workday",
"ats_slug": "",
"country": "DE",
"city": "Berlin",
"status": "researching"
},
{
"key": "darktrace",
"name": "Darktrace",
"domain": "darktrace.com",
"careers_url": "https://www.darktrace.com/careers",
"ats": "workday",
"ats_slug": "",
"country": "GB",
"city": "Cambridge",
"status": "researching"
},
{
"key": "criteo",
"name": "Criteo",
"domain": "criteo.com",
"careers_url": "https://careers.criteo.com/en/working-at-criteo/belonging-inclusion/",
"ats": "workday",
"ats_slug": "",
"country": "FR",
"city": "Paris",
"status": "researching"
},
{
"key": "red-hat",
"name": "Red Hat",
"domain": "redhat.com",
"careers_url": "https://www.redhat.com/en/jobs",
"ats": "workday",
"ats_slug": "",
"country": "CZ",
"city": "Brno",
"status": "researching"
}
]
-166
View File
@@ -1,166 +0,0 @@
# SAP SuccessFactors career sites: reading without a browser
Probed 2026-09-29 with curl and python3 urllib, Chrome User-Agent, at most 1 request/s per host, `-m 20` timeout, no login. About 330 requests in total. No 403, 429, captcha or bot wall was met. Everything below was verified by a live request unless marked "inferred".
## Result table
All 9 rows are SuccessFactors. 8 of 9 are readable with one parser: 7 are the RMK front end on their own host, Novo Nordisk turned out to have one too (`careers.novonordisk.com`), and Signavio is a keyword slice of SAP's site. Bang & Olufsen only links to the back end, which robots.txt disallows.
| Company | Human-facing jobs URL | Served by | Best listing method | Roles seen | Detail method | Verified | Note |
|---|---|---|---|---|---|---|---|
| Vestas | https://careers.vestas.com/search/?q=&locationsearch= | RMK (table layout, 10 per page) | `sitemap.xml` for ids (774) or `/search/?startrow=N` paging (states total) | 661 in search (en_GB), 774 in sitemap | job page microdata JobPosting: `datePosted`, `addressCountry` (ISO), `hiringOrganization`, title, description; no employmentType/salary/JSON-LD | yes | Sitemap is a superset: 113 extra ids are German/French postings of the same roles (e.g. Zistersdorf EN id ...033 and DE id ...933). Also `/googlefeed.xml` (8 MB RSS with descriptions, 774 items). |
| Demant | https://careers.demant.com/search/?q=&locationsearch= | RMK (tile layout, 10 per page) | `sitemap.xml` (308) or search paging | 308 in both, identical id sets | microdata: `datePosted`, `validThrough`, `streetAddress` = "Smørum, DK, 2765" (country is the 2-letter token), `hiringOrganization`; country name also in `data-careersite-propertyid="customfield4"` | yes | Job URLs carry a brand segment: `/demant/job/...`, `/audika_Poland/job/...`. Listing tile has ID (customfield2), Brand, Country, Location. |
| Carlsberg | https://careers.carlsberg.com/search/?q=&locationsearch= | RMK (tile layout, 10 per page) | `sitemap.xml` (105) | 68 in search en_GB, 105 in sitemap | microdata: `datePosted`, `validThrough`, `streetAddress` "Fredericia, DK", `hiringOrganization` | yes | 37 sitemap-only roles exist only in other locales (`locale=fr_FR` gave 16, `de_DE` 11, `sv_SE` 4, `fi_FI` 3 that the English search misses). Sitemap is the only complete list without looping locales. Brand segment in path (`/Kronenbourg/job/`). |
| Systematic | https://jobs.systematic.com/search/?q=&locationsearch= | RMK (table layout, 10 per page) | `sitemap.xml` (47) or search paging | 47 in both | microdata: `datePosted`, `streetAddress` "Bern, CH, 3008", `hiringOrganization`; no `itemprop=title` (use `og:title` or `<h1>`); no validThrough | yes | Given careers_url has a DE+AT+CH filter baked in; the base host lists all 47. |
| Scania | https://jobs.scania.com/search/?q=&locationsearch= | RMK (tile layout, 15 per page) | `sitemap.xml` (760) or search paging | 760 in both, identical id sets | microdata: `datePosted`, `validThrough`, `streetAddress` "Vilnius, LT, LT-02300", `hiringOrganization` | yes | Given careers_url `/go/Jobs-at-Scania/9096601/` is a landing page of the same site; `/search/` on the host works. Tile has location, work area, full/part time, permanent/temporary, date. |
| SAP | https://careers.sap.com/search/?q=&locationsearch= | RMK (table layout, 25 per page, no date column) | `sitemap.xml` is an RSS/Google-Base feed with descriptions: 921 items in one request (1.8 MB gzipped, 15 MB raw); or search paging (33 requests) | 815 in search, 921 in feed | microdata: `datePosted` (has time), `streetAddress` "SANTIAGO DE SURCO, PE, 0033", `hiringOrganization`, `industry`; page is 235 KB | yes | Feed items: `<link>`, `<guid>`/`<g:id>`, `<title>` with "(City, CC, zip)", `<g:location>`, `<g:job_function>`, `<g:expiration_date>` (no posting date). 106 extra items are other-locale postings. Search ignores the date sort. |
| Signavio | https://careers.sap.com/ (same site as SAP) | same as SAP | `/search/?q=signavio` keyword search | 39 hits | as SAP | yes (as a slice) | No separate Signavio site. Only a keyword match on the title/description, so it is not an exact company filter. |
| Novo Nordisk | https://www.novonordisk.com/careers/find-a-job.html (AEM/Akamai, Vue widget) | Own AEM JSON servlet in front; real SF RMK site at `https://careers.novonordisk.com` | RMK `sitemap.xml` (419) or RMK search with 100 per page (5 requests); own JSON `GET /bin/nncorp/careersearch?keyword=&country=&category=&locale=en` returns all rows in one request | 419 RMK; 427 rows / 418 unique req ids in JSON API | RMK job page: HTML only (microdata JobPosting stripped; only `itemprop=title` and `description`), no JSON-LD. AEM job ad page `/content/nncorp/global/en/careers/find-a-job/job-ad.{jobId}.html` is HTML only (og:title, h1) | yes | JSON API has no dates and no URLs; `jobId` = "346501.en_GB" = SF requisition id. AEM job page location is text only. RMK job page location: `p#job-location span.jobGeoLocation` = "Bangalore, Karnataka, IN". See "Novo Nordisk" below. |
| Bang & Olufsen | https://www.bang-olufsen.com/de/de/story/careers | Only a link to SF back end `https://career5.successfactors.eu/career?company=bangolufse` | none allowed | not read | not read | no | Back end host robots.txt is `Disallow: /` and `Disallow: /*company`. No RMK host found (careers./jobs./career.bang-olufsen.com do not resolve). Needs a decision, see below. |
## Anchors for the RMK pages
Both listing layouts use the same job link and total.
- Job link: `href="(/[^"]*?/job/[^"]*?/(\d+)/)"` (the id is the last path segment; unescape `&amp;` in the href; some paths have a brand prefix before `/job/`).
- Table layout (Vestas, Systematic, SAP): rows `<tr class="data-row">`; `a.jobTitle-link` (title text); `span.jobLocation` ("City, Region, CC, zip"); `span.jobDate` (Vestas, Systematic only; "29 Sept 2026"); total in `<span class="paginationLabel" aria-label="Results 1 – 10">Results <b>1 – 10</b> of <b>661</b>`.
- Tile layout (Demant, Carlsberg, Scania): `<li class="job-tile job-id-{id}" data-url="/.../job/.../{id}/">`; fields as `id="job-{id}-desktop-section-{field}-value"` with `{field}` = `date`, `location` or company-specific `customfieldN`, labels in `...-label`; total in `<span id="tile-search-results-label">Showing 1 to 10 of 308 Jobs</span>`. Field set differs per company: Demant has no `location` field but `customfield4` (country name) and `customfield5` (city); Scania and Carlsberg have `location` with a trailing country code.
- Job page: `<div class="jobDisplayShell" itemscope itemtype="http://schema.org/JobPosting">` followed by `<meta itemprop="datePosted" content="Tue Sep 29 02:00:00 UTC 2026">` (Java Date format, parse `%a %b %d %H:%M:%S UTC %Y`), `<meta itemprop="validThrough">` (not always), `<meta itemprop="hiringOrganization">`, `itemprop="jobLocation" > itemprop="address"` with either `addressLocality`/`addressRegion`/`addressCountry` (Vestas) or a single `streetAddress` = "City, CC, zip" (others). Title: `span[itemprop=title]` inside `h1`, fallback `<meta property="og:title">`. Description: `span[itemprop=description] > span.jobdescription` (HTML). No `employmentType`, no `baseSalary`, no JSON-LD on any of the 8 sites; contract-type is at best a custom field in the listing (Scania: "Permanent / Temporary").
## Novo Nordisk
Three layers were seen, all live:
1. `www.novonordisk.com/careers/find-a-job.html` is AEM; a Vue widget calls `/bin/nncorp/careersearch?keyword=&country=&category=&locale=en`. One request returned all 427 rows (341 KB, includes filter dictionaries `subCategories`, `states`, `cities`). Row: `jobId` "346501.en_GB", `jobTitle`, `jobLocale`, `jobLocationLabel` "India, Bangalore", `jobCountry.value`, `jobCategory`, `jobSubCategory`, `jobState`, `jobCity`. `country=Denmark` gave 93 rows (English country name as filter value). Detail URL is built by the widget as `https://www.novonordisk.com/content/nncorp/global/en/careers/find-a-job/job-ad.{jobId}.html` (200, HTML only, description in `.job-display-details`). The API mixes locales (274 en_GB, 140 en_US, 11 ja_JP, 1 de_DE, 1 fr_FR) so req ids repeat; 418 unique.
2. The job page's apply link points to the SF back end `career2.successfactors.eu/career?...career_job_req_id=346501`, which redirects to `https://careers.novonordisk.com/job/Bangalore-Data-Architect-I-Karn/1442129433/`. That host is a normal RMK site, so the generic method works: sitemap 419, search 419 in 5 requests (100 per page).
3. The back end `career2.successfactors.eu/career?company=novonordisk&career_ns=job_listing_summary&resultType=XML` does return an XML listing (2.7 MB, `<Job><JobTitle><Job-Description><ReqId><filter7> country ...`), but only 266 reqs, and robots.txt on that host disallows everything. It was fetched once by accident of probing order and not used further. Do not build on it.
Recommendation: use `careers.novonordisk.com` with the generic RMK method. Novo's RMK job pages have no microdata, so dates and country come from the search listing (date column) and the `span.jobGeoLocation` anchor, or use the JSON API (country in `jobCountry`) as the list and RMK only for dates. Mapping between the two: the RMK URL is only reachable through the redirect from the `career_job_req_id` back-end URL (disallowed) or through the RMK listing; the RMK id (1442129433) is not the req id (346501). So mixing both is not cheap; pick one. Simplest is RMK only.
## Bang & Olufsen
`career5.successfactors.eu/career?company=bangolufse` is the only job entry point on their site. `robots.txt` there is `Disallow: /` and `Disallow: /*company`. One HEAD request returned 200 (page exists, no redirect). It was not read. Inferred, not verified: the same back end supports `career_ns=job_listing_summary&resultType=XML` like Novo's, but that is disallowed too. Options: skip B&O, ask the user to accept the back end, or check whether a `jobs.` host exists behind a differently named CNAME. Not tried: search engines for an RMK host.
## General answers
### One reading method for most
Yes: 7 of 9 rows are RMK (Vestas, Demant, Carlsberg, Systematic, Scania, SAP, plus Novo via `careers.novonordisk.com`), Signavio is a slice of SAP. Algorithm, all URLs relative to the stored base (scheme + host, no trailing slash):
1. Id list (1 request). `GET {base}/sitemap.xml` with `Accept-Encoding: gzip`.
- If the body starts with `<urlset`: for each `<loc>` matching `/job/.*/(\d+)/` take `(url, id)`. Unescape `&amp;`. The URL has to be used as given, since the slug and brand prefix vary.
- If it starts with `<rss` (SAP only): for each `<item>` take `<link>` and `<g:id>`; title, `<g:location>` and `<g:job_function>` come for free; descriptions are escaped HTML.
- The sitemap covers every locale; `<lastmod>` is a constant (the build day) and useless for change detection. It was not stale: every id found by paging the search was also in the sitemap.
2. Diff against stored ids. Ids gone from the sitemap are closed roles.
3. For each new id, `GET` its URL (1 request each, 80 to 250 KB uncompressed) and read the microdata anchors above (datePosted, country, title, description). If no `itemtype=JobPosting` is present (Novo), use `og:title` plus the listing row.
4. Optional cheaper daily path without job pages: the search listing (below) gives title, location, date and country for 10 to 100 roles per request.
Search listing (needed where the sitemap over-counts translations, or if only title/date/location are wanted): `GET {base}/search/?q=&locationsearch=&sortColumn=referencedate&sortDirection=desc&startrow={N}`, N = 0, pagesize, 2*pagesize ... Read the total from the results label (both layouts), stop when `N >= total`. The page size is fixed per site and not settable (`pagesize=100` was ignored): Vestas/Demant/Carlsberg/Systematic 10, Scania 15, SAP 25, Novo 100. Detect it as the number of rows on the first page. Beyond the end, Demant returns 0 rows; Vestas returned a non-empty odd page (`startrow=670` with 661 total), so trust the stated total, not an empty page. Paging to the end was run for all 7 sites: unique ids equalled the stated total each time (815, 760, 661, 419, 308, 68, 47).
Caveat on completeness: the search only lists postings in the site's default locale (`locale=` sets it). Vestas 661 vs 774, Carlsberg 68 vs 105, SAP 815 vs 921. Some of the extras are translations of the same requisition (Vestas), some are roles that exist only in another language (Carlsberg's Kronenbourg/French roles). The app has to choose: sitemap (complete, duplicates per language) or search (deduped per locale, misses local-language-only roles). No requisition id is exposed in the sitemap, so dedupe by translation is not possible without opening the job page (inferred). Recommendation: sitemap as the source of ids.
### Companies with their own method
- SAP: the "sitemap.xml" is an RSS feed with 921 items, 15 MB raw and 1.8 MB gzipped. Use the feed as one request; it is heavier than other sitemaps (Vestas 157 KB) but one request beats 33 pages. It has no posting date; get it from the job page `datePosted`. Signavio: `q=signavio` search only (title/description keyword), or filter SAP results by text in the description (inferred).
- Novo Nordisk: RMK host `careers.novonordisk.com`, job pages have no microdata (see above). The AEM JSON API is an alternative for the whole list in one request and country filter, but no dates.
- Bang & Olufsen: back end only, disallowed. Not readable in good conscience.
### Country filter on the server
Works on all 7 RMK hosts, verified: `&optionsFacetsDD_country=CC` (exact facet, ISO codes on Vestas and Demant; SAP `DE` gave 177) and `&locationsearch=CC` (text match on the location, slightly looser: SAP 186 vs 177). Counts for Denmark: Vestas 62 of 661, Demant 17 of 308, Carlsberg 7, Novo RMK 85, Systematic 24 (that count is DK-only), Scania `SE` 54, SAP `DE` 177. The facet dropdown options are filled by JavaScript, so the valid values are not in the HTML; ISO 2-letter codes worked everywhere tried. Novo API: `country=Denmark` (English name). No country filter for the sitemap; parse the country from `streetAddress` / `addressCountry` on the job page.
### Minimal identifier per company
The site base URL, `https://{host}` (e.g. `https://careers.vestas.com`). Everything else is derived: the layout (table or tile) is auto-detected, the page size from the first page, the sitemap at `/sitemap.xml`. For Signavio also a keyword. Optional: a country code for the server-side filter. Not needed: SF company id, ats_slug. The input `careers_url` values are not the base for four rows (Novo: AEM page; Scania: `/go/...`; Systematic: filtered search URL; Signavio: SAP host); store the host root.
### Requests per daily poll, 500 open roles
- Full re-read by search paging: 50 requests at 10 per page, 34 at 15, 20 at 25, 5 at 100 (Novo). Each page is 43 KB gzipped (Vestas measured), 250 to 300 KB raw.
- Sitemap route: 1 request (Vestas 157 KB raw, Scania 158 KB, SAP feed 1.8 MB gzipped) plus 1 job page per new id. With 5 to 20 new roles a day that is about 6 to 21 requests.
- Read only new roles: the search default order and `sortColumn=referencedate&sortDirection=desc` are date-descending on all 6 sites with a date column (verified by the dates on page 1 and on the last page; `asc` reverses it). SAP shows no date column and ignores the sort. Ids are not ordered with the date (about 50 percent of adjacent pairs ascend; old ids such as 1372305933 sit near the top), because the shown date is the last (re)posting date. So "stop at first known id" is wrong. Use: walk pages while any row date is >= (last poll date minus 1 day) or any id on the page is unknown; typical daily cost 1 to 2 requests plus job pages. Order within one day is not defined. Not verified over several days (only today's data was seen).
- Closed roles are only detected by a full id list (sitemap or full paging).
- No `ETag`, `Last-Modified` or `304` support seen on the sitemap (`Cache-Control: no-cache, no-store`), so conditional GET is not available.
### Rate limiting and bot protection
None met: no 403, 429 or captcha in about 330 requests at 1 per second with a Chrome User-Agent; curl's default User-Agent also got 200 on one Vestas search. RMK hosts: `server: nginx`, `JSESSIONID` cookie, `Cache-Control: no-store`. `www.novonordisk.com` sits behind Akamai (`Akamai-Cache-Status` header) but served the page and JSON API without challenge. One quirk: on the RMK sites any unknown `*.xml` path such as `/googlefeed.xml`, `/sitemal.xml`, `/jobs.xml` returned the same 8 MB Google feed on Vestas (a catch-all); `/feed/` returned an error page. So do not probe feed paths blindly.
### robots.txt
All 6 RMK hosts tested (`careers.vestas.com`, `careers.demant.com`, `careers.carlsberg.com`, `jobs.systematic.com`, `jobs.scania.com`, `careers.sap.com`) plus `careers.novonordisk.com` serve the identical 252-byte file (identical md5), no `Sitemap:` line:
```
User-agent: *
Disallow: /applybutton/
Disallow: /talentcommunity/
Disallow: /mobile/talentcommunity/
Disallow: /emailsubscribe/
Disallow: /email/image/
Disallow: /services/
Disallow: /preapply/
Disallow: /error
Disallow: /unsubscribe/
Disallow: /reset/
```
Consequences: `/search/`, `/job/`, `/sitemap.xml` and `/googlefeed.xml` are allowed. `/services/` is disallowed, which contains the RSS endpoint `/services/rss/job/?locale=en_US&keywords=(...)`; it was therefore not fetched, and its existence is inferred only. `/talentcommunity/apply/{id}/` (the Apply link) is disallowed; never follow it.
`www.novonordisk.com/robots.txt` is a single line: `Sitemap: https://www.novonordisk.com/sitemap.xml` (nothing disallowed, so `/bin/nncorp/careersearch` is allowed).
`career2.successfactors.eu/robots.txt` and `career5.successfactors.eu/robots.txt` (the back end):
```
User-agent: *
Disallow: /
Disallow: /*company
Allow: /login
```
So every `career?company=...` URL is disallowed, including the XML listing.
Deviations from robots: while probing Novo I fetched one back-end job URL (it redirected to the RMK page), one back-end XML listing (2.7 MB, saved as `sf/novo_sf_list.xml` in the scratchpad) and one HEAD on the B&O back end, before I had read that host's robots.txt. Nothing further was requested from those hosts.
## Curl commands that worked
```bash
UA='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36'
B=https://careers.vestas.com # any RMK host: careers.demant.com careers.carlsberg.com jobs.systematic.com jobs.scania.com careers.sap.com careers.novonordisk.com
# robots
curl -sL -m 20 -A "$UA" $B/robots.txt
# id list: sitemap (SAP: RSS feed instead of urlset)
curl -sL -m 30 -A "$UA" -H 'Accept-Encoding: gzip' --compressed $B/sitemap.xml -o sitemap.xml
grep -o '<loc>[^<]*</loc>' sitemap.xml | wc -l
# search listing, date descending, paged; total in the results label
curl -sL -m 20 -A "$UA" "$B/search/?q=&locationsearch=&sortColumn=referencedate&sortDirection=desc&startrow=0"
curl -sL -m 20 -A "$UA" "$B/search/?q=&locationsearch=&startrow=650"
curl -sL -m 20 -A "$UA" "$B/search/?q=&locationsearch=&optionsFacetsDD_country=DK" # server-side country filter
curl -sL -m 20 -A "$UA" "$B/search/?q=signavio&locationsearch=" # keyword (SAP host)
# job page (microdata)
curl -sL -m 20 -A "$UA" "$B/job/Chetwynd-Stock-Keeper-Brit/1432067433/"
# Novo Nordisk JSON API on the AEM host
curl -sL -m 20 -A "$UA" 'https://www.novonordisk.com/bin/nncorp/careersearch?keyword=&country=&category=&locale=en'
curl -sL -m 20 -A "$UA" 'https://www.novonordisk.com/bin/nncorp/careersearch?keyword=&country=Denmark&category=&locale=en'
curl -sL -m 20 -A "$UA" 'https://www.novonordisk.com/content/nncorp/global/en/careers/find-a-job/job-ad.346501.en_GB.html'
```
Extraction that was run (python3): ids `re.findall(r'href="([^"]*/job/[^"]*?/(\d+)/?)"', html)` de-duplicated; total `re.search(r'Results <b>\d+\D+\d+</b> of <b>(\d+)</b>')` or `re.search(r'Showing \d+ to \d+ of (\d+)')`; microdata `re.findall(r'<meta itemprop="([^"]+)" content="([^"]*)"')`.
## Fixtures saved beside this file
- `out-successfactors-list.html`: Vestas search page 1, table layout (661 total, 10 rows).
- `out-successfactors-list-tile.html`: Demant search page 1, tile layout (308 total).
- `out-successfactors-list-sitemap.xml`: Systematic `sitemap.xml` (47 urls).
- `out-successfactors-list-novo.json`: Novo Nordisk JSON API, all 427 rows.
- `out-successfactors-job.html`: Vestas job page (microdata with `addressCountry`).
- `out-successfactors-job-demant.html`: Demant job page (microdata with `streetAddress` "Smørum, DK, 2765", `validThrough`).
- Scratchpad working files (all listing id sets from the full paging, SAP feed, Vestas google feed, other job pages) are in `sf/` beside this file.
## Verified versus inferred
- Verified by live request: every count in the table; paging to the end on 7 hosts; sitemaps and their id-set differences to search; country filters; date sort on 6 hosts; microdata contents on 6 job pages plus Novo RMK and AEM pages; the Novo JSON API; robots.txt on 9 hosts; the SAP feed; the Vestas Google feed; no rate limiting.
- Inferred only: existence and content of `/services/rss/job/` (disallowed, not fetched); whether B&O's back end supports the XML listing; how the sitemap and search behave over several days (only one day observed); that duplicates between locales cannot be merged without opening the page; whether `optionsFacetsDD_country` values other than ISO codes exist.
-252
View File
@@ -1,252 +0,0 @@
# Workday career sites read without a browser
Probed on 2026-09-29 from a plain (non-datacentre, WSL) IP with `curl` and `urllib`, one request per second per host, browser User-Agent, no cookies, no login. About 400 requests in total. "Verified" means I got a 200 with real data from a live request. "Inferred" is marked as such.
## Result per company
All 12 are on Workday and all 12 answer the list and detail endpoints without any token or cookie. Verified = yes means list, offset paging to `total`, and one detail call all worked.
| Company | tenant | wd | site | list total seen | verified | human-facing jobs URL | note |
|---|---|---|---|---|---|---|---|
| Zendesk | zendesk | wd1 | zendesk | 124 | yes | https://zendesk.wd1.myworkdayjobs.com/zendesk | Only site in robots.txt. Entries carry `remoteType`. |
| Unity | unitytech | wd1 | Unity | 123 | yes | https://unitytech.wd1.myworkdayjobs.com/Unity | Tenant is `unitytech`, not `unity`. No country facet, only city-level `locations`. `total` is repeated on every page. |
| LEGO Group | lego | wd103 | LEGO_External | 453 | yes | https://lego.wd103.myworkdayjobs.com/LEGO_External | robots.txt lists `LEGO_Executive` as Disallow (not probed). Country facet is called `Location` (top-level). One posting in the list has only `bulletFields` (no title, no path), so 452 of 453 are readable. Many retail and hourly roles. |
| Saxo Bank | saxobank | wd3 | CareeratSaxoBank | 50 | yes | https://saxobank.wd3.myworkdayjobs.com/CareeratSaxoBank | Second site `Sharedexclusive` (1 role, not the main one). |
| Novonesis | novonesis | wd103 | Novonesis_Careers | 134 | yes | https://novonesis.wd103.myworkdayjobs.com/Novonesis_Careers | The careers page links `novozymes.wd103...`, which answers 422 on the list endpoint and 500 (maintenance page) on the HTML. The working tenant is `novonesis` (found by trying it; the old host name is stale). robots.txt: `Disallow: /Novonesis_Careers/`. |
| GN Group | gn | wd3 | GN-Careers | 80 | yes | https://gn.wd3.myworkdayjobs.com/GN-Careers | Main site. Seven more brand sites on the same tenant: BeltoneCareers 126, JabraCareers 23, ReSoundCareers 24, AudigyCareers 4, SteelSeriesCareers 8, BeltoneIndependentlyOwned 124, ReSoundConversion 0. Overlap with GN-Careers not checked. |
| Equinor | equinor | wd3 | EQNR | 10 | yes | https://equinor.wd3.myworkdayjobs.com/EQNR | Only 10 roles, mostly summer internships and graduate programmes. Second site `EQCONF` has 0. The company careers page links only EQNR. I did not find a separate site for experienced hires, so Equinor's real list may live elsewhere (unverified). |
| Axis Communications | axis | wd3 | External_Career_Site | 98 | yes | https://axis.wd3.myworkdayjobs.com/External_Career_Site | Has `remoteType`, `endDate`. Country facet `locationCountry`; the facet count (37 for Sweden) differed from the filtered total (35). |
| Mister Spex | misterspex | wd103 | Mister_Spex_Careers | 70 | yes | https://misterspex.wd103.myworkdayjobs.com/de-DE/Mister_Spex_Careers | Careers page links use the `/de-DE/` locale segment; without it works too. German descriptions. Mostly stores in Germany. |
| Darktrace | darktrace | wd3 | DarktaceExternal | 73 | yes | https://darktrace.wd3.myworkdayjobs.com/DarktaceExternal | Site name is misspelt by the company ("Darktace"). robots.txt: `Disallow: /DarktaceExternal/`. |
| Criteo | criteo | wd3 | Criteo_Career_Site | 81 | yes | https://criteo.wd3.myworkdayjobs.com/Criteo_Career_Site | robots.txt: `Disallow: /Criteo_Career_Site/`. Three school-fair sites (EPITASchoolFair2026, CentraleSupelec2026, IESEG2026) have 0 roles. |
| Red Hat | redhat | wd5 | jobs | 151 | yes | https://redhat.wd5.myworkdayjobs.com/jobs | Site id is case-insensitive in the API (`jobs` and `Jobs` both work); `externalUrl` uses `Jobs`. Facet keys are single letters (`a` = Country). |
Sum of `total` over the main sites: 1,447 roles.
## General answers
### Is the description only in the detail call, and what does a poll cost?
Verified: the list entry has only `title`, `externalPath`, `locationsText`, `postedOn` (relative text), `bulletFields` (job req id, sometimes a location), and on some tenants `remoteType`. No description, no date, no country id. So the description costs one request per role.
The page size is fixed. `limit` 20 works everywhere; 21, 25, 30, 40, 50 and 100 return HTTP 400 on every tenant tried (all 12 for 50/100, the other 11 for 21). So a board with 800 roles needs 40 list requests to enumerate.
- First import of 800 roles: 40 list + 800 detail = 840 requests, about 14 minutes at one request per second.
- Steady-state daily poll: 40 list requests plus one detail request per new role. With the usual churn (order of 5 to 30 new roles per day) that is 45 to 70 requests.
- With a server-side country filter the list shrinks a lot: LEGO Denmark is 95 roles = 5 list requests instead of 23.
Cheaper new-role detection, verified as far as stated:
- `total` and the ids arrive on page 0 with one request. If `total` and the 20 ids on page 0 are unchanged, nothing new was posted at the top. This can miss the case of one role removed and one added if the new one is not on page 0, which order makes unlikely (see below).
- Default order is newest first, but not strictly. Over the full LEGO list (453) and Red Hat list (151) the `postedOn` day count is nondecreasing with a few violations (LEGO: a "Posted 30+ Days Ago" entry at position 24 followed by a "Posted Today" entry; Red Hat: 28 then 27; several 8-then-7 flips). `postedOn` also saturates at "Posted 30+ Days Ago". So "stop at the first known id" is a heuristic. A safer rule is to stop after one full page (20 ids) with no unknown id. Reposted or amended roles can surface later in the list.
- There is no sort parameter. I tried `sortBy`, `sort` and `orderBy` in the body; the response was 200 with the same `total` and, for `sortBy`, the same order, so unknown keys are ignored. Inferred: order is fixed by Workday.
- `siteMap.xml` (listed in robots.txt for allowed sites) is capped at 100 URLs, has no `lastmod`, and equals the first 5 list pages (checked on LEGO: all 100 URLs are in the list, in list order; Novonesis also 100). It saves at most 4 requests for a large board and is no substitute for paging. Sites that Disallow have no sitemap line (Darktrace, Criteo main; Novonesis serves one anyway, 200).
- The detail response carries `similarJobs` (2 entries, each with `startDate`), not useful for enumeration.
### Offset paging
Verified on 11 tenants (Novonesis paged only to page 0 after the tenant fix, total 134, first page and detail verified): paging `offset` 0, 20, 40, ... until `offset >= total` returns exactly `total` unique `externalPath` values (LEGO: 452 unique plus the one path-less entry). Quirks that matter for code:
- `total` is only reliable on the first page (`offset` 0). On later pages it is `0` on 9 tenants (Zendesk, LEGO, Saxo, GN, Axis, Mister Spex, Darktrace, Criteo, Red Hat) and correct on Unity and Equinor. Read `total` once from page 0.
- Offset at or beyond `total` does not return an empty list. It returns page 0 again (seen on Zendesk: offset 124, 140, 200 gave the first 20 roles and `total` 124). A loop that stops on an empty page never ends; stop on `offset >= total`.
- The path-less entry (LEGO): `{"bulletFields": ["0000036369"]}`. Skip entries without `externalPath`.
### Fields in the detail response
Verified on all 12 (one role each). `jobPostingInfo` always has:
- `title`, `jobDescription` (HTML, 4.8 to 10.9 KB), `location` (text), `postedOn` (relative text, e.g. "Posted 4 Days Ago"), `startDate`, `timeType`, `jobReqId`, `jobPostingId`, `externalUrl`, `country` (`descriptor`, `id`), `jobRequisitionLocation` (`descriptor` plus `country.alpha2Code`), `canApply`, `posted`.
- `startDate` is an ISO date (`2026-09-29`). In all 12 samples it matches the relative `postedOn` text (Posted Today = today, Posted 4 Days Ago = 2026-09-25, Posted Yesterday = 2026-09-28). So it is the usable real posting date. Its meaning is inferred from that match. It is not a job start date.
- `externalUrl` is the canonical human URL, without a locale segment (`https://zendesk.wd1.myworkdayjobs.com/zendesk/job/Lisbon-Portugal/Business-Development-Representative_R35239`).
- `country.alpha2Code` is only inside `jobRequisitionLocation.country`, so the ISO country code is available there.
Optional per tenant:
| field | present on |
|---|---|
| `remoteType` (In Office, Hybrid, Remote, No, On-Site) | Zendesk, Axis, Novonesis, Criteo, Red Hat. Absent on the other 7; no remote/hybrid signal there except the description text. |
| `additionalLocations` (list of text) | Equinor, Darktrace, Red Hat, only when the role has several locations. The list shows "2 Locations" in `locationsText` in that case. |
| `endDate`, `timeLeftToApply`, `jobPostingEndDateAsText` | Axis, LEGO, Novonesis, Equinor, Red Hat (application deadline) |
| `videoInfo` (large blob with signed tokens) | LEGO, Saxo, GN. Ignore it. |
Pay: there is no structured pay field on any tenant. Pay appears only inside the description HTML, when at all. Of 12 sample roles: Unity states a range (`Base Salary Range ... Zone A: $114,600 - $171,800 USD gross`), Red Hat has a generic compensation paragraph, Criteo, Saxo and Mister Spex say "attractive salary"/"Gehalt" without numbers, the other 6 say nothing. Sample size is one role per company, so this says only that pay is text-only.
`timeType` values differ in spelling ("Full time" vs "Full-time" on Mister Spex).
### Server-side country filter
Verified: yes, via `appliedFacets` with the facet parameter name and the value ids from the `facets` array of the first list response. Values in one facet are OR-ed (Darktrace `Country: [UK, US]` gave 53 where the facet counts summed to 54, likely count drift between two requests seconds apart), different facets are AND-ed. Combining with `searchText` works (Darktrace UK plus "engineer" gave 21). An unknown facet key returns HTTP 400.
The parameter name differs per tenant, so it cannot be hard-coded:
| tenant | country facet parameter | where |
|---|---|---|
| zendesk, saxobank, gn, axis, misterspex | `locationCountry` | nested under `locationMainGroup` |
| darktrace, criteo, novonesis | `Country` | top-level facet |
| lego | `Location` | top-level facet, values are countries (`Denmark` 95) |
| redhat | `a` (descriptor "Country") | top-level, single-letter keys |
| unitytech | none; only `locations` (cities) | pick the city ids |
| equinor | none; only `locations` (cities) | pick the city ids |
Filter results checked live (facet count then filtered total): Zendesk Portugal 16 then 16; Unity Copenhagen 11 then 11; LEGO Denmark 95 then 95; Saxo Denmark 14 then 14; Novonesis Denmark 21 then 21; GN Denmark 15 then 15; Equinor Hammerfest 1 then 1; Axis Sweden 37 then 35; Mister Spex Germany 67 then 67; Darktrace UK 35 then 35; Criteo France 20 then 20; Red Hat `a`=United States 53 then 53 (Czechia not in its list today).
The country value ids look tenant-independent: United States `bc33aa3152ec42d4995f4791a106ed09` and United Kingdom `29247e57dbaf46fb855b224e03170bc7` appear on LEGO, Darktrace, Red Hat, Unity; Denmark `49ab063f422741e2aef271de00efeac8` on LEGO, Saxo, Novonesis. Verified for those three, inferred for the rest. Safest is to look up the id by `descriptor` in the first response's `facets` per tenant (one request, which the poll makes anyway), and to fall back to post-filtering on the detail `country` when a tenant has no country facet.
Post-filtering from the list alone is weak: `locationsText` is often a city or "2 Locations" or, on LEGO, only a country name, so the country of a role is cleanly known only from the detail (`country`, `alpha2Code`) or from a facet filter.
### Rate limiting and bot protection
Verified: none met. All hosts are behind Cloudflare (`server: cloudflare`, `cf-ray`, `__cf_bm` cookie), yet no 403, 429 or challenge page came back in about 400 requests at one per second. Every response was 200, or 400/404/422/500 for my own deliberately wrong inputs. I did not send bursts, so the actual rate limit is unknown. I only probed from one IP, which is not a datacentre address; whether a cloud server IP gets a Cloudflare challenge is unverified. Plan for it: treat 403/429/503 as "back off and retry tomorrow", keep the one-request-per-second pacing.
### Does every tenant answer the same way?
The protocol is the same on all 12 (same paths, same JSON shape, no auth). Differences that matter:
- Facet names and country facets (table above); Red Hat uses `a` to `e` as facet parameters.
- `total` on later pages (0 vs repeated); wrap-around at the end.
- Optional detail fields (`remoteType`, `additionalLocations`, `endDate`).
- Unknown or stale tenant or site: HTTP 422 on the list (Novonesis with the stale `novozymes`), HTTP 404 with `errorCode S21` for an unknown site name, HTTP 200 with `total: 0` for a real but empty site (Equinor `EQCONF`, Criteo school sites), HTTP 400 for a bad body, limit above 20, or unknown facet key.
- The site name in the URL is case-insensitive (`jobs` vs `Jobs`).
### Minimal identifier per company
Three parts are needed: tenant, data-centre and site. Proposed string form: `{tenant}.{wd}/{site}`, for example
```
lego.wd103/LEGO_External
unitytech.wd1/Unity
novonesis.wd103/Novonesis_Careers
redhat.wd5/jobs
```
From it the app builds everything: list `https://{tenant}.{wd}.myworkdayjobs.com/wday/cxs/{tenant}/{site}/jobs`, detail `.../wday/cxs/{tenant}/{site}{externalPath}`, human URL `https://{tenant}.{wd}.myworkdayjobs.com/{site}`. The per-role key to store is `jobReqId` (also the last token of `externalPath` and of `bulletFields[0]`); `externalPath` is the stable fetch key (note it can end in `-1`, `-2` on reposts, e.g. Red Hat `..._R-059566-2`). The tenant is not always the company name (`unitytech`, `novonesis` vs the linked `novozymes`), so store it as found and do not derive it from the domain.
Discovery for a new company: fetch the careers page and grep for `myworkdayjobs.com` (worked for all 12 without JavaScript); then `https://{host}/robots.txt` lists all sites of the tenant as `Sitemap:` or `Allow:` lines (worked for 9 of 12; the three that Disallow their main site show the site only in the `Disallow` line). Then verify with the list call. A stale tenant shows as 422.
### robots.txt and terms
Verified: robots.txt exists on every host (fetched on all 12). It is generated per tenant and per site, and only names site paths, never `/wday/cxs/`. Quotes:
LEGO (`lego.wd103`):
```
Sitemap: https://lego.wd103.myworkdayjobs.com/LEGO_External/siteMap.xml
User-agent: *
Allow: /LEGO_External/
Disallow: /LEGO_Executive/
Disallow: /refreshFacet/
```
Novonesis (`novonesis.wd103`):
```
User-agent: *
Disallow: /Novonesis_Careers/
Disallow: /Novonesis_Careers_Private/
Disallow: /refreshFacet/
```
Darktrace: `Disallow: /DarktaceExternal/`. Criteo: `Disallow: /Criteo_Career_Site/` (while allowing its three school sites). The other 9 hosts have `Allow: /{site}/` for their main site and `Disallow: /refreshFacet/` only (Zendesk, Unity, Saxo, GN, Equinor, Axis, Mister Spex, Red Hat, LEGO).
Reading: three of the 12 (Novonesis, Darktrace, Criteo) have switched their main site to Disallow for all user agents, which is Workday's "do not let search engines index this site" setting. Strictly the rule covers the page path `/{site}/`, and the JSON API sits under `/wday/cxs/`, which no robots.txt here mentions; but the intent is that the employer does not want automated crawling of that site. This is a policy decision for you: honour it (skip those three), or treat the cxs endpoint as the page's own backing API for a once-daily poll of public data. I did not find a terms-of-use statement on the hosts (the JSON has no legal text, the HTML app shell does not carry one in the parts I saw; I did not fetch a Workday terms page). No authentication or captcha was touched, and the requests to those three sites were the same handful of calls as for the others.
## Sample responses (trimmed)
List entry (Zendesk, `total` 124, from `out-workday-list.json`):
```json
{
"total": 124,
"jobPostings": [
{
"title": "Business Development Representative",
"externalPath": "/job/Lisbon-Portugal/Business-Development-Representative_R35239",
"locationsText": "Lisbon, Portugal",
"postedOn": "Posted Today",
"remoteType": "In Office",
"bulletFields": ["R35239"]
}
],
"facets": [
{"facetParameter": "remoteType", "descriptor": "Remote Type",
"values": [{"descriptor": "Hybrid", "id": "e92222236f4b100194e5184e4fa00000", "count": 43}]},
{"facetParameter": "locationMainGroup",
"values": [{"facetParameter": "locationCountry", "descriptor": "Country",
"values": [{"descriptor": "Portugal", "id": "2e8c5034deb045d49315417c347472ca", "count": 16}]}]}
]
}
```
Detail (Zendesk, `jobPostingInfo` only, description cut, from `out-workday-detail.json`):
```json
{
"jobPostingInfo": {
"id": "f0833f8f93df1000fd891e465ff10000",
"title": "Business Development Representative",
"jobDescription": "<h2>Job Description</h2><p ...>(10,880 chars of HTML)</p>",
"location": "Lisbon, Portugal",
"postedOn": "Posted Today",
"startDate": "2026-09-29",
"timeType": "Full time",
"jobReqId": "R35239",
"jobPostingId": "Business-Development-Representative_R35239",
"jobPostingSiteId": "zendesk",
"country": {"descriptor": "Portugal", "id": "2e8c5034deb045d49315417c347472ca"},
"jobRequisitionLocation": {"descriptor": "Portugal - Lisbon - Office",
"country": {"descriptor": "Portugal", "id": "2e8c5034deb045d49315417c347472ca", "alpha2Code": "PT"}},
"remoteType": "In Office",
"externalUrl": "https://zendesk.wd1.myworkdayjobs.com/zendesk/job/Lisbon-Portugal/Business-Development-Representative_R35239",
"canApply": true,
"posted": true
},
"hiringOrganization": {"name": "CLEVERLY LDA (Portugal)", "url": ""},
"similarJobs": ["2 entries with title, externalPath, timeType, locationsText, postedOn, startDate"],
"userAuthenticated": false
}
```
The full raw files are `out-workday-list.json` (Zendesk page 0, limit 20) and `out-workday-detail.json` (the same role, R35239), a matching pair to use as fixtures. Full responses of all 12 tenants are also in `wd/raw_{tenant}_list.json` and `wd/raw_{tenant}_detail.json`.
## Commands that worked
Set once: `UA='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36'`. The list POST (LEGO Denmark filter), the Zendesk detail GET, robots.txt, siteMap.xml and the careers-page grep were re-run verbatim with `curl` and gave the results stated above. The paging, limit, sort and facet experiments were run with the equivalent `urllib` calls.
Find the site:
```
curl -sS -m 20 -A "$UA" https://www.equinor.com/careers | grep -o 'https://[a-z0-9.-]*myworkdayjobs.com[^"<> ]*' | sort -u
curl -sS -m 20 -A "$UA" https://gn.wd3.myworkdayjobs.com/robots.txt
```
List (page 0, 20 roles, `total` and `facets` in the response):
```
curl -sS -m 20 -A "$UA" -X POST \
-H 'Content-Type: application/json' -H 'Accept: application/json' \
-d '{"appliedFacets":{},"limit":20,"offset":0,"searchText":""}' \
https://lego.wd103.myworkdayjobs.com/wday/cxs/lego/LEGO_External/jobs
```
List filtered by country (LEGO Denmark, facet parameter `Location`):
```
curl -sS -m 20 -A "$UA" -X POST \
-H 'Content-Type: application/json' -H 'Accept: application/json' \
-d '{"appliedFacets":{"Location":["49ab063f422741e2aef271de00efeac8"]},"limit":20,"offset":0,"searchText":""}' \
https://lego.wd103.myworkdayjobs.com/wday/cxs/lego/LEGO_External/jobs
```
Next page: change `"offset":20`, `40`, ... while `offset < total` (take `total` from page 0).
Detail (`externalPath` from the list entry):
```
curl -sS -m 20 -A "$UA" -H 'Accept: application/json' \
https://zendesk.wd1.myworkdayjobs.com/wday/cxs/zendesk/zendesk/job/Lisbon-Portugal/Business-Development-Representative_R35239
```
Sitemap (100 newest URLs, no dates): `curl -sS -m 20 -A "$UA" https://lego.wd103.myworkdayjobs.com/LEGO_External/siteMap.xml`
Things that failed, on purpose or not: `"limit":21` and up gives HTTP 400; the stale `novozymes.wd103` tenant gives HTTP 422 on the list and a 500 maintenance page on the HTML; an unknown site gives 404 `S21`; an unknown facet key gives 400.
## Not verified
- Behaviour from a cloud or datacentre IP (Cloudflare challenge risk).
- The real rate limit; I never exceeded one request per second per host.
- Whether Equinor has experienced-hire roles on another site or system.
- Whether facet ids for countries are identical across all tenants (checked for US, UK, DK only).
- Whether `startDate` always equals the posting date (matched in all 12 samples, one role each; also verified for the old "Posted 4 Days Ago" case on LEGO and Equinor).
- Any terms-of-service text of Workday or the employers about API access.