diff --git a/docs/decisions.md b/docs/decisions.md index 874d6a9..6b9107b 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -96,6 +96,20 @@ What was decided for roledesk and the one reason for each. The longer reasoning **A board that fails on every fetch for 21 days closes its roles.** A board whose fetch raised was only logged, so a company that moved ATS kept its roles open for good behind a 404. The `ats` plugin now records `ats:failing:` in `state` with the time of the first failure in a row and the latest reason, and a fetch that answers removes it. Once the first failure is `STALE_DAYS` old, the same window a search role gets, the settle closes the board's roles; the next answer reopens what it lists through `upsert_role`, as for a board switched back on. The record is plugin state rather than a column because the plugin only sees `self.state`. `/api/companies` returns it as `failing_since` and `failing_reason`, the Companies list shows "not answering since", and `app companies check` prints it beside its live result. +**A host is a module in `plugins/atshosts/`, and registers by existing.** `plugins/ats.py` held eight hosts in six parallel tables and `app/jobs/discover.py` three more, so two people adding a host edited the same lines. Each host is now one file that states its name, label, URL patterns, slug check, allowed host names and either a feed and a parser or a pager; `ats.py` and `discover.py` build their tables from the package, ordered by the module's `RANK` so the select and the probes kept their order. The package sits beside `ats.py` and is not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module. The contract is the docstring of `plugins/atshosts/__init__.py`, and a module that breaks it fails the import by name. + +**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed. + +**A description stated only on the role's own page is read once, 200 roles per board per pull.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. A module's `detail` is called for roles not yet read, those never tried before those that failed, and the ids read are kept in `ats:detailed:`; a known role is sent without a description on later pulls, which `upsert_role` ignores. `detail` may fill `DETAIL_FIELDS` only (description, location, remote, currency, salary_min, salary_max, posted), the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing, since the listing rewrites them every pull. Completeness is decided by the listing alone, so a board of 800 roles closes its roles from the first pull and has all its text after four. + +**robots.txt is honoured, by RFC 9309.** A module names the page a person opens, and a board whose site disallows that page for `*` or for the app's own token fails with "robots.txt disallows it" and closes its roles after 21 days like any failing board. The matcher takes the longest rule, an Allow on a tie, `*` and `$`, and case-sensitive paths, or paths without case for a host that serves any case, as Workday does. A 4xx means no rules; a 5xx, a 429 or a timeout fails the board for that pull. `urllib.robotparser` was replaced because it takes the first rule and knows no wildcard, which refused Infineon and let `/*company` through. This keeps Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy and Umbraco off the automatic route. The SuccessFactors reader uses `/search/` and `/job/` only, never `/services/` or a `career*.successfactors.eu` back end. + +**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key the page's JavaScript carries, which can change without notice and marks an endpoint that is not open. They are on the Watchlist with reason `key`. + +**A company has one board, and a company inside its owner's board is read by hand.** DFDS has eleven Teamtailor boards by country and is read from the Danish one. Onfido, Tink, Glovo, Cognigy, Contractbook, Signavio and Native Instruments list their roles in Entrust's, Visa's, Delivery Hero's, NICE's, Scrive's, SAP's and inMusic's boards, where no field separates them reliably; they carry reason `group`. + +**The seed's gap was mostly discovery, not parsers.** A survey of the 155 companies without a board (2026-09-29, every careers site opened) found 49 on a host the app already read: the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain or on Lever's EU instance (`lever:eu/`), or the careers page refuses a plain request. 43 were attached by hand from the survey, 46 more with the fourteen hosts added that day, and 286 of 362 companies have a board. Discovery keeps a slug's case for a module with `KEEP_CASE`, since Workday's site names, Gem's, Deel's and Rippling's slugs are case-sensitive; SmartRecruiters and Ashby are not, checked live. + **`pull` is a job that starts the cascade, on a default schedule of every 6h.** A schedule enqueues a job kind and a cascade is started by one, so the `pull` job only asks for the chain; declared beside it in `app/jobs/cascade.py`, the schedule is a starting point that an edit under Runs overrides for good. `pull` is not also a task, which would draw two buttons under Runs. ## Ranking @@ -154,6 +168,8 @@ What was decided for roledesk and the one reason for each. The longer reasoning **Watchlist is a third tab: the companies the app cannot read, rated yes or no.** The owner goes through the careers pages of such companies by hand and wanted that list one click from Roles rather than behind the cog, so `TABS` has three entries and the phone bar's tabs are tighter. A company is on it when it has no board subscription and is not known from a search only, or when its board is failing: the rule of the Company boards filter "Not readable yet", read from `GET /api/companies`. The rating is `companies.watch` (migration 15: NULL unrated, 1 on the list, 0 not), global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. A row rated under Unrated leaves the list at once. On a phone the search field of Roles is left a few letters wide by the third tab, so while it has the focus the tabs and the queue chip step aside. +**A company on the Watchlist says why it is there, and can be added, edited and removed on the tab.** `companies.hand_reason` (migration 16) is one of `unread` (readable, no module yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone` and `added` (by hand or by an import, not looked at); attaching a board clears it. `POST`, `PATCH` and `DELETE /api/companies` and `app companies add`, `edit`, `checked`, `remove` and `by-hand` are what the LinkedIn import will write through. A company with a reason is never "known from a search only", so one added with a name alone is listed. No second table and no `checked_at`: a company has one careers link, and `careers_opened_at` already says when somebody last looked. A careers link somebody typed is kept: discovery and the seed fill it only when empty. A name is changed only while the company has no board, since the subscription is found by the key the name folds to. + **Settings are three sub-tabs: Search agents, Sources, Account.** Six sub-tabs were three setup pages and three system pages for two things, what the app ranks by and where roles come from, so each is one page now and Account stays its own. **The old paths' 301s are gone.** `MOVED` answered 301 for the twelve paths above, kept for the few days between the move and this cut, on the assumption that a bookmark should still land. A browser caches a 301 indefinitely, so a path a future page reused would be unreachable through no fault of that page; a one-person app with no outstanding bookmarks does not need to carry that risk. diff --git a/docs/open.md b/docs/open.md index 167a1db..b0bad5d 100644 --- a/docs/open.md +++ b/docs/open.md @@ -10,3 +10,12 @@ What is left after the phone rework of 2026-09-25, for whoever picks the repo up - **Fit does not read seniority.** With three prompts on the live data of 2026-09-29 an individual contributor's posting ranks beside a head's on the same subject. Switch extract on so `seniority` fills, then add a prefer gate on it, and vote. - **Nine Teamtailor seed companies have no board**, because their `careers_url` is the marketing site: Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial. - **Votes do not train fit.** A vote only moves the rank at read time. Feeding votes into the prompt vectors is worth doing once there are a few hundred. +- **76 companies are read by hand**, on the Watchlist with a careers link opened on 2026-09-29: 35 `unread`, 10 `robots`, 9 Teamtailor ones without a career host, 8 `group`, 7 `blocked`, 4 `gone`, 3 `key`. The remaining readers are planned in `docs/plan-hosts-wave-2.md`. +- **The hosts added on 2026-09-29 were read from this machine only.** Every Workday tenant is behind Cloudflare; read one Workday and one SuccessFactors board from the deployed server before relying on them. +- **An unknown SmartRecruiters or Gem id answers an empty listing, not 404**, so a mistyped slug reads as a complete board with no roles and would close a company's roles. `set_board` reads the board first, which catches a typo only when the person notices "0 roles". +- **`Crawl-delay` is not honoured.** Booking.com's robots.txt asks for 5 seconds; the shared reader waits `PAGE_DELAY`, 0.5. A read of its board is two requests. +- **Eightfold returns 10 roles a page and `MAX_PAGES` is 100**, so Infineon (1,289 roles) and Microsoft (about 2,300) would never be read complete and are not attached. The endpoint takes a location filter; a target has nowhere to state one. +- **A SuccessFactors site on a company's own domain is not discovered from its own pages**, which link only the back end and the image CDN. It needs the rule Teamtailor has in `discover.py`: a page with the markers means the page's host is the board, checked by reading `/search/`. Until then such a board is set by hand. +- **HR Manager's portal disallows reading in robots.txt, its feed host has no robots.txt.** The module checks the feed's host, so a board would be read. Seaborg, the one company on it, is not attached until the owner decides whether the feed counts as allowed. +- **Six new hosts' dates are not trusted** (Eightfold, Jibe, Gem, Deel, HiBob, HR Manager; Pinpoint states none), so their backlog counts as posted when first seen. Check each date on a live board and add the host to `TRUSTED_POSTED_HOSTS`. + diff --git a/docs/plan-hosts-and-by-hand.md b/docs/plan-hosts-and-by-hand.md deleted file mode 100644 index fde9de5..0000000 --- a/docs/plan-hosts-and-by-hand.md +++ /dev/null @@ -1,151 +0,0 @@ -# Plan: a module per host, new hosts, and the list of companies checked by hand - -Planned 2026-09-29. Three waves of work packages with disjoint files, built in parallel worktrees made from `main`, merged by the planner. This file and `docs/plan-hosts/` are folded into `decisions.md` and deleted when the work lands. - -## Why - -164 of the 362 companies have no board attached: 9 on Teamtailor whose career host is not found yet (the other 23 were attached today), and 155 on a host the app cannot read or on no host it could name. They include LEGO, Novo Nordisk, Vestas, Danske Bank, SimCorp, Ørsted, Klarna and Zalando. A company the app cannot read is shown as "not readable yet" with a link that nobody can edit, so the owner has no list to work through by hand. - -## What the survey found - -Nine Sonnet agents read every one of the 155 career sites on 2026-09-29. Their notes are in `docs/plan-hosts/`: `workday.md`, `successfactors.md`, `otherhosts.md`, `own-1.md` to `own-4.md`, and `survey-own.json` with one record per company (verified `jobs_url`, class, host, identifier, listing URL, roles seen, evidence). Raw responses for fixtures are in the session scratchpad, `/tmp/claude-1000/-home-claude-cboxcode-jobboard/1734d897-7509-48fc-844c-efe768acbf7a/scratchpad/`, and a builder copies the ones it needs, trimmed, into `tests/fixtures/ats/`. - -| Route | Companies | Examples | -|---|---|---| -| Board on a host the app reads already | 49 + 9 Teamtailor | Bosch, Wise, Ubisoft, Checkout.com, Lunar, DFDS, Voi, Remote | -| SuccessFactors | 20 | Novo Nordisk, Vestas, Carlsberg, Pandora, Coloplast, Scania, SAP, Tryg | -| Workday | 19 | LEGO, Zeiss, SimCorp, Zendesk, Unity, Saxo Bank, Red Hat | -| Oracle | 4 | Danske Bank, Nokia, Sinch, Milestone | -| Phenom | 3 | Philips, Just Eat Takeaway, King | -| Eightfold | 3 | Microsoft, Ericsson, Infineon | -| Eleven small hosts | 15 | Klarna (Deel), Booking.com (Jibe), sennder (Gem), Einride (Jobylon) | -| Sitemap and structured data | 8 | Arm, Mambu, Odoo, Aiven, AUTO1, ASML | -| Own JSON API | 8 | Ørsted, Nordea, Amazon, Vinted, Dynatrace | -| Own HTML list | 10 | Zalando, Netcompany, Google, Bolt, CHECK24 | -| By hand | about 25 | Revolut, Meta, Maersk, Bang & Olufsen, Visma, HashiCorp | - -The 49 were missed because the slug is not the company's name (`BoschGroup`, `jimdo.com`, `workatbackbase`), the board is on a custom domain, the board is on Lever's EU host, or the careers page refuses a plain request. - -## Decisions - -**A host is a module.** `plugins/ats.py` holds eight hosts in six parallel tables, and `app/jobs/discover.py` holds three more, so two builders adding a host edit the same lines. Each host moves into `plugins/atshosts/.py`, and `ats.py` and `discover.py` build their tables from the package. The package is a sibling of `ats.py` and not `plugins/ats/`, since `desk.sources.registry` globs `plugins/*.py` and the tests patch names on the `ats` module. - -**A host is handed the fetch.** A host module imports nothing from `plugins.ats`. Its functions receive `get`, which `ats.py` binds to its own `_get` at call time, so the allow-list, the redirect rule, the byte cap and the deadline stay in one place and a test that patches `ats._get` reaches every host. - -**A description read per role is read once.** Workday, SuccessFactors and BambooHR state the description only on the role's own page. A host with a `detail` function has it called for roles not yet read in detail, at most `DETAIL_CAP` (200) per board per pull; the ids read are kept in `ats:detailed:`. A known role is sent without a description on later pulls, which `upsert_role` ignores since an empty value never overwrites a filled column. Completeness is decided by the listing alone. - -**robots.txt is honoured.** A host module names the page whose path must be allowed. The shared code reads the site's robots.txt at most once a day per board and fails a board that disallows it for `*`, with the reason "robots.txt disallows it". This takes Novonesis, Darktrace, Criteo and Bang & Olufsen off the automatic route. The SuccessFactors reader uses `/sitemap.xml`, `/search/` and `/job/` only, never `/services/` or the `career*.successfactors.eu` back end. - -**No reader is built on a key lifted from a site's script.** Maersk's API, New Work's and Welcome to the Jungle's search index answer only with a key that the page's JavaScript carries. They go on the by-hand list. - -**The by-hand list is the `companies` table, and the Watchlist tab is the list.** A company is on it when it has no shown board and is not known from a search only. `careers_url`, `notes`, `watch` (the rating) and `careers_opened_at` exist already, the last being the date a person last looked at the careers page; migration 16 adds `hand_reason`. There is no `checked_at`, since opening the page from the Watchlist stamps the date already. No second table, since a company has one careers link and the list needs no history of checks. - -**A link typed by hand is kept.** `add_discovered` and the seed fill `careers_url` only when it is empty. - -**A company has one board.** DFDS has eleven Teamtailor boards by country, and Onfido, Tink and Glovo sit inside their owners' boards. DFDS gets its Danish board; the other three go on the by-hand list with reason `group`. - -## Contracts between packages - -Host module, `plugins/atshosts/.py`, as built by WP0. The docstring of `plugins/atshosts/__init__.py` holds the same contract and is the one to keep current. A module imports `desk` and `._shared` only, never `plugins.ats`. - -```python -NAME = "workday" # the host of a target `workday:`; equals the module's file name -LABEL = "Workday" # the Sources page's select -PATTERNS = (r"https?://...",) # discovery: regexes over a URL, compiled with IGNORECASE -RANK = 100 # optional int, default 100; lower comes first -KEEP_CASE = True # optional, default False; keep a discovered slug's case -ROBOTS_ANY_CASE = True # optional, default False; read robots.txt paths without case - -def slug_ok(slug: str) -> bool: ... # strict, re.fullmatch -def allowed(slug: str) -> Callable[[str], bool]: ... # host names a fetch may reach -def careers_url(slug: str) -> str: ... # the page a person opens - -# one of the two -def feed_url(slug: str) -> str: ... # the whole board in one answer -def parse(body: bytes, slug: str) -> list[Row]: ... - -def page(get: Get, slug: str, cursor: Any) -> Page: ... # a paged board, one page a call -def rows(decoded: Any, slug: str) -> list[Row]: ... - -# optional -def fallback_url(slug: str) -> str: ... # feed hosts: a smaller feed, read once when feed_url is over the byte cap -def robots_page(slug: str) -> str: ... # the URL robots.txt must allow; on a host allowed(slug) admits -def detail(get: Get, slug: str, row: Row) -> dict[str, Any]: ... # fields from the role's own page -def probe_url(slug: str) -> str: ... # discovery: answers 2xx only for a board -def probe_ok(body: bytes) -> bool: ... # discovery: when 2xx is not enough -def probe_slugs(guess: str) -> tuple[str, ...]: ... # the slugs a guessed name is probed as; default (guess,) -``` - -`Get`, `Row` and `Page` are in `plugins/atshosts/_shared.py`, with the helpers several hosts share: `only(host)` for `allowed`, `path_slug_ok` and `label_slug_ok` for `slug_ok`, `parse_dt`, `is_web_url`, `num`, `yearly_amount` and `pay_line`. `plugins/atshosts/__init__.py` lists the modules with `pkgutil.iter_modules`, checks each against the contract at import (an ImportError names the module and what it lacks), and orders `HOSTS` by `(RANK, NAME)`. The eight first hosts hold ranks 10 to 80, which keeps the order they had before the modules: the select opens on Greenhouse, and a guessed slug on two hosts goes to the one probed first. - -Order and names. `RANK` orders the Sources page's select, the plugin's description and discovery's probes. `NAME` equals the file name. A host that `discover.py` names in its own `_UNREADABLE` table is taken from its module as soon as the module exists, so a wave 1 package does not edit `discover.py`. - -Patterns. The slug is group 1, or, when a pattern has more groups, the groups that matched joined by "/" (`https?://([a-z0-9-]+\.wd\d+)\.myworkdayjobs\.com/(?:[a-z]{2}-[A-Z]{2}/)?([A-Za-z0-9_-]+)` gives `lego.wd103/LEGO_External`). Only group 1 is checked against discovery's words that name no board (`api`, `jobs`, `embed`, ...). The slug is lowercased unless the module sets `KEEP_CASE = True`, which a host with case-sensitive paths needs, since robots.txt paths are case-sensitive too. A pattern without a group names the host with no slug. - -Slugs. `slug_ok` uses `re.fullmatch` (a `$` lets a trailing newline through) and refuses anything that could add a path, query, fragment, port or user to the URLs built from it. A slug of several parts joins them with "/": Lever's EU boards are `lever:eu/`. - -`get(url, *, method="GET", payload=None, headers=None) -> bytes` is `ats._get` bound to the board. A POST sends `payload` as JSON; `headers` are added to the default User-Agent and Accept and replace either where they name it. It fetches https on port 443 only, never an IP address or a local name (`localhost`, `.local`, `.internal`, `.lan`, `.home.arpa`), follows at most three redirects each checked against `allowed(slug)`, cuts the body at `MAX_BYTES` and ends within `DEADLINE`. It returns the body and raises `httpx.HTTPStatusError` on a non-2xx answer, `httpx.HTTPError` on a timeout or a connection failure, and another exception, `ValueError` among them, for a refused host, an oversized body or a broken redirect. A module lets these through. - -`Page` is `(decoded, results, total, cursor)`: the decoded page handed to `rows`, its raw results as dicts with an `id` (counted by distinct id for completeness), the total the host states or None, and the next cursor or None after the last page. The first call gets cursor None. A board with a total is complete when as many distinct ids were read, one without when the read ended on an empty page; a 429 after the first page ends the read, incomplete. A paged module may also have `parse(body, slug)` for one page's body, for its tests. - -A `Row` has `source_id`, `title`, `location`, `url`, `posted` (aware datetime or None), `department`, `description` (plain text), `remote` (True, False, a word or None) and `salary_text`, and, where the host states pay, `salary_min`, `salary_max` (yearly integers) and `currency`. - -`detail` returns a dict of what the role's own page states. The shared code calls it for roles not yet read (`ats:detailed:`), at most `DETAIL_CAP` (200) per board per pull, `PAGE_DELAY` apart, the roles never tried before those whose detail failed (`ats:detailfailed:`). Only `DETAIL_FIELDS` are taken: `description`, `location`, `remote`, `currency`, `salary_min`, `salary_max` and `posted`, the keys whose columns survive a later pull that sends the listing's empty value (`upsert_role`'s keep-filled columns, the salary figures, and `posted_at`, which keeps the earliest date). Any other key is dropped with a debug log line: `title`, `url`, `department` and `salary_text` cannot be filled from `detail`, since the listing rewrites them every pull. A role not read in detail on a pull is sent with an empty description. A 404 or 410 counts as read, any other failure is retried on a later pull, and a 429 ends the detail reads for that board and pull. Completeness is decided by the listing alone. - -robots.txt. For a module with `robots_page`, the shared code reads `https:///robots.txt` before the listing, at most once a day per board (`ats:robots:`), following a redirect to the host's `www.` twin or back and no other. It fails the board with "robots.txt disallows it" when the rules for `*` or for the app's own token disallow the page, matched per RFC 9309: the longest matching rule wins, Allow wins a tie, `*` and a final `$` in paths, paths case-sensitive, an empty Disallow allows. A 4xx other than 429 means there are no rules, which allows. A 5xx, a 429, a timeout, a connection error or a redirect elsewhere fails the board with "robots.txt could not be read", and the next pull asks again. - -Slugs: Workday `lego.wd103/LEGO_External`; SuccessFactors, Phenom and the sitemap reader take the site's hostname, with a path where the site has one (`jobs.teradyne.com/MiR`); Oracle `/`. - -Company JSON, in `GET /api/companies`: `CompanyOut` has `domain`, `city`, `watch` and `careers_opened_at` from the Watchlist, and gains `notes`, `hand_reason` and `by_hand` (bool, no board shown and not `from_search`). `hand_reason` is one of `''`, `unread` (readable, no parser yet), `robots`, `blocked` (the site answers a program with a challenge), `key`, `group`, `gone`, `added` (by hand or by an import, not looked at). `from_search` is false for a company with a `hand_reason`. - -| Endpoint | Body | Answer | -|---|---|---| -| `POST /api/companies` | `name`, `careers_url`, `notes`, `hand_reason` (default `added`) | 201 `CompanyOut`; 409 when the key is taken | -| `PATCH /api/companies/{key}` | any of `name`, `careers_url`, `notes`, `hand_reason` | `CompanyOut`; the key stays; 409 on `name` for a company with a board | -| `DELETE /api/companies/{key}` | none | 204; 409 when it has a board or open roles; its `agent_companies` rows go with it | - -`POST /api/companies/{key}/careers` and `POST /api/companies/{key}/opened` of the Watchlist stay and answer a `Company`; `POST .../careers` and `PATCH` check the link with the same function (`check_careers_url` in `app/jobs/sources.py`). `careers_url` is http or https, as `safeUrl` demands on the page. A person looking at the page is `POST .../opened`, which stamps `careers_opened_at`. - -## Work packages - -Wave 0, in parallel, from today's `main`: - -| WP | Branch | Model | Owns | -|---|---|---|---| -| 0 Host modules | `wp0-hosts` | Opus | `plugins/ats.py`, `plugins/atshosts/`, `app/jobs/discover.py`, `tests/test_ats.py`, `tests/test_discover.py`, `tests/test_atshosts.py` | -| 1 By-hand backend | `wp1-byhand-api` | Sonnet | `app/models.py`, `app/storage/schema.py`, `app/storage/store.py`, `app/web/api.py`, `app/jobs/sources.py`, `app/cli.py`, `tests/test_storage.py`, `tests/test_web.py`, `tests/test_sources.py` | -| 2b By-hand editing | `wp2b-byhand-ui` | Opus | `app/static/watchlist.js`, `app/templates/watchlist.html`, `app/static/app.css`, `tests/test_pages.py` | - -Wave 1, in parallel, from `main` after WP0 is merged. Each owns its modules in `plugins/atshosts/`, `tests/test_ats_.py` and `tests/fixtures/ats/*`, and nothing else: - -| WP | Branch | Model | Hosts | -|---|---|---|---| -| 3 | `wp3-workday` | Sonnet | Workday | -| 4 | `wp4-successfactors` | Sonnet | SuccessFactors | -| 5 | `wp5-oracle-phenom` | Sonnet | Oracle, Phenom | -| 6 | `wp6-json-hosts` | Sonnet | Eightfold, Jibe, Pinpoint, BambooHR, Gem | -| 7 | `wp7-small-hosts` | Sonnet | Deel, Jobylon, HR-ON, HR Manager, Rippling, HiBob | - -The sitemap reader (job URLs from a sitemap, the role from the page's JSON-LD or microdata; Arm, Mambu, Odoo, Aiven, AUTO1, ASML) moved from wave 1 to wave 2. A sitemap lists URLs only, so the title would come from the role's own page, and `detail` may not set a title, since the listing rewrites it on every pull. The contract needs a rule for a listing that knows no title before that reader is built. - -Wave 2 is decided after wave 1 by the roles each company would add: a module per company for the own JSON APIs (Ørsted, Nordea, Amazon, Vinted, Dynatrace, RELEX, Siteimprove, Bending Spoons) and the own HTML lists (Zalando, Netcompany, Google with DeepMind, Bolt, CHECK24). Until then these companies are on the by-hand list with reason `unread`. - -**WP0.** The eight hosts move into modules with no change in behaviour: the existing tests pass with edits to imports only. `careers_url`, `is_known_target`, `_allowed_host` and `_paged` lose their Teamtailor branches to the module. `_get` takes `headers`. `detail` and the robots check are added to the shared code with tests against a fake host. `discover.py` takes `READABLE`, `HOSTS`, `_PROBES` and the body checks of `probe` from the modules; the hosts it names but cannot read stay in a table of its own. Discovery gains Lever's EU host. The tests that use `workday:` or `successfactors` as the unreadable host use `taleo` instead. - -**WP1.** Migration 16 with `hand_reason` only, the endpoints `POST /api/companies`, `PATCH` and `DELETE`, `CompanyOut`'s new fields, `remove_company` taking the `agent_companies` rows, `careers_url` filled only when empty in `add_discovered` and `seed_companies`, the seed reading `hand_reason`, `check_careers_url` as the one rule for a typed link, and `app companies add`, `edit`, `checked`, `by-hand` and `remove` in the CLI, where `checked` stamps `careers_opened_at` and `by-hand` lists the never opened first. The `workday:` and `successfactors` fixtures in its three test files become `taleo`. - -**WP2b.** New, Edit and Remove on the Watchlist tab, which is the by-hand list. New puts a form on top of the list: name, careers link, note and reason, sent to `POST /api/companies`. A row gains Edit, which turns it into the same form in place (`PATCH`), and Remove, which asks first (`DELETE`, and a 409 is shown in the row). A row shows the reason and the note. New and Edit use the form pattern of the job boards section (`sources.js`: a `