Mark the decisions that later ones superseded and fix stale words in decisions.md
Each older paragraph that a later decision contradicts (cascade, Jobindex completeness and link-out, host counts, HR Manager and the three Workday companies, summaries, seniority order, trusted dates, tabs, the 175 count) gets one sentence naming what replaced it. The Jobindex robots record names sort=. Em-dashes and a role called a job are reworded. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
1f09d3a61c
commit
597ce9ca39
1 file changed
+19
-19
+19
-19
@@ -16,7 +16,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
## Data
|
||||
|
||||
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one job under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
|
||||
**Two open roles are one posting when company, title, country and a town agree** (`app/dupes.py`). Same non-empty `company_key`, same `title_key`, the same country (the column, or the one the place names when the column is empty or `EU`), and town sets from `places.towns` that share a member, or no town on one side. Host does not matter: on 2026-09-29, 1,080 of 33,881 open roles were copies, 14 of them across hosts (Jobindex or Platsbanken beside the company's board) and the rest one board listing one role under several ids, as SAP does under 3 to 9. The stored `(company_key, title_key, place_key)` found none of the cross-host pairs, since `Ballerup` and `Ballerup, Denmark` differ, and newsdesk's `location_key OR title_key` would merge the same title at two companies. The same title in Berlin and Munich stays two roles, so a place filter finds each. `places.towns` reads the first comma part of each place, so `Mountain View, California` and `San Francisco, California` share no town.
|
||||
|
||||
**A group is rows that all match each other, never a chain.** A row with no place matches Belgium and the Netherlands, a row with no country matches Japan and Malaysia, and joining on one match merged both pairs on the live data. Rows are taken sharpest first (a town, then only a country, then neither) and join a group only when they match every member; a vaguer row joins a group of its own tier, else the one sharper group it fits, else starts its own. A known miss: `Remote - USA NH` and `Remote - USA WA` merge, since a state code is not a town.
|
||||
|
||||
@@ -34,7 +34,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**Structured pay is unpacked the same as any other board field, except that Greenhouse and Ashby need an extra query parameter to get it.** Ashby sends it with `?includeCompensation=true`, Lever's `salaryRange` and Recruitee's `salary` are already on the plain list response, and Greenhouse needs `pay_transparency=true` on the same request it already makes (no per-role fetch); Personio's XML rarely carries a `salaryInformation` block, but reads the same way when it does. Workable and SmartRecruiters give neither, checked live against their real boards (2026-09-29). Every host converts to yearly `salary_min`/`salary_max`/`currency`: a monthly figure is multiplied by 12, an hourly or a daily one is left out rather than annualised into a number nobody stated, and a range with no currency is ignored. Greenhouse can give several ranges for one role (one per office or pay zone); with more than one there is no way to tell which office the role in hand is at, so all of them are skipped rather than one being picked at random.
|
||||
|
||||
**Pay that does not normalise is not dropped, only kept out of the gate columns.** `roles.salary_text` (migration 12) is one human-readable line per parser, written whether or not the board's pay conformed — an hourly or a daily rate, several Greenhouse pay zones, a range with no currency — so a person can read it and do their own sums; it is filled through `raw` like the other columns, but replaced whole on every pull, including with `''` when a board stops stating pay, since the usual keep-filled rule (`_RAW_TEXT`) would otherwise leave a stale line behind. `salary_state` (`app.models.Role`) is derived, not stored: 'normalised' when `salary_min` or `salary_max` is set, 'stated' when only `salary_text` is, `''` otherwise; it is what the feed's Pay filter and badge read, and gates and scoring keep reading only `salary_min`/`salary_max`/`currency`, unchanged.
|
||||
**Pay that does not normalise is not dropped, only kept out of the gate columns.** `roles.salary_text` (migration 12) is one human-readable line per parser, written whether or not the board's pay conformed (an hourly or a daily rate, several Greenhouse pay zones, a range with no currency), so a person can read it and do their own sums; it is filled through `raw` like the other columns, but replaced whole on every pull, including with `''` when a board stops stating pay, since the usual keep-filled rule (`_RAW_TEXT`) would otherwise leave a stale line behind. `salary_state` (`app.models.Role`) is derived, not stored: 'normalised' when `salary_min` or `salary_max` is set, 'stated' when only `salary_text` is, `''` otherwise; it is what the feed's Pay filter and badge read, and gates and scoring keep reading only `salary_min`/`salary_max`/`currency`, unchanged.
|
||||
|
||||
**`extract` reads the board's location too.** The model gets title, the board's `place` text, and the first 3000 characters of the description. Greenhouse's location line ("Berlin, Germany") is often the only place a country is stated, and it is part of the posting, so the "only what the posting states" contract still holds.
|
||||
|
||||
@@ -44,7 +44,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**Postings are persisted.** Storing every posting is what gives `first_seen`, when the app itself first stored the role, and `closed_at`, set by `mark_closed` when a complete fetch of a board no longer lists a role. A role listed again is open again.
|
||||
|
||||
**`posted_at` (migration 10) is the earlier of `first_seen` and a trusted host's `published_at`; `first_seen` is left alone, as when the app found the role.** `first_seen` being the only honest posted date was true of Greenhouse's `updated_at`, which resets on every edit; `plugins.ats` already reads `first_published` instead, and that plus the other hosts' dates — Lever's `createdAt`, Ashby's `publishedAt`, SmartRecruiters' `releasedDate`, Recruitee's `published_at`/`created_at`, Personio's `createdAt`, Workable's `published`, and the two `search` providers' (Jobindex's RSS `pubDate`, Adzuna's `created`) — are creation dates, checked against live boards and the live database (`app.models.TRUSTED_POSTED_HOSTS`, with the evidence in its comment). A date in the future or before 2015 is not trusted either, host or no host. Without this, a newly added board's whole backlog scored as freshly posted for two weeks: on the live database, 10,223 of 15,415 board roles were posted more than 30 days before they were first seen (average gap in days: Ashby 78, Greenhouse 143, Lever 226, Personio 333), and open roles with `recency` (`app.jobs.score.recency`) above 0.5 fell from all 15,558 of them to 3,089 (19.9%) once it read `posted_at` instead. `upsert_role` keeps the earliest `posted_at` a role has ever had: an edit on the board can only move its own date later, never earlier, so it can only move `posted_at` earlier too, when the board finally reports the true one.
|
||||
**`posted_at` (migration 10) is the earlier of `first_seen` and a trusted host's `published_at`; `first_seen` is left alone, as when the app found the role.** `first_seen` being the only honest posted date was true of Greenhouse's `updated_at`, which resets on every edit; `plugins.ats` already reads `first_published` instead, and that plus the other hosts' dates (Lever's `createdAt`, Ashby's `publishedAt`, SmartRecruiters' `releasedDate`, Recruitee's `published_at`/`created_at`, Personio's `createdAt`, Workable's `published`, and the two `search` providers' (Jobindex's RSS `pubDate`, Adzuna's `created`)) are creation dates, checked against live boards and the live database (`app.models.TRUSTED_POSTED_HOSTS`, with the evidence in its comment). A date in the future or before 2015 is not trusted either, host or no host. Without this, a newly added board's whole backlog scored as freshly posted for two weeks: on the live database, 10,223 of 15,415 board roles were posted more than 30 days before they were first seen (average gap in days: Ashby 78, Greenhouse 143, Lever 226, Personio 333), and open roles with `recency` (`app.jobs.score.recency`) above 0.5 fell from all 15,558 of them to 3,089 (19.9%) once it read `posted_at` instead. `upsert_role` keeps the earliest `posted_at` a role has ever had: an edit on the board can only move its own date later, never earlier, so it can only move `posted_at` earlier too, when the board finally reports the true one. `TRUSTED_POSTED_HOSTS` is gone: each reader declares `Fields(posted="trusted")`, read through `app/models.py`, and there are three search providers, Platsbanken's `publication_date` trusted beside the two named.
|
||||
|
||||
**The tracker schema is taken from jobhunt-hq (MIT).** `applications` with status in `saved, applied, screen, technical, onsite, offer, rejected, withdrawn`, `next_action`, `next_date`, and an append-only `application_events` written by the store on every status entered. The events table is what "days to first screen" and conversion by source are computed from.
|
||||
|
||||
@@ -58,7 +58,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**Boards follow companies; searches find them.** `ats` only reads companies someone listed, so it cannot surface a role at a company nobody knew of, and the seed list came from a US-centred project: of 29,000 open roles, 31 were in Denmark. The `search` plugin runs saved queries on boards that span companies, as `search` subscriptions whose `params` are the query, and pulls them in the same task as the boards.
|
||||
|
||||
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list. A re-run attaches the board an entry names to a row that never had one, so a seed update that gains boards reaches an existing database; it never replaces a board, including a detached one, and never touches affinity, notes or status. Its boards go through `set_board` without the fetch, since they were read when the seed was written and `app companies check` finds the ones that died since.
|
||||
**The seed is a list of companies, not of boards.** The first seed was jobhunt-hq's 186 US boards, chosen because they could be read, and 31 of their 29,000 roles were in Denmark. The second is 362 European employers picked by hand, with `board` filled by `app.jobs.discover` from each company's own careers site and checked once by fetching it; 175 have one. The rest stay in the list with the `ats` host their site runs on, so that "cannot read yet" is a filter on the Companies section rather than a company that was never written down, and the next parser to write is whichever host that filter shows most. Teamtailor is first, on this list. A re-run attaches the board an entry names to a row that never had one, so a seed update that gains boards reaches an existing database; it never replaces a board, including a detached one, and never touches affinity, notes or status. Its boards go through `set_board` without the fetch, since they were read when the seed was written and `app companies check` finds the ones that died since. The count of 175 is out of date: the seed has 286 boards, as the paragraph on the seed's gap below says.
|
||||
|
||||
**`discover` reads the site before it guesses.** A careers page that links its board is the authority, so the home page, its own careers links, then the usual paths are read for a link into a known host, and a page's own "open positions" link one level down. Only when no page names a host are Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee and Personio probed with the company's slug, because a board drawn by a script leaves no link. A probe hit on a common name can be another tenant (`recruitee:google` exists, and is not Google), which is why the seed's boards were fetched and read before they landed, and why `discover --add` reports the roles it found. For the same reason a page that names only an unreadable host ends discovery with that host: probing anyway, as the first version did, could replace a known Workday site with a stranger's Greenhouse board.
|
||||
|
||||
@@ -70,9 +70,9 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**Platsbanken is the third search provider, for Sweden.** The JobTech API (`jobsearch.api.jobtechdev.se/search`) needs no key, states a total, so completeness follows Adzuna's rule, and filters by the taxonomy id of a län (`platsbanken.REGIONS`; Skåne is `CaRE_1nn_cSU`). It refuses an offset above 2000, which the page limit never reaches. The ads carry no usable remote field, so `remote` is read off the headline. Free text ranks by all words rather than requiring them; a quoted phrase narrows it.
|
||||
|
||||
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. An Adzuna search listed everything when it read as many distinct results as the `count` it states; a Jobindex search, whose feed states no count, when a page within its page limit came back with fewer than 20 items, counted before the parser drops any, so one malformed item does not pass for the end. Such a search closes like a board; the rest close once no search has listed them for 21 days. A Jobindex answer with no RSS channel raises instead of reading as an empty, complete search. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists.
|
||||
**A search closes a role only when it listed everything.** A search result is not a complete list, so a role missing from one run has not closed. An Adzuna search listed everything when it read as many distinct results as the `count` it states; a Jobindex search, whose feed states no count, when a page within its page limit came back with fewer than 20 items, counted before the parser drops any, so one malformed item does not pass for the end. Such a search closes like a board; the rest close once no search has listed them for 21 days. A Jobindex answer with no RSS channel raises instead of reading as an empty, complete search. A role two searches found belongs to the last one that listed it, so one search never closes a role the other still lists. Jobindex has since been declared `complete="never"` (its feed states no total), so a Jobindex search closes its roles only by the 21 days and the short-page rule above no longer applies to it.
|
||||
|
||||
**Jobindex ads are read from the feed, not followed to the employer.** Each item links to Jobindex's own page, and its `/c?t=` redirect leads to the employer's site or ATS. Following it per role would count as a click in Jobindex's statistics and costs a request per role, so discovering a company's ATS board from a search result is left for later.
|
||||
**Jobindex ads are read from the feed, not followed to the employer.** Each item links to Jobindex's own page, and its `/c?t=` redirect leads to the employer's site or ATS. Following it per role would count as a click in Jobindex's statistics and costs a request per role, so discovering a company's ATS board from a search result is left for later. The on-demand fetch (A search role's posting text is fetched on demand, below) does follow the "Se jobbet" link to the posting, for a role somebody opened; the pull still never does.
|
||||
|
||||
**Switching a board or search off closes its open roles.** Only a pull that reads a target closes its roles, so a target no longer read would keep them open for good. Switched back on, the next pull reopens what it still lists. A batch already in flight when the switch happens reopens them again, so the pull's settle closes every switched-off target once more.
|
||||
|
||||
@@ -88,13 +88,13 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**SmartRecruiters' board list has no description, so its text is read per role by `enrich`.** Its `/postings` endpoint gives title, location and department; the text is only on `/postings/{id}`, whose `jobAd.sections` hold `jobDescription`, `qualifications` and `additionalInformation` (joined) and `companyDescription` (left out, the same blurb on every role). Reading it for every role would multiply a board's requests by its roles, so the reader declares `text="detail"` and `enrich` reads only the ranked roles (2026-09-30; 209 open roles had no text before).
|
||||
|
||||
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing.
|
||||
**The body cap is 96 MB, with one retry for Greenhouse.** openings.go's 8 MB cap was written against arbitrary URLs, while `ats` only fetches a fixed allow-list of seven ATS hosts, and the real run found `?content=true` boards between 9 and 41 MB. A board that still exceeds 96 MB retries once without `?content=true`, trading the description for the listing. The seven hosts became 22 modules, each with its own allow-list of host names; Greenhouse is still the only reader with a fallback feed.
|
||||
|
||||
**A board request is streamed, with a 30 s read timeout and a 60 s deadline.** `_get` counts the body as it arrives and refuses it at `MAX_BYTES`, where reading `response.content` first held the whole body in memory before the check. httpx's timeouts are per phase, so a board that keeps sending a byte now and then never timed out; the deadline covers the whole request, redirects and body included, and is checked between chunks. openings.go's 10 s read timeout failed `greenhouse:mongodb` and `greenhouse:feedzai` in a pull of 2026-09-28. Measured on 2026-09-29, three runs each of those two and the five Greenhouse boards with the most open roles, with and without `pay_transparency=true` (84 requests, every one a CloudFront miss): a body of up to 10.4 MB arrives in under a second once it starts, and the first byte waited up to 16.6 s, with the three slowest at 16.6, 14.4 and 10.9 s. The flag made no measurable difference; the slow first byte is the origin's, with or without it. A read may wait 30 s, about twice the worst seen, and the deadline is 60 s, which leaves a large board room on a slower link.
|
||||
|
||||
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published job object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The job page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by `enrich` for the ranked roles as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id.
|
||||
**Workable's list endpoint has no URL and no text.** The parser was first written to Workable's published role object, since every slug tried in the build answered an empty list; the first populated board (Hugging Face, 2026-09-28) showed the v3 shape instead: `location` an object, `department` a list, a `shortcode` and no URL, and no description. The role page is `apply.workable.com/{slug}/j/{shortcode}/`. The text is on `/api/v2/accounts/{slug}/jobs/{shortcode}` (`description`, `requirements`, `benefits`), read by `enrich` for the ranked roles as for SmartRecruiters; the row keeps the `shortcode` for it, since `source_id` is the numeric id.
|
||||
|
||||
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse.
|
||||
**Workable and SmartRecruiters are read to the end, and a board closes roles only when it read the total it states.** Workable answers 10 roles a page with a `nextPage` token and SmartRecruiters at most 100 a page by `offset`, and both parsers read page one only, so the settle took page one for the whole board and closed what fell off it: Delivery Hero stated 980 roles and 100 were open. `_paged` follows the pages 0.5 s apart, up to 100 pages, and the plugin stamps `ats:complete` only when the distinct ids read equal `total` or `totalFound`. The count is taken on the raw results, so a row the parser drops does not block closure, and on distinct ids, so a role repeated by a shifted offset does not fake completeness. A 429 after page one ends the read: the roles read are stored and nothing closes, because a partial list is no evidence that the rest closed. A 429 on page one is a failure like any other. The other five hosts answer with the whole board and are complete whenever they parse. Since then there are 22 host modules, not seven; a board over 100 pages is swept across pulls and closes by two finished sweeps (below), and `_paged` is `_runner.read_listing`.
|
||||
|
||||
**Teamtailor is read from the career site's `jobs.rss`, and a slug with a dot is the site's own host.** The official API needs a key per company; the feed needs none and carries title, link, guid, `pubDate`, the description, `remoteStatus` and locations, department and role in the `tt:` namespace. `teamtailor:acme` reads `acme.teamtailor.com` and `teamtailor:careers.acme.com` that host, since most career sites run on the company's domain. `_allowed_host` allows exactly that host, redirects included, and a slug must be a DNS name whose last label is not a number; beyond that a target may name any host that resolves, the same trust as any board the owner adds. The feed states no total, so it is read by `per_page=50` and `offset` to an empty page, and a page with nothing new ends the read as incomplete. 23 of the 32 seed companies answered on their `careers_url` host on 2026-09-29; the rest list a marketing site there.
|
||||
|
||||
@@ -104,9 +104,9 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**A host is handed the fetch.** A module imports `desk` and `._shared` only and receives `get`, which `ats.py` binds to its own `_get` at call time. The allow-list, the redirect rule, the byte cap and the deadline stay in one place, and a test that patches `ats._get` reaches every host. `_get` refuses an IP address, a local or numeric host name and any port but 443 before it asks, because a Teamtailor or SuccessFactors slug is a host name somebody typed.
|
||||
|
||||
**Per-role text is read by the enrich run, ranked roles first.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. The pull reads listings only. After scoring, the `enrich` stage reads the role pages of roles without text that an enabled agent ranks in its top `enrich.top` (500) at `enrich.min_score` (0.5) or more, best score first, 200 per board and 1,000 per run, and only the roles it filled are embedded and scored again. It follows every pull, a saved agent and a star change, not a vote. A read that brought no text is tried again after `retry_days` (7). `detail` may fill `DETAIL_FIELDS` only, the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing. Completeness is decided by the listing alone. The pull used to read 200 pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered.
|
||||
**Per-role text is read by the enrich run, ranked roles first.** Workday, SuccessFactors, Oracle, Phenom, BambooHR, Jobylon, Rippling and Gem list a role without its text. The pull reads listings only. After scoring, the `enrich` stage reads the role pages of roles without text that an enabled agent ranks in its top `enrich.top` (500) at `enrich.min_score` (0.5) or more, best score first, 200 per board and 1,000 per run, and only the roles it filled are embedded and scored again. It follows every pull, a saved agent and a star change, not a vote. A read that brought no text is tried again after `retry_days` (7). `detail` may fill `DETAIL_FIELDS` only, the columns a later pull with an empty listing value leaves alone; title, url, department and `salary_text` come from the listing. Completeness is decided by the listing alone. The pull used to read 200 pages per board in the order roles were found, so a board of 800 roles took four pulls to reach the roles that mattered. Eightfold, SmartRecruiters and Workable list their roles without text too and are read the same way.
|
||||
|
||||
**robots.txt is decided when a reader is built, not checked at run time.** The builder reads the host's robots.txt, the owner decides per reader whether to read it anyway, and `READS.robots` records the decision and the date. The runtime matcher went with the readers rework. Jobindex is read in full although its robots.txt disallows `page=` and `geoareaid=`; Adzuna's API host disallows `/` for every agent; the owner decided to read its documented API with the account's key anyway, since the rule is aimed at crawlers. It needs a key in the config and has not run live yet. HR Manager's portal disallows everything but the application form; no attached company uses it. The companies once kept off the automatic route by robots.txt (Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy, Umbraco) stay on the Watchlist with reason `robots` until each is decided. The SuccessFactors reader uses `/search/` and `/job/` only.
|
||||
**robots.txt is decided when a reader is built, not checked at run time.** The builder reads the host's robots.txt, the owner decides per reader whether to read it anyway, and `READS.robots` records the decision and the date. The runtime matcher went with the readers rework. Jobindex is read in full although its robots.txt disallows `sort=`, `page=` and `geoareaid=`; Adzuna's API host disallows `/` for every agent; the owner decided to read its documented API with the account's key anyway, since the rule is aimed at crawlers. It needs a key in the config and has not run live yet. HR Manager's portal disallows everything but the application form; no attached company uses it. The companies once kept off the automatic route by robots.txt (Novonesis, Darktrace, Criteo, Snyk, Bang & Olufsen, Pandora, Terma, Zenegy, Umbraco) stay on the Watchlist with reason `robots` until each is decided. The SuccessFactors reader uses `/search/` and `/job/` only. Since then `hrmanager:teknologisk` (the Danish Technological Institute) has been attached and has 7 open roles, so an attached company does use HR Manager; and Novonesis, Darktrace and Criteo are Workday boards, no longer on the Watchlist.
|
||||
|
||||
**Every source is a reader module that declares what it can do.** The 22 hosts and the 3 search providers are modules in `plugins/readers/`, each with `READS = Reads(...)`: board or search, whole or paged listing, where the text is (listing, excerpt, detail, none), how completeness is known (stated, end, whole, never), page size and the host's ceiling, the filters it takes per dimension (none, one value a request, many, host ids through `facets`), whether it states one place, several, or one posting per place, which fields it states and whether its date is trusted, a key, a provisional title, whether an unknown slug answers an empty board, and the robots decision. Capabilities were implied by which functions a module had, and the search providers sat outside the contract with their own getter. `problems()` checks the declaration against the functions at import, and `tests/test_readers_contract.py` runs every module against fixtures, so a new reader that drops a place or a teaser fails there. The contract is the docstring of `plugins/readers/__init__.py`; the types are `_contract.py`.
|
||||
|
||||
@@ -154,7 +154,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**The embedding backend is effectively required.** The intent is a CV or a paragraph per agent, and TF-IDF cannot relate "Kubernetes" to "container orchestration". The model is `intfloat/multilingual-e5-small`, not the English-only `all-MiniLM-L6-v2` roledesk started with: the corpus turned European, and on a sample of the live roles (all open Danish-country roles plus a random sample of the rest, language labelled by a stop-word count) against the "Copenhagen engineering" agent's intent, MiniLM separates engineering from non-engineering roles with an AUC of 0.68 overall but only 0.74 restricted to Danish-country roles, where a Danish-country engineering role beats the rest's non-engineering median only 68% of the time; multilingual-e5-small reaches an AUC of 0.70 overall and 1.00 on Danish-country roles, for about 3x the CPU time. A second multilingual candidate, `paraphrase-multilingual-MiniLM-L12-v2`, measured *worse* than plain MiniLM (AUC 0.60) and was dropped. e5 needs its own "query: " / "passage: " prefixes, which `app/jobs/embed.py` adds by matching the model name, not a separate setting.
|
||||
|
||||
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty.
|
||||
**Summary and tagging are off; `extract` replaces them.** Every profile declares `SummaryPolicy(enabled=False)` and `TaggingPolicy(enabled=False)`. `extract` is a roledesk `ModelTask` on Haiku that turns title and description into the gate columns, and fills only the columns a plugin left empty. Summaries and tags came back as the `summarise` stage and the Summarise button (below); what stays off is desk's own `SummaryPolicy` and `TaggingPolicy`.
|
||||
|
||||
**A score is the weighted sum over the sum of the weights.** The agent's `fit`, `recency` and `affinity` weights and every prefer gate's `weight` form one denominator, so a score stays in 0..1 whatever the weights are.
|
||||
|
||||
@@ -172,15 +172,15 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**`embed` and `score` run in the server process.** newsdesk isolates `embed` in a child process with a timeout. roledesk's run is one bounded pass over open roles, and in-process keeps the embedder replaceable in tests. The cost is torch's memory staying in the server after the first run; isolate it if that becomes the problem.
|
||||
|
||||
**The pull cascade is `pull → embed → score → extract → score → summarise`.** Every stage is always in the chain and a stage that is off (the switch under Runs, else config.yaml) finds nothing, as in newsdesk. The free stages run first: embed and score cost no tokens, and their score is what bounds the two model stages. Extract reads only what ranks, then score runs again so the gates read what it filled. A stage is addressed by its position in the tuple, since `score` is in it twice. Extract reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly.
|
||||
**The pull cascade is `pull → embed → score → extract → score → summarise`.** Every stage is always in the chain and a stage that is off (the switch under Runs, else config.yaml) finds nothing, as in newsdesk. The free stages run first: embed and score cost no tokens, and their score is what bounds the two model stages. Extract reads only what ranks, then score runs again so the gates read what it filled. A stage is addressed by its position in the tuple, since `score` is in it twice. Extract reads `on_ingest` only as a stage of a pull, so with `auto` and no model reachable a pull passes it quietly. Later decisions added `prep` and `enrich`: the cascade is `pull → prep → embed → score → enrich → prep → embed → score → extract → score → summarise`, and `CASCADES` in `app/jobs/cascade.py` is the source.
|
||||
|
||||
**Extract is bounded like summarise: the top roles of each agent, by that agent's own `limits`.** Extract has short answers but would otherwise touch every open role, and the seed boards alone produced 29,000; at Haiku's rates that is tens of dollars for a first pass over roles nobody will look at. Bounded to the ranking, a pull reads a few hundred. `extract_top` is the backstop (default 500) and `extract_min` the real stop (default 0.55, about 300 roles for the starter agent): a fixed count stays flat whatever the boards produce, where a percentage would not. The bounds are per agent rather than config, because one agent may be a wide monitoring search that should read a thousand and another a narrow hunt that wants fifty; one global number fits neither. A batch dealt named uids finishes them whatever their rank, so a running batch is not cut short by a rescore.
|
||||
|
||||
**`extract_batch` and `summarise_batch` are `isolate=True`, with a timeout.** `context.check()` only runs between the calls `modeltask.one` makes, not inside one, so a call that hangs on either backend was otherwise unkillable short of restarting the server; a process boundary is what lets a Kill, or the timeout itself, actually end it. One call is already bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for summarise): a batch still running after that is stuck, not slow. The cost: an isolated job's context has no working `stop`, `task` or `fan_out` (see `pipeline/child.py`'s `context_for`), so a refusal or an account-level failure inside a batch can no longer stop the rest of that task's queued siblings — they run on to the same failure instead of being spared it.
|
||||
**`extract_batch` and `summarise_batch` are `isolate=True`, with a timeout.** `context.check()` only runs between the calls `modeltask.one` makes, not inside one, so a call that hangs on either backend was otherwise unkillable short of restarting the server; a process boundary is what lets a Kill, or the timeout itself, actually end it. One call is already bounded at `agent.timeout`, so the deadline is a few calls' worth (30 minutes for extract, 15 for summarise): a batch still running after that is stuck, not slow. The cost: an isolated job's context has no working `stop`, `task` or `fan_out` (see `pipeline/child.py`'s `context_for`), so a refusal or an account-level failure inside a batch can no longer stop the rest of that task's queued siblings, which run on to the same failure instead of being spared it.
|
||||
|
||||
## The pages
|
||||
|
||||
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models`, `plugins.ats.FUNCTIONS` and `plugins.ats.HOSTS`, so they cannot drift from what the store and the plugin accept). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change.
|
||||
**Each roledesk tab renders an empty shell; `static/roles.js` fills it and drives every action.** Roles, Applications, Agents and Companies extend the boilerplate's `base.html`; the server renders only select options (from `app.models`, `plugins.ats.FUNCTIONS` and `plugins.ats.HOSTS`, so they cannot drift from what the store and the plugin accept). One code path for "show this" beats a server-rendered first page plus a client re-render after every filter change. Only Roles, Applications and Watchlist are tabs now: Agents and Companies are settings pages, and Companies is a section of Sources (Two tabs and a cog, Watchlist is a third tab and Settings are three sub-tabs, below).
|
||||
|
||||
**The feed loads whether or not its enhancements do.** `/api/agents` and `/api/companies` only dim hidden companies, draw the affinity control and pick the viewing agent's score order; the init block used to await both before ever calling `loadFeed`, so a slow or failed one left the feed blank for good behind a silent `catch (err) { return; }`. Now the feed's own request goes out alongside them rather than after, agents and companies fill in and redraw what they touch (the rail, the feed's dimming, the reader's affinity slot) whenever they land, and a feed already fetched with no agent known reloads once the real one is. A failed feed fetch of its own gets an empty state with a retry (`state.feedError`) instead of the same silent blank.
|
||||
|
||||
@@ -190,7 +190,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**The regions are a hand-written list.** Skåne, the Capital Region of Denmark, Zealand, Øresund (the two capital regions), Jutland and Funen, each with its names and towns. On the data of 2026-09-29 no Danish or Skåne town with more than one role was missing. A region elsewhere is added to `REGIONS`.
|
||||
|
||||
**`seniority` is read from the title until extract has read the role.** Extract had never run, so the column was empty on all 33,881 open roles and a seniority filter or gate had nothing to read. `seniority.from_title` matches whole words in English, Danish, Swedish, German and French and gives 43% of open roles a level. "X Manager" is a level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor; intern wins over a boss named in the title. Known mistakes: "VP" is always exec, "Field CTO" and "Chief of Staff" get none, "Account Director" is director. Extract overwrites the title's level, a level a board states wins over both.
|
||||
**`seniority` is read from the title until extract has read the role.** Extract had never run, so the column was empty on all 33,881 open roles and a seniority filter or gate had nothing to read. `seniority.from_title` matches whole words in English, Danish, Swedish, German and French and gives 43% of open roles a level. "X Manager" is a level only where it clearly names someone who runs a team, since a product or account manager is an individual contributor; intern wins over a boss named in the title. Known mistakes: "VP" is always exec, "Field CTO" and "Chief of Staff" get none, "Account Director" is director. Extract overwrites the title's level, a level a board states wins over both. Since the ladder, `roles.seniority` is the first of the model's level, the host's stated level and the title's, in that order (Seniority has three sources, above), so a stated level no longer wins over the model's.
|
||||
|
||||
**Seniority is a ladder of rungs with two tracks.** The first vocabulary was one line (intern to exec) and put a Principal below a Team Lead, and a Staff engineer below any manager, while the owner counts a Principal Solutions Engineer as very senior. From `staff` up, each rung pairs an individual contributor's level with a people lead's, as big tech firms grade them: `staff` with `lead` (a team lead) and `manager`, `principal` (also Senior Staff) with `senior_manager` (also Associate Director, AVP), `distinguished` (Senior Principal, Distinguished, Technical Fellow) with `director` (also Head of), then `exec`. `LADDER` in `app/models.py` gives each word its rung and track, and `roles.seniority` stays one column of words, so gates, `extract` and the host tables work on words. `lead` changed meaning: "Tech Lead", "Lead Engineer" and "Marketing Lead" are `staff`, and only a team lead, supervisor or shift lead is `lead`. In a title, a people level wins over a grade on its own rung, and a grade above it lifts it one rung ("Principal Engineering Manager" is `senior_manager`); a slash is "or" ("Senior/Staff Engineer" is `staff`). Principal sits one rung above Staff, although at some firms it is the top individual grade and at others a senior manager's peer. Migration 25 cleared the model levels the split made ambiguous (staff, lead, manager, director), and prep `VERSION` 2 mapped every open role again. On 2026-09-30 that gave, over 19,230 open roles without copies: staff 1,756, principal 535, distinguished 23, lead 222, manager 736, senior_manager 312, director 883, exec 101, none 9,565.
|
||||
|
||||
@@ -206,11 +206,11 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**A muted company is excluded in SQL, before `LIMIT`.** Filtering after `LIMIT` shipped first and showed nothing once Databricks' 884 freshly pulled roles filled the newest 200 rows and Databricks was muted.
|
||||
|
||||
**`GET /api/roles` serves a row, not a role.** It used to answer `roles.*`, the whole `Role` — `description`, `feed_text`, `raw` and `article_text` included, about 5.5 KB a role on average — for a feed that only ever shows a title, a few badges and an excerpt; a 1000-row restore was several megabytes of text a phone never renders. `Store.roles_page` selects only the fields a row draws (`RoleRow`), with `excerpt` built by `app.models.excerpt` off a `SUBSTR` of `summary` or `description`, never the column whole. Measured on a copy of the live database: a 60-row page went from 392 KiB to 39 KiB and a 1000-row restore from 6.6 MB to 631 KiB, both about a tenth. The reader is unaffected: it already fetched the whole role by uid when it opened one (`GET /api/roles/{uid}`, still the full `Role`); what changed is that the list row it painted from first, before that fetch lands, no longer carries a body to paint with, so the reader now waits on the fetch for the posting text and the original link the same way it always waited for the summary.
|
||||
**`GET /api/roles` serves a row, not a role.** It used to answer `roles.*`, the whole `Role` (`description`, `feed_text`, `raw` and `article_text` included, about 5.5 KB a role on average) for a feed that only ever shows a title, a few badges and an excerpt; a 1000-row restore was several megabytes of text a phone never renders. `Store.roles_page` selects only the fields a row draws (`RoleRow`), with `excerpt` built by `app.models.excerpt` off a `SUBSTR` of `summary` or `description`, never the column whole. Measured on a copy of the live database: a 60-row page went from 392 KiB to 39 KiB and a 1000-row restore from 6.6 MB to 631 KiB, both about a tenth. The reader is unaffected: it already fetched the whole role by uid when it opened one (`GET /api/roles/{uid}`, still the full `Role`); what changed is that the list row it painted from first, before that fetch lands, no longer carries a body to paint with, so the reader now waits on the fetch for the posting text and the original link the same way it always waited for the summary.
|
||||
|
||||
## The phone
|
||||
|
||||
**Two tabs and a cog.** `TABS` in `app/web/pages.py` becomes Roles and Applications; the other six pages (Agents, Companies, Sources, Jobs, Schedules, Messages, Account) move behind a settings cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and six of them were setup or system pages.
|
||||
**Two tabs and a cog.** `TABS` in `app/web/pages.py` becomes Roles and Applications; the other seven pages (Agents, Companies, Sources, Jobs, Schedules, Messages, Account) move behind a settings cog at `/settings/<name>`, because nine tabs wrapped to three rows on a phone and six of them were setup or system pages. The Watchlist became a third tab and the settings pages became three sub-tabs, both below.
|
||||
|
||||
**Watchlist is a third tab: the companies the app cannot read, rated yes or no.** The owner goes through the careers pages of such companies by hand and wanted that list one click from Roles rather than behind the cog, so `TABS` has three entries and the phone bar's tabs are tighter. A company is on it when it has no board subscription and is not known from a search only, or when its board is failing: the rule of the Company boards filter "Not readable yet", read from `GET /api/companies`. The rating is `companies.watch` (migration 15: NULL unrated, 1 on the list, 0 not), global and apart from affinity, which is per agent and ranks roles the app does read. `careers_opened_at` is stamped when the careers link is followed, and On the list sorts by it, never opened first, so a round starts with what is overdue. A row rated under Unrated leaves the list at once. On a phone the search field of Roles is left a few letters wide by the third tab, so while it has the focus the tabs and the queue chip step aside.
|
||||
|
||||
@@ -240,7 +240,7 @@ What was decided for roledesk and the one reason for each. The longer reasoning
|
||||
|
||||
**`taste` is the liked-minus-passed centroid, as in newsdesk.** An agent's votes become one vector, the mean of the liked roles' embeddings minus the mean of the passed ones', and a role's taste term is its cosine to that vector, calibrated like fit, weighted beside fit, recency and affinity. It is the one term an agent learns rather than is told, and it costs no model call.
|
||||
|
||||
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest.
|
||||
**Roles are a feed with a reader, not links out.** `GET /roles/{uid}` renders the feed with its reader open on that role instead of sending the reader to the board, because 29,194 of 29,196 open board roles already have a `description` averaging 6,300 characters, and a search role's short teaser is worth reading in place before deciding whether to fetch the rest. The figure is of 2026-09-28: hosts that list a role without its text now leave 1,974 of 24,003 open roles without a description on 2026-09-30, until `enrich` reads them.
|
||||
|
||||
**Every link built from someone else's data goes through `safeUrl`.** A board, a search result, a discovered careers page and a saved application's `url` are all free text from outside the app; an `http(s)` check before it lands in an `href` is what keeps a stray `javascript:` link or a bare string typed into a field from being clickable. The reader needed this first, for `original_url`, so `safeUrl` moved to `app.js` beside the app's other shared helpers rather than staying roles.js's own; companies.js (`careers_url`), sources.js (a search sample's `url`) and applications.js (`application.url`) now use the same one.
|
||||
|
||||
|
||||
Reference in new issue
Block a user