Prune done work from the docs: fold the ingest review's open findings into open.md and delete it, cut the hosts wave-2 plan to what is unbuilt, drop the smaller-hosts survey
The plugins README no longer describes a run-time robots.txt check or boards that never close past the page cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a44bf5dacb
commit
802656aca4
8 files changed
+70
-472
No files matched your search
+19
-19
@@ -59,14 +59,14 @@ Numbers are from the live database after the pull of 2026-09-29. For the "list,
|
||||
| Where requests go | https on port 443, only to the host names the host module allows; no IP address, no local name; at most 3 redirects, each checked | A slug cannot point a request anywhere else. |
|
||||
| Size cap | 96 MiB per response | A larger answer fails the board. Greenhouse retries once without text and pay; no other host has a fallback. |
|
||||
| Time limits | 10 s to connect, 30 s between bytes, 60 s for the whole request; 15 s per request for the search providers | A slow board fails for this pull and is asked again next pull. |
|
||||
| Pages per board | 100 | A board above it (1,000 roles at Workable or Eightfold, 2,000 at Workday, 5,000 at Teamtailor, 10,000 at SmartRecruiters) is read incomplete: roles on the first 100 pages are stored and new ones found, but no role of that board is ever closed by list. |
|
||||
| Pages per board | 100 | A board above it is swept across pulls: each pull stores what it read and the next continues from the cursor. A role of a swept board closes only when two finished sweeps in a row missed it, so a new role on a later page waits up to one sweep. A read that reaches the host's `ceiling` (Workday 2,000) is not swept and closes nothing; a filter must bring the board under it. |
|
||||
| Wait | 0.5 s before every page after the first and before every per-role read; none between search pages | The list of a Workday board of 1,000 roles spends about 25 s waiting. |
|
||||
| Per-role reads | 200 per board per pull | The rest wait for the next pull. A 404 or 410 counts as read; another failure is retried on a later pull. |
|
||||
| Closing a role | after a complete read, every open role of the board it did not list | An incomplete read closes nothing. A role listed again reopens. A board switched off closes its roles at once. |
|
||||
| Search roles | a complete search closes what it did not list; otherwise a role closes when no search has listed it for 21 days | A search result is never a company's whole list. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize, robots.txt. |
|
||||
| A board that fails | first failure: logged, roles stay open, the Companies list shows "not answering since" | Any error counts: refused host, timeout, HTTP error, parse error, oversize. |
|
||||
| A board that fails for 21 days | all its open roles close at the next pull | The next answer reopens what it lists. |
|
||||
| robots.txt | read before the list for the 14 hosts that name a page: Workday, SuccessFactors, Oracle, Phenom, Eightfold, Jibe, Gem, BambooHR, Pinpoint, Deel, Jobylon, Rippling, HiBob, HR Manager; kept for one day per board | A disallow fails the board with "robots.txt disallows it" and counts toward the 21 days. An unreachable file (5xx, 429, timeout) fails the board for this pull and is asked again next pull. A 4xx means no rules, which allows. `Crawl-delay` is not honoured. The eight API hosts (Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee, Personio, Teamtailor) and the search providers read no robots.txt. |
|
||||
| robots.txt | not read at run time | Decided per reader when it is built and recorded in `READS.robots` with the date (`docs/decisions.md`). `Crawl-delay` is not honoured. |
|
||||
| 429 | first page: the board fails; a later page: the read stops, keeps what it read and is incomplete; a per-role read: the per-role reads of that board stop until the next pull | Nothing is closed on a list cut short by a 429. A search that fails is skipped for the pull. |
|
||||
|
||||
How far a board is read: a pull asks each board over 300 roles for the reach of the search agents that are switched on (`app/jobs/reach.py`): the union of their `any` gates on country, town (as its country) and remote, or every role when one agent can take a role from anywhere. A company set to All countries is asked for the agents' wide reach, the union of their `require` reach, Europe today; a host that asks one country a request reads such a board whole instead of 30 times, unless fetch terms narrow it. A board's own reach, set on the Companies list, wins. A complete filtered read closes the roles outside the reach, except those with a vote or an application.
|
||||
@@ -171,7 +171,7 @@ What no source does:
|
||||
| Target | the subdomain, e.g. `pinpoint:xeneta` |
|
||||
| Companies | 1 board: Xeneta |
|
||||
| List | `https://<sub>.pinpointhq.com/postings.json`, the whole board in one answer; no total, no paging keys |
|
||||
| Requests per pull | 1, plus robots.txt once a day |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, skills and benefits sections; live: 2 of 2 |
|
||||
| Date | none: the list states none and the role page is not read for it; the role counts as posted when first seen |
|
||||
| Place and country | `location.name` (else `city`) plus `province`, one place; country from that line |
|
||||
@@ -187,7 +187,7 @@ What no source does:
|
||||
| Target | the board slug on `jobs.deel.com`, with case, e.g. `deel:klarna` |
|
||||
| Companies | 1 board: Klarna |
|
||||
| List | the HTML page `https://jobs.deel.com/<slug>`, from which the Next.js server payload is read; the whole board in one answer; no total |
|
||||
| Requests per pull | 1, plus robots.txt once a day |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text (`richtextDescription`); live: 96 of 96 |
|
||||
| Date | the publication's `createdAt`, else the posting's; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every `jobLocations` name, joined with "; "; country from that line (91 of 96) |
|
||||
@@ -203,7 +203,7 @@ What no source does:
|
||||
| Target | the subdomain of `careers.hibob.com`, e.g. `hibob:dixa` |
|
||||
| Companies | 1 board: Dixa |
|
||||
| List | `https://<sub>.careers.hibob.com/api/job-ad` with the header `companyIdentifier: <sub>`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1, plus robots.txt once a day |
|
||||
| Requests per pull | 1 |
|
||||
| Description | in the list, full text: description, responsibilities, requirements, benefits; live: 2 of 2 |
|
||||
| Date | `publishedAt`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `site` plus `country`, one place |
|
||||
@@ -272,7 +272,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the career site's host, e.g. `jibe:jobs.booking.com` |
|
||||
| Companies | 1 board: Booking.com |
|
||||
| List | `https://<host>/api/jobs?limit=100&page=<n>`, 100 per request, the most the host serves; states `totalCount` |
|
||||
| Requests per pull | roles / 100, rounded up, plus robots.txt once a day; 2 for Booking.com |
|
||||
| Requests per pull | roles / 100, rounded up; 2 for Booking.com |
|
||||
| Description | in the list, full text: description and qualifications; live: 140 of 140 |
|
||||
| Date | `posted_date`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | `full_location`, several places joined by the host; country 140 of 140 |
|
||||
@@ -288,14 +288,14 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the portal's customer name, e.g. `hrmanager:stsgroup_tr` |
|
||||
| Companies | none attached; Seaborg, the one seed company on it, is on the Watchlist as `robots` |
|
||||
| List | the RSS feed `api.hr-manager.net/JobPortal.svc/<customer>/PositionList/rss/?incads=true`, 100 per request by `skip`; no total, so the read goes on to an empty page |
|
||||
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page, plus robots.txt once a day |
|
||||
| Requests per pull | roles / 100, rounded up, plus 1 for the empty page |
|
||||
| Description | in the list, the whole advertisement |
|
||||
| Date | `pubDate`; not trusted |
|
||||
| Place and country | none: the feed names no place |
|
||||
| Remote | none |
|
||||
| Pay | none |
|
||||
| Found by discovery | yes, from a portal or API link; not probed |
|
||||
| Limitations | The portal's robots.txt on `candidate.hr-manager.net` disallows everything but the application form; the feed's API host has no robots.txt, so the module's check allows it. The owner decided to leave the board unattached. Without `take` the host answers 25 roles. |
|
||||
| Limitations | The portal's robots.txt on `candidate.hr-manager.net` disallows everything but the application form; the feed's API host has no robots.txt. The owner decided to leave Seaborg unattached. Without `take` the host answers 25 roles. |
|
||||
|
||||
## Hosts read as list, then each role
|
||||
|
||||
@@ -306,7 +306,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the career site's host, or host plus a brand path, e.g. `successfactors:careers.vestas.com`, `successfactors:jobs.teradyne.com/MiR` |
|
||||
| Companies | 14 boards: Novo Nordisk, Vestas, Carlsberg |
|
||||
| List | the site's HTML search page `/search/?startrow=<n>`, parsed in its two known layouts; page size is the site's own (10, 15, 25 or 100) and cannot be set; a total where the pagination label states one |
|
||||
| Requests per pull | roles / the site's page size, rounded up, plus up to 200 role pages, plus robots.txt once a day; per-board sizes are not in these notes |
|
||||
| Requests per pull | roles / the site's page size, rounded up, plus up to 200 role pages; per-board sizes are not in these notes |
|
||||
| Description | per role, the `itemprop="description"` block of the role page; live: 1,651 of 3,738 after one pull |
|
||||
| Date | the list's date column where it parses; the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's location, with a two-letter country code turned into the name; the role page's microdata replaces it; country 3,675 of 3,738 |
|
||||
@@ -323,7 +323,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | `<tenant>.<wdN>/<Site>`, site with case, e.g. `workday:lego.wd103/LEGO_External` |
|
||||
| Companies | 13 boards: LEGO Group, Zendesk, Equinor |
|
||||
| List | a POST to `/wday/cxs/<tenant>/<Site>/jobs`, 20 per request by offset (21 or more answers 400); the first page states the total |
|
||||
| Requests per pull | roles / 20, rounded up, plus up to 200 role reads, plus robots.txt once a day; for a board of 1,000 roles 50 + 200 on the first pull |
|
||||
| Requests per pull | roles / 20, rounded up, plus up to 200 role reads; for a board of 1,000 roles 50 + 200 on the first pull |
|
||||
| Description | per role, `jobDescription` from the role's JSON; live: 1,411 of 2,377 after one pull |
|
||||
| Date | the list's "Posted N Days Ago" as a day, possibly one day off; "Posted 30+ Days Ago" and other languages give none; the role's `startDate` replaces it; trusted |
|
||||
| Place and country | the list has `locationsText` ("Copenhagen" or "2 Locations"); the per-role read adds every location and the country. The country therefore arrives with the per-role read: 1,724 of 2,377 after one pull |
|
||||
@@ -340,7 +340,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | `<pod host>/<siteNumber>`, e.g. `oracle:fa-evmr-saasfaprod1.fa.ocs.oraclecloud.com/CX_1` |
|
||||
| Companies | 4 boards: Danske Bank, Nokia, Milestone Systems, Sinch |
|
||||
| List | the REST resource `recruitingCEJobRequisitions`, 200 per request by offset, with secondary places expanded; states `TotalJobsCount` |
|
||||
| Requests per pull | roles / 200, rounded up, plus up to 200 role reads, plus robots.txt once a day; 3 + 200 for Nokia's roughly 600 on the first pull |
|
||||
| Requests per pull | roles / 200, rounded up, plus up to 200 role reads; 3 + 200 for Nokia's roughly 600 on the first pull |
|
||||
| Description | per role: external description, responsibilities and qualifications joined; the list's one-paragraph `ShortDescriptionStr` is not stored; live: 432 of 826 after one pull |
|
||||
| Date | `PostedDate`; the role's `ExternalPostedStartDate` replaces it when earlier; trusted |
|
||||
| Place and country | the primary place plus every secondary one, joined with "; "; country 769 of 826 |
|
||||
@@ -357,7 +357,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the career site's host plus its locale root, e.g. `phenom:www.careers.philips.com/global/en` |
|
||||
| Companies | 3 boards: King, Just Eat Takeaway.com, Philips |
|
||||
| List | a POST to `https://<host>/widgets`, 100 per request by offset; states `totalHits` |
|
||||
| Requests per pull | roles / 100, rounded up, plus up to 200 role pages, plus robots.txt once a day; 9 + 200 for Philips' 865 on the first pull |
|
||||
| Requests per pull | roles / 100, rounded up, plus up to 200 role pages; 9 + 200 for Philips' 865 on the first pull |
|
||||
| Description | per role, the JSON-LD `JobPosting` description of the role page; the list's `descriptionTeaser` is not stored; live: 335 of 1,007 after one pull |
|
||||
| Date | the list's `postedDate`; the role page's `datePosted` replaces it when earlier (Philips' list gave the day of the read for a role posted 2026-09-08); trusted |
|
||||
| Place and country | every place of `multi_location`, joined with "; "; country 975 of 1,007 |
|
||||
@@ -374,7 +374,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | `<tenant host>/<company domain>`, e.g. `eightfold:jobs.ericsson.com/ericsson.com` |
|
||||
| Companies | 1 board: Ericsson |
|
||||
| List | `https://<host>/api/pcsx/search?domain=<domain>&sort_by=timestamp`, newest first, 10 per request whatever `num` asks; states `count` |
|
||||
| Requests per pull | roles / 10, rounded up, at most 100, plus up to 200 `position_details` reads, plus robots.txt once a day; 63 + 200 for Ericsson's 630 on the first pull |
|
||||
| Requests per pull | roles / 10, rounded up, at most 100, plus up to 200 `position_details` reads; 63 + 200 for Ericsson's 630 on the first pull |
|
||||
| Description | per role, `jobDescription`; live: 200 of 630 after one pull |
|
||||
| Date | `postedTs`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location, repeated names dropped, joined with "; "; country 622 of 630 |
|
||||
@@ -391,7 +391,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the board name on `jobs.gem.com`, with case, e.g. `gem:productboard` |
|
||||
| Companies | 2 boards: sennder, Productboard |
|
||||
| List | a GraphQL POST to `jobs.gem.com/api/public/graphql`, the whole board in one answer; no total |
|
||||
| Requests per pull | 1, plus one GraphQL POST per role not yet read (at most 200), plus robots.txt once a day |
|
||||
| Requests per pull | 1, plus one GraphQL POST per role not yet read (at most 200) |
|
||||
| Description | per role, `descriptionHtml`; live: 44 of 44 |
|
||||
| Date | per role only, `firstPublishedTsSec`; not trusted, so the role counts as posted when first seen |
|
||||
| Place and country | every location's city, joined with "; "; `isoCountry` is fetched and not used; country from the line (41 of 44) |
|
||||
@@ -407,7 +407,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the numeric company id, e.g. `jobylon:3143` |
|
||||
| Companies | 2 boards: Einride, Vipps MobilePay |
|
||||
| List | the careers widget's embed page `cdn.jobylon.com/jobs/companies/<id>/embed/v2/`, whose script literal holds the whole board; no total |
|
||||
| Requests per pull | 1, plus one role page on `emp.jobylon.com` per role not yet read (at most 200), plus robots.txt once a day; 1 + 33 for Einride on the first pull |
|
||||
| Requests per pull | 1, plus one role page on `emp.jobylon.com` per role not yet read (at most 200); 1 + 33 for Einride on the first pull |
|
||||
| Description | per role, the JSON-LD description of the role page; live: 33 of 33 |
|
||||
| Date | the list's day ("Sept. 25, 2026"); the role page's `datePosted` replaces it when earlier; trusted |
|
||||
| Place and country | the list's place names, which can be cut short ("Houston +3 more"); the per-role read fills every place; country 32 of 33 |
|
||||
@@ -423,7 +423,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the board slug on `ats.rippling.com`, with case, e.g. `rippling:prisma-careers` |
|
||||
| Companies | 1 board: Prisma |
|
||||
| List | `api.rippling.com/platform/api/ats/v1/board/<slug>/jobs`, the whole board in one array, one entry per role and place, merged by role; no total |
|
||||
| Requests per pull | 1, plus one role record per role not yet read (at most 200), plus robots.txt once a day; 1 + 2 for Prisma |
|
||||
| Requests per pull | 1, plus one role record per role not yet read (at most 200); 1 + 2 for Prisma |
|
||||
| Description | per role, `description.role`; live: 2 of 2 |
|
||||
| Date | per role only, `createdOn`; trusted |
|
||||
| Place and country | the list's place labels, joined with "; "; the per-role read fills every work location |
|
||||
@@ -439,7 +439,7 @@ The reader's "Fetch the posting" button shows for these roles, since their descr
|
||||
| Target | the subdomain, e.g. `bamboohr:configit` |
|
||||
| Companies | 1 board: Configit |
|
||||
| List | `https://<sub>.bamboohr.com/careers/list`, the whole board in one answer; a list shorter than its `totalCount` fails the board rather than being read as whole |
|
||||
| Requests per pull | 1, plus one `/careers/<id>/detail` per role not yet read (at most 200), plus robots.txt once a day; 1 + 2 for Configit |
|
||||
| Requests per pull | 1, plus one `/careers/<id>/detail` per role not yet read (at most 200); 1 + 2 for Configit |
|
||||
| Description | per role; live: 2 of 2 |
|
||||
| Date | per role only, `datePosted`; trusted |
|
||||
| Place and country | the list's city and state; the country arrives with the per-role read |
|
||||
@@ -513,4 +513,4 @@ Jobnet and Jooble were left out: Jobnet's search answers 401 outside its own sit
|
||||
| `key` | 3 | the only listing answers with a key the page's script carries | Maersk, New Work SE, Yousign (Welcome to the Jungle) |
|
||||
| no reason | 9 | on Teamtailor, but the careers link is a marketing site and the career host is not found | Templafy, Queue-it, Telenor, Polestar, IQM, TeamViewer, Tractive, Starship, Factorial |
|
||||
|
||||
What is planned for the `unread` companies and for the gaps in hosts already read (Eightfold's page cap, SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, a country filter in a target) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
|
||||
What is planned for the `unread` companies and for the gaps in hosts already read (SuccessFactors sites of Tryg, DNB and Allegro, Recruitee on a custom domain, Avature for Siemens) is in `docs/plan-hosts-wave-2.md`. What is open without a plan is in `docs/open.md`.
|
||||
@@ -10,7 +10,7 @@ as they are found. The site is case-insensitive for the API (Red Hat's
|
||||
`jobs` and `Jobs` both work) but its robots.txt paths are not, so the slug
|
||||
keeps the case the URL wrote (`KEEP_CASE`).
|
||||
|
||||
Written from live career sites on 2026-09-29 (`docs/plan-hosts/workday.md`).
|
||||
Written from live career sites on 2026-09-29.
|
||||
The listing carries no description, so `detail` reads each role's own JSON.
|
||||
|
||||
Countries and remote are filtered on the server by the tenant's own facets
|
||||
|
||||
Reference in new issue
Block a user